Skip to main content

Data Source - Bizintel360 Data Lake

Bizintel360 Data Lake is a search engine based NO-SQL database owned by Bizdata. Users can ingest petabyte, zettabyte, yottabyte  of records both structured and unstructured for Analytics, Storage, Machine Learning and deep learning.

Bizintel360 Data Lake Source is a connection pool in eZintegrations platform to retrieve data in JSON format.

Response from Bizintel360 Data Lake source is stored key `bizdata_dataset_response` . If you are using Single Line to Multiple Operations as a next operation in your pipeline then the `Chop key` will have value as 

['bizdata_dataset_response']

Bizintel360 Data Lake Source have following Parameters :

Data Lake Version : Data Lake Version is the Data Lake Name and its version assigned to the organization by Bizdata

Index / Table Name: Index or Table name that you want to retrieve data from Data Lake. For the List of table name or index, check `Datalake` section in Bizintel360 Visualization product.

Pagination Wait Time : By default it is 2m, where m is minute. How long need to wait for next page. If the response of Table/Index is very high ( having 100+ keys in a single record) then try increasing the `Pagination Wait Time'. In general `2m` is sufficient to stream data. Use this when you have high network traffic congestion.

Can also use `h` for hours and `s` for seconds

Timeout : By default it is `2m`, where m is minute. In general `2m` is high enough to get response from Bizintel360 Data Lake. Increase this when response from Bizintel360 Data Lake slow. This can happen when the Data Lake Cluster size is small. Reach Bizdata support team to see increase in cluster size of Bizintel360 Data Lake.

Can also use `h` for hours and `s` for seconds

Size : By default it is `1000`. Size is number of streaming record count from Bizintel360 Data Lake source. The source will stream the records inside pipeline in size of 1000 chunks and move to operations and finally to Data Target. This can be increased to max of 10,000 records in case of use case like one-time historical data loads.

Query : JSON Body based query to retrieve data from tables/index of Bizintel360 Data Lake.

Get all the Records from a table

{ "query": { "match_all": {} } }

Get all Records with specific columns from a table

{
  "_source": ["store_number", "customer_number"],
  "query": {
    "match_all": {}
  }
}

Get Specific column from a table

{
  "query": {
    "match": {
      "employee_id":130
    }
  },
   "_source": {
        "includes": [ "employee_id", "employee_name" ]
      }
}

Get Specific Columns and Filters from a table

{"size":50,"sort":[{}],"_source":["Project","title","Assigned To","Priority","Created By","createdDateTime","dueDateTime"],"query":{"bool":{"must":[{"query_string":{"query":"*"}},{"query_string":{"query":"Project:\"Project ABC\" AND Priority:[* TO *] AND NOT percentComplete:100"}},{"bool":{"should":[]}}],"must_not":[]}}}
Back to top