Advanced Settings

The Advanced Settings section is available under the Summary tab of the Integration Bridge page. It provides additional configuration options for fine-tuning the execution of data streaming jobs.

Advance Setting Feature has the following characteristics:

Select Worker Capacity: Users can select the number of records workers will process.
Number of Workers: This option allows users to select how many workers will work on their data processing. More the number of workers, lesser the time of processing.
Source Response Wait Time: Users can set how long the system should wait for source data to arrive before moving forward.
Log Stream Size per Line (in KB): This option allows the user to specify the amount of data that should be displayed in each log line.
Log Rotation Size (in MB): This option allows users to configure the maximum size of each log file. Once the defined size is reached, the log file is compressed into a zip file, and a new active log file is created.

Example:
If you have a large dataset, say, 1 million records, you can speed up processing by enabling Advanced Settings from the Summary tab in the Integration Bridge.
In this example, the chunk size is 50,000 records, meaning the source system sends 50,000 records per processing cycle.

You can configure the following settings based on your requirements:

Streaming Worker Capacity: 10000
Number of Workers: 4
Source Response Time: 1000 (in milliseconds)
Log Stream Size per Line: 300 KB
Log Rotation Size: 50 MB

With this configuration:

The 50,000-record chunk will be split evenly across 4 workers.
Each worker will process 10,000 records in parallel.
Log lines are capped at 300 KB each.
Once the log file reaches 50 MB, it is automatically zipped, and a new file is created.
This leads to faster data processing and optimized storage.

Activate Parallel Processing

The Activate Parallel Processing allows data to be processed simultaneously for faster and more efficient execution.

Activate Parallel Processing feature has the following characteristics:

Improves Performance for Large Datasets: Efficiently handles high volumes of data by distributing workload across multiple CPU cores.
Minimizes Processing Bottlenecks: Reduces delays caused by single-threaded data operations.
Ideal for Batch Jobs or Heavy Transformations: Especially useful in scenarios where complex data transformations or bulk operations are involved.

Log Stream Size per Line Parameter

This feature allows users to set the maximum size of each log line directly in the eZintegrations UI. This helps control how much information each log line holds, reducing delays caused by very large log entries during data pipeline execution.

Log Stream Size per Line Feature has the following characteristics:

Helps manage logs efficiently in high-volume data integration environments.
Prevents logging from becoming a bottleneck when large data sets pass through the pipeline.
Allows users to configure and limit the size of each log line via the UI.
Reduces delays in file writing caused by overly large log entries.
Helps prevent excessive memory usage during logging.
Improves readability and maintainability of log files for easier debugging.

Note: Log Stream Size per Line should not be less than 20KB.

Log Rotation Size (in MB)

This feature allows users to configure the size of each log file directly from the UI. Users can define the threshold at which a log file should be compressed into a zip file.

Once the specified size is reached, the log file is zipped, and a new active log file is created.
The zipped content will still be reflected in the dashboard for monitoring.
When downloading logs, only the most recent (active) log file is downloaded, while older files are available in compressed format.
By specifying a smaller size, users can optimize storage and improve performance by ensuring older logs are automatically zipped.