Skip to main content

New Filter Operation Scenarios

In this guide, we will be discussing various rules and scenarios in which Filter operation can be used and how the users can add them to their integration bridge. 

Rules for Using Filter Operations 
 
1. When the Target is an API 
 
Single Filter Condition: 

If a single filter condition is used to filter data and the filtered data is sent to the API as the request body (dropping unfiltered data), Data Aggregation operation is not required.

Multiple Filter Conditions: 
 
If multiple filter conditions are used, and all the filtered data needs to be sent to the API as the request body: 

  1. After applying the first Filter condition, use the API operation to send the filtered data. 

  1. Use the Filter End operation to close the first filter flow. 

  1. Apply the second Filter condition and send the resulting data to the API operation or Target.

2. When the Target is Datalake

General Rule: 
 
Whether using one or more filter conditions, always: 

  1. After applying the first Filter condition, use the Datalake operation to send the filtered data. 

  1. Use the Filter End operation to close the first filter flow. 

  1. Apply the second Filter condition and send the resulting data to the Datalake operation or Target. (if want to use second filter) 

Dropping Unfiltered Data:

After applying the Filter operation, send the filtered data to the Datalake using the Datalake operation or target. 

Retaining All Data:

Use the first Filter operation to filter the data, then send the filtered data to the Datalake using the Datalake operation. After the Filter End operation, apply another Filter operation using the opposite condition of the first filter and send the resulting data to the Datalake using the Datalake operation or target. This ensures both filtered and unfiltered data are sent to the Database.

3.When the Target is a Database

General Rule: 
 
For one or more filter conditions, always: 
   

  1. Use Append ops to add a dummy key for aggregation 

  1. Aggregate the data using the Data Aggregation operation. 

  1. Apply the SingleLine to Tuple operation. 

  1. Send the data to the Database. 

Dropping Unfiltered Data: 

After the filter operation, append a dummy key to serve as a group-by key in the Data Aggregation operation. This ensures only filtered data is sent to the Database. 
 
Retaining All Data:

Apply a Filter operation and send the filtered data to the Database after grouping it using Data Aggregation and converting it into a tuple. After the Filter End operation, apply another Filter using the opposite condition and send the remaining data to the Database in the same way. This ensures both filtered and unfiltered data are sent to the Database. 

When Ending a Filter Condition:

Always use the Filter End operation once data transformation and data transfer are completed.

The Filter End operation finalizes the active filter condition. After Filter End, no further filtering is applied, and the original data flow is restored.

Important:
If more than one filter condition is used within the same flow, applying Filter End after each filter block is mandatory to avoid unintended data filtering and to ensure correct data processing.

 

To show these scenarios, a dataset of 12 records has been used in multiple ways. The dataset consists of records: 

Filter1.png



Scenario 1: With Target as Datalake and 1 filter condition 

In this scenario, one filter condition will be applied on the data coming from the source and then the filtered record will be sent to Datalake. 


Filter2.png

Operations Used: 

Filter3.png

  • To send the records to Datalake, we apply SingleLine to Multiline operation as the data coming from source is a SingleLine JSON data.

  • Next, the filter condition is applied as per the requirement (Here, Filter condition applied is ID not equal to 2, which means records associated with ID 2 should not be sent to target) and now send the records to the target. 


Example: Data in Source 

[
{ 
  "id": 1, 
  "email": "george.bluth@reqres.in", 
  "first_name": "George", 
  "last_name": "Bluth" 
}, 
{ 
  "id": 2, 
  "email": "janet.weaver@reqres.in", 
  "first_name": "Janet", 
  "last_name": "Weaver" 
}, 
{ 
  "id": 3, 
  "email": "emma.wong@reqres.in", 
  "first_name": "Emma", 
  "last_name": "Wong" 
}
]

 Filter Condition:  

 data['id'] !=2

Output in Datalake: 

id 

email 

first_name 

last_name 

1 

george.bluth@reqres.in 

George 

Bluth 

3 

emma.wong@reqres.in 

Emma 

Wong 





Scenario 2: With Target as API and 1 filter condition 

In this scenario, one filter condition will be applied to the data coming from the source and then the filtered record will be sent to the target which is API. 

Filter4.png


Filter5.png


  • To send the records to the target (API), we apply SingleLine To Multiline operation as the data coming from source is a SingleLine JSON data.

  • Next, the filter condition is applied as per the requirement (Here, Filter condition applied is ID not equal to 3, which means records associated with ID 3 should not be sent to target) and now send the records to the target. 

Note: Further operations can be used in the integration bridge depending on the Target and data type. 

Example: Data in Source 

[
{ 
      "id": 1, 
      "email": "george.bluth@reqres.in", 
      "first_name": "George", 
      "last_name": "Bluth" 
}, 
{ 
      "id": 2, 
      "email": "janet.weaver@reqres.in", 
      "first_name": "Janet", 
      "last_name": "Weaver" 
}, 
{ 

      "id": 3, 
      "email": "emma.wong@reqres.in", 
      "first_name": "Emma", 
      "last_name": "Wong" 
}
]

Filter Condition: 

data['id'] !=3

Output in Target: 

{ 
"id": 1, 
"email": "george.bluth@reqres.in", 
"first_name": "George", 
"last_name": "Bluth" 

}, 
{ 
"id": 2, 
"email": "janet.weaver@reqres.in", 
"first_name": "Janet", 
"last_name": "Weaver" 
}


Scenario 3: With Target as Database and 1 filter condition 

In this scenario, one filter condition will be applied to the data coming from the source and then the filtered record will be sent to the Database. 




Operations Used: 

Filter7.png


  • To send the records to database, we apply SingleLine to Multiline operation on the data coming from source, as it is a SingleLine JSON data.

  • Next, the filter condition is applied as per the requirement (Here, Filter condition applied is ID not equal to 1, which means records associated with ID 1 should not be sent to target).

  • Once the filter operation is applied, create a new column using Append, which will help to aggregate data.

  • To send data to the Database, a Tuple must be created. Before creating the Tuple, aggregate all required columns using the Data Aggregation operation, then use the key created by the Append operation and store the aggregated data under the Tuple key.

  • Use SingleLine to Tuple and send the records to the Database. 


Example: Data in Source 

[
{ 
"id": 1, 
"email": "george.bluth@reqres.in", 
"first_name": "George", 
"last_name": "Bluth" 
}, 

{ 
"id": 2, 
"email": "janet.weaver@reqres.in", 
"first_name": "Janet", 
"last_name": "Weaver" 
}, 

{ 
"id": 3, 
"email": "emma.wong@reqres.in", 
"first_name": "Emma", 
"last_name": "Wong" 
}
]

Filter Condition:  

data['id'] !=1

Output in Database: 

id 

email 

first_name 

last_name 

2 

janet.weaver@reqres.in 

Janet 

Weaver 

3 

emma.wong@reqres.in 

Emma 

Wong 


Scenario 4: With Target as Datalake but more than 1 filter condition 

In this scenario, more than one filter condition will be applied to the data coming from the source and then the filtered record will be sent to the Datalake. 

Filter8.png

Operations Used: 

Filter9.png

  • To send the records to Datalake, we will use the operation SingleLine to Multiline as your records are in SingleLine JSON data.

  • Apply the Filter condition, send the filtered records to the Datalake, and end the filter using the Filter End operation.

  • Apply another Filter condition and send the resulting data to the Datalake (using the same or a different index). 


Example: Data in Source 

[
{ 
"id": 1, 
"email": "george.bluth@reqres.in", 
"first_name": "George", 
"last_name": "Bluth" 
}, 
{ 
"id": 2, 
"email": "janet.weaver@reqres.in", 
"first_name": "Janet", 
"last_name": "Weaver" 
}, 
{ 
"id": 4, 
"email": "eve.holt@reqres.in", 
"first_name": "Eve", 
"last_name": "Holt" 
}, 
{ 
"id": 5, 
"email": "charles.morris@reqres.in", 
"first_name": "Charles", 
"last_name": "Morris"
}
]

Condition 1

  • Filter Condition:  
data['id’] == 1

Condition 2 

  • Filter Condition:  
data['id’] == 2

Output in Datalake: 

id 

email 

first_name 

last_name 

1 

george.bluth@reqres.in 

George 

Bluth 

2 

janet.weaver@reqres.in 

Janet 

Weaver 


Scenario 5: With Target as Database but more than 1 filter condition 

In this scenario, more than one filter condition will be applied to the data coming from the source and then the filtered record will be sent to the Database. 

Filter10.png

Operations Used: 

Filter11.png


  • To send the records to Database, we will use the operation SingleLine to Multiline as your records are in SingleLine JSON data.

  • Apply the filter conditions along with the Append operation to add a one dummy key-value pair as this will be used to aggregate filtered records

  • Now, aggregate filtered records using the dummy key.

  • To send the records to the Database, create a tuple using SingleLine to Tuple and send the records to the target and end the filter using the Filter End operation.

  • Now, again apply filter condition along with the Append operation to create a one dummy key-value pair.

  • Now, aggregate all the records using the newly created dummy key and create tuple using SingleLine to Tuple and send the records to the target (same or different table). 

Example: Data in Source 

[
{ 
"id": 1, 
"email": "george.bluth@reqres.in", 
"first_name": "George", 
"last_name": "Bluth" 
}, 
{ 
"id": 2, 
"email": "janet.weaver@reqres.in", 
"first_name": "Janet", 
"last_name": "Weaver" 
}, 
{ 
"id": 4, 
"email": "eve.holt@reqres.in", 
"first_name": "Eve", 
"last_name": "Holt" 
}, 
{ 
"id": 5, 
"email": "charles.morris@reqres.in", 
"first_name": "Charles", 
"last_name": "Morris" 
}
]

Condition 1 

  • Filter Condition:  
data['id’] == 2

 
Condition 2 

  • Filter Condition:  
data['id’] == 4


Output in Database: 

id 

email 

first_name 

last_name 

2 

janet.weaver@reqres.in 

Janet 

Weaver 

4 

eve.holt@reqres.in 

Eve 

Holt 

Scenario 6: With Target as API but more than 1 filter condition 

In this scenario, more than one filter condition will be applied to the data coming from the source and then the filtered record will be sent to the target (API). 

Filter12.png


Operations Used:

Filter13.png


  • To send the records to API, we will use the operation Singleline to Multiline as your records are in singleline JSON data.

  • Filter ops is used with 1st filter condition then send to API with filter_ends

  • Filter ops is used with 2nd filter condition or opposite of 1st filter to send to Same API target

Note: Other required operations can also be used in the integration process based on the target and data type. 

Example: Data in Source

[
{ 
"id": 1, 
"email": "george.bluth@reqres.in", 
"first_name": "George", 
"last_name": "Bluth" 
}, 
{ 
"id": 2, 
"email": "janet.weaver@reqres.in", 
"first_name": "Janet", 
"last_name": "Weaver" 
}, 
{ 
"id": 3, 
"email": "emma.wong@reqres.in", 
"first_name": "Emma", 
"last_name": "Wong" 
}, 
{ 
"id": 4, 
"email": "eve.holt@reqres.in", 
"first_name": "Eve", 
"last_name": "Holt" 
}
]

Condition 1 

  • Filter Condition:  

 
Condition 2

  • Filter Condition:  
 data['id'] == 2

Output in API: 

{ 
"id": 2, 
"email": "weaver.janet@reqres.in", 
"first_name": "Janet", 
"last_name": "Weaver" 

}, 
{ 
"id": 4, 
"email": "eve.holt@reqres.in", 
"first_name": "Holt", 
"last_name": "Eve" 
}
Scenario 7: With more than 1 filter condition and multiple Targets

Note:
When multiple targets are configured, the filter logic remains the same regardless of the order in which targets are executed.

To avoid repeating scenarios that differ only by target order, this guide documents the following representative cases:

  • Case A: Database and Datalake

  • Case B: Database and API

  • Case C: API and Datalake

The Datalake → Database flow is not documented separately, as it follows the same filter and data-handling logic as Case A.


Case A: Database and Datalake as Targets 

In this scenario, more than one filter condition will be applied to the data coming from the source and then the filtered record will be sent to multiple targets which are Database and Datalake.

Filter14.png

Operations Used:

Filter15.png

In this scenario, we have two targets, Database and Datalake. Therefore, we will be sending the records to Database, followed by Datalake. 

  • After applying the filter conditions and transforming the data based on the requirements and adding a dummy key to our filtered data.

  • To send data to Database as target, We have to filter the data now with the dummy key-value pair condition after this we will Aggregate the data based on the meta column added using Data aggregation operation and followed by Singleline to Tuple Operation and insert the data to Database.

  • After sending the data we will use a Filter Ends operation.

  • To send data to Datalake as target, We have to filter the data and ingest the data to Datalake. 

Note: This scenario applies even if the execution order of Database and Datalake is reversed.

Note:  In 1st filter we are using append ops to append a dummy key for aggregation, In 2nd filter we can send data directly to datalake. Other required operations can also be used in the integration process based on the target and data type. 

Example: Data in Source

[
{
"id": 1, 
"email": "george.bluth@reqres.in", 
"first_name": "George", 
"last_name": "Bluth" 
}, 
{ 
"id": 2, 
"email": "janet.weaver@reqres.in", 
"first_name": "Janet", 
"last_name": "Weaver" 
}, 
{ 
"id": 3, 
"email": "emma.wong@reqres.in", 
"first_name": "Emma", 
"last_name": "Wong" 
}, 
{ 
"id": 4, 
"email": "eve.holt@reqres.in", 
"first_name": "Eve", 
"last_name": "Holt" 
}
]

Condition 1 

  • Filter Condition:  
 data['id'] ==4

 
Condition 2

  • Filter Condition:  
 data['id'] == 2


Output in Database: 

id 

email 

first_name 

last_name 

4 

eve.holt@reqres.in 

Holt 

Eve 

 

Output in Datalake: 

id 

email 

first_name 

last_name 

2 

weaver.janet@reqres.in 

Janet 

Weaver 


Case B: Database and API as Target 

In this scenario, more than one filter condition will be applied to the data coming from the source and then the filtered record will be sent to multiple targets which are Database and API. 

Filter18.png

Operations Used: 

Filter19.png

In this scenario, we have two targets, Database and API. Therefore, we will be sending the records to Database, followed by API. 

  • After applying the filter conditions and transforming the data based on the requirements and adding a dummy key to our filtered data.

  • To send data to Database as target, We have to filter the data now with the dummy key-value pair condition after this we will Aggregate the data based on the meta column added using Data aggregation operation and followed by Singleline to Tuple Operation and insert the data to Database.

  • After sending the data we will use a Filter Ends operation.

  • To send data to API as target, We have to filter the data and ingest the data to API. 

Note: This scenario applies even if the execution order of Database and API is reversed.

Note:  In 1st filter we are using append ops to append a dummy key for aggregation, In 2nd filter we can send data directly to API. Other required operations can also be used in the integration process based on the target and data type. 


Example: Data in Source

[
{ 
"id": 1, 
"email": "george.bluth@reqres.in", 
"first_name": "George", 
"last_name": "Bluth" 
}, 
{ 
"id": 2, 
"email": "janet.weaver@reqres.in", 
"first_name": "Janet", 
"last_name": "Weaver" 
}, 
{ 
"id": 3, 
"email": "emma.wong@reqres.in", 
"first_name": "Emma", 
"last_name": "Wong" 
}, 
{ 
"id": 4, 
"email": "eve.holt@reqres.in", 
"first_name": "Eve", 
"last_name": "Holt" 
} 
]

Condition 1 

  • Filter Condition:  
 data['id'] ==4

 
Condition 2

  • Filter Condition:  
 data['id'] == 2


Output in Database: 

id 

email 

first_name 

last_name 

4 

eve.holt@reqres.in 

Holt 

Eve 

Output in API: 

{ 
"id": 2, 
"email": "janet.weaver@reqres.in", 
"first_name": "Janet", 
"last_name": "Weaver" 
}


Case C: Datalake and API as Target  

In this scenario, more than one filter condition will be applied to the data coming from the source and then the filtered record will be sent to multiple targets which are API and Datalake.

Filter20.png

 Operations Used:

Filter21.png


In this scenario, we have two targets, API and Datalake. Therefore, we will be sending the records to Database, followed by Database. 

  • To send data to API as target, We have to filter the data and ingest the data to Datalake.

  • Then use filterends and start the 2nd Filter.

  • Send data to Datalake as target

  • After sending the data we will use a Filter Ends operation. 

Note: This scenario applies even if the execution order of Datalake and API is reversed.

Note:  Other required operations can also be used in the integration process based on the target and data type. 

Example: Data in Source 

[
{ 
"id": 1, 
"email": "george.bluth@reqres.in", 
"first_name": "George", 
"last_name": "Bluth" 
}, 
{ 
"id": 2, 
"email": "janet.weaver@reqres.in", 
"first_name": "Janet", 
"last_name": "Weaver" 
}, 
{ 
"id": 3, 
"email": "emma.wong@reqres.in", 
"first_name": "Emma", 
"last_name": "Wong" 
}, 
{ 
"id": 4, 
"email": "eve.holt@reqres.in", 
"first_name": "Eve", 
"last_name": "Holt" 
}
]


Condition 1 

  • Filter Condition:  
 data['id'] ==4

 
Condition 2

  • Filter Condition:  
 data['id'] == 2

Output in API: 

{ 
"id": 4, 
"email": "eve.holt@reqres.in", 
"first_name": "Holt", 
"last_name": "Eve" 
}


Output in Datalake: 

id 

email 

first_name 

last_name 

2 

weaver.janet@reqres.in 

Janet 

Weaver 


Back to top