Data Transformation

Upper Case

Description

This operations helps to convert a given key value into Upper Case from any JSON Dataset. This operations take place while data is in-flight

Number of Parameters : 1

Parameter : Uppercase

Provide comma separated list of keys in double quotes to convert the defined key values into Upper Case

Below is a example where we are converting the value of first_name and last_name into Upper Case

"first_name", "last_name"

Lower Case

Description

This operations helps to convert a given key value into Lower Case from any JSON Dataset. This operations take place while data is in-flight

Number of Parameters : 1

Parameter : Lowercase

Provide comma separated list of keys in double quotes to convert the defined key values into Lower Case

Below is a example where we are converting the value of first_name and last_name into Lower Case

 "first_name", "last_name"

Data Type

Description

Data type operation can convert any key's value into it's data type like 

1. String type Boolean into Boolean data type 
2. String type Float into Float  data type
3. String type Integer into Integer data type 
4. String type Datetime into Datetime data type

Number of Parameters : 4

Parameter : Boolean

A string type Boolean can be converted into Boolean data type .

String type Boolean examples: "True","False","0","1" etc.

Booleans datatype do not contain double quotes.

Below is a example where we are converting the values of test_passed key into Boolean data type .

"test_passed"

Parameter : Float

A string type Float can be converted into Float data type 

Below is a example where we are converting the value of Amount into JSON data type float.

"Amount"

Parameter : Integer

A string type Integer can be converted into Integer data type. 

Below is a example where we are converting the value of Quantity into JSON data type Integer.

"Quantity"

Parameter : Date Time

A string type Date Time can be converted into Date Time data type . 

From date format : The date/datetime format in which the user is defining their date/datetime.

To date format : The date/datetime format in which the user wants to convert their date/datetime.

Example: ["key_name","From date format","To date format"]

Below is a example where we are converting From date format to  To date format inside JSON data type.

"startweekdate1","%Y-%m-%d %H:%M:%S.%f%z","%Y-%m-%d %H:%M:%S","startweekdate2","%Y-%m-%d %H:%M:%S","%Y-%m-%d"

Code	Example	Description
%a	Sun	Weekday as locale’s abbreviated name.
%A	Sunday	Weekday as locale’s full name.
%w	0	Weekday as a decimal number, where 0 is Sunday and 6 is Saturday.
%d	08	Day of the month as a zero-padded decimal number.
%-d	8	Day of the month as a decimal number. (Platform specific)
%b	Sep	Month as locale’s abbreviated name.
%B	September	Month as locale’s full name.
%m	09	Month as a zero-padded decimal number.
%-m	9	Month as a decimal number. (Platform specific)
%y	13	Year without century as a zero-padded decimal number.
%Y	2013	Year with century as a decimal number.
%H	07	Hour (24-hour clock) as a zero-padded decimal number.
%-H	7	Hour (24-hour clock) as a decimal number. (Platform specific)
%I	07	Hour (12-hour clock) as a zero-padded decimal number.
%-I	7	Hour (12-hour clock) as a decimal number. (Platform specific)
%p	AM	Locale’s equivalent of either AM or PM.
%M	06	Minute as a zero-padded decimal number.
%-M	6	Minute as a decimal number. (Platform specific)
%S	05	Second as a zero-padded decimal number.
%-S	5	Second as a decimal number. (Platform specific)
%f	000000	Microsecond as a decimal number, zero-padded to 6 digits.
%z	+0000	UTC offset in the form ±HHMM[SS[.ffffff]] (empty string if the object is naive).
%Z	UTC	Time zone name (empty string if the object is naive).
%j	251	Day of the year as a zero-padded decimal number.
%-j	251	Day of the year as a decimal number. (Platform specific)
%U	36	Week number of the year (Sunday as the first day of the week) as a zero-padded decimal number. All days in a new year preceding the first Sunday are considered to be in week 0.
%-U	36	Week number of the year (Sunday as the first day of the week) as a decimal number. All days in a new year preceding the first Sunday are considered to be in week 0. (Platform specific)
%W	35	Week number of the year (Monday as the first day of the week) as a zero-padded decimal number. All days in a new year preceding the first Monday are considered to be in week 0.
%-W	35	Week number of the year (Monday as the first day of the week) as a decimal number. All days in a new year preceding the first Monday are considered to be in week 0. (Platform specific)
%c	Sun Sep 8 07:06:05 2013	Locale’s appropriate date and time representation.
%x	09/08/13	Locale’s appropriate date representation.
%X	07:06:05	Locale’s appropriate time representation.
%%	%	A literal '%' character.

Append

Description: 

Append operation adds a new key on the fly with it's value as dynamic value or static value. This will add key and it's value for each record.

Number of Parameters : 1

Parameter : Append

By adding new elements to the end of an existing data structure, the append operation can help to extend or modify the data structure in a flexible and efficient way.

For dynamic value the user can use {%column_name%} where column_name is the incoming column in the data pipeline.

For dynamic Integer value use {%^column_name^%} where column_name is the incoming column in the data pipeline.

Below is an example where we are adding a new key in-flight with its fixed/static value.

"export_flag_y":"Y","export_flag_p":"P"

Below is an example where we are adding a new key whose value is Concatenation of two keys value.

"concatenate_key_name":"{%ORDERNUMBER%}{%ORDER_TYPE%}"

Below is an example where we are adding a new key whose value is Concatenation of two keys value separated by a String Pipe (|).

"concatenate_key_name":"{%ORDERNUMBER%}|{%ORDER_TYPE%}"

Below is an example where we are adding an array with key's value with dynamic values to the keys. 

"array_key": [{"transportmode":"{%MODE_OF_TRANSPORT%}","orderType": "{%ORDER_TYPE%}"}]

Below is an example where we are adding an array with key's value with dynamic values to the keys. We can append multiple keys at a time.

"orderLines1":"{%orderLines%}","bill_to":[{"city": "ALTADENA","contactName": "None"}],"array_key":[{"transportmode": "{%MODE_OF_TRANSPORT%}","orderType": "{%ORDER_TYPE%}"}]

Below is an example where we are adding an object with key's value with dynamic values to the keys. We can append multiple keys at a time.

"keyname": {
                "id": "{%id%}",
                "email": "{%email%}",
                "first_name": "{%first_name%}"
            }

Below is an example for a key whose value is a dynamic Integer value and boolean value

In below example the price key was integer and status_flag key was boolean before using Append operation and after using Append operation the data type remains same with the below sprintf feature

"price":"{%^stock_price^%}",
"status_flag":"{%^status_flag^%}",

Below is an example of creating a sentence using the Append operation

"neural_field":"The system facilitates comprehensive tracking of product changes, capturing details from the initial problem identification (Product ID:{%^PRODUCTID^%}, Change Request ID:{%^QCR_ID^%},Change Request Date:{%^CHANGE_REQUEST_DATE^%}& Time:{%^CHANGE_REQUEST_TIME^%}, Issue Description:{%^ISSUE_DESCRIPTION^%}) through to the final verification (Validation Status:{%^VALIDATION_STATUS^%}, Date:{%^VALIDATION_DATE^%} & Time:{%^VALIDATION_TIME^%})."

Note: For creating sentence use this sprintf {%^ISSUE_DESCRIPTION^%} only.Rest of the sprintf syntax and functionality is same as before.

Title Case

Description:

Title Case operation helps in the converting a given key's value into title case. 

Number of Parameters : 1

Parameter : Title Case

Provide comma separated list of keys in double quotes to convert the defined key values into Title Case

Below is an example where we are converting the values of amount and first_name into Title Case.
 

"amount","first_name"

Data Extractor

Description: 

Data Extractor operation is designed to extract specific data from JSON response. 

Number of Parameters : 2

Parameter : Data Extractor

Data Extractor is used to extract keys and its value from a JSON response.

Parameter :Data Extractor Keys

This helps to provide user defined keys. If left blank it will auto generate keys.

Below is an example where we are extracting the values of access_token and feedDocumentId.

"['access_token']","['feedDocumentId']"

Trim 

Description: 

Trim operation helps in removing unnecessary parts from the given key's value as defined by the user.

Number of Parameters : 1

Parameter : Trim Key

Provide comma separated list of keys in double quotes to trim the defined key values.

Below is an example where we will trim the defined key value first_name

"first_name"

JSON to String

Description: 

JSON to String operation is used to convert JSON into a String.

Number of Parameters : 1

Parameter : JSON to String

Provide comma separated list of keys in double quotes to convert the defined key's value from JSON to String.

Below is an example where we are converting the values of key1 and key2 from JSON to String

"key1","key2"

String to JSON

Description: 

String to JSON operation is used to convert a String into a JSON . 

Number of Parameters : 1

Parameter : String to JSON

Provide comma separated list of keys in double quotes to convert the defined key's value from String to JSON

Below is an example where we are converting the value of key1 and key2 from JSON to String

"key1","key2"

JSON to XML

Description: 

JSON to XML operation helps to convert JSON object or value into XML.

Number of Parameters : 2

Parameter : Key Data

Key Data converts the provided key's value from JSON to XML.

Below is an example where we are converting the value of product_data_response from JSON to XML.

product_data_response

 

Parameter : Response Key

Response Key holds the converted XML value under the specified key name which is easy to access.

Below is an example of the key name data_response which will hold the converted XML value, 

data_response

XML to JSON

Description: 

XML to JSON is used to convert XML into JSON. 

Number of Parameters : 2

Parameter : Get key

Get Key converts the provided Key's value from XML to JSON
Below is an example where we are converting the value of product_data_response from XML to JSON.

product_data_response

Parameter : Response key

Response Key take note of the converted value under the specified key name which is easy to access.
Below is an example of the key name data_response which will hold the converted JSON value, 

data_response

Base64 Encoding

Description: 

Base64 operation is used to convert some specific key to base64 encoded, the user can give multiple 
keys if they require.

Number of Parameters : 1

Parameter : Base64 Encode

Used to encode the values of given key into base64.

Below is an example where we are encoding the value of email into Base64.

"email"

Base64 Decoding

Description: 

Base64 decoding operation is used to convert a Base64-encoded string back to its original data format, the user can give multiple 
keys if they require.

Number of Parameters : 1

Parameter : Base64 Decode

Below is an example where we are decoding the Base64 encoded value of email back to original data.

"email"

Generate Array Sequence Number

Description:

Generate Array Sequence Number operation helps in generating sequence number for each row.

Number of Parameters : 2

Parameter : Sequence Key

It is the key name in which the single line data is present whose sequence the user needs to give.
Below is an example of the key_name which holds the single line data.

key_name

Parameter : Sequence Number Key

It is the name of the new key for sequence
Below is an example of the key name DATA which will store the sequence number.

DATA

Send Keys top of Root

Description:

Send Keys top of root operation helps in bringing the given nested key's value to the top of root.

Number of Parameters : 1

Parameter : Column to Root

Provide comma separated key name in double quotes to specify the key's value.

Below is an example where we are giving the nested key as key_name whose value we want to bring at the top of the root.

"key_name"

Today Timestamp

Description:

Today Timestamp operation adds a new on the fly with the value of today's date/ date time as specified by the user.

Number of Parameters : 2

Parameter : Date Format

Date Format is used to pass the required format.
Below is an example of a datetime format which can be modified according to user's need.

%Y-%m-%dT%H:%M:%S.%f%z

Parameter : Datetime Key

Datetime Key is the name of key in which the user wants to save their date/ datetime

Below is an example of key name dl_insert_date which will add a new key on the fly with date format value.

dl_insert_date

Round

Description:

Round operation is used to reduce a decimal number to a specific number of decimal places, where the numbers need to be rounded off.

Number of Parameters : 2

Parameter : Round Keys

Round Keys is used to access the specific key that needs to be rounded off, define by the user.
Below is an example of the key name's float_value and int_value whose value we want to round off. 

"float_value","int_value"

Parameter : Decimal Key Number

Decimal Key Number specifies how much decimal places the user needs.
Below is a example of the decimal key number till where we are  rounding off the value. 

2

Calculator

Description: 

Calculator operation is used when we want to calculate any particular process provided by the user depending upon the values which are in the columns.

Number of Parameters : 2

Parameter : Calculation Keys

Calculation Keys holds the calculated formula based on the column names of the provided data.
Below is a example where we are providing calculated formula's Amount1 - Amount2 and Amount1 + Amount2 based on column names of data.

"Amount1-Amount2","Amount1 + Amount2"

Parameter : New Calculation Keys

New Calculation Keys used to  store the calculated values.

Provide comma separated list of keys in double quotes to specify the value.

Below is an example of new keys name key1 and key2 which will store the calculated values.

"key1","key2"

Date Analytics

Description: 

Date analytics helps us to extract information about the date like the financial year, financial month, quarterly information of year as well as financial year and many other relatable information about the date. .

Number of Parameters : 4

Parameter : Data Field Key

In Data Field Key we provide the key name which holds the date.
Below is the example of the key name Created_datetime which holds the required value.

Created_datetime

Parameter : Fiscal Month Start 

Fiscal Month Start is used to specify the fiscal start month of an organization.
Below is an example of the organization's fiscal start month as April so we are taking the month number of the year i.e. 4.
 

4

Parameter : Date Column

Date Column is used for giving user defined 11 fields for savings data. If left blank by user it will generate 11 fields dynamically.
Below is an example of user defined 11 fields. 

"Calender Month","Calender Month Num","Calender Year","Calender FY Month Num","Calender FY","Calender Week Num","Calender Month Week","Calender Qr Num","Calender FY Qr Num","Calender Qr","Calender FY Qr"

Below is an example when user leaves date_column blank

Parameter : Weekday

Weekday is used to specify the starting of the week, %W (starting of week from Monday) and %U (starting of week from Sunday) .
Below is an example of weekday starting with Monday, 

%W

Repeat First Row Values

Description:

Repeat First Row Values operation is used to repeat the 1st row's value of the specified column's.

Number of Parameters : 2

Parameter : Source Key

Source Key is the key which holds the user's single line data that we need to pass.
Below is the example of  key name that holds the data

['product_data_response']['data']

Parameter : Fields to Repeat Value

Fields To Repeat Value is a list of key names comma separated in double quotes whose first row value we want to repeat.

Below is an example of key names whose 1st row value we want to repeat

"month","customer_site"

Grok Pattern

Description: 

Grok operation is used for parsing log files and extracting structured data from unstructured log lines. It employs predefined patterns to efficiently identify and capture specific types of information. Here is a list of commonly used Grok patterns:

These patterns enable the Grok operation to efficiently process log data and extract relevant information, facilitating better analysis and understanding of system logs. Users can customize their log parsing by leveraging these patterns to suit the specific needs of their applications.

Number of Parameters : 2

Parameter : Input Key

In the Input Key parameter, users are required to specify the key from which they intend to extract the data. This key serves as the reference point for the Grok operation to identify and capture the relevant information based on the predefined patterns.

For instance, when utilizing the Input Key parameter, consider a scenario where the specified key is 'Details.'

Input Key : 

Details

Within the 'Details' key, the data encapsulates an endpoint URL, a MAC address (00:1A:2B:3C:4D:5E), and both IPv4 (192.168.1.1) and IPv6 (2001:0db8:85a3:0000:0000:8a2e:0370:7334) addresses .

Details : This is endpoint url https://www.example.com/path/to/resource for mac add 00:1A:2B:3C:4D:5E and v4 192.168.1.1 and V6 2001:0db8:85a3:0000:0000:8a2e:0370:7334

Parameter : Grok Pattern

In the Grok Pattern parameter, users can specify a predefined pattern to guide the extraction of data. This pattern serves as a template, enabling the Grok operation to accurately identify and capture relevant information from the input data according to the defined structure.

For instance, when utilizing the Grok Pattern parameter, let's consider a scenario where we input the pattern 'grok_pattern.' This specified pattern guides the Grok operation in parsing and extracting data from the input based on the provided template.

Grok Pattern :

This is endpoint url %{URI:endpoint_url} for mac add %{MAC:mac_address} and v4 %{IPV4:ip_address_v4} and V6 %{IPV6:ip_address_v6}

Result :

Details: This is endpoint url https://www.example.com/path/to/resource for mac add 00:1A:2B:3C:4D:5E and v4 192.168.1.1 and V6 2001:0db8:85a3:0000:0000:8a2e:0370:7334
endpoint_url: https://www.example.com/path/to/resource
mac_address: 00:1A:2B:3C:4D:5E
ip_address_v4: 192.168.1.1 
ip_address_v6: 2001:0db8:85a3:0000:0000:8a2e:0370:7334

                                  

PDF Extractor

Description

This operation helps to extract data from PDF files. It takes place while data is in-flight.

Number of Parameters : 2

Parameter : File URL Key
Enter the key name which contains the PDF File URL. In this case the Base64 key will be empty
Example:

Items

Parameter : Base64 Key 
Write the key name which will have the Base64 encoded data. In this case the File URL Key will be empty.
Example:

@xyz.grapgh.downloadUrl

                            

ARRAY COUNT

Description:
This operation is employed to retrieve the record count within an array. It involves specifying the key name associated with an array value within the provided data.

Number of Parameters : 1

Parameter : Array Key Name

Provide the key name where its value is array , so that we'll get the count of the array.

Below is an example where we will get the length of the array of key "data"

['bizdata_dataset_response']['data']

                                


ENCODE DECODE

Description:

This operation facilitates the encoding or decoding of data by inputting the desired key name for encoding or decoding purposes.

Number of Parameters : 3

Parameter : Response Key

Pass the key name which holds the data that we want to encode or decode.

Below is the example how we can pass the response key

"email"

Parameter : Method Type

Give the type in which you want to encode/decode your data.

Below is the example how we can pass the Method Type

"utf-8" or "utf-8-sig" or "latin1"

Parameter : Process

Pass the process type based on the requirement. Possible value for this key is encode or decode.

Below is the example how we can pass the Process

"encode" or "decode"

                                 

RAW SENTENCE GENERATOR 

Description:

The Raw Sentence Generator operation transforms your structured JSON data into Raw sentence.

Number of Parameters : 3

Parameter : Singleline Key

Provide the key name that contains the single-line data, which we aim to utilize for generating the raw sentence.

Below is an example where we provide the data key, containing single-line data.

data

Parameter : Include Keys

Pass the key names separated by commas and enclosed in double quotes for which the raw sentence is to be generated. If all key names are to be included for raw sentence generation, leave this parameter empty.

Below is an example where we specify the key names Name and Commands for generating the sentence.

"Name","Commands"

Parameter: Raw Response Key

Specify the key name where you want to store the generated raw sentence.

Below is an example where we aim to store the generated raw sentence in the Response key.

Response

Various Use Cases for the Parameters:

Case 1:

When it's necessary to incorporate all upcoming keys in sentence generation.

Singleline Key (Empty)

Include Keys (Empty)

Raw Response Key

Response

Case 2:

When not all upcoming keys are needed for sentence generation, only specific keys are required.

Singleline Key (Empty)

Include Keys 

"Name","Commands"

Raw Response Key

Response

Case 3:

When aiming to generate a Raw Sentence from all dictionaries within the Singleline key, the Include Keys parameter remains empty each time. This facilitates the creation of a raw sentence containing all key-value pairs from the dictionaries inside the Singleline key.

Singleline Key (Empty)

data

Include Keys 

Raw Response Key

Response

Example for Case 3:

Input JSON

"data": [
    {
      "id": 1,
      "email": "george.bluth@reqres.in",
      "first_name": "George",
      "last_name": "Bluth",
      "avatar": "https://reqres.in/img/faces/1-image.jpg"
    },
    {
      "id": 2,
      "email": "janet.weaver@reqres.in",
      "first_name": "Janet",
      "last_name": "Weaver",
      "avatar": "https://reqres.in/img/faces/2-image.jpg"
    },
    {
      "id": 3,
      "email": "emma.wong@reqres.in",
      "first_name": "Emma",
      "last_name": "Wong",
      "avatar": "https://reqres.in/img/faces/3-image.jpg"
    }
  ]

Result

"raw_response": "The id is 1, email is george.bluth@reqres.in, first_name is George, last_name is Bluth, avatar is https://reqres.in/img/faces/1-image.jpg, id is 2, email is janet.weaver@reqres.in, first_name is Janet, last_name is Weaver, avatar is https://reqres.in/img/faces/2-image.jpg, id is 3, email is emma.wong@reqres.in, first_name is Emma, last_name is Wong, avatar is https://reqres.in/img/faces/3-image.jpg"

TIME UNITS 

Description:

This operation is designed to extract important information like the year, month, and date from timestamps in supported formats. It analyzes the timestamps you give it and gives back structured data, including the year, month, date, and sometimes more details based on the timestamp you provided.

Number of Parameters: 2

Parameter: Date Timestamp Key

Provide the Key Name which holds the timestamp value.

Below is an example where we provide the timestamp key, containing timestamp value.

timestamp

Parameter: Time Units

Provide the key name for saving data (can be left empty). For user-defined fields, provide the key names in double quotes separated by commas.

Below is an example where we provide a comma-separated list of keys enclosed in double quotes for user-defined fields.

"year", "month", "day", "hour", "minute", "second", "microsecond"

Note : If the user doesn't input any Time Units, the operation dynamically generates seven fields.

Data Chunking

Description:

The Data Chunking operation is used to split text data into smaller chunks of data based on the specified chunk size.

Number of Parameters: 4

Parameter: Process Key

Provide the key name that holds the data for which we need to create the chunks.

Below is an example where we provide the text key, containing the data for which we need to create the chunks. Please note, the user must input the key enclosed in square brackets and single quotes, like ['xxx'].

['text']

Parameter: Chunk Key

Provide the key name that will hold the chunked data.

Below is an example where we specify the chunks key for the Chunk Key parameter. This key will be used to store the individual chunks of data that are created from the original text

chunks

Parameter: Token Size

Provide the size of your chunk.

Below is an example where we provide the value 1000, This setting will create chunks of text, with each chunk being up to 1000 characters long.

For instance, if your input text is a long document or a large dataset, setting the Token Size to 1000 will ensure that the text is divided into smaller, more manageable pieces, each containing a maximum of 1000 characters. 

1000

Parameter: Group By Chunk

Provide the key name that will hold all the chunks made from the data.

Below is an example where we specify the Data_Chunks key for the Group By Chunk parameter.This key will be used to aggregate and store all the chunks created from the original data.

For instance, if your text data is divided into multiple chunks, the Data_Chunks key will serve as a container that holds these individual chunks.

Data_Chunks