Use Case:-

We have a dataset, in which we have columns as follows:-

FIRST_NAME, LAST_NAME, EMAIL, PHONE, GENDER, DEPARTMENT, JOB_TITLE, YEARS_OF_EXPEREIENCE, SALARY. 

Lets sort the column SALARY in ascending order and write to a new table with just 4 columns, FIRST_NAME, LAST_NAME, YEARS_OF_EXPEREIENCE, SALARY.

Pipeline Design:-

1. Snowflake Table (origin)
2. Sort (Processor)
3. Column Remover (Processor)
4. Snowflake Table (Destination)

Pipeline Working:-

1. Snowflake Origin will fetch the table and columns and pass the records to Sort processor
2. Sort processor will sort the data based on the configuration and pass it to Column Remover. (SALARY column, Ascending order)
3. Column Remover will keep or remove the columns based on the configuration.
4. Snowflake Table Destination will write the data to a new snowflake table.