Example pipelines for Delta lake lookup processor


Userlevel 3
Badge +1

 

Input data:

[
  {
    "product_id": 1,
    "unit_price": 23.2
  },
  {
    "product_id": 4,
    "unit_price": 21.64
  },
  {
    "product_id": 2,
    "unit_price": 10.87
  }

]

Delta lake lookup table. This is same for all operations performed

Return all matching rows, generating a record for each match:

 

 

Return first matching row:

There won’t be change in the output from above since there are no duplicates in input.

 Return a count of matching rows:

Return true if matches exist, otherwise false:

Input data:

[
  {

    "product_id": 1,
    "unit_price": 23.2
  },
  {

    "product_id": 2,
    "unit_price": 21.64
  },
  {
    "product_id": 4,
    "unit_price": 10.87
  },
  
  {
    "product_id": 5,
    "unit_price": 99.32
  },
  
  {
    "product_id": 4,
    "unit_price": 20.33
  }
]

Return all matching rows, generating a record for each match:

Return first matching row:

Return true if matches exist, otherwise false:

 

Return a count of matching rows:

 

0 replies

Be the first to reply!

Reply