Question

Job to pick only the changes since the last run

A
Rank

Hi

 

I need your help here.

I have a requirement to build a pipeline where I need to read data from one oracle DB and load the data into another oracle DB. Now for doing this everything I run my job, I should pick only those records from the source database which have been inserted or updated since the last run of my job.

 

Please help me with your inputs.

 

Regards,

Ashish

3 replies

dima
Rank
Userlevel 3
Badge

Sounds like what Incremental mode in our JDBC Multitable Consumer origin does well. 

K
Rank

I believe oracle cdc client can also be used if feasible since you also require change tracking. If the rows contain last updated time then it can be leveraged to track and replicate changes.

Bikram
Rank
Userlevel 5
Badge +1

@Ashish 

 

You can follow full load and delta load concept here.

Please find below the steps for the same.

  1. Read data from source DB and use look up to check if the fetched data from source are available in destination.

     2 . select count(*) form emp where empid = record.value(‘empid’)

 

    3  if count == 0 , no data present in destination table , then here insert dato into it

   4 . if cout > 0 , then data already in destination DB , then use update query and validate the columns value changed and only update the changed column values.

 

5 While inserting data to destination table , please add a column (named as lastchangeddatetime) which will help you to track the changed records from source to destination

 

Reply


Terms & Conditions