Question

Would Streamsets be able to read data from a Standby Log on an oracle instance

  • 8 August 2022
  • 3 replies
  • 65 views

We currently are using an OTLP DB and read the data for CDC via the Archive / Redo Logs. However this is limiting us so if we spin up an Oracle ADG or a CDG instance can we read and process our CDC from the logs that are replicated on an ADG or CDG instance ? 


3 replies

Userlevel 1

Currently, StreamSets’ Oracle CDC connector requires a direct connection to the primary database, in part because Log Miner operations require more than read-only permissions. 

However, if you are concerned with lag time, you can move the SQL parsing to a separate pipeline that can be scaled out using multiple Job Template instances, and avoid performing the SQL parsing in the Oracle CDC Origin.  Please see this example of using a SQL Parser Processor in a separate pipeline.

Thanks for the response and I really appreciate it. Here is the situation I am in . 

 

I have one RAC DB against which we are using streamsets for CDC purposes

Streamsets is limited to 2 Threads or CPU cores per node in the cluster which comes to 4 cores in total 

This Source DB is very chatty and generates a lot of logs especially when they do purging of data or make mods to their tables

As and when that happens with this limited resources that Streamsets would have to play with it starts chocking ( as an example a 197K record based purge activity tool like 28 - 40 hrs of lag on the target side ) 

I am trying to get away from that as 197k records is chump change to begin with but taking on an avg 30 hrs to process that volume is crazy. Once we upgraded to 4.x version of streamsets we are seeing an improvement in the overall data read / sec from 300 rec/sec to 600 rec/sec. Despite of this I am not able to cope up. However we also have a CDG instance on which the logs are in Standby mode so thats why I am trying to see if the option would work for us or not. 

I am going to give it a try with the approach the doc you provided depicts. I would have to discuss with our inhouse architect before we even decide to move forward or no so will keep you posted.

Userlevel 1

Very good!  Your StreamSets account team can help prototype this design pattern

Reply