Skip to main content

Hi team,

Example pipeline

Pipeline 1 -- origin--to--kafka producer
Pipeline 2 -- kafka consumer--to--destination

Need your support for how to create job to merge pipeline1 and pipeline2. 
Please let me know the possibility.

Hi @ujeyakum 

Pipeline 1 can run on its own and Pipeline 2 can run on its own. Since these are kafka pipelines, you just let them run independently. As soon as the Origin from Pipeline 1 creates a new record, it will generate a message in Kafka Producer. On the other hand, Pipeline 2 will continously look for new messages and as soon as it sees a new message, it will send it to the destination. 

You dont have to manually make dependencies.


Reply