Question

how to merge 2 different pipeline

  • 14 February 2023
  • 1 reply
  • 30 views
U
Rank

Hi team,

Example pipeline

Pipeline 1 -- origin--to--kafka producer
Pipeline 2 -- kafka consumer--to--destination

Need your support for how to create job to merge pipeline1 and pipeline2. 
Please let me know the possibility.

1 reply

saleempothiwala
Rank
Userlevel 4
Badge

Hi @ujeyakum 

Pipeline 1 can run on its own and Pipeline 2 can run on its own. Since these are kafka pipelines, you just let them run independently. As soon as the Origin from Pipeline 1 creates a new record, it will generate a message in Kafka Producer. On the other hand, Pipeline 2 will continously look for new messages and as soon as it sees a new message, it will send it to the destination. 

You dont have to manually make dependencies.

Reply


Terms & Conditions