I have numerous Kafka topics that I’m moving to Databricks, but I don’t want the pipelines to continuously run. Is there a way that I can schedule a pipeline to run for a certain amount of time or trigger it to stop after a certain amount of time...say an hour or 2?
Best answer by saleempothiwalaView original
If the source is Kafka and if you want to stop after consuming the messages then you need to stop by using the code .
We don’t have any event to stop the pipeline .
You also can try to one thing ,stop the event by selecting the option data bricks and check if it helps.
Below the snippet for your reference.
Init Script: state['first_batch'] = "true"
Please let me know if I can help you more on the issue.
Thanks & Regards