How is my Cluster Mode Pipeline Parallelized?

In Cluster Streaming mode (currently Kafka Origin only) a separate pipeline instance is launched for each partition in the topic.

For example, if you have 10 partitions in the configured topic, 10 worker pipelines are spawned.

This is equivalent to running 10 standalone pipelines in the same consumer group, but with the added HA features of running on Spark Streaming.

Did this topic help you find an answer to your question?

This topic has been closed for comments

Sign up