I am having 6 API’s where I have to build 3 pipelines for each i.e., API-KAFKA, KAFKA-FILESYSTEM and FS-Target table. Like this For 6 API’s I have to build and execute 18 pipelines (6*3). Could anyone suggest me that is there any approach which reduces the number of pipelines ( to push all 6 APIs data into 1 Kafka topic in 1st pipeline, after that I will generate data file from Kafka to FS and FS to target table).
FYI- my target table is same for all APIs.
If they have similar authentication, more or less similar headers, parameters, etc then it is easy to create 1 pipeline and create multiple jobs and pass parameters that can configure the API calls.
2nd pipe line is KAFKA TO FS and the 3rd one is FS to target table.