Skip to main content

Want a parquet file format in Local FS while reading from Kafka.

 

I tried whole file transformer but its not working as it need whole file but my data format in Kafka is SDC records. 

I tried it to read into AVRO format then tried converting into Parque, This logic worked but it required 2 different pipelines.

Pipeline 1 : Kafka to Avro.

Pipeline 2: Avro to Parque using whole file transformer. 

 

Is there is any way where I can achieve this in single pipeline.

 

Thanks in Advance. 

Be the first to reply!

Reply