Question

how to write Parquet file in local fs from kafka as source

  • 18 June 2023
  • 0 replies
  • 58 views

Want a parquet file format in Local FS while reading from Kafka.

 

I tried whole file transformer but its not working as it need whole file but my data format in Kafka is SDC records. 

I tried it to read into AVRO format then tried converting into Parque, This logic worked but it required 2 different pipelines.

Pipeline 1 : Kafka to Avro.

Pipeline 2: Avro to Parque using whole file transformer. 

 

Is there is any way where I can achieve this in single pipeline.

 

Thanks in Advance. 


0 replies

Be the first to reply!

Reply