multithread processing using datacollector by multiple csv file

@lakshmi_narayanan_t 

Please read *.csv from directory as a whole file and add the filter using streamsets collector .

Then you need to use jython or java script process the fetch the file content and send to destination .

Please  design your pipeline as given   below and do let me know if it helps.

In this case i am reading all csv file from directory as whole file , then filter the filename and by using jython evaluator fetching the file content and sending to destination.

Attached the pipeline for your reference.

Thanks & Regards

Bikram_

@lakshmi_narayanan_t

Instead of ${record:value('/fileInfo/filename')}

use ${record:attribute('filename')}

@lakshmi_narayanan_t

Please read *.csv from directory as a whole file and add the filter using streamsets collector .

Then you need to use jython or java script process the fetch the file content and send to destination .

 

Please  design your pipeline as given   below and do let me know if it helps.

 

 

In this case i am reading all csv file from directory as whole file , then filter the filename and by using jython evaluator fetching the file content and sending to destination.

 

Attached the pipeline for your reference.

 

Thanks & Regards

Bikram_

as you mentioned  above  is possible to  write in  MYSQL database .

@lakshmi_narayanan_t 

Yes, By using script processor , you can read the file content and load to DB.

Any sample file which you need to load , so i can help you on it.