Question

multithread processing using datacollector by multiple csv file

  • 6 June 2023
  • 4 replies
  • 56 views
lakshmi_narayanan_t
Rank
Userlevel 2

as my origin is local director i want read multipile csv and process it by streamselector according to the file name =${record:value('/fileInfo/filename')=='submission_trackers_25523.csv'} but it shows error how can i handle this scenario.

cd

4 replies

Bikram
Rank
Userlevel 5
Badge +1

@lakshmi_narayanan_t 

Please read *.csv from directory as a whole file and add the filter using streamsets collector .

Then you need to use jython or java script process the fetch the file content and send to destination .

 

Please  design your pipeline as given   below and do let me know if it helps.

 

 

In this case i am reading all csv file from directory as whole file , then filter the filename and by using jython evaluator fetching the file content and sending to destination.

 

Attached the pipeline for your reference.

 

Thanks & Regards

Bikram_

1 Attachment
saleempothiwala
Rank
Userlevel 4
Badge

@lakshmi_narayanan_t

 

Instead of ${record:value('/fileInfo/filename')}

 

use ${record:attribute('filename')}

lakshmi_narayanan_t
Rank
Userlevel 2

@lakshmi_narayanan_t

Please read *.csv from directory as a whole file and add the filter using streamsets collector .

Then you need to use jython or java script process the fetch the file content and send to destination .

 

Please  design your pipeline as given   below and do let me know if it helps.

 

 

In this case i am reading all csv file from directory as whole file , then filter the filename and by using jython evaluator fetching the file content and sending to destination.

 

Attached the pipeline for your reference.

 

Thanks & Regards

Bikram_

as you mentioned  above  is possible to  write in  MYSQL database .

Bikram
Rank
Userlevel 5
Badge +1

@lakshmi_narayanan_t 

Yes, By using script processor , you can read the file content and load to DB.

Any sample file which you need to load , so i can help you on it.

Reply


Terms & Conditions