Solved

Post-processing capabilities for GoogleCloudStorage_01

  • 23 March 2023
  • 5 replies
  • 43 views

Hello everyone,

I just wanted to ask that don’t we have post-processing capability for “GoogleCloudStorage_01”?

Basically we are using “streamsets/datacollector:3.18.1” for some testing purpose and we are fetching data from GCS and want to move file to some other location post processing.

icon

Best answer by AkshayJadhav 23 March 2023, 09:44

View original

5 replies

Userlevel 4
Badge

Hello @karansing - Thank you for reaching out to the Streamsets Community.  We do not have post-processing option available in the data collector 3.18.1 however you can try configuring the shell executor to remove the file from the bucket once it is processed.

Thank you - AkshayJ

@AkshayJadhav खूप खूप धन्यवाद जाधव साहेब, Hey I just checked with the documentation (here) but couldn’t find anything useful, can you please point me to the correct one or any sample code which does similar thing?

Userlevel 4
Badge

Hi @karansing - We do not have any sample code as such but you can follow the steps to achieve the same:

  1. Install gsutil on the data collector host.
  2. Create a small script that remove the files from bucket.
  3. Paste the commands in the shell executor and test it with sample data in dev environment.  

Hope this helps.

Thank you - AkshayJ

Sure @AkshayJadhav I will give a try, thank you :) 

Userlevel 4
Badge

Thank you @karansing - I would request you to mark the comment as best answer that will close the thread.

Feel free to reach out to community if you have any further question/issues with StreamSets Product.

Thank you - AkshayJ 

Reply