Post-processing capabilities for GoogleCloudStorage_01

  • 23 March 2023
  • 5 replies

Hello everyone,

I just wanted to ask that don’t we have post-processing capability for “GoogleCloudStorage_01”?

Basically we are using “streamsets/datacollector:3.18.1” for some testing purpose and we are fetching data from GCS and want to move file to some other location post processing.


Best answer by AkshayJadhav 23 March 2023, 09:44

View original

5 replies

Userlevel 4

Hello @karansing - Thank you for reaching out to the Streamsets Community.  We do not have post-processing option available in the data collector 3.18.1 however you can try configuring the shell executor to remove the file from the bucket once it is processed.

Thank you - AkshayJ

@AkshayJadhav खूप खूप धन्यवाद जाधव साहेब, Hey I just checked with the documentation (here) but couldn’t find anything useful, can you please point me to the correct one or any sample code which does similar thing?

Userlevel 4

Hi @karansing - We do not have any sample code as such but you can follow the steps to achieve the same:

  1. Install gsutil on the data collector host.
  2. Create a small script that remove the files from bucket.
  3. Paste the commands in the shell executor and test it with sample data in dev environment.  

Hope this helps.

Thank you - AkshayJ

Sure @AkshayJadhav I will give a try, thank you :) 

Userlevel 4

Thank you @karansing - I would request you to mark the comment as best answer that will close the thread.

Feel free to reach out to community if you have any further question/issues with StreamSets Product.

Thank you - AkshayJ