Skip to main content
Solved

Post-processing capabilities for GoogleCloudStorage_01

karansing
Fan

Hello everyone,

I just wanted to ask that don’t we have post-processing capability for “GoogleCloudStorage_01”?

Basically we are using “streamsets/datacollector:3.18.1” for some testing purpose and we are fetching data from GCS and want to move file to some other location post processing.

Best answer by AkshayJadhav

Hi @karansing - We do not have any sample code as such but you can follow the steps to achieve the same:

  1. Install gsutil on the data collector host.
  2. Create a small script that remove the files from bucket.
  3. Paste the commands in the shell executor and test it with sample data in dev environment.  

Hope this helps.

Thank you - AkshayJ

View original
Did this topic help you find an answer to your question?

5 replies

AkshayJadhav
StreamSets Employee
Forum|alt.badge.img
  • AkshayJadhavStreamSets Employee
  • StreamSets Employee
  • 101 replies
  • March 23, 2023

Hello @karansing - Thank you for reaching out to the Streamsets Community.  We do not have post-processing option available in the data collector 3.18.1 however you can try configuring the shell executor to remove the file from the bucket once it is processed.

Thank you - AkshayJ

karansing
Fan
  • karansingFan
  • Author
  • Fan
  • 2 replies
  • March 23, 2023

@AkshayJadhav खूप खूप धन्यवाद जाधव साहेब, Hey I just checked with the documentation (here) but couldn’t find anything useful, can you please point me to the correct one or any sample code which does similar thing?

AkshayJadhav
1 person likes this
  • AkshayJadhav
    AkshayJadhav
AkshayJadhav
StreamSets Employee
Forum|alt.badge.img
  • AkshayJadhavStreamSets Employee
  • StreamSets Employee
  • 101 replies
  • Answer
  • March 23, 2023

Hi @karansing - We do not have any sample code as such but you can follow the steps to achieve the same:

  1. Install gsutil on the data collector host.
  2. Create a small script that remove the files from bucket.
  3. Paste the commands in the shell executor and test it with sample data in dev environment.  

Hope this helps.

Thank you - AkshayJ

karansing
1 person likes this
  • karansing
    karansing
karansing
Fan
  • karansingFan
  • Author
  • Fan
  • 2 replies
  • March 23, 2023

Sure @AkshayJadhav I will give a try, thank you :) 

AkshayJadhav
StreamSets Employee
Forum|alt.badge.img
  • AkshayJadhavStreamSets Employee
  • StreamSets Employee
  • 101 replies
  • March 23, 2023

Thank you @karansing - I would request you to mark the comment as best answer that will close the thread.

Feel free to reach out to community if you have any further question/issues with StreamSets Product.

Thank you - AkshayJ 

Reply


Terms & Conditions