I want to remove files (whole file data format) from a origin stage (GCS stage) after processing it, but when I activate thge post-processing option and set it to "Delete" and try to start the associated job to the pipeline a have the Error:
GCS_11 - Error validating postprocessing options. If 'Whole File' data format is used, neither 'Delete' nor 'Archive -> Move into ...' can be used.
There are any way to do that directly from a pipeline or the best practice is to add a shell script at the end of the pipeline to remove the files from command line with google CLI commands after processing them?
The error you are obtaining is pretty clear. Unfortunately, what you are trying to do is not possible…
You should try with events & some scripts, bash perhaps.