Issue:
- By default, transformer pipeline is submitted in default queue. Is there a way to submit the transformer pipeline in different queue?
- How to set the spark.yarn.queue=<queue-name> in transformer pipeline?
Resolution:
- In transformer pipeline UI --> Go to Cluster tab --> Add spark.yarn.queue=root.users.test in Extra Spark Configuration.
- Now, transformer pipeline will be submitted in root.users.test queue.
