Issue:

• By default, transformer pipeline is submitted in default queue. Is there a way to submit the transformer pipeline in different queue?
• How to set the spark.yarn.queue=<queue-name> in transformer pipeline?

Resolution:

• In transformer pipeline UI --> Go to Cluster tab --> Add spark.yarn.queue=root.users.test in Extra Spark Configuration.
• Now, transformer pipeline will be submitted in root.users.test queue.