Skip to main content

In Transformer, seeing the error messages in the Yarn logs.


AkshayJadhav
StreamSets Employee
Forum|alt.badge.img

Error:

Error message in the Yarn log: “Dynamic allocation of executors requires the external shuffle service. You may enable this through spark.shuffle.service.enabled.” even though the spark.shuffle.service.enabled setting is set to true on the Cluster

 

Resolution:

You are seeing this error because of the “spark.dynamicAllocation.enabled” property was set to true in the Extra Spark Configuration section under the Cluster tab on the pipeline. But you don’t have the “spark.shuffle.service.enabled” property set to true there as well.

If the “spark.dynamicAllocation.enabled” property is set in the pipeline, then the “spark.dynamicAllocation.enabled” property needs to be set on both the cluster and the pipeline.

 

Did this topic help you find an answer to your question?

0 replies

Be the first to reply!

Reply