Error:
Error message in the Yarn log: “Dynamic allocation of executors requires the external shuffle service. You may enable this through spark.shuffle.service.enabled.” even though the spark.shuffle.service.enabled setting is set to true on the Cluster
Resolution:
You are seeing this error because of the “spark.dynamicAllocation.enabled” property was set to true in the Extra Spark Configuration section under the Cluster tab on the pipeline. But you don’t have the “spark.shuffle.service.enabled” property set to true there as well.
If the “spark.dynamicAllocation.enabled” property is set in the pipeline, then the “spark.dynamicAllocation.enabled” property needs to be set on both the cluster and the pipeline.
