When using the Field Remover processor we can see the following error:
Transformer StackOverFlow error:
This error happens inside org.apache.spark.sql.catalyst because of not enough memory to process that high number of columns. In this case, we would recommend to increase the amount of memory by adding the following configurations for the spark-submit command in the cluster panel:
Spark.executor.extraJavaOptions : -Xss64m
Spark.driver.extraJavaOptions : -Xss64m
