"Failed to compile" exception: 'Compiling "GeneratedClass": Cannot load class ... through the given ClassLoader'.

Issue:

While trying to execute a Transformer pipeline against a Hadoop distribution of Spark, an exception is encountered that halts the pipeline. The pipeline does submit and run successfully, and generates a corresponding Job on the Spark Cluster - however during execution of the pipeline, an exception similar to the following is encountered:

ERROR codegen.CodeGenerator: failed to compile: org.codehaus.janino.InternalCompilerException: Compiling "GeneratedClass": Cannot load class "com.esotericsoftware.kryo.KryoSerializable" through the given ClassLoader
org.codehaus.janino.InternalCompilerException: Compiling "GeneratedClass": Cannot load class "com.esotericsoftware.kryo.KryoSerializable" through the given ClassLoader

Cause:

This is an indication that the environment is incorrectly configured for the host on which Transformer is running.

Resolution:

There are a few steps that can be taken to validate and verify the environment Transformer is running in:

1. Verify that the necessary installation steps have been completed, as listed in the Transformer documentation[1]. In particular, verify that the SPARK_HOME, HADOOP_CONF_DIR, and YARN_CONF_DIR have all been configured as necessary - these can be configured as persistent environment variables on the host itself, or configured in Transformer's 'transformer-env.sh' script (stored under $TRANSFORMER_HOME/libexec/transformer-env.sh).

2. Validate what the target Spark Cluster is using as its SPARK_HOME directory using the spark2-shell:

[user@host]# spark2-shell
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Spark context Web UI available at http://hostname.com:4040
Spark context available as 'sc' (master = yarn, app id = application_1588800000000_0000).
Spark session available as 'spark'.

scala> sc.getConf.get("spark.home")
res0: String = /opt/cloudera/parcels/SPARK2-2.3.0.cloudera2-1.cdh5.13.3.p0.316101/lib/spark2

3. Verify the client configurations directory that the Spark service is using, as these should match what's configured for HADOOP_CONF_DIR and/or YARN_CONF_DIR (which are used by Transformer):

[user@host]# ls -al $SPARK_HOME/conf

4. Verify which client configurations directory Spark is using for YARN by checking the 'spark-env.sh' script stored in the $SPARK_HOME/conf directory, as these should match what's configured for HADOOP_CONF_DIR and/or YARN_CONF_DIR (which are used by Transformer):

[user@host]# cat $SPARK_HOME/conf/spark-env.sh | grep HADOOP_CONF
HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-$SPARK_CONF_DIR/yarn-conf}

[1] https://streamsets.com/documentation/transformer/3.11.x/help/transformer/Installation/Installing.html#concept_es1_hyw_dhb