Skip to main content

how to verify Spark and Scala version in Transformer EMR cluster ?

  • January 10, 2022
  • 0 replies
  • 176 views
AkshayJadhav
StreamSets Employee
Forum|alt.badge.img

Solution: 

1) Start the EMR Cluster. 

2) Start the Transformer pipeline. 

3) In EMR Console, Click on <cluster name>, then  Click Application History. 

4) Click download to the right of the application ID.  you should see below output:

 

{"Event":"SparkListenerLogStart","Spark Version":"2.4.4"}

{"Event":"SparkListenerBlockManagerAdded","Block Manager ID":{"Executor ID":"driver","Host":"ip-10-10-9-23.us-west-2.compute.internal","Port":41581},"Maximum Memory":1078827417,"Timestamp":1585838163140,"Maximum Onheap Memory":1078827417,"Maximum Offheap Memory":0}

{"Event":"SparkListenerEnvironmentUpdate","JVM Information":{"Java Home":"/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.242.b08-0.50.amzn1.x86_64/jre","Java Version":"1.8.0_242 (Oracle Corporation)","Scala Version":"version 2.11.12"},"Spark Properties":

 

Terms & ConditionsAccessibility statement