Map the job submitted in transformer to the one run in Databricks SparkUI

  • 10 March 2023
  • 0 replies
  • 14 views

Userlevel 3
Badge +1

How do we map the job submitted in transformer to the one run in Spark? Or, in other words, what is the corresponding job in the Spark UI for the pipeline/job in question?

A:

In the transformer log, we see something like below once the job is submitted to databricks.

2023-03-02 12:02:18,099 INFO submit run response run id: 108505820

In spark UI, you would see the job group having the id 108505820

Run response id can also be seen as Task Run id in databricks workflow > job.


0 replies

Be the first to reply!

Reply