How do we map the job submitted in transformer to the one run in Spark? Or, in other words, what is the corresponding job in the Spark UI for the pipeline/job in question?
In the transformer log, we see something like below once the job is submitted to databricks.
|2023-03-02 12:02:18,099||INFO||submit run response run id: 108505820|
In spark UI, you would see the job group having the id 108505820
Run response id can also be seen as Task Run id in databricks workflow > job.