How do we map the job submitted in transformer to the one run in Spark? Or, in other words, what is the corresponding job in the Spark UI for the pipeline/job in question?

A:

In the transformer log, we see something like below once the job is submitted to databricks.

In spark UI, you would see the job group having the id 108505820

Run response id can also be seen as Task Run id in databricks workflow > job.