Problem:
How to add make PySpark processor to work on Databricks.
Solution:
This error can be solved by adding the following env variables on Databricks Spark cluster with a runtime that comes bundled with Python:
PYTHON_HOME=/databricks/python3/bin/python3
PYSPARK_PYTHON=/databricks/python3/bin/python3
PYSPARK_DRIVER_PYTHON=/databricks/python3/bin/python3
PYTHONPATH=/databricks/python3/bin/python3
