Environment:
-
Data Collector.
-
Destination Databricks Delta Lake.
Issue:
The pipeline is encountering a "DELTA_PATH_DOES_NOT_EXIST" error, signaling that the specified path does not exist. To resolve this, we have manually created the file in Databricks.
DELTA_LAKE_37 - Runnable process batch failed: DELTA_LAKE_12 - Could not perform SQL operation: java.sql.SQLException: [Databricks][DatabricksJDBCDriver](500051) ERROR processing query/statement. Error Code: 0, SQL state: org.apache.hive.service.cli.HiveSQLException: Error running query: [DELTA_PATH_DOES_NOT_EXIST] org.apache.spark.sql.AnalysisException: dbfs:/dash/DeltaLake/TEST doesn't exist at org.apache.spark.sql.hive.thriftserver.HiveThriftServerErrors$.runningQueryError(HiveThriftServerErrors.scala:48) at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.$anonfun$execute$1(SparkExecuteStatementOperation.scala:694) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) at com.databricks.unity.UCSEphemeralState$Handle.runWith(UCSEphemeralState.scala:41) at com.databricks.unity.HandleImpl.runWith(UCSHandle.scala:99) at
Resolution:
The error indicate that “dbfs:/dash/DeltaLake/TEST” needs to to be created. You can browse the BDFS from the Catalog page and confirm if it is available or not.
To resolve this, run the following command in the notebook and re-run the pipeline.
%fs mkdirs dbfs:/dash/DeltaLake/TEST