Skip to main content

Hi,

I have created a pipeline in the StreamSets Data Collector which reads data from an Apache Kafka topic and inserts it into a Databricks Delta table.

Table Auto Creation has been disabled. I have created the table on the Databricks instance separately and the columns and data types in the table are correct.

But I am getting the following error while validating the pipeline. What could be the reason for the error?

Caused by: com.streamsets.pipeline.api.StageException: DELTA_LAKE_13 - Table 'gov_src_req_sts_v1' column '' unsupported type '' at com.streamsets.pipeline.stage.destination.definitions.JdbcTableDefStore.get(JdbcTableDefStore.java:157) at com.streamsets.pipeline.stage.destination.definitions.CacheTableDefStore$1.load(CacheTableDefStore.java:59) at com.streamsets.pipeline.stage.destination.definitions.CacheTableDefStore$1.load(CacheTableDefStore.java:53) at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3542) at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2323) at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2286) at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2201)

Hi @rohan_b,

Which Cluster version are you using in Databricks? And which version of the enterprise connector?

My guess (from previous customers and users hitting this) is that you have version 7 or higher and an “old” version of the connector.

Can you please try updating the connector to the latest version available?

Hi Alex,

Thanks for the reply.

Databricks Runtime version is 9.1.

How do I update the connector? Can you provide any reference / documentation?

Regards,

Rohan

 

 

Hi Rohan, I think that this is what you are looking for https://docs.streamsets.com/portal/datacollector/3.18.x/help/datacollector/UserGuide/Upgrade/PostUpgrade.html#task_rql_l3y_s3b

Thanks

Reply


Terms & Conditions