How to troubleshoot connection timeout related issues in Azure Synapse stage

Issue: In Azure Synapse stage, there might be some errors that initially look like the issue with loading/writing data to Synapse, but it could be caused by connection timeout:

• AZURE_STORAGE_07 - Could not get a Stage File Writer instance to write the records: 'AZURE_STORAGE_11 - Could not load file to Azure Storage.
• AZURE_STORAGE_02 - Azure Synapse load request failed: 'AZURE_DATA_WAREHOUSE_09 - Could not merge staged file com.streamsets.pipeline.api.StageException: AZURE_DATA_WAREHOUSE_00 - Could not perform SQL operation

To determine the root cause, please turn on debug log by adding the following to log4j to further investigate the issue:

logger.l5.name = com.streamsets.pipeline.stage.common.synapse
logger.l5.level = DEBUG

logger.l6.name = com.streamsets.pipeline.stage.destination.datawarehouse
logger.l6.level = TRACE

You might be able to get more hints in debug log and see if it’s related to connection timeout, such as:

• Connection is not available, request timed out after 30000ms.
• Channel response timed out after 60000 milliseconds.

Solution: If the issue is indeed caused by connection timeout, please upgrade to SDC 5.6.0 or above as we have added the option to provide configuration parameters in SDC 5.6.0. Using this new configuration, you can change connection timeout property along with other driver properties.