Problem:
You might get an error like below while using ADLS Gen2 to write/read from BLOB storage.
RUN_ERROR: HADOOPFS_13 - Error while writing to HDFS: Operation failed: "This request is not authorized to perform this operation.", 403, HEAD, https://abcd.blob.core.windows.net/xxxx//?upn=false&action=getAccessControl&timeout=90
Solution:
Error might be misleading but this could happen whenever you do not have correct permissions on the storage account or your blob is private and behind the firewall.
- Check whether you have a "Storage Blob Data Contributor" role assigned to your service account used to access the AZ blob. here is a good airticle on it https://learn.microsoft.com/en-us/answers/questions/775151/cant-run-azure-synapse-notebook-in-pipeline
- If your blob storage is private and behind the firewall, You can solve this error by adding public IP of the SDC host into the whitelist. You can get the public IP of the SDC host by running command
curl ifconfig.co