Issue:
S3 destination successfully write data to S3 in preview and cluster mode, but fails in local mode with "Caused by: org.apache.hadoop.fs.s3a.commit.PathCommitException: `s3a://<bucket-name>/': Filesystem not supported by this committer" error.
Resolution:
According to this documentation, writing data to S3 requires some buffering before committing data to the S3 filesystem. Below are the options for the committer. The default is "file" committer. (check the attached print screen)
Directory and partitioned committers require HDFS, and "magic" requires DynamoDB as metadata caching. Due to this reason, I'm going to disable the STF test for local mode.
It is possible that S3 Destination may work in local mode with proper configuration
