I’m following the DataOps Platform Fundamentals course and the Build a Pipeline chapter has you enter “/zomato” as a files directory in the configuration of a Directory source.
However, when validating the pipeline I get the error: SPOOLDIR_12-Directory ‘/zomato’ does not exist: conf.spoolDir.
Any solution for this?
Best answer by dima
Make sure the Directory exists. Keep in mind that if you're running Data Collector in Docker, the path would be inside the container.
Make sure the Directory exists. Keep in mind that if you're running Data Collector in Docker, the path would be inside the container.
Thank you.
So there are crucial steps missing from the course. They could have at least mentioned to create the zomato directory in the container and also download the actual csv dataset file from the zomato website.
I’m completely new to streamsets and also to the devops role to be honest, and as beginners, we expect courses to have the complete information needed for the hands on part to actually work.
Just a quick mention, in the course, of creating the directory and downloading the dataset file would have been enough to point me in the right direction to make the pipeline work.
SebastienM - Thanks for the feedback. We’ll review the lab instructions for clarity and make the necessary edits. We incorporate feedback as we receive it so we appreciate you taking the time to document it.
Added the tag “DataOps Platform Fundamentals Course”
I found the source of the error.
This error occurs if you don’t follow the prior “Setup a Deployment” lab exactly.
Step 9 shows:
| sudo docker container ls -n 3 | tail -n +2 | awk '{ print $1 }' | while read line; do sudo docker network connect streamsets-core $line; sudo docker network connect streamsets-integrations $line; sudo docker network connect streamsets-cooked $line ; sudo docker cp training-dependencies/zomato/ $line:zomato; sudo docker update --restart=always $line; echo "set up complete for $line"; done |
That script sets up your Zomato directory so that you can complete later exercises. Hope that helps!
I found the source of the error.
This error occurs if you don’t follow the prior “Setup a Deployment” lab exactly.
Step 9 shows:
| sudo docker container ls -n 3 | tail -n +2 | awk '{ print $1 }' | while read line; do sudo docker network connect streamsets-core $line; sudo docker network connect streamsets-integrations $line; sudo docker network connect streamsets-cooked $line ; sudo docker cp training-dependencies/zomato/ $line:zomato; sudo docker update --restart=always $line; echo "set up complete for $line"; done |
That script sets up your Zomato directory so that you can complete later exercises. Hope that helps!
FYI, I followed every step of the tutorial from the start, and copy, pasted and executed those parts of the tutorial that had terminal commands.
Even after running the above script , “Zomato folder is showing after running the bash command cd training-dependencies and than ls command“ but still same issue is getting while previewing the pipeline
I found the source of the error.
This error occurs if you don’t follow the prior “Setup a Deployment” lab exactly.
Step 9 shows:
| sudo docker container ls -n 3 | tail -n +2 | awk '{ print $1 }' | while read line; do sudo docker network connect streamsets-core $line; sudo docker network connect streamsets-integrations $line; sudo docker network connect streamsets-cooked $line ; sudo docker cp training-dependencies/zomato/ $line:zomato; sudo docker update --restart=always $line; echo "set up complete for $line"; done |
That script sets up your Zomato directory so that you can complete later exercises. Hope that helps!
After running the set of given scripts, it is showing “ training-dependencies/zomato no such file or directory ”
I found the source of the error.
This error occurs if you don’t follow the prior “Setup a Deployment” lab exactly.
Step 9 shows:
| sudo docker container ls -n 3 | tail -n +2 | awk '{ print $1 }' | while read line; do sudo docker network connect streamsets-core $line; sudo docker network connect streamsets-integrations $line; sudo docker network connect streamsets-cooked $line ; sudo docker cp training-dependencies/zomato/ $line:zomato; sudo docker update --restart=always $line; echo "set up complete for $line"; done |
That script sets up your Zomato directory so that you can complete later exercises. Hope that helps!
Hi, I saw this discussion topic was at last 2 years. Unfortunately, I still not able to see the course update with the path guideline in StreamSets Platform Fundamentals.
I am also encountered error below:
Error: [Directory 1 - Files Directory] The directory '/data/dir' configured to read from does not exist (SPOOLDIR_12)
Remediation Steps: Create the directory that you want to read from before starting the pipeline
And I am using Windows Machine.
I am not able to run the scripts you provided in docker command line. I faced below error:
At line:1 char:81
+ ... ainer ls -n 3 | tail -n +2 | awk '{ print $1 }' | while read line; do
+ ~
Missing statement body in do loop.
+ CategoryInfo : ParserError: (:) [], ParentContainsErrorRecordException
+ FullyQualifiedErrorId : MissingLoopStatement
I am totally new to streamset and docker hub, hope to get any advice here...
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.
