StreamSets Academy

I’m following the DataOps Platform Fundamentals course and the Build a Pipeline chapter has you enter “/zomato” as a files directory in the configuration of a Directory source.However, when validating the pipeline I get the error: SPOOLDIR_12-Directory ‘/zomato’ does not exist: conf.spoolDir.Any solution for this?

Environment:StreamSets Data Collector v4.3. Control Hub v3.51.4. (Inactivity period for session termination = 30 Min) Issue:It has been observed that credentials time out is happening on the long running Streaming Pipelines when using Start Jobs stage. This can happen when using the User & Password Authentication type.The following error captured from control hub logs indicates that the token has expired. START_JOB_04 - Failed to start job template for job ID: START_JOB_04 - Failed to start job template for job ID: 1238759hvbv949:Test, status code '401': {"ISSUES":[{"code":"SSO_01","message":"User not authenticated"}]}, status code } Solution:After an internal investigation, found that  we are encountering COLLECTOR-1125, which has been addressed and resolved in data collector v5.1.

While i am working with a table that had no Primary Key and the total number of records were also huge, so i thought of a basic SQL concept that “ To Uniquely Identify a row some times we use Multiple columns “.To use the same in incremental mode in JDBC Query Consumer, I thought of using multiple columns as offset, but i have gone nowhere.Can anybody have any thoughts on this.Thank you.

Hi, I’ve started working with StreamSets recently and I want a Transformer Engine to process data from IBM Cloud Object Storage (COS) and write it to a Hive table managed by Watsonx.data. However, I’m facing challenges connecting StreamSets to the Spark engine in Watsonx.data.Watsonx.data uses an optimized Spark engine, and from what I understand, it behaves as a Standalone Cluster. The issue is that I don’t have access to the Spark Master URL (spark://<master-host>:<port>), which StreamSets requires to connect the Transformer Engine. StreamSets needs a Master URL to send jobs to the Spark cluster. Without this, I haven’t been able to configure the pipeline to process jobs directly in Watsonx.data’s Spark engine.As well I’ve tried with the data collector engine, that I understand that it’s not the best fit but it requires easier connections. It didn’t work neither, I’ll leave logs at the bottom for this situation.Is it possible to connect StreamSets to the Spark engine in W

Hello,I opened my account in cloud.login.streamsets.com with my IBM id (email@ibm.com) and I am performing this course https://academy.streamsets.com/learn/courses/6/streamsets-platform-fundamentals. My account is not privileged enough to perform all the course lessons. For example I cannot schedule jobs. All the scheduling menus are not seen in my streamsets control hub account. I think this is because I do not have the "Scheduler Operator" role in my user roles:my account organization is “IBM StreamSets Hands On Lab”.I want to be able to complete all the tasks in the course, how can I make sure that my account is compatible with the course tasks?Thanks in advance,Aviv

Hi,We are a team of developers. As part of a project we want to use Streamsets Data Collector as a USB between our application and several other third-party applications for Electronic Data Interchanges (EDIs). Therefore, we want to train on this platform in person in France or Morocco. Does anyone know of a training center in these 2 countries?

Hello,We have an issue with cert, error with amazon s3, trying to replace a new certificate in the truststory.jks file does not help, streamsets was restarted from the Cloudera level. Streamsets Collector 3.10.1.

I created a pipeline load data from oracle to snowflake. even though the target table in snowflake does not exist in snowflake , the pipline still show succeeded without any error, also without any data loaded. If checked the log, it is said “Error SnowflakeSQLException: SQL compilation errortable XXXX does not exist or not authorized.”How to make the pipeline failed if found target table is incorrect, or column is incorrect, for such kind of normal errors reporting in data load? 

Hi ,I am unable to access the Lab documents provided in the DataOps Platform Fundamentals Course” and the Lab page is rendered as Blank window. Able to access to Streamset youtube videos for the related lesson, however, could not access the lab. Not sure if this is an issue related to accessing google drive.Please let us know if any workaround to fix this issue. Thanks.  

Hi , I am taking up the treamsets Foundational course and unable to open the Lab Instructions for Streamsets Foundational course . Often getting blank page instead of the actual contents. Any idea whether it is a known issue ? Please suggest if any workaround available . Thanks .

Hi,There is a username and password to login to academy portal using personal account. However there is no MFA enabled. I had thought Auth0 was some form of authentication on top of a password but it does not ask for any extra layer of verification of the account holder. Would like to check if my understanding about Auth0 is correct.

HiI am following the steps generate API Credential, and run the content in the green frame directly, however it return error like below.  curl -X GET https://eu01.hub.streamsets.com/security/rest/v1/currentUser -H "Content-Type:application/json" -H "X-Requested-By:curl" -H "X-SS-REST-CALL:true" -H "X-SS-App-Component-Id: $CRED_ID" -H "X-SS-App-Auth-Token: $CRED_TOKEN" -i  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current                                 Dload  Upload   Total   Spent    Left  Speed100    19  100    19    0     0     22      0 --:--:-- --:--:-- --:--:--    22HTTP/1.1 403 Forbiddencontent-length: 19content-type: text/plaincontent-security-policy: object-src 'none';script-src 'self' https://cdn.cookielaw.org https://privacyportal.onetrust.com https://geolocation.onetrust.com https://app.intercom.io https://widget.intercom.io https://js.intercomcdn.com https://js.userflow.com https://cdn.userflow.com;style-src 'self' https://fonts.googleapis.com h

Hi I have pipeline and it has 1 runtime parameter about table name. I create a job on top of this pipeline, so it also has a runtime parameter, when create job from this pipeline, I provide a default parameter.I try to use python SDK to call the job, if I ran like below, the job will run with the default parameter(TEST_TABLE1) defined in pipeline and job, it is expected:job = sch.jobs.get(job_name = "MY_SDC_JOB")sch.start_job(job) However when I run like this, I want to overwrite the default runtime parameter, but it still using the default parameter(TEST_TABLE1) , how to make it run with the given parameter(TEST_TABLE2)?sch.start_job(job, TABLE_NAME=’TEST_TABLE2')

Organization 5c4e8df2-f1c4-11ee-9efd-39b2ca65500f already has 0 pipelines which is the maximum allowed. Please contact support : PIPELINE_STORE_07

Hi AllI have a trial account of Stream sets cloud.I am going through the streamsets-platform-fundamentals course.So i am currently trying to get hands-on , on the transformer engine.The Transformer Deployment and engines were installed and running yesterday, i also built 2 pipelines there. then i shut down my lab environment provided by StreamSets.When i login today and start the above lab environment, all the data collector engines start automatically, but the Transformer Engine does not start automatically.There is no option to start the transformer engine either. As you can see from the below screenshot.  If i select the Restart Engine or Shut Down Engine i get the below errorEngine cannot be reached. Verify that the engine is running. Because of this now i am not able to proceed further with the course, can someone help me on how to start the transformer engine after restarting my Lab Environment ?

Hi,In my pipeline, I am having a stream selector stage. I want to parameterize it and use the following expression for the condition:${record:value('/rating_text') == '${pipeline_rating_text}'}Here, pipeline_rating_text is my parameter that I have defined for my pipeline. The problem is that when I run the pipeline it does not work. If I use this expression ${record:value('/rating_text') == 'Excellent'}everything will be fine. Can somebody help me, please?

Hi,I have a JDBC connection for a database that is located in a docker container in my local machine. This connection works perfectly when I make a data collector pipeline.I installed a transformer engine in a docker container in my local machine (I have installed the external JDBC libraries). Then, I made a very simple pipeline to read from my database using this JDBC connection. I constantly get this error message that “[JDBC Table 1] Cannot connect to specified database: Communications link failure The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server. (JDBC_00)”.Can anyone help me with solving the problem?

HiI am trying to establishing a connection to jdbc:mysql://mysqldb:3306/zomato   but it is not working.I tried 2 differents approach:at first, I did not want to use Strigo so I was not limited to the 8 hours. So I run my engines in containers within my local machine. The engines are running fine on the Control Hub and I can create and run some pipelines. However, I would like to use the connection cited in one of the lab with mysqldb and zomato. I am not sure how to “install” the zomato db and reviews table locally? I have a container running mysql but from there i dont know what to.         If i try to create the connection using my local engine, i get then the below error.JDBC_00 - Cannot connect to specified database: com.mysql.cj.jdbc.exceptions.CommunicationsException: Communications link failure The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server. I could see a post with the same error message but it does

We are logging pipeline errors /orchestrated task errors in database . Now often, we see error messages like - “for actual error , open the logs “ as runtime errors.Per our solution , this gets logged into the database . We don’t want that ..Instead we want the actual errors to get logged . How can we do that ?

Hi Team,We have some pipelines, which has some configuration like below, How can I get the exact plain test value for these parameter property value? Also how and where can I set these kind of values?   

We‘ve set up a machine for Self-Managed Deployments (specifications below) with Docker as it’s said in the documentation. Machine specifications:Operating system: Linux (ubuntu 22.04) Disk size: 30GiB RAM: 4GiB Performance tier: P10 - 500 IOPS, 100MMB/sWe succesfully created some Data Collector and Transfomer engines. The problem occurs when we try to restart/run all of the engines at the same time. Server just freezes and eventually stops responding.If you look at the metrics of the machine, you can see that it stops usually when “Disk Bytes” peek. My first question is, what could cause this huge usage of resources to crash the server eventually? The next things are Transformer engines. In documentation it says:“Important: When using a Transformer engine that works with a Spark cluster, edit the transformer.base.http.url property on the Transformer Configuration tab. Uncomment the property and set it to the Transformer URL. For more information, see Granting the Spark Cluster Access t

Hello,  I was doing the StreamSets fundamentals training and I have got this notification from Strigo, saying “This course has ended”. Actually It hasn’t ended yet. How can I reaccess the account? I am Software AG employee. 

I am trying to regenerate api credentials and store them in some location(text file). I tried it in many ways but seems to be not working 

Issue:- While connecting to the MongoDB atlas, following error has been occurred:-MONGODB_22 - Mongo Query Error: Command failed with error 115 (CommandNotSupported): 'command insert is unsupported, correlationID = 179543fa79a5cba7e589f5b1' on server atlas-sql-653f56d6e2904d1155c97d0e-e54eg.a.query.mongodb.net:27017. The full response is {"ok": 0, "errmsg": "command insert is unsupported, correlationID = 179543fa79a5cba7e589f5b1", "code": 115, "codeName": "CommandNotSupported"} null Solution:-The query is failing due to the authentication issue. We have to provide a Username and Password in the JDBC string for the MongoDB Atlas.mongodb+srv://<USERNAME>:<PASSWORD>@<MONGODB_ATLAS_URL>/?retryWrites=true&w=majority