Skip to main content

StreamSets Academy

StreamSets Academy course topics and questions

  • 105 Topics
  • 254 Replies
105 Topics
M
Roadie
mr_maniiRoadie
posted in StreamSets Academy
Streamset whitebelt certification

Hi, Im planning to take up exams for Streamset whitebelt certification in the coming days. Is there any inputs that anyone can provide around how to attempt for the same?

3
saleempothiwala
Headliner
1 year ago
SebastienM
Roadie
SebastienMRoadie
asked in StreamSets Academy
Gettting error zomato directory does not exist

I’m following the DataOps Platform Fundamentals course and the Build a Pipeline chapter has you enter “/zomato” as a files directory in the configuration of a Directory source.However, when validating the pipeline I get the error: SPOOLDIR_12-Directory ‘/zomato’ does not exist: conf.spoolDir.Any solution for this?

9
hpppppppp
Fan
18 days ago
A
Roadie
AtousaRoadie
asked in StreamSets Academy
[JDBC Table 1] Cannot connect to specified database: Communications link failure The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server. (JDBC_00)

Hi,I have a JDBC connection for a database that is located in a docker container in my local machine. This connection works perfectly when I make a data collector pipeline.I installed a transformer engine in a docker container in my local machine (I have installed the external JDBC libraries). Then, I made a very simple pipeline to read from my database using this JDBC connection. I constantly get this error message that “[JDBC Table 1] Cannot connect to specified database: Communications link failure The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server. (JDBC_00)”.Can anyone help me with solving the problem?

8
A
Roadie
4 months ago
ufmurphy
Opening Band
ufmurphyOpening Band
asked in StreamSets Academy
Runtime parameters: reset origin and start

Hey Community, I see the option in the Streamsets canvas:Start pipeline Reset origin and start Start with parameters Is it possible to combine the last 2 options?Reset origin and start with parameters

3
ufmurphy
Opening Band
2 years ago
SebastienM
Roadie
SebastienMRoadie
asked in StreamSets Academy
Getting error: VALIDATION_0011 - Stage has open output streams

I get this error, following the steps in the lecture “DataOps Platform Fundamentals”, chapter “Build a Pipeline”, lab “Lab: Build a Pipeline”.When clicking on the error, it takes me to the Stream Selector module, Configuration, General tab. There is nothing highlighted in that section and all seems in working order to me.Could you please advise?

11
Drew Kreiger
Rock star
2 years ago
M
Fan
MadhumithaFan
asked in StreamSets Academy
Any references for preparation of the Streamsets White belt Exam

Can I get any references for the preparation for Streamsets white belt exam other than the Streamsets Academy content…?

4
Praveen P
Fan
1 year ago
M
Discovered Fame
MelanieDiscovered Fame
asked in StreamSets Academy
JDBC Connection not working (Zomato)

HiI am trying to establishing a connection to jdbc:mysql://mysqldb:3306/zomato   but it is not working.I tried 2 differents approach:at first, I did not want to use Strigo so I was not limited to the 8 hours. So I run my engines in containers within my local machine. The engines are running fine on the Control Hub and I can create and run some pipelines. However, I would like to use the connection cited in one of the lab with mysqldb and zomato. I am not sure how to “install” the zomato db and reviews table locally? I have a container running mysql but from there i dont know what to.         If i try to create the connection using my local engine, i get then the below error.JDBC_00 - Cannot connect to specified database: com.mysql.cj.jdbc.exceptions.CommunicationsException: Communications link failure The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server. I could see a post with the same error message but it does

19
A
Roadie
5 months ago
john.durkin
StreamSets Employee
john.durkinStreamSets Employee
posted in StreamSets Academy
"DataOps Platform Fundamentals" - questions from the video course

Hi -Welcome to the StreamSets Academy area of Community!Please post questions related to the self-paced video course “DataOps Platform Fundamentals”. Please use the tag “DataOps Platform Fundamentals Course” when creating your question under the “StreamSets Academy” category. Thank you! 

0
john.durkin
StreamSets Employee
2 years ago
S
Discovered Fame
Santhosh KumarDiscovered Fame
asked in StreamSets Academy
Reattempt Test White Belt

Hi Team, Good Morning, as I participated in the test(White belt), unfortunately I am under scored so, i had another test within the month ,so I requested you to provide a coupon code or detailed information about the procedure for reattempt.

3
Bikram
Headliner
1 year ago
Praveen P
Fan
Praveen PFan
asked in StreamSets Academy
how to write avro file in wholefile format from kafka to local fs?

i trying to write avro format as wholefile format from kafka to localfs but i got error.below  i share the pipeline ,can you please suggest how to done this 

13
Praveen P
Fan
1 year ago
S
Discovered Fame
Santhosh KumarDiscovered Fame
asked in StreamSets Academy
streamsets White Belt certification

How many attempts using the white belt certification.

1
Bikram
Headliner
1 year ago
lakshmi_narayanan_t
Discovered Fame
lakshmi_narayanan_tDiscovered Fame
asked in StreamSets Academy
data collector vs transformer

Anyone can provide a quick overview of pros and cons of data-collector vs transformer apart from streamsets document also.

1
saleempothiwala
Headliner
1 year ago
loveStremSets
Fan
loveStremSetsFan
asked in StreamSets Academy
Strigo Lab - Streamsets Academy:DataOps Platform Fundamentals/Lab Build a JDBC Pipeline

I am following the instructions in the Lab: Build a JDBC Pipeline. As per the instructions, I go on the Right Hand Pane and click on Set Up → Engines and then click on the Engine(s) running on Strigo and then on the top tab I click on “External Resources” and then clicking on the side of the new left panel, I click on “External Libraries” and then the Plus (+) icon, for me to Upload the External Library, I select from the pull-down menu the Stage Library for JDBC and then it asks to Browse File.  At that point the video states in a pop-up block that “Files required to complete this exercise are provided in the lab guide”.I could not find the file which in the video is named “mysql-connector-java-8.0.25.jar”.  I search the Windows environment through the Folder Explorer under the “This PC” for that jar file, using *.jar as the search term, with only the securitypack jar file showing up.I then googled that I could download directly that named jar file from the MySQL website and downloade

4
M
Discovered Fame
1 year ago
D
Opening Band
Dhanashri_BhateOpening Band
asked in StreamSets Academy
Handling multiple destination tables in data copy/CDC pipeline

Hello,Scenario:  Multiple tables from source DB are to be replicated in the target DB. Whatever changes happen in source tables should be reflected in the target tables. When a record gets deleted, an entry with all columns of the record must be saved to a file. Both source and target are postgres DBs.I have implemented this for a single table. Created one pipeline that simply performs one time data copy from source to target. Then created a CDC pipeline like in the picture below. I am using the JDBC Producer destination. Though this works as designed for single table, I am not sure if this is the right approach when working with multiple tables. Though the CDC origin accepts the table name pattern “source_tab%” , the JDBC Producer destination seems to allow just one table name. i.e. though CDC is giving me all updates for all records, I can only use one table name at a time in the JDBC producer destination.I have tried using EL in the table name e.g. ${record:attribute(/'change.table'

2
saleempothiwala
Headliner
1 year ago
D
Opening Band
Dhanashri_BhateOpening Band
asked in StreamSets Academy
Error: Directory path does not exist: file:///data/store_retail , Lab: deploy transformer pipeline using sample

Hello,I am going through the Data Ops Platform Fundamentals course, and currently on Lab: Deploy a Transformer Engine with a Sample Pipeline. I have made sure my deployment is up and running, transformer engine is up. Created a pipeline by using sample pipeline as per lab steps. I am getting these validation errors (attached) - Error: Directory path does not exist: file:///data/store_retail Am I missing any steps? Some more info:I have not changed any parameter settings.  The runtime env for the pipeline runtime conf - It shows origin_directory=/data/store_retailThe origin shows: Directory Path = file://${origin_directory}The transformer engine is running. It does not show any directory /data/store_retailbash-5.0# hostname46ab0f17ee4fbash-5.0# ls /data/transformerbash-5.0# ls /data/transformer/blobstore                      meteringSnapshot.json          pipelines                      runInfo                        stats.jsoncontrol-hub-pushed.properties  opt-stats.json                

5
Bikram
Headliner
1 year ago
D
Opening Band
Dhanashri_BhateOpening Band
asked in StreamSets Academy
Email notifications for alerts are not working

Hello,I have a pipeline that reads a log file and sends alerts when CRITICAL category message is seen.The alert works fine, I get notification on Control Hub UI, I am also getting Webhook notifications.But the email notification does not seem to work.Settings done in the Rules section for email: In the Data Rule - Send Email = true, In the notifications tab a valid email id is given.Is there any other setting that overrides this? Did not see any user level preference setting option for this. Thanks,Dhanashri 

3
Bikram
Headliner
1 year ago
Ritantara
Fan
RitantaraFan
asked in StreamSets Academy
How to fetch actual error message in a Streamsets pipeline?

We are logging pipeline errors /orchestrated task errors in database . Now often, we see error messages like - “for actual error , open the logs “ as runtime errors.Per our solution , this gets logged into the database . We don’t want that ..Instead we want the actual errors to get logged . How can we do that ?

7
Bikram
Headliner
5 months ago
loveStremSets
Fan
loveStremSetsFan
asked in StreamSets Academy
Strigo Lab - Streamsets Academy:DataOps Platform Fundamentals/Setup a Deployment

When I am in the Strigo Lab simulator environment, it has 2 simultaneous sessions running currently (a Windows desktop and a “Streamset Cluster” [Ubuntu shell]). When I am doing Setup a Deployment part of the training, it requires me to copy and paste the into the Ubuntu shell a docker run command to Setup the Environment.  This docker command is quite long (several rows) as it includes the Deployment ID and Tokens.  However I can not copy paste from the Strigo Windows environment into the Ubuntu shell as it is two separate environments. The only way would be to MANUALLY type each and every character which would not work if I did a single typo.  Is there any work around?

5
loveStremSets
Fan
1 year ago
S
Fan
SUDHANSHOOFan
asked in StreamSets Academy
java.lang.IllegalArgumentException: File not found: '/resources/transformer/db_token'

HI Team,Trying a sample Transformer pipeline - “Scala User-Defined Function”. And after creation I found this error with pipeline not with component whichever its used.CREATION_005 - Could not resolve implicit EL expression '${runtime:loadResource('db_token', false)}': com.streamsets.pipeline.api.el.ELEvalException: CTRCMN_0100 - Error evaluating expression ${runtime:loadResource('db_token', false)}: java.lang.IllegalArgumentException: File not found: '/resources/transformer/db_token'I just open the sample pipeline, didn't do anything with it. Does any one have idea regarding this?

1
Rafael Lopez
Fan
1 year ago
kapil24nagar
Fan
kapil24nagarFan
asked in StreamSets Academy
Lab: Build a JDBC Pipeline (zomato database is not found)

where can I get the SQL used to create the review tables under zomato database? or is there a way/script by which it can be made available in mysqldb docker?

1
saad
StreamSets Employee
2 years ago
nikkichow
Fan
nikkichowFan
asked in StreamSets Academy
Handling the error records in the same Kafka topic

Hello All,we have an usecase where we’ll read the json message from the kafka topic and call the api using the http client stage for the data load. Here, when any stage sees any error we want to send it back to the same source kafka topic from where we are reading the json messages  . But we see the error records are sent to the kafka topic in sdc format and since our pipeline is configured to read json data format it’s failing. Is there any way we can route the error records to same source topic in json format instead of SDC format without creating a new failure topic and streamsets pipeline to handle the error records?

3
Sanjeev
StreamSets Employee
1 year ago
S
Discovered Fame
Santhosh KumarDiscovered Fame
asked in StreamSets Academy
file transfer from origin to destination

hi,I want to transfer multiple files from origin to  destination like any other destination ,how can I achieve these use case .any one can suggest  .

8
Bikram
Headliner
1 year ago
D
Opening Band
Dhanashri_BhateOpening Band
asked in StreamSets Academy
Where can I find Transpose processor documentation?

Hello,I am looking for documentation for the transpose processor in transformer pipelines.I searched here : https://docs.streamsets.com/portal/platform-transformer/latest/search.html?searchQuery=transpose but did not find it.Can someone share the link to the relevant documentation?-Dhanashri 

10
Bikram
Headliner
1 year ago
S
Discovered Fame
Santhosh KumarDiscovered Fame
asked in StreamSets Academy
Parquet file Data collector

Hi teami using streamset control hub 3.x ,if possible to read and write parquet file in data collector ? 

2
saleempothiwala
Headliner
1 year ago
SebastienM
Roadie
SebastienMRoadie
asked in StreamSets Academy
In Data Collector Fundamental course, Day 2, Lab - Data Drift, I get error 'HIVE_38 - unexpected authorization method SIMPLE, expected KERBEROS' when validating pipeline

Course and chapter url: https://academy.streamsets.com/courses/streamsets-data-collector/lessons/data-drift/topic/data-drift/ Screenshot of errors attached. 

2
A
Fan
2 years ago

Badge winners

Show all badges
Terms & Conditions