- - Knowledge base
Product Updates
Groups
Join Build It With Brenna

Build it with Brenna: Transformer for Snowflake

(Wed, 31 May, 17:00)

Home
Community overview
Platform Q&A
StreamSets Academy

StreamSets Academy

StreamSets Academy course topics and questions

67 Topics
186 Replies

When you subscribe we will email you when there is a new topic in this category

67 Topics

Most views

Recently active Most replies Newest first

SebastienMRoadie

asked in StreamSets Academy

Gettting error zomato directory does not exist

I’m following the DataOps Platform Fundamentals course and the Build a Pipeline chapter has you enter “/zomato” as a files directory in the configuration of a Directory source.However, when validating the pipeline I get the error: SPOOLDIR_12-Directory ‘/zomato’ does not exist: conf.spoolDir.Any solution for this?

1 year ago

ufmurphyOpening Band

asked in StreamSets Academy

john.durkinStreamSets Employee

posted in StreamSets Academy

"DataOps Platform Fundamentals" - questions from the video course

Hi -Welcome to the StreamSets Academy area of Community!Please post questions related to the self-paced video course “DataOps Platform Fundamentals”. Please use the tag “DataOps Platform Fundamentals Course” when creating your question under the “StreamSets Academy” category. Thank you!

1 year ago

MelanieDiscovered Fame

asked in StreamSets Academy

JDBC Connection not working (Zomato)

HiI am trying to establishing a connection to jdbc:mysql://mysqldb:3306/zomato but it is not working.I tried 2 differents approach:at first, I did not want to use Strigo so I was not limited to the 8 hours. So I run my engines in containers within my local machine. The engines are running fine on the Control Hub and I can create and run some pipelines. However, I would like to use the connection cited in one of the lab with mysqldb and zomato. I am not sure how to “install” the zomato db and reviews table locally? I have a container running mysql but from there i dont know what to. If i try to create the connection using my local engine, i get then the below error.JDBC_00 - Cannot connect to specified database: com.mysql.cj.jdbc.exceptions.CommunicationsException: Communications link failure The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server. I could see a post with the same error message but it does

5 months ago

loveStremSetsFan

asked in StreamSets Academy

Strigo Lab - Streamsets Academy:DataOps Platform Fundamentals/Lab Build a JDBC Pipeline

I am following the instructions in the Lab: Build a JDBC Pipeline. As per the instructions, I go on the Right Hand Pane and click on Set Up → Engines and then click on the Engine(s) running on Strigo and then on the top tab I click on “External Resources” and then clicking on the side of the new left panel, I click on “External Libraries” and then the Plus (+) icon, for me to Upload the External Library, I select from the pull-down menu the Stage Library for JDBC and then it asks to Browse File. At that point the video states in a pop-up block that “Files required to complete this exercise are provided in the lab guide”.I could not find the file which in the video is named “mysql-connector-java-8.0.25.jar”. I search the Windows environment through the Folder Explorer under the “This PC” for that jar file, using *.jar as the search term, with only the securitypack jar file showing up.I then googled that I could download directly that named jar file from the MySQL website and downloade

5 months ago

kapil24nagarFan

asked in StreamSets Academy

Lab: Build a JDBC Pipeline (zomato database is not found)

where can I get the SQL used to create the review tables under zomato database? or is there a way/script by which it can be made available in mysqldb docker?

1 year ago

Dhanashri_BhateOpening Band

asked in StreamSets Academy

Error: Directory path does not exist: file:///data/store_retail , Lab: deploy transformer pipeline using sample

Hello,I am going through the Data Ops Platform Fundamentals course, and currently on Lab: Deploy a Transformer Engine with a Sample Pipeline. I have made sure my deployment is up and running, transformer engine is up. Created a pipeline by using sample pipeline as per lab steps. I am getting these validation errors (attached) - Error: Directory path does not exist: file:///data/store_retail Am I missing any steps? Some more info:I have not changed any parameter settings. The runtime env for the pipeline runtime conf - It shows origin_directory=/data/store_retailThe origin shows: Directory Path = file://${origin_directory}The transformer engine is running. It does not show any directory /data/store_retailbash-5.0# hostname46ab0f17ee4fbash-5.0# ls /data/transformerbash-5.0# ls /data/transformer/blobstore meteringSnapshot.json pipelines runInfo stats.jsoncontrol-hub-pushed.properties opt-stats.json

6 months ago

Praveen PFan

asked in StreamSets Academy

how to write avro file in wholefile format from kafka to local fs?

i trying to write avro format as wholefile format from kafka to localfs but i got error.below i share the pipeline ,can you please suggest how to done this

2 months ago

MadhumithaFan

asked in StreamSets Academy

Any references for preparation of the Streamsets White belt Exam

Can I get any references for the preparation for Streamsets white belt exam other than the Streamsets Academy content…?

2 months ago

SebastienMRoadie

asked in StreamSets Academy

In Data Collector Fundamental course, Day 2, Lab - Data Drift, I get error 'HIVE_38 - unexpected authorization method SIMPLE, expected KERBEROS' when validating pipeline

Course and chapter url: https://academy.streamsets.com/courses/streamsets-data-collector/lessons/data-drift/topic/data-drift/ Screenshot of errors attached.

1 year ago

Santhosh KumarDiscovered Fame

asked in StreamSets Academy

streamsets White Belt certification

How many attempts using the white belt certification.

5 months ago

Santhosh KumarDiscovered Fame

asked in StreamSets Academy

file transfer from origin to destination

hi,I want to transfer multiple files from origin to destination like any other destination ,how can I achieve these use case .any one can suggest .

3 months ago

lakshmi_narayanan_tDiscovered Fame

asked in StreamSets Academy

data collector vs transformer

Anyone can provide a quick overview of pros and cons of data-collector vs transformer apart from streamsets document also.

4 months ago

zhanghaoFan

asked in StreamSets Academy

activate streamset fail

Hello everyone, I'm a new user, and I got an installation package, but I can't activate it. I can access the community normally, but I can't log in normally and activate streamset. Who can give me some advice? Thank you!

1 year ago

Santhosh KumarDiscovered Fame

asked in StreamSets Academy

Reattempt Test White Belt

Hi Team, Good Morning, as I participated in the test(White belt), unfortunately I am under scored so, i had another test within the month ,so I requested you to provide a coupon code or detailed information about the procedure for reattempt.

4 months ago

loveStremSetsFan

asked in StreamSets Academy

Strigo Lab - Streamsets Academy:DataOps Platform Fundamentals/Setup a Deployment

When I am in the Strigo Lab simulator environment, it has 2 simultaneous sessions running currently (a Windows desktop and a “Streamset Cluster” [Ubuntu shell]). When I am doing Setup a Deployment part of the training, it requires me to copy and paste the into the Ubuntu shell a docker run command to Setup the Environment. This docker command is quite long (several rows) as it includes the Deployment ID and Tokens. However I can not copy paste from the Strigo Windows environment into the Ubuntu shell as it is two separate environments. The only way would be to MANUALLY type each and every character which would not work if I did a single typo. Is there any work around?

8 months ago

Dhanashri_BhateOpening Band

asked in StreamSets Academy

Handling multiple destination tables in data copy/CDC pipeline

Hello,Scenario: Multiple tables from source DB are to be replicated in the target DB. Whatever changes happen in source tables should be reflected in the target tables. When a record gets deleted, an entry with all columns of the record must be saved to a file. Both source and target are postgres DBs.I have implemented this for a single table. Created one pipeline that simply performs one time data copy from source to target. Then created a CDC pipeline like in the picture below. I am using the JDBC Producer destination. Though this works as designed for single table, I am not sure if this is the right approach when working with multiple tables. Though the CDC origin accepts the table name pattern “source_tab%” , the JDBC Producer destination seems to allow just one table name. i.e. though CDC is giving me all updates for all records, I can only use one table name at a time in the JDBC producer destination.I have tried using EL in the table name e.g. ${record:attribute(/'change.table'

5 months ago

Dhanashri_BhateOpening Band

asked in StreamSets Academy

Email notifications for alerts are not working

Hello,I have a pipeline that reads a log file and sends alerts when CRITICAL category message is seen.The alert works fine, I get notification on Control Hub UI, I am also getting Webhook notifications.But the email notification does not seem to work.Settings done in the Rules section for email: In the Data Rule - Send Email = true, In the notifications tab a valid email id is given.Is there any other setting that overrides this? Did not see any user level preference setting option for this. Thanks,Dhanashri

5 months ago

Dhanashri_BhateOpening Band

asked in StreamSets Academy

Where can I find Transpose processor documentation?

Hello,I am looking for documentation for the transpose processor in transformer pipelines.I searched here : https://docs.streamsets.com/portal/platform-transformer/latest/search.html?searchQuery=transpose but did not find it.Can someone share the link to the relevant documentation?-Dhanashri

2 months ago

ufmurphyOpening Band

asked in StreamSets Academy

Kafka not running in lab

Hey guys, I am running through the Academy training and seem to have hit a snag. I am not real knowledgeable about Docker/Kubernetes, so I thought I’d ask here. Is Kafka running in the labs? I am getting timeout errors when I try to connect from Control Hub.https://academy.streamsets.com/courses/dataops-platform-fundamentals/lessons/build-a-kafka-pipeline/topic/lab-build-a-kafka-pipeline-2/ Any help is appreciated. Jim

1 year ago

ysunFan

asked in StreamSets Academy

Strigo Lab - Streamsets Academy: DataOps Platform Fundamentals/ Technical Issue with the lab

Hi all,I’m seeming to have issues today with the lab. It was wroking fine yesterday. However when I tried to login to StreamSet Cluster today, I got an error message saying: We have encountered a technical issue while setting up or your lab. Please contact your course provider, as they can best help you resolve this issue.Could you please help me with this issue? Many thanks.

7 months ago

brian parkerFan

asked in StreamSets Academy

Issue with Streamsets Tutorial

In the setup deployment step of the streamsets tutorial, when I run the update-nodes.sh script in the strigo environment, I get the following error every time:Error response from daemon: enpoint with name wonderful_volhard already exists in network streamsets-coreError response from daemon: enpoint with name wonderful_volhard already exists in network streamsets-integrationsError response from daemon: enpoint with name wonderful_volhard already exists in network streamsets-cookedAlso, the tutorial says I should see 3 engines in the control hub, but I only see 2.

1 year ago

ufmurphyOpening Band

asked in StreamSets Academy

DataOps Platform Fundamentals Course - Data Drift missing

Hey guys, In the DataOps Platform Fundamentals Course, the video on Data Drift is missing.https://academy.streamsets.com/courses/dataops-platform-fundamentals/lessons/data-drift-2/ Same with CI/CD:https://academy.streamsets.com/courses/dataops-platform-fundamentals/lessons/implementing-pipeline-ci-cd/ Is there a video, or should I mark it complete and move on? Jim

1 year ago

Page 1 / 3

Badge winners

AkshayJadhavhas earned the badge Eager to help
john.durkinhas earned the badge Eager to help
samhas earned the badge Eager to help
saleempothiwalahas earned the badge Eager to help
Bikramhas earned the badge Community Champion 🏅

Show all badges

Terms & Conditions

Sign up

Already have an account? Login

Social Login

Username *

E-mail address *

What I do... *

Data Leader Data Architect Data Engineer Data Scientist Other

Company *

Country *

Zip Code *

Marketing Communications

Yes No

Password *

I have read and Agree to the Website Terms of Service and I have read and acknowledged the Privacy Policy.

loginBox.register.email_repeat

Login to the community

No account yet? Create an account

Social Login

Username or Email

Password

Remember me

Forgot password?

Enter your username or e-mail address. We'll send you an e-mail with instructions to reset your password.

Username or e-mail

Back to overview

Scanning file for viruses.

Sorry, we're still checking this file's contents to make sure it's safe to download. Please try again in a few minutes.

This file cannot be downloaded

Sorry, our virus scanner detected that this file isn't safe to download.