- - Knowledge base
Product Updates
Events

30-Day Free Trial: It’s Never Been Easier To Get Started With StreamSets

1 month ago

Home
Community overview
StreamSets Platform
StreamSets Academy

StreamSets Academy

StreamSets Academy course topics and questions

84 Topics
212 Replies

When you subscribe we will email you when there is a new topic in this category

84 Topics

Recently active

Most replies Most views Newest first

lakshmi_narayanan_tDiscovered Fame

posted in StreamSets Academy

field renamer

In my FIELD RENAMER processor my input 6 columns looks like (1'result.name' , 2'result.adress.street' , 3'result.adress.city' , 4'result.adress.state', 5'result.adress.zip' , 6'result.phone' ) BUT I WANT these output as a column name like(1’name' 2'street' 3city' 4'state' 5'zip' ,6'phone' )And also want to achieve these output in single field renamer processor

9 months ago

Santhosh KumarDiscovered Fame

asked in StreamSets Academy

Kafka connection not working ...

Hi Team,i using docker its work on JDBC connection but not working on a Kafka connection.

9 months ago

lakshmi_narayanan_tDiscovered Fame

asked in StreamSets Academy

how to saw the local fs in command terminal

below i post the pic ,anybody fix the issue .i dont known whats going wrong in pipeline or command terminal

9 months ago

SUDHANSHOOFan

asked in StreamSets Academy

Build a CDC Pipeline, unable to find - https://academy.streamsets.com/wp-content/uploads/2021/11/externalResources.zip

Hi Streamsets Team,I was using Streamsets Tutorial - Build a CDC Pipeline, Where its mentioned to use External Resource but unable to find, because this hyperlink does not map correctly.Could you please check and update.

9 months ago

SUDHANSHOOFan

asked in StreamSets Academy

java.lang.IllegalArgumentException: File not found: '/resources/transformer/db_token'

HI Team,Trying a sample Transformer pipeline - “Scala User-Defined Function”. And after creation I found this error with pipeline not with component whichever its used.CREATION_005 - Could not resolve implicit EL expression '${runtime:loadResource('db_token', false)}': com.streamsets.pipeline.api.el.ELEvalException: CTRCMN_0100 - Error evaluating expression ${runtime:loadResource('db_token', false)}: java.lang.IllegalArgumentException: File not found: '/resources/transformer/db_token'I just open the sample pipeline, didn't do anything with it. Does any one have idea regarding this?

9 months ago

tejas_surveRoadie

asked in StreamSets Academy

Related With REST API

I was working on StreamSet Learning Labs and after completing some learning tracks, currently I am stuck in REST API part and lab videos and instructions are not enough for me to understand it properly.So please guide me regarding these same things

9 months ago

chrisnrublocFan

asked in StreamSets Academy

SCD behaviour options missing !

HIi am trying to do the snowflake transformer course but the lab wants me to use an option in the scd processor thats not available !behaviour for new fields “Keep for inserted records and populate values from change data for updated records” all i see are remove from change and keep for newest and set null for previous ! is this the replaced version or is something different going to happen ?

9 months ago

SUDHANSHOOFan

asked in StreamSets Academy

Build a Transformer pipeline: org.apache.spark.sql.AnalysisException: Partition column `` not found in schema

Hi Team,While building a Transformer pipeline exercise, I was joining (Inner Join) two MySQL Tables from classicmodels database, employees & customers tables using condition “employees_employeeNumber = customers_salesRepEmployeeNumber”its showing below error: org.apache.spark.sql.AnalysisException: Partition column `` not found in schema struct<employees_lastName:string,employees_firstName:string,employees_email:string,employees_employeeNumber:int,employees_officeCode:string,customers_customerNumber:int,customers_salesRepEmployeeNumber:int,customers_creditLimit:decimal(10,2)>;

9 months ago

SUDHANSHOOFan

asked in StreamSets Academy

In Transformer, joining two small table but preview does not show any output for join processor

Hi All,in Transformer I’ve created a simple pipeline join 2 small MySQL table but join processor doesn’t show any output records, but previews show actual number of records to be saved in destination.While doing the same things with MySQL in backend actual record count is matched what previewed record is showing.

9 months ago

MelanieDiscovered Fame

asked in StreamSets Academy

JDBC Connection not working (Zomato)

HiI am trying to establishing a connection to jdbc:mysql://mysqldb:3306/zomato but it is not working.I tried 2 differents approach:at first, I did not want to use Strigo so I was not limited to the 8 hours. So I run my engines in containers within my local machine. The engines are running fine on the Control Hub and I can create and run some pipelines. However, I would like to use the connection cited in one of the lab with mysqldb and zomato. I am not sure how to “install” the zomato db and reviews table locally? I have a container running mysql but from there i dont know what to. If i try to create the connection using my local engine, i get then the below error.JDBC_00 - Cannot connect to specified database: com.mysql.cj.jdbc.exceptions.CommunicationsException: Communications link failure The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server. I could see a post with the same error message but it does

9 months ago

Santhosh KumarDiscovered Fame

asked in StreamSets Academy

streamsets White Belt certification

How many attempts using the white belt certification.

9 months ago

Dhanashri_BhateOpening Band

asked in StreamSets Academy

Is it possible to use Regex in attribute name?

Hello,Can I use regex patterns to filter records based on a field name pattern? Or use them in Field Type converter? For example: If a record has field names containing “cost” in them e.g. “Average Cost for two”, “Cost range”, “Parking Costs” etc. I want to send these records to a certain destination? Or if the field names including “cost” in them, use field type converter to convert the values to integer.I did not find any specific information on this in Docs. Played around with a few patterns. But even if some passed validation, they failed in preview. -Dhanashri

9 months ago

loveStremSetsFan

asked in StreamSets Academy

Strigo Lab - Streamsets Academy:DataOps Platform Fundamentals/Lab Build a JDBC Pipeline

I am following the instructions in the Lab: Build a JDBC Pipeline. As per the instructions, I go on the Right Hand Pane and click on Set Up → Engines and then click on the Engine(s) running on Strigo and then on the top tab I click on “External Resources” and then clicking on the side of the new left panel, I click on “External Libraries” and then the Plus (+) icon, for me to Upload the External Library, I select from the pull-down menu the Stage Library for JDBC and then it asks to Browse File. At that point the video states in a pop-up block that “Files required to complete this exercise are provided in the lab guide”.I could not find the file which in the video is named “mysql-connector-java-8.0.25.jar”. I search the Windows environment through the Folder Explorer under the “This PC” for that jar file, using *.jar as the search term, with only the securitypack jar file showing up.I then googled that I could download directly that named jar file from the MySQL website and downloade

9 months ago

yash4b4Fan

asked in StreamSets Academy

Error: Directory path does not exist: file:///data/store_retail , Lab: deploy transformer pipeline using sample

Hello,I am going through the Data Ops Platform Fundamentals course, and currently on Lab: Deploy a Transformer Engine with a Sample Pipeline. I have made sure my deployment is up and running, transformer engine is up. Created a pipeline by using sample pipeline as per lab steps. I am getting these validation errors (attached) - Error: Directory path does not exist: file:///data/store_retail Am I missing any steps? Some more info:I have not changed any parameter settings. The runtime env for the pipeline runtime conf - It shows origin_directory=/data/store_retailThe origin shows: Directory Path = file://${origin_directory}The transformer engine is running. It does not show any directory /data/store_retailbash-5.0# hostname46ab0f17ee4fbash-5.0# ls /data/transformerbash-5.0# ls /data/transformer/blobstore meteringSnapshot.json pipelines runInfo stats.jsoncontrol-hub-pushed.properties opt-stats.json

10 months ago

himabinduFan

asked in StreamSets Academy

while loging streamstets facing error

Hi Team,while logging to stream sets i am facing error with this email id himabindu_nukaraju@persistent.comi need to login my course. Please help on this

10 months ago

saikat_chatterjeeDiscovered Fame

asked in StreamSets Academy

Unable to upload the driver in the external libraries and facing same issue while importing a pipeline

In the build jdbc pipeline the step where we need to upload the driver in the external libraries after selecting the mysql-connector-java-8.0.26 when I am clicking on upload a red line in appearing as error but there nothing mentioned of which error it is.The same thing is happening while importing a pipeline, when I am browsing the zip file and clicking on the import button the same error as mentioned above is appearing.

10 months ago

ysunFan

asked in StreamSets Academy

Strigo Lab - Streamsets Academy: DataOps Platform Fundamentals/ Technical Issue with the lab

Hi all,I’m seeming to have issues today with the lab. It was wroking fine yesterday. However when I tried to login to StreamSet Cluster today, I got an error message saying: We have encountered a technical issue while setting up or your lab. Please contact your course provider, as they can best help you resolve this issue.Could you please help me with this issue? Many thanks.

10 months ago

loveStremSetsFan

asked in StreamSets Academy

Strigo Lab - Streamsets Academy:DataOps Platform Fundamentals/Setup a Deployment

When I am in the Strigo Lab simulator environment, it has 2 simultaneous sessions running currently (a Windows desktop and a “Streamset Cluster” [Ubuntu shell]). When I am doing Setup a Deployment part of the training, it requires me to copy and paste the into the Ubuntu shell a docker run command to Setup the Environment. This docker command is quite long (several rows) as it includes the Deployment ID and Tokens. However I can not copy paste from the Strigo Windows environment into the Ubuntu shell as it is two separate environments. The only way would be to MANUALLY type each and every character which would not work if I did a single typo. Is there any work around?

11 months ago

nandhiniFan

asked in StreamSets Academy

Partitioning by date while data ingestion

Hi communityHow can we configure the partitioning option while we transfer the table from one end to other end.I dont want the table to be overwrite into the destination layer.I want to write it as a append mode by date.How can we do that?

1 year ago

vasanthiFan

asked in StreamSets Academy

Not able to access more than one kafka topic

Hi Team,I am not able to access more than one kafka topic in multi topic consumer .And also i am not access the local directory file path in my local docker in windows version. I have created two topics while i try preview the pipeline i am able to view only one kafka topic data and not able to see another kafka topic data please some one help me in this.

1 year ago

SebastienMRoadie

asked in StreamSets Academy

Gettting error zomato directory does not exist

I’m following the DataOps Platform Fundamentals course and the Build a Pipeline chapter has you enter “/zomato” as a files directory in the configuration of a Directory source.However, when validating the pipeline I get the error: SPOOLDIR_12-Directory ‘/zomato’ does not exist: conf.spoolDir.Any solution for this?

1 year ago

Page 3 / 4

Badge winners

Sanjeevhas earned the badge Eager to help
AkshayJadhavhas earned the badge Eager to help
john.durkinhas earned the badge Eager to help
samhas earned the badge Eager to help
saleempothiwalahas earned the badge Eager to help

Show all badges

Terms & Conditions

Sign up

Already have an account? Login

Social Login

Username *

E-mail address *

What I do... *

Data Leader Data Architect Data Engineer Data Scientist Other

Company *

Country *

Zip Code *

Marketing Communications

Yes No

Password *

I have read and Agree to the Website Terms of Service and I have read and acknowledged the Privacy Policy.

loginBox.register.email_repeat

Login to the community

No account yet? Create an account

Social Login

Username or Email

Password

Remember me

Forgot password?

Enter your username or e-mail address. We'll send you an e-mail with instructions to reset your password.

Username or e-mail

Back to overview

Scanning file for viruses.

Sorry, we're still checking this file's contents to make sure it's safe to download. Please try again in a few minutes.

This file cannot be downloaded

Sorry, our virus scanner detected that this file isn't safe to download.