StreamSets Academy

StreamSets Academy course topics and questions

  • 67 Topics
  • 187 Replies

67 Topics

lakshmi_narayanan_t
Discovered Fame
lakshmi_narayanan_tDiscovered Fame
asked in StreamSets Academy

data type change by field mapper

if it possibile to  convert data type by  field mapper 

1
Bikram
Headliner
4 months ago
C
Fan
chrisnrublocFan
asked in StreamSets Academy

SCD behaviour options missing !

HIi am trying to do the snowflake transformer course but the lab wants me to use an option in the scd processor thats not available !behaviour for new fields “Keep for inserted records and populate values from change data for updated records” all i see are remove from change and keep for newest and set null for previous ! is this the replaced version or is something different going to happen ?

1
C
Fan
5 months ago
A
Roadie
AnkurRoadie
asked in StreamSets Academy

HTTP Client origin pagination config

Hi,If the field given in Result Field Path does not exist in the response, then we get an error, which is correct, but how to catch and handle that error to stop the pipeline smoothly.  

1
AkshayJadhav
StreamSets Employee
1 month ago
S
Fan
suresh_venkataFan
asked in StreamSets Academy

Unable to pause Strigo "The workspace cannot be paused"

Hi Team,I’m Unable to pass the strigo last 2hours. its Consumed 2hours time Already Please Help me on this.  

2
S
Fan
6 months ago
N
Fan
nandhiniFan
asked in StreamSets Academy

Partitioning by date while data ingestion

Hi communityHow can we configure the partitioning option while we transfer the table from one end to other end.I dont want the table to be overwrite into the destination layer.I want to write it as a append mode by date.How can we do that?  

1
Kirti
StreamSets Employee
10 months ago
B
Fan
btsshirt5Fan
asked in StreamSets Academy

How to parametrize field remover field names

I'm having trouble parameterizing field remover's field names. In reality, you can give specific field names like /type, /id, and /name. In my use case, however, I would like to parametrize these fields and read the data from a properties file, however I am unable to do so.

1
Bikram
Headliner
3 months ago
S
Fan
SUDHANSHOOFan
asked in StreamSets Academy

Build a CDC Pipeline, unable to find - https://academy.streamsets.com/wp-content/uploads/2021/11/externalResources.zip

Hi Streamsets Team,I was using Streamsets Tutorial - Build a CDC Pipeline, Where its mentioned to use External Resource but unable to find, because this hyperlink does not map correctly.Could you please check and update.

1
john.durkin
StreamSets Employee
5 months ago
D
Opening Band
Dhanashri_BhateOpening Band
asked in StreamSets Academy

Need more info on the two events for SFTP destination?

Hello,I am referring to this documentation : https://docs.streamsets.com/portal/platform-datacollector/latest/datacollector/UserGuide/Destinations/SFTP.html#concept_sgt_m2m_xhbThere are two Event types - Uses one of the following types:file-closed - Generated when the destination closes a file. wholeFileProcessed - Generated when the destination completes streaming a whole file.If I have to add an executor to move the file from this destination then to some other location upon receiving an event, which event I have to use? Are there any more steps between wholeFileProcessed and file-closed? Is there anything more after the event file-closed is triggered? Dhanashri

2
D
Opening Band
1 month ago
S
Fan
SUDHANSHOOFan
asked in StreamSets Academy

Build a Transformer pipeline: org.apache.spark.sql.AnalysisException: Partition column `` not found in schema

Hi Team,While building a Transformer pipeline exercise, I was joining (Inner Join) two MySQL Tables from classicmodels database, employees & customers tables using condition “employees_employeeNumber = customers_salesRepEmployeeNumber”its showing below error: org.apache.spark.sql.AnalysisException: Partition column `` not found in schema struct<employees_lastName:string,employees_firstName:string,employees_email:string,employees_employeeNumber:int,employees_officeCode:string,customers_customerNumber:int,customers_salesRepEmployeeNumber:int,customers_creditLimit:decimal(10,2)>;

0
S
Fan
5 months ago
S
Fan
suresh_venkataFan
asked in StreamSets Academy

How to Create Engine from Control Hub by using SDK Python ?

Hi Team,we Observed, After the Creation Of Deployment using SDK Python Version 5.1.0, we have to Run the “Get Install Script” in Our Local VM for the Creating Engine. Except This Step How to Create the Engine By using SDK Python? can anyone Please help me on this. Thank YouSuresh_venkata

0
S
Fan
4 months ago
A
Roadie
AnkurRoadie
asked in StreamSets Academy

Metric Rule giving Alert on False Condition.

We have a Metric Alert in a JDBC Producer Input Counter with below Condition:${value() < 35}The above rule with “less than operator” is giving alert in both the conditions True or False. If we change the operator from “less than “< to “greater than”>. This it is behaving as expected.${value() > 35} Please provide the suggestion or resolution for ${value() < 35} to work as expected i.e. only on True condition.        

2
A
Roadie
1 month ago
R
Fan
rohit6488Fan
asked in StreamSets Academy

Missing Fragment used in Academy Course for CDC from Mysql to Snowflake

Hi All,i am looking for Fragment which has been used in Academy Course for CDC Pipeline Demo, they have mentioned they will share with us. But i cant find it anywhere. Can someone help me to share that fragment. Regards,Rohit

3
john.durkin
StreamSets Employee
1 day ago
D
Opening Band
Dhanashri_BhateOpening Band
asked in StreamSets Academy

Selection of cloud libraries when pipeline needs 2 different cloud environments

Hello,We’re trying to create a transformer pipeline with Origin: AmazonS3 , Destination: Azure ADLS and compute is on Azure Databricks. We tried to break is down into smaller pieces.S3->ADLS pipeline with Streamsets own cluster worked fine. Random data source to ADLS with Databricks compute worked fine. But when we try to bring in all 3 components together, we are getting stage library specific errors which are conflicting. We have tried different combinations from the list in the attached picture.In this particular scenario, what are the stage libraries we need to choose?  

1
D
Opening Band
3 months ago
lakshmi_narayanan_t
Discovered Fame
lakshmi_narayanan_tDiscovered Fame
asked in StreamSets Academy

how to get data time format from mongodb to mysql .

the json data in mongodb had  a date format as a  timestamp in value  but during  process only date shown not timestamp show. how to fix below is my mongodb data:- but  inside of data updatedAt column has timestamp value but in streamsets not shown.how to get timestamp value from mongodb origin.

1
AkshayJadhav
StreamSets Employee
2 months ago
A
Roadie
AnkurRoadie
asked in StreamSets Academy

Connect to Dynamic 365

Hi,As per the requirement I need to connect Streamsets to Dynamic 365 db.DO we have this connector and if not how we can connect it?  Regards,Ankur

0
A
Roadie
4 months ago
tamilarasup
Discovered Fame
tamilarasupDiscovered Fame
posted in StreamSets Academy

errorhandling

Anyone can provide the topics and some use case OR real-time scenario about the error handling SDC, SCH that will be very useful for my belt preparation.

0
tamilarasup
Discovered Fame
4 months ago
S
Fan
SUDHANSHOOFan
asked in StreamSets Academy

In Transformer, joining two small table but preview does not show any output for join processor

Hi All,in Transformer I’ve created a simple pipeline join 2 small MySQL table but join processor doesn’t show any output records, but previews show actual number of records to be saved in destination.While doing the same things with MySQL in backend actual record count is matched what previewed record is showing. 

0
S
Fan
5 months ago

Badge winners

Show all badges
Terms & Conditions