- - Knowledge base
Product Updates
Groups
Join Build It With Brenna

Build it With Brenna: Hands-on Workshop with a MySQL Origin and a Snowflake Destination

(Wed, 29 Mar, 17:00)

Home
Community overview
Platform Q&A
StreamSets Academy

StreamSets Academy

StreamSets Academy course topics and questions

61 Topics
172 Replies

When you subscribe we will email you when there is a new topic in this category

61 Topics

Recently active

Most replies Most views Newest first

Dhanashri_BhateOpening Band

asked in StreamSets Academy

Santhosh KumarDiscovered Fame

asked in StreamSets Academy

file transfer from origin to destination

hi,I want to transfer multiple files from origin to destination like any other destination ,how can I achieve these use case .any one can suggest .

1 month ago

Santhosh KumarDiscovered Fame

asked in StreamSets Academy

multiple files transfer to directory

i had a multiple json files in same directory ,what i wanted to do is just transfer the files like copy and paste to another directory as whole file format .the origin directory contains two json files these two files copy and paste it to another directoryhow to give file name expression

1 month ago

btsshirt5Fan

asked in StreamSets Academy

How to parametrize field remover field names

I'm having trouble parameterizing field remover's field names. In reality, you can give specific field names like /type, /id, and /name. In my use case, however, I would like to parametrize these fields and read the data from a properties file, however I am unable to do so.

1 month ago

lakshmi_narayanan_tDiscovered Fame

asked in StreamSets Academy

jdbc tee processor not write data in microsoft sql server for control hub 3.x

hi,I am facing issue , to write a data in Microsoft Sql server database by using JDBC tee processor but its not work .I tried the identity insert on and off in database level .if it Possibile or not .suggest any processor to achieve this .ERRORdemo pipelinejdbc tee configuration

1 month ago

Santhosh KumarDiscovered Fame

asked in StreamSets Academy

active file ,file tail

hi,I want to use the file tail origin but i didn’t known the active file path usage in sdc. anyone can provide the configuration of file tail and active file path .

1 month ago

Dhanashri_BhateOpening Band

asked in StreamSets Academy

Selection of cloud libraries when pipeline needs 2 different cloud environments

Hello,We’re trying to create a transformer pipeline with Origin: AmazonS3 , Destination: Azure ADLS and compute is on Azure Databricks. We tried to break is down into smaller pieces.S3->ADLS pipeline with Streamsets own cluster worked fine. Random data source to ADLS with Databricks compute worked fine. But when we try to bring in all 3 components together, we are getting stage library specific errors which are conflicting. We have tried different combinations from the list in the attached picture.In this particular scenario, what are the stage libraries we need to choose?

1 month ago

suresh_venkataFan

asked in StreamSets Academy

How to Create Engine from Control Hub by using SDK Python ?

Hi Team,we Observed, After the Creation Of Deployment using SDK Python Version 5.1.0, we have to Run the “Get Install Script” in Our Local VM for the Creating Engine. Except This Step How to Create the Engine By using SDK Python? can anyone Please help me on this. Thank YouSuresh_venkata

1 month ago

Santhosh KumarDiscovered Fame

asked in StreamSets Academy

Rest API

how to write rest api in python code..

2 months ago

Santhosh KumarDiscovered Fame

asked in StreamSets Academy

Parquet file Data collector

Hi teami using streamset control hub 3.x ,if possible to read and write parquet file in data collector ?

2 months ago

lakshmi_narayanan_tDiscovered Fame

asked in StreamSets Academy

data type change by field mapper

if it possibile to convert data type by field mapper

2 months ago

tamilarasupFan

posted in StreamSets Academy

errorhandling

Anyone can provide the topics and some use case OR real-time scenario about the error handling SDC, SCH that will be very useful for my belt preparation.

2 months ago

lakshmi_narayanan_tDiscovered Fame

asked in StreamSets Academy

cdc possibile are not SQLSERVER

it is possible to Achieve CDC in origin JDBC MULTI TABLE consumer to JDBC Producer Pipeline

2 months ago

AnkurRoadie

asked in StreamSets Academy

Connect to Dynamic 365

Hi,As per the requirement I need to connect Streamsets to Dynamic 365 db.DO we have this connector and if not how we can connect it? Regards,Ankur

2 months ago

lakshmi_narayanan_tDiscovered Fame

asked in StreamSets Academy

datadrift

Difference in data rule vs data drift rule with some scenario, use case?

2 months ago

Santhosh KumarDiscovered Fame

asked in StreamSets Academy

Reattempt Test White Belt

Hi Team, Good Morning, as I participated in the test(White belt), unfortunately I am under scored so, i had another test within the month ,so I requested you to provide a coupon code or detailed information about the procedure for reattempt.

2 months ago

lakshmi_narayanan_tDiscovered Fame

asked in StreamSets Academy

data collector vs transformer

Anyone can provide a quick overview of pros and cons of data-collector vs transformer apart from streamsets document also.

2 months ago

AnkurRoadie

asked in StreamSets Academy

Different types of Events and their names

Hi,In the StreamSets documentation we found very lesser Event Types like:no-more-data Pipeline start Pipeline StopOther than these we are not able to able any other Event Type if it exists in the documentation.Like, if I want to generate an event after the record count is 5000, what should be the event name to put in the below type of equation:${record:eventType()=='no-more-data'} Please share other Event types and their names for the usage.

2 months ago

Dhanashri_BhateOpening Band

asked in StreamSets Academy

Handling multiple destination tables in data copy/CDC pipeline

Hello,Scenario: Multiple tables from source DB are to be replicated in the target DB. Whatever changes happen in source tables should be reflected in the target tables. When a record gets deleted, an entry with all columns of the record must be saved to a file. Both source and target are postgres DBs.I have implemented this for a single table. Created one pipeline that simply performs one time data copy from source to target. Then created a CDC pipeline like in the picture below. I am using the JDBC Producer destination. Though this works as designed for single table, I am not sure if this is the right approach when working with multiple tables. Though the CDC origin accepts the table name pattern “source_tab%” , the JDBC Producer destination seems to allow just one table name. i.e. though CDC is giving me all updates for all records, I can only use one table name at a time in the JDBC producer destination.I have tried using EL in the table name e.g. ${record:attribute(/'change.table'

2 months ago

Page 1 / 3

Badge winners

AkshayJadhavhas earned the badge Eager to help
john.durkinhas earned the badge Eager to help
samhas earned the badge Eager to help
saleempothiwalahas earned the badge Eager to help
Bikramhas earned the badge Community Champion 🏅

Show all badges

Terms & Conditions

Sign up

Already have an account? Login

Social Login

Username *

E-mail address *

What I do... *

Data Leader Data Architect Data Engineer Data Scientist Other

Company *

Country *

Zip Code *

Marketing Communications

Yes No

Password *

I have read and Agree to the Website Terms of Service and I have read and acknowledged the Privacy Policy.

loginBox.register.email_repeat

Login to the community

No account yet? Create an account

Social Login

Username or Email

Password

Remember me

Forgot password?

Enter your username or e-mail address. We'll send you an e-mail with instructions to reset your password.

Username or e-mail

Back to overview

Scanning file for viruses.

Sorry, we're still checking this file's contents to make sure it's safe to download. Please try again in a few minutes.

This file cannot be downloaded

Sorry, our virus scanner detected that this file isn't safe to download.