- - Knowledge base
Product Updates
Groups
Join Build It With Brenna

Build it With Brenna: Hands-on Workshop with a MySQL Origin and a Snowflake Destination

(Wed, 29 Mar, 17:00)

Home
Community overview
Welcome

Welcome

Welcome to the StreamSets Community

121 Topics

Recently active

Most replies Most views Newest first

NileshFan

asked in Community Articles

Oracle CDC to Snowflake while processing CLOB datatype

Hi,Facing error in Data collector pipeline Oracle CDC to Snowflake while processing CLOB datatype table. When inserting value >= 4000 then works fine. But the less than that facing error.SNOWFLAKE_28 - Snowflake MERGE for 'sdc-df26c7ce-fd00-4b5d-a3b1-817b9809e0a3.csv.gz' processed '1' CDC records out of '3'Oracle:create table TEST_LOB_VAL(lobid number(10) primary key, lob_value clob );INSERT INTO TEST_LOB_VAL VALUES (1,to_clob(RPAD('AAAAAAAAAAAAAAAAAA',4000,'A')));INSERT INTO TEST_LOB_VAL VALUES (2,to_clob(RPAD('AAAAAAAAAAAAAAAAAA',2000,'A')));INSERT INTO TEST_LOB_VAL VALUES (3,'AAAAAAAAAA');COMMIT;

9 minutes ago

NileshFan

asked in Community Articles

How to write data from Multiple Snowflkae Tables to Multiple text files (separate file for each table) on Local Server?

I have scenario to write data from multiple DB tables on daily basis to file server. Each table supposed to generate separate file on Local Server

12 hours ago

deepikamani007Fan

asked in Community Articles

BrennaStreamSets Employee

published in Events & Webinars

Business Value of Data Engineering Survey is now openNews

The first Business Value of Data Engineering Survey is now open. We know you’re more than a data ‘plumber’ - we see it every day in the outcomes you help your organizations create. With this survey, we’re looking at how you work with your business colleagues to make it happen. Please take 5-7 minutes to fill it out — you’ll automatically be entered to win one of 5 $100 Amazon gift cards! Take the survey here ▶️ https://tinyurl.com/dataengbv

6 days ago

akanshajain6793Discovered Fame

asked in Community Articles

how to get count of number of parameter passed in a request

Hi teami have column request_param:0: a : “11” b: “12” c: “12” d: “12” how i can get count if above request has 4 parameter it can have 3 also like belowrequest_param:0: a : “11” b: “12” c: “12” based on count i need to hit diff url

6 days ago

PradeepStreamSets Employee

posted in Community Articles

Map the job submitted in transformer to the one run in Databricks SparkUI

How do we map the job submitted in transformer to the one run in Spark? Or, in other words, what is the corresponding job in the Spark UI for the pipeline/job in question?A:In the transformer log, we see something like below once the job is submitted to databricks.2023-03-02 12:02:18,099 INFO submit run response run id: 108505820 In spark UI, you would see the job group having the id 108505820Run response id can also be seen as Task Run id in databricks workflow > job.

11 days ago

akanshajain6793Discovered Fame

asked in Community Articles

how to generate UUID bases on timestamp

Hi Team I have a date datefield storing current datetime in it want to create a UUID using same column how i can create . currently using groovy script to create UUID but want to use same column and create it

17 days ago

akanshajain6793Discovered Fame

asked in Community Articles

Getting error with HTTP client

Getting below error while passing URL as parameter any input for same

17 days ago

PradeepStreamSets Employee

posted in Show us your Pipelines

Example pipelines for Delta lake lookup processor

Input data:[ { "product_id": 1, "unit_price": 23.2 }, { "product_id": 4, "unit_price": 21.64 }, { "product_id": 2, "unit_price": 10.87 }]Delta lake lookup table. This is same for all operations performedReturn all matching rows, generating a record for each match: Return first matching row:There won’t be change in the output from above since there are no duplicates in input. Return a count of matching rows:Return true if matches exist, otherwise false:Input data:[ { "product_id": 1, "unit_price": 23.2 }, { "product_id": 2, "unit_price": 21.64 }, { "product_id": 4, "unit_price": 10.87 }, { "product_id": 5, "unit_price": 99.32 }, { "product_id": 4, "unit_price": 20.33 }]Return all matching rows, generating a record for each match:Return first matching row:Return true if matches exist, otherwise false: Return a count of matching rows:

19 days ago

ujeyakumFan

asked in Community Articles

how to merge 2 different pipeline

Hi team,Example pipelinePipeline 1 -- origin--to--kafka producerPipeline 2 -- kafka consumer--to--destinationNeed your support for how to create job to merge pipeline1 and pipeline2. Please let me know the possibility.

1 month ago

btsmerchstore45Fan

asked in Community Articles

Is it possible to configure the transformer pipeline to run on an AWS EMR cluster?

I have created a simple transformer pipeline to run on an EMR cluster. In Cluster configuration, I have used the necessary configuration details.I am facing the following issue:

1 month ago

akanshajain6793Discovered Fame

asked in Community Articles

how to connect and load data to aerospike from streamsets

Hi Team did any one work on aerospike load from streamsets i see the stage is deprecated can we use jdbc aerospike connector ?

1 month ago

krunchworksFan

asked in Community Articles

How can i get sample code to perform some data transformation for a sample data using snowflake?

I am new to streamsets and looking to explore more on use case where streamsets will help snowflake. I learned that streamset will help in data transformation. Is there some links or codes available to quickly learn from?

1 month ago

harshit_agrawal1Fan

asked in Community Articles

Configuring a trigger to activate when a file is created in a directory

I am trying to create a pipeline which gets triggered when I add a file to a directory. Is there any functionality in the streamsets hub or any kind of trigger which I can directly set to the “Directory” origin which can in turn trigger the pipeline ahead?

1 month ago

akanshajain6793Discovered Fame

asked in Community Articles

Webservice Array request Json

Hi team I have two json request for singleton[{“a”: “1”“b”:”2”“c”:”3”}]and for batch[{“a”: “1”“b”:”2”“c”:”3”},{“x”: “4”“y”:”5”“z”:”6”}] if singleton come i need to pass first request and if batch comes i need to pick the second one and in the second one i need to pick first value. am using json generator how to convert that into array as a request and then for batch how to pick the first value from multiple values.

1 month ago

akshayp2Fan

asked in Community Articles

How to configure the transformer pipeline to run on AWS EMR cluster?

HiI have created a simple transformer pipeline trying to run on EMR cluster. In Cluster configuration I have used required configuration details.Facing below issue, Can anyone help me this?

1 month ago

durairajFan

asked in Community Articles

Postgres CDC client is not processing data, when slot volume reaches 1 GB

We are using postgres CDC client to ingest the data from postges to ADLS. It’s working for other pipeline which are getting the CDC records lesser than 1 GB. While we drop and recreate the slots its working fine. But after sometime CDC files are generating from Streamsets. In pipeline logs we are not find error logs and jobs is active and running.while querying pg_replication_slots volume of data reaches more than 1 GB to several. until drop the replication slots CDC pipeline is not streaming.So, please suggest how to fix the issue. Do, we need to change the Streamsets configuration or postgres configuration. please let us know.Our Streamsets config. as below,Max Batch Size : 15000Streamsets engine : 4.4.1Batch Wait Time : 15000Query Timeout : {45 * MINUTES}Poll Interval: {1 * SECONDS}Status Interval : ${30 * SECONDS}CDC Generator Queue Size : 20000

1 month ago

amarzeetFan

asked in Community Articles

Streamset data collector 3.x tarball download link not available

I want to use stream set data collector legacy version (3.x) on a linux machine. Only option is to install the tar ball. I couldn't find a link on streamset website to download and use it. Can you please provide this information

2 months ago

vgohilFan

asked in Community Articles

JOBRUNNER_103 error when trying to run complex pipeline (with fragment)

I am following the fragment lesson for which i have imported the pipeline and created a duplicate. With the duplicate pipeline, i managed to create a fragment, edit the pipeline and remove the components and add the new fragment. Upon publishing the pipeline i am receiving the following error below... JOBRUNNER_103:Organization 4441d489-761c-11ed-aae0-e11f88355e74 already has 2 running jobs which is the maximum allowed. Please contact your StreamSets account team or email subscriptions@streamsets.com Can anyone please suggest what to do here to successfully start the job.

3 months ago

RishiStreamSets Employee

asked in Community Articles

How to enable ZGC for Streamsets Data Collector ?

What is ZGC ?The Z Garbage Collector, also known as ZGC, is a scalable low latency garbage collector designed to meet the following goals:Sub-millisecond max pause times Pause times do not increase with the heap, live-set or root-set size Handle heaps ranging from a 8MB to 16TB in sizeZGC was initially introduced as an experimental feature in JDK 11, and was declared Production Ready in JDK 15.At a glance, ZGC is:Concurrent Region-based Compacting NUMA-aware Using colored pointers Using load barriersAt its core, ZGC is a concurrent garbage collector, meaning all heavy lifting work is done while Java threads continue to execute. This greatly limits the impact garbage collection will have on your application’s response time. With SDC 5.3.x release, Data Collector supports Java 11 and 17 in addition to Java 8. And hence you can user ZGC if you are using JAVA 17 To enable ZGC just add the following jvm argument -XX:+UseZGC and restart the engine. You can verify this by review $SDC_LOG/g

3 months ago

Santhosh KumarDiscovered Fame

asked in Show us your Pipelines

what is best of streamsets?

Hello Team, My self Santhos kumari'm Beginner of streamsets so which one is best of the learning in depth of streamsets.

3 months ago

lakshmi_narayanan_tDiscovered Fame

asked in Show us your Pipelines

JDBC PIPE LINE NOT RUNING,SHOWS ME ERROR LIKE: JDBC_16 - Table '' does not exist or PDB is incorrect. Make sure the correct PDB was specified

I facing a issue on running JDBC pipeline ,it shows the below error ,I tried so many things from the google but no use anybody can solve .

4 months ago

subashiniStreamSets Employee

asked in Community Articles

StreamSets Hive Metadata issue after upgrading to CDP 7.1.x

Issue: Hive Metadata stage throws error "A managed table's location should be located within managed warehouse root directory or within its database's managedLocationUri" after upgrading to CDP version 7.14 and above. Reason: In CDP7.x or HDP3.x, the database has 2 locations by default(managed and external) and managed tables are always ACID.Whereas HDP-2.6.x or CDH-6.x or lesser, there is no difference in managed/external table location and managed table will not be created as ACID table by default. In CDP7 or HDP3,When user tries to create managed table with custom location, CDP-Hive validates whether the custom location is outside database's managed location and throws above described exception in StreamSets pipeline. StreamSets will not support ACID table read/write. Workaround: Since StreamSets pipeline with Hive Metadata stage does not support Hive Managed table creation, the best solution is to choose the External table option and provide Table path template so t

4 months ago

Page 1 / 5

Badge winners

AkshayJadhavhas earned the badge Eager to help
john.durkinhas earned the badge Eager to help
samhas earned the badge Eager to help
saleempothiwalahas earned the badge Eager to help
Bikramhas earned the badge Community Champion 🏅

Show all badges

Terms & Conditions

Sign up

Already have an account? Login

Social Login

Username *

E-mail address *

What I do... *

Data Leader Data Architect Data Engineer Data Scientist Other

Company *

Country *

Zip Code *

Marketing Communications

Yes No

Password *

I have read and Agree to the Website Terms of Service and I have read and acknowledged the Privacy Policy.

loginBox.register.email_repeat

Login to the community

No account yet? Create an account

Social Login

Username or Email

Password

Remember me

Forgot password?

Enter your username or e-mail address. We'll send you an e-mail with instructions to reset your password.

Username or e-mail

Back to overview

Scanning file for viruses.

Sorry, we're still checking this file's contents to make sure it's safe to download. Please try again in a few minutes.

This file cannot be downloaded

Sorry, our virus scanner detected that this file isn't safe to download.