Skip to main content

StreamSets Platform

Get inspired and gain all the knowledge you need

1218 Topics
S
Fan
santhoshchallaFan
asked in Community Articles and Got a Question?
Find the status of a job using REST API

Hi, I have triggered one of the job suing REST API in StreamSets and I need to the status of the job if it is failed or successful. I am able to get the status of the job but if the job is failed or successful, I am getting status as Inactive in both success and failure case. I need to get exact status like success or failure and error message in case of job failure. Please let me know how I can I achieve this StreamSets.

2916
prasanna_kumar
Roadie
S
Fan
SUDHANSHOOFan
asked in Share your Best Practices
Need to Read ADLS files name in nested folders

Hi All,We have a senario where we need to read all file name in ADLS Origin Stage. We have checked out in Streamsets documentation but its not working Its mentioned in Doc - we can access the files with **/<File Pattern> But its not picking up the filename. If anyone have the solution you can share with us.

282
S
Fan
himanshu1234567
Discovered Fame
himanshu1234567Discovered Fame
asked in Community Articles and Got a Question?
Not able to export all pipelines at once using python sdk

Hi team when i am trying to export all pipelines using python sdk , i am facing below error: {"ISSUES":[{"code":"PIPELINE_STORE_41","message":"Cannot export draft pipelines with plain text credentials stripped"}]}So basically we are not able to export draft pipelines with credentials. Does we have any way where we can export all pipelines whether it is published or draft version.

452
himanshu1234567
Discovered Fame
AdamK
Fan
AdamKFan
asked in Community Articles and Got a Question?
How can I extend default timeout value?

Hi, I’m using StreamsetsSDK in version 3.12.1. In my code which is run on Teamcity agent when I’m publishing changes to pipeline using publish_pipeline() method I’m getting the following error: requests.exceptions.ConnectionError: HTTPSConnectionPool(host='******', port=******): Max retries exceeded with url: /rest/v1/system/info (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f95dca34820>: Failed to establish a new connection: [Errno 110] Connection timed out')) This might be caused by firewall settings because code works fine when run locally. Is there a way to extend timeout limit? On the other hand doing imports using import_jobs() work fine.

1032
AdamK
Fan
A
StreamSets Employee
ajinkyaStreamSets Employee
asked in Community Articles and Got a Question?
How to get a specific pipeline using pipeline id or pipeline name using python SDK?

sch.pipelines.get(pipeline_id='id of pipeline') function can be used to get the specific pipeline using the pipeline id. from streamsets.sdk import ControlHub server_url = 'https://cloud.streamsets.com/' sch = ControlHub(server_url='https://cloud.streamsets.com/',username = 'username@org',password = 'password') desiredpip=sch1.pipelines.get(pipeline_id='00a2a5e6-5125-4477-b022-5507c7f63c59:dpmsupport') desiredpip=sch1.pipelines.get(name='SDC500 - Dev to Trash (batch size 10000)') NOTE:- Fetching a pipeline using pipeline id is much faster than the pipeline name.

690
A
StreamSets Employee
lakshmi_narayanan_t
Discovered Fame
lakshmi_narayanan_tDiscovered Fame
asked in StreamSets Academy
how to separate pincode into specific field?

I had Pincode like alpha numeric how can i separate numeric value into specific fieldthis is field value -4900 B ICI RD BUT I WANT Output like -4990 and remainig another field like-B ICI RD

181
Rishi
StreamSets Employee
J
Opening Band
jyuanOpening Band
asked in Community Articles and Got a Question?
How to get statistics like bytes streamed per job per day?

Is there a way to gather statistics on jobs? I am interested in showing the bytes streamed by each job in the last 24 hours, and the like. Also, if possible, I would like to retrieve the bytes streamed in the last 30 days. From the ControlHub GUI, I don’t see anything. Maybe the solution is available on the SDC side? Maybe we can run a script on the SDC server to gather statistics? Thanks!

300
J
Opening Band
Pavan.b236
Fan
Pavan.b236Fan
asked in Community Articles and Got a Question?
DB2 DB CDC implimention and load into S3 in streamsets

I have a requirement to do IBM DB2 database as origin and needs to implement CDC pipeline to load into S3. so could you please help me here how to implement DB2 database as Origin for CDC implementation and configuration details what parameters required and performance tuning setting for the same .

2543
lex03
Discovered Fame
A
Fan
Akambe21Fan
asked in Community Articles and Got a Question?
Reading XML

I am reading xml from s3 using data format as xml and putting it in snowflake but it is going in a single column. I need to put it multiple columns as node avaliable

566
saleempothiwala
Headliner
southwr62
Fan
southwr62Fan
asked in Community Articles and Got a Question?
Using calculated dates in an HTML request

I need to create a stream where the origin is an HTTP call. I need to pass a start and end date into this. I have 3 date range to run, although these could be 3 pipelines they are in principle the same Start and end date same (Yesterday) Start date 1 month ago, end date yesterday start date 1 year ago, end date yesterday I have worked with static parameters for start and end date, and this is fine, but I need to run the pipelines daily, via the scheduler. I tried to create a calculated date using timeAddition to calculate the date for yesteday (based on the example in the Docs for timeAddition function), but I get an EL invalid message

561
saleempothiwala
Headliner
K
Fan
kushal_upadhyayFan
asked in Community Articles and Got a Question?
APIS to start or stop a schedule

Is there any separate API for start or stop a scheduler task. ?

231
Rishi
StreamSets Employee
som
Fan
somFan
posted in Community Articles and Got a Question?
Control the number of email alerts per job failure.

ISSUE: When the job gets failed, we get many number of email alerts.Below is the subscription set at our end. We have also tried changing the subscription to “max global fail over retries” but it does not give actual error message and gives a generic error, {{"FROM_STATUS":"INACTIVE", "TO_COLOR":"RED", "TO_STATUS":"INACTIVE", "ERROR_MESSAGE":"Job is inactive due to global failover retries exhausted." } } Can we control the number of email alerts with getting actual error message. SOLUTION: We can control the email alert by setting below subscription where we will get only 1-2 email alert when either jobs gives any error or exception and also can capture actual error message. ${TO_STATUS=="INACTIVE" && TO_COLOR=="RED" && str:contains(ERROR_MESSAGE,"RUN_ERROR") || str:contains(ERROR_MESSAGE,"exception")}

320
som
Fan
som
Fan
somFan
posted in Community Articles and Got a Question?
Disable fail over by rest api call in sch.

ISSUE: When we create a job in rest api, by default it creates the job with fail over enabled.Sometimes we need to create the job via rest api with fail over disabled.we can follow below solution to achieve this. SOLUTION: While creating the job, we can change the value of field "migrateOffsets" to “false” and it will create the job with fail over disabled.

60
som
Fan
C
Fan
Ch_7_RFan
asked in Community Articles and Got a Question?
Turn off Heap Dumps recording

How to turn off Heap Dumps recording to var/log/sdc folder on linux node?

341
Bikram
Headliner
A
StreamSets Employee
ajinkyaStreamSets Employee
asked in Share your Best Practices
CONTAINER_0701 - Stage 'OracleCDC_1' initialization error: java.lang.NullPointerException

Issue:- While using the new Oracle CDC Origin, the following error is encountered:- CONTAINER_0701 - Stage 'OracleCDC_1' initialization error: java.lang.NullPointerException Resolution:- Oracle CDC requires following jars to be placed on SDC instance:- ojdbc8.jar orai18n-mapping.jar orai18n.jar Review the section Task:4 Install Required Libraries on the Data Collector in the documentation for more information. Root Cause:- Missing dependency jars.

620
A
StreamSets Employee
A
StreamSets Employee
ajinkyaStreamSets Employee
asked in Share your Best Practices
Why Oracle CDC Origin consuming more than 80% CPU?

Issue:- Oracle CDC Origin is consuming more than 80% CPU while processing lower number of records. Resolution:- Set Buffer Wait Strategy to BLOCKING. Root Cause:- New Oracle CDC Origin comes with new Buffer Wait Strategy options. Busy Spin would use the most amount of the CPU. The strategy can be set as per the environment setup and requirements. Please review the documentation for more information.

260
A
StreamSets Employee
R
Fan
ravitejakshettyFan
asked in Community Articles and Got a Question?
How to query BigQuery table in the pipeline and use the result for further processing

Hi Friends, I am new to StreamSets and trying to build a pipeline to handle event data coming from Google Cloud pub/sub to Google Cloud BigQuery. I am stuck with a problem and need your help. Whenever an event enters the pipeline, I will have to check in the Bigquery tables whether the event already exists in the tables. We have a unique id in the event against which i can query the BigQuery table to check and further continue in the pipeline. I tried various options(including jdbc lookup) but no luck. Can you please help? Thanks in Advance. Regards, Ravi

140
R
Fan
Lavender
StreamSets Employee
LavenderStreamSets Employee
posted in Share your Best Practices
Log in Platform as a local Control Hub user when SAML is enabled

Topic: Org Admin wants to log in Platform as a local Control Hub user when SAML is enabled Solution: In our Platform documentation, when SAML is enabled, it is mentioned that: "After you publish a draft configuration to production, you enable the production configuration to activate it. When enabled, all organization users must log in using SAML authentication.As a user with the Organization Administrator role, you can log in using local or public identity provider authentication to access the Control Hub organization SAML configuration page only." This means that all users must authenticate via SSO SAML, but only Org Admin can log in locally to configure SAML settings when SAML breaks, or to disable SAML. This is different than logging in Control Hub as a local user. In Control Hub, there is a way for users with the Org Admin role or the Control Hub Authentication role to access SAML backdoor and log in Control Hub. Please refer to this article for instructions:

190
Lavender
StreamSets Employee
N
Fan
NileshFan
asked in Share your Best Practices
How to pass runtime value to transformer in Snowflake Destination

Hi, I’m trying to merge table data if data is not matched. I have multiple tables, so need to create dynamic pipeline which will take input as Table_Name and Primary Key columns to Merge Configuration. But in runtime, Snowflake not reading values and create table as parameter name. Transformer Pipeline: Raw Data (which actual table in Oracle DB) Snowflake: But Pipeline running, it creates new table as TABLE_NAME, instead of merging on EMP table

210
N
Fan
N
Fan
naveenkumarYFan
asked in Community Articles and Got a Question?
Register Datacollector to controlhub using LDAP.

Controlhub is enabled with LDAP, now I want to register datacollectors to SCH. Previously datacollectors were registered using following file-based authentication https://docs.streamsets.com/datacollector/latest/help/datacollector/UserGuide/DPM/RegisterSDCwithDPM.html#concept_js4_2z1_cx. After LDAP is enabled in controlhub, I believe we can’t use file-based authentication to register SDC to SCH.

162
N
Fan
K
Fan
kushal_upadhyayFan
asked in Community Articles and Got a Question?
Writing binay file with 0 bytes in S3 bucket when binary format is selected

Send data from KAFKA To OBS (Object storage cloud customized version of AWS S3) in binary format is creating a file with 0 bytes though we have 36 records. The records from Kafka has a format of SDC records. Please help how we can write binary data to S3 bucket when format,

440
K
Fan
P
Fan
priyankarayalaFan
asked in Community Articles and Got a Question?
JDBC Mutlitable consumer converting varbinary data to byte_array[] causing data issue.

Due to this conversion of varbinary to byte_array[] my data is getting distorted. How can I get back the original data i.e., the ASIS varbinary data to be loaded. In SQL server - the data looks like 0xFFD8FFE000104A464946000………….. In Streamsets - It read the data like /9j/4AAQSkZJRgABAQAAAQABAAD// ………………

290
P
Fan
Pritam2612
Fan
Pritam2612Fan
asked in Community Articles and Got a Question?
JDBC_06 - Failed to initialize connection pool: com.zaxxer.hikari.pool.HikariPool$PoolInitializationException

Getting this error While create JDBC connection for GreenPlum Database: JDBC_06 - Failed to initialize connection pool: com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: [Pivotal][Greenplum JDBC Driver][Greenplum]no pg_hba.conf entry for host "172.18.0.2", user "sdc", database "postgres", SSL off. Using Greenplum 9.5 in docker.

38731
antmcmullen
StreamSets Employee
lakshmi_narayanan_t
Discovered Fame
lakshmi_narayanan_tDiscovered Fame
asked in Share your Best Practices
Reading XML file in directory it shows error how to fix?

ERROR Shown Is:- ANy One SuGgest to fix ThIs ErRor Timestamp Error Code Error Message Aug 9, 2023, 5:41:08 PM SPOOLDIR_02 SPOOLDIR_02 - Object in file 'beautified.xml' at offset '0' exceeds maximum length (View Stack Trace... ) Aug 9, 2023, 5:41:08 PM SPOOLDIR_01 SPOOLDIR_01 - Failed to process file '/resources/beautified.xml' at position '0': com.streamsets.pipeline.lib.dirspooler.BadSpoolFileException: com.streamsets.pipeline.lib.parser.DataParserException: XML_PARSER_02 - XML object exceeded maximum length: readerId 'beautified.xml', offset '0', maximum length '4096' (View Stack Trace... )

341
Rishi
StreamSets Employee
R
Fan
rutuja_pradhanFan
asked in Community Articles and Got a Question?
How to create cluster with tarball setup

Hi, How do we put 2 tarball data collector engines in a cluster? Is ‘Enable Failover’ checkbox in job settings the only option?

403
saleempothiwala
Headliner

Most helpful members this week

Badge winners

Show all badges

Popular tags

Terms & Conditions