Skip to main content

StreamSets Platform

Get inspired and gain all the knowledge you need

1210 Topics
K
Fan
kushal_upadhyayFan
asked in Community Articles and Got a Question?
Unable to write to HTTP client

I am trying to write data from mysql database to HTTP client destination (free public client) without authentication but its giving error HTTP_41 - Error sending resource - Not elaborated. Converted the data into map using expression evaluator, then passed it to the field remover and kept the map only , then passing the data to json generator which generated the json Field to serialize - myNewMap containing ID,NAME,address,age fields and then it converted to json. Till Json generator output is fine.   0myNewMap : MAP Record1-Output Record1 : LIST_MAP 0myNewMap : STRING   {"ID":1,"NAME":"Kushal","AGE":35,"ADDRESS":"Bavdhan"} After this the data is not written to HTTP client. I can post the request using ARC successfully with the same data but not through streamsets. Belows is the excerpt from the pipeline..  Please suggest whats going wrong. HTTP client configuration : URL - GivenMethod - POSTHeader - Content-Type - application/json  

1131
Bikram
Headliner
1 year ago
D
Opening Band
Dhanashri_BhateOpening Band
asked in StreamSets Academy
Selection of cloud libraries when pipeline needs 2 different cloud environments

Hello,We’re trying to create a transformer pipeline with Origin: AmazonS3 , Destination: Azure ADLS and compute is on Azure Databricks. We tried to break is down into smaller pieces.S3->ADLS pipeline with Streamsets own cluster worked fine. Random data source to ADLS with Databricks compute worked fine. But when we try to bring in all 3 components together, we are getting stage library specific errors which are conflicting. We have tried different combinations from the list in the attached picture.In this particular scenario, what are the stage libraries we need to choose?  

131
D
Opening Band
1 year ago
S
Discovered Fame
saikat_chatterjeeDiscovered Fame
asked in Community Articles and Got a Question?
Schedule Job using sdk python

After building a pipeline and running a job using sdk python I am unable to get the ways to schedule a job using sdk and also there are no steps given in StreamSets documents for the same.Can anyone guide me on this. 

331
XavierV
StreamSets Employee
1 year ago
A
Discovered Fame
akanshajain6793Discovered Fame
asked in Community Articles and Got a Question?
Webservice Array request Json

Hi team I have two json request for singleton[{“a”: “1”“b”:”2”“c”:”3”}]and for batch[{“a”: “1”“b”:”2”“c”:”3”},{“x”: “4”“y”:”5”“z”:”6”}] if singleton come i need to pass first request and if batch comes i need to pick the second one and in the second one i need to pick first value. am using json generator how to convert that into array as a request and then for batch how to pick the first value from multiple values.   

355
Bikram
Headliner
1 year ago
J
Fan
Jay_66Fan
asked in Community Articles and Got a Question?
INFO Pipeline - Destroying pipeline with reason=FINISHED

Hi, We created a Pipeline that has a CRON scheduler, and calls multiple other pipelines.this pipeline had been running fine, but since today anytime we start the pipeline it would immediately complete with Finished Status, on checking the logs the below is what we found. Complete log attached DEBUG PipelineConfigurationValidator - Pipeline 'BigDaddyOr1ef6605f-4617-42c3-b8e8-6b178394e717' validation. valid=true, canPreview=true, issuesCount=0 [stage:] INFO  StandaloneAndClusterRunnerProviderImpl - Pipeline execution mode is: STANDALONE[runner:] [thread:webserver-29331] [stage:] DEBUG ManagerResource - Status is FINISHED [runner:] [thread:webserver-29331] [stage:] DEBUG LogUtil - Logger 'pipeline.BigDaddyOr1ef6605f-4617-42c3-b8e8-6b178394e717.0.state' already exists

463
Bikram
Headliner
1 year ago
P
Discovered Fame
Priyanka MynepallyDiscovered Fame
asked in Community Articles and Got a Question?
JDBC_02 error

Hi TeamI am facing some issue like incorrect syntax at ‘=’ while doing jdbc lookup.I am trying to lookup column a and fetch b,c columns from Table-X.For some rows, this error comes, But the same rows when executing independently in SQL-server, I am able to fetchBut from SS-its not fetching. Please help me on this. RegardsPriyanka 

551
Bikram
Headliner
1 year ago
S
Fan
suresh_venkataFan
asked in StreamSets Academy
How to Create Engine from Control Hub by using SDK Python ?

Hi Team,we Observed, After the Creation Of Deployment using SDK Python Version 5.1.0, we have to Run the “Get Install Script” in Our Local VM for the Creating Engine. Except This Step How to Create the Engine By using SDK Python? can anyone Please help me on this. Thank YouSuresh_venkata

260
S
Fan
1 year ago
lakshmi_narayanan_t
Discovered Fame
lakshmi_narayanan_tDiscovered Fame
asked in Share your Best Practices
field renamer

when i using field re-namer to perform multipile function with same field it  shows error ,if there is any option to perform operation like caps and remove special characters of field with one processor.

401
saleempothiwala
Headliner
1 year ago
S
Fan
suresh_venkataFan
asked in Community Articles and Got a Question?
How to Create Engine from Control Hub by using SDK Python?

Hi Team,we Observed, After the Creation Of Deployment using SDK Python Version 5.1.0, we have to Run the “Get Install Script” in Our Local VM for the Creating Engine. Except This Step How to Create the Engine By using SDK Python? Requesting to you Please Provide the Information. Thank YouSuresh_venkata   

200
S
Fan
1 year ago
ashok verma
Discovered Fame
ashok vermaDiscovered Fame
asked in Community Articles and Got a Question?
performance is bad when i am using JDBC lookup in DataCollector

i am reading a file from S3 bucket and doing lookup against data from  Postgress DB using JDBC lookup using Data collector.in source i have some 341k records against 341 records in postgress DB.my observations are1.it taking 30min to process 50K records.2.some records are going to error even matching one present in DB3.i have tried by enabling local cache. 

1626
Kartik Medavarapu
Roadie
1 year ago
M
Fan
madhusudan_shastriFan
asked in Community Articles and Got a Question?
Transformer pipeline on EMR - START_ERROR: Application has completed. Clearing staged files..

I am executing a simple transformer pipeline on EMR cluster (6.5.0). EMR logs show that the job is successful, but the transformer pipeline fails with error “START_ERROR: Application has completed. Clearing staged files..”. What could be the issue here? Pipeline logs are as below:2023-01-27 10:44:46,788    INFO    Current Step Status is PENDING. Waiting for 30 seconds before checking status again..    EMRAppLauncher    *842bf273-5a66-11ed-b52a-ab8a97a1d11f@b601f953-5a66-11ed-b8e4-b31442db6663        runner-pool-2-thread-292023-01-27 10:45:16,824    INFO    Application started successfully. Current Status is RUNNING    EMRAppLauncher    *842bf273-5a66-11ed-b52a-ab8a97a1d11f@b601f953-5a66-11ed-b8e4-b31442db6663        runner-pool-2-thread-292023-01-27 10:45:16,824    INFO    DataTransformerLauncher start method finished    DataTransformerLauncher    *842bf273-5a66-11ed-b52a-ab8a97a1d11f@b601f953-5a66-11ed-b8e4-b31442db6663        runner-pool-2-thread-292023-01-27 10:46:16,857    INFO    

631
wilsonshamim
StreamSets Employee
1 year ago
A
Roadie
akshayp2Roadie
asked in Community Articles and Got a Question?
How to configure the transformer pipeline to run on AWS EMR cluster?

HiI have created a simple transformer pipeline trying to run on EMR cluster. In Cluster configuration I have used required configuration details.Facing below issue,  Can anyone help me this?

1013
wilsonshamim
StreamSets Employee
1 year ago
Theodore
Fan
TheodoreFan
asked in Community Articles and Got a Question?
Installing SDC and building pipelines on IOS operating system. 

Hello,Please help if you can. I was given these instructions:  Please go through the installation steps for the common tarball version and build Data Collector pipelines based on the Basic and Extended Tutorials (find the links below). You can use your home computer/laptop as we support a variety of operating systems (except Windows). After creating your account and logging in, select "Download Data Collector," then select "Linux Server - For production use," and then select "Tarball - recommended". This will allow you to download Data Collector version 3.22.3. I am using a IOS operating system, I followed the instructions above,  but I cannot seem to get it to work via the terminal. I am new to the community and I am new to learning about Streamset. I was given a task that I can’t seem to get through. I need assistance installing SDC and building pipelines on my IOS operating system.  Please help

193
Theodore
Fan
1 year ago
J
Fan
Jay_66Fan
asked in Community Articles and Got a Question?
ERROR The Scheduler cannot be restarted after shutdown()

One of the pipelines that we created immediately switces to "FINISHED" state after we start it. This started happening after we "FORCE SHUTDOWN " the pipeline with the CRON. Looks like the Pipeline id got registered somwhere with the CRON Shutdown status and wont allow us to restart.DEBUGStaring pipeline with offset: {}ProductionPipelineRunner ERRORThe Scheduler cannot be restarted after shutdown() has been called.SchedulerPushSourceorg.quartz.SchedulerException: The Scheduler cannot be restarted after shutdown() has been called.    at org.quartz.core.QuartzScheduler.start(QuartzScheduler.java:529)    at org.quartz.impl.StdScheduler.start(StdScheduler.java:142)

1730
J
Fan
1 year ago
C
Roadie
collidRoadie
asked in Community Articles and Got a Question?
Custom Stages in StreamSets Transformer pipelines

How easy would it be to create a custom Scala or PySpark stage, that can output an array of spark data frames (like it can receive for input) rather than just one?

292
C
Roadie
1 year ago
M
Fan
mayankdhiamn27Fan
asked in Community Articles and Got a Question?
How to download and install data collector on a machine

I want to run data collector on machine. Have gone through some git readme.md and some blogs too, they are asking to download the tarball, but I am not able to find the url to download the tarball. Can someone help with this, or with some other way to download the tarball? 

10508
Theodore
Fan
1 year ago
mayank_phopse
Fan
mayank_phopseFan
asked in Community Articles and Got a Question?
Exception handling for data collector using SDK

Suggest any method to check whether data collector is up or down and handle it as a exception in python SDK 

160
mayank_phopse
Fan
1 year ago
mayank_phopse
Fan
mayank_phopseFan
asked in Community Articles and Got a Question?
Exception handling for username and password using SDK

I have tried exceptional handling for invalid username or password using SDK.try: JDBC_consumer.username = "****" JDBC_consumer.password = "********"except: print("Invalid username & password") Following code is not working. Can anyone suggest a different approach?  

140
mayank_phopse
Fan
1 year ago
S
Fan
Satyam_TopeFan
asked in Community Articles and Got a Question?
How to configure JDBC Multitable consumer & Azure data lake storage using python sdk 5.0 ?

We were trying to configure, JDBC multitable consumer using python SDK 5.0. We were able to set below configurations:JDBC_consumer = pipeline_builder.add_stage('JDBC Multitable Consumer')JDBC_consumer.jdbc_connection_string = 'jdbc:mysql://mysqldb:3306/zomato'JDBC_consumer.username = '****'JDBC_consumer.password = '****'However, not able to set table config. Same with Azure Data Lake Storage, we are able to set below configurations:Azure_storage.data_format = 'DELIMITED'Azure_storage.authentication_method = 'Shared Key'Azure_storage.account_shared_key = '***'However, not able to set Account FQDN & Storage Container / File System  Please guide us with these configurations.

1123
S
Fan
1 year ago
Bikram
Headliner
BikramHeadliner
asked in Community Articles and Got a Question?
How to remove special characters from field names in streamsets data collector?

Solution.The issue can be handled by Field mapper processor. Below the syntax for the same. ${str:replaceAll(f:name(), '[.\\-/()!&%^$#@]','')} Attached the screen shot for the reference. NB : The issue also can be handled in different ways e;g(Expression Evaluator, scripting processor etc) 

91215
P
Opening Band
1 year ago
C
Fan
CS_1970Fan
asked in Community Articles and Got a Question?
Orchestration Pipeline - Getting Variable Back from Called Job?

Hello, I am new to Streamsets, I have tried to find answer to this question.I can see a worked example at my place of work where an Orchestration Pipeline can pass a variable down to a Transformer Job in the Runtime Parameters, eg…     Collector Pipeline Fragment (Jython)for record in sdc.records:  try:    if :     ...    else:       ...    record.value['LATEST_DATE'] = ….    sdc.output.write(record) Orchestration Pipeline then executed Transformer Job node “branch_one_run” passing the value as a parameter in the Runtime Parameters"LATEST_DATE": "${record:value("/LATEST_DATE")}", Subsequent Stream Splitter node in the Orchestrator pipeline can determine course of action to take based on whether the call to the Transformer job was successful… ${record:value('/orchestratorTasks/branch_one_run/success')==true}But is it possible for the Orchestrator to also get a bespoke variable containing value / values back from the called Transformer job that it can use to pass as a parameter to subseq

664
C
Fan
1 year ago
Dirk B.
Fan
Dirk B.Fan
asked in Community Articles and Got a Question?
Data Collector going to disconnected mode all the time

I get every 10 seconds the following sequence in the data collector log:DPM reachable but returning '404' HTTP status on login        RemoteSSOService    RemoteSSOService is not active, changing to disconnected mode        FailoverSSOService           RemoteSSOService is active, changing to connected mode        FailoverSSOService Most things seem to work. The engine (v5.3) is connected to Control Hub with a proper setup ( https://eu01.hub.streamsets.com/ ). As the URL is not logged (even in trace) I cannot see what could solve this. Does anyone have an idea what is happening here?

791
Dirk B.
Fan
1 year ago
S
Discovered Fame
Santhosh KumarDiscovered Fame
asked in StreamSets Academy
Rest API

how to write rest api in python code..

461
Bikram
Headliner
1 year ago
S
Discovered Fame
Santhosh KumarDiscovered Fame
asked in StreamSets Academy
Parquet file Data collector

Hi teami using streamset control hub 3.x ,if possible to read and write parquet file in data collector ? 

1042
saleempothiwala
Headliner
1 year ago
S
Discovered Fame
saikat_chatterjeeDiscovered Fame
asked in Community Articles and Got a Question?
Unable to create a deployment using python SDK

After creating an environment through python sdk, when I am trying to create a deployment using the scripts I am getting the error described bellow: >>>deployment_builder = sch.get_deployment_builder(deployment_type='SELF')>>> # sample_environment is an instance of streamsets.sdk.sch_models.SelfManagedEnvironment>>> deployment = deployment_builder.build(deployment_name='Sample Deployment',...                                       environment=sample_environment,...                                       engine_type='DC',...                                       engine_version='4.1.0',...                                       deployment_tags=['self-managed-tag'])Traceback (most recent call last):  File "<stdin>", line 2, in <module>NameError: name 'sample_environment' is not definedPlease help me regarding this.

302
saleempothiwala
Headliner
1 year ago

Badge winners

Show all badges
Terms & Conditions