- - Knowledge base
Groups
Join Build It With Brenna

Home
Community overview
Platform Q&A

Platform Q&A

Get inspired and gain all the knowledge you need

631 Topics

Recently active

Most replies Most views Newest first

Jay_66Fan

asked in Got a Question?

ashok vermaDiscovered Fame

asked in Got a Question?

performance is bad when i am using JDBC lookup in DataCollector

i am reading a file from S3 bucket and doing lookup against data from Postgress DB using JDBC lookup using Data collector.in source i have some 341k records against 341 records in postgress DB.my observations are1.it taking 30min to process 50K records.2.some records are going to error even matching one present in DB3.i have tried by enabling local cache.

2 days ago

madhusudan_shastriFan

asked in Got a Question?

Transformer pipeline on EMR - START_ERROR: Application has completed. Clearing staged files..

I am executing a simple transformer pipeline on EMR cluster (6.5.0). EMR logs show that the job is successful, but the transformer pipeline fails with error “START_ERROR: Application has completed. Clearing staged files..”. What could be the issue here? Pipeline logs are as below:2023-01-27 10:44:46,788 INFO Current Step Status is PENDING. Waiting for 30 seconds before checking status again.. EMRAppLauncher *842bf273-5a66-11ed-b52a-ab8a97a1d11f@b601f953-5a66-11ed-b8e4-b31442db6663 runner-pool-2-thread-292023-01-27 10:45:16,824 INFO Application started successfully. Current Status is RUNNING EMRAppLauncher *842bf273-5a66-11ed-b52a-ab8a97a1d11f@b601f953-5a66-11ed-b8e4-b31442db6663 runner-pool-2-thread-292023-01-27 10:45:16,824 INFO DataTransformerLauncher start method finished DataTransformerLauncher *842bf273-5a66-11ed-b52a-ab8a97a1d11f@b601f953-5a66-11ed-b8e4-b31442db6663 runner-pool-2-thread-292023-01-27 10:46:16,857 INFO

3 days ago

TheodoreFan

asked in Got a Question?

Installing SDC and building pipelines on IOS operating system.

Hello,Please help if you can. I was given these instructions: Please go through the installation steps for the common tarball version and build Data Collector pipelines based on the Basic and Extended Tutorials (find the links below). You can use your home computer/laptop as we support a variety of operating systems (except Windows). After creating your account and logging in, select "Download Data Collector," then select "Linux Server - For production use," and then select "Tarball - recommended". This will allow you to download Data Collector version 3.22.3. I am using a IOS operating system, I followed the instructions above, but I cannot seem to get it to work via the terminal. I am new to the community and I am new to learning about Streamset. I was given a task that I can’t seem to get through. I need assistance installing SDC and building pipelines on my IOS operating system. Please help

3 days ago

Jay_66Fan

asked in Got a Question?

ERROR The Scheduler cannot be restarted after shutdown()

One of the pipelines that we created immediately switces to "FINISHED" state after we start it. This started happening after we "FORCE SHUTDOWN " the pipeline with the CRON. Looks like the Pipeline id got registered somwhere with the CRON Shutdown status and wont allow us to restart.DEBUGStaring pipeline with offset: {}ProductionPipelineRunner ERRORThe Scheduler cannot be restarted after shutdown() has been called.SchedulerPushSourceorg.quartz.SchedulerException: The Scheduler cannot be restarted after shutdown() has been called. at org.quartz.core.QuartzScheduler.start(QuartzScheduler.java:529) at org.quartz.impl.StdScheduler.start(StdScheduler.java:142)

3 days ago

collidRoadie

asked in Got a Question?

Custom Stages in StreamSets Transformer pipelines

How easy would it be to create a custom Scala or PySpark stage, that can output an array of spark data frames (like it can receive for input) rather than just one?

3 days ago

mayankdhiamn27Fan

asked in Got a Question?

How to download and install data collector on a machine

I want to run data collector on machine. Have gone through some git readme.md and some blogs too, they are asking to download the tarball, but I am not able to find the url to download the tarball. Can someone help with this, or with some other way to download the tarball?

4 days ago

mayank_phopseFan

asked in Got a Question?

Exception handling for data collector using SDK

Suggest any method to check whether data collector is up or down and handle it as a exception in python SDK

4 days ago

mayank_phopseFan

asked in Got a Question?

Exception handling for username and password using SDK

I have tried exceptional handling for invalid username or password using SDK.try: JDBC_consumer.username = "****" JDBC_consumer.password = "********"except: print("Invalid username & password") Following code is not working. Can anyone suggest a different approach?

4 days ago

Satyam_TopeFan

asked in Got a Question?

How to configure JDBC Multitable consumer & Azure data lake storage using python sdk 5.0 ?

We were trying to configure, JDBC multitable consumer using python SDK 5.0. We were able to set below configurations:JDBC_consumer = pipeline_builder.add_stage('JDBC Multitable Consumer')JDBC_consumer.jdbc_connection_string = 'jdbc:mysql://mysqldb:3306/zomato'JDBC_consumer.username = '****'JDBC_consumer.password = '****'However, not able to set table config. Same with Azure Data Lake Storage, we are able to set below configurations:Azure_storage.data_format = 'DELIMITED'Azure_storage.authentication_method = 'Shared Key'Azure_storage.account_shared_key = '***'However, not able to set Account FQDN & Storage Container / File System Please guide us with these configurations.

4 days ago

BikramHeadliner

asked in Share your Best Practices

How to remove special characters from field names in streamsets data collector?

Solution.The issue can be handled by Field mapper processor. Below the syntax for the same. ${str:replaceAll(f:name(), '[.\\-/()!&%^$#@]','')} Attached the screen shot for the reference. NB : The issue also can be handled in different ways e;g(Expression Evaluator, scripting processor etc)

5 days ago

CS_1970Fan

asked in Got a Question?

Orchestration Pipeline - Getting Variable Back from Called Job?

Hello, I am new to Streamsets, I have tried to find answer to this question.I can see a worked example at my place of work where an Orchestration Pipeline can pass a variable down to a Transformer Job in the Runtime Parameters, eg… Collector Pipeline Fragment (Jython)for record in sdc.records: try: if : ... else: ... record.value['LATEST_DATE'] = …. sdc.output.write(record) Orchestration Pipeline then executed Transformer Job node “branch_one_run” passing the value as a parameter in the Runtime Parameters"LATEST_DATE": "${record:value("/LATEST_DATE")}", Subsequent Stream Splitter node in the Orchestrator pipeline can determine course of action to take based on whether the call to the Transformer job was successful… ${record:value('/orchestratorTasks/branch_one_run/success')==true}But is it possible for the Orchestrator to also get a bespoke variable containing value / values back from the called Transformer job that it can use to pass as a parameter to subseq

5 days ago

Dirk B.Fan

asked in Got a Question?

Data Collector going to disconnected mode all the time

I get every 10 seconds the following sequence in the data collector log:DPM reachable but returning '404' HTTP status on login RemoteSSOService RemoteSSOService is not active, changing to disconnected mode FailoverSSOService RemoteSSOService is active, changing to connected mode FailoverSSOService Most things seem to work. The engine (v5.3) is connected to Control Hub with a proper setup ( https://eu01.hub.streamsets.com/ ). As the URL is not logged (even in trace) I cannot see what could solve this. Does anyone have an idea what is happening here?

5 days ago

saurabhsakpalbhpFan

asked in Got a Question?

Http Client Pagination Error

I am essentially using a 2 step authentication to run a API call, first step is logging in and storing the cookie as a session id (post request), second is making a get request with that cookie to retrieve all the information. Now when I run this pipeline once it works fine. Problem is when I incorporate pagination, I need to ingest 1300 records and max records per api call is 500, so I will need to run in 3 batches.I keep getting this error:java.lang.ClassCastException: java.util.LinkedHashMap cannot be cast to java.util.List when i set to pagination, is there anyway I can workaround this.Note: Also I notice in data format for Http Client Processor, I only have option to click Json, but no further control e.g Array of json or multiple json objects, but when I use Http Client Origin, I have the option, is this a limitation within stremasets?

5 days ago

Santhosh KumarFan

asked in StreamSets Academy

Rest API

how to write rest api in python code..

10 days ago

Santhosh KumarFan

asked in StreamSets Academy

Parquet file Data collector

Hi teami using streamset control hub 3.x ,if possible to read and write parquet file in data collector ?

10 days ago

saikat_chatterjeeRoadie

asked in Got a Question?

Unable to create a deployment using python SDK

After creating an environment through python sdk, when I am trying to create a deployment using the scripts I am getting the error described bellow: >>>deployment_builder = sch.get_deployment_builder(deployment_type='SELF')>>> # sample_environment is an instance of streamsets.sdk.sch_models.SelfManagedEnvironment>>> deployment = deployment_builder.build(deployment_name='Sample Deployment',... environment=sample_environment,... engine_type='DC',... engine_version='4.1.0',... deployment_tags=['self-managed-tag'])Traceback (most recent call last): File "<stdin>", line 2, in <module>NameError: name 'sample_environment' is not definedPlease help me regarding this.

10 days ago

lakshmi_narayanan_tFan

asked in StreamSets Academy

data type change by field mapper

if it possibile to convert data type by field mapper

11 days ago

saikat_chatterjeeRoadie

asked in Got a Question?

pip3 install

I am trying to run the following command:“pip3 install streamsets~=5.0” in my VM to activate the sdk for python without using the activation key as shown in the documentation but unfortunately I am receiving the following error: WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])"))': /simple/streamsets/WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])"))': /simple/streamsets/WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])"))': /

15 days ago

praveenmittalFan

asked in Got a Question?

Streamset Snowflake staging issue

Hello Everyone I am facing following issue. Any inputs will be helpful. We are using Streamsets to load data into Snowflake. Our Snowflake instance is hosted in the region [EU-WEST-1] and S3 (for Stage) is in [EU-WEST-2] We configured Snowflake connector [Staging] in the StreamSets to use S3 region same as Snowflake. With this setting in place, we see following errors getting logged in S3 bucket (which is meant to capture the errors):"errorMessage": "SNOWFLAKE_59 - Could not get a Stage File Writer instance to write the records: 'The bucket is in this region: eu-west-2. Please use this region to retry the request", If we change S3 Region for the Snowflake connector in the Streamsets to [EU-WEST-2], we see following error in the Streamsets UI.SNOWFLAKE_19 - Could not access the S3 bucket associated with the stage: com.amazonaws.SdkClientException: Unable to execute HTTP request: s3.eu-west-2.aws.amazonaws.com: Name or service not known Streamset documentation says following: “To stage d

16 days ago

Page 1 / 26

Badge winners

AkshayJadhavhas earned the badge Eager to help
john.durkinhas earned the badge Eager to help
samhas earned the badge Eager to help
saleempothiwalahas earned the badge Eager to help
Bikramhas earned the badge Community Champion 🏅

Show all badges

Terms & Conditions

Sign up

Already have an account? Login

Social Login

Username *

E-mail address *

What I do... *

Data Leader Data Architect Data Engineer Data Scientist Other

Company *

Country *

Zip Code *

Marketing Communications

Yes No

Password *

I have read and Agree to the Website Terms of Service and I have read and acknowledged the Privacy Policy.

loginBox.register.email_repeat

Login to the community

No account yet? Create an account

Social Login

Username or Email

Password

Remember me

Forgot password?

Enter your username or e-mail address. We'll send you an e-mail with instructions to reset your password.

Username or e-mail

Back to overview

Scanning file for viruses.

Sorry, we're still checking this file's contents to make sure it's safe to download. Please try again in a few minutes.

This file cannot be downloaded

Sorry, our virus scanner detected that this file isn't safe to download.