- - Knowledge base
Product Updates
Events

30-Day Free Trial: It’s Never Been Easier To Get Started With StreamSets

4 months ago

Home
Community overview
StreamSets Platform
Community Articles and Got a Question?

Community Articles and Got a Question?

Can't find what you're looking for? Ask it here or check out the Community articles

945 Topics
2,248 Replies

When you subscribe we will email you when there is a new topic in this category

945 Topics

Recently active

Most replies Most views Newest first

skyRoadie

asked in Community Articles and Got a Question?

Produce Events Error

When I am testing the Produce Events of a component. When I uncheck the "Produce Events" box, the component configuration displays normally. However, when I check it, I always encounter the following error. Do you know what might be causing this? Has anyone else experienced this issue?

1 month ago

nathanStreamSets Employee

posted in Community Articles and Got a Question?

Technical Service Bulletin 2023-08-25 (TSB) - Azure Synapse, Delta Lake, Google BigQuery, and Snowflake destinations issues with DateTime data

StreamSets Support team would like to inform you of an issue where StreamSets Data Collector pipelines using destinations such as Azure Synapse, Delta Lake, Google BigQuery, or Snowflake might corrupt the data for the DATE/DATETIME/TIME fields.Products Affected: StreamSets Data Collector Releases Affected: StreamSets Data Collector 5.6.0 and prior versions Users Affected: StreamSets Data Collector pipelines using Azure Synapse destination, Delta Lake destination, Google BigQuery destination, and Snowflake destination. Severity: High Description: It has been identified that in some cases, DATE/DATETIME/TIME datatype fields could be corrupted. Users could be finding that their DATE/DATETIME/TIME data fields contain incorrect dates. This issue happens only if all the following conditions take place:The pipeline writes to one of the following stages: Azure Synapse destination Delta Lake destination Google BigQuery destination Snowflake destination. The pipeline writes to multiple tab

1 month ago

nathanStreamSets Employee

posted in Community Articles and Got a Question?

Technical Service Bulletin 2023-07 (TSB) - Changes in Amazon S3 Stage Properties from StreamSets Data Collector version 5.6.0

StreamSets Support Team would like to inform you that starting with Data Collector 5.6.0 pipelines with Amazon S3 stages fail to execute if the "Bucket" or "Bucket and path" property contains a forward slash (/). Products Affected: StreamSets Data Collector Releases Affected: Starting from StreamSets Data Collector version 5.6.0 Users Affected: StreamSets Data Collector pipelines that include the forward slash (/) in the bucket property of Amazon S3 stages. Severity: Medium/High Description: Starting with version 5.6.0, you can no longer include the forward slash (/) in the following properties for Amazon S3 stages due to an Amazon Web Services SDK upgrade:Bucket property for the Amazon S3 origin Bucket and path property for the Amazon S3 destination and executorAdditional details regarding this change can be found in the Upgrade Impact documentation.Immediate action required: YesResolution: Avoid using the forward slash (/) in the bucket name as this may cause signature mismatch error

1 month ago

nathanStreamSets Employee

asked in Community Articles and Got a Question?

paulStreamSets Employee

posted in Community Articles and Got a Question?

SCH 3.x On-Prem - Database upgrade and potential subsequent rollback

Upgrading one or both of the backend databases for SCH 3.x (relational database and timeseries database) has no effect on SCH itself assuming that you are upgrading to a supported version of the database. You will need to either bring down SCH during the upgrade, or put the database into read-only mode, and of course take a backup of the database before beginning the upgrade.In a scenario where a subsequent rollback of the database to the backed-up version is contemplated, be aware that there will be some data loss, even if there were no new jobs created or other changes made (code freeze). Some of this data includes, but is not limited to:Any clean up of the DB Writing to InfluxDB Writing to the PostgreSQL Timseries DB (job metrics that don’t go to InfluxDB but to the SQL DB) Updating jobs statuses Running scheduler tasksThese data would need to be manually reconstructed from the upgraded database and backported to the rolled-back database.If you have any additional questions, please

1 month ago

paulStreamSets Employee

posted in Community Articles and Got a Question?

CONTROL_HUB_API_00 error in SDC log

In SDC versions up to 5.5.x, you may see CONTROL_HUB_API_00 errors which can cause pipelines and jobs to fail. There is no context in the logs as to what was happening that led up the to the error; it’s just a generic error that occurs when any type of API call to SCH fails.In SDC 5.6.x, this error was removed and more helpful logging was added that captures the specific API call that was being made and what the exception was. If you are encountering the CONTROL_HUB_API_00 error in an older version of SDC, StreamSets Support recommends either upgrading to at least 5.6.x, or install SDC 5.6.x or higher in a non-production environment and reproduce the issue. This will allow StreamSets Support to more effectively assist you and find the root cause of the issue.

1 month ago

john.mcavoyStreamSets Employee

asked in Community Articles and Got a Question?

MapR Client Library Upgrade causes "Provider could not be instantiated" or "ClassNotFoundException"

IssueAfter upgrading MapR client libraries or applying a MapR client patch on an SDC node, pipelines using Hadoop stages fail with some Provider could not be instantiated error or ClassNotFoundException at startup, indicating some Hadoop class cannot be found within the Java classpath.BackgroundDue to licensing restrictions, StreamSets cannot distribute MapR libraries with Data Collector. As a result, you must perform additional steps to enable the Data Collector machine to connect to MapR.When the bin/streamsets setup-mapr prerequisite is completed, SDC creates symlinks to the installed MapR Client libraries on the system in the streamsets-libs/streamsets-datacollector-mapr-version-lib/lib directory.When applying a MapR Client patch or upgrading the MapR Client package on the SDC machine, some JAR files in the MapR client installation patch will be renamed which will cause broken symlinks within SDC’s MapR stage library directory.SolutionThe Provider could not be instantiated exceptio

1 month ago

bsayedRoadie

asked in Community Articles and Got a Question?

Field mapper not mapping all fields

I have a field mapper to remap camel case fields to snake case. This is the message i am trying to save:{ "id": "e35fd529-613f-5e79-036f-f37c2801041d", "batchName": "SIT Axle3+ 9_12_test", "batchSize": 35, "numberOfAudited": 35, "batchType": "TRIP_BASED", "batchMode": "RANDOM", "batchFacility": "71", "batchDirection": "West", "batchPlaza": "", "batchLane": "", "fromTimestamp": 1694538060, "toTimestamp": 1694566800, "createdTimestamp": 1694626323, "createdBy": "mmccammon@etcc.com", "completedTimestamp": 1697039389, "completedBy": "asayed@etcc.com", "auditDetailList": [ { "id": "a4f274d5-8de5-4c54-ae70-b6a498ab4e9e", "auditBatchId": "e35fd529-613f-5e79-036f-f37c2801041d", "eventId": "49dcbe2c-51a8-11ee-9348-3cecefd01508", "eventType": "TRIP", "eventTimeStamp": 1694549374, "auditResult": false, "auditCorrections": [], "auditedTimestamp": 1694635817, "auditedBy": "ppradeep@etcc.co

1 month ago

NCollinsStreamSets Employee

published in Community Articles and Got a Question?

Zero to StreamSetsTutorial

An in-depth guide into starting StreamSets, deploying engines, and running pipelines. Overall architecture recapThe control plane of StreamSets Platform, Control Hub, hosted in the cloud. The data plane encompasses StreamSets Data Collectors (SDCs), which should be installed in a customer’s network, be it in the cloud or on-premise. Consider these options for SDC installation (not all of them are recommended for production scenario): Tarball (binary download) Docker container Windows PC not supported OK – the fastest to implement Linux/Mac OK with configured prerequisites Depending on your choice, find below instructions for major steps of tarball and Docker options. Provide and prepare your machineUse your own laptop or create a VM in the cloud of your choice. VM size should be determined by the minimum requirements for SDC installation: 2 vCores, 1GB RAM, 6GB disk space.If on Linux, run a system update command, like yum updat

270

1 month ago

vivek.sharmaFan

asked in Community Articles and Got a Question?

Json to XML with namespaces

hello,I’m new to streamsets so it might be an obvious question!I’m using “data generator” to accept the json input and transform to XML, however I want to add few namespaces in the generated XML output. Please can anyone suggest how to achieve that?This is the output “data generator” is producing:<?xml version="1.0" encoding="UTF-8" standalone="no"?><root> <id>file</id> <value>File</value> <popup> <menuitem> <value>New</value> <onclick>CreateDoc()</onclick> </menuitem> <menuitem> <value>Open</value> <onclick>OpenDoc()</onclick> </menuitem> <menuitem> <value>Save</value> <onclick>SaveDoc()</onclick> </menuitem> </popup></root>What changes do I need to make it something like this (the element id is under “b” namespace):<?xml version="1.0" encoding="UTF-8" standalone="no" ?&g

1 month ago

levanyeFan

asked in Community Articles and Got a Question?

running error pipeline couldn`t force stop

Running Error pipeline couldn`t stop, when I click the button of force stop, nothing happened, what should i do?

1 month ago

uday8770Fan

asked in Community Articles and Got a Question?

Metadata driven ingestion framework

HiI'm currently working on implementing a metadata-driven pipeline. One of the key components of this pipeline is a job that's responsible for transferring data from Teradata to Snowflake. To make this process dynamic and reusable, I've introduced a job which will load the data to snowflake.Furthermore, I've established a separate pipeline that serves the purpose of fetching metadata from Teradata. This metadata contains important information, including the values for ${dbname} and ${Tname}. The intention is to pass these parameter values to the "startjobs" process. However, I'm currently unsure about how these parameter values will be effectively utilized within the job I've created for the data transfer from Teradata to Snowflake. I would greatly appreciate any guidance or assistance in this regard.

1 month ago

Paawan AngraFan

asked in Community Articles and Got a Question?

Start Jobs origin throws error regarding runtime parameters

I am trying to create a job instance from the job template. I have passed the runtime parameters to the start job component but get the following error. - Start failed for job template ID: START_JOB_14 - Start failed for job template ID: abc: com.streamsets.pipeline.lib.ControlHubApiException: Unexpected exception from Control Hub: START_JOB_05 - Failed to parse runtime parameters for job ID: abc, error: 400: I have made sure that the parameter name is correct. Please help me with why is it not able to parse the runtime parameters?

1 month ago

cortiez21Fan

asked in Community Articles and Got a Question?

Common IDP

Hello, AVG has quarantined CefSharp.BrowserSubprocess.exe as a possible IDP—generic infection. Slido doesn't function, and it's been quarantined.How should I proceed?

1 month ago

skyRoadie

asked in Community Articles and Got a Question?

Encrypt and Decrypt Fields 1 error

When I use the Encrypt and Decrypt component to encrypt a string field, I encounter the following error. Could you please tell me what might be causing this issue? Also, when I convert the string type to a byte array or base64 characters, it doesn't work either.

1 month ago

Satish PanchwareFan

asked in Community Articles and Got a Question?

Facing issues while decrypting text

I am working a sample involving encryption and decryption and facing issues.Here is the problem:While using the "Encrypt and Decrypt Fields" processor when we encrypt something using it, it outputs the resulting fields in a ByteArray format, and when we save it somewhere like localFS the ByteArray gets converted into String Format,Now when it comes to the decryption part, The processor needs ByteArray format only , So when I try to convert the String into Byte array it results in some unrecognizable characters which causes error.Encryption:Decryption:Please let me know if you have any pointers.

1 month ago

lizlarssonFan

asked in Community Articles and Got a Question?

Discarding records but passing value to next stage

Hi!We have a pipeline where we need to use two http stages. First we need to get all objects of type A and then for each A object we need to get all objects of type B.1. First we have an http origin where we get all objects of type A. The only thing of interest in the response is the id of object A2. For each A object, we make another http request (using a http client stage) to retrieve all the objects of type B. The url looks like this:https://www.example.com/rest/1.0/foo/${record:value('/id')}/bari.e. we use the id of A which we got in the first stage. We are not really interested in the records we get in stage one. The only thing we are interested in is the id which will be used in the URL in stage two.How can we discard all records received in stage one and just pass the id to stage two? Is it possible to write to an environment variable?Any help would be appreciated!

1 month ago

sudarshan_shinde1Fan

asked in Community Articles and Got a Question?

Get list of pipeline name using Python SDK

Hi Team,I am using below mentioned method to get list of all pipelines.But it is returning all the values like version, commit id.I want only pipeline name.sch = ControlHub(credential_id, token)pipeline_list = sch.pipelines.get_all() I tried sch.pipeline.name but it is not supporting.

1 month ago

Jhon01Fan

asked in Community Articles and Got a Question?

The API functions correctly in Speakatoo POSTMAN but is not working within my integrated application.

"I've noticed that the Speakatoo API functions properly when tested in POSTMAN, but it's not performing as expected within my integrated application. I'm seeking assistance to understand and address the issue with the API integration in my application."

2 months ago

southwr62Fan

asked in Community Articles and Got a Question?

Problem with Pagination

Hi, I am using the HTTP client origin, and since I need to load a lot of records, I need to use pagination.my url ends with ‘auditLogs?offset=${startAt}&pagesize=5&isascorder=false&order=id’ cannot give more for confidentiallity reasons. This is a post requestwhen I set pagination to offset or page number, all I get is the first 5 records, offset is actually used to denote the start record of the data returned, not a page in this instance.If I set pagesize to 1, which I think is how I need to do this, I only get 1 result it doesn’t seem to continue, I have tried with both page and offset set as the pagination method.any ideas on what is wrong are welcomed

2 months ago

abibatoul20Fan

asked in Community Articles and Got a Question?

Enable StreamSets Control Hub

Bonsoir, j’ai installé StreamSets Data Collector mais je veux activer le control hub. Quand j’essaie ça me génère l’erreur suivante : DPM Login failed, status code '403': {"message":"Authentication failed"} ( View Details... )

2 months ago

swapniel22Fan

asked in Community Articles and Got a Question?

Seeking Assistance with Column Renaming and SDC Field Update in StreamSets Pipeline. SDK

Hello,I am currently running a pipeline that reads data from Oracle Bulkload as the source and utilizes JDBC Producer for PostgreSQL as the destination.Now, I am facing an issue where I need to rename a column from 'FIRST_NAME' to 'second_name' in the destination table, which is in PostgreSQL. I've attempted the following methods:1. Using Oracle Bulkload with a Field Renamer stage followed by JDBC Producer. - Error: JDBC_07 - Invalid column mapping from field '/Second_name' to column '/first_name'. 2. Using JDBC Consumer Table with a Field Renamer stage followed by JDBC Producer. - Error: JDBC_07 - Invalid column mapping from field '/Second_name' to column '/first_name'. 3. Using JDBC Consumer Table with a Field Renamer stage followed by JDBC Producer. - Error: com.streamsets.pipeline.api.base.OnRecordErrorException: JDBC_90 - Record doesn't have any columns for table 'table1'. - Note: For option 3, I left the Field to Column Mapping empty in JDBC Producer. ……………………………………………………

2 months ago

SperchFan

asked in Community Articles and Got a Question?

Troubleshooting Jython Evaluator Code

I am trying to use a Jython evaluator stage to remove the formatting and code encapsulation to only present the string content to be inserted in the destination stage (Snowflake). I have been searching online for ideas and guidance but have found very little, and most not helpful and/or pertinent to my quandary. An example of the string incoming to the Jython stage:This position is embedded within our Data Analytics practice. Leverage your curiosity and problem-solving skills to explore, discover, and predict patterns contained within data sets for a wide range of government clients. This includes the derivation of clear narratives that help our clients understand their data and how those insights address their research questions. <span style="color: #00000

2 months ago

Page 2 / 38

Badge winners

Sanjeevhas earned the badge Eager to help
AkshayJadhavhas earned the badge Eager to help
john.durkinhas earned the badge Eager to help
samhas earned the badge Eager to help
saleempothiwalahas earned the badge Eager to help

Show all badges

Terms & Conditions

Sign up

Already have an account? Login

Social Login

Username *

E-mail address *

What I do... *

Data Leader Data Architect Data Engineer Data Scientist Other

Company *

Country *

Zip Code *

Marketing Communications

Yes No

Password *

I have read and Agree to the Website Terms of Service and I have read and acknowledged the Privacy Policy.

loginBox.register.email_repeat

Login to the community

No account yet? Create an account

Social Login

Username or Email

Password

Remember me

Forgot password?

Enter your username or e-mail address. We'll send you an e-mail with instructions to reset your password.

Username or e-mail

Back to overview

Scanning file for viruses.

Sorry, we're still checking this file's contents to make sure it's safe to download. Please try again in a few minutes.

This file cannot be downloaded

Sorry, our virus scanner detected that this file isn't safe to download.