- - Knowledge base
Groups
StreamSets Live
Write for Us

Community
Platform Q&A
Got a Question?

Got a Question?

Can't find what you're looking for? Ask it here!

427 Topics
978 Replies

When you subscribe we will email you when there is a new topic in this category

427 Topics

Recently active

Most replies Most views Newest first

asked in Got a Question?

While establishing JDBC connection.

I am getting this type of error JDBC_00 - Cannot connect to specified database: com.mysql.cj.jdbc.exceptions.CommunicationsException. Can anybody help me in this.

asked in Got a Question?

Boolean Field

how to evaluate Boolean field, whether it is true or not? Any Boolean functions? or can it be achieved with expression evaluator, if so how can it be achieved?

Priyanka MynepallyFan

asked in Got a Question?

Expression-Evaluator

I need to insert map in empty-list using expression, how to achieve it in streamsets. emptyList(), emptyMap() functions I have tried but no luck.Can anyone help me please. Ex:-[{“list”: 1}]

asked in Got a Question?

data drift alert messages

Hi team,I am doing data drift lab exercise in streamset academy. In that i am following the step as they are mentioned in the video , in that video it is displaying as 54 records consuming from that kafka but while i am doing the same as they mentioned but i am able to consume only 4 records(twitter data) And also not able see the alert messages. could anyone help me through this please.

asked in Got a Question?

Groovy Script "Origin Stage" to traverse AWS s3 bucket/directory

I use Streamsets and I need a Groovy Script "Origin" that can read aws s3 bucket.Do you know someone who can help with a sample groovy script to iterate a folder and list objects in s3 bucket?So, I only want a access s3 folder and list all filenames, file path, file size. e.gFilesrecord 1: /path to location/File1.csvrecord 2: /path to location/File2.csvrecord 3: /path to location/File3.csv

N_aliOpening Band

asked in Got a Question?

Cannot Rename Files in SFTP dst Server

Hello , I have a simple pipeline where i want some csv files to be moved from s3 bucket to SFTP server,However the files saved in the SFTP server should have different naming that includes the current date, something like:“mer_YYYYMMDDHHmmss.csv”i tried in the file name expression field in the sftp server destination to come up with expression evaluating the above but it always produces errors that functions like YYYY() is not a function.

pedromanuelOpening Band

asked in Got a Question?

Exception when importing fragment in Control Hub

Hi, When trying to import a fragment I get this error:“The transaction has been rolled back. See the nested exceptions for details on the errors that occurred. “Unfortunately I have no idea as to where I can check this exception. I guess it happens on the Control Hub and there would be no way to access the log file?Any ideas?

asked in Got a Question?

Same data in two fields

How to populate the same field value in two different fields of target?src-->csvfield1-->tgt1field1-->tgt2

Priyanka MynepallyFan

asked in Got a Question?

Groovy evaluator

How do I write else if loop in groovy evaluator Example: the if-else loop in expression evaluator is written in below way: ${record:value('/phone_home[0]') != ' ' ? record:value('/phone_home[0]'):(record:value('/phone_home[1]') != ' ' ? record:value('/phone_home[1]'):(record:value('/phone_home[2]') != ' ' ? record:value('/phone_home[2]'): ' '))} How do I implement Same-Example thing in groovy evaluator,Since there are dynamic number of rows coming each time, [sometimes number is 10, sometimes 100 etc, so need to write for-loop]

asked in Got a Question?

Multiple Header lines

Can we populate multiple header lines(3) with different header names using data collector?I am using csv as source, which will contains one header.Need to convert csv to XLS, and then the xls should have 3 headers which ae not present in source.3 headers followed by data from the source.

Priyanka MynepallyFan

asked in Got a Question?

json to list

How to convert json array into list

pedromanuelOpening Band

asked in Got a Question?

Start Pipline (orchestration) origin/processor not available in Data Collector?

Hi, while getting familiar with orchestration stages found that the “Start Pipeline” origin/processor are not available at Data Collector engines (according to https://docs.streamsets.com/portal/platform-datacollector/latest/datacollector/UserGuide/Orchestration_Pipelines/OrchestrationPipelines_Title.html#OrchestratorStagesActually they not seem to be available anywhere? Where are they actually available if not at the collector engine? Is this a documentation issue?

pedromanuelOpening Band

asked in Got a Question?

Creating a job instance from template without immediatly starting it?

Hi, just wondering, shouldn’t it be possible to create a job instance from a job template without having it started instantly? Make it a choice to either start it right away or leave the start to a later time?

pedromanuelOpening Band

asked in Got a Question?

Authentication issue with Control Hub API credentials

Hi, when trying to run a pipeline with the “Start Job” origin the pipeline fails and the log shows:com.streamsets.pipeline.lib.ControlHubApiError: Control Hub security token expired or invalid: {"ISSUES":[{"code":"SSO_01","message":"User not authenticated"}]} The credentials I used are newly generated credentials and the stage is configured as can be seen in the screen shot with the credential id and token. Why are these credentials rejected by the Control hub? If I use the exact same credentials in a call to CURL I get a perfectly fine response showing my user details(request like: curl -X GET https://na01.hub.streamsets.com/security/rest/v1/currentUser -H "Content-Type:application/json" -H "X-Requested-By:curl" -H "X-SS-REST-CALL:true" -H "X-SS-App-Component-Id: $cred_id" -H "X-SS-App-Auth-Token: $cred_token" -i)Any ideas?

asked in Got a Question?

Aggregation Chart not generated

Hi,I have a pipeline which is using windowing aggregator to perform couple of window aggregations. When I run the pipeline, I don’t see the chart generated but instead I see the below. Am I missing something?Regards,Karthik

asked in Got a Question?

Load Date column from Oracle to Snowflake Failure

Hi team we built a pipeline to load oracle table to Snowflake using StreamSets, there is one date column, and in oracle the value is always like "2012-04-25 00:00:00.000", we create this column as date in snowflake too, however it failed the load, only after I changed the data type of Snowflake corresponding column as TIMESTAMP_NTZ(9) and then it can be loaded.So how can I convert this column to Date in StreamSets when loading, which processor I can use and what is the evaluation of this convert? by the way what is the root cause of failure for this direct date to date column loading from oracle to snowflake with StreamSets?

Priyanka MynepallyFan

asked in Got a Question?

Converting [ to {

In a json file, I need to replace [ with{ in between json file, How to achieve that using exp evaluator Example: { "student": [ { "id":"01", "name": "Tom", "lastname": "Price" }, { "id":"02", "name": "Nick", "lastname": "Thameson" } ]} to{ "student": { { "id":"01", "name": "Tom", "lastname": "Price" }, { "id":"02", "name": "Nick", "lastname": "Thameson" } }}

asked in Got a Question?

DC Pipeline Validation pass but Failed for draft run, log no clear message for rootcause of failure

Hi Team, I am building a DC pipeline to load a small table data from Oracle origin to Snowflake, validation pass but failed for draft run (I click “Reset Origin and Start”), I searched the log, there is only warningWARN Exception thrown while trying to remove temporary directory /opt/streamsets-datacollector/tmp/sdc148432276299548840 ”can you please guide hwo to debug it? Thank you.

asked in Got a Question?

Date Conversions in Data Collector

Getting date (YYYY-MM-DD) in a string field from source, need to convert into date type of format(DD/MM/YYYY).Tried Fieldtype converter, but of no use.Any suggestions?

ramakrishna_raoFan

asked in Got a Question?

Orchestration records schema for start jobs

Can you please share the schema for orchestration record for start jobs, I am not seeing job name in the schema which is required for triggering the downstream jobs. All I can see is the attached image of orch record schema.

asked in Got a Question?

Error when inserting list type to Snowflake

Hello, I have a response from a HTTP client stage that has been pivoted. Some of the fields are lists and these return errors when trying to load into Snowflake.Ideally I’m trying to get the lists (array) inserted as JSON in variant columns for flattening later.I can use a JSON parser however I need to do that for every field and there are some cases where the fields are dynamic.

asked in Got a Question?

How to perform a cyberarkpassword lookup for all hosts in an inventory group and write the keys out to individual pem files?

Ansible version 2.7.9I'm writing an ansible playbook to deploy an piece of software to a linux environment. SSH access to these systems is protected by a CPM (Cyberark), used as an ssh key manager.I've got most of the logic figured out, save for one piece. The playbook needs to loop through hosts in an inventory group, lookup the ssh private key in Cyberark for each host and then use each key to ssh into each host in the inventory group to install the software. I'm struggling with how to make that work in ansible.I've read through the add_host and cyberarkpassword documentation, as well about 4 hours worth of searching stackoverflow and blogs, and couldn't find a single example even close to what I'm trying to do.As far as how I think it should work:Using the cyberarkpassword lookup module, loop through hosts in inventory group specified by {{ env }}. Value for this will be passed in through --extra-args. Retrieve the ssh private key for each host. Register the output from the lookup,

asked in Got a Question?

How to generate row numbers

Hi team,Need to create rank/row number function for each row just like how we do in SQL.How to implement using expression evaluator/groovy etc.can anyone please help me with this .

asked in Got a Question?

ksqlDB

can we use ksqlDB in streamset?

asked in Got a Question?

How to send Form-data in http client for POST Call

I am new to Stremsets.I am looking to perform a POST call using HTTP client and the POST Body is a form-data with file upload . We are able to achieve this in POSTMAN. Is there a way we can do this in streamsets data controller pipeline.

1
2
3
4
5
...
18

Page 2 / 18

Badge winners

samhas earned the badge Eager to help
saleempothiwalahas earned the badge Eager to help
Bikramhas earned the badge Community Champion 🏅
Bikramhas earned the badge Eager to help
Pradeephas earned the badge Eager to help

Show all badges

Powered by inSided

Terms & Conditions

Sign up

Already have an account? Login

Social Login

or

Username *

E-mail address *

What I do... *

Data Leader Data Architect Data Engineer Data Scientist Other

Company *

Country *

Zip Code *

Marketing Communications

Yes No

Password *

I have read and Agree to the Website Terms of Service and I have read and acknowledged the Privacy Policy.

loginBox.register.email_repeat

Login to the community

No account yet? Create an account

Social Login

or

Username or Email

Password

Remember me

Forgot password?

Enter your username or e-mail address. We'll send you an e-mail with instructions to reset your password.

Username or e-mail

Back to overview

Scanning file for viruses.

Sorry, we're still checking this file's contents to make sure it's safe to download. Please try again in a few minutes.

OK

This file cannot be downloaded

Sorry, our virus scanner detected that this file isn't safe to download.

OK