- - Knowledge base
Groups
StreamSets Live
Write for Us

Community
Platform Q&A

Platform Q&A

Get inspired and gain all the knowledge you need

456 Topics

Recently active

Most replies Most views Newest first

Priyanka MynepallyFan

asked in Got a Question?

pedromanuelOpening Band

asked in Got a Question?

JDBC Producer record error: JDBC_90 - Record doesn't have any columns for table 'schema.tablename'

Hi, running the PostgreSQL Metadata processor and the JDBC Producer in a pipeline with the Oracle CDC client I run into the above error when adding a column to a table on the source side.Checking the destination table I found that the Metadata processor altered the target table perfectly but it seems that the JDBC producer does not recognize the new field and assumes that the record, consisting of two columns, the primary key column and a value for the new added column does not carry any relevant data and issues the error.I attach a screenshot with the pipeline and data.May be someone has an idea what’s wrong? I checked the articles about data drift and cannot find any hint about what I may have configured wrongly. Any help is greatly appreciated!Here the stacktrace:com.streamsets.pipeline.api.base.OnRecordErrorException: JDBC_90 - Record doesn't have any columns for table 'bunkerstreamsets.s_product' at com.streamsets.pipeline.lib.jdbc.JdbcGenericRecordWriter.processQueue(JdbcGener

5 days ago

AmreshHalderFan

asked in Got a Question?

File encryption using RSA algorithm

Is there a way we can use RSA public and private key to encrypt and decrypt the contents of a file in a pipeline?

5 days ago

N_aliOpening Band

asked in Got a Question?

cannot write headers in csv file

is there a way to write headers first like the ones in yellow rows in the screenshot before adding the needed columns after in a csv file ?after the 4th row the data should be displayed as a normal csv file, However i cannot figure out how to write the first 3 rows .

6 days ago

rashikhFan

asked in Got a Question?

Error while installating Streamsets python packages

I am getting error like microsoft build c++ not found in microsoft visual code error while installing streamsets python packages .Can anybody provide the solution for the same.pip install streamsets

10 days ago

Priyanka MynepallyFan

asked in Got a Question?

Json to XML

How to convert json file obtaining from source-API into xml file? Example json: ============{"menu": { "id": "file", "value": "File", "popup": { "menuitem": [ {"value": "New", "onclick":"CreateDoc()"}, {"value": "Open", "onclick": "OpenDoc()"}, {"value": "Save", "onclick": "SaveDoc()"} ] } }}

11 days ago

DavidCFan

asked in Got a Question?

Pipeline fails with error - Failed to update metadata aft er 60000 ms.

Hello,Does somebody have experience with this issue?I have pipeline with HTTP server on origin side and Kafka producer on destination side. The pipeline restarts more or less regularly with this error message: 2022-09-07 13:24:26,245 [user:*admin] [pipeline:rx_syslog/rx_syslog] [runner:] [thread:ProductionPipelineRunnable-rx_syslog-rx_syslog] ERROR ProductionPipelineRunnable - An exception occurred while running the pipeline, org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms.There is number of following messages before failure2022-09-07 13:24:23,594 [user:*admin] [pipeline:rx_syslog/rx_syslog] [runner:4] [thread:kafka-producer-network-thread | topicMetadataClient] WARN NetworkClient - [Producer clientId=topicMetadataClient] Error while fetching metadata with correlation id 436 : {avro..syslog.viptela=UNKNOWN_TOPIC_OR_PARTITION}2022-09-07 13:24:23,694 [user:*admin] [pipeline:rx_syslog/rx_syslog] [runner:4] [thread:kafka-producer-network-thread |

12 days ago

AkshayJadhavStreamSets Employee

posted in Share your Best Practices

Docker run command is failing with "Container ID 20159 cannot be mapped to a host ID."

Environment:Data Collector Docker Image. Issue: This has been observed that docker run command with any version of data collector image is failing with following error when the user has restrictions from OS side to, not to use higher UIDs.#docker run -d -e datacollector:4.4.1Unable to find image 'datacollector:4.4.1' locally4.4.1: Pulling from datacollector801bfaa63ef2: Pull complete755713f1a3c1: Pull completeeb3cc996a4d2: Pull completecc65d2e08110: Pull completea78edd797c42: Pull complete2837f4f48b17: Extracting [==================================================>] 425.6 MB/425.6 MBfb4a45fc2613: Download completeee56bb53e5ce: Download complete47f24455e547: Download completee0e00f796eb2: Download completeb134ff785401: Download complete/usr/bin/docker-current: failed to register layer: Error processing tar file(exit status 1): Container ID 20159 cannot be mapped to a host ID.See '/usr/bin/docker-current run --help'.In the Data Collector v4.4.1 docker image, we’ve hardcoded the SDC_UI

15 days ago

SubhaFan

asked in Got a Question?

How to pass parameter from previous stages as input to a fragment parameter

Hi i am trying to pass the value from a field from previous stage as input to the fragment parameter . Basically I have created a fragment with shell stage and i am using this fragment in a pipeline . I want to pass the file path which i have formed by using some expressions in expression evaluator and have assigned to a field called Filename. I want to pass value from this field (“Filename”) to the fragment parameter. Can anyone suggest how to pass this .

16 days ago

chisouFan

asked in Got a Question?

ArrayIndexOutOfBoundsException when renaming array elements

I’m building a data collector pipeline in which I want to rename elements within an array.The input are JSON files read from a data lake, format is a single document with an array of identically structured documents, e.g.{ "entries": [ { "wd:Employee_User_ID": "lol", "wd:Scheduled_Weekly_Hours": "40", "wd:fte": "100" }, { "wd:Employee_User_ID": "rofl", "wd:Scheduled_Weekly_Hours": "37.5", "wd:fte": "100" }]} I need to rename these fields, so I’ve put the following into the Field Renamer processor:/entries[*]/'wd:Employee_User_ID' → /entries[*]/alias But that gives an java.lang.ArrayIndexOutOfBoundsException: -1 at java.util.ArrayList.elementData(ArrayList.java:424) at java.util.ArrayList.get(ArrayList.java:437) at com.streamsets.datacollector.record.RecordImpl.get(RecordImpl.java:309) at com.streamsets.datacollector.record.RecordImpl.has(RecordImpl.java:374) at com.streamsets.pipeline.stage.processor.fieldrenamer.FieldRenamerProcessor.process(Fi

16 days ago

srivasthavaFan

asked in Got a Question?

How to call hashicrop vault in pipeline ?

Hi Team,How to store username and password in hashicrop vault and how to call hashicrop vault in a pipeline and also what to take as origin and as destination for that pipeline ?

16 days ago

ravikumarFan

asked in Got a Question?

How to redirect to jira dashboard from servlet

I want to redirect from servlet to jira's dashboard.What i am trying to do : when the user click create isse and fill all the details and submit, i check if the user with openid, after i check, i need to redirect the user to jira dashboard or i should be above to redirect to any workflow screen.I tried the redirect with the following lines.response.sendRedirect("/Dashboard.jspa");after the servlet finishes, i want to redirect, i have the HttpServletResponse object. when i use the above redirect method it doesn't redirect.please help.

16 days ago

tdeshappriyaFan

asked in Got a Question?

Configuring Kafka Producer to send data to confluent cloud

I’m building a pipeline which data source will be the twitter API and data destination will be the confluent cloud. I have added the broker URL according to my Confluent Cloud Server endpoint. I’m bit stucked in configuring the security details in Streamsets. Currently I’m getting the below issue and I appreciate if anyone could help me on this. From confluent cloud, I have a API key and Secret to authenticate. @Marty Norris

16 days ago

pedromanuelOpening Band

asked in Got a Question?

Jython Evaluator erases field attributes?

Hi, just want to get familiar with the jython evaluator since I want to do some bulk operations on fields with a certain type which would be hard to do otherwise.Playing with the default script code provided (iterating through the records and writing back each record unchanged) I noticed that all the field attributes (e.g. precision and scale for Decimal types) are gone.Is this a defect or do I miss something? Would be very inconvenient to recreate these attributes.Please refer to the Preview screen shot.Any comments?

17 days ago

SebasRFan

asked in Got a Question?

CDC Oracle Problem

Hi everyone, my name is Sebastian and rigth know Im start working with Streamsets, the plataform looks good but I have some problems y want to bring data from Oracle Database Insert Update and Delete, all good at this point the problem is i want to read only one table but i watch the CDC reads al SCN of the datatable and when make the operation (Insert, Update, Delete) the SCN is 2238200632 and in the pipeline is 2237969670 so is huge space that the pipeline needs to record to take the operation, so my question is if is possible to improve or increased speed of read to try to catch the SCN of database? To operation on the DBB and go to Streamset is like 1 or 2 hours and is impossible to work with this teams in a table that only have 300 registers. I hope one of you can help me!! Thank you

18 days ago

Priyanka MynepallyFan

asked in Got a Question?

Control Charecters replacement

How to replace control charecters in streamsets.Data is coming from Kafka in UTF-8 charset and along with “i, Inverted Question-mark,½ symbol”How to replace this using Groovy/Expression Evaluator /Jython

18 days ago

BhaskarFan

asked in Got a Question?

Tensor flow model creation and import into streamsets

Hi - Looking to explore the option of Tensor Flow evaluator. Looks like we need to build a model in tensorflow and then import. Can we import the model in Strigo lab ? any detailed steps would help on this.

18 days ago

nidheeshFan

asked in Got a Question?

Write to sdc in groovy scripting

I am pretty new to Streamsets and Groovy and I am updating my “Dev Raw Data Source” to “Groovy Scripting” as my input. The issue which I face here is, I have the below code in my “Groovy Scripting” and it throws an error. Groovy Scripting:import com.streamsets.pipeline.stage.origin.scripting.*record = sdc.createRecord("bulkUpdateRequestTemplate")bulkUpdateRequestTemplate = "{ \"payload\": { \"objects\": { \"filter\": \"(equals(type,'configuration/entityTypes/Contract') and equals(attributes.ContractStatus, 'ACTIVE') and lt (attributes.ExpirationDate,'currentTimestampEpoch'))\", \"options\": \"searchByOv,ovOnly\" }, \"actions\": [ { \"operation\": \"UpdateAttribute\", \"operationParameters\": { \"attributeURI\": \"configuration/entityTypes/Contract/attributes/ContractStatus\", \"attributeValue\": \"INACTIVE\" } } ] } }"record.value = bulkUpdateRequestTemplatesdc.state['bulkUpdateRequestTemplate'] = record Error:com.streamsets.pipeline.api.StageException: SCRIPTING_10 - Script error in u

18 days ago

collidRoadie

asked in Got a Question?

StreamSets SDK - building fragments

I’ve been looking into building a fragment via the Python SDK, but I keep getting a key Error regarding the ‘pipelineFragmentConfig’.What’s the best way to be able to specify this config for the fragment i’m building?If i export the JSON having built a simple fragment in the UI, i can see these details, but I cant seem to figure out how to set them from the SDK.Thanks

18 days ago

BendjleeFan

asked in Got a Question?

Date Manipulation

What is the best practise approach to manipulating dates? Given/now = ${time:now()} I would like to modify “/now” to be -1/+1 Days and/or hours from the current timestamp. the only examples i could find of this are to first convert the date into milliseconds then followed by adding or subtracting the required number of Milliseconds to achieve the desired result (Looking for a better way to achieve this)

19 days ago

Page 1 / 19

Badge winners

samhas earned the badge Eager to help
saleempothiwalahas earned the badge Eager to help
Bikramhas earned the badge Community Champion 🏅
Bikramhas earned the badge Eager to help
Pradeephas earned the badge Eager to help

Show all badges

Terms & Conditions

Sign up

Already have an account? Login

Social Login

Username *

E-mail address *

What I do... *

Data Leader Data Architect Data Engineer Data Scientist Other

Company *

Country *

Zip Code *

Marketing Communications

Yes No

Password *

I have read and Agree to the Website Terms of Service and I have read and acknowledged the Privacy Policy.

loginBox.register.email_repeat

Login to the community

No account yet? Create an account

Social Login

Username or Email

Password

Remember me

Forgot password?

Enter your username or e-mail address. We'll send you an e-mail with instructions to reset your password.

Username or e-mail

Back to overview

Scanning file for viruses.

Sorry, we're still checking this file's contents to make sure it's safe to download. Please try again in a few minutes.

This file cannot be downloaded

Sorry, our virus scanner detected that this file isn't safe to download.