Skip to main content

Community Articles and Got a Question?

Can't find what you're looking for? Ask it here or check out the Community articles

  • 1,059 Topics
  • 2,327 Replies
1052 Topics
truder66.again
Fan
truder66.againFan
asked in Community Articles and Got a Question?
joining files through kafka

How to join 2 files using Kafka ?

905
saleempothiwala
Headliner
anthony.souza
Fan
anthony.souzaFan
asked in Community Articles and Got a Question?
Engine memory usage not being freed

 After running a few draft runs, pausing and running it again, the memory reaches 80% and I can't run anymore and I have to restart the Engine.How to make the memory free automatically since even with no more draft runs running the memory remains at 80%

661
Rishi
StreamSets Employee
D
Fan
dinesh
asked in Community Articles and Got a Question?
i am getting an error "cant get metadata for the object: salesforce_object" when trying to read salesforce object data

i am getting an error "cant get metadata for the object: salesforce_object" when trying to read salesforce object data

1611
Rishi
StreamSets Employee
A
Fan
AnjaleeFan
asked in Community Articles and Got a Question?
JSON to CSV conversion for dynamic nested CSV

HiI am new to Streamsets and there are many more to learn. I am trying to convert Json to CSV having dynamic Json file. Sample Json:{    "order":    [        {            "order_id": "100001",            "customer_id": "1045",            "order_date": "20180701",            "ship_date": "20180701",            "item":                [                    {                     "product_id": "5001",                     "quantity": 1                    },                    {                     "product_id": "5021",                     "quantity": 3                    },                    {                     "product_id": "3331",                     "quantity": 4                    }                ]        },        {            "order_id": "100002",            "customer_id": "1088",            "order_date": "20180701",            "ship_date": "20180702",            "freight": "5.00"        }    ]} Expected Converted CSV file is:order.order_id,order.customer_id,order.order_date,order.s

5363
A
Fan
alvachaitanya
Fan
alvachaitanyaFan
asked in Community Articles and Got a Question?
Creating a new version of SDC pipeline using SDK

I am trying to update the existing pipeline with a new version using SDK. Trying to replicate the version control feature to keep track of all changes. Is this option available in SDK ?

671
alvachaitanya
Fan
A
Fan
AshishFan
asked in Community Articles and Got a Question?
Job to pick only the changes since the last run

Hi I need your help here.I have a requirement to build a pipeline where I need to read data from one oracle DB and load the data into another oracle DB. Now for doing this everything I run my job, I should pick only those records from the source database which have been inserted or updated since the last run of my job. Please help me with your inputs. Regards,Ashish

673
Bikram
Headliner
B
Roadie
BharathiMaddelaRoadie
asked in Community Articles and Got a Question?
Oracle JDBC Origin - Transformer Pipeline

Hi, I am using Oracle JDBC Origin in Transformer and having an issue to save down the data frame into a Hive table. Below is the error screenshot of the same.However, data is saved onto a table when i run pipeline in preview mode. In preview mode, i can see all the data types are of string, decimal and date timestamp.I have tried reading only one string field and still can see same error message. Any help/suggesting would be appreciated.Transformer version is 3.13. 

706
B
Roadie
M
Discovered Fame
MelanieDiscovered Fame
asked in Community Articles and Got a Question?
missing one engine

HiIn the setup Deployment step, at the last step it says i should have 3 engines. However i only have 2 engines.Please note that at step 9 i did not get a “command not found” error.What is the 3rd engine i should get please? 

411
john.durkin
StreamSets Employee
A
Discovered Fame
ana00159Discovered Fame
asked in Community Articles and Got a Question?
Generate file footer

If I read from an api and generate a csv file, how can I add the number of records read as a footer in the csv file?

483
Bikram
Headliner
AkshayJadhav
StreamSets Employee
AkshayJadhavStreamSets Employee
published in Community Articles and Got a Question?
Reset password Control Hub on Cloud.

Question:How can I reset the Control Hub password if I forgot?Clicked on Forgot Password and it says to contact System Admin: Answer: Ask someone of  StreamSets who has authorization in order to reset the password, for example, Jerry Yu, and provide them the email address of the user is trying to access

511
Arthidevi Balavignesh
Fan
john.elmasry
Fan
john.elmasryFan
asked in Community Articles and Got a Question?
Error in Setting up streamset data collector,

Hello, I am installing Streamsets data collector using docker on my windows machine.the command I used is docker run --restart on-failure -p 18630:18630 -d --name sdc streamsets/datacollectorAnd the image is created successfully, but while browsing the localhost I found this page which is not mentioned anywhere in any toturialI don’t have an enterprise account so I tried to connect to contol hub, using data in my accountBut then I receive this errorDPM Login failed, status code '500': {"EXCEPTION":{"rawMessage":"java.lang.IllegalArgumentException: Invalid/unsupported hash version 'UNKNOWN'","className":"java.lang.IllegalArgumentException","message":"Invalid/unsupported hash version 'UNKNOWN'","stackTrace":"java.lang.IllegalArgumentException: Invalid/unsupported hash version 'UNKNOWN'\n\tat com.streamsets.lib.security.http.PasswordHasher.getValueToHash(PasswordHasher.java:131)\n\tat com.streamsets.lib.security.http.PasswordHasher.verify(PasswordHasher.java:167)\n\tat com.streamsets.apps

3162
Jesse
StreamSets Employee
M
Fan
MaheshFan
asked in Community Articles and Got a Question?
Origin related question

i have to get data from a server and i can only get 100 tables at a time from it. I have to run the pipeline again if there are more than 100 tables in it. how can i check whether the server is empty or has data left in it? how to automate the checking process 

523
Bikram
Headliner
V
Fan
vicky20Fan
asked in Community Articles and Got a Question?
Check a key field exists in JSON

How can I check if a key field exists in nested JSON? For example I have a json like this,    "Ticket": {        "Price": {            "Adult": "18",            "Child": [{                "FromAge": "0",                "ToAge": "12",                "Price": "10"            }]        }    }}From source I might get the json as follows as well without one among the key fields (child missing).    "Ticket": {        "Price": {            "Adult": "18"                    }    }}In this case how can check if key field exists and if not I want to introduce "Child": [{                "FromAge": "0",                "ToAge": "12",                "Price": "10"            }] to the json to standardise the structure. How can I achieve this in StreamSets? If I couldn’t standardize, at least I want to identify the records with missing key in the json. Thanks!  

5746
Bikram
Headliner
B
Roadie
BharathiMaddelaRoadie
asked in Community Articles and Got a Question?
Authorising transformer update in Transformer Jobs in Control Hub

Hi,is it possible to change the Authorising Transformer of  a job on control hub?we have a scenario to change another Authorising Transformer  in all the Transformer Jobs, when we have changed it  in a underneath pipeline and chose to update the Job while doing check-in, it does not reflect on the job.

333
Rishi
StreamSets Employee
nikkichow
Fan
nikkichowFan
asked in Community Articles and Got a Question?
Can we write the records into/as a text file in Streamsets?

I have an usecase where we need to read from the source and write all the records into a text file using streamsets. Is it possible to write all the records as a text file?

764
Bikram
Headliner
nikkichow
Fan
nikkichowFan
asked in Community Articles and Got a Question?
How to copy a file from one folder to other folder in NAS(sftp) location

I’m trying to copy a file which is in one folder to another folder in the NAS location . I tried using shell script executor but I see the below error . Might be issue with my shell script . Need help for any alternate approach or a shell script which executes successfully to copy the file from one folder to another in the NAS location.

921
Pradeep
StreamSets Employee
M
Fan
michal.kertesFan
asked in Community Articles and Got a Question?
Streamsets Data Collector 3.16 | java.lang.NoClassDefFoundError after MapR upgrade

Hello, We performed mapr-setup as per documentation, but unfortunatelly we are facing with pretty limited functionality of sdc and following entries in sdc.log: #####4-9963-6352a0fc023c-tsystems_maprfs_test] [stage:] ERROR ProductionPipelineRunnable - An exception occurred while running the pipeline, com.streamsets.datacollector.runner.PipelineRuntimeException: CONTAINER_0702 - Pipeline initialization error: java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.security.UserGroupInformationcom.streamsets.datacollector.runner.PipelineRuntimeException: CONTAINER_0702 - Pipeline initialization error: java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.security.UserGroupInformation        at com.streamsets.datacollector.execution.runner.common.ProductionPipeline.run(ProductionPipeline.java:106)        at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunnable.run(ProductionPipelineRunnable.java:75)        at com.stream

810
M
Fan
M
Fan
michal.kertesFan
asked in Community Articles and Got a Question?
Streamsets Data Collector 3.16 | MapR 6.1.1 compatibility

Hello, I comming with question about SDC 3.16 compatibility.We can see in support matrix for MapR environment the latest supported version for MapR core 6.1.0. Since support matrix is from 2019, when MapR 6.1.1 wasn’t even released, I’d like to ask whether version of SDC 3.16.is compatibile with  MapR 6.1.1  (and MEP 6.3.6)As we can see in documentation for MapR setup ( https://docs.streamsets.com/platform-datacollector/latest/datacollector/UserGuide/Installation/MapR-Prerequisites.html,) we can run Set Up command to setup configuration files and create requeired symbolic links. Is it possible to perform these steps even if version of MapR is 6.1.1?  ThanksMichal K

742
M
Fan
A
Discovered Fame
ana00159Discovered Fame
asked in Community Articles and Got a Question?
Convert a JSON string to JSON

I am reading from Kafka in text format (as JSON format not supported) and trying to convert the JSON string into an actual JSON for flattening and pivoting. My sample data looks like below.{"text":"\d0000\d0000\d0000\d00007{\"Employee\":{\"Address\":[{\"Addressid\":\"3897ef\"}],\"Role\":{\"Designation\":{\"name\":\"Manager\"}}} I want to convert as follows{    "Employee": {        "Address": [            {                "Addressid": "3897ef"            }        ],        "Role": {            "Designation": {                "name": "Manager"            }        },} How can I covert this into a list map JSON hierarchy as above?

1221
Bikram
Headliner
MohammedSohail
Roadie
MohammedSohailRoadie
asked in Community Articles and Got a Question?
Few questions regarding the Data Collector

As I am new to streamsets, few doubts , can anyone help me here  Q1:  How to call a pipeline from another pipeline?Q2: How to store data in streamset like Local Variable (I want to use later part of pipeline)?@Rishi 

1841
Bikram
Headliner
rajneesh75
Fan
rajneesh75Fan
asked in Community Articles and Got a Question?
How to open and read a json file using Javascript scripting stage

I need to open few json files in side streamsets datacollector pipeline to do some custom processing. Datacollector doesn't allow multiple origins, so I am looking at Javascript scripting stage to open multiple file objects.  Also directory origin is opening and displaying only one .json file, so that's too is not working. Any ideas. Thanks

1585
Bikram
Headliner
P
Opening Band
pkandraOpening Band
asked in Community Articles and Got a Question?
Configuring AWS S3 Bucket

I’m trying to write some simple output from a database to an Amazon S3 bucket.  I have the access key and shared secret setup properly, but what values do I need to use for bucket and path?  The documentation isn’t too clear and the rows seem to all go to Stage Error on a pipeline run.

1596
Bikram
Headliner
MohammedSohail
Roadie
MohammedSohailRoadie
asked in Community Articles and Got a Question?
Need Help in Reading data from SFTP location and loading the same data Another system VIA REST call

Need Help in Reading data from SFTP location and loading the same data into another system VIA REST call (HTTP)  Does someone help me with sample pipeline? 

1594
Rishi
StreamSets Employee
S
Headliner
Srinivasan SankarHeadliner
asked in Community Articles and Got a Question?
Error seen while executing a Job Instance in DataOps Platform

Hi there,  Any help on the below error, that I see in the Job Instances section in DataOps Platform,SAVE_PIPELINETIMEDOUTUnexpected error. Ack tracker of type 'SAVE_PIPELINE' timed out after '600000' ms. Data Collectors is present in the database and it's responding. Data Collectors last reported time is 2022-06-17T05:00:54.535Z (epoch millis 1655442054535)START_PIPELINETIMEDOUTUnexpected error. Ack tracker of type 'START_PIPELINE' timed out after '600000' ms. Data Collectors is present in the database and it's responding. Data Collectors last reported time is 2022-06-17T04:59:54.531Z (epoch millis 1655441994531) The errors disappear quickly and the job instance is in ACTIVE (Green) state again but I am trying to understand the reason behind the error.  FYI - I ran a job instance that triggers 15 other job instances. The parent job instance is running. I see this error in the child job instances that are triggered by the parent job instance. But as mentioned earlier, the error disappea

591
Rishi
StreamSets Employee
MohammedSohail
Roadie
MohammedSohailRoadie
asked in Community Articles and Got a Question?
Need help in removing special character

[{"domain":"{\n\t\t\"id\": \"7330-2ade99\"\n\t\t\n}"}]  How to remove “\n\t\t”

1776
Rishi
StreamSets Employee

Badge winners

Show all badges
Terms & Conditions