Got a Question?

Can't find what you're looking for? Ask it here!

  • 298 Topics
  • 651 Replies

298 Topics

jk56273
Fan
jk56273Fan
asked in Got a Question?

Truncate and load ADSL folder

Hi, I want to truncate and load ADSL folder using Streamsets pipeline. Can you suggest some way to achieve truncate and load feature in Streamsets

2
R
StreamSets Employee
23 days ago
Rishi
StreamSets Employee
RishiStreamSets Employee
posted in Got a Question?

What is pipeline Retry Exponential Backoff time ?

By default, when Data Collector encounters a stage-level error that might cause a standalone pipeline to fail, it retries the pipeline. That is, it waits a period of time, and then tries again to run the pipeline. A stage-level error might include a stage not being able to connect to an external system because the system or network is down. You can define the maximum number of pipeline retries that Data Collector attempts.The wait time between retries begins at 15 seconds and doubles in length until reaching a maximum of 5 minutes.

0
Rishi
StreamSets Employee
23 days ago
A
Fan
AnjaleeFan
asked in Got a Question?

JSON to CSV conversion for dynamic nested CSV

HiI am new to Streamsets and there are many more to learn. I am trying to convert Json to CSV having dynamic Json file. Sample Json:{    "order":    [        {            "order_id": "100001",            "customer_id": "1045",            "order_date": "20180701",            "ship_date": "20180701",            "item":                [                    {                     "product_id": "5001",                     "quantity": 1                    },                    {                     "product_id": "5021",                     "quantity": 3                    },                    {                     "product_id": "3331",                     "quantity": 4                    }                ]        },        {            "order_id": "100002",            "customer_id": "1088",            "order_date": "20180701",            "ship_date": "20180702",            "freight": "5.00"        }    ]} Expected Converted CSV file is:order.order_id,order.customer_id,order.order_date,order.s

0
A
Fan
23 days ago
Drew Kreiger
StreamSets Employee
Drew KreigerSenior Community Builder at StreamSets
asked in Got a Question?

What communities and meetup groups are you part of?

What other communities are you part of?  What meetup groups are you part of?  What resources do you find useful? Add them below within this thread

0
Drew Kreiger
StreamSets Employee
23 days ago
A
Discovered Fame
ana00159Discovered Fame
asked in Got a Question?

Join multiple api calls in streamsets

I have multiple API’s to be invoked. Can I use the multiple clients within same pipeline and integrate all the result set into 1 csv file or generate multiple csv files within the same pipeline? How can this be accomplished?

3
Bikram
Opening Band
23 days ago
A
Discovered Fame
ana00159Discovered Fame
asked in Got a Question?

Pagination issue with http client

I am trying to make a GET call using http client. Say I have 50 pages to be read, my requirement is to read all the pages (50 in this case) and send that to target in the initial run itself. But the pipeline reads only 1st page and comes to an end even without a pipeline finisher. I tried with batch mode and streaming mode, but no luck. Tried with pagination set in all different modes, didn't help. How can I read the entire pages available in a single batch run?

0
A
Discovered Fame
23 days ago
A
Discovered Fame
ana00159Discovered Fame
asked in Got a Question?

Docker configuration for streamsets installation

I am trying to create a deployment in na01.hub.streamsets.com. I have downloaded docker desktop to run the deployment script. Can you guide me how to proceed with the installation process?  

3
A
Discovered Fame
24 days ago
S
Fan
sunweiFan
asked in Got a Question?

how to update mongodb

I want to update the data to the existing data in mongodb according to "userid" and "createDate". What should I do? 

0
S
Fan
24 days ago
A
Discovered Fame
ana00159Discovered Fame
asked in Got a Question?

Convert nested JSON into csv

How can I convert a nested json into a csv file? Could you show with exact settings for the stages?

5
A
Discovered Fame
25 days ago
A
Discovered Fame
ana00159Discovered Fame
asked in Got a Question?

Invoke a get request in csv format

I want to send a GET request to a web url and get the result set in csv format which is the output format. I need to have the pagination set to read all the records at that point of time. My token will get expired every 30 minutes and its needs to be refreshed to get a new token. Any insight on how to implement this? Also Can I call multiple api’s in the same pipeline and merge all of them together?

0
A
Discovered Fame
25 days ago
S
Headliner
Srinivasan SankarHeadliner
asked in Got a Question?

REST API - Create Job - Enable Failover Option NOT Found

Hi there,I am using the following REST API,https://ap01.hub.streamsets.com/jobrunner/rest/v1/jobsI do not see an option to “Enable Failover” in the request body.How do I set “Enable Failover = true” when creating a Job using the above REST API? Cheers,Srini

2
S
Headliner
26 days ago
Yago Aparecido
Discovered Fame
Yago AparecidoDiscovered Fame
asked in Got a Question?

error when trying to create docker container from streamsets docker-compose

when we run the command "docker-compose up" it shows the error below: # docker-compose upBuilding with native build. Learn about native build in Compose here: https://docs.docker.com/go/compose-native-build/Creating sdc_2 ... doneAttaching to sdc_2sdc_2  | /docker-entrypoint.sh: line 22: /opt/streamsets-datacollector-3.22.3/bin/streamsets: No such file or directory contents of docker-compose.yml fileversion: '3'services:  sdc:    image: streamsets/datacollector:3.22.3    container_name: sdc_2    hostname: sdc002    restart: unless-stopped    network_mode: bridge    ports:      - 18630:18630/tcp    volumes:       - ./sdc-data:/data    environment:       SDC_CONF_HTTP_AUTHENTICATION: 'form' what could be causing this problem? the docker-entrypoint.sh file that is inside the sdc 3.22.3 image may be having a problem? when we create the sdc docker container manually with "docker run...", as below, the container works 100%: docker run -d -p 18630:18630 \-h sdc002 \--name sdc_2 \-e SDC_CONF_h

3
dima
StreamSets Employee
26 days ago
R
Fan
ramkumar_bvrFan
posted in Got a Question?

How to Pass Index value to Field Renamer Stage instead of source field name

I’m trying to automate our data restatement process. Based on the source field we are changing it to generic field as field1,field2,field3…..fieldN. But if we use Field Renamer stage it is mandatorily asking the exact source field name. Will it be possible to pass the index value instead of Source Field Name. Please let me know, if you need any further information on this. 

0
R
Fan
26 days ago
C
Fan
cmckenzieFan
asked in Got a Question?

OAuth 2 API to inContact

Hello, I attempting to use the HTTP client destination to send data to a could platform which uses OAuth2 for authentication. I’ve appended the headers that should be sent to the API to receive the token."Password" grant REQUESTRequest URL: https://api.incontact.com/InContactAuthorizationServer/Token Request Method: POSTRequest Headers: POST /InContactAuthorizationServer/Token HTTP/1.1Host: api.incontact.comConnection: keep-aliveContent-Length: 98Origin: http://myapp.example.com Authorization: basic U3VwZXJXaWRnZXRAV2lkZ2V0c0luYy5jb206MTIzNDU2 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.95 Safari/537.11 Content-Type: application/json; charset=UTF-8 Accept: application/json, text/javascript, */*; q=0.01 Referer: http://myapp.example.com/mainPage Accept-Encoding: gzip,deflate,sdch Accept-Language: en-US,en;q=0.8 Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3Request Payload: {   "grant_type" : "password",    "username

2
dima
StreamSets Employee
1 month ago
Yago Aparecido
Discovered Fame
Yago AparecidoDiscovered Fame
asked in Got a Question?

pipeline in StreamSets Data Collector 3.12.0 is not stopping normally, even clicking "force stop"

pipeline in StreamSets Data Collector 3.12.0 is not stopping normally, even clicking "force stop", we can only stop the pipeline if we restart the streamsets service why is StreamSets Data Collector 3.12.0 having this issue and how to fix it? PS: StreamSets Data Collector 3.12.0 is installed in Cloudera Express 6.3.

8
Yago Aparecido
Discovered Fame
1 month ago
Pradeep
StreamSets Employee
PradeepStreamSets Employee
asked in Got a Question?

How to fix error: USER doesn't have READ permission on the engine which appears in sdc.log

Symptoms of the issue described as below.When trying to access listed data collector(s) in page Administration > Data Collectors it redirects to the control hub login page and sdc.log shows the below.2022-05-23 10:29:08,574 [user:] [pipeline:] [runner:] [thread:webserver-48607] [stage:] ERROR AbstractSSOService - Exception while doing remote validation for token 'TOKEN:00a623cd-fe81-45...' component '-':  Could not connect to security service: com.streamsets.lib.security.http.ForbiddenException: ISSUES=code=DPM_02, message=User validation failed2022-05-23 10:29:09,387 [user:] [pipeline:] [runner:] [thread:webserver-48530] [stage:] WARN RemoteSSOService - Could not do User session validation, going inactive: com.streamsets.lib.security.http.ForbiddenException: ISSUES=code=DPM_04, message=USER doesn't have READ permission on the engine2022-05-23 10:29:09,387 [user:] [pipeline:] [runner:] [thread:webserver-48530] [stage:] ERROR AbstractSSOService - Exception while doing remote validati

1
Pradeep
StreamSets Employee
1 month ago
I
Fan
inalotoabFan
asked in Got a Question?

Linux commands for monitor/administer StreamSets

Is there a compiled note on commands that can be used to monitor/administer StreamSets from terminal? I am new here and want to get up to speed learning.

3
I
Fan
1 month ago
P
Fan
Priyanka MynepallyFan
asked in Got a Question?

Data-drifts

While working on datadrifts lab, and validating pipeline I got below error: Please help me with resolution steps Http failure response for https://eu01.hub.streamsets.com/tunneling/rest/2a5ca0cd-c8e1-4d7f-9fa8-939e294f1e69/rest/v1/pipeline/testRun__c26db2a6-aa95-494c-9732-bc0843dae308__c65d9b96-d66f-11ec-8f07-8576feff1283__a3fd417c-d66f-11ec-bf45-0fe868dac346%40c65d9b96-d66f-11ec-8f07-8576feff1283?autoGeneratePipelineId=false&draft=false&description=New%20Pipeline&TUNNELING_INSTANCE_ID=tunneling-3: 500 OK

1
R
StreamSets Employee
1 month ago
M
Fan
mstoffelFan
asked in Got a Question?

Error on kafka consumer

Hi, I try to combine a datacollector and a transformer pipeline (due to the missing parquet s3 support in datacollector). I read data from a Postgres db and publish it to Kafka using the datacollector. I have a Kafka connection that I want to use for both pipelines. Pushing Data to Kafka works but when I run the transfom pipeline (using the same connection) I get the following error: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in stage 0.0 (TID 0, 53f742dd118a, executor driver): java.lang.NoSuchMethodError: ‘void org.apache.spark.sql.kafka010.consumer.InternalKafkaConsumerPool$PoolConfig.setEvictionPolicy(org.apache.commons.pool2.impl.EvictionPolicy)' at org.apache.spark.sql.kafka010.consumer.InternalKafkaConsumerPool$PoolConfig.init(InternalKafkaConsumerPool.scala:191) at org.apache.spark.sql.kafka010.consumer.InternalKafkaConsumerPool$PoolConfig.<init>(InternalKafkaConsumerPool.scala:1

4
R
StreamSets Employee
1 month ago
P
Fan
PaawanFan
asked in Got a Question?

Need help with getting data from API and writing it in Parquet format.

Hi, I am trying to get some data from a SOAP API, I get the data in the from of an XML response. The requirement is to convert the data into Parquet format and store it on an ADLS GEN 2 storage.As far as I understand, I can use the data collector to write files in AVRO and then convert the AVRO to Parquet using a whole file transformer.I know that transformer can write in parquet directly, so is there any way for me to skip the intermediate AVRO file creation? 

1
R
StreamSets Employee
1 month ago
M
Fan
mstoffelFan
asked in Got a Question?

JDBC Query Consumer ignores query Interval

Hi Streamsets Community, just started out and tried the JDBC Query Consumer in DataCollector. My query executes correctly (offset etc.) but I is continuously executed regardless what I put into the query interval. I tried different values like ${5 * MINUTES} ${5 * HOURS} and also just plain integers like 10000. Any Idea what I’m doing wrong? Thanks Marco

3
M
Fan
1 month ago
J
Fan
jmanoFan
posted in Got a Question?

DC 5.0 load data from db2 to Oracle DB using multitable consumer example

Hellois it possible to use DC pipeline to move data from DB2 to Oracle DB using multitable consumner.? is there any example pipe line ?oris there any other way? thanksJames

0
J
Fan
1 month ago
S
Headliner
Srinivasan SankarHeadliner
asked in Got a Question?

HTTP Client Origin - Max Object Length Exceeded

Is there a way to increase the amount of data that a HTTP client can return in StreamSets? It seems to be limited (I have maxed out the Max Object Length field and still get this error). The call returns the data OK in postman - the source file is only a 1.9Mb Excel with 29k rows.WE ARE ON DATA COLLECTOR VERSION 5.0.0.

2
S
Headliner
1 month ago
S
Headliner
Srinivasan SankarHeadliner
asked in Got a Question?

StreamSets 5.0.0 Documentation - Discrepancy

Hi there, I was reading the below documentation,https://docs.streamsets.com/portal/platform-datacollector/5.0.x/datacollector/UserGuide/Processors/WaitJob.html#concept_xv5_xbd_zlband I could not find a processor stage named “Start Pipelines” in version 5.0.0. Note - I am on the latest version of DataOps Platform and I am building my Data Collector pipeline on the DataOps Platform / Control Hub GUI. Could you please verify and advise whether I need any additional stage libraries to be installed on my 5.0.0 engine for “Start Pipelines” processor stage? I am able to see other Orchestration stages as described in the below documentation,https://docs.streamsets.com/portal/platform-datacollector/5.0.x/datacollector/UserGuide/Orchestration_Pipelines/OrchestrationPipelines_Title.html#OrchestratorStages I can only assume this is a discrepancy in the documentation (refer first link), please advise otherwise. Cheers,Srini

2
S
Headliner
1 month ago
L
Fan
levanyeFan
asked in Got a Question?

running error pipeline couldn`t force stop

 Running Error pipeline couldn`t stop, when I click the button of force stop, nothing happened, what should i do?

6
S
Headliner
1 month ago

Badge winners

Show all badges
Terms & ConditionsCookie settings