StreamSets Platform

Question:When trying to apply the activation key for Control Hub On-Pre from the CLI, one might receive an error:ERROR: java.io.IOException: java.security.SignatureException: Signature length not correct: got 512 but was expecting 256 Answer:Control Hub is unable to read or reach the streamset-activation.jar file in the lib directory. Under the $DPM_HOME/apps/security/lib, check to see if there is both a 'streamsets-activation-<version>.jar' and a 'streamsets-activation-dev-<version>.jar'.  If there is a 'streamsets-activation-dev-<version>.jar', remove the dev jar, as it is meant only for the dev builds.  If there is no jar in the directory, a user may have deleted the file or was absent from the installation process.  To retrieve a new jar file, extract the jar file from a tarb

How can I schedule and trigger StreamSets jobs from Control + M scheduler? 

Hello alls, i has writen many connection string to Snowflake, but it’s fail all. Someone can give me some advices or samples.Thanks for reading.

I am using the SDC to make an API call which returns an XML file. My final requirement is to write the XML output as a parquet file on Azure datalake.I had thought of converting the XML into Avro and then using a whole file converter to convert the Avro into Parquet. The schema generator for the avro file creation is giving the following error.SCHEMA_GEN_0007 - Map '/XMLData' have different schemas for items. First schema: '"string"', Second schema: '{"type":"array","items":{"type":"map","values":{"type":"array","items":{"type":"map","values":{"type":"array","items":{"type":"map","values":"string"}}}}}}' How do I resolve this error?Attaching the XML preview below -  Also if there is a more efficient way to do all this, I’d really appreciate the input. 

Hi I am trying to execute the python SDK code but am facing the below error. It works fine without the pipeline builder line of code.Code:import argparseimport osfrom streamsets.sdk import ControlHub, DataCollectordef main(args):    # Connect to the Control Hub instance you want to interact with, and show that the ControlHub object was    # successfully created by displaying it's URL.    sch = ControlHub("xxxxx", username=args.uid, password=args.pwd)    print(sch.server_url)    # Instantiate the PipelineBuilder instance to build the pipeline, and add stages to the pipeline.    pipeline_builder = sch.get_pipeline_builder()    #dc = DataCollector(server_url="xxxxxx", control_hub=sch)    #builder = dc.get_pipeline_builder()if __name__ == '__main__':    os.environ['STREAMSETS_SDK_ACTIVATION_KEY'] = "xxxxxxx"    parser = argparse.ArgumentParser()    parser.add_argument("--uid

To reduce the serverless package size I am trying to exclude aws-sdk from being included with deployment.I was under the impression that the aws-sdk npm package was included by default in the aws services lambda runtime. When I make a new function this looks to be the case, I can console log the aws-sdk version just fine.When my larger lambda, uploaded with serverless, is executed it comes back with a an import error for not finding aws-sdk

Hi there, I have a requirement to read data from Snowflake DB and then use a specific column value as a filter condition for reading from a SQL Server DB. Has anyone dealt with such design in a data collector pipeline?I am stuck, since I can ONLY read from one source at a time using the JDBC Query Origin and I don’t think, I can pass values of one query as a parameter to another. Example:Get the max date of a column from a Snowflake table and use that date value as a filter condition for a SQL Server table. Thanks,Srini

Unable to execute a scheduled task on a running job. Getting the following error:Error trying to STOP xxxxxxxx Error: JOBRUNNER_00 Cannot change state for job xxxxxxx 

HADOOPFS_13: Error While writing to HDFS: Java.io.FilenotFoundException  while processing _TMP_Tablename_0 While extracting SQL server table with 10million rows. Pipeline is restarting again and again after extracting few records in ADLS with this error. It goes on for hours.  Using JDBC query consumer to connect to sql server and ADLS gen 2 as sink. please help to fix the issue.

I am running my SDC engine on a docker image installed on an Ubuntu WSL in Windows. I am using local fs in my jobs and generating few files. Where can I see these files?

Hi, I want to truncate and load ADSL folder using Streamsets pipeline. Can you suggest some way to achieve truncate and load feature in Streamsets

By default, when Data Collector encounters a stage-level error that might cause a standalone pipeline to fail, it retries the pipeline. That is, it waits a period of time, and then tries again to run the pipeline. A stage-level error might include a stage not being able to connect to an external system because the system or network is down. You can define the maximum number of pipeline retries that Data Collector attempts.The wait time between retries begins at 15 seconds and doubles in length until reaching a maximum of 5 minutes.

What other communities are you part of?  What meetup groups are you part of?  What resources do you find useful? Add them below within this thread

I have multiple API’s to be invoked. Can I use the multiple clients within same pipeline and integrate all the result set into 1 csv file or generate multiple csv files within the same pipeline? How can this be accomplished?

I am trying to make a GET call using http client. Say I have 50 pages to be read, my requirement is to read all the pages (50 in this case) and send that to target in the initial run itself. But the pipeline reads only 1st page and comes to an end even without a pipeline finisher. I tried with batch mode and streaming mode, but no luck. Tried with pagination set in all different modes, didn't help. How can I read the entire pages available in a single batch run?

I am trying to create a deployment in na01.hub.streamsets.com. I have downloaded docker desktop to run the deployment script. Can you guide me how to proceed with the installation process?  

I want to update the data to the existing data in mongodb according to "userid" and "createDate". What should I do? 

How can I convert a nested json into a csv file? Could you show with exact settings for the stages?

I want to send a GET request to a web url and get the result set in csv format which is the output format. I need to have the pagination set to read all the records at that point of time. My token will get expired every 30 minutes and its needs to be refreshed to get a new token. Any insight on how to implement this? Also Can I call multiple api’s in the same pipeline and merge all of them together?

Hi there,I am using the following REST API,https://ap01.hub.streamsets.com/jobrunner/rest/v1/jobsI do not see an option to “Enable Failover” in the request body.How do I set “Enable Failover = true” when creating a Job using the above REST API? Cheers,Srini

when we run the command "docker-compose up" it shows the error below: # docker-compose upBuilding with native build. Learn about native build in Compose here: https://docs.docker.com/go/compose-native-build/Creating sdc_2 ... doneAttaching to sdc_2sdc_2  | /docker-entrypoint.sh: line 22: /opt/streamsets-datacollector-3.22.3/bin/streamsets: No such file or directory contents of docker-compose.yml fileversion: '3'services:  sdc:    image: streamsets/datacollector:3.22.3    container_name: sdc_2    hostname: sdc002    restart: unless-stopped    network_mode: bridge    ports:      - 18630:18630/tcp    volumes:       - ./sdc-data:/data    environment:       SDC_CONF_HTTP_AUTHENTICATION: 'form' what could be causing this problem? the docker-entrypoint.sh file that is inside the sdc 3.22.3 image

Issue:The SDC is showing following error while start the process on sone of the java version however this will not stop the data collector process from booting. This can be seen while booting only.# WARNING: Unable to get Instrumentation. Dynamic Attach failed. You may add this JAR as -javaagent manually, or supply -Djdk.attach.allowAttachSelf# WARNING: Unable to attach Serviceability Agent. Unable to attach even with module exceptions: [org.openjdk.jol.vm.sa.SASupportException: Sense failed., org.openjdk.jol.vm.sa.SASupportException: Sense failed., org.openjdk.jol.vm.sa.SASupportException: Sense failed.] Solution:There are 2 ways to suppress the warning message:We can added the global variable in linux which will enable this property for all the java process running on linux machine, however the JAVA_TOOL_OPTIONS variable will be unset after the reboot. If you want

Hello, I attempting to use the HTTP client destination to send data to a could platform which uses OAuth2 for authentication. I’ve appended the headers that should be sent to the API to receive the token."Password" grant REQUESTRequest URL: https://api.incontact.com/InContactAuthorizationServer/Token Request Method: POSTRequest Headers: POST /InContactAuthorizationServer/Token HTTP/1.1Host: api.incontact.comConnection: keep-aliveContent-Length: 98Origin: http://myapp.example.com Authorization: basic U3VwZXJXaWRnZXRAV2lkZ2V0c0luYy5jb206MTIzNDU2 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.95 Safari/537.11 Content-Type: application/json; charset=UTF-8 Accept: application/json, text/javascript, */*; q=0.01

pipeline in StreamSets Data Collector 3.12.0 is not stopping normally, even clicking "force stop", we can only stop the pipeline if we restart the streamsets service why is StreamSets Data Collector 3.12.0 having this issue and how to fix it? PS: StreamSets Data Collector 3.12.0 is installed in Cloudera Express 6.3.

Symptoms of the issue described as below.When trying to access listed data collector(s) in page Administration > Data Collectors it redirects to the control hub login page and sdc.log shows the below.2022-05-23 10:29:08,574 [user:] [pipeline:] [runner:] [thread:webserver-48607] [stage:] ERROR AbstractSSOService - Exception while doing remote validation for token 'TOKEN:00a623cd-fe81-45...' component '-':  Could not connect to security service: com.streamsets.lib.security.http.ForbiddenException: ISSUES=code=DPM_02, message=User validation failed2022-05-23 10:29:09,387 [user:] [pipeline:] [runner:] [thread:webserver-48530] [stage:] WARN RemoteSSOService - Could not do User session validation, going inactive: com.streamsets.lib.security.http.ForbiddenException: ISSUES=code=DPM_04, message=USER doesn't have READ permission on the engine2022-05-23 10:29:09,387 [user:] [pipeline:] [runner:] [thread:webserver-48530] [stage:] ERROR Abstra