Platform Q&A

Get inspired and gain all the knowledge you need

320 Topics

nikkichow
Fan
nikkichowFan
asked in Got a Question?

Can we write the records into/as a text file in Streamsets?

I have an usecase where we need to read from the source and write all the records into a text file using streamsets. Is it possible to write all the records as a text file?

3
dima
StreamSets Employee
1 day ago
S
Headliner
Srinivasan SankarHeadliner
asked in Got a Question?

Looping in Data Collector pipelines

Hi there, Any design suggestions to build a loop in StreamSets Data Collector pipelines ?  Example:I have two API endpoints, the first one takes in a request to perform an activity and sends me an ID for the request placed. As a second step, I need to poll a different endpoint with the ID obtained from the previous step to check the status of my earlier request (i.e. placed in the first step). Ideally, I would like to loop that second endpoint in my StreamSets pipeline until I get the status as “succeeded” or “error”. Any suggestions on how I achieve this in a data collector pipeline? Cheers,Srini

0
S
Headliner
2 days ago
john.elmasry
Fan
john.elmasryFan
asked in Got a Question?

Error in Setting up streamset data collector,

Hello, I am installing Streamsets data collector using docker on my windows machine.the command I used is docker run --restart on-failure -p 18630:18630 -d --name sdc streamsets/datacollectorAnd the image is created successfully, but while browsing the localhost I found this page which is not mentioned anywhere in any toturialI don’t have an enterprise account so I tried to connect to contol hub, using data in my accountBut then I receive this errorDPM Login failed, status code '500': {"EXCEPTION":{"rawMessage":"java.lang.IllegalArgumentException: Invalid/unsupported hash version 'UNKNOWN'","className":"java.lang.IllegalArgumentException","message":"Invalid/unsupported hash version 'UNKNOWN'","stackTrace":"java.lang.IllegalArgumentException: Invalid/unsupported hash version 'UNKNOWN'\n\tat com.streamsets.lib.security.http.PasswordHasher.getValueToHash(PasswordHasher.java:131)\n\tat com.streamsets.lib.security.http.PasswordHasher.verify(PasswordHasher.java:167)\n\tat com.streamsets.apps

1
dima
StreamSets Employee
4 days ago
nikkichow
Fan
nikkichowFan
asked in Got a Question?

How to copy a file from one folder to other folder in NAS(sftp) location

I’m trying to copy a file which is in one folder to another folder in the NAS location . I tried using shell script executor but I see the below error . Might be issue with my shell script . Need help for any alternate approach or a shell script which executes successfully to copy the file from one folder to another in the NAS location.

1
Pradeep
StreamSets Employee
4 days ago
M
Fan
michal.kertesFan
asked in Got a Question?

Streamsets Data Collector 3.16 | java.lang.NoClassDefFoundError after MapR upgrade

Hello, We performed mapr-setup as per documentation, but unfortunatelly we are facing with pretty limited functionality of sdc and following entries in sdc.log: #####4-9963-6352a0fc023c-tsystems_maprfs_test] [stage:] ERROR ProductionPipelineRunnable - An exception occurred while running the pipeline, com.streamsets.datacollector.runner.PipelineRuntimeException: CONTAINER_0702 - Pipeline initialization error: java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.security.UserGroupInformationcom.streamsets.datacollector.runner.PipelineRuntimeException: CONTAINER_0702 - Pipeline initialization error: java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.security.UserGroupInformation        at com.streamsets.datacollector.execution.runner.common.ProductionPipeline.run(ProductionPipeline.java:106)        at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunnable.run(ProductionPipelineRunnable.java:75)        at com.stream

0
M
Fan
5 days ago
M
Fan
michal.kertesFan
asked in Got a Question?

Streamsets Data Collector 3.16 | MapR 6.1.1 compatibility

Hello, I comming with question about SDC 3.16 compatibility.We can see in support matrix for MapR environment the latest supported version for MapR core 6.1.0. Since support matrix is from 2019, when MapR 6.1.1 wasn’t even released, I’d like to ask whether version of SDC 3.16.is compatibile with  MapR 6.1.1  (and MEP 6.3.6)As we can see in documentation for MapR setup ( https://docs.streamsets.com/platform-datacollector/latest/datacollector/UserGuide/Installation/MapR-Prerequisites.html,) we can run Set Up command to setup configuration files and create requeired symbolic links. Is it possible to perform these steps even if version of MapR is 6.1.1?  ThanksMichal K

2
M
Fan
5 days ago
A
Discovered Fame
ana00159Discovered Fame
asked in Got a Question?

Convert a JSON string to JSON

I am reading from Kafka in text format (as JSON format not supported) and trying to convert the JSON string into an actual JSON for flattening and pivoting. My sample data looks like below.{"text":"\d0000\d0000\d0000\d00007{\"Employee\":{\"Address\":[{\"Addressid\":\"3897ef\"}],\"Role\":{\"Designation\":{\"name\":\"Manager\"}}} I want to convert as follows{    "Employee": {        "Address": [            {                "Addressid": "3897ef"            }        ],        "Role": {            "Designation": {                "name": "Manager"            }        },} How can I covert this into a list map JSON hierarchy as above?

1
Bikram
Opening Band
5 days ago
MohammedSohail
Roadie
MohammedSohailRoadie
asked in Got a Question?

Few questions regarding the Data Collector

As I am new to streamsets, few doubts , can anyone help me here  Q1:  How to call a pipeline from another pipeline?Q2: How to store data in streamset like Local Variable (I want to use later part of pipeline)?@Rishi 

1
Bikram
Opening Band
5 days ago
rajneesh75
Fan
rajneesh75Fan
asked in Got a Question?

How to open and read a json file using Javascript scripting stage

I need to open few json files in side streamsets datacollector pipeline to do some custom processing. Datacollector doesn't allow multiple origins, so I am looking at Javascript scripting stage to open multiple file objects.  Also directory origin is opening and displaying only one .json file, so that's too is not working. Any ideas. Thanks

5
Bikram
Opening Band
6 days ago
P
Fan
pkandraFan
asked in Got a Question?

Configuring AWS S3 Bucket

I’m trying to write some simple output from a database to an Amazon S3 bucket.  I have the access key and shared secret setup properly, but what values do I need to use for bucket and path?  The documentation isn’t too clear and the rows seem to all go to Stage Error on a pipeline run.

6
Bikram
Opening Band
6 days ago
MohammedSohail
Roadie
MohammedSohailRoadie
asked in Got a Question?

Need Help in Reading data from SFTP location and loading the same data Another system VIA REST call

Need Help in Reading data from SFTP location and loading the same data into another system VIA REST call (HTTP)  Does someone help me with sample pipeline? 

4
Rishi
StreamSets Employee
9 days ago
S
Headliner
Srinivasan SankarHeadliner
asked in Got a Question?

Error seen while executing a Job Instance in DataOps Platform

Hi there,  Any help on the below error, that I see in the Job Instances section in DataOps Platform,SAVE_PIPELINETIMEDOUTUnexpected error. Ack tracker of type 'SAVE_PIPELINE' timed out after '600000' ms. Data Collectors is present in the database and it's responding. Data Collectors last reported time is 2022-06-17T05:00:54.535Z (epoch millis 1655442054535)START_PIPELINETIMEDOUTUnexpected error. Ack tracker of type 'START_PIPELINE' timed out after '600000' ms. Data Collectors is present in the database and it's responding. Data Collectors last reported time is 2022-06-17T04:59:54.531Z (epoch millis 1655441994531) The errors disappear quickly and the job instance is in ACTIVE (Green) state again but I am trying to understand the reason behind the error.  FYI - I ran a job instance that triggers 15 other job instances. The parent job instance is running. I see this error in the child job instances that are triggered by the parent job instance. But as mentioned earlier, the error disappea

1
Rishi
StreamSets Employee
9 days ago
MohammedSohail
Roadie
MohammedSohailRoadie
asked in Got a Question?

Need help in removing special character

[{"domain":"{\n\t\t\"id\": \"7330-2ade99\"\n\t\t\n}"}]  How to remove “\n\t\t”

6
Rishi
StreamSets Employee
10 days ago
S
Headliner
Srinivasan SankarHeadliner
asked in Got a Question?

Credential Store Configuration

Hi there, I’ve set the following property to TRUE. # Requires a group secret for each secretcredentialStore.aws.config.enforceEntryGroup=true When I configure the groups for the secret name using the convention,ch/plant/sourceDb-groups=mysensitive-group I get an error message with the code CSP_075 as below,Advanced configuration credential-stores.properties has an error : ch/plant/sourceDb-groups=mysensitive-group : CSP_075 Looks like the config, does not like / in the secret name. AWS allows / in secret names. Please advise how to get around this issue?

1
R
StreamSets Employee
10 days ago
A
Fan
Anusha SunkavalliFan
posted in Got a Question?

Is it possible to read data from SOAPUI using http client

As part of my project requirement I have to read data from SOAPUI and I tried implementing it using http client but not able to fetch data. Steps Followed.Http client is source component WSDL details are provided in reource url path HTTP method is POST SOAP request is provided in Request body. 5.Soap Action details are provided header tab. Error message is "HTTP_01 - Error fetching resource. HTTP-Status: 415 Reason: Unsupported Media Type :"

1
wilsonshamim
StreamSets Employee
11 days ago
AkshayJadhav
StreamSets Employee
AkshayJadhavStreamSets Employee
published in Share your Best Practices

STF - Configuration of maximum open file limit is too low: 1024 (expected at least 32768).

Issue:When using the StreamSets Test Framework and trying to run a test it fails with the following error:$ stf test --sdc-version 3.4.0 -s stage/test_dev_stages.py 2019-01-28 09:03:16 AM [INFO] [streamsets.testframework.cli] Pulling Docker image streamsets/testframework:master ... INFO:streamsets.testframework.cli:Pulling Docker image streamsets/testframework:master ... ============================= test session starts ============================== platform linux -- Python 3.6.8, pytest-3.0.4, py-1.7.0, pluggy-0.4.0 benchmark: 3.1.1 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000) rootdir: /root/tests, inifile: plugins: benchmark-3.1.1 collected 7 itemsstage/test_dev_stages.py 2019-01-28 09:03:27 AM [INFO] [streamsets.testframework.sdc] Starting StreamSets Data Collector 3.4.0 container ... 2019-01-28 09:05:41 AM [ERROR] [streamsets.testframework.sdc] Error reaching Data Col

60
AkshayJadhav
StreamSets Employee
13 days ago
AkshayJadhav
StreamSets Employee
AkshayJadhavStreamSets Employee
posted in Share your Best Practices

Unable to activate Control Hub with “ERROR: java.io.IOException: java.security.SignatureException: Signature length not correct: got 512 but was expecting 256”

Question:When trying to apply the activation key for Control Hub On-Pre from the CLI, one might receive an error:ERROR: java.io.IOException: java.security.SignatureException: Signature length not correct: got 512 but was expecting 256 Answer:Control Hub is unable to read or reach the streamset-activation.jar file in the lib directory. Under the $DPM_HOME/apps/security/lib, check to see if there is both a 'streamsets-activation-<version>.jar' and a 'streamsets-activation-dev-<version>.jar'.  If there is a 'streamsets-activation-dev-<version>.jar', remove the dev jar, as it is meant only for the dev builds.  If there is no jar in the directory, a user may have deleted the file or was absent from the installation process.  To retrieve a new jar file, extract the jar file from a tarball of the Control Hub On-Prem version that one is using.

0
AkshayJadhav
StreamSets Employee
13 days ago
A
Discovered Fame
ana00159Discovered Fame
asked in Got a Question?

Invoke StreamSets jobs from Control M

How can I schedule and trigger StreamSets jobs from Control + M scheduler? 

5
T
StreamSets Employee
16 days ago
jufi
Fan
jufiFan
asked in Got a Question?

ConnectionString to Snowflake

Hello alls, i has writen many connection string to Snowflake, but it’s fail all. Someone can give me some advices or samples.Thanks for reading.

2
Pradeep
StreamSets Employee
17 days ago
V
Fan
vicky20Fan
asked in Got a Question?

Check a key field exists in JSON

How can I check if a key field exists in nested JSON? For example I have a json like this,    "Ticket": {        "Price": {            "Adult": "18",            "Child": [{                "FromAge": "0",                "ToAge": "12",                "Price": "10"            }]        }    }}From source I might get the json as follows as well without one among the key fields (child missing).    "Ticket": {        "Price": {            "Adult": "18"                    }    }}In this case how can check if key field exists and if not I want to introduce "Child": [{                "FromAge": "0",                "ToAge": "12",                "Price": "10"            }] to the json to standardise the structure. How can I achieve this in StreamSets? If I couldn’t standardize, at least I want to identify the records with missing key in the json. Thanks!  

5
S
Headliner
17 days ago
P
Fan
PaawanFan
asked in Got a Question?

How to resolve this error SCHEMA_GEN_0007 - Map '/XMLData' have different schemas for items.

I am using the SDC to make an API call which returns an XML file. My final requirement is to write the XML output as a parquet file on Azure datalake.I had thought of converting the XML into Avro and then using a whole file converter to convert the Avro into Parquet. The schema generator for the avro file creation is giving the following error.SCHEMA_GEN_0007 - Map '/XMLData' have different schemas for items. First schema: '"string"', Second schema: '{"type":"array","items":{"type":"map","values":{"type":"array","items":{"type":"map","values":{"type":"array","items":{"type":"map","values":"string"}}}}}}' How do I resolve this error?Attaching the XML preview below -  Also if there is a more efficient way to do all this, I’d really appreciate the input. 

4
P
Fan
18 days ago
D
Fan
DimpleFan
asked in Got a Question?

Streamsets SDK error

Hi I am trying to execute the python SDK code but am facing the below error. It works fine without the pipeline builder line of code.Code:import argparseimport osfrom streamsets.sdk import ControlHub, DataCollectordef main(args):    # Connect to the Control Hub instance you want to interact with, and show that the ControlHub object was    # successfully created by displaying it's URL.    sch = ControlHub("xxxxx", username=args.uid, password=args.pwd)    print(sch.server_url)    # Instantiate the PipelineBuilder instance to build the pipeline, and add stages to the pipeline.    pipeline_builder = sch.get_pipeline_builder()    #dc = DataCollector(server_url="xxxxxx", control_hub=sch)    #builder = dc.get_pipeline_builder()if __name__ == '__main__':    os.environ['STREAMSETS_SDK_ACTIVATION_KEY'] = "xxxxxxx"    parser = argparse.ArgumentParser()    parser.add_argument("--uid", type=str, required=True, help='Type the Streamsets username')    parser.add_argument("--pwd", type=str, required=T

2
D
Fan
18 days ago
S
Headliner
Srinivasan SankarHeadliner
asked in Got a Question?

Pass the value from one JDBC query to another

Hi there, I have a requirement to read data from Snowflake DB and then use a specific column value as a filter condition for reading from a SQL Server DB. Has anyone dealt with such design in a data collector pipeline?I am stuck, since I can ONLY read from one source at a time using the JDBC Query Origin and I don’t think, I can pass values of one query as a parameter to another. Example:Get the max date of a column from a Snowflake table and use that date value as a filter condition for a SQL Server table. Thanks,Srini

5
Rishi
StreamSets Employee
18 days ago
F
Fan
feras.jumaFan
asked in Got a Question?

Scheduled task failed - JOBRUNNER Cannot change state for job

Unable to execute a scheduled task on a running job. Getting the following error:Error trying to STOP xxxxxxxx Error: JOBRUNNER_00 Cannot change state for job xxxxxxx 

2
F
Fan
19 days ago
S
Fan
sanchitFan
asked in Got a Question?

HADOOPFS_13: Error While writing to HDFS: Java.io.FilenotFoundException while processing _TMP_Tablename_0

HADOOPFS_13: Error While writing to HDFS: Java.io.FilenotFoundException  while processing _TMP_Tablename_0 While extracting SQL server table with 10million rows. Pipeline is restarting again and again after extracting few records in ADLS with this error. It goes on for hours.  Using JDBC query consumer to connect to sql server and ADLS gen 2 as sink. please help to fix the issue.

2
S
Fan
21 days ago

Badge winners

Show all badges
Terms & ConditionsCookie settings