Got a Question?

Can't find what you're looking for? Ask it here!

  • 684 Topics
  • 1,769 Replies

684 Topics

R
Roadie
Rachit_2022Roadie
asked in Got a Question?

Referring a variable in Spark SQL Stage

Use Case - We have to pass dynamic date in Spark SQL Query stage in Streamsets,We have got an output as a dynamic date using pyspark code, now we need to use output in Spark SQL stages within SQL scripts. How to refer this date output variable in SQL scripts.Any help and advise would be great.

3
Bikram
Headliner
22 days ago
S
Discovered Fame
saikat_chatterjeeDiscovered Fame
asked in Got a Question?

pip3 install

I am trying to run the following command:“pip3 install streamsets~=5.0” in my VM to activate the sdk for python without using the activation key as shown in the documentation but unfortunately I am receiving the following error: WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])"))': /simple/streamsets/WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])"))': /simple/streamsets/WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])"))': /

9
K
Fan
23 days ago
M
Fan
Mihir KaleFan
asked in Got a Question?

Error while creating sdc instance in Python SDK

sch = ControlHub(API_CREDENTIAL_ID, API_TOKEN)sdc=DataCollector(server_url='http://023aee033abf:18630',control_hub=sch)sdc instance creation is taking lot of time to execute, when I force stop run, it shows the following err,-------------------------------------------------------------------------------------Traceback (most recent call last):  File "C:\Users\mihir_kale\PycharmProjects\Informatica_to_SS\sdc_instance.py", line 15, in <module>    sdc=DataCollector(server_url='http://023aee033abf:18630',control_hub=sch)        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  File "C:\Users\mihir_kale\PycharmProjects\Informatica_to_SS\venv\Lib\site-packages\streamsets\sdk\sdc.py", line 90, in __init__    self.api_client = sdc_api.ApiClient(server_url=current_server_url,                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  File "C:\Users\mihir_kale\PycharmProjects\Informatica_to_SS\venv\Lib\site-packages\streamsets\sdk\sdc_api.py", line 97, in __

9
Bikram
Headliner
23 days ago
K
Fan
kushal_upadhyayFan
asked in Got a Question?

Unable to fetch data using Kafka consumer random group

Unable to fetch data from kafka when trying kafka consumer to Local FS and getting follwoing error :iorg2.org] [runner:] [thread:ProductionPipelineRunnable-KAFKATOL__494a67a7-94ef-49e1-a41b-385c79551625__apiorg2.org-KAFKA_TO_LOCALFS] [stage:] INFO  AbstractRunner - Scheduling retry in '239996' milliseconds2023-05-11 23:21:31,730 [user:*admin@apiorg2.org] [pipeline:KAFKA_TO_LOCALFS/KAFKATOL__494a67a7-94ef-49e1-a41b-385c79551625__apiorg2.org] [runner:] [thread:ProductionPipelineRunnable-KAFKATOL__494a67a7-94ef-49e1-a41b-385c79551625__apiorg2.org-KAFKA_TO_LOCALFS] [stage:] ERROR ProductionPipelineRunnable - An exception occurred while running the pipeline, com.streamsets.datacollector.runner.PipelineRuntimeException: CONTAINER_0800 - Can't start pipeline due 1 validation error(s). First one: KAFKA_41 - Could not get partition count for topic 'tst2' : com.streamsets.pipeline.api.StageException: KAFKA_41 - Could not get partition count for topic 'tst2' : org.apache.kafka.common.errors.Timeo

1
AkshayJadhav
StreamSets Employee
23 days ago
P
Discovered Fame
Priyanka MynepallyDiscovered Fame
asked in Got a Question?

curl command in shell

Hi Team  I need to post file1.json and file2.csv using curl.I tried this but no luckPlease help me  metadata=@/tmp/file1.json \files=@/tmp/file2.csv \curl -kiv --trace -w --location --request POST "<Location>" \--header "accept:application/json,application/hal+json, application/json, application/*+json, text/plain, application/octet-stream, application/xml, text/xml, application/x-www-form-urlencoded, application/*+xml, multipart/form-data, multipart/mixed, */*" \--header "Accept-Charset: UTF-8" \--header "Content-Type:application/json" \--form  metadata="$metadata" \--header "Accept-Charset: UTF-8" \--header "Content-Type:multipart/form-data,text/csv" \--form  files="$files"

4
Bikram
Headliner
24 days ago
D
Fan
dhanraj_shindeFan
asked in Got a Question?

Edit an existing pipeline using python SDK

Hi TeamCan we edit a pipeline using python SDK. I tried creating new pipeline it worked. i tried to retrieving pipelines from control hub it worked!,but it only contain metadata of a pipeline like (name ,id commit id ,labels , version, ... etc ),I did not see any property related to stages and it’s configuration.  if it is possible, could you please share a code snippet of same.     

0
D
Fan
25 days ago
S
Fan
shivrajFan
asked in Got a Question?

How to get stage configuration data in python SDk?

I have a pipeline object which was imported using pipeline Id. I need it’s stage configuration data.I have imported a pipeline but I wasn’t able find destination.How can I get a destination where file is getting stored after running pipeline. I want that path in runtime on python SDK.Please help me put with this concern asap.

0
S
Fan
25 days ago
Saurabh Suman
Fan
Saurabh SumanFan
asked in Got a Question?

File generation on local

how can i generate a file from any source to local (on my computer) or a given unix server. 

1
R
StreamSets Employee
25 days ago
lakshmi_narayanan_t
Discovered Fame
lakshmi_narayanan_tDiscovered Fame
asked in Got a Question?

cdc operation without use of mongodb oplog origin

If it is possible to achieve CDC (Change Data Capture) operations in MongoDB Atlas without using MongoDB Oplog, and only by utilizing MongoDB Atlas or other alternative methods, please provide the correct information or alternatives ,I am encountering an error when using MongoDB Oplog in my CDC operation in MongoDB Atlas. Although I have created a capped collection, the error still persists. Therefore, I am looking for alternative methods or processors to achieve CDC operations in MongoDB Atlas without relying on MongoDB Oplog.

0
lakshmi_narayanan_t
Discovered Fame
27 days ago
K
Fan
kushal_upadhyayFan
asked in Got a Question?

Noclassfound error while connecting to some customer connector (similar like AWS)

While Creating connection with one of the customized cloud connector in Control HUB (same like AWS), we are facing issues like java.lang.NoClassDefFoundError: Could not initialize class com.amazonaws.auth.DefaultAWSCredentialsProviderChainThey have created some custom libraries and put it into the external folder, it is working fine in some other envrionment.. we have checked the java version was different which is 11_0_19 at our end but it was 1_8_ on other envrionment but even after installing 1.8 the error didn’t go away. Please help 

1
AkshayJadhav
StreamSets Employee
30 days ago
M
Fan
MadhumithaFan
asked in Got a Question?

Updating the excel spreadsheet directly from streamsets

I generally run the streamsets pipeline after which the couchbase gets updated and I export the JSON file, convert it and update my google spreadsheet which I will be using further for visualizations. My question here is, Is there any chances I can update the spreadsheet directly through Streamsets, so as to avoid the export of JSON and convert it to excel and update the sheet.  

1
Bikram
Headliner
30 days ago
ychundawat
Fan
ychundawatFan
asked in Got a Question?

All JDBC executor steps executing at the end

I have a pipeline with multiple JDBC query executors and I’m looking for execution in the order it is mentioned in the pipeline however all executors are executing at the end. I have tried batch commit - enable/disable but still facing the same issue, Any clue?  

2
Bikram
Headliner
1 month ago
M
Fan
Mihir KaleFan
asked in Got a Question?

Unexpected output when validating multiple pipelines simultaneously

Using SDK, we are trying to bulk validate multiple pipelines in a multi threading approach on the same engine.But few pipelines are giving unexpected msg: “Expecting value: line 1 column 1 (char 0)”. It is only after re validating these pipelines sequentially, correct validation results are obtained.What is this error?Is it possible to call validate_method() of sch by multiple pipelines at the same time?Uploaded screenshot contains the unexpected error,invalid and valid pipeline output messages. 

0
M
Fan
1 month ago
N
Fan
neelam_choudharyFan
posted in Got a Question?

Connection downloading and importing to different control hub?

Export and import of connection from one control hub to another is possible ?With command like connections.get_all() . We get only id, type and name . What about other configuration like (url , credential, tags)And for importing this to different control is also not working as we tried connection_builder mannually providing the values for only id and name . Not sure what to pass on the sdc url? Any suggestions here

0
N
Fan
1 month ago
SungChul
Fan
SungChulFan
asked in Got a Question?

How far is it to use in a private environment?

I want to use streamsets in a private environment that is not accessed externally.But I can't find the file to install directly anywhere.I need help.thank you

1
AkshayJadhav
StreamSets Employee
1 month ago
SungChul
Fan
SungChulFan
asked in Got a Question?

Installation inquiry, StreamSets Data Collector

Is there any way to install it locally?Can't login to StreamSets Control Hub after installing with dockerThe login error is as follows.java.lang.RuntimeException: DPM Login failed, status code '403': {"message":"Authentication failed"} at com.streamsets.datacollector.cli.sch.SchAdmin.retrieveUserToken(SchAdmin.java:264) at com.streamsets.datacollector.cli.sch.SchAdmin.enableDPM(SchAdmin.java:82) at com.streamsets.datacollector.restapi.AdminResource.enableDPM(AdminResource.java:142) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.r

3
dima
StreamSets Employee
1 month ago
S
Fan
shobhit_pathakFan
asked in Got a Question?

Export the connection from one control hub and import to another control hub using sdk python script

Export script :  ( please ignore the api key and token being provided as hard coded value , this can be parameterized ) from streamsets.sdk import ControlHubimport jsonAPI_CREDENTIAL_ID = "**"API_TOKEN = "**"sch = ControlHub(API_CREDENTIAL_ID, API_TOKEN)connections = sch.connections.get_all()for connection in connections:    filename = connection.name.lower().replace(' ', '_') + '.json'    file= open(filename, "a")    file.write(str(connection))    file.close() With above code we are able to export the connection of pipeline from one control hub in Json file Now to import the connection in another control hub . I am not sure if its feasible to do so? as we are not having enough information in json file. It carries only  (Connection id , name and type) . But it don't have Connection URL  and credential. Without that how can the connection be imported to another control hub ? Any suggestion please help . Also incase if I want to import the connection directly without bringing the details

0
S
Fan
1 month ago
P
Discovered Fame
Priyanka MynepallyDiscovered Fame
asked in Got a Question?

File upload to HTTP

I have a file in resources directory.I need to upload whole file through HTTP.What options to cnfigure. Able to do it in postman

2
P
Discovered Fame
1 month ago
P
Discovered Fame
Priyanka MynepallyDiscovered Fame
asked in Got a Question?

Directory/Groovy

Hi Team I need to read 2 whole files from directory.abc.json and abc.csv at a time. I gave abc.* in filename pattern but no luck.Can someone help me please.. Or can you suggest an alternative using groovy. RegardsPriyanka  

2
P
Discovered Fame
1 month ago
Vikash
Fan
VikashFan
posted in Got a Question?

hcb y u can be in the same time I will be in

Gghtt y jjj and trucks and 

0
Vikash
Fan
1 month ago
P
Discovered Fame
Priyanka MynepallyDiscovered Fame
asked in Got a Question?

POST call http

Hi Team I am able to extract 2 files abc.json and abc.csv from directory as whole files.Now I need to insert both in POST call through HTTP.I tried form data, but its not working. In postman I gave 2 keys “xyz”: (uploaded abc.json), “xyz1”:{uploaded csv file}And its working How to do same thing in streamsets? I used groovy but its reading whole file and converting to string Hi Team Can someone help me on this please? RegardsPriyanka

0
P
Discovered Fame
1 month ago
D
Fan
dhanraj_shindeFan
asked in Got a Question?

In pythonSDK how to get type of pipeline

In python SDK after fetching pipeline details with use of ControlHub object (sch.pipelines).How can we get type of pipeline (Data collector or transformer)?  

1
saleempothiwala
Opening Band
1 month ago
S
Fan
shivrajFan
asked in Got a Question?

Getting this error while imporing stub?

from streamsets.testframework.decorators import stub from streamsets.testframework.utils import get_random_stringBoth commands are not working.Everytime I run the code Seeing same error even after installing the module.ModuleNotFoundError: No module named 'streamsets.testframework.decoratorsPlease help out with this?

4
S
Fan
1 month ago
V
Fan
VIreshFan
asked in Got a Question?

Resuming pipeline after pipeline failure and Batch processing behaviour

I need to build a fail safe pipeline where In case the pipeline is stopped abruptly due to some issue on the VM side or the SDC, There needs to be a safe exit.The batch it was working on should not be lost or partial data should not be written.When restarted I need to make sure it does not start from the 1st batch again but needs to pick up from where it left off.What does streamsets have to deal with problem like this??

1
saleempothiwala
Opening Band
1 month ago
Victor01
Fan
Victor01Fan
asked in Got a Question?

How to enable the acl for aws s3 destination bucket in streamsets data collector

Hi Team, I was facing the issue to keep my whole file to s3 destination bucket using private endpoint with acl. I am using streamsets datacollector 5.3 version. The customer s3 destination bucket requires canned acl to put file on that destination bucket path. The required canned acl would be bucket-owner-full-control. Can you please help me with detailed steps how to configure acl with s3 private endpoint link on destination bucket. We use Instance Profile without access keys and secret keys. Thanks.

1
saleempothiwala
Opening Band
1 month ago

Badge winners

Show all badges
Terms & Conditions