Skip to main content

Community Articles and Got a Question?

Can't find what you're looking for? Ask it here or check out the Community articles

  • 1,027 Topics
  • 2,316 Replies
1027 Topics
R
Fan
RohithFan
asked in Community Articles and Got a Question?
File watcher event in streamsets

I want my pipeline to automatically trigger only when the source file is placed on the server. How can we achieve this in streamsets?

2
saleempothiwala
Headliner
1 year ago
P
Fan
Priya151997Fan
asked in Community Articles and Got a Question?
How to configure REST service

Hi Team,As, I am new to Streamsets want to have more information about Rest Service.We have databases on on-prem, We run procedure on database that initiate the rest API call to tomcat(service published on this) web server on port 8908 redirectedNow we receive one url which contain different parameter. I need to configure the same in Streamsets.Can you please let me know, How we can configure same in rest service.  

1
Bikram
Headliner
1 year ago
D
Discovered Fame
DolphinDiscovered Fame
asked in Community Articles and Got a Question?
Field renamer failure

Hi team,I am using field renamer for mssql column name rename from origin mssql “regionid” to target snowflake “RegionID”.Attchment 1 is my configure for Field Rename configuration. When I ran “previewer”, did not show any errors.Attachment 2 is log.Cannot figured out what is the exact reason for the failure from the log.Can you suggest?

15
Bikram
Headliner
1 year ago
sanket.bhendarkar
Roadie
sanket.bhendarkarRoadie
asked in Community Articles and Got a Question?
java.lang.NoClassDefFoundError: Could not initialize class com.datastax.driver.core.Cluster

Hi, community,  I am using Streamset data collector for data migration from PostgreSQL to Cassandra But while testing the Cassandra connection I am encountering the above error. 

12
AkshayJadhav
StreamSets Employee
1 year ago
P
Fan
Priya151997Fan
asked in Community Articles and Got a Question?
How to initiate the emr cluster through streamset

How to initiate the emr cluster through streamset

6
P
Fan
1 year ago
ravikumar
Fan
ravikumarFan
asked in Community Articles and Got a Question?
How to get WSUS server details from CyberArk Vault server?

When checked at path Computer Configuration -> Administrative Templates -> Windows Components -> Windows Update, I don't see any detail for WSUS. Please tell what's the other way to check WSUS server detailsCyberArk Vault 

1
AkshayJadhav
StreamSets Employee
1 year ago
L
Fan
LeoWoodSideFan
asked in Community Articles and Got a Question?
GraphQL

Has anybody used GraphQL as data source? I’ve been tried to connect it using StreamSets, but, no luck! 

2
L
Fan
1 year ago
xarnthehero
Fan
xarntheheroFan
asked in Community Articles and Got a Question?
JDBC Lookup SQL from record

I’m looking to make a pipeline fragment where the query to be run will be passed in as a pipeline fragment parameter.It looks like it is not possible specify the SQL Query section as something like ${query} since these values get sanitized. You can only do something like SELECT * FROM TABLE WHERE ID=${id_param}. Is it possible to set a value in an expression evaluator like/query=”SELECT * FROM DUAL”and have it run by referencing the /query variable on the record?

3
Bikram
Headliner
1 year ago
Y
Discovered Fame
yogesh0590
asked in Community Articles and Got a Question?
Update “globalMaxRetries” job config using REST API Call..

Hello Team,I need to update one job config which is “globalMaxRetries” using rest call.I refer Restful section in Control Hub but that Rest call for “updateJob” requires many other configs also needs to mention in Request Body. could you please assist me with this ?

1
Rishi
StreamSets Employee
1 year ago
M
Fan
Madhavan PFan
asked in Community Articles and Got a Question?
Find the week number

HI Team, Please help me to find the week number from given date.how to use weeknum functions and lookup’s ThanksMadhavan P

5
Bikram
Headliner
1 year ago
HåkonD
Fan
HåkonDFan
asked in Community Articles and Got a Question?
How to pass a parameter with a list of strings to URL in HTTP client origin

The HTTP client origin I’m using has a parameter called “tags”, which is a list of strings (see image below). I’m trying to pass a parameter to the URL to only get records with a certain tag, but it’s either giving me an error message or not filtering at all.When I’m trying to pass the tags parameter directly, like https://……./v1/ssns?tags=’NO’, the pipeline fails. When I’m trying to pass the list item directly, like https://……./v1/ssns?tags[0]=’NO’, the pipeline runs, but it’s not filtering out records at all, and returns all tags.I’m guessing it’s probably a syntax issue, but I’ve tried many different variations without success, and I can’t find any documentation about what the proper syntax is.  

3
mskarthik648
Fan
1 year ago
mblahay
Discovered Fame
mblahayDiscovered Fame
asked in Community Articles and Got a Question?
Why does converting a null from string to date result in 1/1/1970?

When running strings through the Field Type Converter, going from string to date, if the string value is null the resultant date value is 1/1/1970. I would expect the date to also be null. Why am I not receiving a null output? Are dates not capable of being null?

4
mblahay
Discovered Fame
1 year ago
mblahay
Discovered Fame
mblahayDiscovered Fame
asked in Community Articles and Got a Question?
Where is documentation for com.streamsets.pipeline.api?

The various code evaluators offer an option for “Record Type” of “Data Collector Records.” This is important as it avoids any unexpected type changes that occur otherwise.  However, I have been unable to find any documentation on the Api. Most of my success has been a result of good guessing starting from the three lines of example provided in the processor documentation. I have reached a point where guessing is not enough; where can I find the api documentation?

2
mblahay
Discovered Fame
1 year ago
M
Fan
meghrajFan
asked in Community Articles and Got a Question?
Directory origin does not find directory

Even though directory exists and given permision to all. on preview , SDC giving error “Directory doesnot exists.

3
M
Fan
1 year ago
JoeMeszaros
Fan
JoeMeszarosFan
asked in Community Articles and Got a Question?
How to increase Throughput for JDBC Multitable

Hello, I was wondering if there was a way to increase the throughput for the JDBC multitable consumer origin? I know that when it comes to the multitable origin it only runs 1 table per thread so increasing the number of threads doesnt do anything. If there another way to increase the throughput without increasing the number of threads?

3
saleempothiwala
Headliner
1 year ago
HåkonD
Fan
HåkonDFan
asked in Community Articles and Got a Question?
Dynamic unix timestamp in request URL for HTTP client origin

The origin has a parameter ‘since’ that is a UNIX timestamp, which tells the API to give all records since the given timestamp. Looks something like https://……./v1/ssns?since=1667257200000. I would like this value to be dynamic. For example, if I set my pipeline to run once each day I would like the timestamp to give me only records from the last 24 hours.I’ve tried variations of like ?since={time:dateTimeToMilliseconds(time:now())-86400000}, but I can’t find the right syntax. Is it possible to put this kind of statement in the resource URL that returns a unix timestamp integer? Or if not, any other way to do this?

1
saleempothiwala
Headliner
1 year ago
miguelrezende7rox
Fan
miguelrezende7roxFan
asked in Community Articles and Got a Question?
Inject params to pipeline

Hi guys! I'm building a pipeline that uses a token that is updated hourly. So when I start the pipeline I have to input it as a parameter (or in another way that works). I'm facing problem to inject the token using the api or another pipeline, anyone could help?    

2
saleempothiwala
Headliner
1 year ago
URIBPA
Fan
URIBPAFan
asked in Community Articles and Got a Question?
Store Procedure in JDBC Query consumer - incomplete data

Hi all. I have a stored procedure (quite lengthy that creates a temporary table and then deletes it) that I have managed to get to run in JDBC Query Consumer (in full mode). This procedure performs the query to SQL Server and on average it takes about 9 minutes to get the result of the execution from a database manager. The issue is that when I perform a "Preview" it does not bring me the same data and what it brings, it brings it incomplete. Do you know what could be affecting the result? I used the query directly and the query improved a little but not completely.I also tried in "Test Run" but it does not give me the result of the query.What could it be?  

1
saleempothiwala
Headliner
1 year ago
mskarthik648
Fan
mskarthik648Fan
asked in Community Articles and Got a Question?
REMOTE_UPLOAD_03 - Upload error while sending the file to SFTP location

I am using streamsets pipeline to upload a file from GCS bucket to SFTP location, occasionally i am getting the below error. the error is causing due to renaming the file after uploading to the SFTP location. not sure how to fix it. "errorCode":"REMOTE_UPLOAD_03","errorMessage":"REMOTE_UPLOAD_03 - Error happened when writing to file 'net.schmizz.sshj.sftp.SFTPException: Resource busy. Failed to temp/_tmp_testTrigger.txt to temp/testTrigger.txt because the file is currently in progress.'. Reason: {}","errorTimestamp":1667283307037,"errorStackTrace":"net.schmizz.sshj.sftp.SFTPException: Resource busy. Failed to rename temp/_tmp_testTrigger.txt to temp/testTrigger.txt because the file is currently in progress.\n\tat net.schmizz.sshj.sftp.Response.error(Response.java:140)\n\tat in progress.\n\tat net.schmizz.sshj.sftp.Response.error(Response.java:140)\n\tat net.schmizz.sshj.sftp.Response.ensureStatusIs(Response.java:133)\n\tat net.schmizz.sshj.sftp.Response.ensureStatusPacketIsOK(Response.

3
mskarthik648
Fan
1 year ago
P
Discovered Fame
Priyanka MynepallyDiscovered Fame
asked in Community Articles and Got a Question?
Groovy/Javascript query

Hi TeamI need to discard records which have even one field null and other records should pass. This should happen in loop. If I use pre-condition/stream selector, even if 1 record fails conditions, rest pass, even then entire flow gets discarded. records are comming as below in javascriptrecords[i].value.A[0][‘x1’]records[i].value.A[0][‘x2’]records[i].value.A[1][‘x1’]records[i].value.A[1][‘x2’]If records[i].value.A[0][‘x1’] is null/records[i].value.A[0][‘x2’] is null, only A[0] should be discarded. and A[1] should flow this way I need  In this case can anyone help me how to achieve this in groovy? javascript?

1
Bikram
Headliner
1 year ago
R
Fan
ramkumar_bvrFan
posted in Community Articles and Got a Question?
How to Pass Index value to Field Renamer Stage instead of source field name

I’m trying to automate our data restatement process. Based on the source field we are changing it to generic field as field1,field2,field3…..fieldN. But if we use Field Renamer stage it is mandatorily asking the exact source field name. Will it be possible to pass the index value instead of Source Field Name. Please let me know, if you need any further information on this. 

2
Bikram
Headliner
1 year ago
A
Discovered Fame
akanshajain6793Discovered Fame
asked in Community Articles and Got a Question?
Conversion failed when converting date from character string

Hi Team am trying to process a file and in that i have date column for which two records am getting date as 00000130 and 00000706  the year in date field has value as 0000 which is causing the records to drop the problem is due to this two records streamsets is not processing other good records too if i 1 million in records in file its not loading around 1k records after those two error records   

1
Bikram
Headliner
1 year ago
mblahay
Discovered Fame
mblahayDiscovered Fame
asked in Community Articles and Got a Question?
How does one remove all "Pipeline State Changes" from the Notifications tab?

It has been found that when attempting to clear out the list of “Pipeline State Changes” from the Notifications tab of the pipeline configuration, an error is displayed saying that “" (empty string) is not a valid value. How can this error be avoided?  Of note is that this error does not occur if editing a pipeline directly on the data collector. This would seem to indicate that there might be an issue with the Control Hub UI.

1
S
StreamSets Employee
1 year ago
chisou
Roadie
chisouRoadie
asked in Community Articles and Got a Question?
Can't connect to Azure SQL using JDBC Producer

Hi everybody,I have trouble writing to a Azure SQL database using JDBC. This is the error message:Reference to database and/or server name in 'dbo.schema.table' is not supported in this version of SQL Server.There is not much information about this error, it seems to have something with Azure SQL not being a full-blown MS SQL server. Also adding `dbo` is not necessary in this case.My JDBC string looks like this:jdbc:sqlserver://censored.database.windows.net:1433;database=censored;encrypt=true;trustServerCertificate=false;hostNameInCertificate=*.database.windows.net;loginTimeout=30;I’ve configured the JDBC connection like this: This method (putting `dbo` as schema and `schema.table` as table name) is the only one that appears to work. All other combinations omitting `dbo` result in errors like these:Table 'dbo.schema.table' does not exist or PDB is incorrect. Make sure the correct PDB was specifiedAny ideas? Has anybody else already connected Azure SQL via JDBC? How did you configure it

7
chisou
Roadie
1 year ago
E
Fan
emily.suchanFan
asked in Community Articles and Got a Question?
Programmatically Add Labels to Data Collectors

I am trying to use the StreamSets SDK python package to add labels to my data collectors. When I call the directory of the Control Hub instance, it shows an attribute “update_data_collector_labels”, it only takes one argument which is the data collector. Is there some syntax I am missing to be able to add the labels to this data collector? Does this functionality exist? or is there a workaround I could implement to achieve this? Sample Code: from streamsets.sdk import DataCollectorfrom streamsets.sdk import ControlHubcontrol_hub = ControlHub(control_hub_link, username=username,password=password)datacollectors= control_hub.data_collectorsdata_collector= datacollectors.get()#this line shows the current labels for the data collectorscontrol_hub.get_data_collector_labels(data_collector)#the function only takes one argument. I was hoping I could use this line to update the data collector labelscontrol_hub.update_data_collector_labels(data_collector) Thank you! 

2
E
Fan
1 year ago

Badge winners

Show all badges
Terms & Conditions