- - Knowledge base
Product Updates
Groups
Join Build It With Brenna

Build it With Brenna: Hands-on Workshop with a MySQL Origin and a Snowflake Destination

(Wed, 29 Mar, 17:00)

Home
Community overview
Platform Q&A
Got a Question?

Got a Question?

Can't find what you're looking for? Ask it here!

612 Topics
1,570 Replies

When you subscribe we will email you when there is a new topic in this category

612 Topics

Recently active

Most replies Most views Newest first

Drew KreigerSenior Community Builder at StreamSets

asked in Got a Question?

How do I downloading a local version of StreamSets on my laptop?

How do I downloading a local version of StreamSets on my laptop?

Drew KreigerSenior Community Builder at StreamSets

asked in Got a Question?

I deployed a StreamSets data collector image from GCP marketplace but can't log in using the default initial username and password (admin, admin). Where can I find the correct initial username/password for my deployment?

I deployed a StreamSets data collector image from GCP marketplace but can't log in using the default initial username and password (admin, admin). Where can I find the correct initial username/password for my deployment?

Drew KreigerSenior Community Builder at StreamSets

asked in Got a Question?

What language is the expression editor in?

What language is the expression editor in?

Drew KreigerSenior Community Builder at StreamSets

asked in Got a Question?

How do you read the data from an API beyond the rows of records?

What if you need to read your data from an API that has a more complex structure than just rows of records?

Drew KreigerSenior Community Builder at StreamSets

asked in Got a Question?

Is there an option to pretty print the JSON in the raw data editor?

Is there an option to pretty print the JSON in the raw data editor?

Drew KreigerSenior Community Builder at StreamSets

asked in Got a Question?

How do you access Neo4j?

How do you access Neo4j?

Drew KreigerSenior Community Builder at StreamSets

asked in Got a Question?

How do I install StreamSets on a Windows 10 machine?

I am trying to install on StreamSets on a Windows 10 machine. As per the documents for self-managed deployment supported OS versions do not include Windows 10. So, I am using docker image. But it is still not working. Could you please guide me how I can use StreamSets on Windows 10 platform?

Drew KreigerSenior Community Builder at StreamSets

asked in Got a Question?

Is it required that my EC2 Instances have internet access for this to work?

I am planning to use StreamSets Control Hub with AWS EC2 Instance. Is it required that my EC2 Instances have internet access for this to work?

Drew KreigerSenior Community Builder at StreamSets

asked in Got a Question?

How to setup a continuous data replication pipeline from SQL Server to Snowflake?

We would like to setup a continuous data replication pipeline from SQL Server to Snowflake, completed with historical data for several hundred tables. Would like some documentation/assistance with creating a test pipeline for our use case.

Drew KreigerSenior Community Builder at StreamSets

asked in Got a Question?

I want to know how to migrate the data from my Oracle local database to PostgreSQL.

I want to know how to migrate the data from my Oracle local database to PostgreSQL.

Drew KreigerSenior Community Builder at StreamSets

asked in Got a Question?

Is there a way to configure the Snowflake destination to make periodic updates on Snowflake rather than make real-time?

We are trying to replicate data from SQL Server to Snowflake. Is there a way to configure the Snowflake destination to make periodic updates on Snowflake rather than make real-time? Having the pipeline run in real-time would mean high credit usage on the snowflake warehouse.

Drew KreigerSenior Community Builder at StreamSets

asked in Got a Question?

Does StreamSets support automatic propagation of DDL changes when using the CDC Client without data loss?

Does StreamSets support automatic propagation of DDL changes when using the CDC Client without data loss?

Drew KreigerSenior Community Builder at StreamSets

asked in Got a Question?

Autocreate Tables to Replicate to Snowflake Error

We have hundreds of tables to replicate to Snowflake as our destination, so it makes sense to have StreamSets autocreate these at the destination with the correct DDL but the functionality does not seem to be working. Besides checking the 'Table Auto Create' box are there any other steps we should take?

Drew KreigerSenior Community Builder at StreamSets

asked in Got a Question?

Is there any way to sort records?

Is there any way to sort records?

Drew KreigerSenior Community Builder at StreamSets

asked in Got a Question?

Can I use a variable in the 'Request URL' of an HTTP client?

Can I use a variable in the 'Request URL' of an HTTP client?

asked in Got a Question?

Unreachable web interface

Whenever I try to log in the web interface I get a 502 error.This is happening when I try to reload a biggish table from a Mysql snapshot.VM is under reasonable load (25% cpu, 50% RAM)Scd service is up and running and apparently the pipelines are running as usual.

ashok vermaDiscovered Fame

asked in Got a Question?

refreshing pipeline launch using SDK

After launching pipeline using SDK, if i have any changes in pipeline, i want to do it from SDK instead of UI and those changes has to reflect in UI. how can i achieve this one without again launching pipeline.some more queries1.preview using SDKhow to know stage has no errors in SDK

asked in Got a Question?

Invoking a SDC job with a set of input parameters

Hi, We know a SDC job be involked through HTTP endpoint and can a set of parameters be passed as input parameter?Effectively we want the following :Originating system X > invoke SDC job with a set of parameters > SDC job looks up a database table with the passed on parameters > Returns the results to a different external system Y[ note its not the originating system X] > Send a response back to Originating system with a pass/fail status.Any demo pipeline will help. Are we advised to use REST Service origin for https://docs.streamsets.com/portal/controlhub/latest/help/datacollector/UserGuide/Microservice/Microservice_Title.htmlRegards, Anirban

ashok vermaDiscovered Fame

asked in Got a Question?

issue while reading a file

i am reading a fixed width file from s3 and trying to parse with Jython evaluator in Data collector.when i try to run the pipeline i am getting below isue. Script error while processing batch: javax.script.ScriptException: java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: Cannot create PyString with non-byte value in <script> at line number 60i am reading file using UTF-8 option.

swayamDiscovered Fame

asked in Got a Question?

Unable to get data from Shopify via HTTP Client Origin using pagination

Hi,I’m trying to fetch data from Shopify using HTTP Clent Origin with pagination set to “Link in HTTP Header”.As per Shopify (refer link here - https://shopify.dev/api/usage/pagination-rest#link-headers) - the parameters that comes back in “link” contain the page_info parameter that we should be using while querying the next page.Using StreamSets HTTP Client Origin, the pagination option using “Link in HTTP Header” is not working. I think this is because SDC is expecting the parameter name as “page” and hence this issue.Has anybody faced this earlier and if so how have you been able to make progress here?Anyone from StreamSets who can comment on this?

asked in Got a Question?

Integrate Bit bucket for already written module based pyspark code

Already have module based pyspark code ..So how can we use the same without creating and adding stage in streamset pipeline.Can we dump or integrate with bitbucket repo?

asked in Got a Question?

NullPointerException when starting a pipeline

We are encountering a NullPointerException error when starting a pipeline which is moving the data into a BigQuery table from a postgres database. After a while (an hour) the pipeline is starting. We are using StreamSets 3.8.0

asked in Got a Question?

HOW TO FIX ERROR THAT IS NOT CORRESPOND WITH MARIADB/MYSQL VERSION FOR THE RIGHT SYNTAX.. First error: SQLState: 42000 Error Code: 1064

My Streamset loading get this error but i don't know how to fix it. In detail, i paste the error and pineline image here. Many thanks for helping com.streamsets.pipeline.api.StageException: JDBC_77 - SQLSyntaxErrorException attempting to execute query 'SELECT ID AS product_id, AMOUNT AS amount, MODIFIED_DATE FROM TBL_PRODUCT WHERE MODIFIED_DATE > 2021-12-27 17:31:43 ORDER BY MODIFIED_DATE'. Giving up after 1 errors as per stage configuration. First error: SQLState: 42000 Error Code: 1064 Message: You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near '17:31:43 ORDER BY MODIFIED_DATE' at line 6 This error appear when i have verified and the pineline have run, the Mariadb database received full data successfully. But then this error came

Diana LacatusFan

asked in Got a Question?

Transfer of data

Hi everyone,I would like to know if it is possible to introduce a delay between the transfer of the records, in order to not send the whole information at once. I want to transfer the data record by record.Could you help me, please?

swayamDiscovered Fame

asked in Got a Question?

How to ignore the unexpected line in CSV file and proceed to next?

Hi - while processing a CSV file via Directory Origin, I encountered the following error "SPOOLDIR_01 - Failed to process file '/tmp/out/customer_daily_incremental/CUSTOMER/CUSTOMER_20092021.csv' at position '177394': com.streamsets.pipeline.lib.dirspooler.BadSpoolFileException: java.io.IOException: (line 1138) invalid char between encapsulated token and delimiter" - I now know that the records present in line 1138 doesn't comply to the expected format (I see there is an additional double quote). But I don't want SDC to completely abort this file. Is there a configuration available where I can opt to log the record with error message and proceed to next line in the file?

1
...
17
18
19
20
21
22
23
...
25

Page 20 / 25

Badge winners

AkshayJadhavhas earned the badge Eager to help
john.durkinhas earned the badge Eager to help
samhas earned the badge Eager to help
saleempothiwalahas earned the badge Eager to help
Bikramhas earned the badge Community Champion 🏅

Show all badges

Powered by inSided

Terms & Conditions

Sign up

Already have an account? Login

Social Login

or

Username *

E-mail address *

What I do... *

Data Leader Data Architect Data Engineer Data Scientist Other

Company *

Country *

Zip Code *

Marketing Communications

Yes No

Password *

I have read and Agree to the Website Terms of Service and I have read and acknowledged the Privacy Policy.

loginBox.register.email_repeat

Login to the community

No account yet? Create an account

Social Login

or

Username or Email

Password

Remember me

Forgot password?

Enter your username or e-mail address. We'll send you an e-mail with instructions to reset your password.

Username or e-mail

Back to overview

Scanning file for viruses.

Sorry, we're still checking this file's contents to make sure it's safe to download. Please try again in a few minutes.

OK

This file cannot be downloaded

Sorry, our virus scanner detected that this file isn't safe to download.

OK