Hi, I am new to StreamSets so I may miss a thing here, if this is the case please apologize!Wanted to configure the PostgreSQL metadata processor and found that the only way to connect is to directly configure the JDBC connection string and credentials. There seems no way to use a connection previously configured, like when setting up a JDBC (multitable) consumer or JDBC Producer.Is this a defect/missing feature or do I misunderstand something? Thank you in advance!
We currently are using an OTLP DB and read the data for CDC via the Archive / Redo Logs. However this is limiting us so if we spin up an Oracle ADG or a CDG instance can we read and process our CDC from the logs that are replicated on an ADG or CDG instance ?
Learn to perform a SAML trace and generate SAML Trace JSON file using browser plug-ins. Streamsets support team uses the file to trace the SAML assertions occurring between your identity provider and Control-Hub to troubleshoot your sign-in issues. Use the links below to download and install the SAML tracer plug-in for your browser: Note: The links and steps provided here were correct at time of publishing. Mozilla Firefox: https://addons.mozilla.org/en-US/firefox/addon/saml-tracer/ Google Chrome: https://chrome.google.com/webstore/detail/saml-tracer/mpdajninpobndbfcldcmbpnnbhibjmch When the plug-in is added, Click the newly added SAML tracer icon on the upper-right add-in menu for your respective browser. This opens the SAML Tracer dialog box. The SAML tracer dialog box records and displays details as shown below. Note the occasional SAML tags shown at the right indicating SAML assertions being passed. Re-produce the issue. use the SAML Tracer dialog to navigate to Export tab
i try to connect neo4j to python and it successfully connected but then i dont know how to get hive data into neo4j through python
My JDBC producer stage is giving the error java.lang.NoClassDefFoundError: org/apache/commons/lang3/StringUtils Has anyone had a similar issue before?
I am trying to call a SOAP API which requires page number and number of records to be mentioned in request body.We have multiple pages in response however streamsets pipeline just ingest the first page. resource URL - just URL of the API call , no variable for pagination here .Headers : blankmode : batchHTTP Method : POSTRequest body : complete request body with page number <Response Filter> <Page>1<Page> <Count>999<Count></Response Filter> Pagination: By page number initial page :1 rest default can someone help how to retrieve all available pages.tried using value ${startAt} for page in request body.
Hi experts,I have one question and here is the scenario.I am trying to connect from Mssql to AWS s3. I m going to do incremental load for 2 tables for example. Everything will be done in 1 schema.Table Name Pattern List - Table A , Table BTable AOffset column - LastUpdatedDate , Initial Offset Column - LastUpdatedDate , Value - “-2208988800000” which is “1900-01-01 00:00:00.000”Table BOffset column, Initial Offset Column, Value are all same with Table A. How JDBC MultiTable handle the incremental load for two tables with same offset columns. I try to update the rows in Table A, and I start the pipeline, it will get me the updated file and StreamSets will set the offset value itself. When I update or insert new records again, it will start reading data based on previous offset value and get the another updated file However, when I try to update the rows in Table B, and I start the pipeline, I didn’t get anything.What’s the problems here ? My thought is like Table A , use first offset
Hi experts, is it possible to delete objects from an s3 bucket using a DataCollector Pipeline? Didn't find anything… Thanks, Marco
Hi, I try to combine a datacollector and a transformer pipeline (due to the missing parquet s3 support in datacollector). I read data from a Postgres db and publish it to Kafka using the datacollector. I have a Kafka connection that I want to use for both pipelines. Pushing Data to Kafka works but when I run the transfom pipeline (using the same connection) I get the following error: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in stage 0.0 (TID 0, 53f742dd118a, executor driver): java.lang.NoSuchMethodError: ‘void org.apache.spark.sql.kafka010.consumer.InternalKafkaConsumerPool$PoolConfig.setEvictionPolicy(org.apache.commons.pool2.impl.EvictionPolicy)' at org.apache.spark.sql.kafka010.consumer.InternalKafkaConsumerPool$PoolConfig.init(InternalKafkaConsumerPool.scala:191) at org.apache.spark.sql.kafka010.consumer.InternalKafkaConsumerPool$PoolConfig.<init>(InternalKafkaConsumerPool.scala:1
Hello, could you please help me out?In Transformer pipelines adding a Kafka consumer throwsjava.lang.NoClassDefFoundError: Could not initialize class org.apache.spark.sql.kafka010.consumer.KafkaDataConsumer$The same connection works fine with Data Collector.I am using local Spark in docker Thanks!
I would like to stream data from an API and write the output to Oracle autonomous data warehouse (ADW). Anyone else have experience doing this?
Hello , i created a pipeline and it runs without errors , the output is as expected which is 2777 records however when i run preview it shows no data records except for the one in the screenshot below .and i couldn’t know why this happens .
The JDBC Lookup documentation explains how to configure the cache and how the eviction works, but there is no discussion about how the cache actually works. Is the query being evaluated and then hashed, with the hash value being used as a key? or is something else happening?
Become a leader!
Learn how to make the most of StreamSets with user guides and tutorials
Get StreamSets certified to expand your skills and accelerate your success.
Contact our support team and we'll be happy to help you get up and running!
Already have an account? Login
No account yet? Create an account
Enter your username or e-mail address. We'll send you an e-mail with instructions to reset your password.
Sorry, we're still checking this file's contents to make sure it's safe to download. Please try again in a few minutes.
Sorry, our virus scanner detected that this file isn't safe to download.
