Have a Question about StreamSets? | StreamSets Community

58 Topics

Unable to inject large XML data (4MB size) from S3 to Snowflake (Data collector)

While trying to inject XML data from S3 into snowflake, facing the below error :S3_SPOOLDIR_01 - Failed to process object 'UBO/GSRL_Sample_XML.xml' at position '0': com.streamsets.pipeline.stage.origin.s3.BadSpoolObjectException: com.streamsets.pipeline.api.service.dataformats.DataParserException: X...

XSLT/Freemarker Transformation

Dear StreamSetsWe have an requirement to transform Complex XML data into JSON using XSLT. This needs to be done in DataCollector. The incoming file will contain millions of records and for each record, we need to apply XSLT and write the output to S3 location.I could not find resource on support for...

XML to JSON and derivations

Hi, I have a XML as shown below<events> <event> <type>online</type> <event_date>1-Jan-21</event_date> <feedback_status>Closed</feedback_status> </event> <event> <type>online</type>...

Standard out-of-box origins for SaaS apps

Hi StreamSets,I would like to know whether there is an initiative to introduce standard origins for various popular SaaS apps like Shopify, Magento, Branch etc? If there is a space where we can vote for these connectors based on which they can be prioritised for development, please do share that inf...

Swayam's Micro-blogging: Auto detection of date format

Dear Community users,SDC offers a great out-of-box processor “Field Type Converter” to convert an incoming field of one data type to another data type. This is a very useful transformation function that is required before we send the data to final destination. However, sometime we may have to write...

StreamSets Airflow Integration

Does StreamSets Data Collector and Transformer support integration with Airlfow? Could you please share details on the integration? i.e. Prebuild StreamSets Operator for Airflow

S3 source connection issue

I have configured streamets transformer larger on aws, i am trying to create pipe line , to get data from s3 source , we are getting an error like expecting file:/// we gave it s3://<bucket name> and also tried s3a://<bucket name>

Gradle Build Problem with Golang go-gradle plugin

I am using legacy version till now and facing this issue of build recently.I was trying to build the datacollector-edge-oss version 3.14 from source with commandgradlew goClean dist publishToMavenLocal --build-cache --stacktrace --info --scanand the build is failing as per below scan results : gradl...

Unreachable web interface

Whenever I try to log in the web interface I get a 502 error.This is happening when I try to reload a biggish table from a Mysql snapshot.VM is under reasonable load (25% cpu, 50% RAM)Scd service is up and running and apparently the pipelines are running as usual.

Unable to Read Data Using S3origin in Data Transformer

Hi Team,I am facing issue i.e. reading data through S3origin within data transformer. I am able to read data through s3 origin in data collector.Trying to read data from S3 origin and copy the same in different location using S3 destination. I am using EMR as an computing engine. Job runs for sever...

How StreamSets will communicate/connect to Immuta governance Tool ?

Team...How we can connect Immuta governance by using Streamsets to do the required ETL Operations.

Swayam's Micro-blogging: Combine multiple JSON records into a single one

Dear Community users,To extend the last topic where I shared how to convert the CSV to JSON, what if you have to combine multiple JSON records in a batch into a single JSON?I often came with requirements where we have to call an external API which suports both single as well as batch requests and mo...

Any specific templates to write a blog

We are using 3.21 OSS StreamSets in a unique way where we are always creating pipelines as templates. We have given a custom UI in customer's hand to pick their preffered source. Based on their choice, we call Streami REST APIs to create customer specific pipelines in real time. That's awesome me an...

Scheduler page in SCH

When a scheduled job is deleted, will the record associated with that job in the scheduler page get automatically removed?

Running Error pipeline couldn`t stop

Running Error pipeline couldn`t stop,must restart sdc?anyone meet this problem?

Any insights on having two copies of same pipeline with different names in same SCH

I have the JSON file of a streamset pipeline.I need to have two copies of this pipeline with different names in a single SCH. We tried to rename the title in the json file with the first name and imported to SCH. The next time I changed the title and on import to the same SCH, the version of the pip...

What is the value of the StreamSets DataOps Platform?

What does it mean for Data Collector to be close source?

What is the StreamSets Summer ‘21 free service tier?

Is the StreamSets Summer ‘21 free service tier completely equivalent to the Data Collector 3.x download?

What does StreamSets Summer ‘21 include?

What are the key capabilities of StreamSets Summer `21?

Where can I find sample pipelines?

Where can I find some sample pipelines to start using them as templates in StreamSets DataOps Platform?

Where can I learn more about StreamSets DataOps Platform?

1
2
3

Page 2 / 3

Got a Question?

58 Topics

Exception while using Windowing Aggregator

Logging mechanism for Data Transformer and Data Collector pipelines

conversion error

Unable to inject large XML data (4MB size) from S3 to Snowflake (Data collector)

XSLT/Freemarker Transformation

XML to JSON and derivations

Standard out-of-box origins for SaaS apps

Swayam's Micro-blogging: Auto detection of date format

StreamSets Airflow Integration

S3 source connection issue

Gradle Build Problem with Golang go-gradle plugin

Unreachable web interface

Unable to Read Data Using S3origin in Data Transformer

How StreamSets will communicate/connect to Immuta governance Tool ?

Swayam's Micro-blogging: Combine multiple JSON records into a single one

Any specific templates to write a blog

Scheduler page in SCH

Running Error pipeline couldn`t stop

Any insights on having two copies of same pipeline with different names in same SCH

What is the value of the StreamSets DataOps Platform?

What does it mean for Data Collector to be close source?

What is the StreamSets Summer ‘21 free service tier?

What does StreamSets Summer ‘21 include?

Where can I find sample pipelines?

Where can I learn more about StreamSets DataOps Platform?

Cookie policy

Cookie settings

Got a Question?

Sign up

Social Login

Login to the community

Social Login

Scanning file for viruses.

This file cannot be downloaded

Cookie policy

Cookie settings