- - Knowledge base
Product Updates
Groups
Join Build It With Brenna

Build it With Brenna: Hands-on Workshop with a MySQL Origin and a Snowflake Destination

(Wed, 29 Mar, 17:00)

Home
Community overview
Welcome

Welcome

Welcome to the StreamSets Community

121 Topics

Recently active

Most replies Most views Newest first

tipharcoFan

asked in Community Articles

CÔNG TY CỔ PHẦN DƯỢC PHẨM TIPHARCO

Tipharco đã phát triển 3 nhà máy sản xuất Dược phẩm bao gồm Non Betalactam, Betalactam và nhà máy sản xuất thuốc Dược liệu đạt tiêu chuẩn GMP-WHO. Thông tin liên hệ: Web: tipharco.vn SĐT: 02733872972, Email: info@tipharco.vn, Địa chỉ: Lô 08, 09 CCN và TTCN Tân Mỹ Chánh, P.9, TP Mỹ Tho, Tiền Giang, Việt NamXem thêm:https://tipharco.vn/https://www.instagram.com/tipharco/https://www.facebook.com/tipharco/https://twitter.com/CTipharhttps://www.linkedin.com/in/tipharco-tg/https://www.youtube.com/channel/UCKn1EwVLYGO0ooLE95xKR7ghttps://www.flickr.com/people/tipharco/https://www.pinterest.com/tipharco/https://www.tumblr.com/tipharco

4 months ago

john.mcavoyStreamSets Employee

posted in Community Articles

UTF-8 Special Characters being replaced by Question Marks (�)in Pipelines

Problem DescriptionWhen running an SDC pipeline which processes records with Strings containing UTF-8 special characters, these special characters are being replaced by question marks (? or �) in various parts of your pipeline.ExampleInput record:{"productName": "My Product™"}Output record:{"productName": "My Product�"}Root CauseThis problem indicates that your Java Runtime Environment is using a non-UTF-8 character set which does not support these special characters, so the special characters are being replaced by the � (U+FFFD) REPLACEMENT CHARACTER.By default, SDC will try to set your JAVA_OPTS to use the UTF-8 encoding, however it is possible that the JAVA_OPTS parameters are being set outside of SDC and are overriding either of the following JVM parameters which tells the JRE what encoding to use at runtime: file.encoding and sun.jnu.encoding.The JRE also has a default character set which it will use if these two parameters are not specified. This default character set can vary be

5 months ago

BrennaStreamSets Employee

published in Events & Webinars

StreamSets Roadshow - San Francisco and New YorkNews

I’m excited to write that we’ve announced dates for the StreamSets Roadshow. (Details here and in your right hand tool bar) First upcoming are San Francisco and New York. I wanted to personally invite all my fellow data engineers to join me at my training sessions during these events.If you use my code BuuckSF50 for the San Francisco Roadshow or BuuckNYC50 for New York you can get 50% off a 4 hour training and a half-day of info on best practices and innovation with StreamSets. I hope to see you there!

230

5 months ago

Yago AparecidoDiscovered Fame

asked in Community Articles

versions of streamsets available for free download

hello cloudera community, what is the latest version of streamsets available for free download? ps: rpm download to install on linux what would be the repository link to download all the free versions?

5 months ago

Yago AparecidoDiscovered Fame

asked in Community Articles

how to check jvm usage of streamsets by command line?

hello streamsets community, the streamsets is installed in cloudera manager, but it does not have graphics that show the memory usage of the jvm with that, i need to know how to query the jvm memory usage of streamsets by the command line.

5 months ago

Yago AparecidoDiscovered Fame

asked in Community Articles

list all streamsets pipelines by rest api

hi streamsets community, we need to export the listing of all pipelines that exist in streamsets by rest api How can we do that?

5 months ago

swayamDiscovered Fame

posted in Show us your Pipelines

Convert multiple CSV records into a single JSON and make an HTTP bulk API POST call

Hi,The above pipeline reads the CSV files uploaded into an S3 bucket for new contact file uploads. Based on the new contacts available as a CSV, it converts them into a JSON. But the critical part here is that it merges each batches (each batch size is 1000) into a single JSON of multiple records and finally make a single HTTP POST call to one of our internal system to carry out a CONTACT update call.

5 months ago

RishiStreamSets Employee

posted in Community Articles

ERROR StageLibraryUtil - Failed to read repository manifest json

During SDC startup by default SDC try connecting to repository manifest URL 2022-09-12T21:38:54,103 [user:] [pipeline:] [runner:] [thread:ManifestFetcher] [stage:]INFO StageLibraryUtil - Reading from Repository Manifest URL: http://archives.streamsets.com/datacollector/5.0.0/tarball/repository.manifest.jsonNow, if your SDC doesn’t have internet connectivity it will failed with the below error, However, SDC will start without any issue 2022-09-12T21:38:56,714 [user:] [pipeline:] [runner:] [thread:ManifestFetcher] [stage:] ERROR StageLibraryUtil - Failed to read repository manifest jsonjavax.ws.rs.ProcessingException: java.net.SocketTimeoutException: connect timed out at org.glassfish.jersey.client.internal.HttpUrlConnector.apply(HttpUrlConnector.java:287) ~[jersey-client-2.25.1.jar:?] at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:252) ~[jersey-client-2.25.1.jar:?] at org.glassfish.jersey.client.JerseyInvocation$1.call(JerseyInvocation.java:

6 months ago

N_aliOpening Band

asked in Community Articles

aggregate the records into a single array of objects

I have some json objects that looks as follows and i want them to be all merged in an array like :data: [{nstaskid: “...”,awb: “….”},{nstaskid: “...”,awb: “….”},] i tried to use javascript evalutor , loop over the records and push them in an array and write this record array , but it didn’t work.Can you please help me with that .

6 months ago

PradeepStreamSets Employee

posted in Community Articles

STARTING to RUNNING state of a pipeline

When a data collector pipeline is in STARTING state it will validate each stage, create required clients and also do initialization.Each stage has init() method which initializes the objects needed for stages to run. Validation runs static operations. For example, running the query given in JDBC origin stages with limited results to validate the query, its schema, offset column etc. Do note that validation is specific to each stage.If you notice any slowness in STARTING to RUNNING stage, it is possible that validation is taking sometime and it usually happens when the datasets in configured query are larger. This is an example of slowness for JDBC or Relational DB stages. If the slowness is unacceptable you might consider disabling the query validation.

6 months ago

AkshayJadhavStreamSets Employee

posted in Community Articles

Connecting to Microsoft SQL Server from SDC v5.1 is failing "unable to find valid certification path to requested target"

Environment:Streamsets Data Collector 5.1. Producer or consumer Library for Microsoft SQL Server. Issue: The pipeline starts failing with following messages when try to fetch or produce the data to Microsoft SQL Server.This happens on data collector is 5.1 but the same pipeline is working fine on 4.x versions.com.microsoft.sqlserver.jdbc.SQLServerException: The driver could not establish a secure connection to SQL Server by using Secure Sockets Layer (SSL) encryption. Error: "sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target". ClientConnectionId:af15a6cf-f28d-4e21-adb3-141826becad0 at com.microsoft.sqlserver.jdbc.SQLServerConnection.terminate(SQLServerConnection.java:3680) ~[mssql-jdbc-10.2.1.jre8.jar:?] at com.microsoft.sqlserver.jdbc.TDSChannel.enableSSL(IOBuffer.java:2113) ~[mssql-jdbc-10.2.1.jre8.jar:?] at com.microsoft.sqlserver.jdbc.SQLServer

6 months ago

nick_nilzFan

asked in Community Articles

How do you introduce (write) new columns to an existing hadoop table?

Hello community,My pipeline is being upgraded to ingest more fields and I want to write these new fields to the existing table in Hadoop. I have searched for the metadata processor but it does not exist and I have enabled data drift. None of them are able to write new columns, which means I have to write an ALTER TABLE statement to capture these new fields. With time we will get further requirements to expand our tables and introduce new fields. Is StreamSets able to write new fields automatically?Kind regards,Nick

6 months ago

PradeepStreamSets Employee

posted in Community Articles

(DataOps) Downloading support bundle when Websocket tunneling is enabled

When Websocket Tunneling is enabled, you won’t be able to download support bundle through dataops platform UI. We are shown below message when we try to access it from Engine > Support Bundle page.The support bundle allows you to generate an archive file with the information required to troubleshoot various issues with the engine. To download a support bundle, you must set up direct engine access and turn off Websocket Tunneling in your Browser Settings.Follow below steps to download support bundle:Generate an API Credential from DataOpsDataOps Platform -> Manage -> API Credentials -> Add New API Credential -> (Note Credential ID and Token)Run the following commands on the engine(SDC or Transformer) (replace CRED_ID and CRED_TOKEN values) where the relevant pipeline has been run.export CRED_ID=<xxxxxxxx>export CRED_TOKEN=<xxxxxxxx>curl -X GET '<URL-of-the-engine>/rest/v1/system/bundle/generate?generators=SdcInfoContentGenerator,PipelineContentGenerator,

6 months ago

DashSenior Technical Evangelist and Developer Advocate at Snowflake

posted in Show us your Pipelines

Orchestrate Initial Bulk Load to Change Data Capture

This pipeline is designed to orchestrate initial bulk load to change data capture workload from on-prem Oracle database to Snowflake Data Cloud. The pipeline takes into consideration the completion status of the initial bulk load job before proceeding to the next (CDC) step and also sends success/failure notifications. The pipeline also takes advantage of the platform’s event framework to automatically stop the pipeline when there’s no more data to be ingested.

6 months ago

ana00159Discovered Fame

asked in Community Articles

Yago AparecidoDiscovered Fame

asked in Community Articles

jvm memory exceed values defined in xmx and xms

Hello streamset community, I configured xmx and xms with 35GB, like this: SDC_JAVA_OPTS="-Xmx35840m -Xms35840m means that 35G has been configured, but querying the service status with "systemctl status sdc" command, the memory value exceeds 37G, as shown below: why does this occur? does the memory value that appears with the "systemctl status sdc" command have a different conversion or does this memory add to another type of memory?

6 months ago

xagavinFan

asked in Community Articles

can not register sdc by error 403 message Forbidden

after install StreamSets Data Collector 3.16.1 , and open it by chrome , i have show register page.when i submit infomation , the page is reload and show Register again , check network, i fount register request get 403 error . how to do ?

7 months ago

sujatha mogiliFan

asked in Community Articles

The streamsets pipeline fails with java.lang.OutOfMemoryError: Java heap space

The streamsets pipeline fails on a regular basis with the below error. Please advise if you ran into this issue and the resolution. We have increased Heap size couple of times but not helping.“ERROR A JVM error occurred while running the pipeline, java.lang.OutOfMemoryError: Java heap space”

7 months ago

gkognoleFan

asked in Show us your Pipelines

Unable to ingest data from Azure SQL (CDC) to Azure Data bricks using Stream Sets.

Trying to build data pipeline for Azure SQL Server DB (CDC) as source and Azure Data bricks (Delta tables) as destinationI have referred data pipeline sample fromhttps://github.com/streamsets/pipeline-library/tree/master/datacollector/sample-pipelines/pipelines/SQLServer%20CDC%20to%20Delta%20Lake Getting below error for few records in Schema preview as-well:DELTA_LAKE_34 - Databricks Delta Lake load request failed: 'DELTA_LAKE_32 - Could not copy staged file 'sdc-4a076fce-7a73-45ba-8dd7-29e58848cf23.csv': java.sql.SQLException: [Simba][SparkJDBCDriver](500051) ERROR processing query/statement. Error Code: 0, SQL state: org.apache.hive.service.cli.HiveSQLException: Error running query: org.apache.spark.sql.AnalysisException: Unable to infer schema for CSV. It must be specified manually. Note : On Preview/Draft Run → Pipeline is able to capture changes from Source DB, successfully created files in stage (ADLS container) and created Delta tables at destination but it it fails to ingest re

7 months ago

RishiStreamSets Employee

posted in Community Articles

How to perform a SAML Trace

Learn to perform a SAML trace and generate SAML Trace JSON file using browser plug-ins. Streamsets support team uses the file to trace the SAML assertions occurring between your identity provider and Control-Hub to troubleshoot your sign-in issues. Use the links below to download and install the SAML tracer plug-in for your browser: Note: The links and steps provided here were correct at time of publishing. Mozilla Firefox: https://addons.mozilla.org/en-US/firefox/addon/saml-tracer/ Google Chrome: https://chrome.google.com/webstore/detail/saml-tracer/mpdajninpobndbfcldcmbpnnbhibjmch When the plug-in is added, Click the newly added SAML tracer icon on the upper-right add-in menu for your respective browser. This opens the SAML Tracer dialog box. The SAML tracer dialog box records and displays details as shown below. Note the occasional SAML tags shown at the right indicating SAML assertions being passed. Re-produce the issue. use the SAML Tracer dialog to navigate to Export tab

7 months ago

sperandioFan

asked in Community Articles

Global variable to reduce memory usage and API calls

I have a pipeline migrating oracle db to salesforce which gets a couple of product attributes from salesforce for all previously migrated products in salesforce. Each record in the pipeline is processing one product and needs to reference this map, which uses local caching to only call salesforce once to build the product map of everything (rather than call for every product being processed - which is tioo slow and uses too many API calls).With this approach, every record gets the map of all products. The approach works with small batch sizes of 50, but runs out of heap memory with batch sizes bigger than that. Is there a way for every record to access this large map of Prouct information loaded from salesforce without duplicating it inside of every record?

7 months ago

RishiStreamSets Employee

posted in Community Articles

java.lang.IllegalStateException: ConfigMap is not a supported kind in the deployment spec

Kubernetes Deployment fails with the below errorAck Error for Event '64627d8f-ccc8-494e-bbd3-e9750f361a68:dpmsupport'. Reason : java.lang.IllegalStateException: ConfigMap is not a supported kind in the deployment specControl-agent (Provisioning agent ) cannot create config-maps or secrets. The only resources control agent can create are deployments, services, ingress and hpa.If you want to use these resources then you need to create this separately and call it in deployment manifest.

7 months ago

iaakashbansalFan

asked in Show us your Pipelines

HTTP Client Origin - Stop Condition for Pagination

I need to extract data from an API using OAuth2 connection. As per the data, they provide /cursor at the end of each page and that cursor can be used to get the records from next page.In Pagination tab, I used Link with Response field and tried to add Stop Condition with /cursor. but, not able to handle this scenario.Can someone please help. Thanks in Advance!!

7 months ago

Page 2 / 5

Badge winners

AkshayJadhavhas earned the badge Eager to help
john.durkinhas earned the badge Eager to help
samhas earned the badge Eager to help
saleempothiwalahas earned the badge Eager to help
Bikramhas earned the badge Community Champion 🏅

Show all badges

Terms & Conditions

Sign up

Already have an account? Login

Social Login

Username *

E-mail address *

What I do... *

Data Leader Data Architect Data Engineer Data Scientist Other

Company *

Country *

Zip Code *

Marketing Communications

Yes No

Password *

I have read and Agree to the Website Terms of Service and I have read and acknowledged the Privacy Policy.

loginBox.register.email_repeat

Login to the community

No account yet? Create an account

Social Login

Username or Email

Password

Remember me

Forgot password?

Enter your username or e-mail address. We'll send you an e-mail with instructions to reset your password.

Username or e-mail

Back to overview

Scanning file for viruses.

Sorry, we're still checking this file's contents to make sure it's safe to download. Please try again in a few minutes.

This file cannot be downloaded

Sorry, our virus scanner detected that this file isn't safe to download.