30-Day Free Trial: It’s Never Been Easier To Get Started With StreamSets

11 months ago

Home
Community overview
StreamSets Platform
Community Articles and Got a Question?

Community Articles and Got a Question?

Can't find what you're looking for? Ask it here or check out the Community articles

1,020 Topics
2,305 Replies

When you subscribe we will email you when there is a new topic in this category

1020 Topics

Most replies

Recently active Most views Newest first

N_aliOpening Band

asked in Community Articles and Got a Question?

Pagination reset in HTTPs client

Is there a way to reset the page number to 0 when there are no more data in the response ? in https client processor ? if i am using the pagination mode : “ByPageNumber”. is there a way to include today’s date as a string value to a query parameter in the http URL ? For Example : https://test?date=”2022-09-30”

1 year ago

PaawanFan

asked in Community Articles and Got a Question?

Array out of bounds exception while doing column mapping in JDBC lookup?

Hi, I’m trying to add all the columns from the result of a jdbc lookup query into one map. I am getting an array out of bounds exception error even though my fields are all maps. Attaching the source data and error screen captures. Could someone please point out where I am going wrong?

1 year ago

Yago AparecidoDiscovered Fame

asked in Community Articles and Got a Question?

mblahayDiscovered Fame

posted in Community Articles and Got a Question?

Providing field names for headerless file origins

I have recently been working with origins what ingest delimited files with no header. As such, the fields come into the pipeline with names that are derived from a sequential number list. Since such field names are useless when dumping records into a database, a rename operation must be performed. I have already stated in another thread that the renaming operation is expensive, and it would be ideal to avoid if possible. For this situation it seems ideal to me to provide a field name list via the origin that could override the numerical field names. In this fashion the correct field names can be assigned to the records as they are being read from the file and avoid the need for a renamer stage.What say others on this? Are there maybe other ways to solve this? Perhaps the resolution is to make the renamer more efficient.

1 year ago

subashiniStreamSets Employee

asked in Community Articles and Got a Question?

StreamSets Hive Metadata issue after upgrading to CDP 7.1.x

Issue: Hive Metadata stage throws error "A managed table's location should be located within managed warehouse root directory or within its database's managedLocationUri" after upgrading to CDP version 7.14 and above. Reason: In CDP7.x or HDP3.x, the database has 2 locations by default(managed and external) and managed tables are always ACID.Whereas HDP-2.6.x or CDH-6.x or lesser, there is no difference in managed/external table location and managed table will not be created as ACID table by default. In CDP7 or HDP3,When user tries to create managed table with custom location, CDP-Hive validates whether the custom location is outside database's managed location and throws above described exception in StreamSets pipeline. StreamSets will not support ACID table read/write. Workaround: Since StreamSets pipeline with Hive Metadata stage does not support Hive Managed table creation, the best solution is to choose the External table option and provide Table path template so t

1 year ago

tushar.walunjFan

asked in Community Articles and Got a Question?

Snowflake executor not working with private_key_file

We are using snowflake executor to truncate the table before loading data into it.We have configured snowflake destination and executor with same credentials by using private key file. If I remove executor, pipeline is able to load data into snowflake but when I am adding snowflake executor, its giving error as “password can’t be empty” for executor.kindly help. Thanks in advance.

1 year ago

RishiStreamSets Employee

asked in Community Articles and Got a Question?

How to enable ZGC for Streamsets Data Collector ?

What is ZGC ?The Z Garbage Collector, also known as ZGC, is a scalable low latency garbage collector designed to meet the following goals:Sub-millisecond max pause times Pause times do not increase with the heap, live-set or root-set size Handle heaps ranging from a 8MB to 16TB in sizeZGC was initially introduced as an experimental feature in JDK 11, and was declared Production Ready in JDK 15.At a glance, ZGC is:Concurrent Region-based Compacting NUMA-aware Using colored pointers Using load barriersAt its core, ZGC is a concurrent garbage collector, meaning all heavy lifting work is done while Java threads continue to execute. This greatly limits the impact garbage collection will have on your application’s response time. With SDC 5.3.x release, Data Collector supports Java 11 and 17 in addition to Java 8. And hence you can user ZGC if you are using JAVA 17 To enable ZGC just add the following jvm argument -XX:+UseZGC and restart the engine. You can verify this by review $SDC_LOG/g

1 year ago

himanshuFan

asked in Community Articles and Got a Question?

Skip or Automate registration on 3.22.3

I am upgrading streamsets data collector from 3.1.3.0 to 3.22.3. Initially it was asking for registration and I added SDC_CONF_http_authentication=form as env variable, it worked with admin/admin creds and did not ask for registration. Now I configured it with ldap authentication and it’s asking for registration again.Is there anyway to skip the registration or automate the registration on first run as I’ll be running this on kubernetes and won’t be able to register manually each time the pod gets deployed.

1 year ago

amusunuruFan

asked in Community Articles and Got a Question?

Intermittent Issue - Server not responding on time seeing Read Timeouts

Hi Team,We have a situation where backend server is not responding on time/network latency happening intermittently and HTTP client processor is stopping the execution and throwing the below error.Looking at the error its coming back with a NULL HTTP status code which is not letting us setup a Pre-Status Action for this scenario. Could you pls suggest us if there is a better way we could handle this error and retry the request back so we don’t loose the data and continue with the process.com.streamsets.pipeline.api.base.OnRecordErrorException: HTTP_03 - Error fetching resource. HTTP-Status: NULL Reason: java.util.concurrent.ExecutionException: javax.ws.rs.ProcessingException: java.net.SocketTimeoutException: Read timed out at com.streamsets.pipeline.stage.processor.http.HttpProcessor.processResponse(HttpProcessor.java:825) at com.streamsets.pipeline.stage.processor.http.HttpProcessor.process(HttpProcessor.java:358) at com.streamsets.pipeline.api.base.SingleLaneProcessor.proces

1 year ago

xagavinFan

asked in Community Articles and Got a Question?

can not register sdc by error 403 message Forbidden

after install StreamSets Data Collector 3.16.1 , and open it by chrome , i have show register page.when i submit infomation , the page is reload and show Register again , check network, i fount register request get 403 error . how to do ?

1 year ago

ravikumarFan

asked in Community Articles and Got a Question?

Comparison of Ab Initio and Data Stage

I am gathering comparisons of different ETL tools(Informatica, DataStage, Ab Initio) with respect to their usability and performance, I have worked on informatica and Ab Initio and with the web help I am able to find the key essential factors and differences between the two, But I am unable to find any useful stuff on DataStage Vs Ab Initio, what I have done is below:

1 year ago

SubhaFan

asked in Community Articles and Got a Question?

How to send Form-data in http client for POST Call

I am new to Stremsets.I am looking to perform a POST call using HTTP client and the POST Body is a form-data with file upload . We are able to achieve this in POSTMAN. Is there a way we can do this in streamsets data controller pipeline.

1 year ago

ravikumarFan

asked in Community Articles and Got a Question?

How to perform a cyberarkpassword lookup for all hosts in an inventory group and write the keys out to individual pem files?

Ansible version 2.7.9I'm writing an ansible playbook to deploy an piece of software to a linux environment. SSH access to these systems is protected by a CPM (Cyberark), used as an ssh key manager.I've got most of the logic figured out, save for one piece. The playbook needs to loop through hosts in an inventory group, lookup the ssh private key in Cyberark for each host and then use each key to ssh into each host in the inventory group to install the software. I'm struggling with how to make that work in ansible.I've read through the add_host and cyberarkpassword documentation, as well about 4 hours worth of searching stackoverflow and blogs, and couldn't find a single example even close to what I'm trying to do.As far as how I think it should work:Using the cyberarkpassword lookup module, loop through hosts in inventory group specified by {{ env }}. Value for this will be passed in through --extra-args. Retrieve the ssh private key for each host. Register the output from the lookup,

1 year ago

deepikaFan

posted in Community Articles and Got a Question?

DevOps Vs SRE

DevOps and site reliability engineering (SRE) are two approaches that enhance the product release cycle through enhanced collaboration, automation, and monitoring. Both approaches utilize automation and collaboration to help teams build resilient and reliable software—but there are fundamental differences in what these approaches offer and how they operate. About HKR TrainingHKR Trainings excel at providing you the best online classes with high quality facilities at a low price without any compromise on quality. What can you expect from us? A dedicated learning platform with 24*7 support, best in class training materials to help you learn advanced techniques and practical knowledge of all IT Technologies. Our courses are specifically curated for both professionals as well as job-seekers. Online classes conducted by the best knowledgeable and certified trainers helps you earn certification at your convenience. Key featuresFlexible TimingsHands On Experience24/7 SupportCertified & In

1 year ago

ramakrishna_raoFan

asked in Community Articles and Got a Question?

Orchestration records schema for start jobs

Can you please share the schema for orchestration record for start jobs, I am not seeing job name in the schema which is required for triggering the downstream jobs. All I can see is the attached image of orch record schema.

1 year ago

ana00159Discovered Fame

asked in Community Articles and Got a Question?

Reverse pivot

How to do reverse pivot in streamsets? My rows are as follows.ABC,X,,,100ABC,,Y,,100ABC,,,Z,100I want it as one rowABC,X,Y,Z,100 How this can be done?

1 year ago

abbeygadFan

asked in Community Articles and Got a Question?

Groovy Script "Origin Stage" to traverse AWS s3 bucket/directory

I use Streamsets and I need a Groovy Script "Origin" that can read aws s3 bucket.Do you know someone who can help with a sample groovy script to iterate a folder and list objects in s3 bucket?So, I only want a access s3 folder and list all filenames, file path, file size. e.gFilesrecord 1: /path to location/File1.csvrecord 2: /path to location/File2.csvrecord 3: /path to location/File3.csv

1 year ago

PradeepStreamSets Employee

posted in Community Articles and Got a Question?

(DataOps) Downloading support bundle when Websocket tunneling is enabled

When Websocket Tunneling is enabled, you won’t be able to download support bundle through dataops platform UI. We are shown below message when we try to access it from Engine > Support Bundle page.The support bundle allows you to generate an archive file with the information required to troubleshoot various issues with the engine. To download a support bundle, you must set up direct engine access and turn off Websocket Tunneling in your Browser Settings.Follow below steps to download support bundle:Generate an API Credential from DataOpsDataOps Platform -> Manage -> API Credentials -> Add New API Credential -> (Note Credential ID and Token)Run the following commands on the engine(SDC or Transformer) (replace CRED_ID and CRED_TOKEN values) where the relevant pipeline has been run.export CRED_ID=<xxxxxxxx>export CRED_TOKEN=<xxxxxxxx>curl -X GET '<URL-of-the-engine>/rest/v1/system/bundle/generate?generators=SdcInfoContentGenerator,PipelineContentGenerator,

1 year ago

BhaskarFan

asked in Community Articles and Got a Question?

Tensor flow model creation and import into streamsets

Hi - Looking to explore the option of Tensor Flow evaluator. Looks like we need to build a model in tensorflow and then import. Can we import the model in Strigo lab ? any detailed steps would help on this.

1 year ago

AkshayJadhavStreamSets Employee

posted in Community Articles and Got a Question?

Connecting to Microsoft SQL Server from SDC v5.1 is failing "unable to find valid certification path to requested target"

Environment:Streamsets Data Collector 5.1. Producer or consumer Library for Microsoft SQL Server. Issue: The pipeline starts failing with following messages when try to fetch or produce the data to Microsoft SQL Server.This happens on data collector is 5.1 but the same pipeline is working fine on 4.x versions.com.microsoft.sqlserver.jdbc.SQLServerException: The driver could not establish a secure connection to SQL Server by using Secure Sockets Layer (SSL) encryption. Error: "sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target". ClientConnectionId:af15a6cf-f28d-4e21-adb3-141826becad0 at com.microsoft.sqlserver.jdbc.SQLServerConnection.terminate(SQLServerConnection.java:3680) ~[mssql-jdbc-10.2.1.jre8.jar:?] at com.microsoft.sqlserver.jdbc.TDSChannel.enableSSL(IOBuffer.java:2113) ~[mssql-jdbc-10.2.1.jre8.jar:?] at com.microsoft.sqlserver.jdbc.SQLServer

1 year ago

PradeepStreamSets Employee

posted in Community Articles and Got a Question?

STARTING to RUNNING state of a pipeline

When a data collector pipeline is in STARTING state it will validate each stage, create required clients and also do initialization.Each stage has init() method which initializes the objects needed for stages to run. Validation runs static operations. For example, running the query given in JDBC origin stages with limited results to validate the query, its schema, offset column etc. Do note that validation is specific to each stage.If you notice any slowness in STARTING to RUNNING stage, it is possible that validation is taking sometime and it usually happens when the datasets in configured query are larger. This is an example of slowness for JDBC or Relational DB stages. If the slowness is unacceptable you might consider disabling the query validation.

1 year ago

RishiStreamSets Employee

posted in Community Articles and Got a Question?

ERROR StageLibraryUtil - Failed to read repository manifest json

During SDC startup by default SDC try connecting to repository manifest URL 2022-09-12T21:38:54,103 [user:] [pipeline:] [runner:] [thread:ManifestFetcher] [stage:]INFO StageLibraryUtil - Reading from Repository Manifest URL: http://archives.streamsets.com/datacollector/5.0.0/tarball/repository.manifest.jsonNow, if your SDC doesn’t have internet connectivity it will failed with the below error, However, SDC will start without any issue 2022-09-12T21:38:56,714 [user:] [pipeline:] [runner:] [thread:ManifestFetcher] [stage:] ERROR StageLibraryUtil - Failed to read repository manifest jsonjavax.ws.rs.ProcessingException: java.net.SocketTimeoutException: connect timed out at org.glassfish.jersey.client.internal.HttpUrlConnector.apply(HttpUrlConnector.java:287) ~[jersey-client-2.25.1.jar:?] at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:252) ~[jersey-client-2.25.1.jar:?] at org.glassfish.jersey.client.JerseyInvocation$1.call(JerseyInvocation.java:

1 year ago

Page 34 / 41

Badge winners

vishwesh.margasahayamhas earned the badge Product expert
ajinkyahas earned the badge Innovator
Sanjeevhas earned the badge Eager to help
AkshayJadhavhas earned the badge Eager to help
john.durkinhas earned the badge Eager to help

Show all badges

Terms & Conditions

Sign up

Already have an account? Login

Social Login

Username *

E-mail address *

What I do... *

Data Leader Data Architect Data Engineer Data Scientist Other

Company *

Country *

Zip Code *

Marketing Communications

Yes No

Password *

I have read and Agree to the Website Terms of Service and I have read and acknowledged the Privacy Policy.

loginBox.register.email_repeat

Login to the community

No account yet? Create an account

Social Login

Username or Email

Password

Remember me

Forgot password?

Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.

Enter your e-mail address

Back to overview

Scanning file for viruses.

Sorry, we're still checking this file's contents to make sure it's safe to download. Please try again in a few minutes.

This file cannot be downloaded

Sorry, our virus scanner detected that this file isn't safe to download.