- - Knowledge base
Product Updates
Groups
Join Build It With Brenna

Build it with Brenna: Transformer for Snowflake

(Wed, 31 May, 17:00)

Home
Community overview
Platform Q&A

Platform Q&A

Get inspired and gain all the knowledge you need

775 Topics

Recently active

Most replies Most views Newest first

rohit6488Fan

asked in StreamSets Academy

Reading avro file from a bucket similar to S3 and trying to convert into parquet

I am reading an avro file from a custom cloud similar to S3 and trying to conver it into parquet file using whole file evaluator but its giving an error Record1-Error Record1 CONVERT_01 - Failed to validate record is a whole file data format : java.lang.IllegalArgumentException: Record does not contain the mandatory fields /fileRef,/fileInfo,/fileInfo/size for Whole File Format. : (View Stack Trace...

2 days ago

realmatchaOpening Band

asked in Got a Question?

JDBC Multitable Consumer error

Hi, I am new to Streamsets, started using it today.I have a task to create some backup data to hive and hadoop using jdbc multitable consumer.the problem that occurs is that there is a table name with '-' instead of '_' example: call-outcomme_id should the correct one be call_outcomme_id.I have used the field remover to maintain the table name with 'keep listed fields' when I start an error occurs, but when I select 'remove listed fields' the column that I targeted was deleted and it worked.thanks in advance

2 days ago

realmatchaOpening Band

asked in Got a Question?

Error : Java heap Space

I have 98 columns/fields and 250,000 rows, every time I run it always error, what i'm doing now is to reduce the :Max Batch Size (Records) = 10Max Clob Size (Characters) = 10Max Blob Size (Bytes) = 10Fetch Size = 10because the default is 1000 and an error will occur. How to handle that problem?

2 days ago

akshayp2Fan

asked in StreamSets Academy

Subscriptions created for create incident for failed job in ServiceNow has not triggered.

I have created subscription as suggested in related topic but that subscription is not triggered. I am able to create incident at pipeline level when pipeline fails, using the notification tab by giving webhook details for incident table.Also, I want some information about job failure in incident table for e.g. Short description like job_name is ‘abc’ failed and their error details. Is this possible using subscription? If yes then please help me on the same.

3 days ago

LavenderStreamSets Employee

posted in Share your Best Practices

How to troubleshoot connection timeout related issues in Azure Synapse stage

Issue: In Azure Synapse stage, there might be some errors that initially look like the issue with loading/writing data to Synapse, but it could be caused by connection timeout:AZURE_STORAGE_07 - Could not get a Stage File Writer instance to write the records: 'AZURE_STORAGE_11 - Could not load file to Azure Storage. AZURE_STORAGE_02 - Azure Synapse load request failed: 'AZURE_DATA_WAREHOUSE_09 - Could not merge staged file com.streamsets.pipeline.api.StageException: AZURE_DATA_WAREHOUSE_00 - Could not perform SQL operationTo determine the root cause, please turn on debug log by adding the following to log4j to further investigate the issue:logger.l5.name = com.streamsets.pipeline.stage.common.synapselogger.l5.level = DEBUGlogger.l6.name = com.streamsets.pipeline.stage.destination.datawarehouselogger.l6.level = TRACEYou might be able to get more hints in debug log and see if it’s related to connection timeout, such as:Connection is not available, request timed out after 30000ms. Channel

3 days ago

jyuanOpening Band

asked in Got a Question?

Connection management - is it possible to switch environments seamlessly?

In comparison with Informatica and SSIS, I am puzzled by the connection management in Streamsets.If I build a pipeline that reads from a MSSQL, and a connection for this server and database is created, I can use it as the source. However, at the beginning, I only want to connect to a DEV instance, and the connection is dedicated to it. After I finish testing and want to get into user acceptance testing (UAT), I can’t use this connection. It feels that my pipeline would have to be modified so that I use a connection that’s dedicated to UAT. But, this isn’t right! The pipeline shouldn’t need to be changed!In Informatica or SSIS, when I need to switch from DEV to UAT, my “pipeline” doesn’t need to be changed. There is always a mechanism that enables the connection to be switched from DEV to UAT, seamlessly.I imagine Steamsets also has a way to enable seamless switching of environments with regard to connections. But, I can’t find it. I would appreciate it very much if someone could share

4 days ago

Mihir KaleFan

asked in Got a Question?

Incorrect job metrics in orchestration pipeline

I want to evaluate the job metrics field of orchestrator tasks JSON, which is the output of start job origin in orchestration pipeline. The start job origin contains pipeline which moves 5 records from csv file from directory to snowflake table.I need to confirm from the job metrics field that the pipeline output count equals that of input count.But when I preview the orchestration pipeline, both the input and output records count show 0.Is there any additional configuration required that I have missed? I have attached the preview SS along with the job instance pipeline.

4 days ago

lakshmi_narayanan_tDiscovered Fame

asked in Got a Question?

cron expression how to write in scheduler

i want to schedule my job in monday to friday and the time is 10 am to 6 pm IST and also every 2 minutes , how to schedule ,anyone suggest.

5 days ago

realmatchaOpening Band

asked in Got a Question?

Install Streamsets on ubuntu 20.04

Hello, I am new to Streamsets. How to install streamsets data collector on server ubuntu 20.04?

5 days ago

LavenderStreamSets Employee

posted in Share your Best Practices

Error HTTP_42 - Failing stage. Connection timeout as per configuration

Issue: Error HTTP_42 - Failing stage. Connection timeout as per configuration in HTTP Client processor/destination.Solution: We recommend to increase the "Max Request Timeout" configuration in the HTTP stage. (Default is 60 seconds).If increasing "Max Request Timeout" doesn’t solve the error, please perform networking testings such as generating tcpdump to investigate your networking environment and open a ticket with Support for further troubleshooting.

8 days ago

dvmishraFan

asked in Got a Question?

Cloud Architecture

Hi Team, I have few queries on Architecture and feature support: When using Streamsets cloud ( SaaS ) , Can i deploy control pane in our network? or Control pane resides in Streamsets boundaries wheres as the processing takes place in Clients AWS account. For later, i have a followup question, that will client needs to install agent to communicate with control pane or does control pane requires direct access using some kind of cross account role to spin up / manage and spin down resources like EMR etc ? Does any data ( in preview or Debug Mode ) goes back to Control Pane or Streamsets cloud infra? Is CDC supported for MongoDB, DynamoDB, PostgreSQL, AuroraDB ? With Kafka, does it supports Kerberos based authentication and authorisation?Can i replay the data any any point in the pipeline?Does it offers connectivity to on-premise databases over TCPS protocol?Does it offers push based processing for sources like Oracle, SQL-Server, Snowflake? Finally , does Streamsets supports

9 days ago

lex03Fan

asked in Got a Question?

Write to a already existing text file in LocalFS

I have a text file in my “/tmp” directory named “data.txt”Now every time when I run the pipeline, I want to write data to the same file.I was trying to use LocalFS but I couldnt find anyway wherein I could append data to the same file.Can anyone help me out with this?

9 days ago

aditya_ghongadeFan

asked in Got a Question?

Connecting to Streamsets OPC UA

Hi,I was trying to connect to OPC UA client via different OPC UA free servers like Ignition, Prosys OPC UA Simulation Server, Integration Objects OPC UA Server Simulator but was unable to connect to it. The Streamsets OPC Client was refusing to connect to it. However, I tried connecting Integration objects OPC client with its server and it was able to connect. Can someone guide me through the steps to connect to OPC UA Client and also mention any OPC UA servers.

9 days ago

Kartik MedavarapuRoadie

asked in Got a Question?

DataOps Platform - REST API Origin - How to invoke pipeline?

Hi All,I created a microservice pipeline with REST API Origin using the DataOps platform as per video: https://www.youtube.com/watch?v=wIZWMV1bMl4Now, I am unable to invoke the pipeline using 3rd party tools like Postman. I have chosen default settings in SDC and headers as (X-SDC-APPLICATION-ID:sdc_microservice & Content-Type:application/json) yet I’m facing issue as “Error: connect ETIMEDOUT xx.xx.xx.xx:8000”.However, when I use the same URL in a curl command where SDC container is running, I see, 200 OK response. curl -i -X GET http://xx.xx.xx.xx:8000/rest/v1/user --header "X-SDC-APPLICATION-ID:sdc_microservice"There is no connectivity issue. So, can anyone tell me why I am unable to use Postman tool or if I am missing anything here?Below are the images showing the same:

10 days ago

saikat_chatterjeeDiscovered Fame

asked in Got a Question?

Facing error in On-prem setup of sch

I was trying to do the on-prem setup of SCH in which I have completed till configuring the control hub, relational databases, system data collector etc.Now when I am going to create the required tables for each databases using “sudo dev/01-initdb.sh” command the tables are getting created for the first database i.e. “Security” and it is throwing error in the next database i.e. “PipelineStore”From the marked red arrow the error is staring.Can someone help me regarding this?

10 days ago

paulStreamSets Employee

posted in Got a Question?

Details on SDC Metrics

QuestionHow are the SDC Metrics collected and what do they represent?AnswerThe graphs on the SDC Metrics page in StreamSets Data Collector and the Metrics tab of the Execution Engine page in Control Hub are system-level metrics collected from standard Java core libraries. They should roughly correspond with metrics reported by other system-level tools, such as top and uptime at the command line, as well as external monitoring tools that show system-level metrics. Note that the numbers reported by different tools won’t match each other exactly due to differences in reporting intervals and other factors, but there should be a correlation.

11 days ago

HaripriyaFan

asked in Got a Question?

Oracle CDC - Not fetching any data due to missing Redo Log files for Time Range

We are using Streamsets CDC stage for fetching Oracle CDC Data. We are received error like, Missing redo log files for time range and pipeline is not fetching any data for long.Question:What is the reason for receiving missing redo log error? What step need to take when we receive this kind of error?

12 days ago

southwr62Fan

asked in Got a Question?

How to create and use a variable

Hi, I am new to Streamsets, started using it today.I have a task to create some entries in a big query table, that represent the number of records in returned by an API call.I can do this using power automate, without issue, but power automate doesn’t support posting the BigQuery.roughly speaking the process I have in power automate is….Http get array is returned get length of the array post the result to DB table I need to use Streamsets and BQ as these have been specified as a requirement.thanks in advance

13 days ago

kushal_upadhyayFan

asked in Got a Question?

Masking

Need to mask the fields using fixed length,custom and regular expression….Suppose I have given the field_to_mask parameter in the pipeline, how can I mask it ...For e.g in order to do the masking of fixed length the field to mask string is like this in the picture but its giving syntax error. Can you please provide and example how to do the masking in fixed,custom and regular expressionCan you please provide the examples of custom and regular expression

13 days ago

meowwFan

asked in Got a Question?

Error for Kafka Connection using aws secret manager as credential store

if i create a kafka connection using simple string then it works.. but if i create connection with these pramaters..${str:concat(str:concat(str:concat("org.apache.kafka.common.security.scram.ScramLoginModule required username='", credential:get("aws", "all@myorg.org", "MyStreamsetsSecret&kafka_username")), str:concat("' password='", credential:get("aws", "all@myorg.org", "MyStreamsetsSecret&kafka_password"))), "';")} then i am getting an below error if i test a connection KAFKA_67 - Error connecting to Kafka Brokers 'org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient'

17 days ago

Page 1 / 31

Badge winners

AkshayJadhavhas earned the badge Eager to help
john.durkinhas earned the badge Eager to help
samhas earned the badge Eager to help
saleempothiwalahas earned the badge Eager to help
Bikramhas earned the badge Community Champion 🏅

Show all badges

Terms & Conditions

Sign up

Already have an account? Login

Social Login

Username *

E-mail address *

What I do... *

Data Leader Data Architect Data Engineer Data Scientist Other

Company *

Country *

Zip Code *

Marketing Communications

Yes No

Password *

I have read and Agree to the Website Terms of Service and I have read and acknowledged the Privacy Policy.

loginBox.register.email_repeat

Login to the community

No account yet? Create an account

Social Login

Username or Email

Password

Remember me

Forgot password?

Enter your username or e-mail address. We'll send you an e-mail with instructions to reset your password.

Username or e-mail

Back to overview

Scanning file for viruses.

Sorry, we're still checking this file's contents to make sure it's safe to download. Please try again in a few minutes.

This file cannot be downloaded

Sorry, our virus scanner detected that this file isn't safe to download.