StreamSets Platform

I am trying to capture change data from SQL Server database for which I have enabled the CDC for the source table in SQL Server database and loading the data to Snowflake destination in StreamSets, I am getting below error while running/previewing the pipeline in StreamSets Control Hub Platform. Please find the below error: SNOWFLAKE_27 - CDC record is missing the ‘PERSONID’ key fieldNote: PERSONID is the Primary key specified both in SQL Server and Snowflake tables I tried with checking and unchecking the “Get Primary Key Information” option and similarly with table auto create option. But still, I am facing the same error. Could anyone please help me fix this issue. Thanks in advance. Happy to share more details if required. 

Hi all,I need to receive a confirmation from Kafka after sending a message to proceed with a different business logic. The Kafka producer is a destination that requires a REST Service to send the response, but my current origin is a Groovy Scripting. I have tried to implement this process with a Groovy Processor Evaluator but the Kafka packages are not available. Does anyone know how to achieve this challenge?Thanks,Edwin

while connecting with apache hudi through pyspark processor facing error ie:- An error occurred while calling o131.save. : org.apache.hudi.exception.HoodieException: hoodie only support org.apache.spark.serializer.KryoSerializer as spark.serializer at org.apache.hudi.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:154) at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:150) at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:46) at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70) at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68) at org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:90) at org.apache.spark.sql.execution.SparkPlan.$anonfun$execute$1(SparkPlan.scala:175) at org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1(SparkPlan.scala:213) at org.apache.sp

(8 Cores x 16 Threads) x 1 CPU = 128 vCPUs

Hi,Currently exploring Kerberos authentication with Streamsets. For HDFS.Could you please provide some sample values that need to be set for below properties.kerberos.client.enabled kerberos.client.principal kerberos.client.keytabGetting below error while starting up sdc when kerberos is enabled -Refreshing Kerberos configurationKey for the principal <sdc/hostname> not available in sdc.keytab                 [Krb5LoginModule] authentication failedUnable to obtain password from user

I need to process a zip file hosted on a website. This zip file has multiple csv files of different formats along with some pdf files.I need to process csv files into different tables in snowflake. how can this request be achieved.

Hi All,i am looking for Fragment which has been used in Academy Course for CDC Pipeline Demo, they have mentioned they will share with us. But i cant find it anywhere. Can someone help me to share that fragment. Regards,Rohit

I would like a job to loop in parallel around a pipeline using data from a control table as the parameters.I know you can create an orchestration with multiple jobs and run in parallel but the issue is that the control table will be updated with more data and hence more job swill need to be added to the orchestration manually which is not ideal.I have thought about looping using pyspark but then the job wont be run in parallel.Is there a way to loop around a pipeline in parallel without creating multiple jobs ? 

Can you please help with how I can address this problem. I am using SDC.Summary: Parent api and for each parent there is a child api which needs to be processed.I have a rest api pipeline which returns thousands of rows and each row has has high level information along with url field which needs to be processed to get additional information. Currently I am processing parent url to snowflake. Manually downloading all url’s to a file and processing the url’s to get additional information to snowflake. Is there a better approach for the process which can be automated.  

HiThere is a processor bundle called "Protector" in streamsets .Has anyone ever worked on it or has knowledge about it.I am trying to use the "Protector : Hash" and "Protector : Custom Mask" processor , but there is not enough documentation about it.If anyone has any idea about these processors kindly let me know.

Hi Guys, I have a requirement where I have thousands of json file url’s in a file(all files are of same format). I need to process the data for every file and load data in to destination. Ex: local_dir/file_list.txthttps://example1.jsonhttps://example2.jsonhttps://example3.jsonhttps://example4.json

I schedule my job during running it shows the  bellow error. how can i fix this.

I am using field type convertor to convert the column into time stamp and write into mysql but it only write 2023 year details how can i write 1970 year ,fix it by any other processor.this is my configration 1970 year data not writable in mysqlin mysql only 2023 data will be write  

   when writing in mysql it through the error below i shown   

Hi, I am new to Streamsets, started using it today.I have a task to create some backup data to hive and hadoop using jdbc multitable consumer.the problem that occurs is that there is a table name with '-' instead of '_' example: call-outcomme_id should the correct one be call_outcomme_id.I have used the field remover to maintain the table name with 'keep listed fields' when I start an error occurs, but when I select 'remove listed fields' the column that I targeted was deleted and it worked.thanks in advance

I have 98 columns/fields and 250,000 rows, every time I run it always error, what i'm doing now is to reduce the :Max Batch Size (Records) = 10Max Clob Size (Characters) = 10Max Blob Size (Bytes) = 10Fetch Size = 10because the default is 1000 and an error will occur. How to handle that problem?

I created a pipeline for primary transformers that would run on an EMR cluster. I used the crucial setup information when configuring the cluster.I'm experiencing the following issue:Transformer's pipeline Engine Transformer 

I have created subscription as suggested in related topic but that subscription is not triggered.  I am able to create incident at pipeline level when pipeline fails, using the notification tab by giving webhook details for incident table.Also, I want some information about job failure in incident table for e.g. Short description like job_name is ‘abc’ failed and their error details. Is this possible using subscription? If yes then please help me on the same.

Issue: In Azure Synapse stage, there might be some errors that initially look like the issue with loading/writing data to Synapse, but it could be caused by connection timeout:AZURE_STORAGE_07 - Could not get a Stage File Writer instance to write the records: 'AZURE_STORAGE_11 - Could not load file to Azure Storage. AZURE_STORAGE_02 - Azure Synapse load request failed: 'AZURE_DATA_WAREHOUSE_09 - Could not merge staged file com.streamsets.pipeline.api.StageException: AZURE_DATA_WAREHOUSE_00 - Could not perform SQL operationTo determine the root cause, please turn on debug log by adding the following to log4j to further investigate the issue:logger.l5.name = com.streamsets.pipeline.stage.common.synapselogger.l5.level = DEBUGlogger.l6.name = com.streamsets.pipeline.stage.destination.datawarehouselogger.l6.level = TRACEYou might be able to get more hints in debug log and see if it’s related to connection timeout, such as:Connection is not available, request timed out after 30000ms. Channel

How to troubleshoot: JDBC Lookup Performance Issues When facing performance issues within the JDBC Lookup stage there are a variety of different variables which may cause or contribute to the issue. Provided below are some initial techniques which you can add to your tool belt to assist you in diagnosing this issue.* To note, while this article is targeted specifically for diagnosing JDBC Lookup issues the same principles of steps 1 & 2 can be used towards JDBC Query Consumers.Step 1: Enabling DEBUG LoggingEnabling DEBUG can often provide a lot of the information you need to determine a next action for investigation. It can provide additional context to error messages which you may have observed previously, but it may also show errors which were previously not available. Additionally, it can also provide information which - while not an obvious error - can provide crucial insight into the pipeline operation such as statistical data. Debug can be configured in Data Collector via the

In comparison with Informatica and SSIS, I am puzzled by the connection management in Streamsets.If I build a pipeline that reads from a MSSQL, and a connection for this server and database is created, I can use it as the source. However, at the beginning, I only want to connect to a DEV instance, and the connection is dedicated to it. After I finish testing and want to get into user acceptance testing (UAT), I can’t use this connection. It feels that my pipeline would have to be modified so that I use a connection that’s dedicated to UAT. But, this isn’t right! The pipeline shouldn’t need to be changed!In Informatica or SSIS, when I need to switch from DEV to UAT, my “pipeline” doesn’t need to be changed. There is always a mechanism that enables the connection to be switched from DEV to UAT, seamlessly.I imagine Steamsets also has a way to enable seamless switching of environments with regard to connections. But, I can’t find it. I would appreciate it very much if someone could share

Hello, I am new to Streamsets. How to install streamsets data collector on server ubuntu 20.04?

Issue: You encounter error HTTP_42 - Failing stage. Connection timeout as per configuration in HTTP Client destination.Analysis: This is related to Maximum Request Time stage setting. Maximum Request Time can be viewed as allocated for the entire batch. While the stage is processing responses sequentially, if at any point the Maximum Request Time is exceeded, then that record and any records after it are hit with an HTTP_42. Please increase Maximum Request Time to a higher value to clear out this error.However, from SDC 5.6.0 and below, we discover that there's a bug which makes Maximum Request Time setting conflicting with One request per Batch in HTTP Client Destination. This issue makes Maximum Request Time configuration not picked up when One request per Batch option is enabled. Solution: This bug will be fixed in SDC 5.7.0 and above.Workaround: 1st workaround: Please disable One request per Batch option. 2nd workaround (only applicable to SDC 5.4.0): Please replace these attached

 Hi Team, I have few queries on Architecture and feature support: When using Streamsets cloud ( SaaS ) , Can i deploy control pane in our network? or Control pane resides in Streamsets boundaries wheres as the processing takes place in Clients AWS account.             For later, i have a followup question, that will client needs to install agent to communicate with control pane or does control pane requires direct access using some kind of cross account role to spin up / manage and spin down resources like EMR etc ? Does any data ( in preview or Debug Mode ) goes back to Control Pane or Streamsets cloud infra? Is CDC supported for MongoDB, DynamoDB, PostgreSQL, AuroraDB ? With Kafka, does it supports Kerberos based authentication and authorisation?Can i replay the data any any point in the pipeline?Does it offers connectivity to on-premise databases over TCPS protocol?Does it offers push based processing for sources like Oracle, SQL-Server, Snowflake? Finally , does Streamsets supports

I have a text file in my “/tmp” directory named “data.txt”Now every time when I run the pipeline, I want to write data to the same file.I was trying to use LocalFS but I couldnt find anyway wherein I could append data to the same file.Can anyone help me out with this?