StreamSets Platform

I am using the python SDK and I when I use the method “.committer” or “.last_modified_by” on a pipeline object, I am expecting to get back an email that I am able to see under “Committed By” on the platform. However, I get back a long string that I can’t make out:sudo code of what I am doing: sch = ControlHub(credential_id="cred_id,token = token)print(sch.pipelines.get(commit_id = commit_id).last_modified_by) and instead of getting a string of an email returned, I am getting something like this returned:a83b3195-61f4-11ee-a6e4-…..@b5f95dcd-5f00-11ec-b405-…… Would you happen to know how I can turn this string into a valid email? I couldn’t find anything on this topic in the docs.  

IssueA Kafka stage fails to connect to the broker due to the follow error stack trace:INFO ConsumerCoordinator - (Re-)joining groupINFO NetworkClient - Disconnecting from node xxx due to socket connection setup timeout. The timeout value is 8424 ms.INFO ConsumerCoordinator - Group coordinator is unavailable or invalid due to cause: null. isDisconnected: true. Rediscovery will be attempted.2023-09-27 13:07:58,776 INFO ConsumerCoordinator - Request joining group due to: rebalance failed due to 'null' (DisconnectException)BackgroundThis specific Kafka Client exception appears when the Kafka Client’s socket-connection-setup-timeout-max-ms is set too low, causing it to attempt to re-join the group.The maximum amount of time the client will wait for the socket connection to be established. The connection setup timeout will increase exponentially for each consecutive connection failure up to this maximum. SolutionWhen this property is set too low, the Kafka Client will time out during the

Can we configure multiple tables in same pipeline when we use JDBC Query consumer 1 as origin and JDBC producer 1 as destination. in my case i need to send 20 tables data to send from on-premise DB (db2) to cloud DB (Azure postgresql)?. i am thinking to avoid multiple pipelines creation in streamsets.

We are facing some connection issue from STF side while executing a test code, that might be regarding the organization ID not sure why this issue is occurring it saying for organization the url STF  calling is “api/security/v3/orgs/e4cbaa88-611a-11ed-b310-c96332d26f9a” but url must be ”http://api/security/v3/orgs/e4cbaa88-611a-11ed-b310-c96332d26f9a?”. I’m not sure how to fix this Error. E           requests.exceptions.MissingSchema: Invalid URL 

Error: If your Snowflake account name contains underscores and you're using Snowflake JDBC driver version 3.13.25 or higher, you might encounter the following error message:SNOWFLAKE_11 error. This error message indicates a communication problem with the message: "JDBC driver encountered a communication error. Message: Exception encountered for HTTP request: my-account.us-west-2.privatelink.snowflakecomputing.com: Name or service not known."Solution: To prevent this error, explicitly set allowUnderscoresInHost to 'true' in your JDBC connection string or properties. Refer to the Snowflake Community article for additional guidance.Affected Environments: This issue impacts environments meeting the following criteria:Snowflake account name contains underscores. JDBC is used for private link connections via the 'classic' regioned URL (e.g., https://my_account.us-west-2.privatelink.snowflakecomputing.com). Upgrading Snowflake JDBC driver from version 3.13.24 or lower to 3.13.25 or higher.

I am reading a file from s3 and after doing some manipulation in file i wanted to save it different location in s3 but with same name .how can i do this?please help. 

QuestionThere are rpm packages for Red Hat 6 and 7, but not for Red Hat 8? AnswerStreamSets has tested the RHEL 7 package on 8 and certified it for use. There are no plans to release a separate rpm package specifically labeled for RHEL 8.

Hi All,we are new in streamsets !!In our usecase we are uploading file in s3 which predefined name as abcd_sggshs_31-04-2018.csvwe need to capture this above parameter which are in our file name,we are generating events after uploading file in s3,using expression evaluator we are trying to capture this parameter in json file in s3.but we are not getting desired output.below are the expression we are using please suggest the correct expression to get the file name.   

The offset is always stored in the Control Hub (SCH). The Data Collector (SDC) stores the offset to the offset.json file and sends the information to the SCH periodically. When 'Enable Failover' is checked, the offset is migratable i.e the SCH can start a job on a different SDC (SDC_2) with the last-saved offset (from the previous SDC - SDC_1, which is not available anymore). If 'Enable Failover' is not checked, the offset is tied to a specific SDC ID and it does not migrate i.e the SCH cannot start a job with the last-saved offset from another SDC (SDC_1) on a different SDC (SDC_2). On a new SDC, the job will run the pipeline from the beginning. Failover needs to be enabled for the offset to be migrated.

I am reading an avro file from a custom cloud similar to S3 and trying to conver it into parquet file using whole file evaluator but its giving an error  Record1-Error Record1 CONVERT_01 - Failed to validate record is a whole file data format : java.lang.IllegalArgumentException: Record does not contain the mandatory fields /fileRef,/fileInfo,/fileInfo/size for Whole File Format. : (View Stack Trace...   

Here we will be covering how to convert an AVRO file with Schema, to a Parquet file.STEP 1Set your pipeline up as follows;Directory - Whole File Transformer - Local FS STEP 2Configure your Directory Origin as follows; GENERAL TAB FILES TAB Please ensure you make the amendments necessary to certain fields to suit your own set up, e.g. File Name Pattern. Keep the rest as in the screenshot. POST PROCESSING TAB DATA FORMAT TAB STEP 3Configure Whole File Transformer stage as follows; GENERAL TAB JOB TAB*amend file directory as necessary AVRO TO PARQUET TAB If you preview the pipeline, ensure to select the following boxes, which will allow you to click in to each record and schema;  You should now have a .parquet file from your original AVRO.

When I am testing the Produce Events of a component. When I uncheck the "Produce Events" box, the component configuration displays normally. However, when I check it, I always encounter the following error. Do you know what might be causing this? Has anyone else experienced this issue?  

StreamSets Support team would like to inform you of an issue where StreamSets Data Collector pipelines using destinations such as Azure Synapse, Delta Lake, Google BigQuery, or Snowflake might corrupt the data for the DATE/DATETIME/TIME fields.Products Affected: StreamSets Data Collector Releases Affected:  StreamSets Data Collector 5.6.0 and prior versions Users Affected: StreamSets Data Collector pipelines using Azure Synapse destination, Delta Lake destination, Google BigQuery destination, and Snowflake destination. Severity: High Description: It has been identified that in some cases, DATE/DATETIME/TIME datatype fields could be corrupted. Users could be finding that their DATE/DATETIME/TIME data fields contain incorrect dates. This issue happens only if all the following conditions take place:The pipeline writes to one of the following stages: Azure Synapse destination Delta Lake destination Google BigQuery destination Snowflake destination.  The pipeline writes to multiple tab

StreamSets Support Team would like to inform you that starting with Data Collector 5.6.0 pipelines with Amazon S3 stages fail to execute if the "Bucket" or "Bucket and path" property contains a forward slash (/). Products Affected: StreamSets Data Collector Releases Affected: Starting from StreamSets Data Collector version 5.6.0 Users Affected: StreamSets Data Collector pipelines that include the forward slash (/) in the bucket property of Amazon S3 stages. Severity: Medium/High Description: Starting with version 5.6.0, you can no longer include the forward slash (/) in the following properties for Amazon S3 stages due to an Amazon Web Services SDK upgrade:Bucket property for the Amazon S3 origin Bucket and path property for the Amazon S3 destination and executorAdditional details regarding this change can be found in the Upgrade Impact documentation.Immediate action required: YesResolution: Avoid using the forward slash (/) in the bucket name as this may cause signature mismatch error

StreamSets Support Team would like to inform you of an issue where StreamSets Data Collector pipelines using Snowflake stages might have problems with threads not being released.  Products Affected: StreamSets Data Collector Releases Affected: StreamSets Data Collector version 5.6.0 and 5.6.1 Users Affected: StreamSets Data Collector pipelines using Snowflake stages. Severity: High Description: A minor thread leak has been detected in all the Snowflake stages. If a user environment involves a substantial number of Snowflake pipelines, users may observe this thread leak behavior, which can impact the overall system performance and resource utilization.Immediate action required: Yes, upgrade to SDC 5.6.3 or 5.7.X.Workaround:  NoneResolution: Fix for this issue has been included in SDC 5.6.3 and 5.7.x release. For your questions, please contact our Support Team via our Support portal or at support@streamsets.com.

When trying to connect to Snowflake in SDC, you may encounter the following error:SNOWFLAKE_11 - Could not create SQL DataSource: com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: JDBC driver encountered communication error. Message: Exception encountered for HTTP request: Remote host terminated the handshake.This error may be caused by some configuration mistakes in the stage. One possible cause is underscore(s) in the account name. The Snowflake JDBC driver does not support the use of underscores in URLs, which include the account name. If your account name does contain underscores, and you are receiving a similar error, a workaround would be to use a hyphen instead.

while running python program using streamsets python sdk we are getting error File "/var/lib/jenkins/.local/lib/python3.8/site-packages/streamsets/sdk/exceptions.py", line 5, in <module> from requests.exceptions import HTTPError, JSONDecodeError ImportError: cannot import name 'JSONDecodeError' from 'requests.exceptions'we have found solution for this but we want to know why we are getting this error.Solution for this is manually we are replacing line “from requests.exceptions import HTTPError, JSONDecodeError”   with “from json import JSONDecodeError”

Can we configure multiple tables in same pipeline when we use JDBC Query consumer 1 as origin and JDBC producer 1 as destination. in my case i need to send 20 tables data to send from on-premise DB (db2) to cloud DB (Azure postgresql)?. i am thinking to avoid multiple pipelines creation in streamsets.

Issue: Pipeline can run successfully but Pipeline Finisher stage shows error “Unsatisfied precondition(s) '${record:eventType() == 'no-more-data'}'”Resolution:This error is due to the configuration in Pipeline Finisher stage, On Record Error tab as Send to Error. Therefore, this error COMMON_0001 - Stage precondition: CONTAINER_0051 - Unsatisfied precondition(s) '${record:eventType() == 'no-more-data'}' is an expected behavior. In this case, the pipeline would still end when it receives no-more-data event, but it would keep generating these error records until no-more-data condition is met.To get rid of this error, you can change in On Record Error tab from Send to Error to Discard. You can read more about this configuration in our documentation: Stopping a Pipeline After Processing All Available Data. 

Problem:When attempting to log in to Legacy Control Hub with SAML enabled, users encounter an HTTP 401 error after trying to authenticate with the Identity Provider. The Login Audit trail provides the following information:HTTP Status Code: 401 Error Type: WEB_SSO_204 Response Status: urn:oasis:names:tc:SAML:2.0:status:RequesterSolution:The above error basically means that the Identity Provider has not granted authorization to the user, resulting in a 401 error response to Legacy Control Hub.This issue may stem from a misconfiguration of SAML settings between Legacy Control Hub and the Identity Provider. In most cases, the problem is related to the SAML keys used by Control Hub. Ensure that the public keys are valid and that the Identity Provider is utilizing the same keys. We recommend exchanging fresh copies of metadata and public keys between Legacy Control Hub and the Identity Provider. Afterward, clear your browser's cache and attempt to log in to Control Hub again. If you encount

Environment:  Data Collector. Destination Databricks Delta Lake. Issue:  The pipeline is encountering a "DELTA_PATH_DOES_NOT_EXIST" error, signaling that the specified path does not exist. To resolve this, we have manually created the file in Databricks.DELTA_LAKE_37 - Runnable process batch failed: DELTA_LAKE_12 - Could not perform SQL operation: java.sql.SQLException: [Databricks][DatabricksJDBCDriver](500051) ERROR processing query/statement. Error Code: 0, SQL state: org.apache.hive.service.cli.HiveSQLException: Error running query: [DELTA_PATH_DOES_NOT_EXIST] org.apache.spark.sql.AnalysisException: dbfs:/dash/DeltaLake/TEST doesn't exist at org.apache.spark.sql.hive.thriftserver.HiveThriftServerErrors$.runningQueryError(HiveThriftServerErrors.scala:48) at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.$anonfun$execute$1(SparkExecuteStatementOperation.scala:694) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) at com.databricks.unit

In this article, we will explore the process of creating a pipeline to transfer data from a MySQL database to a Delta Lake tables on Databricks.Environment:Data Collector on DataOps platform. Destination Databricks Delta Lake.These are the steps we need to follow to develop the pipeline: Login to Databricks UI » Compute » Create new compute then fill out all the details such as spark version, type of cluster, worker type. Once everything is finalised, click on create compute.Creating compute will take some sweet time.Now, we will have to fetch the following information from the cluster and populate into the pipeline configuration section. Token: In your Databricks workspace, click your Databricks username in the top bar, and then select User Settings from the dropdown » Develop » Token » Generate a new token. JDBC String: Click on the Databricks newly created cluster, scroll down and expand Advance Option tab » JDBC/ODBC. For staging purpose, we need to use AWS S3, Google Storage or

Upgrading one or both of the backend databases for SCH 3.x (relational database and timeseries database) has no effect on SCH itself assuming that you are upgrading to a supported version of the database. You will need to either bring down SCH during the upgrade, or put the database into read-only mode, and of course take a backup of the database before beginning the upgrade.In a scenario where a subsequent rollback of the database to the backed-up version is contemplated, be aware that there will be some data loss, even if there were no new jobs created or other changes made (code freeze). Some of this data includes, but is not limited to:Any clean up of the DB Writing to InfluxDB Writing to the PostgreSQL Timseries DB (job metrics that don’t go to InfluxDB but to the SQL DB) Updating jobs statuses Running scheduler tasksThese data would need to be manually reconstructed from the upgraded database and backported to the rolled-back database.If you have any additional questions, please

In SDC versions up to 5.5.x, you may see CONTROL_HUB_API_00 errors which can cause pipelines and jobs to fail. There is no context in the logs as to what was happening that led up the to the error; it’s just a generic error that occurs when any type of API call to SCH fails.In SDC 5.6.x, this error was removed and more helpful logging was added that captures the specific API call that was being made and what the exception was. If you are encountering the CONTROL_HUB_API_00 error in an older version of SDC, StreamSets Support recommends either upgrading to at least 5.6.x, or install SDC 5.6.x or higher in a non-production environment and reproduce the issue. This will allow StreamSets Support to more effectively assist you and find the root cause of the issue.

IssueAfter upgrading MapR client libraries or applying a MapR client patch on an SDC node, pipelines using Hadoop stages fail with some Provider could not be instantiated error or ClassNotFoundException at startup, indicating some Hadoop class cannot be found within the Java classpath.BackgroundDue to licensing restrictions, StreamSets cannot distribute MapR libraries with Data Collector. As a result, you must perform additional steps to enable the Data Collector machine to connect to MapR.When the bin/streamsets setup-mapr prerequisite is completed, SDC creates symlinks to the installed MapR Client libraries on the system in the streamsets-libs/streamsets-datacollector-mapr-version-lib/lib directory.When applying a MapR Client patch or upgrading the MapR Client package on the SDC machine, some JAR files in the MapR client installation patch will be renamed which will cause broken symlinks within SDC’s MapR stage library directory.SolutionThe Provider could not be instantiated exceptio