30-Day Free Trial: It’s Never Been Easier To Get Started With StreamSets
Doing the tutorial, fragments section (https://academy.streamsets.com/learn/courses/6/streamsets-platform-fundamentals/lessons/172/lab-create-a-new-fragment) I’m invite to import a pipeline, but I don’t have the box “What are pipelines?” with the import link. Maybe I closed the box at some stage, but how doI get it back?Screenshot of my Pipelines
The title sums up what I am looking for.The context is that I have a pipeline with a column that is always NULL but the business owner wants it to be populated with an auto incrementing integer to use as a primary key in the Snowflake table destination. I tried permitting the column to accept NULLs in Snowflake while still having it configured to add an integer for each record, but it just uses the NULLs. Taking the field out of the pipeline of course fails due to the missing field.I am hoping to avoid staging this data in another table and then using INSERT to get the auto increment if StreamSets has the capabilities to do this via regex or some other means.Thank you in advance for your time to anyone that reads this.
Is there a way to create a pipeline to successfully call StreamSets API endpoints that does not require the Python SDK for StreamSets to be leveraged? I have tried the REST Service and HTTP Client origins, and even attempted (horribly) a Jython Scripting origin to no avail. I am fairly certain that authentication is the problem since the API works when I use them through the SCH RESTful API section manually, and receive the following as an error when attempting to use a pipeline stage:com.fasterxml.jackson.core.JsonParseException: Unexpected character ('<' (code 60)): expected a valid value (JSON String, Number, Array, Object or token 'null', 'true' or 'false') at [Source: REDACTED (`StreamReadFeature.INCLUDE_SOURCE_IN_LOCATION` disabled); line: 1, column: 2]My ask for a UI-only solution is due to 1) it would allow less experienced admins to troubleshoot without immediately calling on me if problems arise, 2) allow us to build sequences for business- or management-oriented users to
I am familiar with this inquiry from a year ago, but I was hoping to be able to do this via API to avoid additional steps in automating the ETL process of that data being ingested into a database. Is there a means to do this via API that I am just missing? I am familiar with /pipelinestore/rest/v1/pipelines and use that already to get data about pipelines, but see nothing that sticks out to me as fulfilling the desired stage configuration information.Thank you in advance.
I’m following the DataOps Platform Fundamentals course and the Build a Pipeline chapter has you enter “/zomato” as a files directory in the configuration of a Directory source.However, when validating the pipeline I get the error: SPOOLDIR_12-Directory ‘/zomato’ does not exist: conf.spoolDir.Any solution for this?
Hi AllI have a trial account of Stream sets cloud.I am going through the streamsets-platform-fundamentals course.So i am currently trying to get hands-on , on the transformer engine.The Transformer Deployment and engines were installed and running yesterday, i also built 2 pipelines there. then i shut down my lab environment provided by StreamSets.When i login today and start the above lab environment, all the data collector engines start automatically, but the Transformer Engine does not start automatically.There is no option to start the transformer engine either. As you can see from the below screenshot. If i select the Restart Engine or Shut Down Engine i get the below errorEngine cannot be reached. Verify that the engine is running. Because of this now i am not able to proceed further with the course, can someone help me on how to start the transformer engine after restarting my Lab Environment ?
The HTTP Client in my pipeline is not processing all of the input records that it gets. For eg: Input to HTTP client is 1430 records, but the output records processed in the same client are 1360 records only, with 0 error records. Not sure if I am missing any configuration to be added so that I can balance the I/O records, and send the error records to the error stage.
Hi,In my pipeline, I am having a stream selector stage. I want to parameterize it and use the following expression for the condition:${record:value('/rating_text') == '${pipeline_rating_text}'}Here, pipeline_rating_text is my parameter that I have defined for my pipeline. The problem is that when I run the pipeline it does not work. If I use this expression ${record:value('/rating_text') == 'Excellent'}everything will be fine. Can somebody help me, please?
I have passed a parameter to my pipeline which has a default value when I start the pipeline. I need to change it's value in from one of processors. Is it possible to change the runtime parameter? If yes, can you please provide the steps to do that.
HiI am trying to establishing a connection to jdbc:mysql://mysqldb:3306/zomato but it is not working.I tried 2 differents approach:at first, I did not want to use Strigo so I was not limited to the 8 hours. So I run my engines in containers within my local machine. The engines are running fine on the Control Hub and I can create and run some pipelines. However, I would like to use the connection cited in one of the lab with mysqldb and zomato. I am not sure how to “install” the zomato db and reviews table locally? I have a container running mysql but from there i dont know what to. If i try to create the connection using my local engine, i get then the below error.JDBC_00 - Cannot connect to specified database: com.mysql.cj.jdbc.exceptions.CommunicationsException: Communications link failure The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server. I could see a post with the same error message but it does
Hello,I'm currently working on a simple pipeline to ingest kafka messages inside a log file.I'm trying to consume all the data from the beginning of a topic but i'm only getting newer data added to this topic.Once consumed, the previous topic messages are not accessible anymore.I've already test all the different "Auto Offset Reset" properties.Same for simple and multi topic consumers.In the official documentation docs.streamsets.com :auto.commit.interval.ms bootstrap.servers enable.auto.commit group.id max.poll.recordsIf I understand correctly all those parameters are locked so I can't disable the offset management and process all the data from the beginning of a topic.Is there an additionnal Kafka configuration property to use or do I need to configure the topic directly via kafka CLI ??StreamSets Data Collector version : 3.14.0Kafka Consumer version : 2.0.0Regards.
Hello Team,I need to update one job config which is “globalMaxRetries” using rest call.I refer Restful section in Control Hub but that Rest call for “updateJob” requires many other configs also needs to mention in Request Body. could you please assist me with this ?
Problem description: By default https://<control-hub or platform endpoint>:<port-number>/pipelinestore/rest/v1/pipelines returns the pipelines which are committed. Pipeline count which you note in below image is the number committed pipelines. What if we want the pipelines api to return non-committed or draft pipelines too?
We are running the SDC on Linux servers. I noticed that many temporary directories are created under the /tmp directory. I assume that this is the default location for these sub-directories. However, I would like to use a different location for them. For example, maybe I can use /data/tmp?What’s the procedure to specify another location for those temporary sub-directories?Thanks!
How to solve the below error in SDC logs faced by pipeline(s)?JdbcQueryExecutor - Can't execute querynet.snowflake.client.jdbc.SnowflakeSQLException: Execution error in store procedure PROC_AUTO_MERGE:Statement '01b0b234-0504-da19-0032-6b8704c17d96' has locked table 'TEST_TABLE' in transaction 1701430332198000000 and this lock has not yet been released.Your statement '01b0b234-0504-d944-0032-6b8704c16e32' was aborted because the number of waiters for this lock exceeds the 20 statements limit. At Statement.execute, line 155 position 17
Solution.The issue can be handled by Field mapper processor. Below the syntax for the same. ${str:replaceAll(f:name(), '[.\\-/()!&%^$#@]','')} Attached the screen shot for the reference. NB : The issue also can be handled in different ways e;g(Expression Evaluator, scripting processor etc)
I use Streamsets Data Collector to Load data from Stage 2 database tables (JDBC Query Consumer) using a query and Write loaded data to another Stage 2 Database table (JDBC Producer). I use Init Query as below to delete the previous records before loading data. But this does not delete any record from the table. It would be great if someone can help me.
I have a field mapper to remap camel case fields to snake case. This is the message i am trying to save:{ "id": "e35fd529-613f-5e79-036f-f37c2801041d", "batchName": "SIT Axle3+ 9_12_test", "batchSize": 35, "numberOfAudited": 35, "batchType": "TRIP_BASED", "batchMode": "RANDOM", "batchFacility": "71", "batchDirection": "West", "batchPlaza": "", "batchLane": "", "fromTimestamp": 1694538060, "toTimestamp": 1694566800, "createdTimestamp": 1694626323, "createdBy": "mmccammon@etcc.com", "completedTimestamp": 1697039389, "completedBy": "asayed@etcc.com", "auditDetailList": [ { "id": "a4f274d5-8de5-4c54-ae70-b6a498ab4e9e", "auditBatchId": "e35fd529-613f-5e79-036f-f37c2801041d", "eventId": "49dcbe2c-51a8-11ee-9348-3cecefd01508", "eventType": "TRIP", "eventTimeStamp": 1694549374, "auditResult": false, "auditCorrections": [], "auditedTimestamp": 1694635817, "auditedBy": "ppradeep@etcc.co
Running Error pipeline couldn`t stop, when I click the button of force stop, nothing happened, what should i do?
I am trying to create a job instance from the job template. I have passed the runtime parameters to the start job component but get the following error. - Start failed for job template ID: START_JOB_14 - Start failed for job template ID: abc: com.streamsets.pipeline.lib.ControlHubApiException: Unexpected exception from Control Hub: START_JOB_05 - Failed to parse runtime parameters for job ID: abc, error: 400: I have made sure that the parameter name is correct. Please help me with why is it not able to parse the runtime parameters?
Hi Team,I am using below mentioned method to get list of all pipelines.But it is returning all the values like version, commit id.I want only pipeline name.sch = ControlHub(credential_id, token)pipeline_list = sch.pipelines.get_all() I tried sch.pipeline.name but it is not supporting.
I am trying to use a Jython evaluator stage to remove the formatting and code encapsulation to only present the string content to be inserted in the destination stage (Snowflake). I have been searching online for ideas and guidance but have found very little, and most not helpful and/or pertinent to my quandary. An example of the string incoming to the Jython stage:<p style="margin: 0in 0in 0pt;"><span style="color: #000000; font-family: verdana, geneva; font-size: 10pt;">This position is embedded within our Data Analytics practice. Leverage your curiosity and problem-solving skills to explore, discover, and predict patterns contained within data sets for a wide range of government clients. This includes the derivation of clear narratives that help our clients understand their data and how those insights address their research questions.</span></p><p style="margin: 0in 0in 0pt;"> </p><p style="margin: 0in 0in 0pt;"><span style="color: #00000
I am trying to use Oracle Bulkload Origin to read multiple tables from Oracle database. I want to use the tablenames from source to write to destination as files names. In the preview I can see some attributes but when I try to capture them in Expression Evaluator they appear null. Here is my config for expression evaluator Is it something to do with oracle version? or data collector version?I am using Oracle Database 21c Enterprise Edition Release 21.0.0.0.0 - ProductionVersion 21.3.0.0.0SDC version 5.5.0 I can see the jdbc.tables if i use JDBC Multitable Consumer but I want to use Bulkload.
i used expression evaluator to give or insert blank record values into Null Values ,how can i achieve this .this is my incoming data from origin. these type field had Blank ,but i want to feed Null ..i used this expression:${str:isNullOrEmpty(record:value('/TYPE_1099'))?NULL:record:value('/TYPE_1099')} but it fails .
My original data is in the XLSX spreadsheet file format. However, I would like to save it locally as an XLSX file format, but I am currently unable to do so. Is there an alternative processor or method I can use to achieve this desired file format conversion?
Already have an account? Login
No account yet? Create an account
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.
Sorry, we're still checking this file's contents to make sure it's safe to download. Please try again in a few minutes.
Sorry, our virus scanner detected that this file isn't safe to download.
