30-Day Free Trial: It’s Never Been Easier To Get Started With StreamSets
The data pipeline is jdbc → hive_metadata → hadoop file system and hivemetastore. Data moves from oracle database to hadoop file system. Schema is also getting created but there is no data in the tables. Tried changing the number idle time settings to 10 seconds.Any help in this regard will be greatly appreciated.
I have configured streamets transformer larger on aws, i am trying to create pipe line , to get data from s3 source , we are getting an error like expecting file:/// we gave it s3://<bucket name> and also tried s3a://<bucket name>
We are ingesting a table uisng streamsets which has a column containing xml data.we have added an extra stage to create a hash value for all the records coming through.In our target we could see all xml data is getting suffixed with hash value, and seperate hash column.How to get only the xml value in xml column without suffixing hash value?
How we can check the condition in Stream selector for which service is not giving any response because of some exception at service end.If any exception at service end Http client is giving “No output records produced.” message so based on this message how we can further process like if success response comes then we can move to Trash if this message is coming “No output records produced.” we may insert into our sql table. So how we can identify this message in Stream selector or any other way.
I am trying to create a pipeline with origin PostgreSQL CDC client. The connection was not initializing until unless i make the postgre port to public. What is the exact solution for this ? can anyone help on this ?
How does JDBC destination handle fields without matching columns? I am encountering a situation where it appears that the fields without matching columns are ignored, but this is not defined in the documentation and seems counter to what I would expect (an exception complaining about a column not existing in the table). Please provide a deeper description of what is happening here.
Existe um diretório criado no IIS (Windows) e publicado via FTPS.Ao tentar usar este diretório em StreamSets usando o componente SFTP|FTP|FTPS, ele retorna o seguinte erro: "REMOTE_11 - Unable to connect to remote host 'ftps://ftps.hostname.net:921/PLV' with given credentials. Please verify if the host is reachable, and the credentials and other configuration are valid. The logs may have more details. Message: Could not list the contents of "ftps://ftps.hostname.net:921/PLV" because it is not a folder. : conf.remoteConfig.remoteAddress" As credenciais estão OK, pois permite abrir este diretório usando o serviço LFTP no LinuxEstá faltando alguma configuração de StreamSets para corrigir esse problema?
im using oracle as a source and s3 as destination.im ingesting the records from the source and adding table name as a column through expression evalutaor. i want to use this table name and create a folder in s3 dynamically before dropping those records in s3 bucketso the folder name should be created dynamically through streamsets fetching the name of the table. what should be the approach for the same?for ex: if I'm fetching records from table abc , I need to create a folder called “abc” and drop all the records inside that folder.
Can you please help me on below points.How we can get to know how many times StreamSets has retried for failure records ?what is their data ?What value we should give in Base backoff interval field ? and what all settings we have to do. bcz for me i am not seeing the incoming data to streamsets is matching with processed records. the difference between both is keeps on increasing. Can you please suggest something on this.
HI, I have tried copy all the files from one folder to another folder with in same s3 bucket using streamsets job. But I am seeing only 1 or 2 files copied into destination folder compared to source folder(like in source folder if 7 files are there, but in destination folder I am seeing 1 or 2 files are copying). Can any one help me on this issue. ThanksMurali
Lookups (into DeltaTable) giving extremely bad performances (sometime it stays in pre-execution stage forever) when used in Transformer with origin of 1000 records, although, it works decent enough in streaming mode which i guess is due to the lesser number of incoming records.
Hi team,how to add new stages to an existing pipelines using streamsets python SDK?
I’m looking for a tool to help prevent missing job version updates when moving code from one environment to another (development / UAT / production).If we update a pipeline and job in our development environment but move ancillary code to our test environment a week later, I’ve seen it is easy for our team to miss the test environment job update, resulting in wasted testing time. It would be helpful if we could see at a glance the differences in job versions before doing a deployment to help validate that we have the correct list of jobs to update as a part of that deployment.I see the REST API and could put together a script to compare versions, but I was wondering if there is any way we could visually see that in Control Hub, or even a way to build a pipeline or report that could be run to give this information. Thanks in advance!-Spyder
I am trying to create transformer pipeline using python sdk , unable to connect transformer engine.. i am getting two id and url for sch.tramsformers command . Please help me
HI, I have tried copy all the files from one folder to another folder with in same s3 bucket using streamsets job. But I am seeing more files copied into destination folder compared to source folder(like in source folder if 7 files are there, but in destination folder I am seeing more than 7 like … 8 or 10 or 12). But this issue is coming only for first time of the day. If I run same job again for the day I am seeing record count matching between source and destination. Can any one help me on this issue. ThanksMurali
i have declared s3 data format as below but in UI/Control hub data format is showing blank.pipeline_builder = sch.get_pipeline_builder(engine_type='data_collector', engine_url=”XXXX”)s3_origin = pipeline_builder.add_stage('Amazon S3', type='origin')--s3_origin.data_format ='Text'.How to see values allowed for any particular component in SDK.(ex s3_origin.data_format ,s3_origin.delimiter etc..)
How to add multiple origins to an existing pipeline using python SDK
I am installing Control Hub and when I run 01-iitdb.sh, I get error of DbUtils.DBType.POSTGRESSLF4J: Found binding in [jar:file:/opt/streamsets-dpm/app-lib/slf4j-log4j12-1.7. 7.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/opt/streamsets-dpm/server-lib/slf4j-log4j12-1 .7.7.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]Error while running 'BuildSchemaCommand': java.lang.IllegalArgumentException: No enum constant com.streamsets.lib.util.DbUtils.DBType.POSTGRES----------------------------------------java.lang.IllegalArgumentException: No enum constant com.streamsets.lib.util.DbU tils.DBType.POSTGRES at java.lang.Enum.valueOf(Enum.java:238) at com.streamsets.lib.util.DbUtils$DBType.valueOf(DbUtils.java:180)
We are using cron scheduler , for which we have to give the cron expression .As per our requirement pipeline has to run for every 15 minsis the below cron expression is correct for the pipeline to run for every 15 mins.if not please suggest the changes Cron expression used in the pipeline : 0 /15 * * * * * but when i do the test run pipeline it is not working, can anyone explain how cron scheduler works and mentioned above cron expression is correct or not
Hello,We are getting an error in Hadoop FS HDFS destination while writing a file. It works fine when we use the Stage Library version CDH 5.16 but fails if we use any higher version like CDH 6.x or CDP 7.1. What may be causing this if the same setup works in SDC using CDH5.16 stage library. Also it works fine for version CDP 7.1 if we do not use HDFS encrypted directory since it wont call KMS provider.The error in the log states: WARN KMS provider at [http://x.y.z:16000/kms/v1/] threw an IOException:java.io.IOException: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt at org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:487) at org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:788)
When I am using jdbc producer to insert data into Sybase ASE its throwing me an error : JDBC_17 -failed to lookup primary keys for table ‘xxxxxx’ : SQLState: ZZZZZ Error code:2762 Message: The ‘CREATE TABLE’ command is not allowed within a multi-statement transaction in the ‘tempdb’ database. SQLState: 42000 Error code:208 Message:#keys not found.’ Please have a look and advice.
Is it possible to perform a whole file scp using the Data Collector?
Hello everyone, I have a csv file and I want to send data to a REST API.My final data looks like this:I will give you an example of how I want my final result to be.For 3 records in my csv file, every record will look similar with the one above. For 1 record in my csv file, I want to sent firstly a json with serial, interface and userData_dep fields, and then another json with serial, interface and userData_lan fields.To do that I used 2 http client processors, one for the dep and one for the lan. But I saw that the jsons weren’t sent as I want. My pipeline send 3 json with _dep field, and then another 3 json with _lan field.I want to send 1 json with dep field, then 1 json with lan field, then the next json with the dep field, and the next json with the lan field and so on.I want to send them sequentially.Could you help me with some ideas?Thank you!
In order to send the file from source system to SFTP/MFT servers , we need to select the data format as “Whole file” in the origin because the sftp processor only accept the data format as whole file.
Team, We are processing data from the Hadoop Avro file to SQL server DB. Whenever the SS process is running, SQL DB performance is degrading due to Mapper execution. We checked with DBA and they suggested decreasing the Insert connections. We need to decrease the mappers (stable) for this data load. Is there any way to limit the mapper creation/insert operation? Appreciate your help!
Already have an account? Login
No account yet? Create an account
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.
Sorry, we're still checking this file's contents to make sure it's safe to download. Please try again in a few minutes.
Sorry, our virus scanner detected that this file isn't safe to download.
