Hi, I am new to Streamsets, started using it today.
I have a task to create some backup data to hive and hadoop using jdbc multitable consumer.
the problem that occurs is that there is a table name with '-' instead of '_' example: call-outcomme_id should the correct one be call_outcomme_id.
I have used the field remover to maintain the table name with 'keep listed fields' when I start an error occurs, but when I select 'remove listed fields' the column that I targeted was deleted and it worked.
thanks in advance
Page 1 / 1
@realmatcha
kindly provide error details, so i can help you on it.
com.streamsets.pipeline.stage.lib.hive.exceptions.HiveStageCheckedException: HIVE_30 - Invalid column name call-outcomme_id
@realmatcha
The error seems like the column name is not matching with your destination table name.
Kindly try to rename the column name by using “Field Rename” processor and check if it avoid the issue.
but it gives syntax error. can you please provide and example how to do proper "Field Rename"
@realmatcha
Please try like below .
I've tried but it's still an error, you can see what each processor contains
@realmatcha
May know if all row contains the column name or not , if it contains then it will rename it else throw error.
You can add a check that if call-outcomme_id is not null then rename it else no need to do anything.
The error is because of the column name is missing for many rows and we are trying rename it .
@realmatcha
You can add filter like below based on the field and it will resolve your issues.
@realmatcha
I can see that in your rename, you are using /’call-comme_id’
Remove the quotes and try.
Remove the quotes for source and target both.
can you please screenshot what is filled in each processor?
i have replaced it but still error
you will have to select each processor and show the properties and screenshot
here`s each one of the processor details
i have replaced it but still error
use quotation after a slash
example: /’call_outcome_id’
@realmatcha
we need quote for the fields but you need to filter out the fields for which you are trying to rename if the field name exists in your record else no need to anything.
I will provide you the work around for your case . You just try that and i hope it will help.
@realmatcha
Please find attached the pipeline and try to implement in your pipeline and i hope it will solve your issues without any error.
cant install library. please help me
@realmatcha
If you are having problem in after importing my pipeline then you just use the processors has been used in my pipeline in your env.
Please follow below steps and i hope it will help. Let me know in case of any issues.
step1 :
Origin as Dev Raw data source and inputs are
{ "call-comme_id": "abc"
} { "call_comme_id": "xyz"
} { "call-comme-id": "ABC"
}
Step2 : Use the filed mapper processor to rename the fields.
Your pipeline should look like below .
I've followed and it's running, but I can't see a result like this
@realmatcha
You can see the above result if you preview your pipeline.
Just preview and see the result. :)
@Bikram@saleempothiwala@tamilarasup thank you all for helping a lot, everything is done and running well