Solved

JDBC Multitable Consumer error


Hi, I am new to Streamsets, started using it today.

I have a task to create some backup data to hive and hadoop using jdbc multitable consumer.

the problem that occurs is that there is a table name with '-' instead of '_' example: call-outcomme_id should the correct one be call_outcomme_id.

I have used the field remover to maintain the table name with 'keep listed fields' when I start an error occurs, but when I select 'remove listed fields' the column that I targeted was deleted and it worked.

thanks in advance

icon

Best answer by realmatcha 25 May 2023, 09:27

View original

23 replies

@Bikram @saleempothiwala @tamilarasup thank you all for helping a lot, everything is done and running well

Userlevel 5
Badge +1

@realmatcha 

You can see the above result if you preview your pipeline.

Just preview and see the result. :)

I've followed and it's running, but I can't see a result like this

 

 

Userlevel 5
Badge +1

@realmatcha 

If you are having problem in after importing my pipeline then you just use the processors has been used in my pipeline in your env.

 

Please follow below steps and i hope it will help. Let me know in case of any issues.

step1 : 

Origin as Dev Raw data source and inputs are 

 

{
  "call-comme_id": "abc"
  
}
{
  "call_comme_id": "xyz"
  
}
{
  "call-comme-id": "ABC"
  
}

Step2 : Use the filed mapper processor to rename the fields.

 

 

 

Your pipeline should look like below .

 

 

 

cant install library. please help me

Userlevel 5
Badge +1

 

Userlevel 5
Badge +1

@realmatcha 

 

Please find attached the pipeline and try to implement in your pipeline and i hope it will solve your issues without any error.

 

 

Userlevel 5
Badge +1

@realmatcha 

 

we need quote for the fields but you need to filter out the fields for which you are trying to rename if the field name exists in your record else no need to anything.

I will provide you the work around for your case . You just try that and i hope it will help.

i have replaced it but still error

use quotation after a slash 

example:           /’call_outcome_id’

here`s each one of the processor details

Userlevel 4
Badge

you will have to select each processor and show the properties and screenshot

i have replaced it but still error

can you please screenshot what is filled in each processor?

Userlevel 4
Badge

Remove the quotes for source and target both.

Userlevel 4
Badge

@realmatcha 

I can see that in your rename, you are using /’call-comme_id’ 

Remove the quotes and try.

Userlevel 5
Badge +1

@realmatcha 

You can add filter like below based on the field and it will resolve your issues.

 

 

Userlevel 5
Badge +1

@realmatcha 

 

May know if all row contains the column name or not , if it contains then it will rename it else throw error.

You can add a check that if  call-outcomme_id is not null then rename it else no need to do anything.

 

The error is because of the column name is missing for many rows and we are trying rename it .

I've tried but it's still an error, you can see what each processor contains

Userlevel 5
Badge +1

@realmatcha 

 

Please try like below .

 

 

but it gives syntax error. can you please provide and example how to do proper "Field Rename"
Userlevel 5
Badge +1

@realmatcha 

The error seems like the column name is not matching with your destination table name.

Kindly try to rename the column name by using “Field Rename” processor and check if it avoid the issue.


com.streamsets.pipeline.stage.lib.hive.exceptions.HiveStageCheckedException: HIVE_30 - Invalid column name call-outcomme_id

Userlevel 5
Badge +1

@realmatcha 

kindly provide error details, so i can help you on it.

Reply