Skip to main content
Solved

JDBC Multitable Consumer error

Hi, I am new to Streamsets, started using it today.

I have a task to create some backup data to hive and hadoop using jdbc multitable consumer.

the problem that occurs is that there is a table name with '-' instead of '_' example: call-outcomme_id should the correct one be call_outcomme_id.

I have used the field remover to maintain the table name with 'keep listed fields' when I start an error occurs, but when I select 'remove listed fields' the column that I targeted was deleted and it worked.

thanks in advance

23 replies

Bikram
Rank
Userlevel 5
Badge +1

@realmatcha 

kindly provide error details, so i can help you on it.

realmatcha
Rank


com.streamsets.pipeline.stage.lib.hive.exceptions.HiveStageCheckedException: HIVE_30 - Invalid column name call-outcomme_id

Bikram
Rank
Userlevel 5
Badge +1

@realmatcha 

The error seems like the column name is not matching with your destination table name.

Kindly try to rename the column name by using “Field Rename” processor and check if it avoid the issue.

realmatcha
Rank
 
but it gives syntax error. can you please provide and example how to do proper "Field Rename"
Bikram
Rank
Userlevel 5
Badge +1

@realmatcha 

 

Please try like below .

 

 

realmatcha
Rank

I've tried but it's still an error, you can see what each processor contains

Bikram
Rank
Userlevel 5
Badge +1

@realmatcha 

 

May know if all row contains the column name or not , if it contains then it will rename it else throw error.

You can add a check that if  call-outcomme_id is not null then rename it else no need to do anything.

 

The error is because of the column name is missing for many rows and we are trying rename it .

Bikram
Rank
Userlevel 5
Badge +1

@realmatcha 

You can add filter like below based on the field and it will resolve your issues.

 

 

saleempothiwala
Rank
Userlevel 4
Badge

@realmatcha 

I can see that in your rename, you are using /’call-comme_id’ 

Remove the quotes and try.

saleempothiwala
Rank
Userlevel 4
Badge

Remove the quotes for source and target both.

realmatcha
Rank

can you please screenshot what is filled in each processor?

realmatcha
Rank

i have replaced it but still error

saleempothiwala
Rank
Userlevel 4
Badge

you will have to select each processor and show the properties and screenshot

realmatcha
Rank

here`s each one of the processor details

tamilarasup
Rank

i have replaced it but still error

use quotation after a slash 

example:           /’call_outcome_id’

Bikram
Rank
Userlevel 5
Badge +1

@realmatcha 

 

we need quote for the fields but you need to filter out the fields for which you are trying to rename if the field name exists in your record else no need to anything.

I will provide you the work around for your case . You just try that and i hope it will help.

Bikram
Rank
Userlevel 5
Badge +1

@realmatcha 

 

Please find attached the pipeline and try to implement in your pipeline and i hope it will solve your issues without any error.

 

 

1 Attachment
Bikram
Rank
Userlevel 5
Badge +1

 

realmatcha
Rank

 

cant install library. please help me

Bikram
Rank
Userlevel 5
Badge +1

@realmatcha 

If you are having problem in after importing my pipeline then you just use the processors has been used in my pipeline in your env.

 

Please follow below steps and i hope it will help. Let me know in case of any issues.

step1 : 

Origin as Dev Raw data source and inputs are 

 

{
  "call-comme_id": "abc"
  
}
{
  "call_comme_id": "xyz"
  
}
{
  "call-comme-id": "ABC"
  
}

Step2 : Use the filed mapper processor to rename the fields.

 

 

 

Your pipeline should look like below .

 

 

realmatcha
Rank

I've followed and it's running, but I can't see a result like this

 

 

Bikram
Rank
Userlevel 5
Badge +1

@realmatcha 

You can see the above result if you preview your pipeline.

Just preview and see the result. :)

realmatcha
Rank

@Bikram @saleempothiwala @tamilarasup thank you all for helping a lot, everything is done and running well

Reply


Terms & Conditions