Skip to main content
Solved

JDBC Multitable Consumer error


realmatcha
Opening Band

Hi, I am new to Streamsets, started using it today.

I have a task to create some backup data to hive and hadoop using jdbc multitable consumer.

the problem that occurs is that there is a table name with '-' instead of '_' example: call-outcomme_id should the correct one be call_outcomme_id.

I have used the field remover to maintain the table name with 'keep listed fields' when I start an error occurs, but when I select 'remove listed fields' the column that I targeted was deleted and it worked.

thanks in advance

Best answer by realmatcha

I've followed and it's running, but I can't see a result like this

 

 

View original
Did this topic help you find an answer to your question?

23 replies

Bikram
Headliner
Forum|alt.badge.img+1
  • Headliner
  • 486 replies
  • May 22, 2023

@realmatcha 

kindly provide error details, so i can help you on it.


realmatcha
Opening Band
  • Author
  • Opening Band
  • 13 replies
  • May 22, 2023


com.streamsets.pipeline.stage.lib.hive.exceptions.HiveStageCheckedException: HIVE_30 - Invalid column name call-outcomme_id


Bikram
Headliner
Forum|alt.badge.img+1
  • Headliner
  • 486 replies
  • May 22, 2023

@realmatcha 

The error seems like the column name is not matching with your destination table name.

Kindly try to rename the column name by using “Field Rename” processor and check if it avoid the issue.


realmatcha
Opening Band
  • Author
  • Opening Band
  • 13 replies
  • May 23, 2023
but it gives syntax error. can you please provide and example how to do proper "Field Rename"

Bikram
Headliner
Forum|alt.badge.img+1
  • Headliner
  • 486 replies
  • May 23, 2023

@realmatcha 

 

Please try like below .

 

 


realmatcha
Opening Band
  • Author
  • Opening Band
  • 13 replies
  • May 23, 2023

I've tried but it's still an error, you can see what each processor contains


Bikram
Headliner
Forum|alt.badge.img+1
  • Headliner
  • 486 replies
  • May 23, 2023

@realmatcha 

 

May know if all row contains the column name or not , if it contains then it will rename it else throw error.

You can add a check that if  call-outcomme_id is not null then rename it else no need to do anything.

 

The error is because of the column name is missing for many rows and we are trying rename it .


Bikram
Headliner
Forum|alt.badge.img+1
  • Headliner
  • 486 replies
  • May 24, 2023

@realmatcha 

You can add filter like below based on the field and it will resolve your issues.

 

 


saleempothiwala
Headliner
Forum|alt.badge.img

@realmatcha 

I can see that in your rename, you are using /’call-comme_id’ 

Remove the quotes and try.


saleempothiwala
Headliner
Forum|alt.badge.img

Remove the quotes for source and target both.


realmatcha
Opening Band
  • Author
  • Opening Band
  • 13 replies
  • May 24, 2023

can you please screenshot what is filled in each processor?


realmatcha
Opening Band
  • Author
  • Opening Band
  • 13 replies
  • May 24, 2023

i have replaced it but still error


saleempothiwala
Headliner
Forum|alt.badge.img

you will have to select each processor and show the properties and screenshot


realmatcha
Opening Band
  • Author
  • Opening Band
  • 13 replies
  • May 24, 2023

here`s each one of the processor details


tamilarasup
Discovered Fame
  • Discovered Fame
  • 11 replies
  • May 24, 2023
realmatcha wrote:

i have replaced it but still error

use quotation after a slash 

example:           /’call_outcome_id’


Bikram
Headliner
Forum|alt.badge.img+1
  • Headliner
  • 486 replies
  • May 24, 2023

@realmatcha 

 

we need quote for the fields but you need to filter out the fields for which you are trying to rename if the field name exists in your record else no need to anything.

I will provide you the work around for your case . You just try that and i hope it will help.


Bikram
Headliner
Forum|alt.badge.img+1
  • Headliner
  • 486 replies
  • May 24, 2023

@realmatcha 

 

Please find attached the pipeline and try to implement in your pipeline and i hope it will solve your issues without any error.

 

 


Bikram
Headliner
Forum|alt.badge.img+1
  • Headliner
  • 486 replies
  • May 24, 2023

 


realmatcha
Opening Band
  • Author
  • Opening Band
  • 13 replies
  • May 25, 2023

 

cant install library. please help me


Bikram
Headliner
Forum|alt.badge.img+1
  • Headliner
  • 486 replies
  • May 25, 2023

@realmatcha 

If you are having problem in after importing my pipeline then you just use the processors has been used in my pipeline in your env.

 

Please follow below steps and i hope it will help. Let me know in case of any issues.

step1 : 

Origin as Dev Raw data source and inputs are 

 

{
  "call-comme_id": "abc"
  
}
{
  "call_comme_id": "xyz"
  
}
{
  "call-comme-id": "ABC"
  
}

Step2 : Use the filed mapper processor to rename the fields.

 

 

 

Your pipeline should look like below .

 

 


realmatcha
Opening Band
  • Author
  • Opening Band
  • 13 replies
  • Answer
  • May 25, 2023

I've followed and it's running, but I can't see a result like this

 

 


Bikram
Headliner
Forum|alt.badge.img+1
  • Headliner
  • 486 replies
  • May 25, 2023

@realmatcha 

You can see the above result if you preview your pipeline.

Just preview and see the result. :)


realmatcha
Opening Band
  • Author
  • Opening Band
  • 13 replies
  • June 2, 2023

@Bikram @saleempothiwala @tamilarasup thank you all for helping a lot, everything is done and running well


Reply