Question

How do you introduce (write) new columns to an existing hadoop table?

  • 9 September 2022
  • 1 reply
  • 47 views
N
Rank

Hello community,

My pipeline is being upgraded to ingest more fields and I want to write these new fields to the existing table in Hadoop. I have searched for the metadata processor but it does not exist and I have enabled data drift. None of them are able to write new columns, which means I have to write an ALTER TABLE statement to capture these new fields. 

With time we will get further requirements to expand our tables and introduce new fields. Is StreamSets able to write new fields automatically?

Kind regards,

Nick

1 reply

N
Rank

Hello community,

I think I managed to resolve this issue. Just had to run the pipeline over and over till it forced the amendments in Hadoop. Not such a technical solution but it worked.

Many thanks,

Nick

Reply


Terms & Conditions