Question

StreamSets Hive Metadata issue after upgrading to CDP 7.1.x

  • 25 October 2022
  • 0 replies
  • 79 views

Badge

Issue:

 

   Hive Metadata stage throws error "A managed table's location should be located within managed warehouse root directory or within its database's managedLocationUri" after upgrading to CDP version 7.14 and above.

 

Reason:

 

    In CDP7.x or HDP3.x, the database has 2 locations by default(managed and external) and managed tables are always ACID.

Whereas HDP-2.6.x or CDH-6.x or lesser, there is no difference in managed/external table location and managed table will not be created as ACID table by default.

 

In CDP7 or HDP3,

When user tries to create managed table with custom location, CDP-Hive validates whether the custom location is outside database's managed location and throws above described exception in StreamSets pipeline.

 

StreamSets will not support ACID table read/write.

 

Workaround:    

 

     Since StreamSets pipeline with Hive Metadata stage does not support Hive Managed table creation, the best solution is to choose the External table option and provide Table path template so that external table is created without any issues.


0 replies

Be the first to reply!

Reply