Skip to main content

Product: Streamsets Data Collector 

 

Issue: When reading from Oracle and writing into Hive destination the primary key coulmn was defined as NUMBER(12,0)

When you have a pipeline with Field type convertor that performs a Decimal to Float conversion before writing to Hive. 


Versions affected: All 

 

Solution:

Due to Field Type Convertor the incoming PK column value was being converted to float
This was changing the data value and writing wrong values to the end point.

Basically if you have a field something line

SOURCE RECORDOBJECT_ID       OBJECT_NAME1456098345.0       T13491348976.0       T24359803400.0       T3

The same records get written into the target as shown below when you have a Field Type Convertor performing a Decimal to Float values conversion

TARGET RECORDOBJECT_ID        OBJECT_NAME1456098304        T13491348992        T24359803392        T3

If you focus on the Input and output records the OBJECT_ID which is primary key is getting changed because of the Field Type Convertor. If you set any conversion based on the Datatype ensure that the primary key columns are not affected because of that

Be the first to reply!