Skip to main content

I’m building a data collector pipeline in which I want to rename elements within an array.
The input are JSON files read from a data lake, format is a single document with an array of identically structured documents, e.g.


{ "entries": :

  {

     "wd:Employee_User_ID": "lol",

     "wd:Scheduled_Weekly_Hours": "40",

     "wd:fte": "100" },

  {

     "wd:Employee_User_ID": "rofl",

     "wd:Scheduled_Weekly_Hours": "37.5",

     "wd:fte": "100" }

]}

 

I need to rename these fields, so I’ve put the following into the Field Renamer processor:

/entries

  • /'wd:Employee_User_ID' → /entries
  • /alias

     

    But that gives an 

    java.lang.ArrayIndexOutOfBoundsException: -1

        at java.util.ArrayList.elementData(ArrayList.java:424)

        at java.util.ArrayList.get(ArrayList.java:437)

        at com.streamsets.datacollector.record.RecordImpl.get(RecordImpl.java:309)

        at com.streamsets.datacollector.record.RecordImpl.has(RecordImpl.java:374)

        at com.streamsets.pipeline.stage.processor.fieldrenamer.FieldRenamerProcessor.process(FieldRenamerProcessor.java:300)

     

    It works quite well (for the first array element) if I replace the asterisk with a 0 in the expressions. Isn’t this the correct syntax for renaming all array elements? 

    • @chisou Try Field Mapper processor

    Thanks, @saleempothiwala 
    I did that:

    And it works, but it’s not renaming, but copying. Also - just for one, and having a Field Mapper for each field seems so inefficient.

    Is the Field Renamer not able to do this? The documentation states “To rename an array or map, you can specify a single array index or map element, or you can use the asterisk wildcard to represent all array indices and map elements.” 

     

    @chisou 

     

    See this below:

     

    Output:

     

    so if you can write an expression to identify the values to be changed then it can be done for multiple fields. 

    for e.g. if you want to take out wd: from all the fields then simply use something like:

     

    Ah, replace, thanks! Unfortunately is not a pattern like this, so I would need to go with multiple Mappers for now. 

    Do you happen to know whether Renamer is considered legacy or something? It seems like a pretty basic requirement, being able to just rename record labels. I most cases you’d probably want to do that within Snowflake directly I guess, but still ...

    Reply


    Terms & Conditions