Dear Community users,

To extend the last topic where I shared how to convert the CSV to JSON, what if you have to combine multiple JSON records in a batch into a single JSON?

I often came with requirements where we have to call an external API which suports both single as well as batch requests and most often batch is preffered to avoid the multiple round-trips and increae performance. This topic can be very useful for developers who wants to convert the multiple CSV records coming in a batch to a single JSON list array and send them as a single record to a target destination.

Problem Statement

Convert the 2 input CSV records into a single JSON.

Input CSV Records:

NAME|AGE|DEPARTMENT|ADDRESS
"John"|"23"|"HR"|"31, 2nd Cross, 4th Street, Wilson-543234"
"Amanda"|"26"|"HR"|"24, 1st Cross, Wilson Garden-543234"

Output Single JSON Record:

>
  {
    "data": {
      "NAME": "John",
      "AGE": "23",
      "DEPARTMENT": "HR",
      "ADDRESS": "31, 2nd Cross, 4th Street, Wilson-543234"
    }
  },
  {
    "data": {
      "NAME": "Amanda",
      "AGE": "26",
      "DEPARTMENT": "HR",
      "ADDRESS": "24, 1st Cross, Wilson Garden-543234"
    }
  }
]

Pipeline (Extending with just one more Jython evaluator) from the previous blog:

Past Blog Reference: 

The new Jython Eveluator combines all the JSON records coming in a batch into a single record.

whole_data = ""
for record in records:
  try:
    # Access each record field and its value    
    for (field,val) in record.value.iteritems():
      whole_data = whole_data+ val + ","

    # Write record to processor output
    t_whole_data = whole_data.rstrip(whole_dataa-1])
    record.valuel'data'] = t_whole_data

    #output.write(record)

  except Exception as e:
    # Send record to error
    error.write(record, str(e))

output.write(record)

As you can see in the pipeline review, both input records are now combined into a single record.

Conclusion

Simple. Honestly, I’ve seen developers spending time to carry out quick PoCs but spending long hours to find how they can do it inside SDC. I’m sure there may be other better ways to do it, but idea here is to make a quick start to see the power of data and tools like SDC that helps us.