Convert multiple CSV records into a single JSON and make an HTTP bulk API POST call

  • 13 March 2022
  • 5 replies
  • 447 views

Userlevel 3
Badge

Hi,

The above pipeline reads the CSV files uploaded into an S3 bucket for new contact file uploads. Based on the new contacts available as a CSV, it converts them into a JSON. But the critical part here is that it merges each batches (each batch size is 1000) into a single JSON of multiple records and finally make a single HTTP POST call to one of our internal system to carry out a CONTACT update call.


5 replies

Hi Swayam,

Can you share the stages configuration.

Userlevel 3
Badge

Send me your email address and I'll share the pipeline.

Userlevel 3
Badge

Can you import the attached pipeline and try? Just unzip and import the pipeline json.

@Rohith 

Thanks Swayam 

Hi Swayam,

Thanks for sharing the pipeline.

I have a requirement where my input json will look like this.

[

{

“name”:”xyz”,

“age”:”21”

},

{

“name”:”xyz”,

“age”:”21”

},

.

.

.

]

 

And I want the output to be in the below json format

{
    "data": {
        "columns": {
            "name": "character",
            "age": "numeric"
        },
        "records": [

{

“name”:”xyz”,

“age”:”21”

},

{

“name”:”xyz”,

“age”:”21”

},

.

.

.

]

    },
    "options": {
        "purge": false
    }
}

Can you help me with this? Is it possible using a single a pipeline?

Thanks in Advance

Reply