How to use Content Type ‘multipart/form-data’ in the HTTP Client stage

  • 22 May 2023
  • 1 reply

Userlevel 1
  • StreamSets Employee
  • 0 replies

SDC’s HTTP Client stage does not have a native option to send HTTP requests using the Content Type ‘multipart/form-data’. Using the default options will result in SDC making the call seemingly correctly but it will not add the required boundary characters that a request using this kind of Content Type requires. 

However, we can force SDC to send the call in the correct multipart/form-data’ format. To do this you will need to set these parameters in the following way:

*** For this example, the request SDC would make is equivalent to the following curl command:

curl --location '' \
--header 'Accept: application/json' \
--header 'Authorization: Bearer *********************' \
--header 'Cookie: staging-connections-canary="45412ba154f512d34"' \
--form 'options="{\"options\":{},\"url\":\"\",\"sslVersion\":\"TLS 1.2\",\"password\":xxxxxxxxxx,\"userId\":xxxxxxxxxx,\"displayName\":\"Pedro\",\"jobName\":\"pedro_test_job\",\"batchsize\":1000,\"waitTimeoutms\":10000}"'

- Default Request Content Type:

multipart/form-data; boundary=------------------------514ca32f61f8d64f

- Headers: set them just like in the curl command

- Request Body:

--------------------------514ca32f61f8d64f${str:unescapeJava('\\u000D')}${str:unescapeJava('\\u000A')}Content-Disposition: form-data; name="options"${str:unescapeJava('\\u000D')}${str:unescapeJava('\\u000A')}${str:unescapeJava('\\u000D')}${str:unescapeJava('\\u000A')}{"options":{},"url":"","sslVersion":"TLS 1.2","password":xxxxxxxxxx,"userId":xxxxxxxxxx,"displayName":"Pedro","jobName":"pedro_test_job","batchsize":1000,"waitTimeoutms":10000}${str:unescapeJava('\\u000D')}${str:unescapeJava('\\u000A')}--------------------------514ca32f61f8d64f--${str:unescapeJava('\\u000D')}${str:unescapeJava('\\u000A')}


Note that the request body is a single line. This body specifies the boundary character, as specified in the Default Request Content Type field needed for the correct ‘multipart/form-data’ format. Also, each of the required spaces are entered as escaped characters so that the format is correctly sent. Adding regular spacing instead of this can result in the message being sent in the incorrect format and the HTTP server not being able to recognize it. 

If you analyze the requests sent by executing an SDC pipeline with this configuration and the curl command shown above with a tool such as Wireshark should show that both requests are equivalent.
*** There will still be some differences between them as they are made with different clients and such but the actual request message should be the same.

1 reply

Hello @pedro 
Can you help me transform this curl :
curl --location '' \
--header 'Authorization: Basic test==' \
--header 'Cookie: full_name=Guest; sid=Guest; system_user=no; user_id=Guest; user_image=' \
--form 'doctype="gg file"' \
--form 'docname="4d3faa9cb2"' \
--form 'file_name="testfromapi.txt"' \
--form 'fieldname="attach"' \
--form 'filedata="SGVsbG8gV29ybGQ="'