I have a use case to read a JSON file from ADLS location and write the entire content of the JSON file to a VARIANT field in Snowflake. I have used ADLS origin stage , Jython Processor and Snowflake Destination Stage. In the ADLS stage, I have the data format as “WholeFile” to get /fileRef.
I used the Jython processor to read /FileRef and convert it as String before loading this field to Snowflake.
I am getting error in Jython Processor Stage
Script sent record to error: 'com.streamsets.pipeline.stage.util.scripting.Scrip' object has no attribute 'createInputStream
fileRef : FILE_REFIobject Object]
How do I read entire JSON file and load the contents to a VARIANT field in Snowflake
Sample JSON:
{
"MbrDetails" :
{ "id" : "100001",
"ProgramName" : "test"
},
"NotificationInformation" :
{ "ProgramStatusDate" : "2024-11-13",
"Category" : "Pilot Program,
"DateTime" : "2024-11-13",
"ProgramStatus" : "Eligible"
}
}
Expected Outcome in Snowflake Table:
| full_json | file_name |
|---|---|
| { "MbrDetails" : }, } | filename_2024_11_13.json |