Question

How to resolve this error SCHEMA_GEN_0007 - Map '/XMLData' have different schemas for items.

P
Rank

I am using the SDC to make an API call which returns an XML file. My final requirement is to write the XML output as a parquet file on Azure datalake.

I had thought of converting the XML into Avro and then using a whole file converter to convert the Avro into Parquet. The schema generator for the avro file creation is giving the following error.

SCHEMA_GEN_0007 - Map '/XMLData' have different schemas for items. First schema: '"string"', Second schema: '{"type":"array","items":{"type":"map","values":{"type":"array","items":{"type":"map","values":{"type":"array","items":{"type":"map","values":"string"}}}}}}'

 

How do I resolve this error?

Attaching the XML preview below -

 

 

Also if there is a more efficient way to do all this, I’d really appreciate the input.

 

4 replies

alex.sanchez
Rank
Userlevel 2
Badge

Hi @Paawan,

 

Based on the error, looks like different records have different schemas, and it is failing to create the Avro records for that.

Alex - StreamSets Engineering Manager @ Collector
P
Rank

Hi @alex.sanchez, thanks for taking out the time.

 

The record basically has 3 lists which further contain maps. I tried removing string and providing only the three lists but that didn’t work too. I am hoping to create a single avro file for the whole record, how should I go about debugging this? I am not sure where to start.

Bikram
Rank
Userlevel 5
Badge +1

Hi @Paawan 

Your avro schema should be as per input data.

In your case it couldn’t handle your data in the schema defined for them.

please modify your schema and retry it , i hope it will work.

 

Thanks & Regards,

Bikram_

P
Rank

Hi @Paawan 

Your avro schema should be as per input data.

In your case it couldn’t handle your data in the schema defined for them.

please modify your schema and retry it , i hope it will work.

 

Thanks & Regards,

Bikram_

Hey @Bikram , as you may see in the screenshot I am not specifying the schema manually. Since I am expecting data with a dynamic schema I have used the schema generator component to create a schema. It is giving me an error WHILE creating the schema.

Reply


Terms & Conditions