Skip to main content
Question

parsing a file which special delimiter


ashok verma
Discovered Fame

i have file which has records delimited by pipe and around pipe i have double quotes.

how can is plit this file and store into snowflake or parquet.

 

ex :

1”|”USA”|”ABC company“|”Resident

2”|”CAN”|”NORther “|”Rsdiendtn

5 replies

antmcmullen
StreamSets Employee
  • StreamSets Employee
  • 16 replies
  • March 16, 2022

Hi Ashok,

Have you tried setting the file format as Delimited and the Delimiter Format Type to Custom?

 

I think this will let you process a pipe delimited file like your example

 


ashok verma
Discovered Fame
  • Author
  • Discovered Fame
  • 13 replies
  • March 16, 2022

i am able to process but i see double quotes on my first field name at target and rest all fields are populating correctly.

ex :

1”

2”


Giuseppe Mura
StreamSets Employee
Forum|alt.badge.img
  • StreamSets Employee
  • 37 replies
  • March 16, 2022

I have just tried with this data as input:
ID"|"Country"|"Company"|"Status
1"|"USA"|"ABC company"|"Resident
2"|"CAN"|"NORther "|"Rsdiendtn
 

I have added a header row, just so that I can get proper field names.

I have then configured the format as follows:
 

And the result is:

 

Which, I believe, is what you want.

 

 


ashok verma
Discovered Fame
  • Author
  • Discovered Fame
  • 13 replies
  • March 17, 2022

thanks you, if i put schema as DDL format, the values are populating as null

 

SEQUENCE_NUMBER    Integer,
country    String,
company    String,
status    String,


ashok verma
Discovered Fame
  • Author
  • Discovered Fame
  • 13 replies
  • March 17, 2022

Giuseppe Mura, without adding header were you able to achieve?


Reply