Question

SOOLDIR_01 and SPOOLDIR_35 in Hadoop FS standalone

  • 11 July 2023
  • 1 reply
  • 27 views
msc
Rank

Hi,

I’m using Hadoop FS standalone as origin in the pipeline. With read order last modified timestamp
The file directory is : /user/msc/*           File name pattern is : *

Under msc there are multiple folders and in these folders the hadoop will read all the files present, some functions are ran on those file and move the files to some other locations. The pipeline is working fine but sometimes I get a error like SPOOLDIR_01- failed to process file. Even though the file is read and processed , I’m getting this error.

I’m also getting a error like Running error : SPOOLDIR_35- spool directory runner failed reason java.io.Filenotfoundexception: file does not exist. After this the pipeline restarts itself.

 

Please help me out, if anyone know the reason.

 

Thanks,
Madhusudan

 

1 reply

Bikram
Rank
Userlevel 5
Badge +1

@msc 

The connectivity issue with your origin is the cause of this occurrence, which can happen occasionally.

To successfully execute the pipeline, you can try rerunning it with the reset origin option.

If you encounter this issue frequently, it is recommended to raise a support ticket with StreamSets to investigate the problem.

Please ensure that your configuration is correct before submitting a ticket to StreamSets.

Reply


Terms & Conditions