Skip to main content

Hi, I need to know more about the sdc.log files which are available there : 

Administration/Logs/Downloads

I need to know : 

  • where can I configure the number of files ? 10 is a default value ?
  • when does a new file sdc.log.<number> is created ? Is there a limit size ?

Then, I would like to know if the logs are lost after a Restart Data collector (manually)

 

Thanks, Flo
 

 

Hi @fdeborde - Thank you for reaching out to the Streamsets Community.

 

You can control the logs file size and the total log files by updating the sdc-log4j.properties file. The same can be available via SDC UI » Administration » Log Config » Logs section.

appender.streamsets.policies.size.size=2MB ==> Log file size.

appender.streamsets.strategy.max = 4 ===> Total files.

 

The logger would not delete any log files from the data collector log directory.

 

Thank you - AkshayJ

 

 

 

 

 

 

 

Thank you @AkshayJadhav for your quick answer :)

Hello again, I have an additional question.

With the following configuration :

  • log4j.appender.streamsets.MaxFileSize=256MB
  • log4j.appender.streamsets.MaxBackupIndex=10

This means that whenever an sdc.log file reaches 256MB : 

  • a new sdc.log file is created
  • the old file sdc.log → sdc.log1
  • sdc.log1 → sdc.log2
  • ...
  • sdc.log9 → sdc.log10
  • the old file sdc.log10 is not kept

And this happens because of the parameter MaxBackupIndex, right ?

 

*The files I am talking about are presented in the first comment in the first picture.

@fdeborde - Whenever an sdc.log file reaches 256MB:

  1. sdc.log fille be sdc-1.log.
  2. New sdc.log file will be created.
  3. if new file reaches to 256MB then sdc.log will be renamed with sdc-2.log and new sdc.log will be created.

 

You would find the latest logs in the sdc.log then sdc-9.log then sdc-8.log. Here is the last log entry on all the files which explains the same.

#tail -n1 sdc-1.log
2023-04-03 21:18:50,445 4user:] rpipeline:] erunner:] rthread:main] istage:] DEBUG ModelConverterContextImpl - resolveProperty rcollection type; class java.util.List, contains isimple type, class com.streamsets.datacollector.restapi.rbean.lang.BaseMsg]]


#tail -n1 sdc-2.log
2023-04-03 21:18:51,771 7user:] rpipeline:] erunner:] rthread:main] istage:] DEBUG ModelResolver - Can't check class asimple type, class java.lang.Object], java.lang.Object


#tail -n1 sdc-3.log
2023-04-03 21:18:51,996 9user:] rpipeline:] erunner:] rthread:main] istage:] DEBUG ModelConverterContextImpl - resolve lsimple type, class com.streamsets.datacollector.restapi.bean.AntennaDoctorMessageJson]


#tail -n1 sdc-4.log
2023-04-03 21:18:52,238 2user:] rpipeline:] erunner:] rthread:main] istage:] DEBUG ModelConverterContextImpl - resolveProperty rcollection type; class java.util.List, contains isimple type, class com.streamsets.datacollector.restapi.rbean.lang.BaseMsg]]


#tail -n1 sdc.log
2023-04-03 22:27:57,211 2user:*?] *pipeline:-] :runner:] rthread:managerExecutor-pool-3-thread-1] -stage:] DEBUG StandaloneAndClusterPipelineManager - Triggering previewer cache cleanup

It is happening because of the SizeBasedTriggeringPolicy policy and max files are set to 10.

Hope this helps.

Thank you - AkshayJ

Hi @fdeborde  - I just wanted to reach out to you to check if you have any question on this thread.

If there no further question, please mark the comment as a best answer that will close the thread.

 

Thank you,

Akshay Jadhav

Hi @AkshayJadhav , it is done. Thanks

Reply


Terms & Conditions