Currently, the log files are rolled over by size. This is specified in sdc-log4j.properties file in $SDC_DIST/etc directory (you can modify the configuration also from the SDC UI > Administration icon > Logs > Log config) by property:
log4j.appender.streamsets=org.apache.log4j.RollingFileAppender
and customized by these two properties:
log4j.appender.streamsets.MaxFileSize=256MBlog4j.appender.streamsets.MaxBackupIndex=10
These default values define that the active log file is rolled over when it reaches 256 MB and the log directory can contain one active and 10 older log files.When there are already 10 older log files, the oldest logs are automatically removed.
According to this link, it is possible to schedule the rollover even by the DatePattern option - monthly, weekly, half-daily, daily, hourly, or minutely rollover schedules (please see the link to find the pattern for your preferred option).
Let's say I want to set the rollover at the beginning of each month. Then I have to modify the sdc-log4j.properties file as follows:
- Add these two lines:
log4j.appender.streamsets=org.apache.log4j.DailyRollingFileAppenderlog4j.appender.streamsets.DatePattern='.'yyyy-MM-
Make sure you comment these three lines:
#log4j.appender.streamsets=org.apache.log4j.RollingFileAppender#log4j.appender.streamsets.MaxFileSize=256MB#log4j.appender.streamsets.MaxBackupIndex=10
However, please be careful - DailyRollingFileAppender has been observed to exhibit synchronization issues and data loss, please see here.
The size of the log files might be very large and less readable after each month. You should make sure that you have enough space on the disk; otherwise, the SDC stops writing the logs to the file. Also, if you want to remove the oldest logs to make some space on the disk, you will have to do it manually.
