Logs are a fundamental piece of the web server architecture for debugging, monitoring and alerting purposes.
Some of the infrastructure at Skyscanner Hotels’ Backend makes use of logster to send logs to Graphite and generate alerts through Seyren enabling ChatOps through Slack. Although we usually check the different dashboards in Kibana, having a well-maintained set of logs is important for quickly looking up and investigating an event.
Logrotate is a small utility that allows to rotate the logs: archiving the current log, start a fresh one and delete old logs.
Due to different reasons we were having problems with some of our logs and the fix turned out to be to schedule the log rotation at a specific time.
In order to schedule the log rotate in our CentOS systems we followed these steps:
Understand that logrotate is executed once a day. If we check the logrotate RPM
rpm -ql logrotatewe see something like the following:
/etc/cron.daily/logrotate /etc/logrotate.conf /etc/logrotate.d /usr/sbin/logrotate /usr/share/doc/logrotate-3.8.6 /usr/share/doc/logrotate-3.8.6/CHANGES /usr/share/doc/logrotate-3.8.6/COPYING /usr/share/man/man5/logrotate.conf.5.gz /usr/share/man/man8/logrotate.8.gz /var/lib/logrotate.status
The reason for
logrotatebeing executed once a day can be seen in the previous list. The RPM creates a configuration entry in the
/cron.daily/directory. Consequently, logrotate will be executed daily.
We would expect cron to be in charge of the scheduling of this task. However, for CentOS systems, anacron is the one in charge of the scheduling for daily events (among others). Furthermore, anacron introduces by default a random delay and it only starts running the scheduled tasks during the
START_HOURS_RANGEas configured in
- Set the logrotate at a specific time: moving the logrotate configuration out of the
/logrotate.daily/directory and scheduling its execution through cron.
- name: check logrotate config exists in cron.daily stat: path=/etc/cron.daily/logrotate register: logrotate_config - name: move logrotate config if existing command: mv /etc/cron.daily/logrotate /opt/home/logrotate.cronjob when: logrotate_config.stat.exists - name: configure daily logrotate cron cron: name="schedule logrotate" minute="0" hour="3" job="/opt/home/logrotate.cronjob"
This way, logrotate will be executed every day at 3 am.
I shall thank my colleague Kampde for his detailed walkthrough about the intricacies of cron, anacron and RPMs among others.