Bug #5991
closed
On a freshly installed node, rsyslog is dropping messages
Added by Nicolas CHARLES almost 10 years ago.
Updated over 9 years ago.
Category:
System techniques
Description
On a fresh install, on CentOS, i have node which are not correctly sending reports, and /var/log/messages tells me:
Dec 12 15:15:18 node1 rsyslogd-2177: imuxsock begins to drop messages from pid 7891 due to rate-limiting
Dec 12 15:15:24 node1 rsyslogd-2177: imuxsock lost 8 messages from pid 7891 due to rate-limiting
Dec 12 15:16:39 node1 rsyslogd-2177: imuxsock begins to drop messages from pid 9244 due to rate-limiting
Dec 12 15:17:06 node1 rsyslogd-2177: imuxsock begins to drop messages from pid 10052 due to rate-limiting
Dec 12 15:18:12 node1 rsyslogd-2177: imuxsock begins to drop messages from pid 10970 due to rate-limiting
The config files are correct, and rsyslog was indeed restarted before at 15:14
Restarting rsyslog solves the issue
I had this problem on 3.0, but i've seen before weird reporting bug on 2.11 as well, where restarting rsyslog solved the issue
- Status changed from New to Discussion
- Assignee set to Nicolas CHARLES
Restarting rsyslog probably didn't solve the problem, just reset the rate limiting counters. Rate limiting in rsyslog is described in http://www.rsyslog.com/doc/master/configuration/modules/imuxsock.html. Basically, it counts number of messages during a given time period. So by restarting rsyslog, you reset both - but this could still end up not working anymore after a few seconds/minutes.
We do have a workaround for this (force rate limiting off), introduced in #3913.
This creates a file in /etc/rsyslog.d/remove_limit.conf that should contain "$SystemLogRateLimitInterval 0". Could you check the existance and content of this file on the system that has this problem?
If it doesn't exist, /var/rudder/tools/check_rsyslog_version may be at cause. Could you run it? It should output "+rsyslog_greater_than_5_7_1".
- Subject changed from On a fresh node install, rsyslog is droping message to On a freshly installed node, rsyslog is dropping messages
The file does exist, but I suspect that it was added after the restart
- Target version changed from 2.11.6 to 2.10.9
Ok, so on !(SuSE|fedora), we are running /etc/init.d/rsyslog restart
This command doesn't exist on Centos 7, and it may be deprecated on Centos 6
Note: on 3.0 we are calling service_restart("rsyslog"), so it should not fail. I'll investigate a bit more
Ok, the bug is tricky: we are calling twice the service_restart bundle; once when we put the rudder-agent.conf rsyslog file, and one when we remove the rate limiter
However, as we execute the second bundle, nothing happen ( it enters the bundle serv_cerestart, but doesn't do anything as all other promises have been checked )
We should enforce the restart with action immediate
Ok, adding action => immediate is not enough, i still have in the logs:
2014-12-18T17:36:43+0000 verbose: /default/check_rsyslog_version/methods: Evaluating promise 'restart_syslog_ng'
2014-12-18T17:36:43+0000 verbose: /default/check_rsyslog_version/methods/'restart_syslog_ng'/default/service_restart: Augment scope 'service_restart' with variable 'service_name' (type: s)
2014-12-18T17:36:43+0000 verbose: /default/check_rsyslog_version/methods/'restart_syslog_ng'/default/service_restart: Evaluating bundle pass 1
2014-12-18T17:36:43+0000 verbose: /default/check_rsyslog_version/methods/'restart_syslog_ng'/default/service_restart/methods: Evaluating promise 'wrap'
2014-12-18T17:36:43+0000 verbose: /default/check_rsyslog_version/methods/'restart_syslog_ng'/default/service_restart: Evaluating bundle pass 2
2014-12-18T17:36:43+0000 verbose: /default/check_rsyslog_version/methods/'restart_syslog_ng'/default/service_restart/methods: Evaluating promise 'wrap'
2014-12-18T17:36:43+0000 verbose: /default/check_rsyslog_version/methods/'restart_syslog_ng'/default/service_restart: Evaluating bundle pass 3
2014-12-18T17:36:43+0000 verbose: /default/check_rsyslog_version/methods/'restart_syslog_ng'/default/service_restart/methods: Evaluating promise 'wrap'
- Target version changed from 2.10.9 to 3.0.0~rc1
Ok, this exists only in 3.0 (2.11 behaves correctly)
It was introduced by #5846
- Target version changed from 3.0.0~rc1 to 3.0.0
As a workaround, I propose to put check_log_system in "end" instead of "bs" in the bundlesequence.
Since the modification done in check_log_system (add one line) is already done in install_rsyslogd this will produce the final file first and check_log_system won't need to modify it and restart rsyslog again.
- Status changed from Discussion to Pending technical review
- Assignee changed from Nicolas CHARLES to Benoît PECCATTE
- Pull Request set to https://github.com/Normation/rudder-techniques/pull/629
- Status changed from Pending technical review to Pending release
- % Done changed from 0 to 100
Applied in changeset policy-templates:commit:60a29992e4b192c241e80c96118eb1e567476373.
Applied in changeset policy-templates:commit:c031e86f255e2bbfdfbb77f6fc297937a220e0db.
- Status changed from Pending release to Released
This bug has been fixed in Rudder 3.0.0, which was released on 2015/02/16
- Related to Bug #7098: When syslog-ng is replaced by rsyslogd on SuSE, service is not restarted added
Also available in: Atom
PDF
Updated by Jonathan CLARKE 
Updated by 
Updated by 
Updated by 
Updated by