Project

General

Profile

Actions

Bug #5991

closed

On a freshly installed node, rsyslog is dropping messages

Added by Nicolas CHARLES over 6 years ago. Updated about 6 years ago.

Status:
Released
Priority:
N/A
Category:
System techniques
Target version:
Severity:
User visibility:
Effort required:
Priority:

Description

On a fresh install, on CentOS, i have node which are not correctly sending reports, and /var/log/messages tells me:
Dec 12 15:15:18 node1 rsyslogd-2177: imuxsock begins to drop messages from pid 7891 due to rate-limiting
Dec 12 15:15:24 node1 rsyslogd-2177: imuxsock lost 8 messages from pid 7891 due to rate-limiting
Dec 12 15:16:39 node1 rsyslogd-2177: imuxsock begins to drop messages from pid 9244 due to rate-limiting
Dec 12 15:17:06 node1 rsyslogd-2177: imuxsock begins to drop messages from pid 10052 due to rate-limiting
Dec 12 15:18:12 node1 rsyslogd-2177: imuxsock begins to drop messages from pid 10970 due to rate-limiting

The config files are correct, and rsyslog was indeed restarted before at 15:14

Restarting rsyslog solves the issue

I had this problem on 3.0, but i've seen before weird reporting bug on 2.11 as well, where restarting rsyslog solved the issue


Related issues

Related to Rudder - Bug #3913: Rsyslog 5.7.1 drops reports when they come to fastReleasedMatthieu CERDA2013-09-06Actions
Related to Rudder - Architecture #5846: Add support for systemd in the Techniques (System ones)ReleasedJonathan CLARKE2014-11-27Actions
Related to ncf - User story #6099: We should be able to call actions as many times as we wantIn progressFélix DALLIDETActions
Related to ncf - Architecture #6101: We should be able to call ncf generic methods more than onceRejected2015-01-08Actions
Related to Rudder - Bug #7098: When syslog-ng is replaced by rsyslogd on SuSE, service is not restartedReleasedMatthieu CERDA2015-08-12Actions
Actions #1

Updated by Jonathan CLARKE over 6 years ago

  • Status changed from New to Discussion
  • Assignee set to Nicolas CHARLES

Restarting rsyslog probably didn't solve the problem, just reset the rate limiting counters. Rate limiting in rsyslog is described in http://www.rsyslog.com/doc/master/configuration/modules/imuxsock.html. Basically, it counts number of messages during a given time period. So by restarting rsyslog, you reset both - but this could still end up not working anymore after a few seconds/minutes.

We do have a workaround for this (force rate limiting off), introduced in #3913.

This creates a file in /etc/rsyslog.d/remove_limit.conf that should contain "$SystemLogRateLimitInterval 0". Could you check the existance and content of this file on the system that has this problem?

If it doesn't exist, /var/rudder/tools/check_rsyslog_version may be at cause. Could you run it? It should output "+rsyslog_greater_than_5_7_1".

Actions #2

Updated by Jonathan CLARKE over 6 years ago

  • Subject changed from On a fresh node install, rsyslog is droping message to On a freshly installed node, rsyslog is dropping messages
Actions #3

Updated by Nicolas CHARLES over 6 years ago

The file does exist, but I suspect that it was added after the restart

Actions #4

Updated by Nicolas CHARLES over 6 years ago

  • Target version changed from 2.11.6 to 2.10.9

Ok, so on !(SuSE|fedora), we are running /etc/init.d/rsyslog restart
This command doesn't exist on Centos 7, and it may be deprecated on Centos 6

Note: on 3.0 we are calling service_restart("rsyslog"), so it should not fail. I'll investigate a bit more

Actions #5

Updated by Nicolas CHARLES over 6 years ago

Ok, the bug is tricky: we are calling twice the service_restart bundle; once when we put the rudder-agent.conf rsyslog file, and one when we remove the rate limiter

However, as we execute the second bundle, nothing happen ( it enters the bundle serv_cerestart, but doesn't do anything as all other promises have been checked )
We should enforce the restart with action immediate

Actions #6

Updated by Nicolas CHARLES over 6 years ago

Ok, adding action => immediate is not enough, i still have in the logs:

2014-12-18T17:36:43+0000  verbose: /default/check_rsyslog_version/methods: Evaluating promise 'restart_syslog_ng'
2014-12-18T17:36:43+0000  verbose: /default/check_rsyslog_version/methods/'restart_syslog_ng'/default/service_restart: Augment scope 'service_restart' with variable 'service_name' (type: s)
2014-12-18T17:36:43+0000  verbose: /default/check_rsyslog_version/methods/'restart_syslog_ng'/default/service_restart: Evaluating bundle pass 1
2014-12-18T17:36:43+0000  verbose: /default/check_rsyslog_version/methods/'restart_syslog_ng'/default/service_restart/methods: Evaluating promise 'wrap'
2014-12-18T17:36:43+0000  verbose: /default/check_rsyslog_version/methods/'restart_syslog_ng'/default/service_restart: Evaluating bundle pass 2
2014-12-18T17:36:43+0000  verbose: /default/check_rsyslog_version/methods/'restart_syslog_ng'/default/service_restart/methods: Evaluating promise 'wrap'
2014-12-18T17:36:43+0000  verbose: /default/check_rsyslog_version/methods/'restart_syslog_ng'/default/service_restart: Evaluating bundle pass 3
2014-12-18T17:36:43+0000  verbose: /default/check_rsyslog_version/methods/'restart_syslog_ng'/default/service_restart/methods: Evaluating promise 'wrap'

Actions #7

Updated by Nicolas CHARLES over 6 years ago

  • Target version changed from 2.10.9 to 3.0.0~rc1

Ok, this exists only in 3.0 (2.11 behaves correctly)
It was introduced by #5846

Actions #8

Updated by Nicolas CHARLES over 6 years ago

See reference bug in cfengine bugtracker:
https://dev.cfengine.com/issues/6893

Actions #9

Updated by Vincent MEMBRÉ over 6 years ago

  • Target version changed from 3.0.0~rc1 to 3.0.0
Actions #10

Updated by Benoît PECCATTE over 6 years ago

As a workaround, I propose to put check_log_system in "end" instead of "bs" in the bundlesequence.

Since the modification done in check_log_system (add one line) is already done in install_rsyslogd this will produce the final file first and check_log_system won't need to modify it and restart rsyslog again.

Actions #11

Updated by Nicolas CHARLES over 6 years ago

  • Status changed from Discussion to Pending technical review
  • Assignee changed from Nicolas CHARLES to Benoît PECCATTE
  • Pull Request set to https://github.com/Normation/rudder-techniques/pull/629
Actions #12

Updated by Nicolas CHARLES over 6 years ago

  • Status changed from Pending technical review to Pending release
  • % Done changed from 0 to 100

Applied in changeset policy-templates:commit:60a29992e4b192c241e80c96118eb1e567476373.

Actions #13

Updated by Benoît PECCATTE over 6 years ago

Applied in changeset policy-templates:commit:c031e86f255e2bbfdfbb77f6fc297937a220e0db.

Actions #14

Updated by Vincent MEMBRÉ about 6 years ago

  • Status changed from Pending release to Released

This bug has been fixed in Rudder 3.0.0, which was released on 2015/02/16

Actions #15

Updated by Alexis MOUSSET over 5 years ago

  • Related to Bug #7098: When syslog-ng is replaced by rsyslogd on SuSE, service is not restarted added
Actions

Also available in: Atom PDF