Project

General

Profile

Actions

Bug #4281

closed

Rsyslog filters reports when too many reports arrive simultaneously

Added by Dennis Cabooter about 9 years ago. Updated almost 9 years ago.

Status:
Released
Priority:
1
Category:
Web - Compliance & node report
Target version:
Severity:
UX impact:
User visibility:
Effort required:
Priority:
Regression:

Description

Issue:
Logs from rsyslog are not stored in postgres

Rsyslog version:

# dpkg -l |grep rsyslog
ii  rsyslog                                5.8.6-1ubuntu8.6                  reliable system and kernel logging daemon
ii  rsyslog-pgsql                          5.8.6-1ubuntu8.6                  PostgreSQL output plugin for rsyslog

Postgres error log:

2013-12-19 11:16:09 CET ERROR:  invalid input syntax for integer: "**NO MATCH**" at character 228
2013-12-19 11:16:09 CET STATEMENT:  insert into RudderSysEvents (executionDate, nodeId, ruleId, directiveId, serial, Component, KeyValue, executionTimeStamp, eventType, msg, Policy) values ('2013-12-19T11:15:42.010588+01:00','**NO MATCH**', '**NO MATCH**' , '0', '**NO MATCH**', '**NO MATCH**', '**NO MATCH**', '**NO MATCH**', '**NO MATCH**', '**NO MATCH**', '**NO MATCH**' )

Solution:

# sed -i 's#$RepeatedMsgReduction on#$RepeatedMsgReduction off#' /etc/rsyslog.conf
# /etc/init.d/rsyslog restart


Related issues 2 (0 open2 closed)

Related to Rudder - Bug #6421: Messages can be dropped on the node, resulting in Unknown reports on the Web InterfaceReleasedJonathan CLARKEActions
Related to Rudder - Bug #8264: Disable Repeated message reduction or reports may be lost on the relayReleasedJonathan CLARKE2016-05-10Actions
Actions #1

Updated by Nicolas CHARLES about 9 years ago

This is a problem that impacts rsyslog 5.8.5 to 5.8.13.
The issue is that rsyslog wrongly considers that some lines are identical (when they are not) and creates line "last message repeated n times", without hostname
So it wrongly matches the regexp, and put invalid content in fields, that can't get to postgres, hence the insert error.
Since inserts are made in batch, it drops the batch

The workaround is to insert the following line on top of the file /etc/rsyslog.conf:

$RepeatedMsgReduction off

We need to enforce this within Rudder as well on policy server

Actions #2

Updated by Matthieu CERDA about 9 years ago

  • Category set to 39
  • Status changed from New to 8
  • Assignee set to Vincent MEMBRÉ
  • Priority changed from N/A to 1
  • Target version set to 2.6.10

This needs to be adressed, as it directly impacts Rudder's reporting. I think this is something we saw before, we need to check if it is not already corrected, and fix it if not.

Vince, since I'm not here, can you lend me an eye on this ?

Actions #3

Updated by Vincent MEMBRÉ about 9 years ago

  • Target version changed from 2.6.10 to 2.6.11
Actions #4

Updated by Vincent MEMBRÉ almost 9 years ago

  • Status changed from 8 to Pending technical review
  • Assignee changed from Vincent MEMBRÉ to Jonathan CLARKE
  • Pull Request set to https://github.com/Normation/rudder-techniques/pull/295

As suggested by Jonathan it was clever to edit /etc/rsyslog.d/rudder.conf to have that parameter only impact rudder related messages, not the whole configuration of rsyslog.

See http://stackoverflow.com/questions/20542758/rsyslog-conditional-repeatedmsgreduction for more details

PR is here: https://github.com/Normation/rudder-techniques/pull/295

Actions #5

Updated by Jonathan CLARKE almost 9 years ago

Vincent MEMBRÉ wrote:

As suggested by Jonathan it was clever to edit /etc/rsyslog.d/rudder.conf to have that parameter only impact rudder related messages, not the whole configuration of rsyslog.

See http://stackoverflow.com/questions/20542758/rsyslog-conditional-repeatedmsgreduction for more details

PR is here: https://github.com/Normation/rudder-techniques/pull/295

There is a misunderstanding here: what the docs say is that specifying "$RepeatedMsgReduction off" will be valid until the next definition of that parameter. This is in no way limited to a file. See the example in your link to clarify how this works.

This change will change this setting for all rsyslog configurations read in after the /etc/rsyslog.d/rudder.conf file.

I'm fine with this change, since we only apply it to the Rudder server, and it is necessary for Rudder, and not a very impacting change for the rest of syslog. But it is important to be clear about what we do, and don't change.

Actions #6

Updated by Jonathan CLARKE almost 9 years ago

  • Subject changed from Logs from rsyslog are not stored in postgres to Logs disappear with many simulateous messages because rsyslog wrongly "reduces" them
Actions #7

Updated by Vincent MEMBRÉ almost 9 years ago

I just say that it does not change the behavior of all other message treatment contrary to if we added it in /etc/rsyslog.conf.

More to that if it was only in /etc/rsyslog.conf it could be overrided by any other file in /etc/rsyslog.d (or any included file) before parsing rudder.conf leading to having no reports.

What would be great would be to restore the state of repeatedMsg at the end of rudder.conf, but I don't know to do it and if it's possible.

I think it was designed to switch the option when needed

Actions #8

Updated by Vincent MEMBRÉ almost 9 years ago

  • Status changed from Pending technical review to Pending release
  • % Done changed from 0 to 100

Applied in changeset policy-templates:commit:ec290e49842f33fa366312691b5dd0d6b2deeec4.

Actions #9

Updated by Vincent MEMBRÉ almost 9 years ago

Applied in changeset policy-templates:commit:c787c271deddeba0c2bdfbfeef7cb16d5d341992.

Actions #10

Updated by Vincent MEMBRÉ almost 9 years ago

  • Subject changed from Logs disappear with many simulateous messages because rsyslog wrongly "reduces" them to Rsyslog filters reports when too many reports arrive simultaneously
Actions #11

Updated by Vincent MEMBRÉ almost 9 years ago

  • Category changed from 39 to Web - Compliance & node report
Actions #12

Updated by Vincent MEMBRÉ almost 9 years ago

  • Status changed from Pending release to Released

This bug has been fixed in Rudder 2.6.11, which was released today.
Check out:

Actions #13

Updated by Nicolas CHARLES over 7 years ago

  • Related to Bug #6421: Messages can be dropped on the node, resulting in Unknown reports on the Web Interface added
Actions #14

Updated by Alexis Mousset over 6 years ago

  • Related to Bug #8264: Disable Repeated message reduction or reports may be lost on the relay added
Actions

Also available in: Atom PDF