Bug #4281: Rsyslog filters reports when too many reports arrive simultaneously - Rudder - Issue Tracker

Actions

Copy link

Bug #4281

closed

Rsyslog filters reports when too many reports arrive simultaneously

Added by Dennis Cabooter about 11 years ago. Updated almost 11 years ago.

Status:

Released

Priority:

1 (highest)

Assignee:

Jonathan CLARKE

Category:

Web - Compliance & node report

Target version:

2.6.11

Pull Request:

https://github.com/Normation/rudder-t...

Severity:

UX impact:

User visibility:

Effort required:

Priority:

Name check:

Fix check:

Regression:

Description

Issue:
Logs from rsyslog are not stored in postgres

Rsyslog version:

# dpkg -l |grep rsyslog
ii  rsyslog                                5.8.6-1ubuntu8.6                  reliable system and kernel logging daemon
ii  rsyslog-pgsql                          5.8.6-1ubuntu8.6                  PostgreSQL output plugin for rsyslog

Postgres error log:

2013-12-19 11:16:09 CET ERROR:  invalid input syntax for integer: "**NO MATCH**" at character 228
2013-12-19 11:16:09 CET STATEMENT:  insert into RudderSysEvents (executionDate, nodeId, ruleId, directiveId, serial, Component, KeyValue, executionTimeStamp, eventType, msg, Policy) values ('2013-12-19T11:15:42.010588+01:00','**NO MATCH**', '**NO MATCH**' , '0', '**NO MATCH**', '**NO MATCH**', '**NO MATCH**', '**NO MATCH**', '**NO MATCH**', '**NO MATCH**', '**NO MATCH**' )

Solution:

# sed -i 's#$RepeatedMsgReduction on#$RepeatedMsgReduction off#' /etc/rsyslog.conf
# /etc/init.d/rsyslog restart

Related issues 2 (0 open — 2 closed)

Actions

Copy link

Updated by Nicolas CHARLES about 11 years ago

This is a problem that impacts rsyslog 5.8.5 to 5.8.13.
The issue is that rsyslog wrongly considers that some lines are identical (when they are not) and creates line "last message repeated n times", without hostname
So it wrongly matches the regexp, and put invalid content in fields, that can't get to postgres, hence the insert error.
Since inserts are made in batch, it drops the batch

The workaround is to insert the following line on top of the file /etc/rsyslog.conf:

$RepeatedMsgReduction off

We need to enforce this within Rudder as well on policy server

Actions

Copy link

Updated by Matthieu CERDA about 11 years ago

Category set to 39
Status changed from New to 8
Assignee set to Vincent MEMBRÉ
Priority changed from N/A to 1 (highest)
Target version set to 2.6.10

This needs to be adressed, as it directly impacts Rudder's reporting. I think this is something we saw before, we need to check if it is not already corrected, and fix it if not.

Vince, since I'm not here, can you lend me an eye on this ?

Actions

Copy link

Updated by Vincent MEMBRÉ almost 11 years ago

Target version changed from 2.6.10 to 2.6.11

Actions

Copy link

Updated by Vincent MEMBRÉ almost 11 years ago

Status changed from 8 to Pending technical review
Assignee changed from Vincent MEMBRÉ to Jonathan CLARKE
Pull Request set to https://github.com/Normation/rudder-techniques/pull/295

As suggested by Jonathan it was clever to edit /etc/rsyslog.d/rudder.conf to have that parameter only impact rudder related messages, not the whole configuration of rsyslog.

See http://stackoverflow.com/questions/20542758/rsyslog-conditional-repeatedmsgreduction for more details

PR is here: https://github.com/Normation/rudder-techniques/pull/295

Actions

Copy link

Updated by Jonathan CLARKE almost 11 years ago

Vincent MEMBRÉ wrote:

As suggested by Jonathan it was clever to edit /etc/rsyslog.d/rudder.conf to have that parameter only impact rudder related messages, not the whole configuration of rsyslog.

See http://stackoverflow.com/questions/20542758/rsyslog-conditional-repeatedmsgreduction for more details

PR is here: https://github.com/Normation/rudder-techniques/pull/295

There is a misunderstanding here: what the docs say is that specifying "$RepeatedMsgReduction off" will be valid until the next definition of that parameter. This is in no way limited to a file. See the example in your link to clarify how this works.

This change will change this setting for all rsyslog configurations read in after the /etc/rsyslog.d/rudder.conf file.

I'm fine with this change, since we only apply it to the Rudder server, and it is necessary for Rudder, and not a very impacting change for the rest of syslog. But it is important to be clear about what we do, and don't change.

Actions

Copy link

Updated by Jonathan CLARKE almost 11 years ago

Subject changed from Logs from rsyslog are not stored in postgres to Logs disappear with many simulateous messages because rsyslog wrongly "reduces" them

Actions

Copy link

Updated by Vincent MEMBRÉ almost 11 years ago

I just say that it does not change the behavior of all other message treatment contrary to if we added it in /etc/rsyslog.conf.

More to that if it was only in /etc/rsyslog.conf it could be overrided by any other file in /etc/rsyslog.d (or any included file) before parsing rudder.conf leading to having no reports.

What would be great would be to restore the state of repeatedMsg at the end of rudder.conf, but I don't know to do it and if it's possible.

I think it was designed to switch the option when needed

Actions

Copy link

Updated by Vincent MEMBRÉ almost 11 years ago

Status changed from Pending technical review to Pending release
% Done changed from 0 to 100

Applied in changeset policy-templates:commit:ec290e49842f33fa366312691b5dd0d6b2deeec4.

Actions

Copy link

Updated by Vincent MEMBRÉ almost 11 years ago

Applied in changeset policy-templates:commit:c787c271deddeba0c2bdfbfeef7cb16d5d341992.

Actions

Copy link

#10

Updated by Vincent MEMBRÉ almost 11 years ago

Subject changed from Logs disappear with many simulateous messages because rsyslog wrongly "reduces" them to Rsyslog filters reports when too many reports arrive simultaneously

Actions

Copy link

#11