Bug #4281
closed
Rsyslog filters reports when too many reports arrive simultaneously
Description
Issue:
Logs from rsyslog are not stored in postgres
Rsyslog version:
# dpkg -l |grep rsyslog ii rsyslog 5.8.6-1ubuntu8.6 reliable system and kernel logging daemon ii rsyslog-pgsql 5.8.6-1ubuntu8.6 PostgreSQL output plugin for rsyslog
Postgres error log:
2013-12-19 11:16:09 CET ERROR: invalid input syntax for integer: "**NO MATCH**" at character 228
2013-12-19 11:16:09 CET STATEMENT: insert into RudderSysEvents (executionDate, nodeId, ruleId, directiveId, serial, Component, KeyValue, executionTimeStamp, eventType, msg, Policy) values ('2013-12-19T11:15:42.010588+01:00','**NO MATCH**', '**NO MATCH**' , '0', '**NO MATCH**', '**NO MATCH**', '**NO MATCH**', '**NO MATCH**', '**NO MATCH**', '**NO MATCH**', '**NO MATCH**' )
Solution:
# sed -i 's#$RepeatedMsgReduction on#$RepeatedMsgReduction off#' /etc/rsyslog.conf # /etc/init.d/rsyslog restart
Updated by Nicolas CHARLES over 10 years ago
This is a problem that impacts rsyslog 5.8.5 to 5.8.13.
The issue is that rsyslog wrongly considers that some lines are identical (when they are not) and creates line "last message repeated n times", without hostname
So it wrongly matches the regexp, and put invalid content in fields, that can't get to postgres, hence the insert error.
Since inserts are made in batch, it drops the batch
The workaround is to insert the following line on top of the file /etc/rsyslog.conf:
$RepeatedMsgReduction off
We need to enforce this within Rudder as well on policy server
Updated by Matthieu CERDA over 10 years ago
- Category set to 39
- Status changed from New to 8
- Assignee set to Vincent MEMBRÉ
- Priority changed from N/A to 1
- Target version set to 2.6.10
This needs to be adressed, as it directly impacts Rudder's reporting. I think this is something we saw before, we need to check if it is not already corrected, and fix it if not.
Vince, since I'm not here, can you lend me an eye on this ?
Updated by Vincent MEMBRÉ over 10 years ago
- Target version changed from 2.6.10 to 2.6.11
Updated by Vincent MEMBRÉ about 10 years ago
- Status changed from 8 to Pending technical review
- Assignee changed from Vincent MEMBRÉ to Jonathan CLARKE
- Pull Request set to https://github.com/Normation/rudder-techniques/pull/295
As suggested by Jonathan it was clever to edit /etc/rsyslog.d/rudder.conf to have that parameter only impact rudder related messages, not the whole configuration of rsyslog.
See http://stackoverflow.com/questions/20542758/rsyslog-conditional-repeatedmsgreduction for more details
PR is here: https://github.com/Normation/rudder-techniques/pull/295
Updated by Jonathan CLARKE about 10 years ago
Vincent MEMBRÉ wrote:
As suggested by Jonathan it was clever to edit /etc/rsyslog.d/rudder.conf to have that parameter only impact rudder related messages, not the whole configuration of rsyslog.
See http://stackoverflow.com/questions/20542758/rsyslog-conditional-repeatedmsgreduction for more details
PR is here: https://github.com/Normation/rudder-techniques/pull/295
There is a misunderstanding here: what the docs say is that specifying "$RepeatedMsgReduction off" will be valid until the next definition of that parameter. This is in no way limited to a file. See the example in your link to clarify how this works.
This change will change this setting for all rsyslog configurations read in after the /etc/rsyslog.d/rudder.conf file.
I'm fine with this change, since we only apply it to the Rudder server, and it is necessary for Rudder, and not a very impacting change for the rest of syslog. But it is important to be clear about what we do, and don't change.
Updated by Jonathan CLARKE about 10 years ago
- Subject changed from Logs from rsyslog are not stored in postgres to Logs disappear with many simulateous messages because rsyslog wrongly "reduces" them
Updated by Vincent MEMBRÉ about 10 years ago
I just say that it does not change the behavior of all other message treatment contrary to if we added it in /etc/rsyslog.conf.
More to that if it was only in /etc/rsyslog.conf it could be overrided by any other file in /etc/rsyslog.d (or any included file) before parsing rudder.conf leading to having no reports.
What would be great would be to restore the state of repeatedMsg at the end of rudder.conf, but I don't know to do it and if it's possible.
I think it was designed to switch the option when needed
Updated by Vincent MEMBRÉ about 10 years ago
- Status changed from Pending technical review to Pending release
- % Done changed from 0 to 100
Applied in changeset policy-templates:commit:ec290e49842f33fa366312691b5dd0d6b2deeec4.
Updated by Vincent MEMBRÉ about 10 years ago
Applied in changeset policy-templates:commit:c787c271deddeba0c2bdfbfeef7cb16d5d341992.
Updated by Vincent MEMBRÉ about 10 years ago
- Subject changed from Logs disappear with many simulateous messages because rsyslog wrongly "reduces" them to Rsyslog filters reports when too many reports arrive simultaneously
Updated by Vincent MEMBRÉ about 10 years ago
- Category changed from 39 to Web - Compliance & node report
Updated by Vincent MEMBRÉ about 10 years ago
- Status changed from Pending release to Released
This bug has been fixed in Rudder 2.6.11, which was released today.
Check out:
- The release announcement: http://www.rudder-project.org/pipermail/rudder-announce/2014-March/000077.html
- The full ChangeLog: http://www.rudder-project.org/foswiki/bin/view/System/Documentation:ChangeLog26
- Download information: https://www.rudder-project.org/site/get-rudder/downloads/
Updated by Nicolas CHARLES almost 9 years ago
- Related to Bug #6421: Messages can be dropped on the node, resulting in Unknown reports on the Web Interface added
Updated by Alexis Mousset almost 8 years ago
- Related to Bug #8264: Disable Repeated message reduction or reports may be lost on the relay added