Project

General

Profile

Bug #5775

Messages sent from an AIX machine via a relay are not forwarded correctly

Added by Matthieu CERDA over 5 years ago. Updated over 5 years ago.

Status:
Released
Priority:
1
Category:
System techniques
Target version:
Severity:
User visibility:
Effort required:
Priority:

Description

AIX does something more compared to the other syslog implementations: it adds a tiny "Message forwarded from <hostname>" to the syslog message, making it non-parseable for rsyslog.

It breaks the relay servers.

We should add the following configuration to rsyslog on relays/root servers:

# for AIX clients
# (strips "Message forwarded from" from the message)
$ModLoad pmaixforwardedfrom
$RulesetParser rsyslog.aixforwardedfrom
$RulesetParser rsyslog.rfc5424
$RulesetParser rsyslog.rfc3164

#1

Updated by Matthieu CERDA over 5 years ago

  • Status changed from In progress to Pending technical review
  • Assignee changed from Matthieu CERDA to Jonathan CLARKE
  • % Done changed from 0 to 100
  • Pull Request set to https://github.com/Normation/rudder-techniques/pull/559
#2

Updated by Nicolas CHARLES over 5 years ago

Thumb up !

#3

Updated by Matthieu CERDA over 5 years ago

  • Pull Request changed from https://github.com/Normation/rudder-techniques/pull/559 to https://github.com/Normation/rudder-techniques/pull/560
#4

Updated by Jonathan CLARKE over 5 years ago

  • Status changed from Pending technical review to Discussion
  • Assignee changed from Jonathan CLARKE to Matthieu CERDA

As requested in this new PR, this needs a lot more explaining. To start with, the ticket description includes a proposed solution, which is always a bad sign, because the explanation of why and how this problem occurs had been glossed over.

Please explain exactly what happens and why this is a problem. So, there's a prefix added to log lines. What didn't like that? A regex in our syslog config? Psql? How come it only affects relays and not the root server?

Once we've covered the "why", let's start looking at solutions. It looks to me like this could potentially be fixed in any one of three places (on the node that adds the prefix, on the relay or on the root server). Please explore these options and explain if/if not possible and why the solution you propose is best.

Last but not least, what is a simple workaround?

#5

Updated by Matthieu CERDA over 5 years ago

  • Status changed from Discussion to Pending technical review
  • Assignee changed from Matthieu CERDA to Jonathan CLARKE
Basically, the fact is:
  • The AIX syslog version, by default, prefixes the forwarded syslog messages with "Message forwarded from <machine hostname>:"
  • This causes no problem on Rudder server, as on it we only rely on the content of the message and discard what's before.

But, on a relay server, we only forward messages identified by the "rudder' program name.

A normal syslog message is:
facility.priority date program: message

A forwarded AIX one is:
facilty.priority date Message forwarded from foobar: program: message

Basically, what's happens then is that rsyslog on the relay tries to parse the message to get the program name, and understands that the program name is "forwarded". We only want "rudder", so rsyslog skips the send to remote syslog step, and worse, stores the message locally instead (like any other message by default).

There are basically three solutions:
  • Stop the AIX syslog from doing this (-n argument)
    • We cannot just ask people to change their syslog default arguments, or worse, do it ourselves sneakily
  • Use a rsyslog module that does it automatically
    • There is the pmaixforwardedfrom.so module, but it is far from compiled everywhere, especially on RHEL ...
  • Be a little more generic about which kind of messages we want to forward
    • This is the safest and most efficient method: try to match more messages

What I do here is say to rsyslog "forward any message that has the program name "rudder" OR contains "rudder" in the message, thus enabling rsyslog to forward them without issue.

#6

Updated by Jonathan CLARKE over 5 years ago

  • Status changed from Pending technical review to Discussion
  • Assignee changed from Jonathan CLARKE to Matthieu CERDA

Thank you for clarifying this Matthieu. This makes this bug easy to understand, and following your analysis is simple. Now I can help.

I understand and confirm your analysis of the problem. However, there seems to be one thing missing: if "it works" on the root server, why don't we make the relay config for rsyslog more like the root server config for rsyslog?

In particular, I note that the root server config doesn't check the programname at all, it just uses a (admittedly complex) regex. Shouldn't we be doing the same on the relay, for consistency? And if we need to change the programname check on the relay, I also note that the root server config ends with a check on the programname (to discard logs from being logged elsewhere) - surely that should be changed also.

#7

Updated by Matthieu CERDA over 5 years ago

  • Status changed from Discussion to In progress
  • % Done changed from 100 to 80

Well, your idea is good, and would be less surprising when maintenance is needed, and more precise about what we forward.

Let's do it !

#8

Updated by Matthieu CERDA over 5 years ago

  • Status changed from In progress to Pending technical review
  • Assignee changed from Matthieu CERDA to Jonathan CLARKE
  • % Done changed from 80 to 100

Actually, I used a mix of my idea and your suggestion, to make sure every remaining Rudder log (not matching the report formar) sent to a relay OR a root server get dropped, since even with the report filtering, regular CFEngine warnings would still pass through (when started using -KI for example ...)

Example:

2014-11-21T17:10:54+0100   notice: R: @@DistributePolicy@@result_success@@root-DP@@root-distributePolicy@@162@@Check WebDAV credentials@@None@@2014-11-21 17:10:53+01:00##root@#The Rudder WebDAV user and password are OK

=> Would not pass through and be sent directly to the database, or forwarded to the root if we are a relay

2014-11-21T17:10:54+0100     info: Executing 'no timeout' ... '/usr/bin/curl -s http://localhost:8080/endpoint/api/status |/bin/grep -q OK'

=> Would not be sent to the DB, but would still be stored in the local log files since they do not match the report regex

Thus I added a condition to make sure that "every remaining log, rudder related message or rudder AIX log with "<forwarded> from XXX: rudder:" in it would get dropped.

Using this, we make sure NOTHING we do not want actually pass through (good for hdd / ssd storage - constrained environments.

PR updated !

#9

Updated by Matthieu CERDA over 5 years ago

  • Status changed from Pending technical review to Pending release

Applied in changeset policy-templates:commit:5f1a4782785521bde9d293de9f43ebd2427b521b.

#10

Updated by Jonathan CLARKE over 5 years ago

Applied in changeset policy-templates:commit:61d3f594fa648ee7b2e1a367974e7564213fd85a.

#11

Updated by Vincent MEMBRÉ over 5 years ago

  • Status changed from Pending release to Released

This bug has been fixed in Rudder 2.10.8 and 2.11.5, which were released today (16/12/14)

Also available in: Atom PDF