Rudder

PR is ready: https://github.com/Normation/rudder-techniques/pull/559

PR has been replaced :) https://github.com/Normation/rudder-techniques/pull/560

As requested in this new PR, this needs a lot more explaining. To start with, the ticket description includes a proposed solution, which is always a bad sign, because the explanation of why and how this problem occurs had been glossed over.

Please explain exactly what happens and why this is a problem. So, there's a prefix added to log lines. What didn't like that? A regex in our syslog config? Psql? How come it only affects relays and not the root server?

Once we've covered the "why", let's start looking at solutions. It looks to me like this could potentially be fixed in any one of three places (on the node that adds the prefix, on the relay or on the root server). Please explore these options and explain if/if not possible and why the solution you propose is best.

Last but not least, what is a simple workaround?

• The AIX syslog version, by default, prefixes the forwarded syslog messages with "Message forwarded from <machine hostname>:"
• This causes no problem on Rudder server, as on it we only rely on the content of the message and discard what's before.

But, on a relay server, we only forward messages identified by the "rudder' program name.

A normal syslog message is:
facility.priority date program: message

A forwarded AIX one is:
facilty.priority date Message forwarded from foobar: program: message

Basically, what's happens then is that rsyslog on the relay tries to parse the message to get the program name, and understands that the program name is "forwarded". We only want "rudder", so rsyslog skips the send to remote syslog step, and worse, stores the message locally instead (like any other message by default).

• Stop the AIX syslog from doing this (-n argument)
  • We cannot just ask people to change their syslog default arguments, or worse, do it ourselves sneakily
• Use a rsyslog module that does it automatically
  • There is the pmaixforwardedfrom.so module, but it is far from compiled everywhere, especially on RHEL ...
• Be a little more generic about which kind of messages we want to forward
  • This is the safest and most efficient method: try to match more messages

What I do here is say to rsyslog "forward any message that has the program name "rudder" OR contains "rudder" in the message, thus enabling rsyslog to forward them without issue.

Thank you for clarifying this Matthieu. This makes this bug easy to understand, and following your analysis is simple. Now I can help.

I understand and confirm your analysis of the problem. However, there seems to be one thing missing: if "it works" on the root server, why don't we make the relay config for rsyslog more like the root server config for rsyslog?

In particular, I note that the root server config doesn't check the programname at all, it just uses a (admittedly complex) regex. Shouldn't we be doing the same on the relay, for consistency? And if we need to change the programname check on the relay, I also note that the root server config ends with a check on the programname (to discard logs from being logged elsewhere) - surely that should be changed also.

Well, your idea is good, and would be less surprising when maintenance is needed, and more precise about what we forward.

Let's do it !

Actually, I used a mix of my idea and your suggestion, to make sure every remaining Rudder log (not matching the report formar) sent to a relay OR a root server get dropped, since even with the report filtering, regular CFEngine warnings would still pass through (when started using -KI for example ...)

Example:

2014-11-21T17:10:54+0100   notice: R: @@DistributePolicy@@result_success@@root-DP@@root-distributePolicy@@162@@Check WebDAV credentials@@None@@2014-11-21 17:10:53+01:00##root@#The Rudder WebDAV user and password are OK

2014-11-21T17:10:54+0100     info: Executing 'no timeout' ... '/usr/bin/curl -s http://localhost:8080/endpoint/api/status |/bin/grep -q OK'

Thus I added a condition to make sure that "every remaining log, rudder related message or rudder AIX log with "<forwarded> from XXX: rudder:" in it would get dropped.

Using this, we make sure NOTHING we do not want actually pass through (good for hdd / ssd storage - constrained environments.

PR updated !

Applied in changeset policy-templates:commit:5f1a4782785521bde9d293de9f43ebd2427b521b.

This bug has been fixed in Rudder 2.10.8 and 2.11.5, which were released today (16/12/14)

• Announcement 2.10 2.11
• Changelog 2.10 2.11
• Download information: https://www.rudder-project.org/site/get-rudder/downloads/