Rudder

First question :

Do we need only one file for report logging (which would contain every kind of reports) or do we need multiple file (one file per report kind) ?

Having only one file makes a more complicated file to understand, but it also centralize history which is very important.

Second question :

What kind of reports should be in log file ?

In Rudder we have 6 kinds of reports: "Success","Repaired","Unknown","Error","Pending","NoAnswer"

Succes and Pending should not be in the log file

Error and Repaired surely shall be in it.

Unkown and NoAnswer should be in it too.

So the question is mainly about those to last type

We may also use raw report type : "log_warn","log_repaired","result_error","result_repaired". But I think they are less interesting than rudder types.

Last question :

There is different ways to implement that system :

First solution: rsyslog rule

• Write the log file directly thanks to rsyslog on Rudder server (and regexp that match expected reports).

This is the easiest way (just add a rsyslog rul), but we would get the raw report type (ie : log_warn, result_error, UUID only)

Second solution: select on existing table

• Select reports from the "RudderSysEvents" table (the table in the database containing all reports).
  • actually, we need to select only new reports from the last select, so we should keep track of what has already been inserted somewhere
• Process each report line, getting what is missing (names, etc)

Main advantage: it's really easy to do.
Main problem with that solution: the report database is huge and already a contention point, so adding big select on it may not be the wiser thing to do (but we don't have numbers to check that feeling).

We could have any kind of rudder reports, and they could be very readable for users (rule and directive name, node hostname).

Third solution: create a new "failed report" table in database

• Create a new "report logging" table in the database,
• Filter report from syslog and insert only whose matching an error or repaired (log_warn,log_repaired,result_error,result_repaired)
• Process that table every 5 minutes to get the latest reports, process them, add the result in the log file,
• Remove the reports from database (so that the disk space used, even without a vaccuum "full", stay low)

Advantages:

• ressource usage would be less than for the second solution,
• reports would be very readable for users (rule and directive name, node hostname)

Disavantage:

• that's a new data table, with a lot of cloned report line from syslog
• we will have only "error" and "repaired" status (so no possibility to calculate "no answer" lines)

Fourth solution: fully duplicate report table in database

• Insert all reports (even Success) in a new "report logging" table in the database, so that we actually have a full duplication of what is added in that table and in RudderSysEvents one,
• Process that table every 5 minutes and get the latest reports, process them.
• Then add them in the log file, and remove them.

Ressource usage would be less than second solution, and there would be every kind of reports (so that we will be able to calculate "no answer" if we want)
Reports would be very readable for users (rule and directive name, node hostname)

I'll go for the fourth solution
This may be more complicated, but this solution offers the best result (every kind of reports, readable reports) without having too much impact on the system (not handling the huuuuuge "RudderSysEvents" table.

So which one do you consider the best ?

There's also some questions about the format we want for those logs

For the moment, my logger format is :

LOG_DATE LOGGER_HOSTNAME rudder-reports[ LOGGER_NAME ]: [ LOG_LEVEL ] Report sent by node ' NODE_HOSTNAME ' executed on EXECUTION_DATE for Rule ' RULE_NAME ' is in status ' REPORT_STATUS ' on directive ' DIRECTIVE_NAME ', component ' COMPONENT_NAME ', value ' COMPONENT_KEY ' with message : ' REPORT_MESSAGE '

nov. 16 15:56:22 mutex rudder-reports[repaired]: [WARN]  Report sent by node 'server.rudder.local' executed on 2012-11-16 15:55:30 for Rule 'distributePolicy' is in status 'result_repaired' on directive 'Distribute Policy', component 'Check WebDAV credentials', value 'None' with message : 'The Rudder WebDAV user and password were updated'

LOGGER NAME / REPORT STATUS / LOG LEVEL are linked. They are organized like that for the moment.
error / result_error / ERROR
repaired / result_repaired / WARN
log / log_warn/repair / INFO

May be some log should be treated as DEBUG or TRACE Level

For this version solution kept will be :

One log file
With raw reports status : result_error, result_repaired, log_repaired, log_warn
Will not create a new table and use RuddersysEvents table to log reports
Will not analyse reports

Log format is [Execution date] N:nodeId(Node hostname) S:report_status R:ruleId(rule name) D:directiveId(directive name) C:component name V:keyValue message

here is an example log :

[20/nov./2012 09:41 +0100] N:root(server.rudder.local) S:result_repaired R:hasPolicyServer-root(Rudder system policy: basic setup (common)) D:common-root(Common) C:Security parameters V:None Some internal security parameters were adjusted
[20/nov./2012 09:41 +0100] N:root(server.rudder.local) S:result_repaired R:root-DP(distributePolicy) D:root-distributePolicy(Distribute Policy) C:Check WebDAV credentials V:None The Rudder WebDAV user and password were updated
[20/nov./2012 09:45 +0100] N:root(server.rudder.local) S:result_repaired R:hasPolicyServer-root(Rudder system policy: basic setup (common)) D:common-root(Common) C:Security parameters V:None Some internal security parameters were adjusted
[20/nov./2012 09:45 +0100] N:root(server.rudder.local) S:result_repaired R:root-DP(distributePolicy) D:root-distributePolicy(Distribute Policy) C:Check WebDAV credentials V:None The Rudder WebDAV user and password were updated
[20/nov./2012 09:46 +0100] N:8fee32c0-00c8-4ae8-8597-5a54a9b6a523(node1.rudder.local) S:result_error R:d0c4a57e-4715-467b-a036-a907c0d9deff(Rule42) D:87097897-95cc-4868-ab39-690b6d11076d(apache) C:apacheServer V:None Could not restart Apache HTTPD
[20/nov./2012 09:46 +0100] N:8fee32c0-00c8-4ae8-8597-5a54a9b6a523(node1.rudder.local) S:result_error R:d0c4a57e-4715-467b-a036-a907c0d9deff(Rule42) D:87097897-95cc-4868-ab39-690b6d11076d(apache) C:apacheServer V:None Apache binary is not present. Something is wrong (installation failure ?)

This is now awaiting the Technical Review on #3018 (development). Integration is all done.