Rudder

I found the same status, on a node managed, An update component was on NoAnswer.

But when I ran the agent it produced a report, and now the component is in Success.

 >> Using command line specified bundlesequence
R: @@Common@@result_success@@hasPolicyServer-root@@common-root@@73@@Update@@None@@2013-06-03 13:56:12+02:00##37cc2d4e-d5f2-401e-af72-a7c44eca00ae@#Policy and dependencies already up to date. No action required.

The status is now Succes. I wonder why the update component did get the no answer status and I wonder if that does not come back.

The status was success for a little while and is now back in No Answer state.

Have you changed a Rule/Directive/group ? does a redeployment happened before having the NoAnswer ?

This is happening on all nodes ? or a part of them ? only to one ?

I didn't change a Rule/Directive/group lately - it happened to occur after the upgrade from 2.5 to 2.6. I don't know if a redeployment happened before having the NoAnswer. It's only happening on some nodes, not all.

Dennis Cabooter wrote:

I didn't change a Rule/Directive/group lately - it happened to occur after the upgrade from 2.5 to 2.6. I don't know if a redeployment happened before having the NoAnswer. It's only happening on some nodes, not all.

Ok, could you describe us the particularity of these nodes ? Are OS, OS version/Service Pack different from the other nodes ?
Are the system Techniques up to date ? (Techniques from /opt/rudder/share/techniques/system/ should be identical to /var/rudder/configuration-repository/system/)

Maybe we missed a case in the reporting of update.

  reports:
    server_ok::
      "@@Common@@log_repaired@@&TRACKINGKEY&@@Update@@None@@${g.execRun}##${g.uuid}@#Started the server (cf-serverd)";
    executor_ok::
      "@@Common@@log_repaired@@&TRACKINGKEY&@@Update@@None@@${g.execRun}##${g.uuid}@#Started the scheduler (cf-execd)";

    no_update::
      "@@Common@@result_error@@&TRACKINGKEY&@@Update@@None@@${g.execRun}##${g.uuid}@#Cannot update node's policy (CFEngine promises)";

    rudder_dependencies_update_error::
      "@@Common@@result_error@@&TRACKINGKEY&@@Update@@None@@${g.execRun}##${g.uuid}@#Cannot update dependencies";

    rudder_promises_generated_error::
      "@@Common@@result_error@@&TRACKINGKEY&@@Update@@None@@${g.execRun}##${g.uuid}@#Cannot update node's policy or dependencies";

    (rudder_promises_generated_ok|(rudder_dependencies_updated_ok.config_ok)).!(rudder_promises_generated_repaired|rudder_promises_generated_error|rudder_dependencies_updated|rudder_dependencies_update_error|config|no_update)::
      "@@Common@@result_success@@&TRACKINGKEY&@@Update@@None@@${g.execRun}##${g.uuid}@#Policy and dependencies already up to date. No action required.";

    rudder_dependencies_updated::
      "@@Common@@log_repaired@@&TRACKINGKEY&@@Update@@None@@${g.execRun}##${g.uuid}@#Dependencies updated";

    config::
      "@@Common@@log_repaired@@&TRACKINGKEY&@@Update@@None@@${g.execRun}##${g.uuid}@#Node's policy (CFEngine promises) updated";

    rudder_promises_generated_repaired|config|rudder_dependencies_updated|server_ok|executor_ok::
      "@@Common@@result_repaired@@&TRACKINGKEY&@@Update@@None@@${g.execRun}##${g.uuid}@#Policy or dependencies were updated or CFEngine service restarted";

    policy_server::
      "@@Common@@result_success@@&TRACKINGKEY&@@Update@@None@@${g.execRun}##${g.uuid}@#Policy server doesn't need to be updated";

I really suspect that some condition are missing here :

    (rudder_promises_generated_ok|(rudder_dependencies_updated_ok.config_ok)).!(rudder_promises_generated_repaired|rudder_promises_generated_error|rudder_dependencies_updated|rudder_dependencies_update_error|config|no_update)::
      "@@Common@@result_success@@&TRACKINGKEY&@@Update@@None@@${g.execRun}##${g.uuid}@#Policy and dependencies already up to date. No action required.";

Anyway, the reporting should indicate that the report is missing (unknwon reports), not showing a no answer. But that should be another bug.

If I run cf-agent -KI, Rudder reports success. However, later on it reports No Anwer again. Not on all nodes.

Dennis Cabooter wrote:

If I run cf-agent -KI, Rudder reports success. However, later on it reports No Anwer again. Not on all nodes.

Ok, but what are the difference between these nodes ? Are the OS identical ?

It happens on Ubuntu 12.04 Precise LTS and RHEL 5.x nodes.

It seems difficult to reproduce this bug but It could be possible that we have it in our lab.
We can't provoke it, so we will add some debugging messages and let it works some hours in observation.

We have notice that the difference between a Success and No Answer is really due to an absence of reporting from the component Update althought it is defined to be launched by the bundlesequence:

R: @@Common@@result_success@@hasPolicyServer-root@@common-root@@33@@Update@@None@@2013-06-05 12:56:00+02:00##36076ea3-281c-480e-be25-1392bfc39b71@#Policy and dependencies already up to date. No action required.
R: Class 'rudder_promises_generated_ok' exist now
R: Update is launched
R: @@Common@@log_info@@hasPolicyServer-root@@common-root@@33@@common@@StartRun@@2013-06-05 12:56:00+02:00##36076ea3-281c-480e-be25-1392bfc39b71@#Start execution
R: @@Common@@result_success@@hasPolicyServer-root@@common-root@@33@@Security parameters@@None@@2013-06-05 12:56:00+02:00##36076ea3-281c-480e-be25-1392bfc39b71@#The internal environment security is acceptable
R: @@Common@@result_success@@hasPolicyServer-root@@common-root@@33@@Red Button@@None@@2013-06-05 12:56:00+02:00##36076ea3-281c-480e-be25-1392bfc39b71@#Red Button is not in effect, continuing as normal...
R: @@Common@@result_success@@hasPolicyServer-root@@common-root@@33@@Process checking@@None@@2013-06-05 12:56:00+02:00##36076ea3-281c-480e-be25-1392bfc39b71@#There is an acceptable number of CFEngine processes running on the machine
R: @@Common@@result_success@@hasPolicyServer-root@@common-root@@33@@CRON Daemon@@None@@2013-06-05 12:56:00+02:00##36076ea3-281c-480e-be25-1392bfc39b71@#The CRON daemon is running
R: @@Common@@result_success@@hasPolicyServer-root@@common-root@@33@@Binaries update@@None@@2013-06-05 12:56:00+02:00##36076ea3-281c-480e-be25-1392bfc39b71@#The CFengine binaries in /var/rudder/cfengine-community/bin are up to date
R: @@Common@@log_info@@hasPolicyServer-root@@common-root@@33@@Log system for reports@@None@@2013-06-05 12:56:00+02:00##36076ea3-281c-480e-be25-1392bfc39b71@#Detected running syslog as syslog-ng
R: @@Common@@result_success@@hasPolicyServer-root@@common-root@@33@@Log system for reports@@None@@2013-06-05 12:56:00+02:00##36076ea3-281c-480e-be25-1392bfc39b71@#Logging system for report centralization is already correctly configured
[...]

The first component 'Update' is launched by failsafe, since cf-exec launch failsafe.cf AND promises.cf.

After forcing cf-execd to launch cf-agent in verbose mode, we have noticed that Update was launched but that no class is thrown:

[...]
rudder>    =========================================================
rudder>    reports in bundle update (1)
rudder>    =========================================================
[...]
rudder> Skipping whole next promise (@@Common@@result_error@@hasPolicyServer-root@@common-root@@33@@Update@@None@@2013-06-05 13:56:17+02:00##36076ea3-281c-480e-be25-1392bfc39b71@#Cannot update node's policy or dependencies), as context rudder_promises_generated_error is not relevant
rudder> . . . . . . . . . . . . . . . . . . . . . . . . . . . .
rudder>  XX Nothing promised here [last.update.reports.-centos-6-64.__Commo] (0/1 minutes elapsed)
rudder>
rudder> . . . . . . . . . . . . . . . . . . . . . . . . . . . .
rudder> Skipping whole next promise (@@Common@@log_repaired@@hasPolicyServer-root@@common-root@@33@@Update@@None@@2013-06-05 13:56:17+02:00##36076ea3-281c-480e-be25-1392bfc39b71@#Dependencies updated), as context rudder_dependencies_updated is not relevant
[...]
rudder>
rudder> . . . . . . . . . . . . . . . . . . . . . . . . . . . .
rudder> Skipping whole next promise (Class 'no_update' exist now), as context no_update is not relevant
rudder> . . . . . . . . . . . . . . . . . . . . . . . . . . . .
rudder>  XX Nothing promised here [last.update.reports.-centos-6-64.Class__] (0/1 minutes elapsed)
rudder>
rudder> . . . . . . . . . . . . . . . . . . . . . . . . . . . .
rudder> Skipping whole next promise (*********************************************************************************
* rudder-agent could not get an updated configuration from the policy server.   *
* This can be caused by a network issue, an unavailable server, or if this      *
* node was deleted from the Rudder root server.                                 *
* Any existing configuration policy will continue to be applied without change. *
*********************************************************************************), as context rudder_promises_generated_error|no_update is not relevant
[...]
rudder>
rudder>    =========================================================
rudder>    reports in bundle update (2)
rudder>    =========================================================
rudder>
rudder>
[...]
rudder> Skipping whole next promise (Class 'config' exist now), as context config is not relevant
rudder> . . . . . . . . . . . . . . . . . . . . . . . . . . . .
rudder>
rudder> . . . . . . . . . . . . . . . . . . . . . . . . . . . .
rudder> Skipping whole next promise (Class 'no_update' exist now), as context no_update is not relevant
rudder> . . . . . . . . . . . . . . . . . . . . . . . . . . . .
rudder>
rudder> . . . . . . . . . . . . . . . . . . . . . . . . . . . .
[...]

The must odd part is that we can see XX Nothing promised here [last.update.reports.-centos-6-64.Class_] (0/1 minutes elapsed)_ which seems to be related to lock files. Furthermore, this expression appear in place of a class expression which concern a negation.

Extract of CFEngine 3 code

[...]
    no_update::
      "Class 'no_update' exist now";

    !no_update::
      "Class 'no_update' does not exist now";
[...]

We should have had an output like this:

[...]
rudder> Skipping whole next promise (Class 'no_update' exist now), as context no_update is not relevant
rudder> . . . . . . . . . . . . . . . . . . . . . . . . . . . .
rudder>
rudder> . . . . . . . . . . . . . . . . . . . . . . . . . . . .
rudder>     Promise's handle:
rudder>     Promise made by: "Class 'no_update' does not exist now" 
rudder>     .........................................................
rudder>
rudder> Report: Class 'no_update' does not exist now
rudder> R: Class 'no_update' does not exist now
[...]

Instead of:

[...]
rudder> Skipping whole next promise (Class 'no_update' exist now), as context no_update is not relevant
rudder> . . . . . . . . . . . . . . . . . . . . . . . . . . . .
rudder>  XX Nothing promised here [last.update.reports.-centos-6-64.Class__] (0/1 minutes elapsed)
rudder>
rudder> . . . . . . . . . . . . . . . . . . . . . . . . . . . .
[...]

Finally, the second pass of update show that no more the class !no_update is checked:

[...]
rudder> Skipping whole next promise (Class 'no_update' exist now), as context no_update is not relevant
rudder> . . . . . . . . . . . . . . . . . . . . . . . . . . . .
rudder>
rudder> . . . . . . . . . . . . . . . . . . . . . . . . . . . .
rudder> Skipping whole next promise (*********************************************************************************
* rudder-agent could not get an updated configuration from the policy server.   *
* This can be caused by a network issue, an unavailable server, or if this      *
* node was deleted from the Rudder root server.                                 *
* Any existing configuration policy will continue to be applied without change. *
*********************************************************************************), as context rudder_promises_generated_error|no_update is not relevant
[...]

Oh, I thought I solved this issue before it was an issue by persisting the classes used by reporting. I must have missed something somehow

Ok, I guess I found the problem !
There is a lock, because the same reports are made both in failsafe and in update of regular promises; so on regular promises the reporting is not done, hence the no answer

Applied in changeset commit:bbe2d1816cc356350bcf3631c51e91abc41d92a2.

This bug has been fixed in Rudder 2.4.7, which was released today.
Check out:

• The release announcement: http://www.rudder-project.org/pipermail/rudder-announce/2013-July/000041.html
• The full ChangeLog: http://www.rudder-project.org/foswiki/bin/view/System/Documentation:ChangeLog24
• Download information: http://www.rudder-project.org/foswiki/Download/