Rudder

Ha. I've been able to reproduce this bug.
It happens when the promises are invalid AND the cfengine databases (BDB) is corrupted (most likely because it has been killed) (and that would explains why files are missing : killed during a copy)

The failsafe doesn't starts, and when running manually the agent, the output contains

/var/rudder/cfengine-community/state/cf_lock.db page 14599 is on free list with type 5
PANIC: Invalid argument
rudder> BDB_WriteComplexKeyDB: Error trying to write database: DB_RUNRECOVERY: Fatal error, run database recovery
PANIC: fatal region error detected; run recovery
rudder> BDB_CloseDB: Unable to close database: DB_RUNRECOVERY: Fatal error, run database recovery
rudder> CloseDB: Could not close DB handle.
rudder> CloseDB: Trying to remove handle from open pool anyway.

Killing the .db files solve the issue, but it's inconvenient

I'm afraid we'll have to live with this until we move to a version of CFEngine with a correct database

Nicolas CHARLES wrote:

Jon, I'm not really sure of what to do, since there are no obvious solution ...

I think we should have an external test (ie, in cron) that detects this case and kill the BDB database if necessary. This would avoid having to have manual intervention to repair things, and is only a temporary workaround until we have a version of CFengine that can use something other than BDB.

Can you please explain the steps to reproduce the situation, so that we can see what error code cf-agent returns and see if we can build something from that?

To reproduce it, you need to kill -9 a cf-agent while it is running, and all the cfengine component, and then delete a promise file (like userManagement/1.0/userManagement.cf)
Do it several time if you are not sure.
Then the cf-execd won't be able to restart

/var/rudder/cfengine-community/bin/cf-execd
Can't stat file "/var/rudder/cfengine-community/inputs/userManagement/1.0/userManagement.cf" for parsing
 !!! System error for stat: "No such file or directory" 

The error code is 1
You can test this on debian-5-32.labo.normation.com

Thinking of that, the executor daemon is not supposed to fire the failsafe if something is wrong in the promises ... shouldn't we execute cf-agent is cf-execd is not running ???

I've had a look at this, and cf-execd and cf-agent both have the decency to exit with error code 1 when this happens. Unless you run failsafe.cf, in which case they block...

But actually, cf-agent only blocks when it gets to the part about downloading files:

rudder> *****************************************************************
rudder> BUNDLE update
rudder> *****************************************************************
rudder> 
rudder> 
rudder>    =========================================================
rudder>    vars in bundle update (1)
rudder>    =========================================================
rudder> 
rudder> 
rudder> . . . . . . . . . . . . . . . . . . . . . . . . . . . .
rudder> Skipping whole next promise (server_inputs), as context nova_edition is not relevant
rudder> . . . . . . . . . . . . . . . . . . . . . . . . . . . .
rudder> 
rudder>      +  Private classes augmented:
rudder> 
rudder>      -  Private classes diminished:
rudder> 
rudder> 
rudder> 
rudder>    =========================================================
rudder>    files in bundle update (1)
rudder>    =========================================================
rudder> 
rudder> 
rudder>     .........................................................
rudder>     Promise handle: 
rudder>     Promise made by: /var/rudder/cfengine-community/inputs
rudder>     .........................................................
rudder> 

^Crudder> Received signal 2 (SIGINT) while doing []
rudder> Logical start time Mon Sep 24 16:28:20 2012
rudder> This sub-task started really at Mon Sep 24 16:28:21 2012
rudder> Trying to remove lock - try
rudder> Can't open lock-log file
rudder>  !!! System error for fopen: "No such file or directory" 

It actually executes the previous bundle just fine. So maybe we could implement a promise in another bundle that would be executed first, to somehow detect the corruption, and clean up the BerkeleyDB files... And then run "cf-agent -f failsafe.cf && cf-agent" via cron, instead of cf-execd.

What do you think?

If there is some way to detect the corruption and clean them with the agent, that would be awesome.
And I totally agree that the cron should not run the executor, rather "cf-agent -f failsafe && cf-agent" if the executor is not running, and the agent would run the executor, which would in turn run the agent

Jonathan CLARKE wrote:

Here's an idea:

• We change the cronjob that tests if cf-execd is running to launch "cf-agent -f failsafe.cf && cf-agent" instead of launching cf-execd. This gives the failsafe.cf the chance to repair any problems with the promises files that might stop cf-execd from running.
• Every time we correctly update the promises, we touch a file in /var/rudder/cfengine-community
• The first action in the failsafe would be to check the file above, and if it is older than, say, 1 hour, we remove cf_lock.db (and only this DB), to give CFEngine a chance to run properly again

The logic has been implemented.

I didn't successfully reproduce the bug with the error message about cf_lock.db but I have a locked state while lauching cf-agent at the step "startExecution". Our logic has unlocked the state and all has been returning to normal state.

I suppose we can consider the issue as fixed ?