Rudder

I have one example:

1. My promises on a new agent (cf_lock ~ 1Mb), takes 10 seconds to run, no problems on reporting or whatever
2. With time, cf_lock grows big ! ( ~140 Mb) and the agent
   • when ran by cf-execd now takes 4 minutes,
   • reporting was ok during the last minute of before the next agent run
   • manually it stays quite fast ( ~ 20 seconds).
3. Waiting a little, more (in my case there it was at ~155 Mb),
   • when cf-agent is ran by cf-execd it now takes 11 minutes,
   • Reporting was never Ok, always some missing reports
   • and manual run takes 4 minutes

In all cases, the last_successful_update file is updated correctly, even with the 11 minutes.

I don't have agent piling in the two first cases, only the last one is causing piling up (and there was only 4 at the same time, so quite far from 8 agents)

In all cases, cfagent uses 100% cpu during the whole duration of the execution, leading in the last case in the usage of four cpus, impacting my other applications (maybe it's why the agent manually run cannot work correctly...)

As said in #4752, coredumb and dnns, are using a cron to check if the size of that file is over 10Mb, and if over, clean (rm) the whole state directory.

They don't have any problems since, and they haven't met any downside using it.

Maybe the size, and the method to clean are not the good one, but i clearly think this is the idea we should add.

About the size: According to coredumb, agent start to slow at 10Mb, and at 100Mb its always slow.

I don't know if there is a strict rule, of if it gets slower randomly ... What i have seen is: the more you have, the slower it can be

Some datas, maybe need to be confirmed

• <1Mb => 10 seconds
• 35 Mb: ~ 1 minute
• 140 Mb: 4 minutes
• 150 Mb: 11 minutes (maybe some corruption occured here ?)
• 230Mb: 10 minutes

About the clean methods:

• I don't think we should delete the whole state dir.
• I tried using 'tchmgr optimize', but that changed nothing at all (kept the same size, same time), same with -df option
• I would only delete cf_lock.tcdb files ( that file and tcdb.lock file)

Another lead would be to use let cfengine optimize itself tcdb using TCDB_OPTIMIZE_PERCENT varaible.

That variable will make cfengine check if it has to optimize tcdb, https://github.com/cfengine/core/blob/master/libpromises/dbm_tokyocab.c#L128

To use a cron to delete cf_lock when over 10Mb

put in file /etc/cron.d/clean-rudder-locks

*/5 * * * * root if [ `ls -l /var/rudder/cfengine-community/state/cf_lock.tcdb | cut -f 5 -d " "` -gt "10485760" ]; then rm -rf /var/rudder/cfengine-community/state/cf_lock.tcdb* && /opt/rudder/bin/cf-agent -KI > /dev/null 2>&1; fi

The rudder-clean works correctly on servers.

I have some nodes with TCDB_OPTIMIZE_PERCENT set to 25

my tcdb were reduced from 40Mb to 1Mb, (execution time lowered from 1 minute to 10 seconds!) and now tcdb is growing but really slowly. (from 1Mb to 1.9 in few hours, it was 0.15Mb by run before)

So I still need to look for the behavior in the long run.

The variable does not prevent the error with cf_lock growing big... The growth rate is slower (1Mb in 45 min, instead of 30 minutes), when the file gets bug (over 10Mb we get some errors too)

The check on the size seems the only solution to me, maybe helped by the variable

The impact of killing the cf_lock database is that all locks on promises are removed, meaning everything that uses ifelapsed may fail.
It does not have any impact on persistent classes, nor package list.

Applied in changeset packages:commit:0d10ba0561ec64aae16a359102b0c1d498926887.