Actions
Bug #22442
open
Could not open database txn cf_state.lmdb: Invalid argument
Pull Request:
Severity:
Major - prevents use of part of Rudder | no simple workaround
UX impact:
User visibility:
Effort required:
Priority:
0
Regression:
No
Description
Hi,
after upgrading Rudder from 6.2.13 to 7.2.4 we found the following cf-agent errors in our logs.
This happend on CentOS 7/RHEL 7, SLES12 SP5, SLES15 SP3, SLES15 SP4
[tux@linux ~]$ sudo journalctl --priority err
[...]
Mar 08 07:44:41 linux cf-agent[11801]: CFEngine(agent) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 08 07:44:41 linux cf-agent[11801]: CFEngine(agent) rudder While persisting class, unable to open database at '/var/rudder/cfengine-community/state/cf_state.lmdb' (OpenDB: Resource temporarily unavailable)
Mar 08 07:44:41 linux cf-agent[11801]: CFEngine(agent) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 08 07:44:41 linux cf-agent[11801]: CFEngine(agent) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 08 07:44:41 linux cf-agent[11801]: CFEngine(agent) rudder While persisting class, unable to open database at '/var/rudder/cfengine-community/state/cf_state.lmdb' (OpenDB: Resource temporarily unavailable)
Mar 08 07:44:41 linux cf-agent[11801]: CFEngine(agent) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 08 07:44:41 linux cf-agent[11801]: CFEngine(agent) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 08 07:44:41 linux cf-agent[11801]: CFEngine(agent) rudder While persisting class, unable to open database at '/var/rudder/cfengine-community/state/cf_state.lmdb' (OpenDB: Resource temporarily unavailable)
Mar 08 07:44:41 linux cf-agent[11801]: CFEngine(agent) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 08 07:44:41 linux cf-agent[11801]: CFEngine(agent) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 08 07:44:41 linux cf-agent[11801]: CFEngine(agent) rudder While persisting class, unable to open database at '/var/rudder/cfengine-community/state/cf_state.lmdb' (OpenDB: Resource temporarily unavailable)
Mar 08 07:44:41 linux cf-agent[11801]: CFEngine(agent) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
[...]
[tux@linux ~]$ sudo rudder agent info
General
Hostname: linux.example.com
UUID: 32cd0812-248c-473c-b289-20ad2b5f24eb
Policy server: rudder.example.com
Role: node
Version: 7.2.4
Policies
Status: enabled
Report mode: changes-only
Run interval: 10 min
Forced audit mode: no
Configuration id: 20230303-122535-cfc4ea7a
Policy updated: 2023-03-08 07:49:28
Inventory sent: 2023-03-08 07:42:41
Key/Certificate
Key hash: MD5=298988589d5b126a23b5867f11f81069
Key hash: sha256//+Mfzd8IWnZHyTGPXvD8ZST2i04H8k3CtRZ0xjLeykr4=
Cert. fingerprint: EF:DA:1D:BC:29:53:36:C1:D1:D7:06:94:73:4F:EC:6A:05:C6:9D:1C
Key pinning: full
Cert. creation: 2020-09-10
Cert. expiration: 2030-09-08
[tux@linux ~]$
[tux@linux ~]$ sudo rudder agent health
OK
[tux@linux ~]$ sudo /var/rudder/cfengine-community/bin/cf-agent --self-diagnostics
self-diagnostics for agent using workdir '/var/rudder/cfengine-community'
self-diagnostics for agent using inputdir '/var/rudder/cfengine-community/inputs'
self-diagnostics for agent using logdir '/var/rudder/cfengine-community'
self-diagnostics for agent using statedir '/var/rudder/cfengine-community/state'
[ YES ] Check that agent is bootstrapped: rudder.example.com
[ NO ] Check if agent is acting as a policy server: Not acting as a policy server
[ YES ] Check private key: OK at '/var/rudder/cfengine-community/ppkeys/localhost.priv'
[ NO ] Check public key: Public key at '/var/rudder/cfengine-community/ppkeys/localhost.pub' had size 775 bytes, expected 426 bytes
[ NO ] Check persistent classes DB: Unable to diagnose LMDB file (not implemented) for '/var/rudder/cfengine-community/state/cf_state.lmdb'
[ NO ] Check checksums DB: Unable to diagnose LMDB file (not implemented) for '/var/rudder/cfengine-community/state/checksum_digests.lmdb'
[ NO ] Check observations DB: Unable to diagnose LMDB file (not implemented) for '/var/rudder/cfengine-community/state/cf_observations.lmdb'
[ NO ] Check file stats DB: Unable to diagnose LMDB file (not implemented) for '/var/rudder/cfengine-community/state/stats.lmdb'
[ NO ] Check locks DB: Unable to diagnose LMDB file (not implemented) for '/var/rudder/cfengine-community/state/cf_lock.lmdb'
[ NO ] Check performance DB: Unable to diagnose LMDB file (not implemented) for '/var/rudder/cfengine-community/state/performance.lmdb'
[ NO ] Check lastseen DB: Unable to diagnose LMDB file (not implemented) for '/var/rudder/cfengine-community/state/cf_lastseen.lmdb'
[tux@linux ~]$
Updated by Alexis Mousset 11 days ago
Does this persist in subsequent agent runs?
Updated by Alexander Brunhirl 11 days ago
Hi Alexis,
yes, we have this since the update to 7.2.4 in our logs. For better readability I removed the duplicated entries.
[tux@linux ~]$ sudo journalctl -p err --since="4 days ago"|sort -u |egrep "cf-(agent|serverd|promises)"
Mar 05 16:43:37 linux cf-promises[16627]: CFEngine(common) Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 05 17:42:25 linux cf-promises[14843]: CFEngine(common) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 05 18:03:27 linux cf-promises[24907]: CFEngine(common) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 05 18:33:41 linux cf-promises[7725]: CFEngine(common) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 05 19:02:32 linux cf-promises[21446]: CFEngine(common) Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 05 22:03:34 linux cf-promises[12942]: CFEngine(common) Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 05 23:43:20 linux cf-promises[29111]: CFEngine(common) Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 06 05:33:20 linux cf-agent[6091]: CFEngine(agent) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 06 05:33:20 linux cf-promises[6414]: CFEngine(common) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 06 10:23:13 linux cf-promises[19921]: CFEngine(common) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 06 16:33:24 linux cf-agent[7460]: CFEngine(agent) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 06 16:33:24 linux cf-promises[7982]: CFEngine(common) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 06 18:53:33 linux cf-promises[12883]: CFEngine(common) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 06 20:43:33 linux cf-promises[1205]: CFEngine(common) Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 06 20:43:34 linux cf-agent[831]: CFEngine(agent) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 06 20:43:34 linux cf-promises[1212]: CFEngine(common) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 06 21:33:24 linux cf-agent[24835]: CFEngine(agent) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 06 21:33:24 linux cf-promises[25104]: CFEngine(common) Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 06 23:13:28 linux cf-promises[9238]: CFEngine(common) Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 06 23:32:54 linux cf-promises[18547]: CFEngine(common) Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 06 23:32:55 linux cf-agent[18248]: CFEngine(agent) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 06 23:32:55 linux cf-promises[18554]: CFEngine(common) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 07 16:42:45 linux cf-agent[24761]: CFEngine(agent) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 08 05:03:00 linux cf-promises[11134]: CFEngine(common) Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 08 06:33:18 linux cf-agent[12022]: CFEngine(agent) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 08 11:12:49 linux cf-agent[24451]: CFEngine(agent) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 08 11:12:49 linux cf-promises[24710]: CFEngine(common) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 08 13:13:39 linux cf-agent[17719]: CFEngine(agent) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 08 13:13:39 linux cf-promises[18335]: CFEngine(common) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 08 17:23:07 linux cf-agent[8173]: CFEngine(agent) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 08 17:23:08 linux cf-promises[8804]: CFEngine(common) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 08 20:03:07 linux cf-agent[28123]: CFEngine(agent) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 08 20:03:07 linux cf-promises[28652]: CFEngine(common) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 09 04:23:01 linux cf-promises[7887]: CFEngine(common) Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 09 06:03:02 linux cf-promises[17248]: CFEngine(common) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 09 07:22:50 linux cf-promises[11345]: CFEngine(common) Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 09 07:22:51 linux cf-promises[11353]: CFEngine(common) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 09 08:23:04 linux cf-agent[22164]: CFEngine(agent) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 09 08:23:04 linux cf-promises[22456]: CFEngine(common) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 09 14:13:10 linux cf-agent[26409]: CFEngine(agent) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 09 14:13:10 linux cf-promises[26519]: CFEngine(common) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
[tux@linux ~]$
Updated by Alexander Brunhirl 11 days ago
One of the best ways to reproduce the behavior is by triggering a rudder agent reset.
A user triggered rudder agent run can create this as well but not always.
[tux@linux ~]$ sudo sh -c 'date; rudder agent reset; date; rudder agent update; date; rudder agent run -qu; date'
Thu Mar 9 15:21:14 UTC 2023
R: Initial policies have been successfully downloaded from the policy server
ok: Rudder agent policies were updated.
ok: Rudder agent state has been reset.
Thu Mar 9 15:21:17 UTC 2023
error: Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
error: Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
error: While persisting class, unable to open database at '/var/rudder/cfengine-community/state/cf_state.lmdb' (OpenDB: Resource temporarily unavailable)
error: Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
error: Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
error: While persisting class, unable to open database at '/var/rudder/cfengine-community/state/cf_state.lmdb' (OpenDB: Resource temporarily unavailable)
error: Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
error: Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
error: While persisting class, unable to open database at '/var/rudder/cfengine-community/state/cf_state.lmdb' (OpenDB: Resource temporarily unavailable)
error: Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
error: Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
error: While persisting class, unable to open database at '/var/rudder/cfengine-community/state/cf_state.lmdb' (OpenDB: Resource temporarily unavailable)
error: Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
error: Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
error: While persisting class, unable to open database at '/var/rudder/cfengine-community/state/cf_state.lmdb' (OpenDB: Resource temporarily unavailable)
error: Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
error: While persisting class, unable to open database at '/var/rudder/cfengine-community/state/cf_state.lmdb' (OpenDB: Resource temporarily unavailable)
error: Rudder agent policies could not be updated.
Thu Mar 9 15:21:31 UTC 2023
Rudder agent 7.2.4
Node uuid: 396bb049-df8b-411a-a7f5-93dbf0b5cbe1
ok: Rudder agent policies were updated.
Start execution with config [20230303-122605-6ae8bb94]
M| State Technique Component Key Message
E| repaired Common Update Policy or configuration library were updated
## Summary #####################################################################
Not all components were displayed because we are not in full compliance mode. Please run with -g to force full compliance mode.
=> 1 components in Enforce mode
-> 1 repaired
Execution time: 39.71s
################################################################################
Reports sent.
Thu Mar 9 15:22:19 UTC 2023
[tux@linux ~]$
[tux@linux ~]$ sudo journalctl -p 0..3 --follow
-- Logs begin at Sat 2023-02-04 22:47:15 UTC. --
Mar 09 15:21:25 linux cf-promises[24431]: CFEngine(common) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 09 15:21:25 linux cf-agent[24321]: CFEngine(agent) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 09 15:21:25 linux cf-agent[24321]: CFEngine(agent) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 09 15:21:25 linux cf-agent[24321]: CFEngine(agent) rudder While persisting class, unable to open database at '/var/rudder/cfengine-community/state/cf_state.lmdb' (OpenDB: Resource temporarily unavailable)
Mar 09 15:21:25 linux cf-agent[24321]: CFEngine(agent) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 09 15:21:25 linux cf-agent[24321]: CFEngine(agent) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 09 15:21:25 linux cf-agent[24321]: CFEngine(agent) rudder While persisting class, unable to open database at '/var/rudder/cfengine-community/state/cf_state.lmdb' (OpenDB: Resource temporarily unavailable)
Mar 09 15:21:25 linux cf-agent[24321]: CFEngine(agent) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 09 15:21:25 linux cf-agent[24321]: CFEngine(agent) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 09 15:21:25 linux cf-agent[24321]: CFEngine(agent) rudder While persisting class, unable to open database at '/var/rudder/cfengine-community/state/cf_state.lmdb' (OpenDB: Resource temporarily unavailable)
Mar 09 15:21:25 linux cf-agent[24321]: CFEngine(agent) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 09 15:21:25 linux cf-agent[24321]: CFEngine(agent) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 09 15:21:25 linux cf-agent[24321]: CFEngine(agent) rudder While persisting class, unable to open database at '/var/rudder/cfengine-community/state/cf_state.lmdb' (OpenDB: Resource temporarily unavailable)
Mar 09 15:21:25 linux cf-agent[24321]: CFEngine(agent) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 09 15:21:25 linux cf-agent[24321]: CFEngine(agent) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 09 15:21:25 linux cf-agent[24321]: CFEngine(agent) rudder While persisting class, unable to open database at '/var/rudder/cfengine-community/state/cf_state.lmdb' (OpenDB: Resource temporarily unavailable)
Mar 09 15:21:25 linux cf-agent[24321]: CFEngine(agent) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 09 15:21:25 linux cf-agent[24321]: CFEngine(agent) rudder While persisting class, unable to open database at '/var/rudder/cfengine-community/state/cf_state.lmdb' (OpenDB: Resource temporarily unavailable)
Mar 09 15:21:25 linux cf-promises[24431]: CFEngine(common) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 09 15:21:25 linux cf-promises[24431]: CFEngine(common) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 09 15:21:25 linux cf-promises[24431]: CFEngine(common) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 09 15:21:25 linux cf-promises[24431]: CFEngine(common) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 09 15:21:25 linux cf-promises[24431]: CFEngine(common) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Actions