Project

General

Profile

Actions

Bug #22442

open

Could not open database txn cf_state.lmdb: Invalid argument

Added by Alexander Brunhirl 12 days ago. Updated 11 days ago.

Status:
New
Priority:
N/A
Category:
Agent
Target version:
Severity:
Major - prevents use of part of Rudder | no simple workaround
UX impact:
User visibility:
Effort required:
Priority:
0
Regression:
No

Description

Hi,

after upgrading Rudder from 6.2.13 to 7.2.4 we found the following cf-agent errors in our logs.
This happend on CentOS 7/RHEL 7, SLES12 SP5, SLES15 SP3, SLES15 SP4

[tux@linux ~]$ sudo journalctl --priority err
       [...]
Mar 08 07:44:41 linux cf-agent[11801]: CFEngine(agent) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 08 07:44:41 linux cf-agent[11801]: CFEngine(agent) rudder While persisting class, unable to open database at '/var/rudder/cfengine-community/state/cf_state.lmdb' (OpenDB: Resource temporarily unavailable)
Mar 08 07:44:41 linux cf-agent[11801]: CFEngine(agent) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 08 07:44:41 linux cf-agent[11801]: CFEngine(agent) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 08 07:44:41 linux cf-agent[11801]: CFEngine(agent) rudder While persisting class, unable to open database at '/var/rudder/cfengine-community/state/cf_state.lmdb' (OpenDB: Resource temporarily unavailable)
Mar 08 07:44:41 linux cf-agent[11801]: CFEngine(agent) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 08 07:44:41 linux cf-agent[11801]: CFEngine(agent) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 08 07:44:41 linux cf-agent[11801]: CFEngine(agent) rudder While persisting class, unable to open database at '/var/rudder/cfengine-community/state/cf_state.lmdb' (OpenDB: Resource temporarily unavailable)
Mar 08 07:44:41 linux cf-agent[11801]: CFEngine(agent) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 08 07:44:41 linux cf-agent[11801]: CFEngine(agent) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 08 07:44:41 linux cf-agent[11801]: CFEngine(agent) rudder While persisting class, unable to open database at '/var/rudder/cfengine-community/state/cf_state.lmdb' (OpenDB: Resource temporarily unavailable)
Mar 08 07:44:41 linux cf-agent[11801]: CFEngine(agent) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
       [...]
[tux@linux ~]$ sudo rudder agent info

General
           Hostname: linux.example.com
               UUID: 32cd0812-248c-473c-b289-20ad2b5f24eb
      Policy server: rudder.example.com
               Role: node
            Version: 7.2.4

Policies
             Status: enabled
        Report mode: changes-only
       Run interval: 10 min
  Forced audit mode: no
   Configuration id: 20230303-122535-cfc4ea7a
     Policy updated: 2023-03-08 07:49:28
     Inventory sent: 2023-03-08 07:42:41

Key/Certificate
           Key hash: MD5=298988589d5b126a23b5867f11f81069
           Key hash: sha256//+Mfzd8IWnZHyTGPXvD8ZST2i04H8k3CtRZ0xjLeykr4=
  Cert. fingerprint: EF:DA:1D:BC:29:53:36:C1:D1:D7:06:94:73:4F:EC:6A:05:C6:9D:1C
        Key pinning: full
     Cert. creation: 2020-09-10
   Cert. expiration: 2030-09-08

[tux@linux ~]$ 
[tux@linux ~]$ sudo rudder agent health
OK
[tux@linux ~]$ sudo /var/rudder/cfengine-community/bin/cf-agent --self-diagnostics
self-diagnostics for agent using workdir '/var/rudder/cfengine-community'
self-diagnostics for agent using inputdir '/var/rudder/cfengine-community/inputs'
self-diagnostics for agent using logdir '/var/rudder/cfengine-community'
self-diagnostics for agent using statedir '/var/rudder/cfengine-community/state'
[ YES ] Check that agent is bootstrapped: rudder.example.com
[ NO  ] Check if agent is acting as a policy server: Not acting as a policy server
[ YES ] Check private key: OK at '/var/rudder/cfengine-community/ppkeys/localhost.priv'
[ NO  ] Check public key: Public key at '/var/rudder/cfengine-community/ppkeys/localhost.pub' had size 775 bytes, expected 426 bytes
[ NO  ] Check persistent classes DB: Unable to diagnose LMDB file (not implemented) for '/var/rudder/cfengine-community/state/cf_state.lmdb'
[ NO  ] Check checksums DB: Unable to diagnose LMDB file (not implemented) for '/var/rudder/cfengine-community/state/checksum_digests.lmdb'
[ NO  ] Check observations DB: Unable to diagnose LMDB file (not implemented) for '/var/rudder/cfengine-community/state/cf_observations.lmdb'
[ NO  ] Check file stats DB: Unable to diagnose LMDB file (not implemented) for '/var/rudder/cfengine-community/state/stats.lmdb'
[ NO  ] Check locks DB: Unable to diagnose LMDB file (not implemented) for '/var/rudder/cfengine-community/state/cf_lock.lmdb'
[ NO  ] Check performance DB: Unable to diagnose LMDB file (not implemented) for '/var/rudder/cfengine-community/state/performance.lmdb'
[ NO  ] Check lastseen DB: Unable to diagnose LMDB file (not implemented) for '/var/rudder/cfengine-community/state/cf_lastseen.lmdb'
[tux@linux ~]$ 
Actions #1

Updated by Alexis Mousset 11 days ago

  • Assignee set to Alexis Mousset
Actions #2

Updated by Alexis Mousset 11 days ago

Does this persist in subsequent agent runs?

Actions #3

Updated by Alexander Brunhirl 11 days ago

Hi Alexis,

yes, we have this since the update to 7.2.4 in our logs. For better readability I removed the duplicated entries.

[tux@linux ~]$ sudo journalctl -p err --since="4 days ago"|sort -u |egrep "cf-(agent|serverd|promises)" 
Mar 05 16:43:37 linux cf-promises[16627]: CFEngine(common)  Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 05 17:42:25 linux cf-promises[14843]: CFEngine(common) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 05 18:03:27 linux cf-promises[24907]: CFEngine(common) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 05 18:33:41 linux cf-promises[7725]: CFEngine(common) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 05 19:02:32 linux cf-promises[21446]: CFEngine(common)  Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 05 22:03:34 linux cf-promises[12942]: CFEngine(common)  Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 05 23:43:20 linux cf-promises[29111]: CFEngine(common)  Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 06 05:33:20 linux cf-agent[6091]: CFEngine(agent) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 06 05:33:20 linux cf-promises[6414]: CFEngine(common) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 06 10:23:13 linux cf-promises[19921]: CFEngine(common) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 06 16:33:24 linux cf-agent[7460]: CFEngine(agent) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 06 16:33:24 linux cf-promises[7982]: CFEngine(common) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 06 18:53:33 linux cf-promises[12883]: CFEngine(common) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 06 20:43:33 linux cf-promises[1205]: CFEngine(common)  Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 06 20:43:34 linux cf-agent[831]: CFEngine(agent) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 06 20:43:34 linux cf-promises[1212]: CFEngine(common) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 06 21:33:24 linux cf-agent[24835]: CFEngine(agent) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 06 21:33:24 linux cf-promises[25104]: CFEngine(common)  Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 06 23:13:28 linux cf-promises[9238]: CFEngine(common)  Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 06 23:32:54 linux cf-promises[18547]: CFEngine(common)  Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 06 23:32:55 linux cf-agent[18248]: CFEngine(agent) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 06 23:32:55 linux cf-promises[18554]: CFEngine(common) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 07 16:42:45 linux cf-agent[24761]: CFEngine(agent) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 08 05:03:00 linux cf-promises[11134]: CFEngine(common)  Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 08 06:33:18 linux cf-agent[12022]: CFEngine(agent) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 08 11:12:49 linux cf-agent[24451]: CFEngine(agent) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 08 11:12:49 linux cf-promises[24710]: CFEngine(common) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 08 13:13:39 linux cf-agent[17719]: CFEngine(agent) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 08 13:13:39 linux cf-promises[18335]: CFEngine(common) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 08 17:23:07 linux cf-agent[8173]: CFEngine(agent) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 08 17:23:08 linux cf-promises[8804]: CFEngine(common) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 08 20:03:07 linux cf-agent[28123]: CFEngine(agent) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 08 20:03:07 linux cf-promises[28652]: CFEngine(common) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 09 04:23:01 linux cf-promises[7887]: CFEngine(common)  Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 09 06:03:02 linux cf-promises[17248]: CFEngine(common) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 09 07:22:50 linux cf-promises[11345]: CFEngine(common)  Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 09 07:22:51 linux cf-promises[11353]: CFEngine(common) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 09 08:23:04 linux cf-agent[22164]: CFEngine(agent) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 09 08:23:04 linux cf-promises[22456]: CFEngine(common) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 09 14:13:10 linux cf-agent[26409]: CFEngine(agent) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 09 14:13:10 linux cf-promises[26519]: CFEngine(common) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
[tux@linux ~]$
Actions #4

Updated by Alexander Brunhirl 11 days ago

One of the best ways to reproduce the behavior is by triggering a rudder agent reset.
A user triggered rudder agent run can create this as well but not always.

[tux@linux ~]$ sudo sh -c 'date; rudder agent reset; date; rudder agent update; date; rudder agent run -qu; date'
Thu Mar  9 15:21:14 UTC 2023
R: Initial policies have been successfully downloaded from the policy server
ok: Rudder agent policies were updated.
ok: Rudder agent state has been reset.
Thu Mar  9 15:21:17 UTC 2023
   error: Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
   error: Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
   error: While persisting class, unable to open database at '/var/rudder/cfengine-community/state/cf_state.lmdb' (OpenDB: Resource temporarily unavailable)
   error: Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
   error: Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
   error: While persisting class, unable to open database at '/var/rudder/cfengine-community/state/cf_state.lmdb' (OpenDB: Resource temporarily unavailable)
   error: Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
   error: Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
   error: While persisting class, unable to open database at '/var/rudder/cfengine-community/state/cf_state.lmdb' (OpenDB: Resource temporarily unavailable)
   error: Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
   error: Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
   error: While persisting class, unable to open database at '/var/rudder/cfengine-community/state/cf_state.lmdb' (OpenDB: Resource temporarily unavailable)
   error: Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
   error: Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
   error: While persisting class, unable to open database at '/var/rudder/cfengine-community/state/cf_state.lmdb' (OpenDB: Resource temporarily unavailable)
   error: Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
   error: While persisting class, unable to open database at '/var/rudder/cfengine-community/state/cf_state.lmdb' (OpenDB: Resource temporarily unavailable)

error: Rudder agent policies could not be updated.
Thu Mar  9 15:21:31 UTC 2023
Rudder agent 7.2.4
Node uuid: 396bb049-df8b-411a-a7f5-93dbf0b5cbe1
ok: Rudder agent policies were updated.
Start execution with config [20230303-122605-6ae8bb94]

M| State         Technique                 Component                 Key                Message
E| repaired      Common                    Update                                       Policy or configuration library were updated

## Summary #####################################################################
Not all components were displayed because we are not in full compliance mode. Please run with -g to force full compliance mode.
   => 1 components in Enforce mode
      -> 1 repaired
Execution time: 39.71s
################################################################################
Reports sent.
Thu Mar  9 15:22:19 UTC 2023
[tux@linux ~]$
[tux@linux ~]$ sudo journalctl -p 0..3 --follow
-- Logs begin at Sat 2023-02-04 22:47:15 UTC. --
Mar 09 15:21:25 linux cf-promises[24431]: CFEngine(common) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 09 15:21:25 linux cf-agent[24321]: CFEngine(agent) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 09 15:21:25 linux cf-agent[24321]: CFEngine(agent) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 09 15:21:25 linux cf-agent[24321]: CFEngine(agent) rudder While persisting class, unable to open database at '/var/rudder/cfengine-community/state/cf_state.lmdb' (OpenDB: Resource temporarily unavailable)
Mar 09 15:21:25 linux cf-agent[24321]: CFEngine(agent) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 09 15:21:25 linux cf-agent[24321]: CFEngine(agent) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 09 15:21:25 linux cf-agent[24321]: CFEngine(agent) rudder While persisting class, unable to open database at '/var/rudder/cfengine-community/state/cf_state.lmdb' (OpenDB: Resource temporarily unavailable)
Mar 09 15:21:25 linux cf-agent[24321]: CFEngine(agent) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 09 15:21:25 linux cf-agent[24321]: CFEngine(agent) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 09 15:21:25 linux cf-agent[24321]: CFEngine(agent) rudder While persisting class, unable to open database at '/var/rudder/cfengine-community/state/cf_state.lmdb' (OpenDB: Resource temporarily unavailable)
Mar 09 15:21:25 linux cf-agent[24321]: CFEngine(agent) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 09 15:21:25 linux cf-agent[24321]: CFEngine(agent) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 09 15:21:25 linux cf-agent[24321]: CFEngine(agent) rudder While persisting class, unable to open database at '/var/rudder/cfengine-community/state/cf_state.lmdb' (OpenDB: Resource temporarily unavailable)
Mar 09 15:21:25 linux cf-agent[24321]: CFEngine(agent) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 09 15:21:25 linux cf-agent[24321]: CFEngine(agent) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 09 15:21:25 linux cf-agent[24321]: CFEngine(agent) rudder While persisting class, unable to open database at '/var/rudder/cfengine-community/state/cf_state.lmdb' (OpenDB: Resource temporarily unavailable)
Mar 09 15:21:25 linux cf-agent[24321]: CFEngine(agent) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 09 15:21:25 linux cf-agent[24321]: CFEngine(agent) rudder While persisting class, unable to open database at '/var/rudder/cfengine-community/state/cf_state.lmdb' (OpenDB: Resource temporarily unavailable)
Mar 09 15:21:25 linux cf-promises[24431]: CFEngine(common) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 09 15:21:25 linux cf-promises[24431]: CFEngine(common) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 09 15:21:25 linux cf-promises[24431]: CFEngine(common) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 09 15:21:25 linux cf-promises[24431]: CFEngine(common) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Mar 09 15:21:25 linux cf-promises[24431]: CFEngine(common) rudder Could not open database txn /var/rudder/cfengine-community/state/cf_state.lmdb: Invalid argument
Actions

Also available in: Atom PDF