Rudder

Does it happen when running the agent (rudder agent run) or during an inventory (rudder agent inventory)?

Do you have a list of the processes (with ps auxf or similar) opened when the problem occurs? It would help find the source of the problem.

We actually have two problems here:

• Memory leak/exhaustion
• Lack of timeout for fdisk -l call

Tested with a simple sleep:

root      5085  0.0  0.2  12832   604 pts/0    S+   13:56   0:00  |                   \_ /bin/sh /opt/rudder/share/commands/agent-inventory
root      5125  0.0  0.2  12832   604 pts/0    S+   13:56   0:00  |                       \_ /bin/sh /opt/rudder/share/commands/agent-run -N -D force_inventory
root      5224  5.5  4.2 120716  9788 ?        Ss   13:56   0:00  |                           \_ /opt/rudder/bin/cf-agent -I -D info -Cnever -K -b doInventory 
root      5273  0.0  0.4  12832  1000 ?        S    13:56   0:00  |                           |   \_ /bin/sh /opt/rudder/bin/run-inventory --local=/var/rudder/
root      5319  4.5 24.9 154868 57652 ?        S    13:56   0:00  |                           |       \_ fusioninventory-agent: running task Inventory         
root      5360  0.0  0.5   9772  1184 ?        S    13:56   0:00  |                           |           \_ /bin/sh /sbin/fdisk -v
root      5361  0.0  0.3   4356   720 ?        S    13:56   0:00  |                           |               \_ sleep 500

It seems to only affect the inventory.

We can confirm that it's specific to 6.2.x
Actually no, it is also on 6.1.8-nightly at least

Memory exhaustion could come from inventory processes piling up.

Alexis MOUSSET wrote in #note-5:

Tested with a simple sleep:

[...]

It seems to only affect the inventory.

yes, i can confirm. we had exactly this situation. many stranded cf-agent processes that piled up.

Tested on 6.1.7, 6.1.8-rc, and 6.2.0:

- switch fdisk with sleep 1000,
- start a bunch of rudder agent inventory.

In all case, after some time, rudder agent inventory returns with the following message:

info     Rudder agent was run on a subset of policies - not all policies were checked

But background blockink fdisk is still running and not killed (as shown in message #15).

There are several problems here:

• We authorize several inventories to pile up, but the inventory process uses a lot of memory, so this can leads to DoSing the node
• We authorize inventories to last forever. Using CFEngine's timeout is not enough as it leaves zombies processes and may fail.

What we could do:

• Prevent running a new inventory if another one is already running. If done reliably, it prevents the DoS.
• Adding a watchdog for inventory process in rudder agent check, to kill inventories lasting for more than X minutes and all of their child processes.