Project

General

Profile

Actions

Bug #5621

closed

Cannot manage more than 4993 nodes, due to CFEngine limitation

Added by Lionel Le Folgoc over 7 years ago. Updated over 7 years ago.

Status:
Released
Priority:
N/A
Assignee:
Matthieu CERDA
Category:
System techniques
Target version:
Severity:
UX impact:
User visibility:
Effort required:
Priority:

Description

Hi,

# cf-promises -f /var/rudder/cfengine-community/inputs/promises.cf
/var/rudder/cfengine-community/inputs/common/1.0/cf-served.cf:25101:44: error: memory exhausted
                    "MY-SEXY-HOSTNAME", 
/var/rudder/cfengine-community/inputs/common/1.0/cf-served.cf:25101:45: error: syntax error
                    "MY-SEXY-HOSTNAME", 
[...]
follow several lines of syntax errors...

There are several lists in this file: "trustkeysfrom", "allowconnects", "allowallconnects", and "hosts", that contain all accepted hostnames, and it seems cfengine is choking on some limit.
(here with 5000 nodes, the file is 40146 lines long)

That happens during policy generation on the policy server (if cf-promises is activated), and also on the relaytop server.

Thanks.


Subtasks 1 (0 open1 closed)

Bug #6102: Apply #5621 to branch 3.0ReleasedMatthieu CERDA2015-01-08Actions

Related issues 1 (0 open1 closed)

Related to Rudder - Bug #6132: Rudder agent compilation fails to build on AIXReleasedBenoît PECCATTE2015-01-15Actions
Actions #1

Updated by Lionel Le Folgoc over 7 years ago

Sooooo, the parsing code of cfengine is stupid (yyparse(), from the look of it it's autogenerated with/flex and scary things like that).

Fails with the memory error:

bundle common lol
{
  vars:
    "ponies" ilist => {
      1,
      2,
      ... up to ...
      4999,
      5000
    };
}

But, cf-promises works if you split the array and assemble it later:

bundle common lol
{
  vars:
    "ponies" ilist => {
      1,
      2,
      ... up to ...
      4993
    };

    "all_ponies" ilist => {
      @(ponies),
      4994,
      4995,
      4996,
      4997,
      4998,
      4999,
      5000
    };
}

Note that the result is similar with slist and rlist...

That means you need to split all hostname lists each time they reach 4993 elements! :)
(not even funny)

Actions #2

Updated by Nicolas CHARLES over 7 years ago

  • Category set to System techniques
  • Assignee set to Nicolas CHARLES
  • Target version set to 2.10.7

Bug opened at CFEngine
https://dev.cfengine.com/issues/6672

let's see how they react

Actions #3

Updated by Vincent MEMBRÉ over 7 years ago

  • Target version changed from 2.10.7 to 2.10.8
Actions #4

Updated by Jonathan CLARKE over 7 years ago

  • Subject changed from cf-promises fails with "error: memory exhausted" to CFEngine fails with lists over 4993 elements long - in particular over 4993 managed nodes
Actions #5

Updated by Vincent MEMBRÉ over 7 years ago

  • Target version changed from 2.10.8 to 2.10.9
Actions #6

Updated by Benoît PECCATTE over 7 years ago

  • Target version changed from 2.10.9 to 2.11.6

Re targeting to 2.11
We can't apply the patch to 2.10 and people wanting to use more than 5000 nodes may have already updated to 2.11.

Actions #7

Updated by Benoît PECCATTE over 7 years ago

  • Status changed from New to Pending technical review
  • Assignee changed from Nicolas CHARLES to Matthieu CERDA
  • Pull Request set to https://github.com/Normation/rudder-packages/pull/582
Actions #8

Updated by Benoît PECCATTE over 7 years ago

  • Status changed from Pending technical review to Pending release
  • % Done changed from 0 to 100

Applied in changeset packages:rudder-packages|commit:ebb0ec2a62419244a7b7b6fb29a96012b541cd47.

Actions #9

Updated by Matthieu CERDA over 7 years ago

Applied in changeset packages:rudder-packages|commit:78283803e83e7928846f2ea74d2cec7d38bd0b7e.

Actions #10

Updated by Vincent MEMBRÉ over 7 years ago

  • Target version changed from 2.11.6 to 2.11.7
Actions #11

Updated by Vincent MEMBRÉ over 7 years ago

  • Subject changed from CFEngine fails with lists over 4993 elements long - in particular over 4993 managed nodes to Cannot manage more than 4993 nodes, due to CFEngine limitation
Actions #12

Updated by Vincent MEMBRÉ over 7 years ago

  • Status changed from Pending release to Released

This bug has been fixed in Rudder 2.11.7, which was released these days.

Actions

Also available in: Atom PDF