Project

General

Profile

Actions
Copy link

Bug #5621

closed

Cannot manage more than 4993 nodes, due to CFEngine limitation

Added by Lionel Le Folgoc over 9 years ago. Updated about 9 years ago.

Status:
Released
Priority:
N/A
Assignee:
Matthieu CERDA
Category:
System techniques
Target version:
2.11.7
Pull Request:
https://github.com/Normation/rudder-p...
Severity:
UX impact:
User visibility:
Effort required:
Priority:
Name check:
Fix check:
Regression:

Description

Hi,

# cf-promises -f /var/rudder/cfengine-community/inputs/promises.cf
/var/rudder/cfengine-community/inputs/common/1.0/cf-served.cf:25101:44: error: memory exhausted
                    "MY-SEXY-HOSTNAME", 
/var/rudder/cfengine-community/inputs/common/1.0/cf-served.cf:25101:45: error: syntax error
                    "MY-SEXY-HOSTNAME", 
[...]
follow several lines of syntax errors...

There are several lists in this file: "trustkeysfrom", "allowconnects", "allowallconnects", and "hosts", that contain all accepted hostnames, and it seems cfengine is choking on some limit.
(here with 5000 nodes, the file is 40146 lines long)

That happens during policy generation on the policy server (if cf-promises is activated), and also on the relaytop server.

Thanks.

Subtasks 1 (0 open — 1 closed)

Bug #6102: Apply #5621 to branch 3.0ReleasedMatthieu CERDA2015-01-08Actions

Related issues 1 (0 open — 1 closed)

Related to Rudder - Bug #6132: Rudder agent compilation fails to build on AIXReleasedBenoît PECCATTE2015-01-15Actions
Actions
Copy link
 #1

Updated by Lionel Le Folgoc over 9 years ago

Sooooo, the parsing code of cfengine is stupid (yyparse(), from the look of it it's autogenerated with/flex and scary things like that).

Fails with the memory error:

bundle common lol
{
  vars:
    "ponies" ilist => {
      1,
      2,
      ... up to ...
      4999,
      5000
    };
}

But, cf-promises works if you split the array and assemble it later:

bundle common lol
{
  vars:
    "ponies" ilist => {
      1,
      2,
      ... up to ...
      4993
    };

    "all_ponies" ilist => {
      @(ponies),
      4994,
      4995,
      4996,
      4997,
      4998,
      4999,
      5000
    };
}

Note that the result is similar with slist and rlist...

That means you need to split all hostname lists each time they reach 4993 elements! :)
(not even funny)

Actions
Copy link
 #2

Updated by Nicolas CHARLES over 9 years ago

  • Category set to System techniques
  • Assignee set to Nicolas CHARLES
  • Target version set to 2.10.7

Bug opened at CFEngine
https://dev.cfengine.com/issues/6672

let's see how they react

Actions
Copy link
 #3

Updated by Vincent MEMBRÉ over 9 years ago

  • Target version changed from 2.10.7 to 2.10.8
Actions
Copy link
 #4

Updated by Jonathan CLARKE over 9 years ago

  • Subject changed from cf-promises fails with "error: memory exhausted" to CFEngine fails with lists over 4993 elements long - in particular over 4993 managed nodes
Actions
Copy link
 #5

Updated by Vincent MEMBRÉ over 9 years ago

  • Target version changed from 2.10.8 to 2.10.9
Actions
Copy link
 #6

Updated by Benoît PECCATTE over 9 years ago

  • Target version changed from 2.10.9 to 2.11.6

Re targeting to 2.11
We can't apply the patch to 2.10 and people wanting to use more than 5000 nodes may have already updated to 2.11.

Actions
Copy link
 #7

Updated by Benoît PECCATTE over 9 years ago

  • Status changed from New to Pending technical review
  • Assignee changed from Nicolas CHARLES to Matthieu CERDA
  • Pull Request set to https://github.com/Normation/rudder-packages/pull/582
Actions
Copy link
 #8

Updated by Benoît PECCATTE over 9 years ago

  • Status changed from Pending technical review to Pending release
  • % Done changed from 0 to 100

Applied in changeset packages:rudder-packages|commit:ebb0ec2a62419244a7b7b6fb29a96012b541cd47.

Actions
Copy link
 #9

Updated by Matthieu CERDA over 9 years ago

Applied in changeset packages:rudder-packages|commit:78283803e83e7928846f2ea74d2cec7d38bd0b7e.

Actions
Copy link
 #10

Updated by Vincent MEMBRÉ about 9 years ago

  • Target version changed from 2.11.6 to 2.11.7
Actions
Copy link
 #11

Updated by Vincent MEMBRÉ about 9 years ago

  • Subject changed from CFEngine fails with lists over 4993 elements long - in particular over 4993 managed nodes to Cannot manage more than 4993 nodes, due to CFEngine limitation
Actions
Copy link
 #12

Updated by Vincent MEMBRÉ about 9 years ago

  • Status changed from Pending release to Released

This bug has been fixed in Rudder 2.11.7, which was released these days.

Actions
Copy link

Also available in: Atom PDF