Rudder

This is indeed terribly slow, and I will look to that.

But it seems that the major problem is the missing api call "bulk delete node1, node2, .... node5625".
I susepct that that some part of the latency is due to 1/ network, 2/generating rules, and neither have to be paid each time.

But it seems that the major problem is the missing api call "bulk delete node1, node2, .... node5625".

(that would be nice ; then again, I don't think I'm going to perform mass bulk deletion of my nodes everyday ;-)

I susepct that that some part of the latency is due to 1/ network, 2/generating rules, and neither have to be paid each time.

1/ I'm launching the script directly on the policy server.
2/ The whole rule regeneration takes 20 minutes here so that might not be the cause of the 1min delay (my script doesn't force a rule generation anyway).

I missed the 20 minutes. Wow. That is awful. Which version of Rudder ? Could you please add the line

<logger name="com.normation.rudder.services.policies.DeploymentServiceImpl" level="debug" />

On the logback.xml configuration file to get timing about what is going so slowly ?

Could you also please run the following scrip so that we get some insight about the size of things: https://github.com/fanf/rudder-tools/blob/rudder_server_metric_reporting_script/scripts/rudder_metrics_reporting.sh

Thanks,

I got some information, not sure if it was for that system, thought:

Number of expected reports (components*directives*nodes): 17848
Number of rules: 6
Number of directives: 6
Number of nodes: 5002
Number of reports for one day: 36993
Report database size: 114 MB
Number of lines in reports table: 108464
Full database size: 2192 MB
Archiving reports:
archive.TTL=3
delete.TTL=90
frequency=daily

With a 4Go (ec2) server.

So, in this regard, I think we are reaching the limit of the system and that is what is causing über slowness. With a little of bad luck the system is swapping, making everything even slower.

The promise generation is taking so long because of the cf-promises growing lineary with the number of nodes. In such installation, #4427 is a showstopper.

So, first thing to do: let user know the server size to use for such big installation. We are recomanding 2Go for the JVM for 500 nodes. Of course, things are not linear, but for 5000 nodes, a multi-server installation with a 8Go server for the webapp machine seems to be a minimum, with 6Go for the JVM. This is a first raw estimate (that could be a big underestimate if some linear or worst memory consumption happen), we will work on better dimensionning metrics.

I discovered later than slapd was wrecked on the server (at some point rudder runtime logs filled the partition, and slapd was in a bizarre state because of that).

I reinitialised everything since then, so when I'm done with the current tests, I'll try again a deletion. I think it'll be significantly faster.

Okay, I retried with 5000 accepted nodes, and deleting one node through the API took less than 4 seconds, so I guess slapd was in a weird state and was the cause of the slowdown.
Thanks, feel free to close this issue.

OK, that seems far better!

I think a lot of other monster slowness should have been present when slapd was in that state (unresponsive UI, etc).

So, we have to find a way to automatically know that it is happening.

The slapd log was full of lines similar to this one (several times per second):
Sep 30 08:28:25 rudderldap01 rudder-slapd¹²⁴⁸: connection_input: conn=1004 deferring operation: too many executing