Rudder

in the reports by Node, the Rule doesn't show up (because we don't have reports from this configid most probably)

Oh, and not all nodes appears in the list of nodes where the rule apply
It showed centos-6-64, where it should show node1 and centos-6-64

Ok, i don't have all the node that appears, because one of the node is pending a change from another rule, so it is looking for the bad nodeconfigid ( CheckChanges(2014-12-17T14:55:23.000+01:00,NodeConfigIdInfo(NodeConfigId(1660048484),2014-12-17T14:36:14.223+01:00,Some(2014-12-17T15:00:39.226+01:00)))) rather than NodeConfigIdInfo(NodeConfigId(-190266289))

I confirm that in compliance mode, it works as expected

It's more likelly a similar occurence than in #6040.

The problem is that in order to have an accurate "pending" state for nodes and rules, we need to know what configuration changed compared to what we received in the last run of corresponding agent. It's actually "received" and not "ask", because (for example) we don't wan to remove a pending state of something whose config didn't changed in the last run (but do changed in the n-1 one) and still don't have any reports for).

In #6040, that was manageable since we have the whole state given by the received reports - that's the goal of compliace.

Here, we would need to rebuild the compliance for policy at "t-1", then compute what state are switchet to pending with the new config. This is an abomination in performance, making Rudder unusable.

So what we really need is a cache of "last" (nodeId, configId, run) -> compliance (at least until rule level). I'm not sur about how big is "last", but we need at least one by node, and for each node, at least the last run for current configId, and something like the last run for the previous config id.

This cache would be invalidated/computed again each time a new run reach Rudder.

Version commit:9aee9a2 on my branch compiles, even if a lot of tests were disable for that.

The performance for a big number of node (2000) are NOT good. On my test machine, even with no reports in base (safe for the root server), the Postgres Request used to get expected reports takes > 20s.

Getting compliance for one node is quick (<50ms for the request)

You could try to explain analyze the query to see what is not going right

And i believe that generate_subscript may not be what you need, for instance if you have this table

rudder=> select * from expectedreportsnodes where nodejoinkey = 49;
 nodejoinkey |                nodeid                |                nodeconfigids                 
-------------+--------------------------------------+----------------------------------------------
          49 | 06da3556-5204-4bd7-b3b0-fa5e7bcfbbea | {-1884617228}
          49 | 769cd476-b96b-4546-b2fb-0196e333cbc7 | {-1959368040}
          49 | root                                 | {-1684204762,321158356,935965435,1829297815}

you obtain this result

 nodejoinkey |                nodeid                |                nodeconfigids                 | v 
-------------+--------------------------------------+----------------------------------------------+---
          49 | 06da3556-5204-4bd7-b3b0-fa5e7bcfbbea | {-1884617228}                                | 1
          49 | 769cd476-b96b-4546-b2fb-0196e333cbc7 | {-1959368040}                                | 1
          49 | root                                 | {-1684204762,321158356,935965435,1829297815} | 1
          49 | root                                 | {-1684204762,321158356,935965435,1829297815} | 2
          49 | root                                 | {-1684204762,321158356,935965435,1829297815} | 3
          49 | root                                 | {-1684204762,321158356,935965435,1829297815} | 4

So you get more lines, and i'm not sure you are using them in the query nor in the result

could it be that you need to use unnest rather that generate_subscripts, and so have the values outside of the array to query on it (see http://blog.heapanalytics.com/dont-iterate-over-a-postgres-array-with-a-loop/ )

also, using ANY to search the arrays (if you don't want to use unnest), and even better ANY (VALUES (nodeConfigIds) ) might help a lot (see https://www.datadoghq.com/2013/08/100x-faster-postgres-performance-by-changing-1-line/ )

I tried several variation of the query in commit 9e9e9d1 of my branch.

unnest does not work, throwing an exception for no obvious reason.
generate_subscript seems to be quite efficient after all.

The result is reasonnably quick. 4s to compute compliance with 2000 nodes (~80k expected reports for all nodes).

Now, the next clear optimisation is a cache.

I opened a stackoverflow question here: http://stackoverflow.com/questions/28025753/looking-in-array-with-a-big-set-of-input-values

There is other trail of optimization that can be tested, but the general idea for now is:

- we have an usable solution, and whatever optimisation we set on postgres, we will still have hundreds of millisec of pur computation & rendering
- so the next step is to set-up the cache, so that we just have the correct data, always
- we can still opimize the query after that

In the last commit (5dc5955 around now), I ported most of the old tests, which are passing.
I also added a "debug_compliance" logger that allows to see what the compliance computer is trying to merge with what. Enable it by adding in logback.xml:

    <logger name="debug_compliance" level="debug" />

The logger named change to "explain_compliance", so now use:

  <logger name="explain_compliance" level="trace" />  

Some more cleaning and a big correction on the rule grid displayed in validation pop-up (ex when directives change).