Rudder

Caches have been cleared on both rudder-server-root

Some Categories / Groups are also missing on the Server A (same that on server B).

I think that the issue cames during the "upgrade" of the server A [from 2.5 to 2.6.10] and it's due to a DB issue.

As you know, Server A is a clone of our prod server and server B is a fresh installed server.

I found a difference between these 3 servers (the DB size) :

Server prod [2.5.4] =>
[root@servprod base]# pwd
/var/lib/pgsql/data/base
[root@servprod base]# du -sh *
5,5M 1
5,5M 11563
5,5M 11564
81G 16385
4,0K pgsql_tmp

Server A [2.5.4 upgraded in 2.5.5 and finally in 2.6.10] =>
[root@servA base]# pwd
/var/lib/pgsql/data/base
[root@serv1 base]# du -sh *
5,4M 1
5,4M 11563
5,5M 11564
54G 16385
4,0K pgsql_tmp

Server B [2.6.10] =>
[root@servB base]# pwd
/var/lib/pgsql/data/base
[root@servB base]# du -sh *
5,3M 1
5,3M 11563
5,4M 11564
26M 16385

I hope these informations will help to find the solution ! :)

As requested, please find more informations :

[root@servprod report]# ./rudder_metrics_reporting.sh
Number of expected reports (components*directives*nodes): 32643
Number of rules: 119 => 129 defined
Number of directives: 261
Number of nodes: 119
Number of reports for one day: 8026788
Report database size: 76 GB
Number of lines in reports table: 6.3087e+07
Full database size: 80 GB
Archiving reports:
archive.TTL=30
delete.TTL=90
frequency=daily

[root@servA report]# ./rudder_metrics_reporting.sh
Number of expected reports (components*directives*nodes): 32184
Number of rules: 116 => 125 defined
Number of directives: 256
Number of nodes: 118
Number of reports for one day: 10174
Report database size: 49 GB
Number of lines in reports table: 33611
Full database size: 54 GB
Archiving reports:
archive.TTL=30
delete.TTL=90
frequency=daily

[root@servB report]# ./rudder_metrics_reporting.sh
Number of expected reports (components*directives*nodes): 31
Number of rules: 3 => 125 defined
Number of directives: 3
Number of nodes: 0
Number of reports for one day: 9887
Report database size: 17 MB
Number of lines in reports table: 32730
Full database size: 25 MB
Archiving reports:
archive.TTL=30
delete.TTL=90
frequency=daily

In bold the values displayed in the web interface...

I'm available to do new test, give more informations. I can also clone (again) the prod server to relanch the update from 2.5.4 to 2.6.10 and give you more logs.

I hope there is a solution because for the moment we can't abandon our 2.5.4 to a 2.6.10

I have cleaned reports before leaving the office on 31/01.
TTL has been changed on each /opt/rudder/etc/rudder-web.properties file.
DB size has not been reduce.

Other point, We still don't know why all categories / groups has not been recreated (our first issue)...

FYI (new metrics reports) =>

[root@servprod report]# ./rudder_metrics_reporting.sh
Number of expected reports (components*directives*nodes): 32639
Number of rules: 119
Number of directives: 266
Number of nodes: 119
Number of reports for one day: 8000424
Report database size: 76 GB
Number of lines in reports table: 2.24742e+07
Full database size: 80 GB
Archiving reports:
archive.TTL=6
delete.TTL=7

root@servA report]# ./rudder_metrics_reporting.sh
Number of expected reports (components*directives*nodes): 32184
Number of rules: 116
Number of directives: 256
Number of nodes: 118
Number of reports for one day: 10244
Report database size: 49 GB
Number of lines in reports table: 33399
Full database size: 54 GB
Archiving reports:
archive.TTL=6
delete.TTL=7
frequency=daily

[root@servB report]# ./rudder_metrics_reporting.sh
Number of expected reports (components*directives*nodes): 31
Number of rules: 3
Number of directives: 3
Number of nodes: 0
Number of reports for one day: 9852
Report database size: 16 MB
Number of lines in reports table: 31822
Full database size: 24 MB
Archiving reports:
archive.TTL=6
delete.TTL=7
frequency=daily

Hello,

For the categories, I don't succeeded in reproducing the problem, so I'm wondering if there is something special with your dataset that I don't think to. If that is possible, could you send us (privatly if you want/need, and if it's big we can use a protected sharing site like dl.free.fr) the archive that produce the bug, so that we can try to import it ?

For the database size, there seems to be an incoherency between the two first reports:

- first base: 32639 components, 8 000 000 reports by day, 2e7 lines (so seems almost coherent for 6 or 7 days)
- second base: 32184 components, 10 000 reports by day, 33399 lines

Does the second base have been cleaned up recently ?

Also, something to know about postgress: it does not give back the memory used, you have to reclam it (else postgress keep the memory for itsefl and just mark it as "clean, can be used to store other things"). But has the clean-up process locks the base for the duration of the cleanup, we don't do it automatically. If you want to do it, please see that part of the manual: http://www.rudder-project.org/rudder-doc-2.9/rudder-doc.html#_database_maintenance

Hop it helps,