Project

General

Profile

Actions

Enhancement #9752

closed

Document stopping process before Postgres restore maintenance to avoid primary key constrain get corrupted

Enhancement #9752: Document stopping process before Postgres restore maintenance to avoid primary key constrain get corrupted

Added by Nicolas CHARLES about 9 years ago. Updated over 1 year ago.

Status:
Released
Priority:
1 (highest)
Category:
Web - Compliance & node report
Target version:
UX impact:
Suggestion strength:
User visibility:
Infrequent - complex configurations | third party integrations
Effort required:
Very Small
Name check:
Fix check:
Checked
Regression:
No

Description

After some time, we understood that the problem seems to be that some primary key constraints get dropped during upgrade. So we can have a totally wrong schema!

Logs contains following error:

"[2016-11-30 13:59:04] ERROR com.normation.rudder.batch.AutomaticReportLogger$LAAutomaticReportLogger - could not fetch last id, don't log anything, wait next run, cause is Error when parsing property 'reportLoggerLastId'" 

and indeed, entry is duplicated in database:

select value from rudderproperties where name='reportLoggerLastId';
value
-----------
153729188
153729188

It occured during an upgrade, with the upgrade script that failed due to invalid db name used

Updated by François ARMAND about 9 years ago Actions #1

How can it happens, the property name has an unicity constraint on it?

Updated by Alexandre Anriot about 9 years ago Actions #2

I don't see any such constraint. I did this twice a few minutes ago:

rudder=# INSERT INTO rudderproperties VALUES ('reportLoggerLastId', 161072834);
INSERT 0 1

and got:

rudder=# select * from rudderproperties where name='reportLoggerLastId';
        name        |   value
--------------------+-----------
 reportLoggerLastId | 161072834
 reportLoggerLastId | 161072834
 reportLoggerLastId | 161072834
(3 rows)

Could it be a DB issue on our side?

François ARMAND wrote:

How can it happens, the property name has an unicity constraint on it?

Updated by Vincent MEMBRÉ about 9 years ago Actions #3

  • Target version changed from 4.0.1 to 4.0.2

Updated by François ARMAND about 9 years ago Actions #4

hum.
I just checked, and there is something strange.

On a fresh installation, I have

rudder=> \d rudderproperties;
Table "public.rudderproperties" 
 Column | Type | Modifiers
--------+------+-----------
 name   | text | not null
 value  | text |
Indexes:
    "rudderproperties_pkey" PRIMARY KEY, btree (name)

And trying to insert the same properties fails on pkey violation constraint.

But on the schema definition, we do have the primary key, and we have it since the addition of the table in ca2380af1529a0a717df88c6c57de54e78f31e4e

So I don't get it. You can correct the problem with the following table alteration:

ALTER TABLE RudderProperties ADD PRIMARY KEY (name);

But I don't understand why there is no pk declaration.

For information, could you provide the result of the following query, to see if other pk constraint are missing:

select constraint_name from information_schema.table_constraints where constraint_type = 'PRIMARY KEY');

Updated by Alexandre Anriot about 9 years ago Actions #5

François,

It's probably a local issue, as I have 23 primary keys on the database of another Rudder instance, VS 21 pkeys on this database.

I just added the one you mentionned, as well as statusupdate_pkey.

I guess that you can now close the ticket.

Thanks a lot.

François ARMAND wrote:

hum.
I just checked, and there is something strange.

On a fresh installation, I have
[...]

And trying to insert the same properties fails on pkey violation constraint.

But on the schema definition, we do have the primary key, and we have it since the addition of the table in ca2380af1529a0a717df88c6c57de54e78f31e4e

So I don't get it. You can correct the problem with the following table alteration:

[...]

But I don't understand why there is no pk declaration.

For information, could you provide the result of the following query, to see if other pk constraint are missing:
[...]

Updated by François ARMAND about 9 years ago Actions #6

  • Status changed from New to Discussion
  • Assignee set to Nicolas CHARLES

Wow, this is extremelly strange and worrying. I wasn't aware that postgres could create a schema with some missing constraints (with possibly grave consequences, as it is seen here).

Not sure if we should do something ? Nicolas, any insight on that?

Updated by Nicolas CHARLES about 9 years ago Actions #7

This is weird indeed
my guess: table was restored or copied, and constraint was dropped during restore (possibly, concurrent writing of data prevented constraint to apply)

Updated by Alexandre Anriot about 9 years ago Actions #8

Several changes have been done when the platforms were setup, it's indeed probably something like this.

I think that Rudder is not guilty.

Nicolas CHARLES wrote:

This is weird indeed
my guess: table was restored or copied, and constraint was dropped during restore (possibly, concurrent writing of data prevented constraint to apply)

Updated by François ARMAND about 9 years ago Actions #9

  • Status changed from Discussion to Rejected

OK, thanks for the insight. So as asked, I'm closing the issue.

Updated by François ARMAND almost 7 years ago Actions #10

  • Status changed from Rejected to New
  • Target version changed from 4.0.2 to 5.0.9
  • Severity set to Major - prevents use of part of Rudder | no simple workaround
  • User visibility set to Operational - other Techniques | Rudder settings | Plugins
  • Priority set to 43

I'm reopening that issue because we get the same bug with:

- backup postgres
- upgrade Unbuntu 16.04 => 18.04
- upgrade postgres 9.6 => 10.5
- restore postgres backup
- upgrade Rudder.

Updated by Vincent MEMBRÉ almost 7 years ago Actions #11

  • Target version changed from 5.0.9 to 5.0.10

Updated by Nicolas CHARLES almost 7 years ago Actions #12

Hypothesis is that Rudder restarted during the restore (stopping rudder doesn't disable it - rudder agent check is still run every 5 minutes), causing Rudder to compute the compliance and writing in database
once data got inserted by restore, constraint is broken

Updated by Vincent MEMBRÉ over 6 years ago Actions #13

  • Target version changed from 5.0.10 to 5.0.11

Updated by Vincent MEMBRÉ over 6 years ago Actions #14

  • Target version changed from 5.0.11 to 5.0.12

Updated by Vincent MEMBRÉ over 6 years ago Actions #15

  • Target version changed from 5.0.12 to 5.0.13

Updated by Vincent MEMBRÉ over 6 years ago Actions #16

  • Target version changed from 5.0.13 to 5.0.14

Updated by Vincent MEMBRÉ over 6 years ago Actions #17

  • Target version changed from 5.0.14 to 5.0.15

Updated by François ARMAND about 6 years ago Actions #18

  • Subject changed from Compliance not computed because of duplicate entry in rudderproperties to Postegres primary key constrain get corrupted during rudder upgrade
  • Description updated (diff)

Updated by Vincent MEMBRÉ about 6 years ago Actions #19

  • Target version changed from 5.0.15 to 5.0.16

Updated by Alexis Mousset almost 6 years ago Actions #20

  • Target version changed from 5.0.16 to 5.0.17
  • Priority changed from 43 to 44

Updated by Vincent MEMBRÉ almost 6 years ago Actions #21

  • Target version changed from 5.0.17 to 5.0.18
  • Priority changed from 44 to 45

Updated by Vincent MEMBRÉ over 5 years ago Actions #22

  • Target version changed from 5.0.18 to 5.0.19
  • Priority changed from 45 to 46

Updated by Vincent MEMBRÉ over 5 years ago Actions #23

  • Target version changed from 5.0.19 to 5.0.20

Updated by Vincent MEMBRÉ about 5 years ago Actions #24

  • Target version changed from 5.0.20 to 797
  • Priority changed from 46 to 47

Updated by Alexis Mousset about 5 years ago Actions #25

  • Subject changed from Postegres primary key constrain get corrupted during rudder upgrade to Postgres primary key constrain get corrupted during rudder upgrade
  • Priority changed from 47 to 48

Updated by Benoît PECCATTE over 4 years ago Actions #26

  • Target version changed from 797 to 6.1.14
  • Priority changed from 48 to 50

Updated by Vincent MEMBRÉ over 4 years ago Actions #27

  • Target version changed from 6.1.14 to 6.1.15

Updated by Vincent MEMBRÉ over 4 years ago Actions #28

  • Target version changed from 6.1.15 to 6.1.16

Updated by Vincent MEMBRÉ over 4 years ago Actions #29

  • Target version changed from 6.1.16 to 6.1.17
  • Priority changed from 50 to 51

Updated by Vincent MEMBRÉ about 4 years ago Actions #30

  • Target version changed from 6.1.17 to 6.1.18
  • Priority changed from 51 to 52

Updated by Vincent MEMBRÉ about 4 years ago Actions #31

  • Target version changed from 6.1.18 to 6.1.19

Updated by François ARMAND almost 4 years ago Actions #32

  • Subject changed from Postgres primary key constrain get corrupted during rudder upgrade to Document stopping process before Postgres restore maintenance to avoir primary key constrain get corrupted
  • User visibility changed from Operational - other Techniques | Rudder settings | Plugins to Infrequent - complex configurations | third party integrations
  • Effort required set to Very Small
  • Priority changed from 52 to 70

So, we believe that upgrade are now correctly done, but we need to document the fact that if you want to restore things in postgres, all services that may interact with it (webapp, relayd, and rudder agent that restarts them) need to be disabled.

Updated by François ARMAND almost 4 years ago Actions #33

  • Tracker changed from Bug to Enhancement
  • Subject changed from Document stopping process before Postgres restore maintenance to avoir primary key constrain get corrupted to Document stopping process before Postgres restore maintenance to avoid primary key constrain get corrupted
  • Severity deleted (Major - prevents use of part of Rudder | no simple workaround)
  • Priority deleted (70)

Updated by Vincent MEMBRÉ almost 4 years ago Actions #34

  • Target version changed from 6.1.19 to 6.1.20

Updated by Vincent MEMBRÉ over 3 years ago Actions #35

  • Target version changed from 6.1.20 to 6.1.21

Updated by Vincent MEMBRÉ over 3 years ago Actions #36

  • Target version changed from 6.1.21 to old 6.1 issues to relocate

Updated by Alexis Mousset almost 3 years ago Actions #37

  • Target version changed from old 6.1 issues to relocate to 7.2.5

Updated by Vincent MEMBRÉ almost 3 years ago Actions #38

  • Target version changed from 7.2.5 to 7.2.6

Updated by Vincent MEMBRÉ over 2 years ago Actions #39

  • Target version changed from 7.2.6 to 7.2.7

Updated by Vincent MEMBRÉ over 2 years ago Actions #40

  • Target version changed from 7.2.7 to 7.2.8

Updated by Vincent MEMBRÉ over 2 years ago Actions #41

  • Target version changed from 7.2.8 to 7.2.9

Updated by Vincent MEMBRÉ over 2 years ago Actions #42

  • Target version changed from 7.2.9 to 7.2.10

Updated by Alexis Mousset over 2 years ago Actions #43

  • Target version changed from 7.2.10 to 7.2.11

Updated by Vincent MEMBRÉ over 2 years ago Actions #44

  • Target version changed from 7.2.11 to 1046

Updated by Alexis Mousset over 2 years ago Actions #45

  • Target version changed from 1046 to Ideas (not version specific)

Updated by Elaad FURREEDAN almost 2 years ago Actions #46

  • Target version changed from Ideas (not version specific) to 7.3.14
  • Regression set to No

Updated by Elaad FURREEDAN almost 2 years ago Actions #47

  • Status changed from New to In progress
  • Assignee changed from Nicolas CHARLES to Elaad FURREEDAN

Updated by Elaad FURREEDAN almost 2 years ago Actions #48

  • Status changed from In progress to Pending technical review
  • Assignee changed from Elaad FURREEDAN to Alexis Mousset
  • Pull Request set to https://github.com/Normation/rudder-doc/pull/1032

Updated by Elaad FURREEDAN almost 2 years ago Actions #49

  • Assignee changed from Alexis Mousset to Nicolas CHARLES

Updated by Elaad FURREEDAN almost 2 years ago Actions #50

  • Assignee changed from Nicolas CHARLES to Alexis Mousset

Updated by Elaad FURREEDAN almost 2 years ago Actions #52

  • Assignee changed from Alexis Mousset to Nicolas CHARLES

Updated by Elaad FURREEDAN almost 2 years ago Actions #53

  • Assignee changed from Nicolas CHARLES to Alexis Mousset

Updated by Anonymous almost 2 years ago Actions #54

  • Status changed from Pending technical review to Pending release

Updated by Alexis Mousset almost 2 years ago Actions #55

  • Fix check set to Checked

Updated by Vincent MEMBRÉ over 1 year ago Actions #56

  • Status changed from Pending release to Released

This bug has been fixed in Rudder 7.3.14, 8.0.8 and 8.1.1 which were released today.

Actions

Also available in: PDF Atom