Project

General

Profile

Actions
Copy link

Architecture #16051

open

Re-authentication error in postgres doesn't kill existing pool

Added by Elaad FURREEDAN about 5 years ago. Updated 20 days ago.

Status:
New
Priority:
N/A
Assignee:
-
Category:
Server components
Target version:
8.1.11
Pull Request:
Effort required:
Name check:
To do
Fix check:
To do
Regression:

Files

thread_jvm.log (125 KB) thread_jvm.log Elaad FURREEDAN, 2019-10-24 14:30

Related issues 2 (0 open — 2 closed)

Related to Rudder - Bug #16055: java.lang.Error null in PooledConnectionProviderRejectedActions
Related to Rudder - Bug #12489: Cannot login into 2 different Rudder with the same URL - apocalypse ensuesRejectedActions
Actions
Copy link
 #1

Updated by Nicolas CHARLES about 5 years ago

there are no obvious locking there

Actions
Copy link
 #2

Updated by Elaad FURREEDAN about 5 years ago

  • File before_error.log added
Actions
Copy link
 #3

Updated by Elaad FURREEDAN about 5 years ago

  • File deleted (before_error.log)
Actions
Copy link
 #4

Updated by François ARMAND about 5 years ago

  • Related to Bug #16055: java.lang.Error null in PooledConnectionProvider added
Actions
Copy link
 #5

Updated by François ARMAND about 5 years ago

  • Subject changed from Rudder is very slow to Authentication error to postgres cause major slowdown
  • Category set to System integration
  • Target version changed from 6.0.0~beta1 to 6.0.0

The problem is linked to Postgres DB connexion dying and the pool does not seems to notice. We can see it very clearly in the log here: https://issues.rudder.io/attachments/1549

So we have the connexion starting to fails around 2019-10-24 12:35:38+0200 - most likely rudder agent is running and reconfigured something on pg, restarting the db after that.
Then, at 2019-10-24 14:16:20+0200 we finaly have:

ERROR com.normation.rudder.rest.v1.RestTechniqueReload - Root exception cause was:
org.postgresql.util.PSQLException: FATAL: password authentication failed for user "rudder" 
    at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:520)
    at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:141)
    at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:192)

And then, everything related to PG fails. It's as if PG was restarted to take care of new auth info, but the connexion didn't noticed the pb. Moreover, Rudder should have been restarted to get the new info, but wasn't.

Actions
Copy link
 #6

Updated by Elaad FURREEDAN about 5 years ago

  • Related to Bug #12489: Cannot login into 2 different Rudder with the same URL - apocalypse ensues added
Actions
Copy link
 #7

Updated by Elaad FURREEDAN about 5 years ago

When I have two Rudder's env running, after ~10mins the bug occurs

Actions
Copy link
 #8

Updated by Elaad FURREEDAN about 5 years ago

  • Category changed from System integration to Web - Technique editor
Actions
Copy link
 #9

Updated by Alexis Mousset about 5 years ago

  • Target version changed from 6.0.0 to 6.0.1
Actions
Copy link
 #10

Updated by Vincent MEMBRÉ about 5 years ago

  • Target version changed from 6.0.1 to 6.0.2
Actions
Copy link
 #11

Updated by Vincent MEMBRÉ almost 5 years ago

  • Target version changed from 6.0.2 to 6.0.3
Actions
Copy link
 #12

Updated by Vincent MEMBRÉ almost 5 years ago

  • Target version changed from 6.0.3 to 6.0.4
Actions
Copy link
 #13

Updated by Vincent MEMBRÉ almost 5 years ago

  • Target version changed from 6.0.4 to 6.0.5
Actions
Copy link
 #14

Updated by Vincent MEMBRÉ over 4 years ago

  • Target version changed from 6.0.5 to 6.0.6
Actions
Copy link
 #15

Updated by Vincent MEMBRÉ over 4 years ago

  • Target version changed from 6.0.6 to 6.0.7
Actions
Copy link
 #16

Updated by Vincent MEMBRÉ over 4 years ago

  • Target version changed from 6.0.7 to 6.0.8
Actions
Copy link
 #17

Updated by François ARMAND over 4 years ago

  • Subject changed from Authentication error to postgres cause major slowdown to Re-authentication error in postgres doesn't kill existing pool
  • Severity set to Major - prevents use of part of Rudder | no simple workaround
  • User visibility set to Operational - other Techniques | Rudder settings | Plugins
  • Priority changed from 0 to 46
Actions
Copy link
 #18

Updated by Vincent MEMBRÉ over 4 years ago

  • Target version changed from 6.0.8 to 6.0.9
  • Priority changed from 46 to 45
Actions
Copy link
 #19

Updated by Vincent MEMBRÉ over 4 years ago

  • Target version changed from 6.0.9 to 6.0.10
  • Priority changed from 45 to 44
Actions
Copy link
 #20

Updated by Vincent MEMBRÉ about 4 years ago

  • Target version changed from 6.0.10 to 798
  • Priority changed from 44 to 43
Actions
Copy link
 #21

Updated by Benoît PECCATTE over 3 years ago

  • Target version changed from 798 to 6.1.14
Actions
Copy link
 #22

Updated by Vincent MEMBRÉ over 3 years ago

  • Target version changed from 6.1.14 to 6.1.15
Actions
Copy link
 #23

Updated by Vincent MEMBRÉ over 3 years ago

  • Target version changed from 6.1.15 to 6.1.16
Actions
Copy link
 #24

Updated by Vincent MEMBRÉ over 3 years ago

  • Target version changed from 6.1.16 to 6.1.17
Actions
Copy link
 #25

Updated by Vincent MEMBRÉ about 3 years ago

  • Target version changed from 6.1.17 to 6.1.18
Actions
Copy link
 #26

Updated by Vincent MEMBRÉ about 3 years ago

  • Target version changed from 6.1.18 to 6.1.19
Actions
Copy link
 #27

Updated by Alexis Mousset about 3 years ago

  • Category changed from Web - Technique editor to Server components
Actions
Copy link
 #28

Updated by François ARMAND almost 3 years ago

  • Tracker changed from Bug to Architecture
  • Severity deleted (Major - prevents use of part of Rudder | no simple workaround)
  • User visibility deleted (Operational - other Techniques | Rudder settings | Plugins)
  • Priority deleted (43)
Actions
Copy link
 #29

Updated by Vincent MEMBRÉ over 2 years ago

  • Target version changed from 6.1.19 to 6.1.20
Actions
Copy link
 #30

Updated by Vincent MEMBRÉ over 2 years ago

  • Target version changed from 6.1.20 to 6.1.21
Actions
Copy link
 #31

Updated by Vincent MEMBRÉ over 2 years ago

  • Target version changed from 6.1.21 to old 6.1 issues to relocate
Actions
Copy link
 #32

Updated by Alexis Mousset almost 2 years ago

  • Target version changed from old 6.1 issues to relocate to 8.0.0~alpha1
Actions
Copy link
 #33

Updated by Vincent MEMBRÉ over 1 year ago

  • Target version changed from 8.0.0~alpha1 to 8.0.0~beta1
Actions
Copy link
 #34

Updated by Alexis Mousset over 1 year ago

  • Target version changed from 8.0.0~beta1 to 8.1.0~alpha1
Actions
Copy link
 #35

Updated by Vincent MEMBRÉ 11 months ago

  • Target version changed from 8.1.0~alpha1 to 8.1.0~beta1
Actions
Copy link
 #36

Updated by Vincent MEMBRÉ 10 months ago

  • Target version changed from 8.1.0~beta1 to 8.1.0~beta2
Actions
Copy link
 #37

Updated by Vincent MEMBRÉ 9 months ago

  • Target version changed from 8.1.0~beta2 to 8.1.0~rc1
Actions
Copy link
 #38

Updated by Vincent MEMBRÉ 9 months ago

  • Target version changed from 8.1.0~rc1 to 8.1.0
Actions
Copy link
 #39

Updated by Vincent MEMBRÉ 9 months ago

  • Target version changed from 8.1.0 to 8.1.1
Actions
Copy link
 #40

Updated by Vincent MEMBRÉ 8 months ago

  • Target version changed from 8.1.1 to 8.1.2
Actions
Copy link
 #41

Updated by Vincent MEMBRÉ 8 months ago

  • Target version changed from 8.1.2 to 8.1.3
Actions
Copy link
 #42

Updated by Vincent MEMBRÉ 7 months ago

  • Target version changed from 8.1.3 to 8.1.4
Actions
Copy link
 #43

Updated by Vincent MEMBRÉ 6 months ago

  • Target version changed from 8.1.4 to 8.1.5
Actions
Copy link
 #44

Updated by Vincent MEMBRÉ 6 months ago

  • Target version changed from 8.1.5 to 8.1.6
Actions
Copy link
 #45

Updated by Vincent MEMBRÉ 5 months ago

  • Target version changed from 8.1.6 to 8.1.7
Actions
Copy link
 #46

Updated by Vincent MEMBRÉ 3 months ago

  • Target version changed from 8.1.7 to 8.1.8
Actions
Copy link
 #47

Updated by Vincent MEMBRÉ about 1 month ago

  • Target version changed from 8.1.8 to 8.1.9
Actions
Copy link
 #48

Updated by Vincent MEMBRÉ 25 days ago

  • Target version changed from 8.1.9 to 8.1.10
Actions
Copy link
 #49

Updated by Vincent MEMBRÉ 20 days ago

  • Target version changed from 8.1.10 to 8.1.11
Actions
Copy link

Also available in: Atom PDF