Project

General

Profile

Actions
Copy link

Bug #21219

closed

relayd can't connect to postgresql on Ubuntu 22.04 server

Added by Alexis Mousset almost 2 years ago. Updated almost 2 years ago.

Status:
Released
Priority:
N/A
Assignee:
Benoît PECCATTE
Category:
Relay server or API
Target version:
7.1.2
Pull Request:
https://github.com/Normation/rudder-p...
Severity:
Major - prevents use of part of Rudder | no simple workaround
UX impact:
User visibility:
First impressions of Rudder
Effort required:
Priority:
100
Name check:
To do
Fix check:
Checked
Regression:

Description

NOTE: A workaround is available

Attached binaries are builds of relayd (from the given tags) that can be used as workaround.

sha256sums:

7.1.1  ae4a5625e28867ff70a08541a3af3d34151d9f367a813c1d1b930ca07ec4962e 
7.1.2  5eeadf7c02b39ea3e75c46ae083cb6b09dd48b903bb94338c119ac0f4026a32c

To fix the problem, on your Rudder root server:

systemctl stop rudder-relayd
cp rudder-relayd /opt/rudder/bin/rudder-relayd
systemctl start rudder-relayd

You relayd service should now run correctly

Server installs correctly, but relayd refuses to connect:

rudder-relayd[15117]: ERROR r2d2: connection to server at "localhost" (127.0.0.1), port 5432 failed: SSL error: i2d ecpkparameters failure
rudder-relayd[15117]: connection to server at "localhost" (127.0.0.1), port 5432 failed: FATAL:  password authentication failed for user "rudder"

and on the postgresql side:

rudder@rudder DETAIL:  Connection matched pg_hba.conf line 95: "host    all             rudder          127.0.0.1/32            md5" 
rudder@rudder FATAL:  password authentication failed for user "rudder"

which looks like a standard "wrong password" case (and the matched acl is the right one).

When adding an sslmode=disable the SSL error disappears, but the connection still fails.

ERROR r2d2: connection to server at "localhost" (127.0.0.1), port 5432 failed: FATAL:  password authentication failed for user "rudder"

The connection details look correct:

# grep -A3 output.database /opt/rudder/etc/relayd/main.conf 
[output.database]
url = "postgresql://rudder@localhost:5432/rudder" 
password = "7cb3e8fad6afd0a07efa" 

# grep PSQL /opt/rudder/etc/rudder-passwords.conf 
RUDDER_PSQL_PASSWORD:7cb3e8fad6afd0a07efa

And connection works with this password:

$ psql postgresql://rudder@localhost:5432/rudder?password=7cb3e8fad6afd0a07efa
psql (14.3 (Ubuntu 14.3-0ubuntu0.22.04.1))
SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, bits: 256, compression: off)
Type "help" for help.
rudder=>

This has only been seen on Ubuntu 22.04 for now (with Rudder 7.1.1). This is notably our most recent supported OS (with openssl 3.0 and postgresql 14).

relayd uses the system libpq:

# ldd /opt/rudder/bin/rudder-relayd | grep pq
    libpq.so.5 => /lib/x86_64-linux-gnu/libpq.so.5 (0x00007fd4513e1000)

# dpkg -l  | grep postg
ii  postgresql                      14+238                                  all          object-relational SQL database (supported version)
ii  postgresql-14                   14.3-0ubuntu0.22.04.1                   amd64        The World's Most Advanced Open Source Relational Database
ii  postgresql-client               14+238                                  all          front-end programs for PostgreSQL (supported version)
ii  postgresql-client-14            14.3-0ubuntu0.22.04.1                   amd64        front-end programs for PostgreSQL 14
ii  postgresql-client-common        238                                     all          manager for multiple PostgreSQL client versions
ii  postgresql-common               238                                     all          PostgreSQL database-cluster manager

Files

Download all files
rudder-relayd (11.5 MB) rudder-relayd 7.1.1-1 build for Ubuntu 22.04 Alexis Mousset, 2022-06-08 12:34
rudder-relayd (11.5 MB) rudder-relayd 7.1.2 build for Ubuntu 22.04 Alexis Mousset, 2022-06-19 19:30
Actions
Copy link
 #1

Updated by Alexis Mousset almost 2 years ago

  • Subject changed from relayd can't connect to postgresal on Ubuntu 22.04 server to relayd can't connect to postgresql on Ubuntu 22.04 server
Actions
Copy link
 #2

Updated by Alexis Mousset almost 2 years ago

  • Description updated (diff)
Actions
Copy link
 #3

Updated by Alexis Mousset almost 2 years ago

  • Description updated (diff)
Actions
Copy link
 #4

Updated by Alexis Mousset almost 2 years ago

  • Description updated (diff)
Actions
Copy link
 #5

Updated by Alexis Mousset almost 2 years ago

  • Description updated (diff)
Actions
Copy link
 #6

Updated by Alexis Mousset almost 2 years ago

A network dump shows:

this

Actions
Copy link
 #7

Updated by Alexis Mousset almost 2 years ago

A working connection looks like:

this

Actions
Copy link
 #8

Updated by Alexis Mousset almost 2 years ago

Postgresql 14.0 changes the default password_encryption:

Change the default of the password_encryption server parameter to scram-sha-256 (Peter Eisentraut)
Previously it was md5. All new passwords will be stored as SHA256 unless this server setting is changed or the password is specified in MD5 format. Also, the legacy (and undocumented) Boolean-like values ??which were previously synonyms for md5 are no longer accepted.
-- https://www.postgresql.org/docs/release/14.0/

And is is indeed what is used by relayd (and psql), but the difference may be linked to this.

Actions
Copy link
 #9

Updated by Alexis Mousset almost 2 years ago

It's not, fails with md5 hashing too.

Actions
Copy link
 #10

Updated by Alexis Mousset almost 2 years ago

but it works with clear-test password setting in ph_hba.conf...

Actions
Copy link
 #11

Updated by Alexis Mousset almost 2 years ago

  • Severity set to Major - prevents use of part of Rudder | no simple workaround
  • User visibility set to First impressions of Rudder
  • Priority changed from 0 to 100
Actions
Copy link
 #12

Updated by Alexis Mousset almost 2 years ago

  • Status changed from New to In progress

I'll try to reproduce with a minimal program.

use diesel::{
    pg::PgConnection,
    r2d2::{ConnectionManager, Pool},
};

fn main() {
    let manager = ConnectionManager::<PgConnection>::new(
        "postgresql://rudder@localhost:5432/rudder?password=f040c1d753051b08e6d9",
    );
    Pool::builder().max_size(10).build(manager).unwrap();
    println!("OK");
}
Actions
Copy link
 #13

Updated by Alexis Mousset almost 2 years ago

It works when recompiling the 7.1.1 tag inside the test VM, but when reinstalling the package the problem comes back.

Actions
Copy link
 #14

Updated by Alexis Mousset almost 2 years ago 

# from 7.1.1 package
-rwxr-xr-x 1 root    root    11594024 Nov 22  2017 /opt/rudder/bin/rudder-relayd*
# nightly from 3th June
-rwxr-xr-x 1 vagrant vagrant 11577640 Nov 22  2017 /vagrant/rudder-relayd*
# built with cargo build --release --locked | strip -s
-rwxr-xr-x 2 root    root    12078456 Jun  8 09:45 target/release/rudder-relayd*

this is strange...

Nightly fails too.

Actions
Copy link
 #15

Updated by Alexis Mousset almost 2 years ago

The build log shows:

11:44:09 Selecting previously unselected package libpq5:amd64.
11:44:09 Preparing to unpack .../44-libpq5_14.3-0ubuntu0.22.04.1_amd64.deb ...
11:44:09 Unpacking libpq5:amd64 (14.3-0ubuntu0.22.04.1) ...
...
11:44:36 RUSTFLAGS="--codegen link-arg=-Wl,--strip-all" cargo build --release --locked
11:44:37    Compiling proc-macro2 v1.0.36
11:44:37    Compiling cfg-if v1.0.0
11:44:37    Compiling unicode-xid v0.2.2
11:44:37    Compiling syn v1.0.85
11:44:37    Compiling autocfg v1.0.1
...
11:46:32    Compiling hyper-tls v0.5.0
11:46:33    Compiling reqwest v0.11.9
11:46:35    Compiling rudder-relayd v7.1.1 (/srv/jenkins/workspace/Rudder-release-Ubuntu-22.04/rudder-packages/rudder-server-relay/BUILD/rudder/relay/sources/relayd)
11:48:21     Finished release [optimized] target(s) in 3m 44s

so it looks normal.

Actions
Copy link
 #16

Updated by Alexis Mousset almost 2 years ago

Database config looks OK with package binary:

DatabaseConfig {
  url: "postgresql://rudder@localhost:5432/rudder",
  password: "******",
  max_pool_size: 10,
}
Actions
Copy link
 #17

Updated by Alexis Mousset almost 2 years ago

Attached binary is a 7.1.1-1 build of relayd that can be used as workaround.

sha256sum:

ae4a5625e28867ff70a08541a3af3d34151d9f367a813c1d1b930ca07ec4962e

To fix the problem, on your Rudder root server:

systemctl stop rudder-relayd
cp rudder-relayd /opt/rudder/bin/rudder-relayd
systemctl start rudder-relayd
Actions
Copy link
 #18

Updated by Alexis Mousset almost 2 years ago

Note: All my tests were done in the official vagrant box ubuntu/jammy64 with up-to-date packages.

Actions
Copy link
 #19

Updated by Alexis Mousset almost 2 years ago

Next steps:

  • Try to reproduce the behavior change when switching to clear-text password
  • Try to build relayd manually in the chroot used for packaging
Actions
Copy link
 #20

Updated by Alexis Mousset almost 2 years ago

  • Description updated (diff)
Actions
Copy link
 #21

Updated by Alexis Mousset almost 2 years ago

  • Description updated (diff)
Actions
Copy link
 #22

Updated by Alexis Mousset almost 2 years ago

  • Description updated (diff)
Actions
Copy link
 #23

Updated by Vincent MEMBRÉ almost 2 years ago

  • Target version changed from 7.1.2 to 7.1.3
Actions
Copy link
 #24

Updated by Alexis Mousset almost 2 years ago

I can reproduce the problem by rebuilding the 7.1.1 .deb package in my test VM, so the problem lies somewhere in the deb build process.

Actions
Copy link
 #26

Updated by Alexis Mousset almost 2 years ago

  • Description updated (diff)
Actions
Copy link
 #27

Updated by Alexis Mousset almost 2 years ago

The output of the Rust build is indeed different:

-rwxr-xr-x 2 root root 11524504 Jun 19 18:35 /root/rudder-packages/rudder-server-relay/BUILD/rudder/relay/sources/relayd/target/release/rudder-relayd*
-rwxr-xr-x 2 root root 12070264 Jun 19 17:27 /root/rudder/relay/sources/relayd/target/release/rudder-relayd*

Rebuilding in the package sources produce a working binary.

Actions
Copy link
 #28

Updated by Alexis Mousset almost 2 years ago

  • Status changed from In progress to Pending technical review
  • Assignee changed from Alexis Mousset to Benoît PECCATTE
  • Pull Request set to https://github.com/Normation/rudder-packages/pull/2620
Actions
Copy link
 #29

Updated by Alexis Mousset almost 2 years ago

The minimal set of options to reproduce the problem was CFLAGS="-flto=auto -ffat-lto-objects", and it hapens because Ubuntu activated LTO by default since 21.04. This somehow breaks something between relayd and libpq.

As explained in Ubuntu wiki this can be disabled with export DEB_BUILD_MAINT_OPTIONS=optimize=-lto in debian/rules.

Actions
Copy link
 #30

Updated by Alexis Mousset almost 2 years ago

  • Status changed from Pending technical review to Pending release
Actions
Copy link
 #31

Updated by Vincent MEMBRÉ almost 2 years ago

  • Target version changed from 7.1.3 to 7.1.2
Actions
Copy link
 #32

Updated by Vincent MEMBRÉ almost 2 years ago

  • Fix check changed from To do to Checked
Actions
Copy link
 #33

Updated by Vincent MEMBRÉ almost 2 years ago

  • Status changed from Pending release to Released

This bug has been fixed in Rudder 7.1.2 which was released today.

Actions
Copy link

Also available in: Atom PDF