Bug #13690
closed
Impossible to update promises when using a debian9 or Ubuntu 18 server and older distributions as Nodes (incompatible openssl version)
Added by Thomas CAILHE over 6 years ago. Updated 12 days ago.
Description
Hi,
I've got the same error on 2 fresh servers with centos6
error: Failed to establish TLS connection: (0 SSL_ERROR_SSL) tlsv1 alert internal error error: No suitable server found error: Failed to establish TLS connection: (0 SSL_ERROR_SSL) tlsv1 alert internal error error: No suitable server found
error: Rudder agent promises could not be updated. Start execution with config [0]
- server: OpenSSL 1.1.0f 25 May 2017 debian 9
- client : OpenSSL 1.0.1e-fips 11 Feb 2013 (well...) centos 6
UPDATE/RESOLUTION:
In comment 20 belo`we (https://issues.rudder.io/issues/13690#note-20), we though we had a solution for everything, but it wasn't sufficient because 1.0.1 is still not compatible with 1.1.1.
So we ended up embeding OpenSSL everywhere, with:
- version 1.0.2 for very old distros (AIX 5, Centos 3, centos 5..)
- version 1.1.1 everywhere.
It still means that new server (in libssl 1.1.1) won't work for people with agent relying on a OpenSSL 1.0.1 (centos 6 for rudder 5.0.2 and system ssl for ex).
At least new servers work correctly with agent in openssl 1.1.0 (so for ex agent 4.3 on ubuntu 18.04 works with server on 5.0.3).
Files
| agent-debug (236 KB) agent-debug | Nicolas CHARLES, 2018-11-06 17:01 | ||
| server-debug (1.77 MB) server-debug | Nicolas CHARLES, 2018-11-06 17:01 |
Updated by Alexis Mousset over 6 years ago
- Subject changed from Openssl version is to old on centos 6 to Openssl version is too old on CentOS 6
Updated by François ARMAND over 6 years ago
- Severity set to Critical - prevents main use of Rudder | no workaround | data loss | security
- User visibility set to Operational - other Techniques | Technique editor | Rudder settings
- Priority changed from 0 to 76
If confirmed, this one is critical, because it forbids the use of Rudder on centos 6 which is still widelly used.
We got more information by gitter: https://gitter.im/normation/rudder?at=5bc9ad3a435c2a518ecf1193
So, we need to reproduce ASAP:
- server debian 9: OpenSSL 1.1.0f
- client centos 6 : OpenSSL openssl-1.0.1e-57.el6.x86_64
And depending of the result, we may need to embed openssl for centos 6.
Updated by François ARMAND over 6 years ago
- User visibility changed from Operational - other Techniques | Technique editor | Rudder settings to Getting started - demo | first install | level 1 Techniques
- Priority changed from 76 to 94
Updated by François ARMAND over 6 years ago
- Assignee set to Benoît PECCATTE
- Target version set to 5.0.2
Updated by Vincent MEMBRÉ over 6 years ago
- Target version changed from 5.0.2 to 5.0.3
Updated by François ARMAND over 6 years ago
- Status changed from New to In progress
- Assignee changed from Benoît PECCATTE to Vincent MEMBRÉ
We were able to reproduce. There is something strange in the debian 9 (and perhaps ubuntu 18.04) cfengine binary. It seems to be linked to both OpenSSL 1.0 and 1.1. But that does not explains why exactly "debian x to debian 9" works but not "centos 6 to debian 9" does not.
We are working on the analysis of pairs that doesn't not work.
It may be the same root cause as #13766 where the server is ubuntu 18.04 / openssl 1.1, and the agent are in ubuntu 18.04 / openssl 1.0.
Updated by Nicolas CHARLES over 6 years ago
A server Rudder 5.0 on Debian9 with an Agent Centos 6 (5.0 ot 4.3) fails
On the Server side, the logs say:
rudder verbose: Setting minimum acceptable TLS version: 1.0 rudder verbose: Setting cipher list for incoming TLS connections to: AES256-GCM-SHA384:AES256-SHA rudder verbose: Listening for connections on socket descriptor 6 ... notice: Server is starting... rudder verbose: Obtained IP address of '192.168.41.5' on socket 7 from accept rudder verbose: New connection (from 192.168.41.5, sd 7), spawning new thread... rudder info: 192.168.41.5> Accepting connection rudder verbose: 192.168.41.5> Setting socket timeout to 600 seconds. rudder verbose: 192.168.41.5> Peeked nothing important in TCP stream, considering the protocol as TLS error: 192.168.41.5> Failed to accept TLS connection: (-1 SSL_ERROR_SSL) illegal zero content rudder verbose: Obtained IP address of '192.168.41.5' on socket 7 from accept
on the agent side
error: Failed to establish TLS connection: (0 SSL_ERROR_SSL) tlsv1 alert internal error error: No suitable server found error: Failed to establish TLS connection: (0 SSL_ERROR_SSL) tlsv1 alert internal error error: No suitable server found
Updated by Nicolas CHARLES over 6 years ago
- Related to Bug #13766: 5.0 agent on ubuntu 18 not able to connect to 4.3 master added
Updated by François ARMAND over 6 years ago
Some more pair tested: on a Rudder 5.0, ubuntu 18.04:
- centos 7.5, debian 8.9, debian 9.5, ubuntu 18.04: OK
- centos 6.9: not ok.
Updated by Nicolas CHARLES over 6 years ago
- File agent-debug agent-debug added
- File server-debug server-debug added
debug logs of the agent & server
Updated by Nicolas CHARLES over 6 years ago
Ldd results View details...View details...
Server:
# ldd /var/rudder/cfengine-community/bin/cf-serverd
linux-vdso.so.1 (0x00007ffe3215a000)
liblmdb.so => /opt/rudder/lib/liblmdb.so (0x00007f4734dac000)
libpcre.so.3 => /lib/x86_64-linux-gnu/libpcre.so.3 (0x00007f4734b34000)
libssl.so.1.1 => /usr/lib/x86_64-linux-gnu/libssl.so.1.1 (0x00007f47348c8000)
libcrypto.so.1.1 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 (0x00007f4734435000)
libacl.so.1 => /lib/x86_64-linux-gnu/libacl.so.1 (0x00007f473422c000)
libyaml-0.so.2 => /usr/lib/x86_64-linux-gnu/libyaml-0.so.2 (0x00007f473400b000)
libcurl.so.4 => /usr/lib/x86_64-linux-gnu/libcurl.so.4 (0x00007f4733d8b000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f4733b87000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f473397f000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f473367b000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f473345e000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f47330bd000)
libattr.so.1 => /lib/x86_64-linux-gnu/libattr.so.1 (0x00007f4732eb8000)
libnghttp2.so.14 => /usr/lib/x86_64-linux-gnu/libnghttp2.so.14 (0x00007f4732c92000)
libidn2.so.0 => /usr/lib/x86_64-linux-gnu/libidn2.so.0 (0x00007f4732a70000)
librtmp.so.1 => /usr/lib/x86_64-linux-gnu/librtmp.so.1 (0x00007f4732853000)
libssh2.so.1 => /usr/lib/x86_64-linux-gnu/libssh2.so.1 (0x00007f4732627000)
libpsl.so.5 => /usr/lib/x86_64-linux-gnu/libpsl.so.5 (0x00007f4732417000)
libssl.so.1.0.2 => /usr/lib/x86_64-linux-gnu/libssl.so.1.0.2 (0x00007f47321ae000)
libcrypto.so.1.0.2 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.2 (0x00007f4731d4a000)
libgssapi_krb5.so.2 => /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2 (0x00007f4731aff000)
libkrb5.so.3 => /usr/lib/x86_64-linux-gnu/libkrb5.so.3 (0x00007f4731825000)
libk5crypto.so.3 => /usr/lib/x86_64-linux-gnu/libk5crypto.so.3 (0x00007f47315f2000)
libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2 (0x00007f47313ec000)
liblber-2.4.so.2 => /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2 (0x00007f47311dd000)
libldap_r-2.4.so.2 => /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2 (0x00007f4730f8c000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f4730d72000)
/lib64/ld-linux-x86-64.so.2 (0x000055e612175000)
libunistring.so.0 => /usr/lib/x86_64-linux-gnu/libunistring.so.0 (0x00007f4730a5b000)
libgnutls.so.30 => /usr/lib/x86_64-linux-gnu/libgnutls.so.30 (0x00007f47306c0000)
libhogweed.so.4 => /usr/lib/x86_64-linux-gnu/libhogweed.so.4 (0x00007f473048b000)
libnettle.so.6 => /usr/lib/x86_64-linux-gnu/libnettle.so.6 (0x00007f4730254000)
libgmp.so.10 => /usr/lib/x86_64-linux-gnu/libgmp.so.10 (0x00007f472ffd1000)
libgcrypt.so.20 => /lib/x86_64-linux-gnu/libgcrypt.so.20 (0x00007f472fcc1000)
libkrb5support.so.0 => /usr/lib/x86_64-linux-gnu/libkrb5support.so.0 (0x00007f472fab5000)
libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1 (0x00007f472f8af000)
libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007f472f698000)
libsasl2.so.2 => /usr/lib/x86_64-linux-gnu/libsasl2.so.2 (0x00007f472f47d000)
libp11-kit.so.0 => /usr/lib/x86_64-linux-gnu/libp11-kit.so.0 (0x00007f472f218000)
libidn.so.11 => /lib/x86_64-linux-gnu/libidn.so.11 (0x00007f472efe4000)
libtasn1.so.6 => /usr/lib/x86_64-linux-gnu/libtasn1.so.6 (0x00007f472edcf000)
libgpg-error.so.0 => /lib/x86_64-linux-gnu/libgpg-error.so.0 (0x00007f472ebbb000)
libffi.so.6 => /usr/lib/x86_64-linux-gnu/libffi.so.6 (0x00007f472e9b2000)
Node
# ldd /var/rudder/cfengine-community/bin/cf-agent
linux-vdso.so.1 => (0x00007fff22073000)
liblmdb.so => /opt/rudder/lib/liblmdb.so (0x00007f0117c26000)
libacl.so.1 => /lib64/libacl.so.1 (0x00007f0117a19000)
libyaml-0.so.2 => /usr/lib64/libyaml-0.so.2 (0x00007f01177fa000)
libcurl.so.4 => /usr/lib64/libcurl.so.4 (0x00007f01175a5000)
libssl.so.10 => /usr/lib64/libssl.so.10 (0x00007f0117338000)
libcrypto.so.10 => /usr/lib64/libcrypto.so.10 (0x00007f0116f55000)
libpcre.so.0 => /lib64/libpcre.so.0 (0x00007f0116d29000)
libxml2.so.2 => /usr/lib64/libxml2.so.2 (0x00007f01169d5000)
libpam.so.0 => /lib64/libpam.so.0 (0x00007f01167c7000)
libnss_nis.so.2 => /lib64/libnss_nis.so.2 (0x00007f01165bc000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f01163b7000)
librt.so.1 => /lib64/librt.so.1 (0x00007f01161af000)
libm.so.6 => /lib64/libm.so.6 (0x00007f0115f2b000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f0115d0d000)
libc.so.6 => /lib64/libc.so.6 (0x00007f0115979000)
libattr.so.1 => /lib64/libattr.so.1 (0x00007f0115774000)
libidn.so.11 => /lib64/libidn.so.11 (0x00007f0115541000)
libldap-2.4.so.2 => /lib64/libldap-2.4.so.2 (0x00007f01152f1000)
libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007f01150ad000)
libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007f0114dc6000)
libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007f0114b9a000)
libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007f0114996000)
libz.so.1 => /lib64/libz.so.1 (0x00007f011477f000)
libssl3.so => /usr/lib64/libssl3.so (0x00007f0114540000)
libsmime3.so => /usr/lib64/libsmime3.so (0x00007f0114314000)
libnss3.so => /usr/lib64/libnss3.so (0x00007f0113fd4000)
libnssutil3.so => /usr/lib64/libnssutil3.so (0x00007f0113da8000)
libplds4.so => /lib64/libplds4.so (0x00007f0113ba4000)
libplc4.so => /lib64/libplc4.so (0x00007f011399e000)
libnspr4.so => /lib64/libnspr4.so (0x00007f0113760000)
libssh2.so.1 => /usr/lib64/libssh2.so.1 (0x00007f0113538000)
libaudit.so.1 => /lib64/libaudit.so.1 (0x00007f0113314000)
libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007f01130dd000)
libnsl.so.1 => /lib64/libnsl.so.1 (0x00007f0112ec3000)
libnss_files.so.2 => /lib64/libnss_files.so.2 (0x00007f0112cb5000)
/lib64/ld-linux-x86-64.so.2 (0x00007f0117e3c000)
liblber-2.4.so.2 => /lib64/liblber-2.4.so.2 (0x00007f0112aa6000)
libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f011288b000)
libsasl2.so.2 => /usr/lib64/libsasl2.so.2 (0x00007f0112671000)
libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007f0112466000)
libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007f0112262000)
libfreebl3.so => /lib64/libfreebl3.so (0x00007f011205f000)
libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f0111e3f000)
Updated by Nicolas CHARLES over 6 years ago
I tried to set tls_ciphers => "AES128-SHA"; as a workaround, without any success
Updated by Alexis Mousset over 6 years ago
- Subject changed from Openssl version is too old on CentOS 6 to Connection error between agents and servers using openssl 1.0.x <-> 1.1.0
Updated by Alexis Mousset over 6 years ago
- Related to deleted (Bug #13766: 5.0 agent on ubuntu 18 not able to connect to 4.3 master)
Updated by Alexis Mousset over 6 years ago
- Related to Bug #13766: 5.0 agent on ubuntu 18 not able to connect to 4.3 master added
Updated by Alexis Mousset over 6 years ago
- Related to deleted (Bug #13766: 5.0 agent on ubuntu 18 not able to connect to 4.3 master)
Updated by Alexis Mousset over 6 years ago
- Has duplicate Bug #13766: 5.0 agent on ubuntu 18 not able to connect to 4.3 master added
Updated by François ARMAND over 6 years ago
Putting back relevant information from #13766:
- the bug is in OpenSSL certificat serialisation format incompatibility between openssl 1.0 and openssl 1.1.0. OpenSSL was producing not strictly exact certificate serialization which are now rejected.
- it is tracked on openssl: https://github.com/openssl/openssl/issues/7134
- it will be corrected in openssl 1.1.1: https://github.com/openssl/openssl/commit/ca89174bc92c16f0a2a7eb86359b6c6fd1dd7a4d
- other projects have the same problem, for ex: https://monitoring-portal.org/woltlab/index.php?thread/41664-ca-crt-verification-error-with-openssl-1-1-0-illegal-zero-content-in-field-seria/
For Rudder, it means that:
- Agent with openssl 1.0 can't connect to Rudder root server with openssl 1.1.0 (resp agent with openssl 1.1.0 can't connect to root server with openssl 1.0).
- openssl 1.1.0 is used in Rudder 5.0 on ubuntu 18_04, debian 9, and SLES 15
- so you can't mix these versions for root server with any other agent version (included agents on ubuntu 18_04/debian 9/SLES 15 on rudder 4.3 or older), nor you can use agent on these version with an server on any other os/rudder version.
As no distribs will be packaging openssl 1.1.1 until a long time, we can't rely on the distribution support.
If we choose to go for an homogeneous version of openssl, it can only be 1.0 (sinve we support os for agent which don't have 1.1.0 at all), but that means that for ex rudder server 5.0.1 on ubuntu 18_04 won't be able to discuss with rudder agent 5.0.with-the-correction on ubuntu 18_04. This is not possible.
So, the only path forward is to statically compile rudder with openssl 1.1.1 on ubuntu 18_04, debian 9 and SLES 15, for both agent and server.
Updated by Benoît PECCATTE over 6 years ago
- Assignee changed from Vincent MEMBRÉ to Benoît PECCATTE
- Priority changed from 94 to 0
Updated by Benoît PECCATTE over 6 years ago
- Status changed from In progress to Pending technical review
- Assignee changed from Benoît PECCATTE to Alexis Mousset
- Pull Request set to https://github.com/Normation/rudder-packages/pull/1709
Updated by Benoît PECCATTE over 6 years ago
- Status changed from Pending technical review to Pending release
Applied in changeset rudder-packages|544408cbb76f0281660fbf0c32be216553bceeb7.
Updated by Vincent MEMBRÉ over 6 years ago
- Subject changed from Connection error between agents and servers using openssl 1.0.x <-> 1.1.0 to Impossible to update promises when using a debian9 or Ubuntu 18 server and older distributions as Nodes (incomaptible openssl version)
Updated by Vincent MEMBRÉ over 6 years ago
- Subject changed from Impossible to update promises when using a debian9 or Ubuntu 18 server and older distributions as Nodes (incomaptible openssl version) to Impossible to update promises when using a debian9 or Ubuntu 18 server and older distributions as Nodes (incompatible openssl version)
Updated by François ARMAND over 6 years ago
- Description updated (diff)
In comment 20 above (https://issues.rudder.io/issues/13690#note-20), we though we had a solution for everything, but it wasn't sufficient because 1.0.1 is still not compatible with 1.1.1.
So we ended up embeding OpenSSL everywhere, with:
- version 1.0.2 for very old distros (AIX 5, Centos 3, centos 5..)
- version 1.1.1 everywhere.
It still means that people with agent relying on a OpenSSL 1.0.1.
It works correctly with openssl 1.1.0.
Updated by Vincent MEMBRÉ over 6 years ago
- Status changed from Pending release to Released
Changelog
Updated by Félix DALLIDET over 6 years ago
- Related to Bug #14570: Build openssl for Slackware, so the agent can update promises added
Updated by François ARMAND over 5 years ago
- Related to Bug #16224: Missing documentation on openssl incompatibilities between 4.x and 5.0 added
Updated by about 2 years ago · Edited
- Description updated (diff)
- Category changed from Security to Server components
- Severity changed from Critical - prevents main use of Rudder | no workaround | data loss | security to Trivial - no functional impact | cosmetic
- UX impact set to It bothers me each time
- User visibility changed from Getting started - demo | first install | level 1 Techniques to Getting started - demo | first install | Technique editor and level 1 Techniques
- Priority changed from 0 to 57
- Regression set to No
Updated by Alexis Mousset 12 days ago
- Description updated (diff)
- Priority changed from 57 to 59
Updated by Alexis Mousset 12 days ago
- Category changed from Server components to Security
- Severity changed from Trivial - no functional impact | cosmetic to Critical - prevents main use of Rudder | no workaround | data loss | security
- Priority changed from 59 to 106