Bug #13690: Impossible to update promises when using a debian9 or Ubuntu 18 server and older distributions as Nodes (incompatible openssl version) - Rudder - Issue Tracker

Actions

Copy link

#1

Updated by Alexis Mousset about 6 years ago

Subject changed from Openssl version is to old on centos 6 to Openssl version is too old on CentOS 6

Actions

Copy link

#2

Updated by François ARMAND about 6 years ago

Severity set to Critical - prevents main use of Rudder | no workaround | data loss | security
User visibility set to Operational - other Techniques | Technique editor | Rudder settings
Priority changed from 0 to 76

If confirmed, this one is critical, because it forbids the use of Rudder on centos 6 which is still widelly used.

We got more information by gitter: https://gitter.im/normation/rudder?at=5bc9ad3a435c2a518ecf1193

So, we need to reproduce ASAP:

- server debian 9: OpenSSL 1.1.0f
- client centos 6 : OpenSSL openssl-1.0.1e-57.el6.x86_64

And depending of the result, we may need to embed openssl for centos 6.

Actions

Copy link

#3

Updated by François ARMAND about 6 years ago

Category set to Security

Actions

Copy link

#4

Updated by François ARMAND about 6 years ago

User visibility changed from Operational - other Techniques | Technique editor | Rudder settings to Getting started - demo | first install | level 1 Techniques
Priority changed from 76 to 94

Actions

Copy link

#5

Updated by François ARMAND about 6 years ago

Assignee set to Benoît PECCATTE
Target version set to 5.0.2

Actions

Copy link

#6

Updated by Vincent MEMBRÉ about 6 years ago

Target version changed from 5.0.2 to 5.0.3

Actions

Copy link

#7

Updated by François ARMAND about 6 years ago

Status changed from New to In progress
Assignee changed from Benoît PECCATTE to Vincent MEMBRÉ

We were able to reproduce. There is something strange in the debian 9 (and perhaps ubuntu 18.04) cfengine binary. It seems to be linked to both OpenSSL 1.0 and 1.1. But that does not explains why exactly "debian x to debian 9" works but not "centos 6 to debian 9" does not.

We are working on the analysis of pairs that doesn't not work.
It may be the same root cause as #13766 where the server is ubuntu 18.04 / openssl 1.1, and the agent are in ubuntu 18.04 / openssl 1.0.

Actions

Copy link

#8

Updated by Nicolas CHARLES about 6 years ago

A server Rudder 5.0 on Debian9 with an Agent Centos 6 (5.0 ot 4.3) fails
On the Server side, the logs say:

rudder  verbose: Setting minimum acceptable TLS version: 1.0
rudder  verbose: Setting cipher list for incoming TLS connections to: AES256-GCM-SHA384:AES256-SHA
rudder  verbose: Listening for connections on socket descriptor 6 ...
  notice: Server is starting...
rudder  verbose: Obtained IP address of '192.168.41.5' on socket 7 from accept
rudder  verbose: New connection (from 192.168.41.5, sd 7), spawning new thread...
rudder     info: 192.168.41.5>    Accepting connection
rudder  verbose: 192.168.41.5>    Setting socket timeout to 600 seconds.
rudder  verbose: 192.168.41.5>    Peeked nothing important in TCP stream, considering the protocol as TLS
   error: 192.168.41.5>    Failed to accept TLS connection: (-1 SSL_ERROR_SSL) illegal zero content 
rudder  verbose: Obtained IP address of '192.168.41.5' on socket 7 from accept

on the agent side

   error: Failed to establish TLS connection: (0 SSL_ERROR_SSL) tlsv1 alert internal error 
   error: No suitable server found
   error: Failed to establish TLS connection: (0 SSL_ERROR_SSL) tlsv1 alert internal error 
   error: No suitable server found

Actions

Copy link

#9

Updated by Nicolas CHARLES about 6 years ago

Related to Bug #13766: 5.0 agent on ubuntu 18 not able to connect to 4.3 master added

Actions

Copy link

#10

Updated by François ARMAND about 6 years ago

Some more pair tested: on a Rudder 5.0, ubuntu 18.04:

- centos 7.5, debian 8.9, debian 9.5, ubuntu 18.04: OK
- centos 6.9: not ok.

Actions

Copy link Download all files

#11

Updated by Nicolas CHARLES about 6 years ago

File agent-debug agent-debug added
File server-debug server-debug added

debug logs of the agent & server

Actions

Copy link

#12

Updated by Nicolas CHARLES about 6 years ago

Ldd results View details...View details...

Server:

# ldd /var/rudder/cfengine-community/bin/cf-serverd 
    linux-vdso.so.1 (0x00007ffe3215a000)
    liblmdb.so => /opt/rudder/lib/liblmdb.so (0x00007f4734dac000)
    libpcre.so.3 => /lib/x86_64-linux-gnu/libpcre.so.3 (0x00007f4734b34000)
    libssl.so.1.1 => /usr/lib/x86_64-linux-gnu/libssl.so.1.1 (0x00007f47348c8000)
    libcrypto.so.1.1 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 (0x00007f4734435000)
    libacl.so.1 => /lib/x86_64-linux-gnu/libacl.so.1 (0x00007f473422c000)
    libyaml-0.so.2 => /usr/lib/x86_64-linux-gnu/libyaml-0.so.2 (0x00007f473400b000)
    libcurl.so.4 => /usr/lib/x86_64-linux-gnu/libcurl.so.4 (0x00007f4733d8b000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f4733b87000)
    librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f473397f000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f473367b000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f473345e000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f47330bd000)
    libattr.so.1 => /lib/x86_64-linux-gnu/libattr.so.1 (0x00007f4732eb8000)
    libnghttp2.so.14 => /usr/lib/x86_64-linux-gnu/libnghttp2.so.14 (0x00007f4732c92000)
    libidn2.so.0 => /usr/lib/x86_64-linux-gnu/libidn2.so.0 (0x00007f4732a70000)
    librtmp.so.1 => /usr/lib/x86_64-linux-gnu/librtmp.so.1 (0x00007f4732853000)
    libssh2.so.1 => /usr/lib/x86_64-linux-gnu/libssh2.so.1 (0x00007f4732627000)
    libpsl.so.5 => /usr/lib/x86_64-linux-gnu/libpsl.so.5 (0x00007f4732417000)
    libssl.so.1.0.2 => /usr/lib/x86_64-linux-gnu/libssl.so.1.0.2 (0x00007f47321ae000)
    libcrypto.so.1.0.2 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.2 (0x00007f4731d4a000)
    libgssapi_krb5.so.2 => /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2 (0x00007f4731aff000)
    libkrb5.so.3 => /usr/lib/x86_64-linux-gnu/libkrb5.so.3 (0x00007f4731825000)
    libk5crypto.so.3 => /usr/lib/x86_64-linux-gnu/libk5crypto.so.3 (0x00007f47315f2000)
    libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2 (0x00007f47313ec000)
    liblber-2.4.so.2 => /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2 (0x00007f47311dd000)
    libldap_r-2.4.so.2 => /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2 (0x00007f4730f8c000)
    libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f4730d72000)
    /lib64/ld-linux-x86-64.so.2 (0x000055e612175000)
    libunistring.so.0 => /usr/lib/x86_64-linux-gnu/libunistring.so.0 (0x00007f4730a5b000)
    libgnutls.so.30 => /usr/lib/x86_64-linux-gnu/libgnutls.so.30 (0x00007f47306c0000)
    libhogweed.so.4 => /usr/lib/x86_64-linux-gnu/libhogweed.so.4 (0x00007f473048b000)
    libnettle.so.6 => /usr/lib/x86_64-linux-gnu/libnettle.so.6 (0x00007f4730254000)
    libgmp.so.10 => /usr/lib/x86_64-linux-gnu/libgmp.so.10 (0x00007f472ffd1000)
    libgcrypt.so.20 => /lib/x86_64-linux-gnu/libgcrypt.so.20 (0x00007f472fcc1000)
    libkrb5support.so.0 => /usr/lib/x86_64-linux-gnu/libkrb5support.so.0 (0x00007f472fab5000)
    libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1 (0x00007f472f8af000)
    libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007f472f698000)
    libsasl2.so.2 => /usr/lib/x86_64-linux-gnu/libsasl2.so.2 (0x00007f472f47d000)
    libp11-kit.so.0 => /usr/lib/x86_64-linux-gnu/libp11-kit.so.0 (0x00007f472f218000)
    libidn.so.11 => /lib/x86_64-linux-gnu/libidn.so.11 (0x00007f472efe4000)
    libtasn1.so.6 => /usr/lib/x86_64-linux-gnu/libtasn1.so.6 (0x00007f472edcf000)
    libgpg-error.so.0 => /lib/x86_64-linux-gnu/libgpg-error.so.0 (0x00007f472ebbb000)
    libffi.so.6 => /usr/lib/x86_64-linux-gnu/libffi.so.6 (0x00007f472e9b2000)

Node

# ldd /var/rudder/cfengine-community/bin/cf-agent 
    linux-vdso.so.1 =>  (0x00007fff22073000)
    liblmdb.so => /opt/rudder/lib/liblmdb.so (0x00007f0117c26000)
    libacl.so.1 => /lib64/libacl.so.1 (0x00007f0117a19000)
    libyaml-0.so.2 => /usr/lib64/libyaml-0.so.2 (0x00007f01177fa000)
    libcurl.so.4 => /usr/lib64/libcurl.so.4 (0x00007f01175a5000)
    libssl.so.10 => /usr/lib64/libssl.so.10 (0x00007f0117338000)
    libcrypto.so.10 => /usr/lib64/libcrypto.so.10 (0x00007f0116f55000)
    libpcre.so.0 => /lib64/libpcre.so.0 (0x00007f0116d29000)
    libxml2.so.2 => /usr/lib64/libxml2.so.2 (0x00007f01169d5000)
    libpam.so.0 => /lib64/libpam.so.0 (0x00007f01167c7000)
    libnss_nis.so.2 => /lib64/libnss_nis.so.2 (0x00007f01165bc000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007f01163b7000)
    librt.so.1 => /lib64/librt.so.1 (0x00007f01161af000)
    libm.so.6 => /lib64/libm.so.6 (0x00007f0115f2b000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f0115d0d000)
    libc.so.6 => /lib64/libc.so.6 (0x00007f0115979000)
    libattr.so.1 => /lib64/libattr.so.1 (0x00007f0115774000)
    libidn.so.11 => /lib64/libidn.so.11 (0x00007f0115541000)
    libldap-2.4.so.2 => /lib64/libldap-2.4.so.2 (0x00007f01152f1000)
    libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007f01150ad000)
    libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007f0114dc6000)
    libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007f0114b9a000)
    libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007f0114996000)
    libz.so.1 => /lib64/libz.so.1 (0x00007f011477f000)
    libssl3.so => /usr/lib64/libssl3.so (0x00007f0114540000)
    libsmime3.so => /usr/lib64/libsmime3.so (0x00007f0114314000)
    libnss3.so => /usr/lib64/libnss3.so (0x00007f0113fd4000)
    libnssutil3.so => /usr/lib64/libnssutil3.so (0x00007f0113da8000)
    libplds4.so => /lib64/libplds4.so (0x00007f0113ba4000)
    libplc4.so => /lib64/libplc4.so (0x00007f011399e000)
    libnspr4.so => /lib64/libnspr4.so (0x00007f0113760000)
    libssh2.so.1 => /usr/lib64/libssh2.so.1 (0x00007f0113538000)
    libaudit.so.1 => /lib64/libaudit.so.1 (0x00007f0113314000)
    libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007f01130dd000)
    libnsl.so.1 => /lib64/libnsl.so.1 (0x00007f0112ec3000)
    libnss_files.so.2 => /lib64/libnss_files.so.2 (0x00007f0112cb5000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f0117e3c000)
    liblber-2.4.so.2 => /lib64/liblber-2.4.so.2 (0x00007f0112aa6000)
    libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f011288b000)
    libsasl2.so.2 => /usr/lib64/libsasl2.so.2 (0x00007f0112671000)
    libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007f0112466000)
    libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007f0112262000)
    libfreebl3.so => /lib64/libfreebl3.so (0x00007f011205f000)
    libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f0111e3f000)

Actions

Copy link

#13

Updated by Nicolas CHARLES about 6 years ago

I tried to set tls_ciphers => "AES128-SHA"; as a workaround, without any success

Actions

Copy link

#14

Updated by Alexis Mousset about 6 years ago

Subject changed from Openssl version is too old on CentOS 6 to Connection error between agents and servers using openssl 1.0.x <-> 1.1.0

Actions

Copy link

#15

Updated by Alexis Mousset about 6 years ago

Description updated (diff)

Actions

Copy link

#16

Updated by Alexis Mousset about 6 years ago

Related to deleted (Bug #13766: 5.0 agent on ubuntu 18 not able to connect to 4.3 master)

Actions

Copy link

#17

Updated by Alexis Mousset about 6 years ago

Related to Bug #13766: 5.0 agent on ubuntu 18 not able to connect to 4.3 master added

Actions

Copy link

#18

Updated by Alexis Mousset about 6 years ago

Related to deleted (Bug #13766: 5.0 agent on ubuntu 18 not able to connect to 4.3 master)

Actions

Copy link

#19

Updated by Alexis Mousset about 6 years ago

Has duplicate Bug #13766: 5.0 agent on ubuntu 18 not able to connect to 4.3 master added

Actions

Copy link

#20

Updated by François ARMAND about 6 years ago

Putting back relevant information from #13766:

- the bug is in OpenSSL certificat serialisation format incompatibility between openssl 1.0 and openssl 1.1.0. OpenSSL was producing not strictly exact certificate serialization which are now rejected.
- it is tracked on openssl: https://github.com/openssl/openssl/issues/7134
- it will be corrected in openssl 1.1.1: https://github.com/openssl/openssl/commit/ca89174bc92c16f0a2a7eb86359b6c6fd1dd7a4d
- other projects have the same problem, for ex: https://monitoring-portal.org/woltlab/index.php?thread/41664-ca-crt-verification-error-with-openssl-1-1-0-illegal-zero-content-in-field-seria/

For Rudder, it means that:

- Agent with openssl 1.0 can't connect to Rudder root server with openssl 1.1.0 (resp agent with openssl 1.1.0 can't connect to root server with openssl 1.0).
- openssl 1.1.0 is used in Rudder 5.0 on ubuntu 18_04, debian 9, and SLES 15
- so you can't mix these versions for root server with any other agent version (included agents on ubuntu 18_04/debian 9/SLES 15 on rudder 4.3 or older), nor you can use agent on these version with an server on any other os/rudder version.

As no distribs will be packaging openssl 1.1.1 until a long time, we can't rely on the distribution support.

If we choose to go for an homogeneous version of openssl, it can only be 1.0 (sinve we support os for agent which don't have 1.1.0 at all), but that means that for ex rudder server 5.0.1 on ubuntu 18_04 won't be able to discuss with rudder agent 5.0.with-the-correction on ubuntu 18_04. This is not possible.

So, the only path forward is to statically compile rudder with openssl 1.1.1 on ubuntu 18_04, debian 9 and SLES 15, for both agent and server.

Actions

Copy link

#21

Updated by Benoît PECCATTE about 6 years ago

Assignee changed from Vincent MEMBRÉ to Benoît PECCATTE
Priority changed from 94 to 0

Actions

Copy link

#22

Updated by Benoît PECCATTE about 6 years ago

Status changed from In progress to Pending technical review
Assignee changed from Benoît PECCATTE to Alexis Mousset
Pull Request set to https://github.com/Normation/rudder-packages/pull/1709

PR https://github.com/Normation/rudder-packages/pull/1709

Actions

Copy link

#23

Updated by Benoît PECCATTE about 6 years ago

Status changed from Pending technical review to Pending release

Applied in changeset rudder-packages|544408cbb76f0281660fbf0c32be216553bceeb7.

Actions

Copy link

#24

Updated by Vincent MEMBRÉ almost 6 years ago

Subject changed from Connection error between agents and servers using openssl 1.0.x <-> 1.1.0 to Impossible to update promises when using a debian9 or Ubuntu 18 server and older distributions as Nodes (incomaptible openssl version)

Actions

Copy link

#25

Updated by Vincent MEMBRÉ almost 6 years ago

Subject changed from Impossible to update promises when using a debian9 or Ubuntu 18 server and older distributions as Nodes (incomaptible openssl version) to Impossible to update promises when using a debian9 or Ubuntu 18 server and older distributions as Nodes (incompatible openssl version)

Actions

Copy link

#26

Updated by François ARMAND almost 6 years ago

Description updated (diff)

In comment 20 above (https://issues.rudder.io/issues/13690#note-20), we though we had a solution for everything, but it wasn't sufficient because 1.0.1 is still not compatible with 1.1.1.

So we ended up embeding OpenSSL everywhere, with:

- version 1.0.2 for very old distros (AIX 5, Centos 3, centos 5..)
- version 1.1.1 everywhere.

It still means that people with agent relying on a OpenSSL 1.0.1.
It works correctly with openssl 1.1.0.

Actions

Copy link

#27

Updated by François ARMAND almost 6 years ago

Description updated (diff)

Actions

Copy link

#28

Updated by Vincent MEMBRÉ almost 6 years ago

Status changed from Pending release to Released

This bug has been fixed in Rudder 5.0.3 which was released today.

Download: https://www.rudder-project.org/site/get-rudder/downloads/

Changelog

Actions

Copy link

#29

Updated by Félix DALLIDET over 5 years ago

Related to Bug #14570: Build openssl for Slackware, so the agent can update promises added

Actions

Copy link

#30

Updated by François ARMAND about 5 years ago

Related to Bug #16224: Missing documentation on openssl incompatibilities between 4.x and 5.0 added

Actions

Copy link

#31

Updated by over 1 year ago

Description updated (diff)
Category changed from Security to Server components
Severity changed from Critical - prevents main use of Rudder | no workaround | data loss | security to Trivial - no functional impact | cosmetic
UX impact set to It bothers me each time
User visibility changed from Getting started - demo | first install | level 1 Techniques to Getting started - demo | first install | Technique editor and level 1 Techniques
Priority changed from 0 to 57
Regression set to No

а вот http://limasd.ru наш новый сайт.

agent-debug (236 KB) agent-debug		Nicolas CHARLES, 2018-11-06 17:01
server-debug (1.77 MB) server-debug		Nicolas CHARLES, 2018-11-06 17:01

Project

General

Profile

Rudder

Custom queries

Bug #13690

Impossible to update promises when using a debian9 or Ubuntu 18 server and older distributions as Nodes (incompatible openssl version)

Updated by Alexis Mousset about 6 years ago

Updated by François ARMAND about 6 years ago

Updated by François ARMAND about 6 years ago

Updated by François ARMAND about 6 years ago

Updated by François ARMAND about 6 years ago

Updated by Vincent MEMBRÉ about 6 years ago

Updated by François ARMAND about 6 years ago

Updated by Nicolas CHARLES about 6 years ago

Updated by Nicolas CHARLES about 6 years ago

Updated by François ARMAND about 6 years ago

Updated by Nicolas CHARLES about 6 years ago

Updated by Nicolas CHARLES about 6 years ago

Updated by Nicolas CHARLES about 6 years ago

Updated by Alexis Mousset about 6 years ago

Updated by Alexis Mousset about 6 years ago

Updated by Alexis Mousset about 6 years ago

Updated by Alexis Mousset about 6 years ago

Updated by Alexis Mousset about 6 years ago

Updated by Alexis Mousset about 6 years ago

Updated by François ARMAND about 6 years ago

Updated by Benoît PECCATTE about 6 years ago

Updated by Benoît PECCATTE about 6 years ago

Updated by Benoît PECCATTE about 6 years ago

Updated by Vincent MEMBRÉ almost 6 years ago

Updated by Vincent MEMBRÉ almost 6 years ago

Updated by François ARMAND almost 6 years ago

Updated by François ARMAND almost 6 years ago

Updated by Vincent MEMBRÉ almost 6 years ago

Updated by Félix DALLIDET over 5 years ago

Updated by François ARMAND about 5 years ago

Updated by over 1 year ago

Bug #13808: rudder-agent Build error on after openssl upgrade to 1.1.1 (at least on RHEL6)	Released	Benoît PECCATTE	Actions
Bug #13811: Broken build with -fPIE	Released	Benoît PECCATTE	Actions
Bug #13817: Removing -fPIE breaks lmdb build	Released	Benoît PECCATTE	Actions
Bug #13829: Broken curl build without -fPIE	Released	Benoît PECCATTE	Actions
Bug #13831: Add -fPIE for cfengine build	Released	Benoît PECCATTE	Actions
Bug #13842: Use openssl 1.0.2 on old agents	Released	Alexis Mousset	Actions
Bug #13853: missing one makefile parameter to build openssl 1.0	Released	Alexis Mousset	Actions
Bug #13864: open ssl build variable name should be different between 1.0.2 and 1.1.1	Released	Alexis Mousset	Actions