Project

General

Profile

Bug #13690

Impossible to update promises when using a debian9 or Ubuntu 18 server and older distributions as Nodes (incompatible openssl version)

Added by Thomas CAILHE about 2 months ago. Updated 12 days ago.

Status:
Released
Priority:
N/A
Category:
Security
Target version:
Severity:
Critical - prevents main use of Rudder | no workaround | data loss | security
User visibility:
Getting started - demo | first install | level 1 Techniques
Effort required:
Priority:
0

Description

Hi,

I've got the same error on 2 fresh servers with centos6
error: Failed to establish TLS connection: (0 SSL_ERROR_SSL) tlsv1 alert internal error error: No suitable server found error: Failed to establish TLS connection: (0 SSL_ERROR_SSL) tlsv1 alert internal error error: No suitable server found
error: Rudder agent promises could not be updated. Start execution with config [0]

  • server: OpenSSL 1.1.0f 25 May 2017 debian 9
  • client : OpenSSL 1.0.1e-fips 11 Feb 2013 (well...) centos 6

UPDATE/RESOLUTION:
In comment 20 belove (https://issues.rudder.io/issues/13690#note-20), we though we had a solution for everything, but it wasn't sufficient because 1.0.1 is still not compatible with 1.1.1.

So we ended up embeding OpenSSL everywhere, with:

- version 1.0.2 for very old distros (AIX 5, Centos 3, centos 5..)
- version 1.1.1 everywhere.

It still means that new server (in libssl 1.1.1) won't work for people with agent relying on a OpenSSL 1.0.1 (centos 6 for rudder 5.0.2 and system ssl for ex).
At least new servers work correctly with agent in openssl 1.1.0 (so for ex agent 4.3 on ubuntu 18.04 works with server on 5.0.3).

agent-debug (236 KB) agent-debug Nicolas CHARLES, 2018-11-06 17:01
server-debug (1.77 MB) server-debug Nicolas CHARLES, 2018-11-06 17:01

Subtasks

Bug #13808: rudder-agent Build error on after openssl upgrade to 1.1.1 (at least on RHEL6)ReleasedBenoît PECCATTE
Bug #13811: Broken build with -fPIEReleasedBenoît PECCATTE
Bug #13817: Removing -fPIE breaks lmdb buildReleasedBenoît PECCATTE
Bug #13829: Broken curl build without -fPIEReleasedBenoît PECCATTE
Bug #13831: Add -fPIE for cfengine buildReleasedBenoît PECCATTE
Bug #13842: Use openssl 1.0.2 on old agentsReleasedAlexis MOUSSET
Bug #13853: missing one makefile parameter to build openssl 1.0ReleasedAlexis MOUSSET
Bug #13864: open ssl build variable name should be different between 1.0.2 and 1.1.1ReleasedAlexis MOUSSET

Related issues

Has duplicate Rudder - Bug #13766: 5.0 agent on ubuntu 18 not able to connect to 4.3 masterRejected

Associated revisions

Revision 544408cb (diff)
Added by Benoît PECCATTE about 1 month ago

Fixes #13690: Connection error between agents and servers using openssl 1.0.x <-> 1.1.0

History

#1 Updated by Alexis MOUSSET about 2 months ago

  • Subject changed from Openssl version is to old on centos 6 to Openssl version is too old on CentOS 6

#2 Updated by François ARMAND about 2 months ago

  • Severity set to Critical - prevents main use of Rudder | no workaround | data loss | security
  • User visibility set to Operational - other Techniques | Technique editor | Rudder settings
  • Priority changed from 0 to 76

If confirmed, this one is critical, because it forbids the use of Rudder on centos 6 which is still widelly used.

We got more information by gitter: https://gitter.im/normation/rudder?at=5bc9ad3a435c2a518ecf1193

So, we need to reproduce ASAP:

- server debian 9: OpenSSL 1.1.0f
- client centos 6 : OpenSSL openssl-1.0.1e-57.el6.x86_64

And depending of the result, we may need to embed openssl for centos 6.

#3 Updated by François ARMAND about 2 months ago

  • Category set to Security

#4 Updated by François ARMAND about 2 months ago

  • User visibility changed from Operational - other Techniques | Technique editor | Rudder settings to Getting started - demo | first install | level 1 Techniques
  • Priority changed from 76 to 94

#5 Updated by François ARMAND about 2 months ago

  • Assignee set to Benoît PECCATTE
  • Target version set to 5.0.2

#6 Updated by Vincent MEMBRÉ about 1 month ago

  • Target version changed from 5.0.2 to 5.0.3

#7 Updated by François ARMAND about 1 month ago

  • Status changed from New to In progress
  • Assignee changed from Benoît PECCATTE to Vincent MEMBRÉ

We were able to reproduce. There is something strange in the debian 9 (and perhaps ubuntu 18.04) cfengine binary. It seems to be linked to both OpenSSL 1.0 and 1.1. But that does not explains why exactly "debian x to debian 9" works but not "centos 6 to debian 9" does not.

We are working on the analysis of pairs that doesn't not work.
It may be the same root cause as #13766 where the server is ubuntu 18.04 / openssl 1.1, and the agent are in ubuntu 18.04 / openssl 1.0.

#8 Updated by Nicolas CHARLES about 1 month ago

A server Rudder 5.0 on Debian9 with an Agent Centos 6 (5.0 ot 4.3) fails
On the Server side, the logs say:

rudder  verbose: Setting minimum acceptable TLS version: 1.0
rudder  verbose: Setting cipher list for incoming TLS connections to: AES256-GCM-SHA384:AES256-SHA
rudder  verbose: Listening for connections on socket descriptor 6 ...
  notice: Server is starting...
rudder  verbose: Obtained IP address of '192.168.41.5' on socket 7 from accept
rudder  verbose: New connection (from 192.168.41.5, sd 7), spawning new thread...
rudder     info: 192.168.41.5>    Accepting connection
rudder  verbose: 192.168.41.5>    Setting socket timeout to 600 seconds.
rudder  verbose: 192.168.41.5>    Peeked nothing important in TCP stream, considering the protocol as TLS
   error: 192.168.41.5>    Failed to accept TLS connection: (-1 SSL_ERROR_SSL) illegal zero content 
rudder  verbose: Obtained IP address of '192.168.41.5' on socket 7 from accept

on the agent side

   error: Failed to establish TLS connection: (0 SSL_ERROR_SSL) tlsv1 alert internal error 
   error: No suitable server found
   error: Failed to establish TLS connection: (0 SSL_ERROR_SSL) tlsv1 alert internal error 
   error: No suitable server found

#9 Updated by Nicolas CHARLES about 1 month ago

  • Related to Bug #13766: 5.0 agent on ubuntu 18 not able to connect to 4.3 master added

#10 Updated by François ARMAND about 1 month ago

Some more pair tested: on a Rudder 5.0, ubuntu 18.04:

- centos 7.5, debian 8.9, debian 9.5, ubuntu 18.04: OK
- centos 6.9: not ok.

#11 Updated by Nicolas CHARLES about 1 month ago

debug logs of the agent & server

#12 Updated by Nicolas CHARLES about 1 month ago

Ldd results

#13 Updated by Nicolas CHARLES about 1 month ago

I tried to set tls_ciphers => "AES128-SHA"; as a workaround, without any success

#14 Updated by Alexis MOUSSET about 1 month ago

  • Subject changed from Openssl version is too old on CentOS 6 to Connection error between agents and servers using openssl 1.0.x <-> 1.1.0

#15 Updated by Alexis MOUSSET about 1 month ago

  • Description updated (diff)

#16 Updated by Alexis MOUSSET about 1 month ago

  • Related to deleted (Bug #13766: 5.0 agent on ubuntu 18 not able to connect to 4.3 master)

#17 Updated by Alexis MOUSSET about 1 month ago

  • Related to Bug #13766: 5.0 agent on ubuntu 18 not able to connect to 4.3 master added

#18 Updated by Alexis MOUSSET about 1 month ago

  • Related to deleted (Bug #13766: 5.0 agent on ubuntu 18 not able to connect to 4.3 master)

#19 Updated by Alexis MOUSSET about 1 month ago

  • Has duplicate Bug #13766: 5.0 agent on ubuntu 18 not able to connect to 4.3 master added

#20 Updated by François ARMAND about 1 month ago

Putting back relevant information from #13766:

- the bug is in OpenSSL certificat serialisation format incompatibility between openssl 1.0 and openssl 1.1.0. OpenSSL was producing not strictly exact certificate serialization which are now rejected.
- it is tracked on openssl: https://github.com/openssl/openssl/issues/7134
- it will be corrected in openssl 1.1.1: https://github.com/openssl/openssl/commit/ca89174bc92c16f0a2a7eb86359b6c6fd1dd7a4d
- other projects have the same problem, for ex: https://monitoring-portal.org/woltlab/index.php?thread/41664-ca-crt-verification-error-with-openssl-1-1-0-illegal-zero-content-in-field-seria/

For Rudder, it means that:

- Agent with openssl 1.0 can't connect to Rudder root server with openssl 1.1.0 (resp agent with openssl 1.1.0 can't connect to root server with openssl 1.0).
- openssl 1.1.0 is used in Rudder 5.0 on ubuntu 18_04, debian 9, and SLES 15
- so you can't mix these versions for root server with any other agent version (included agents on ubuntu 18_04/debian 9/SLES 15 on rudder 4.3 or older), nor you can use agent on these version with an server on any other os/rudder version.

As no distribs will be packaging openssl 1.1.1 until a long time, we can't rely on the distribution support.

If we choose to go for an homogeneous version of openssl, it can only be 1.0 (sinve we support os for agent which don't have 1.1.0 at all), but that means that for ex rudder server 5.0.1 on ubuntu 18_04 won't be able to discuss with rudder agent 5.0.with-the-correction on ubuntu 18_04. This is not possible.

So, the only path forward is to statically compile rudder with openssl 1.1.1 on ubuntu 18_04, debian 9 and SLES 15, for both agent and server.

#21 Updated by Benoît PECCATTE about 1 month ago

  • Assignee changed from Vincent MEMBRÉ to Benoît PECCATTE
  • Priority changed from 94 to 0

#22 Updated by Benoît PECCATTE about 1 month ago

  • Status changed from In progress to Pending technical review
  • Assignee changed from Benoît PECCATTE to Alexis MOUSSET
  • Pull Request set to https://github.com/Normation/rudder-packages/pull/1709

#23 Updated by Benoît PECCATTE about 1 month ago

  • Status changed from Pending technical review to Pending release

#24 Updated by Vincent MEMBRÉ 13 days ago

  • Subject changed from Connection error between agents and servers using openssl 1.0.x <-> 1.1.0 to Impossible to update promises when using a debian9 or Ubuntu 18 server and older distributions as Nodes (incomaptible openssl version)

#25 Updated by Vincent MEMBRÉ 13 days ago

  • Subject changed from Impossible to update promises when using a debian9 or Ubuntu 18 server and older distributions as Nodes (incomaptible openssl version) to Impossible to update promises when using a debian9 or Ubuntu 18 server and older distributions as Nodes (incompatible openssl version)

#26 Updated by François ARMAND 13 days ago

  • Description updated (diff)

In comment 20 above (https://issues.rudder.io/issues/13690#note-20), we though we had a solution for everything, but it wasn't sufficient because 1.0.1 is still not compatible with 1.1.1.

So we ended up embeding OpenSSL everywhere, with:

- version 1.0.2 for very old distros (AIX 5, Centos 3, centos 5..)
- version 1.1.1 everywhere.

It still means that people with agent relying on a OpenSSL 1.0.1.
It works correctly with openssl 1.1.0.

#27 Updated by François ARMAND 13 days ago

  • Description updated (diff)

#28 Updated by Vincent MEMBRÉ 12 days ago

  • Status changed from Pending release to Released
This bug has been fixed in Rudder 5.0.3 which was released today.
Changelog

Also available in: Atom PDF