Project

General

Profile

Bug #13690

Impossible to update promises when using a debian9 or Ubuntu 18 server and older distributions as Nodes (incompatible openssl version)

Added by Thomas CAILHE about 1 year ago. Updated about 1 year ago.

Status:
Released
Priority:
N/A
Category:
Security
Target version:
Severity:
Critical - prevents main use of Rudder | no workaround | data loss | security
User visibility:
Getting started - demo | first install | level 1 Techniques
Effort required:
Priority:
0

Description

Hi,

I've got the same error on 2 fresh servers with centos6
error: Failed to establish TLS connection: (0 SSL_ERROR_SSL) tlsv1 alert internal error error: No suitable server found error: Failed to establish TLS connection: (0 SSL_ERROR_SSL) tlsv1 alert internal error error: No suitable server found
error: Rudder agent promises could not be updated. Start execution with config [0]

  • server: OpenSSL 1.1.0f 25 May 2017 debian 9
  • client : OpenSSL 1.0.1e-fips 11 Feb 2013 (well...) centos 6

UPDATE/RESOLUTION:
In comment 20 belove (https://issues.rudder.io/issues/13690#note-20), we though we had a solution for everything, but it wasn't sufficient because 1.0.1 is still not compatible with 1.1.1.

So we ended up embeding OpenSSL everywhere, with:

- version 1.0.2 for very old distros (AIX 5, Centos 3, centos 5..)
- version 1.1.1 everywhere.

It still means that new server (in libssl 1.1.1) won't work for people with agent relying on a OpenSSL 1.0.1 (centos 6 for rudder 5.0.2 and system ssl for ex).
At least new servers work correctly with agent in openssl 1.1.0 (so for ex agent 4.3 on ubuntu 18.04 works with server on 5.0.3).


Files

agent-debug (236 KB) agent-debug Nicolas CHARLES, 2018-11-06 17:01
server-debug (1.77 MB) server-debug Nicolas CHARLES, 2018-11-06 17:01

Subtasks

Bug #13808: rudder-agent Build error on after openssl upgrade to 1.1.1 (at least on RHEL6)ReleasedBenoît PECCATTEActions
Bug #13811: Broken build with -fPIEReleasedBenoît PECCATTEActions
Bug #13817: Removing -fPIE breaks lmdb buildReleasedBenoît PECCATTEActions
Bug #13829: Broken curl build without -fPIEReleasedBenoît PECCATTEActions
Bug #13831: Add -fPIE for cfengine buildReleasedBenoît PECCATTEActions
Bug #13842: Use openssl 1.0.2 on old agentsReleasedAlexis MOUSSETActions
Bug #13853: missing one makefile parameter to build openssl 1.0ReleasedAlexis MOUSSETActions
Bug #13864: open ssl build variable name should be different between 1.0.2 and 1.1.1ReleasedAlexis MOUSSETActions

Related issues

Related to Rudder - Bug #14570: Build openssl for Slackware, so the agent can update promisesReleasedActions
Related to Rudder - Bug #16224: Missing documentation on openssl incompatibilities between 4.x and 5.0ReleasedActions
Has duplicate Rudder - Bug #13766: 5.0 agent on ubuntu 18 not able to connect to 4.3 masterRejectedActions

Associated revisions

Revision 544408cb (diff)
Added by Benoît PECCATTE about 1 year ago

Fixes #13690: Connection error between agents and servers using openssl 1.0.x <-> 1.1.0

History

#1

Updated by Alexis MOUSSET about 1 year ago

  • Subject changed from Openssl version is to old on centos 6 to Openssl version is too old on CentOS 6
#2

Updated by François ARMAND about 1 year ago

  • Severity set to Critical - prevents main use of Rudder | no workaround | data loss | security
  • User visibility set to Operational - other Techniques | Technique editor | Rudder settings
  • Priority changed from 0 to 76

If confirmed, this one is critical, because it forbids the use of Rudder on centos 6 which is still widelly used.

We got more information by gitter: https://gitter.im/normation/rudder?at=5bc9ad3a435c2a518ecf1193

So, we need to reproduce ASAP:

- server debian 9: OpenSSL 1.1.0f
- client centos 6 : OpenSSL openssl-1.0.1e-57.el6.x86_64

And depending of the result, we may need to embed openssl for centos 6.

#3

Updated by François ARMAND about 1 year ago

  • Category set to Security
#4

Updated by François ARMAND about 1 year ago

  • User visibility changed from Operational - other Techniques | Technique editor | Rudder settings to Getting started - demo | first install | level 1 Techniques
  • Priority changed from 76 to 94
#5

Updated by François ARMAND about 1 year ago

  • Assignee set to Benoît PECCATTE
  • Target version set to 5.0.2
#6

Updated by Vincent MEMBRÉ about 1 year ago

  • Target version changed from 5.0.2 to 5.0.3
#7

Updated by François ARMAND about 1 year ago

  • Status changed from New to In progress
  • Assignee changed from Benoît PECCATTE to Vincent MEMBRÉ

We were able to reproduce. There is something strange in the debian 9 (and perhaps ubuntu 18.04) cfengine binary. It seems to be linked to both OpenSSL 1.0 and 1.1. But that does not explains why exactly "debian x to debian 9" works but not "centos 6 to debian 9" does not.

We are working on the analysis of pairs that doesn't not work.
It may be the same root cause as #13766 where the server is ubuntu 18.04 / openssl 1.1, and the agent are in ubuntu 18.04 / openssl 1.0.

#8

Updated by Nicolas CHARLES about 1 year ago

A server Rudder 5.0 on Debian9 with an Agent Centos 6 (5.0 ot 4.3) fails
On the Server side, the logs say:

rudder  verbose: Setting minimum acceptable TLS version: 1.0
rudder  verbose: Setting cipher list for incoming TLS connections to: AES256-GCM-SHA384:AES256-SHA
rudder  verbose: Listening for connections on socket descriptor 6 ...
  notice: Server is starting...
rudder  verbose: Obtained IP address of '192.168.41.5' on socket 7 from accept
rudder  verbose: New connection (from 192.168.41.5, sd 7), spawning new thread...
rudder     info: 192.168.41.5>    Accepting connection
rudder  verbose: 192.168.41.5>    Setting socket timeout to 600 seconds.
rudder  verbose: 192.168.41.5>    Peeked nothing important in TCP stream, considering the protocol as TLS
   error: 192.168.41.5>    Failed to accept TLS connection: (-1 SSL_ERROR_SSL) illegal zero content 
rudder  verbose: Obtained IP address of '192.168.41.5' on socket 7 from accept

on the agent side

   error: Failed to establish TLS connection: (0 SSL_ERROR_SSL) tlsv1 alert internal error 
   error: No suitable server found
   error: Failed to establish TLS connection: (0 SSL_ERROR_SSL) tlsv1 alert internal error 
   error: No suitable server found

#9

Updated by Nicolas CHARLES about 1 year ago

  • Related to Bug #13766: 5.0 agent on ubuntu 18 not able to connect to 4.3 master added
#10

Updated by François ARMAND about 1 year ago

Some more pair tested: on a Rudder 5.0, ubuntu 18.04:

- centos 7.5, debian 8.9, debian 9.5, ubuntu 18.04: OK
- centos 6.9: not ok.

#11

Updated by Nicolas CHARLES about 1 year ago

debug logs of the agent & server

#12

Updated by Nicolas CHARLES about 1 year ago

Ldd results

#13

Updated by Nicolas CHARLES about 1 year ago

I tried to set tls_ciphers => "AES128-SHA"; as a workaround, without any success

#14

Updated by Alexis MOUSSET about 1 year ago

  • Subject changed from Openssl version is too old on CentOS 6 to Connection error between agents and servers using openssl 1.0.x <-> 1.1.0
#15

Updated by Alexis MOUSSET about 1 year ago

  • Description updated (diff)
#16

Updated by Alexis MOUSSET about 1 year ago

  • Related to deleted (Bug #13766: 5.0 agent on ubuntu 18 not able to connect to 4.3 master)
#17

Updated by Alexis MOUSSET about 1 year ago

  • Related to Bug #13766: 5.0 agent on ubuntu 18 not able to connect to 4.3 master added
#18

Updated by Alexis MOUSSET about 1 year ago

  • Related to deleted (Bug #13766: 5.0 agent on ubuntu 18 not able to connect to 4.3 master)
#19

Updated by Alexis MOUSSET about 1 year ago

  • Has duplicate Bug #13766: 5.0 agent on ubuntu 18 not able to connect to 4.3 master added
#20

Updated by François ARMAND about 1 year ago

Putting back relevant information from #13766:

- the bug is in OpenSSL certificat serialisation format incompatibility between openssl 1.0 and openssl 1.1.0. OpenSSL was producing not strictly exact certificate serialization which are now rejected.
- it is tracked on openssl: https://github.com/openssl/openssl/issues/7134
- it will be corrected in openssl 1.1.1: https://github.com/openssl/openssl/commit/ca89174bc92c16f0a2a7eb86359b6c6fd1dd7a4d
- other projects have the same problem, for ex: https://monitoring-portal.org/woltlab/index.php?thread/41664-ca-crt-verification-error-with-openssl-1-1-0-illegal-zero-content-in-field-seria/

For Rudder, it means that:

- Agent with openssl 1.0 can't connect to Rudder root server with openssl 1.1.0 (resp agent with openssl 1.1.0 can't connect to root server with openssl 1.0).
- openssl 1.1.0 is used in Rudder 5.0 on ubuntu 18_04, debian 9, and SLES 15
- so you can't mix these versions for root server with any other agent version (included agents on ubuntu 18_04/debian 9/SLES 15 on rudder 4.3 or older), nor you can use agent on these version with an server on any other os/rudder version.

As no distribs will be packaging openssl 1.1.1 until a long time, we can't rely on the distribution support.

If we choose to go for an homogeneous version of openssl, it can only be 1.0 (sinve we support os for agent which don't have 1.1.0 at all), but that means that for ex rudder server 5.0.1 on ubuntu 18_04 won't be able to discuss with rudder agent 5.0.with-the-correction on ubuntu 18_04. This is not possible.

So, the only path forward is to statically compile rudder with openssl 1.1.1 on ubuntu 18_04, debian 9 and SLES 15, for both agent and server.

#21

Updated by Benoît PECCATTE about 1 year ago

  • Assignee changed from Vincent MEMBRÉ to Benoît PECCATTE
  • Priority changed from 94 to 0
#22

Updated by Benoît PECCATTE about 1 year ago

  • Status changed from In progress to Pending technical review
  • Assignee changed from Benoît PECCATTE to Alexis MOUSSET
  • Pull Request set to https://github.com/Normation/rudder-packages/pull/1709
#23

Updated by Benoît PECCATTE about 1 year ago

  • Status changed from Pending technical review to Pending release
#24

Updated by Vincent MEMBRÉ about 1 year ago

  • Subject changed from Connection error between agents and servers using openssl 1.0.x <-> 1.1.0 to Impossible to update promises when using a debian9 or Ubuntu 18 server and older distributions as Nodes (incomaptible openssl version)
#25

Updated by Vincent MEMBRÉ about 1 year ago

  • Subject changed from Impossible to update promises when using a debian9 or Ubuntu 18 server and older distributions as Nodes (incomaptible openssl version) to Impossible to update promises when using a debian9 or Ubuntu 18 server and older distributions as Nodes (incompatible openssl version)
#26

Updated by François ARMAND about 1 year ago

  • Description updated (diff)

In comment 20 above (https://issues.rudder.io/issues/13690#note-20), we though we had a solution for everything, but it wasn't sufficient because 1.0.1 is still not compatible with 1.1.1.

So we ended up embeding OpenSSL everywhere, with:

- version 1.0.2 for very old distros (AIX 5, Centos 3, centos 5..)
- version 1.1.1 everywhere.

It still means that people with agent relying on a OpenSSL 1.0.1.
It works correctly with openssl 1.1.0.

#27

Updated by François ARMAND about 1 year ago

  • Description updated (diff)
#28

Updated by Vincent MEMBRÉ about 1 year ago

  • Status changed from Pending release to Released
This bug has been fixed in Rudder 5.0.3 which was released today.
Changelog
#29

Updated by Félix DALLIDET 8 months ago

  • Related to Bug #14570: Build openssl for Slackware, so the agent can update promises added
#30

Updated by François ARMAND 16 days ago

  • Related to Bug #16224: Missing documentation on openssl incompatibilities between 4.x and 5.0 added

Also available in: Atom PDF