Rudder

--- firefox bug and workaround ---

There is ticket opened on mozilla bug tracker: https://bugzilla.mozilla.org/show_bug.cgi?id=1240548

Edit: there was an other bug open in firefox bugtracked, without any chance to find it given the title: https://bugzilla.mozilla.org/show_bug.cgi?id=1056341

The general workaround (because that problem can happen in other case than Rudder one, so if any body land here:) is to:

• solution 1: create a new profile (firefox -F) and use it
• solution 2: remove all existing occurence of the certificate in firefox internal base: preference -> advanced choice -> certificates -> view certificates -> Servers tab and likelly they are in the (unknonw) entry
• solution 3: solution 2 may not be sufficient if the cert base is somehow broken (see comment #13), so: rm ~/.mozilla/firefox/xxxx.default/cert8.db (or cert9.db)
• solution 4: sometimes, even if you remove certX.db, ff will stall. You will have to do:

cd  ~/.mozilla/firefox/xxxx.default/
certutil -d sql:. -L | grep hostname-of-problematic-server
[you should see several entries for it]
certutil -d sql:. -D -n hostname-of-problematic-server

Description of the problem

At the end of the installation process, Rudder generates a self signed certificate and install it into Apache.
Firefox is enable to connect to Rudder in https with that certificate. Of course, it does not happen with all Firefox, but we are several to have experienced the problem with version of Firefox from at 39.x to the latest 43.0.4.
The behaviour is that Firefox status bar display "connected to 192.168.42.2...." and nothing else happen, even if we wait several minutes (i.e: we don't even get the security notification page that the site is not secured, do you want to add an exception are you really sur).

Of course, using Chrome/chromium of even Opera works without any problem.

On the network level, we can see that the TLS handshacks is ok, but then a reset happen.
On apache side, the logs say:

[Mon Jan 18 16:30:02 2016] [info] [client 192.168.42.1] Connection to child 10 established (server server.rudder.local:443)
[Mon Jan 18 16:30:02 2016] [info] Seeding PRNG with 656 bytes of entropy
[Mon Jan 18 16:30:02 2016] [debug] ssl_engine_kernel.c(1818): OpenSSL: Handshake: start
[Mon Jan 18 16:30:02 2016] [debug] ssl_engine_kernel.c(1826): OpenSSL: Loop: before/accept initialization
[Mon Jan 18 16:30:02 2016] [debug] ssl_engine_io.c(1897): OpenSSL: read 11/11 bytes from BIO#7fa515eeae20 [mem: 7fa515f964a0] (BIO dump follows)
[Mon Jan 18 16:30:02 2016] [debug] ssl_engine_io.c(1830): +-------------------------------------------------------------------------+
[Mon Jan 18 16:30:02 2016] [debug] ssl_engine_io.c(1869): | 0000: 16 03 01 00 c0 01 00 00-bc 03 03                 ...........      |
[Mon Jan 18 16:30:02 2016] [debug] ssl_engine_io.c(1875): +-------------------------------------------------------------------------+
[Mon Jan 18 16:30:02 2016] [debug] ssl_engine_io.c(1897): OpenSSL: read 186/186 bytes from BIO#7fa515eeae20 [mem: 7fa515f964ae] (BIO dump follows)
[Mon Jan 18 16:30:02 2016] [debug] ssl_engine_io.c(1830): +-------------------------------------------------------------------------+
[Mon Jan 18 16:30:02 2016] [debug] ssl_engine_io.c(1869): | 0000: 20 d6 04 93 11 ad 47 1f-a3 7d a6 be 26 7a 33 38   .....G..}..&z38 |
[Mon Jan 18 16:30:02 2016] [debug] ssl_engine_io.c(1869): | 0010: f3 0a 9c a9 15 ae 91 4c-47 10 a7 4f 3d 90 f2 50  .......LG..O=..P |
[Mon Jan 18 16:30:02 2016] [debug] ssl_engine_io.c(1869): | 0020: 20 48 c8 ee c8 23 68 a3-0b 85 04 11 61 ea 42 8b   H...#h.....a.B. |
[Mon Jan 18 16:30:02 2016] [debug] ssl_engine_io.c(1869): | 0030: 07 74 e6 9d 9f cc 2f b6-fc d8 2d 8e b1 a1 d2 ff  .t..../...-..... |
[Mon Jan 18 16:30:02 2016] [debug] ssl_engine_io.c(1869): | 0040: dc 00 16 c0 2b c0 2f c0-0a c0 09 c0 13 c0 14 00  ....+./......... |
[Mon Jan 18 16:30:02 2016] [debug] ssl_engine_io.c(1869): | 0050: 33 00 39 00 2f 00 35 00-0a 01 00 00 5d ff 01 00  3.9./.5.....]... |
[Mon Jan 18 16:30:02 2016] [debug] ssl_engine_io.c(1869): | 0060: 01 00 00 0a 00 08 00 06-00 17 00 18 00 19 00 0b  ................ |
[Mon Jan 18 16:30:02 2016] [debug] ssl_engine_io.c(1869): | 0070: 00 02 01 00 00 23 00 00-33 74 00 00 00 10 00 17  .....#..3t...... |
[Mon Jan 18 16:30:02 2016] [debug] ssl_engine_io.c(1869): | 0080: 00 15 02 68 32 08 73 70-64 79 2f 33 2e 31 08 68  ...h2.spdy/3.1.h |
[Mon Jan 18 16:30:02 2016] [debug] ssl_engine_io.c(1869): | 0090: 74 74 70 2f 31 2e 31 00-05 00 05 01 00 00 00 00  ttp/1.1......... |
[Mon Jan 18 16:30:02 2016] [debug] ssl_engine_io.c(1869): | 00a0: 00 0d 00 16 00 14 04 01-05 01 06 01 02 01 04 03  ................ |
[Mon Jan 18 16:30:02 2016] [debug] ssl_engine_io.c(1869): | 00b0: 05 03 06 03 02 03 04 02-02 02                    ..........       |
[Mon Jan 18 16:30:02 2016] [debug] ssl_engine_io.c(1875): +-------------------------------------------------------------------------+
[Mon Jan 18 16:30:02 2016] [debug] ssl_scache_shmcb.c(393): ssl_scache_shmcb_retrieve (0x48 -> subcache 8)
[Mon Jan 18 16:30:02 2016] [debug] ssl_scache_shmcb.c(708): shmcb_subcache_retrieve found no match
[Mon Jan 18 16:30:02 2016] [debug] ssl_scache_shmcb.c(408): leaving ssl_scache_shmcb_retrieve successfully
[Mon Jan 18 16:30:02 2016] [debug] ssl_engine_kernel.c(1684): Inter-Process Session Cache: request=GET status=MISSED id=48C8EEC82368A30B85041161EA428B0774E69D9FCC2FB6FCD82D8EB1A1D2FFDC (session
renewal)
[Mon Jan 18 16:30:02 2016] [debug] ssl_engine_kernel.c(1826): OpenSSL: Loop: SSLv3 read client hello A
[Mon Jan 18 16:30:02 2016] [debug] ssl_engine_kernel.c(1826): OpenSSL: Loop: SSLv3 write server hello A
[Mon Jan 18 16:30:02 2016] [debug] ssl_engine_kernel.c(1826): OpenSSL: Loop: SSLv3 write certificate A
[Mon Jan 18 16:30:02 2016] [debug] ssl_engine_kernel.c(1826): OpenSSL: Loop: SSLv3 write key exchange A
[Mon Jan 18 16:30:02 2016] [debug] ssl_engine_kernel.c(1826): OpenSSL: Loop: SSLv3 write server done A
[Mon Jan 18 16:30:02 2016] [debug] ssl_engine_kernel.c(1826): OpenSSL: Loop: SSLv3 flush data
[Mon Jan 18 16:30:02 2016] [debug] ssl_engine_io.c(1897): OpenSSL: read 5/5 bytes from BIO#7fa515eeae20 [mem: 7fa515f964a3] (BIO dump follows)
[Mon Jan 18 16:30:02 2016] [debug] ssl_engine_io.c(1830): +-------------------------------------------------------------------------+
[Mon Jan 18 16:30:02 2016] [debug] ssl_engine_io.c(1869): | 0000: 16 03 03 00 46                                   ....F            |
[Mon Jan 18 16:30:02 2016] [debug] ssl_engine_io.c(1875): +-------------------------------------------------------------------------+
[Mon Jan 18 16:30:02 2016] [debug] ssl_engine_io.c(1897): OpenSSL: read 70/70 bytes from BIO#7fa515eeae20 [mem: 7fa515f964a8] (BIO dump follows)
[Mon Jan 18 16:30:02 2016] [debug] ssl_engine_io.c(1830): +-------------------------------------------------------------------------+
[Mon Jan 18 16:30:02 2016] [debug] ssl_engine_io.c(1869): | 0000: 10 00 00 42 41 04 c6 23-da 59 a5 c0 ef fa 46 74  ...BA..#.Y....Ft |
[Mon Jan 18 16:30:02 2016] [debug] ssl_engine_io.c(1869): | 0010: 25 b3 95 49 bf ef c7 58-8d 25 98 e9 5e db e1 1a  %..I...X.%..^... |
[Mon Jan 18 16:30:02 2016] [debug] ssl_engine_io.c(1869): | 0020: 73 0b 22 74 ba 55 e0 9c-d9 e3 43 76 a7 13 99 63  s."t.U....Cv...c |
[Mon Jan 18 16:30:02 2016] [debug] ssl_engine_io.c(1869): | 0030: d7 09 d3 7c aa 9c 7b 8b-9e 72 0b 23 60 03 41 28  ...|..{..r.#`.A( |
[Mon Jan 18 16:30:02 2016] [debug] ssl_engine_io.c(1869): | 0040: 2d 6d 44 1b 0d 7a                                -mD..z           |
[Mon Jan 18 16:30:02 2016] [debug] ssl_engine_io.c(1875): +-------------------------------------------------------------------------+
[Mon Jan 18 16:30:02 2016] [debug] ssl_engine_kernel.c(1826): OpenSSL: Loop: SSLv3 read client key exchange A
[Mon Jan 18 16:30:02 2016] [debug] ssl_engine_io.c(1897): OpenSSL: read 5/5 bytes from BIO#7fa515eeae20 [mem: 7fa515f964a3] (BIO dump follows)
[Mon Jan 18 16:30:02 2016] [debug] ssl_engine_io.c(1830): +-------------------------------------------------------------------------+
[Mon Jan 18 16:30:02 2016] [debug] ssl_engine_io.c(1869): | 0000: 14 03 03 00 01                                   .....            |
[Mon Jan 18 16:30:02 2016] [debug] ssl_engine_io.c(1875): +-------------------------------------------------------------------------+
[Mon Jan 18 16:30:02 2016] [debug] ssl_engine_io.c(1897): OpenSSL: read 1/1 bytes from BIO#7fa515eeae20 [mem: 7fa515f964a8] (BIO dump follows)
[Mon Jan 18 16:30:02 2016] [debug] ssl_engine_io.c(1830): +-------------------------------------------------------------------------+
[Mon Jan 18 16:30:02 2016] [debug] ssl_engine_io.c(1869): | 0000: 01                                               .                |
[Mon Jan 18 16:30:02 2016] [debug] ssl_engine_io.c(1875): +-------------------------------------------------------------------------+
[Mon Jan 18 16:30:02 2016] [debug] ssl_engine_io.c(1897): OpenSSL: read 5/5 bytes from BIO#7fa515eeae20 [mem: 7fa515f964a3] (BIO dump follows)
[Mon Jan 18 16:30:02 2016] [debug] ssl_engine_io.c(1830): +-------------------------------------------------------------------------+
[Mon Jan 18 16:30:02 2016] [debug] ssl_engine_io.c(1869): | 0000: 16 03 03 00 28                                   ....(            |
[Mon Jan 18 16:30:02 2016] [debug] ssl_engine_io.c(1875): +-------------------------------------------------------------------------+
[Mon Jan 18 16:30:02 2016] [debug] ssl_engine_io.c(1897): OpenSSL: read 40/40 bytes from BIO#7fa515eeae20 [mem: 7fa515f964a8] (BIO dump follows)
[Mon Jan 18 16:30:02 2016] [debug] ssl_engine_io.c(1830): +-------------------------------------------------------------------------+
[Mon Jan 18 16:30:02 2016] [debug] ssl_engine_io.c(1869): | 0000: 00 00 00 00 00 00 00 00-89 ff fb f5 ab 7d ec 89  .............}.. |
[Mon Jan 18 16:30:02 2016] [debug] ssl_engine_io.c(1869): | 0010: 7f d9 b9 f3 cc 6e ea f3-17 88 2b 37 90 92 44 1d  .....n....+7..D. |
[Mon Jan 18 16:30:02 2016] [debug] ssl_engine_io.c(1869): | 0020: 16 1f 83 62 8f e7 ed f6-                         ...b....         |
[Mon Jan 18 16:30:02 2016] [debug] ssl_engine_io.c(1875): +-------------------------------------------------------------------------+
[Mon Jan 18 16:30:02 2016] [debug] ssl_engine_kernel.c(1826): OpenSSL: Loop: SSLv3 read finished A
[Mon Jan 18 16:30:02 2016] [debug] ssl_engine_kernel.c(1826): OpenSSL: Loop: SSLv3 write session ticket A
[Mon Jan 18 16:30:02 2016] [debug] ssl_engine_kernel.c(1826): OpenSSL: Loop: SSLv3 write change cipher spec A
[Mon Jan 18 16:30:02 2016] [debug] ssl_engine_kernel.c(1826): OpenSSL: Loop: SSLv3 write finished A
[Mon Jan 18 16:30:02 2016] [debug] ssl_engine_kernel.c(1826): OpenSSL: Loop: SSLv3 flush data
[Mon Jan 18 16:30:02 2016] [debug] ssl_engine_kernel.c(1822): OpenSSL: Handshake: done
[Mon Jan 18 16:30:02 2016] [info] Connection: Client IP: 192.168.42.1, Protocol: TLSv1.2, Cipher: ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)

[.... wait wait wait ....]

[Mon Jan 18 16:30:23 2016] [info] [client 192.168.42.1] Request header read timeout
[Mon Jan 18 16:30:23 2016] [debug] ssl_engine_io.c(1908): OpenSSL: I/O error, 5 bytes expected to read on BIO#7fa515eeae20 [mem: 7fa515f964a3]
[Mon Jan 18 16:30:23 2016] [info] [client 192.168.42.1] (70007)The timeout specified has expired: SSL input filter read failed.
[Mon Jan 18 16:30:23 2016] [debug] ssl_engine_kernel.c(1836): OpenSSL: Write: SSL negotiation finished successfully
[Mon Jan 18 16:30:23 2016] [info] [client 192.168.42.1] Connection closed to child 10 with standard shutdown (server server.rudder.local:443)

After loads of debugging, we found that the self-signed certificate in use is faulty, and that just adding an email to the subject make every thing works.

Not working certificate generation command:

openssl req -new -x509 -newkey rsa:2048 -subj "/CN=$(hostname --fqdn)/" -keyout /opt/rudder/etc/ssl/rudder-webapp.key -out /opt/rudder/etc/ssl/rudder-webapp.crt -days 1460 -nodes -sha256

And the working one:

$ openssl req -new -x509 -newkey rsa:2048 -subj "/CN=$(hostname --fqdn)/emailAddress=root@localhost/"  -keyout rudder-webapp.key -out rudder-webapp.crt -days 1460 -nodes -sha256

(attached a couple of working and a couple of not working certificate/key).

Some more information: it happens for firefox on at least Archlinux, Mint, Fedora (but perhaps not always Debian or Ubuntu). It seems to not be related with the Apache version or server distribution (happens in both centos and debian at least).