User story #7220: Document nofiles dependency for syslog/tcp on master and relays - Rudder - Issue Tracker

Actions

Copy link

User story #7220

closed

Document nofiles dependency for syslog/tcp on master and relays

Added by Florian Heigl about 9 years ago. Updated almost 9 years ago.

Status:

Released

Priority:

N/A

Assignee:

Jonathan CLARKE

Category:

Documentation

Target version:

2.11.18

Pull Request:

https://github.com/Normation/rudder-d...

UX impact:

Suggestion strength:

User visibility:

Effort required:

Name check:

Fix check:

Regression:

Description

If syslog over tcp is used you need to raise the open files limits affecting the number of sockets open.

This affects relays, or if none are used, also the master.
Can you add that to the rudder relay setup docs?

Just put in a recommendation to raise the open files limits to something like 10000.

(No: You don't want to see what happens it it's too low.)

Subtasks 1 (0 open — 1 closed)

Actions

Copy link

Updated by Florian Heigl about 9 years ago

ah, and turning off syncookies can also be needed under load :)

Actions

Copy link

Updated by Janos Mattyasovszky about 9 years ago

Well, on SUSE's rsyslogd (v 5.8.7-0.5.5) you'd get messages like:

Sep 22 16:39:38 relayserver rsyslogd-2163: last message repeated 401230 times
Sep 22 16:39:38 relayserver rsyslogd-2163: epoll_ctl failed on fd 30, id 0/0x7f2c1000e650, op 1 with File exists
: File exists [try http://www.rsyslog.com/e/2163 ]
Sep 22 16:39:42 relayserver rsyslogd-2163: last message repeated 143065 times
Sep 22 16:39:42 relayserver rsyslogd-2163: epoll_ctl failed on fd 14, id 0/0x7f2c100021a0, op 1 with File exists
: File exists [try http://www.rsyslog.com/e/2163 ]
Sep 22 16:39:43 relayserver rsyslogd-2163: last message repeated 9846 times
Sep 22 16:39:42 relayserver rsyslogd-2163: epoll_ctl failed on fd 20, id 0/0x7f2c10008d30, op 1 with File exists
: File exists [try http://www.rsyslog.com/e/2163 ]
Sep 22 16:39:42 relayserver rsyslogd-2163: epoll_ctl failed on fd 14, id 0/0x7f2c100021a0, op 1 with File exists
: File exists [try http://www.rsyslog.com/e/2163 ]

Setting it to 10.000 was not enough for us, I had to set soft to 30k and hard to 100k, but this depends on the relay, how much clients it serves and how well it is fit with CPU/Memory. (This example Relay has approx ~2700 Clients)

And as Flo already mentioned, if you (by mistake) change a lot of systems to TCP instead of UDP, and they start sending the rudder messages via TCP, you might also experience Messages like this:

Sep 22 17:59:02 relayserver kernel: [7695000.930982] TCP: Possible SYN flooding on port 514. Sending cookies.
Sep 22 17:59:02 relayserver kernel: [7695000.931004] TCP: Possible SYN flooding on port 514. Sending cookies.
Sep 22 17:59:02 relayserver kernel: [7695000.931061] TCP: Possible SYN flooding on port 514. Sending cookies.
Sep 22 17:59:02 relayserver kernel: [7695000.931075] TCP: Possible SYN flooding on port 514. Sending cookies.
Sep 22 17:59:02 relayserver kernel: [7695000.931151] TCP: Possible SYN flooding on port 514. Sending cookies.
Sep 22 17:59:02 relayserver kernel: [7695000.931165] TCP: Possible SYN flooding on port 514. Sending cookies.
Sep 22 17:59:02 relayserver kernel: [7695000.931398] TCP: Possible SYN flooding on port 514. Sending cookies.
Sep 22 17:59:02 relayserver kernel: [7695000.931439] TCP: Possible SYN flooding on port 514. Sending cookies.
Sep 22 17:59:02 relayserver kernel: [7695000.931446] TCP: Possible SYN flooding on port 514. Sending cookies.
Sep 22 17:59:02 relayserver kernel: [7695000.931452] TCP: Possible SYN flooding on port 514. Sending cookies.

Also pay attention to the conntrack table, which seems to have a pretty high value of 65k, but this is shared among all processes, so if also handling additional workloads with many connections, this might also be worth to monitor (basically /proc/sys/net/netfilter/nf_conntrack_count).

Actions

Copy link