User story #7220
closedDocument nofiles dependency for syslog/tcp on master and relays
Added by Florian Heigl about 9 years ago. Updated over 8 years ago.
Description
If syslog over tcp is used you need to raise the open files limits affecting the number of sockets open.
This affects relays, or if none are used, also the master.
Can you add that to the rudder relay setup docs?
Just put in a recommendation to raise the open files limits to something like 10000.
(No: You don't want to see what happens it it's too low.)
Updated by Florian Heigl about 9 years ago
ah, and turning off syncookies can also be needed under load :)
Updated by Janos Mattyasovszky about 9 years ago
Well, on SUSE's rsyslogd (v 5.8.7-0.5.5) you'd get messages like:
Sep 22 16:39:38 relayserver rsyslogd-2163: last message repeated 401230 times Sep 22 16:39:38 relayserver rsyslogd-2163: epoll_ctl failed on fd 30, id 0/0x7f2c1000e650, op 1 with File exists : File exists [try http://www.rsyslog.com/e/2163 ] Sep 22 16:39:42 relayserver rsyslogd-2163: last message repeated 143065 times Sep 22 16:39:42 relayserver rsyslogd-2163: epoll_ctl failed on fd 14, id 0/0x7f2c100021a0, op 1 with File exists : File exists [try http://www.rsyslog.com/e/2163 ] Sep 22 16:39:43 relayserver rsyslogd-2163: last message repeated 9846 times Sep 22 16:39:42 relayserver rsyslogd-2163: epoll_ctl failed on fd 20, id 0/0x7f2c10008d30, op 1 with File exists : File exists [try http://www.rsyslog.com/e/2163 ] Sep 22 16:39:42 relayserver rsyslogd-2163: epoll_ctl failed on fd 14, id 0/0x7f2c100021a0, op 1 with File exists : File exists [try http://www.rsyslog.com/e/2163 ]
Setting it to 10.000 was not enough for us, I had to set soft to 30k and hard to 100k, but this depends on the relay, how much clients it serves and how well it is fit with CPU/Memory. (This example Relay has approx ~2700 Clients)
And as Flo already mentioned, if you (by mistake) change a lot of systems to TCP instead of UDP, and they start sending the rudder messages via TCP, you might also experience Messages like this:
Sep 22 17:59:02 relayserver kernel: [7695000.930982] TCP: Possible SYN flooding on port 514. Sending cookies. Sep 22 17:59:02 relayserver kernel: [7695000.931004] TCP: Possible SYN flooding on port 514. Sending cookies. Sep 22 17:59:02 relayserver kernel: [7695000.931061] TCP: Possible SYN flooding on port 514. Sending cookies. Sep 22 17:59:02 relayserver kernel: [7695000.931075] TCP: Possible SYN flooding on port 514. Sending cookies. Sep 22 17:59:02 relayserver kernel: [7695000.931151] TCP: Possible SYN flooding on port 514. Sending cookies. Sep 22 17:59:02 relayserver kernel: [7695000.931165] TCP: Possible SYN flooding on port 514. Sending cookies. Sep 22 17:59:02 relayserver kernel: [7695000.931398] TCP: Possible SYN flooding on port 514. Sending cookies. Sep 22 17:59:02 relayserver kernel: [7695000.931439] TCP: Possible SYN flooding on port 514. Sending cookies. Sep 22 17:59:02 relayserver kernel: [7695000.931446] TCP: Possible SYN flooding on port 514. Sending cookies. Sep 22 17:59:02 relayserver kernel: [7695000.931452] TCP: Possible SYN flooding on port 514. Sending cookies.
Also pay attention to the conntrack table, which seems to have a pretty high value of 65k, but this is shared among all processes, so if also handling additional workloads with many connections, this might also be worth to monitor (basically /proc/sys/net/netfilter/nf_conntrack_count).
Updated by Vincent MEMBRÉ almost 9 years ago
- Assignee set to François ARMAND
- Target version set to 2.11.15
François, who do you think would be the best to document this ?
Updated by Vincent MEMBRÉ almost 9 years ago
- Target version changed from 2.11.15 to 2.11.16
Updated by Vincent MEMBRÉ almost 9 years ago
- Target version changed from 2.11.16 to 2.11.17
Updated by Vincent MEMBRÉ almost 9 years ago
- Target version changed from 2.11.17 to 2.11.18
Updated by Jonathan CLARKE almost 9 years ago
- Assignee changed from François ARMAND to Alexis Mousset
Updated by Alexis Mousset almost 9 years ago
- Status changed from New to In progress
Updated by Alexis Mousset almost 9 years ago
- Status changed from In progress to Pending technical review
- Assignee changed from Alexis Mousset to Benoît PECCATTE
- Pull Request set to https://github.com/Normation/rudder-doc/pull/148
Updated by Alexis Mousset over 8 years ago
- Assignee changed from Benoît PECCATTE to François ARMAND
Updated by Alexis Mousset over 8 years ago
- Assignee changed from François ARMAND to Benoît PECCATTE
Updated by Alexis Mousset over 8 years ago
- Assignee changed from Benoît PECCATTE to Jonathan CLARKE
Updated by Alexis Mousset over 8 years ago
- Status changed from Pending technical review to Pending release
- % Done changed from 0 to 100
Applied in changeset rudder-doc|0dffbb020e43100bd1986e21a6d27fc929e04bbf.
Updated by Vincent MEMBRÉ over 8 years ago
- Tracker changed from Bug to User story
Updated by Vincent MEMBRÉ over 8 years ago
- Status changed from Pending release to Released