User story #7220: Document nofiles dependency for syslog/tcp on master and relays - Rudder - Issue Tracker

Custom queries

8.1 beta
Bugs triage
CS - by priority
CS - to review
My TR
Need technical review
Open bugs (all)
Open bugs (UI - UX)
Open UX defects (all)
Opened by users

Actions

Copy link

User story #7220

closed

Document nofiles dependency for syslog/tcp on master and relays

Added by Florian Heigl about 9 years ago. Updated almost 9 years ago.

Status:

Released

Priority:

N/A

Assignee:

Jonathan CLARKE

Category:

Documentation

Target version:

2.11.18

Pull Request:

https://github.com/Normation/rudder-d...

UX impact:

Suggestion strength:

User visibility:

Effort required:

Name check:

Fix check:

Regression:

Description

If syslog over tcp is used you need to raise the open files limits affecting the number of sockets open.

This affects relays, or if none are used, also the master.
Can you add that to the rudder relay setup docs?

Just put in a recommendation to raise the open files limits to something like 10000.

(No: You don't want to see what happens it it's too low.)

Subtasks 1 (0 open — 1 closed)

User story #7728: Document how to switch to UDP for reporting when upgrading to >= 3.1

Released

Jonathan CLARKE

2016-01-06

Actions

History
Notes
Property changes
Associated revisions

Actions

Copy link

Updated by Florian Heigl about 9 years ago

ah, and turning off syncookies can also be needed under load :)

Actions

Copy link

Updated by Janos Mattyasovszky about 9 years ago

Well, on SUSE's rsyslogd (v 5.8.7-0.5.5) you'd get messages like:

Sep 22 16:39:38 relayserver rsyslogd-2163: last message repeated 401230 times
Sep 22 16:39:38 relayserver rsyslogd-2163: epoll_ctl failed on fd 30, id 0/0x7f2c1000e650, op 1 with File exists
: File exists [try http://www.rsyslog.com/e/2163 ]
Sep 22 16:39:42 relayserver rsyslogd-2163: last message repeated 143065 times
Sep 22 16:39:42 relayserver rsyslogd-2163: epoll_ctl failed on fd 14, id 0/0x7f2c100021a0, op 1 with File exists
: File exists [try http://www.rsyslog.com/e/2163 ]
Sep 22 16:39:43 relayserver rsyslogd-2163: last message repeated 9846 times
Sep 22 16:39:42 relayserver rsyslogd-2163: epoll_ctl failed on fd 20, id 0/0x7f2c10008d30, op 1 with File exists
: File exists [try http://www.rsyslog.com/e/2163 ]
Sep 22 16:39:42 relayserver rsyslogd-2163: epoll_ctl failed on fd 14, id 0/0x7f2c100021a0, op 1 with File exists
: File exists [try http://www.rsyslog.com/e/2163 ]

Setting it to 10.000 was not enough for us, I had to set soft to 30k and hard to 100k, but this depends on the relay, how much clients it serves and how well it is fit with CPU/Memory. (This example Relay has approx ~2700 Clients)

And as Flo already mentioned, if you (by mistake) change a lot of systems to TCP instead of UDP, and they start sending the rudder messages via TCP, you might also experience Messages like this:

Sep 22 17:59:02 relayserver kernel: [7695000.930982] TCP: Possible SYN flooding on port 514. Sending cookies.
Sep 22 17:59:02 relayserver kernel: [7695000.931004] TCP: Possible SYN flooding on port 514. Sending cookies.
Sep 22 17:59:02 relayserver kernel: [7695000.931061] TCP: Possible SYN flooding on port 514. Sending cookies.
Sep 22 17:59:02 relayserver kernel: [7695000.931075] TCP: Possible SYN flooding on port 514. Sending cookies.
Sep 22 17:59:02 relayserver kernel: [7695000.931151] TCP: Possible SYN flooding on port 514. Sending cookies.
Sep 22 17:59:02 relayserver kernel: [7695000.931165] TCP: Possible SYN flooding on port 514. Sending cookies.
Sep 22 17:59:02 relayserver kernel: [7695000.931398] TCP: Possible SYN flooding on port 514. Sending cookies.
Sep 22 17:59:02 relayserver kernel: [7695000.931439] TCP: Possible SYN flooding on port 514. Sending cookies.
Sep 22 17:59:02 relayserver kernel: [7695000.931446] TCP: Possible SYN flooding on port 514. Sending cookies.
Sep 22 17:59:02 relayserver kernel: [7695000.931452] TCP: Possible SYN flooding on port 514. Sending cookies.

Also pay attention to the conntrack table, which seems to have a pretty high value of 65k, but this is shared among all processes, so if also handling additional workloads with many connections, this might also be worth to monitor (basically /proc/sys/net/netfilter/nf_conntrack_count).

Actions

Copy link