Improve 'rsyslog' to manage larger load of reports
We use DirectQueue in rsyslog, which is quite inefficient for very large load.
Messages can be dropped in case of continuous burst, or sustained high quantity of message, especially on old version of rsyslog
We need to have also several workers to consume these message - but we'll put a concervative 2 workers in this change, for small installation
Receiving 160 000 message with a DirectQueue of 100 000 elements with rsyslog 8.1901.0 causes it to drop (udp level) 45 000 message, while with LinkedList, only 100 are lost
To correctly handle 1400 messages/s (sustained), it is required to have rsyslog 8.1901.0 at least, with a LinkedList of 100 000 element, enlarged udp buffer ( net.core.rmem_max=26214400 and net.core.rmem_default=26214400 ) and at least 8 CPUs + 2 for PosgreSQL
A large part of the cost comes from the regular expression parsing of the messages received (can reach 200% CPUs just for the parsing) which prevents mesage to enqueue.
Note that the new protocol upcoming in Rudder 5.1 will not suffer from this issue, as syslog won't be used anymore
Updated by Nicolas CHARLES 2 months ago
- Status changed from In progress to Pending technical review
- Assignee changed from Nicolas CHARLES to Alexis MOUSSET
- Pull Request set to https://github.com/Normation/rudder-techniques/pull/1423
Updated by Nicolas CHARLES about 2 months ago
- Status changed from Pending technical review to Pending release
Applied in changeset rudder-techniques|743d0697ce0db1e38aa2332b3efecba30b597d1a.