Project

General

Profile

Bug #14506

Improve 'rsyslog' to manage larger load of reports

Added by Nicolas CHARLES 6 months ago. Updated 5 months ago.

Status:
Released
Priority:
N/A
Category:
Performance and scalability
Target version:
Severity:
User visibility:
Effort required:
Priority:
0

Description

We use DirectQueue in rsyslog, which is quite inefficient for very large load.
Messages can be dropped in case of continuous burst, or sustained high quantity of message, especially on old version of rsyslog

We need to have also several workers to consume these message - but we'll put a concervative 2 workers in this change, for small installation

Some metrics:
Receiving 160 000 message with a DirectQueue of 100 000 elements with rsyslog 8.1901.0 causes it to drop (udp level) 45 000 message, while with LinkedList, only 100 are lost

To correctly handle 1400 messages/s (sustained), it is required to have rsyslog 8.1901.0 at least, with a LinkedList of 100 000 element, enlarged udp buffer ( net.core.rmem_max=26214400 and net.core.rmem_default=26214400 ) and at least 8 CPUs + 2 for PosgreSQL

A large part of the cost comes from the regular expression parsing of the messages received (can reach 200% CPUs just for the parsing) which prevents mesage to enqueue.

Note that the new protocol upcoming in Rudder 5.1 will not suffer from this issue, as syslog won't be used anymore


Subtasks

Bug #14621: rsyslog complains about low discardMarkReleasedBenoît PECCATTEActions

Related issues

Related to Rudder - Bug #15028: Improve rsyslog performance by simplifying parsing of reportsReleasedActions

Associated revisions

Revision 743d0697 (diff)
Added by Nicolas CHARLES 6 months ago

Fixes #14506: rsyslog configuration is quite inefficient for large load

History

#1

Updated by Nicolas CHARLES 6 months ago

  • Status changed from New to In progress
  • Assignee set to Nicolas CHARLES
#2

Updated by Nicolas CHARLES 6 months ago

  • Status changed from In progress to Pending technical review
  • Assignee changed from Nicolas CHARLES to Alexis MOUSSET
  • Pull Request set to https://github.com/Normation/rudder-techniques/pull/1423
#3

Updated by Nicolas CHARLES 6 months ago

  • Status changed from Pending technical review to Pending release
#4

Updated by Vincent MEMBRÉ 5 months ago

  • Subject changed from rsyslog configuration is quite inefficient for large load to Improve 'rsyslog' to manage larger load of reports
#5

Updated by Vincent MEMBRÉ 5 months ago

  • Status changed from Pending release to Released

This bug has been fixed in Rudder 5.0.9 which was released today.

#6

Updated by Nicolas CHARLES 4 months ago

  • Related to Bug #15028: Improve rsyslog performance by simplifying parsing of reports added

Also available in: Atom PDF