Project

General

Profile

Actions

Architecture #11906

closed

rsyslog handling of received log on rudder server is inefficient

Added by Nicolas CHARLES almost 7 years ago. Updated over 6 years ago.

Status:
Rejected
Priority:
N/A
Assignee:
-
Category:
Performance and scalability
Target version:
Effort required:
Name check:
Fix check:
Regression:

Description

On the Rudder server, each time rsyslog receives a message, it writes it in the database, doing a transaction for only one log message
This is highly inefficient, both for rsyslog and database, and may lead to massive slowdown of infra if rsyslog is used in tcp

Some pointer on how to do it there:
http://www.rsyslog.com/doc/v8-stable/concepts/queues.html
http://www.rsyslog.com/doc/v8-stable/rainerscript/queue_parameters.html
http://rsyslog-users.1305293.n2.nabble.com/Rsyslog-Disk-Queue-Flush-Issue-td7580422.html

Open questions are:
  • how can we force the message to get fast to the database, for it to be handled by Rudder and not lag there for 1 hours if there are few nodes ?
  • memory usage of storing in memory logs ?
Actions

Also available in: Atom PDF