Architecture #11906
closed
rsyslog handling of received log on rudder server is inefficient
Description
On the Rudder server, each time rsyslog receives a message, it writes it in the database, doing a transaction for only one log message
This is highly inefficient, both for rsyslog and database, and may lead to massive slowdown of infra if rsyslog is used in tcp
Some pointer on how to do it there:
http://www.rsyslog.com/doc/v8-stable/concepts/queues.html
http://www.rsyslog.com/doc/v8-stable/rainerscript/queue_parameters.html
http://rsyslog-users.1305293.n2.nabble.com/Rsyslog-Disk-Queue-Flush-Issue-td7580422.html
- how can we force the message to get fast to the database, for it to be handled by Rudder and not lag there for 1 hours if there are few nodes ?
- memory usage of storing in memory logs ?
Updated by Nicolas CHARLES almost 7 years ago
apparently, we cannot do multiple insert in database, see https://github.com/rsyslog/rsyslog/issues/399
but we can still use a queue to handle burst of data that might appear
Updated by Nicolas CHARLES almost 7 years ago
transaction were implemented in https://github.com/rsyslog/rsyslog/pull/2172
Updated by Vincent MEMBRÉ almost 7 years ago
- Target version changed from 4.3.0~beta1 to 4.3.0~rc1
Updated by Vincent MEMBRÉ almost 7 years ago
- Target version changed from 4.3.0~rc1 to 4.3.0~rc2
Updated by François ARMAND almost 7 years ago
This one seems risky (big potential impact) and syslog will disappear in next major. Should we not close it?
Updated by Nicolas CHARLES almost 7 years ago
- Status changed from New to Rejected
Yes, let's close it - the timeframe before new rsyslog-psql will become available in maintained distribution is after we'll stop using syslog