Project

General

Profile

Bug #6127

Rsyslog problem on Ubuntu 14.04 fresh install

Added by Dennis Cabooter about 6 years ago. Updated over 5 years ago.

Status:
Released
Priority:
1
Category:
System integration
Target version:
Severity:
User visibility:
Effort required:
Priority:

Description

Basicly there are 2 problems:

  • After upgrading the Rudder server, it doesn't recieve any logs from nodes. Restarting rsyslog does not solve the problem, rebooting does
  • After installing a new node, it doesn't send any logs to the Rudder server. Again, restarting rsyslog does not solve the problem, rebooting does

I'm not sure if it's a rudder or an Ubuntu/rsyslog issue. Please keep in mind Ubuntu 14.04 used Upstart, don't use /etc/init.d/rsyslog restart, but service syslog restart.


Related issues

Has duplicate Rudder - Bug #6014: When upgrading Rudder server on Ubuntu, all nodes gets to no reportRejectedBenoît PECCATTE2014-12-16Actions
#1

Updated by Dennis Cabooter about 6 years ago

  • Category set to System integration
  • Priority changed from N/A to 3
  • Target version set to 2.11.6
#2

Updated by François ARMAND about 6 years ago

  • Assignee set to Benoît PECCATTE

OK, so first step with that one: trying to reproduce it :)

Benoit, is it a mission that you think is possible to handle ?

#3

Updated by Vincent MEMBRÉ about 6 years ago

  • Target version changed from 2.11.6 to 2.11.7
#4

Updated by Vincent MEMBRÉ about 6 years ago

  • Target version changed from 2.11.7 to 2.11.8
#5

Updated by Vincent MEMBRÉ almost 6 years ago

  • Target version changed from 2.11.8 to 2.11.9
#6

Updated by Vincent MEMBRÉ almost 6 years ago

  • Target version changed from 2.11.9 to 2.11.10
#7

Updated by Nicolas CHARLES almost 6 years ago

  • Priority changed from 3 to 1

This is quite painful for users ... can someone have a real look at this one please ?

#8

Updated by Nicolas CHARLES almost 6 years ago

could it be a file permission issue ? http://serverfault.com/questions/596558/ubuntu-14-04-system-logging
if the file as not the right permission, then rsyslog fails to write in the file and skip ?

#9

Updated by Nicolas CHARLES almost 6 years ago

  • Assignee changed from Benoît PECCATTE to Matthieu CERDA

Reassigning to Matthieu

#10

Updated by Vincent MEMBRÉ almost 6 years ago

  • Target version changed from 2.11.10 to 2.11.11
#11

Updated by Vincent MEMBRÉ almost 6 years ago

  • Target version changed from 2.11.11 to 2.11.12
#12

Updated by Vincent MEMBRÉ over 5 years ago

  • Target version changed from 2.11.12 to 2.11.13
#13

Updated by Nicolas CHARLES over 5 years ago

I had to manually restart rsyslog for it to work, but not to reboot (on Rudder 3.0.7 and ubuntu 14.04)
however, services were super slow at start

#14

Updated by Nicolas CHARLES over 5 years ago

ok, restarting rsyslog unclog some messages, but not for a long time ...

#15

Updated by Nicolas CHARLES over 5 years ago

restarting rsyslog does
Jul 13 17:20:26 server rsyslogd: [origin software="rsyslogd" swVersion="7.4.4" x-pid="8060" x-info="http://www.rsyslog.com"] exiting on signal 15.
Jul 13 17:20:26 server rsyslogd: [origin software="rsyslogd" swVersion="7.4.4" x-pid="8347" x-info="http://www.rsyslog.com"] start
Jul 13 17:20:26 server rsyslogd-2307: warning: ~ action is deprecated, consider using the 'stop' statement instead [try http://www.rsyslog.com/e/2307 ]
Jul 13 17:20:26 server rsyslogd-2307: warning: ~ action is deprecated, consider using the 'stop' statement instead [try http://www.rsyslog.com/e/2307 ]
Jul 13 17:20:26 server rsyslogd-2307: warning: ~ action is deprecated, consider using the 'stop' statement instead [try http://www.rsyslog.com/e/2307 ]
Jul 13 17:20:26 server rsyslogd: rsyslogd's groupid changed to 104
Jul 13 17:20:26 server rsyslogd: rsyslogd's userid changed to 101
Jul 13 17:20:27 server rudder8131: R: @server-roles@result_success@server-roles@server-roles-directive@1@Check logrotate configuration@None@2015-07-13 17:20:25+00:00##root@#The logrotate configuration is correct

send it seems to stop

#16

Updated by Nicolas CHARLES over 5 years ago

one important step is to
chown syslog:adm /var/log/rudder/reports and /var/log/rudder/ldap

#17

Updated by Nicolas CHARLES over 5 years ago

it seems that the rsyslog stops or block itself at reception of rudder logs when inserting them in database.
if I comment out the writing to DB, it does work find, otherwise it blocks itself

#18

Updated by Nicolas CHARLES over 5 years ago

restarting the ubuntu server did fix the issue
log don't tell me why it changed
I git init /etc before, and I only get very minor changes (caches for apparmor, or host file changing, and nologin file deleted)

#19

Updated by Nicolas CHARLES over 5 years ago

  • Assignee changed from Matthieu CERDA to Benoît PECCATTE

Benoit, could you try to fix this ? this is a major pain for new users ...

#20

Updated by Alexis MOUSSET over 5 years ago

The problem is in ompgsql, and was fixed in rsyslog 7.6.0 and 7.4.10: http://www.rsyslog.com/changelog-for-7-6-0-v7-stable/

ompgsql bugfix: improper handling of auto-backgrounding mode
If rsyslog was set to auto-background itself (default code behaviour, but many distros now turn it off for good reason), ompgsql could not properly connect. This could even lead to a segfault. The core reason was that a PG session handle was kept open over a fork, something that is explicitely forbidden in the PG API.

The fix is available here, and was tested on 7.4.4/ubuntu 14.04.

#21

Updated by Alexis MOUSSET over 5 years ago

  • Has duplicate Bug #6014: When upgrading Rudder server on Ubuntu, all nodes gets to no report added
#22

Updated by Alexis MOUSSET over 5 years ago

  • Status changed from New to In progress
  • Assignee changed from Benoît PECCATTE to Alexis MOUSSET

Upstream bug reported for the Ubuntu package.

Building packages including the fix for the next Rudder release.

#23

Updated by Benoît PECCATTE over 5 years ago

  • Assignee changed from Alexis MOUSSET to Matthieu CERDA

Alexis made a package with the patch.

Matthieu, can you update rudder dependencies to depend on a version >= to Alexis's version on ubuntu 14.04

#24

Updated by Matthieu CERDA over 5 years ago

  • Status changed from In progress to Pending technical review
  • Assignee changed from Matthieu CERDA to Benoît PECCATTE
  • Pull Request set to https://github.com/Normation/rudder-packages/pull/731
#25

Updated by Matthieu CERDA over 5 years ago

  • Status changed from Pending technical review to Pending release
  • % Done changed from 0 to 100
#26

Updated by Matthieu CERDA over 5 years ago

#28

Updated by François ARMAND over 5 years ago

  • Status changed from Pending release to Released

This bug has been fixed in Rudder 2.11.13, 3.0.8 and 3.1.1 which were released today.

Also available in: Atom PDF