Bug #4473
closedJetty cache files in /tmp directory are removed by the OS tmpwatch cron job, preventing access to the application
Description
I was not able to work on Rudder for over a week and when I tried to log in today I've got a white screen with a couple of screen and no images.
The apache logs show the following errors (just an example, there are more):
10.32.48.196 - - [14/Feb/2014:14:05:33 -0800] "GET /rudder/javascript/jquery/jquery-1.8.3.min.js HTTP/1.1" 404 286 "https://sjc1-ops-proxmox01-ve-104.carrieriq.com/rudder/" "Mozilla/5.0 (X11; Linux i686; rv:24.0) Gecko/20100101 Firefox/24.0" 10.32.48.196 - - [14/Feb/2014:14:05:33 -0800] "GET /rudder/images/themes/ui-lightness/jquery.ui.all.css HTTP/1.1" 404 293 "https://sjc1-ops-proxmox01-ve-104.carrieriq.com/rudder/" "Mozilla/5.0 (X11; Linux i686; rv:24.0) Gecko/20100101 Firefox/24.0" 10.32.48.196 - - [14/Feb/2014:14:05:33 -0800] "GET /rudder/style/rudder-login.css HTTP/1.1" 404 271 "https://sjc1-ops-proxmox01-ve-104.carrieriq.com/rudder/" "Mozilla/5.0 (X11; Linux i686; rv:24.0) Gecko/20100101 Firefox/24.0" 10.32.48.196 - - [14/Feb/2014:14:05:33 -0800] "GET /rudder/images/logoIndexRudder.jpg HTTP/1.1" 404 275 "https://sjc1-ops-proxmox01-ve-104.carrieriq.com/rudder/" "Mozilla/5.0 (X11; Linux i686; rv:24.0) Gecko/20100101 Firefox/24.0"
The problem went away after restarting jetty; I was able to track it down to the directory /tmp/jetty-127.0.0.1-8080-rudder.war-_rudder-any- - if I manually delete this directory, the effect is the same.
So apparently this was the daily cron job (I am using CentOS 6) which removes files not accessed for certain period of time.
The problem could be addressed in one of two ways:
- Create an exception for the tmpwatch in /etc/cron.daily/tmpwatch to exclude certain directories (which exactly?)
- Create temp files elsewhere, where the tmpwatch does not apply.
Either approach has its own pros and cons - i.e. it could be made a matter of policy to maintain the tmpwatch file (but rpm upgrades will be fixing it all the time); on the other hand deploying the war and whatnot in the application directory may be subject to upgrade problems.
I do not know the right solution at the moment but thought I would bring it up just to raise awareness of the issue.
Updated by François ARMAND almost 11 years ago
- Project changed from 24 to Rudder
- Subject changed from Web server files in /tmp directory are removed by the OS tmpwatch cron job, preventing access to the application to Letty cache files in /tmp directory are removed by the OS tmpwatch cron job, preventing access to the application
- Category set to Web - Maintenance
- Status changed from New to 8
- Assignee set to François ARMAND
- Priority changed from N/A to 2
- Target version set to 2.6.11
Ho, that's a really, really nice catch. I believe that we could also make jetty be smarter, and don't try to serve deleted file. I'm taking that one, it should just not happen.
Thanks for reporting it !
Updated by Matthieu CERDA almost 11 years ago
- Subject changed from Letty cache files in /tmp directory are removed by the OS tmpwatch cron job, preventing access to the application to Jetty cache files in /tmp directory are removed by the OS tmpwatch cron job, preventing access to the application
Updated by Vincent MEMBRÉ over 10 years ago
Jetty needs to extract some files of the war (I hope not the whole war) and by default extract to /tmp. ( see http://stackoverflow.com/questions/19232182/jetty-starts-in-c-temp/19232771#19232771)
The directory is defined by a java option defined at jetty startup -Djava.io.tmpdir
If we defined that variable to /var/lib/jetty (or rudder-jetty as 2.10) that would fix the issue.
Before doing that we need to look how jetty cleanup its temporary file, because we do not want to have them staying forever.
Updated by Vincent MEMBRÉ over 10 years ago
- Target version changed from 2.6.11 to 2.6.12
Updated by Vincent MEMBRÉ over 10 years ago
- Target version changed from 2.6.12 to 2.6.13
Updated by Vincent MEMBRÉ over 10 years ago
- Target version changed from 2.6.13 to 2.6.14
Updated by Jonathan CLARKE over 10 years ago
- Target version changed from 2.6.14 to 2.6.16
Updated by Jonathan CLARKE over 10 years ago
- Target version changed from 2.6.16 to 2.6.17
Updated by Nicolas PERRON over 10 years ago
- Target version changed from 2.6.17 to 2.6.18
Updated by Matthieu CERDA about 10 years ago
- Target version changed from 2.6.18 to 2.6.19
Updated by Vincent MEMBRÉ about 10 years ago
- Target version changed from 2.6.19 to 2.6.20
Updated by François ARMAND almost 10 years ago
- Assignee deleted (
François ARMAND) - Target version changed from 2.6.20 to 2.10.10
This one is important, it will bite us somewhere.
Updated by Vincent MEMBRÉ almost 10 years ago
- Target version changed from 2.10.10 to 2.10.11
Updated by Vincent MEMBRÉ over 9 years ago
- Target version changed from 2.10.11 to 2.10.12
Updated by Vincent MEMBRÉ over 9 years ago
- Target version changed from 2.10.12 to 2.10.13
Updated by Vincent MEMBRÉ over 9 years ago
- Target version changed from 2.10.13 to 2.10.14
Updated by Vincent MEMBRÉ over 9 years ago
- Target version changed from 2.10.14 to 2.10.15
Updated by Vincent MEMBRÉ over 9 years ago
- Target version changed from 2.10.15 to 2.10.16
Updated by Vincent MEMBRÉ over 9 years ago
- Target version changed from 2.10.16 to 2.10.17
Updated by Vincent MEMBRÉ about 9 years ago
- Target version changed from 2.10.17 to 2.10.18
Updated by Vincent MEMBRÉ about 9 years ago
- Target version changed from 2.10.18 to 2.10.19
Updated by Vincent MEMBRÉ about 9 years ago
- Target version changed from 2.10.19 to 2.10.20
Updated by Vincent MEMBRÉ almost 9 years ago
- Target version changed from 2.10.20 to 2.11.18
Updated by Jonathan CLARKE almost 9 years ago
- Status changed from New to In progress
- Assignee set to Jonathan CLARKE
Updated by Nicolas CHARLES almost 9 years ago
The folder is normally defined by the environment variable TMPDIR
Updated by Jonathan CLARKE almost 9 years ago
- Status changed from In progress to Pending technical review
- Assignee changed from Jonathan CLARKE to Matthieu CERDA
- Pull Request set to https://github.com/Normation/rudder-packages/pull/868
Updated by Jonathan CLARKE almost 9 years ago
This happened because jetty's init script looks for a variable called TMPDIR, and if not defined it uses "/tmp" by default.
I've adapted it to usr /var/rudder/tmp/jetty instead.
Updated by Jonathan CLARKE almost 9 years ago
- Status changed from Pending technical review to Pending release
- % Done changed from 0 to 100
Applied in changeset rudder-packages|13ccd9a44ff45c002ce14184f7390d232b89c18c.
Updated by Vincent MEMBRÉ almost 9 years ago
- Status changed from Pending release to Released