Project

General

Profile

Actions

Bug #4473

closed

Jetty cache files in /tmp directory are removed by the OS tmpwatch cron job, preventing access to the application

Added by Alex Tkachenko about 7 years ago. Updated over 5 years ago.

Status:
Released
Priority:
2
Assignee:
Matthieu CERDA
Category:
Web - Maintenance
Target version:
Severity:
User visibility:
Effort required:
Priority:

Description

I was not able to work on Rudder for over a week and when I tried to log in today I've got a white screen with a couple of screen and no images.

The apache logs show the following errors (just an example, there are more):

10.32.48.196 - - [14/Feb/2014:14:05:33 -0800] "GET /rudder/javascript/jquery/jquery-1.8.3.min.js HTTP/1.1" 404 286 "https://sjc1-ops-proxmox01-ve-104.carrieriq.com/rudder/" "Mozilla/5.0 (X11; Linux i686; rv:24.0) Gecko/20100101 Firefox/24.0" 
10.32.48.196 - - [14/Feb/2014:14:05:33 -0800] "GET /rudder/images/themes/ui-lightness/jquery.ui.all.css HTTP/1.1" 404 293 "https://sjc1-ops-proxmox01-ve-104.carrieriq.com/rudder/" "Mozilla/5.0 (X11; Linux i686; rv:24.0) Gecko/20100101 Firefox/24.0" 
10.32.48.196 - - [14/Feb/2014:14:05:33 -0800] "GET /rudder/style/rudder-login.css HTTP/1.1" 404 271 "https://sjc1-ops-proxmox01-ve-104.carrieriq.com/rudder/" "Mozilla/5.0 (X11; Linux i686; rv:24.0) Gecko/20100101 Firefox/24.0" 
10.32.48.196 - - [14/Feb/2014:14:05:33 -0800] "GET /rudder/images/logoIndexRudder.jpg HTTP/1.1" 404 275 "https://sjc1-ops-proxmox01-ve-104.carrieriq.com/rudder/" "Mozilla/5.0 (X11; Linux i686; rv:24.0) Gecko/20100101 Firefox/24.0" 

The problem went away after restarting jetty; I was able to track it down to the directory /tmp/jetty-127.0.0.1-8080-rudder.war-_rudder-any- - if I manually delete this directory, the effect is the same.

So apparently this was the daily cron job (I am using CentOS 6) which removes files not accessed for certain period of time.

The problem could be addressed in one of two ways:

  1. Create an exception for the tmpwatch in /etc/cron.daily/tmpwatch to exclude certain directories (which exactly?)
  2. Create temp files elsewhere, where the tmpwatch does not apply.

Either approach has its own pros and cons - i.e. it could be made a matter of policy to maintain the tmpwatch file (but rpm upgrades will be fixing it all the time); on the other hand deploying the war and whatnot in the application directory may be subject to upgrade problems.

I do not know the right solution at the moment but thought I would bring it up just to raise awareness of the issue.

Actions #1

Updated by François ARMAND about 7 years ago

  • Project changed from 24 to Rudder
  • Subject changed from Web server files in /tmp directory are removed by the OS tmpwatch cron job, preventing access to the application to Letty cache files in /tmp directory are removed by the OS tmpwatch cron job, preventing access to the application
  • Category set to Web - Maintenance
  • Status changed from New to 8
  • Assignee set to François ARMAND
  • Priority changed from N/A to 2
  • Target version set to 2.6.11

Ho, that's a really, really nice catch. I believe that we could also make jetty be smarter, and don't try to serve deleted file. I'm taking that one, it should just not happen.

Thanks for reporting it !

Actions #2

Updated by Matthieu CERDA about 7 years ago

  • Subject changed from Letty cache files in /tmp directory are removed by the OS tmpwatch cron job, preventing access to the application to Jetty cache files in /tmp directory are removed by the OS tmpwatch cron job, preventing access to the application
Actions #3

Updated by Vincent MEMBRÉ about 7 years ago

Jetty needs to extract some files of the war (I hope not the whole war) and by default extract to /tmp. ( see http://stackoverflow.com/questions/19232182/jetty-starts-in-c-temp/19232771#19232771)

The directory is defined by a java option defined at jetty startup -Djava.io.tmpdir

If we defined that variable to /var/lib/jetty (or rudder-jetty as 2.10) that would fix the issue.

Before doing that we need to look how jetty cleanup its temporary file, because we do not want to have them staying forever.

Actions #4

Updated by Vincent MEMBRÉ about 7 years ago

  • Target version changed from 2.6.11 to 2.6.12
Actions #5

Updated by Vincent MEMBRÉ about 7 years ago

  • Target version changed from 2.6.12 to 2.6.13
Actions #6

Updated by Vincent MEMBRÉ almost 7 years ago

  • Target version changed from 2.6.13 to 2.6.14
Actions #7

Updated by Jonathan CLARKE almost 7 years ago

  • Target version changed from 2.6.14 to 2.6.16
Actions #8

Updated by Jonathan CLARKE almost 7 years ago

  • Target version changed from 2.6.16 to 2.6.17
Actions #9

Updated by Nicolas PERRON over 6 years ago

  • Target version changed from 2.6.17 to 2.6.18
Actions #10

Updated by Matthieu CERDA over 6 years ago

  • Target version changed from 2.6.18 to 2.6.19
Actions #11

Updated by Vincent MEMBRÉ over 6 years ago

  • Target version changed from 2.6.19 to 2.6.20
Actions #12

Updated by François ARMAND over 6 years ago

  • Assignee deleted (François ARMAND)
  • Target version changed from 2.6.20 to 2.10.10

This one is important, it will bite us somewhere.

Actions #13

Updated by Vincent MEMBRÉ over 6 years ago

  • Target version changed from 2.10.10 to 2.10.11
Actions #14

Updated by Vincent MEMBRÉ about 6 years ago

  • Target version changed from 2.10.11 to 2.10.12
Actions #15

Updated by Vincent MEMBRÉ about 6 years ago

  • Target version changed from 2.10.12 to 2.10.13
Actions #16

Updated by Benoît PECCATTE about 6 years ago

  • Status changed from 8 to New
Actions #17

Updated by Vincent MEMBRÉ about 6 years ago

  • Target version changed from 2.10.13 to 2.10.14
Actions #18

Updated by Vincent MEMBRÉ almost 6 years ago

  • Target version changed from 2.10.14 to 2.10.15
Actions #19

Updated by Vincent MEMBRÉ almost 6 years ago

  • Target version changed from 2.10.15 to 2.10.16
Actions #20

Updated by Vincent MEMBRÉ over 5 years ago

  • Target version changed from 2.10.16 to 2.10.17
Actions #21

Updated by Vincent MEMBRÉ over 5 years ago

  • Target version changed from 2.10.17 to 2.10.18
Actions #22

Updated by Vincent MEMBRÉ over 5 years ago

  • Target version changed from 2.10.18 to 2.10.19
Actions #23

Updated by Vincent MEMBRÉ over 5 years ago

  • Target version changed from 2.10.19 to 2.10.20
Actions #24

Updated by Vincent MEMBRÉ over 5 years ago

  • Target version changed from 2.10.20 to 2.11.18
Actions #25

Updated by Jonathan CLARKE over 5 years ago

  • Status changed from New to In progress
  • Assignee set to Jonathan CLARKE
Actions #26

Updated by Nicolas CHARLES over 5 years ago

The folder is normally defined by the environment variable TMPDIR

Actions #27

Updated by Jonathan CLARKE over 5 years ago

  • Status changed from In progress to Pending technical review
  • Assignee changed from Jonathan CLARKE to Matthieu CERDA
  • Pull Request set to https://github.com/Normation/rudder-packages/pull/868
Actions #28

Updated by Jonathan CLARKE over 5 years ago

This happened because jetty's init script looks for a variable called TMPDIR, and if not defined it uses "/tmp" by default.

I've adapted it to usr /var/rudder/tmp/jetty instead.

Actions #29

Updated by Jonathan CLARKE over 5 years ago

  • Status changed from Pending technical review to Pending release
  • % Done changed from 0 to 100
Actions #30

Updated by Vincent MEMBRÉ over 5 years ago

  • Status changed from Pending release to Released

This bug has been fixed in Rudder 2.11.18, 3.0.13, 3.1.6 and 3.2.0 which were released today.

Actions

Also available in: Atom PDF