Project

General

Profile

Actions

Bug #19161

open

Rudder agent consumes 100% CPU when copying a file to a file system with no space left

Added by Lars Koenen about 3 years ago. Updated about 1 month ago.

Status:
New
Priority:
N/A
Assignee:
-
Category:
Agent
Target version:
Severity:
Critical - prevents main use of Rudder | no workaround | data loss | security
UX impact:
User visibility:
Operational - other Techniques | Rudder settings | Plugins
Effort required:
Priority:
0
Name check:
To do
Fix check:
To do
Regression:
No

Description

While testing with Rudder Version 6.2.4 I noticed the following problem.

I used File download (Rudder server) to copy a large file (~1GB) to a node (each time slightly modified by appending a single character).
The files in /var/rudder/modified-files/ eventually filled up the filesystem. (as expected)
When I then try again to copy the file with rudder to the filed up filesystem of the node, the execution of rudder agent run takes very long (in my case ~1000 seconds) and needs 100% CPU.

rudder agent run terminates at some point, but there are still cf-agent processes left over that continue to use 100% CPU.

2021-04-18T15:58:10+00:00    error: Failed to write to destination file (write: No space left on device)
2021-04-18T15:58:10+00:00    error: Local disk write failed copying '<SERVER_IP>:/var/rudder/configuration-repository/shared-files/file2.txt' to '/root/file2.txt.cfnew'
2021-04-18T16:13:09+00:00    error: Was not able to copy '/var/rudder/configuration-repository/shared-files/file2.txt' to '/root/file2.txt'
907.43s E| error         copyFile                  Copy file                 /root/file2.txt    The content or permissions of the file(s) could not have been repaired (file file2.txt not found?)
  0.02s E| n/a           copyFile                  Post-modification hook    /root/file2.txt    No post-hook command for copy of file2.txt to /root/file2.txt was defined, not executing
2021-04-18T16:13:10+00:00    error: SSL read after retries: underlying network error ()
Actions #1

Updated by Lars Koenen about 3 years ago

  • Subject changed from Rudder agent consumes 100% CPU when copying a file to a full file system to Rudder agent consumes 100% CPU when copying a file to a file system with no space left
Actions #2

Updated by Alexis Mousset about 3 years ago

  • Severity set to Critical - prevents main use of Rudder | no workaround | data loss | security
  • User visibility set to Operational - other Techniques | Rudder settings | Plugins
  • Priority changed from 0 to 76

Thanks for the detailed report, we'll try to reproduce the problem.

Actions #3

Updated by Vincent MEMBRÉ about 3 years ago

  • Target version changed from 6.2.6 to 6.2.7
Actions #4

Updated by Vincent MEMBRÉ almost 3 years ago

  • Target version changed from 6.2.7 to 6.2.8
Actions #5

Updated by Vincent MEMBRÉ almost 3 years ago

  • Target version changed from 6.2.8 to 6.2.9
  • Priority changed from 76 to 74
Actions #6

Updated by Vincent MEMBRÉ almost 3 years ago

  • Target version changed from 6.2.9 to 6.2.10
  • Priority changed from 74 to 73
Actions #7

Updated by Vincent MEMBRÉ over 2 years ago

  • Target version changed from 6.2.10 to 6.2.11
  • Priority changed from 73 to 72
Actions #8

Updated by Vincent MEMBRÉ over 2 years ago

  • Target version changed from 6.2.11 to 6.2.12
  • Priority changed from 72 to 69
Actions #9

Updated by Vincent MEMBRÉ over 2 years ago

  • Target version changed from 6.2.12 to 6.2.13
  • Priority changed from 69 to 68
Actions #10

Updated by Alexis Mousset about 2 years ago

  • Priority changed from 68 to 66

I did not not managed to reproduce it when I tried, but it looks like a protocol error on agent side.

Actions #11

Updated by Alexis Mousset about 2 years ago

  • Priority changed from 66 to 65
Actions #12

Updated by Vincent MEMBRÉ about 2 years ago

  • Target version changed from 6.2.13 to 6.2.14
  • Priority changed from 65 to 63
Actions #13

Updated by Vincent MEMBRÉ almost 2 years ago

  • Target version changed from 6.2.14 to 6.2.15
Actions #14

Updated by Vincent MEMBRÉ almost 2 years ago

  • Target version changed from 6.2.15 to 6.2.16
Actions #15

Updated by Alexis Mousset over 1 year ago

  • Target version changed from 6.2.16 to 6.2.17
Actions #16

Updated by Vincent MEMBRÉ over 1 year ago

  • Target version changed from 6.2.17 to 997
  • Priority changed from 63 to 0
Actions #17

Updated by Vincent MEMBRÉ over 1 year ago

  • Target version changed from 997 to 6.2.18
Actions #18

Updated by Vincent MEMBRÉ over 1 year ago

  • Target version changed from 6.2.18 to 6.2.19
Actions #19

Updated by Vincent MEMBRÉ over 1 year ago

  • Target version changed from 6.2.19 to 6.2.20
Actions #20

Updated by Vincent MEMBRÉ over 1 year ago

  • Target version changed from 6.2.20 to old 6.2 issues to relocate
Actions #21

Updated by Alexis Mousset 9 months ago

  • Target version changed from old 6.2 issues to relocate to 7.2.10
Actions #22

Updated by Alexis Mousset 9 months ago

  • Target version changed from 7.2.10 to 7.2.11
Actions #23

Updated by Vincent MEMBRÉ 7 months ago

  • Target version changed from 7.2.11 to 1046
Actions #24

Updated by Benoît PECCATTE 7 months ago

  • Regression set to No

Tried on rudder 7.3 centos 8, i was unable to reproduce, I just get an error on copy and a "write: No space left on device" message

Actions #25

Updated by Alexis Mousset 6 months ago

  • Target version changed from 1046 to 7.3.8
Actions #26

Updated by Vincent MEMBRÉ 6 months ago

  • Target version changed from 7.3.8 to 7.3.9
Actions #27

Updated by Vincent MEMBRÉ 6 months ago

  • Target version changed from 7.3.9 to 7.3.10
Actions #28

Updated by Vincent MEMBRÉ 4 months ago

  • Target version changed from 7.3.10 to 7.3.11
Actions #29

Updated by Vincent MEMBRÉ 3 months ago

  • Target version changed from 7.3.11 to 7.3.12
Actions #30

Updated by Vincent MEMBRÉ about 2 months ago

  • Target version changed from 7.3.12 to 7.3.13
Actions #31

Updated by Vincent MEMBRÉ about 1 month ago

  • Target version changed from 7.3.13 to 7.3.14
Actions

Also available in: Atom PDF