Project

General

Profile

Actions

Bug #16256

closed

Deadlock on compliance computing

Added by Nicolas CHARLES almost 5 years ago. Updated almost 5 years ago.

Status:
Released
Priority:
N/A
Category:
Web - Compliance & node report
Target version:
Severity:
UX impact:
User visibility:
Effort required:
Priority:
0
Name check:
Reviewed
Fix check:
Checked
Regression:

Description

I generated policies, accepted nodes and the same time, and went to home page several time
after a while, compliance didn't show up...

running jstack show that a lot of thread are BLOCKED (see attachement)


Files

thread_dump (212 KB) thread_dump Nicolas CHARLES, 2019-11-22 23:48
API_casse (497 KB) API_casse Nicolas CHARLES, 2019-11-25 11:08

Related issues 1 (0 open1 closed)

Related to Rudder - Bug #16200: Missing timing info in compliance computation logsReleasedFrançois ARMANDActions
Actions #1

Updated by Nicolas CHARLES almost 5 years ago

I had a query 'idle in transaction' in postgresql: select nodeid, nodeconfigid, begindate, configuration from nodeconfigurations where enddate is null;
this was because of the load script

Actions #2

Updated by Nicolas CHARLES almost 5 years ago

There is also an issue on the policy generation: calling policy/regenerate every 30 minutes causes it to lock after first generation after midnight: every subsequent call results in

<title>502 Proxy Error</title>
</head><body>
<h1>Proxy Error</h1>
<p>The proxy server received an invalid
response from an upstream server.<br />
The proxy server could not handle the request <em><a href="/rudder/api/latest/system/regenerate/policies">POST&nbsp;/rudder/api/latest/system/regenerate/policies</a></em>.<p>
Reason: <strong>Error reading from remote server</strong></p></p>
</body></html>

compliance doesn't show up either

thread dump is attached

Actions #3

Updated by Nicolas CHARLES almost 5 years ago

  • Related to Bug #16200: Missing timing info in compliance computation logs added
Actions #4

Updated by Nicolas CHARLES almost 5 years ago

  • Status changed from New to In progress
  • Assignee set to Nicolas CHARLES
Actions #5

Updated by Nicolas CHARLES almost 5 years ago

  • Status changed from In progress to Pending technical review
  • Assignee changed from Nicolas CHARLES to François ARMAND
  • Pull Request set to https://github.com/Normation/rudder/pull/2632
Actions #6

Updated by Nicolas CHARLES almost 5 years ago

this does have a negative impact on perfs, but make it works.

Actions #7

Updated by Nicolas CHARLES almost 5 years ago

  • Status changed from Pending technical review to Pending release
Actions #8

Updated by Nicolas CHARLES almost 5 years ago

  • Fix check changed from To do to Checked

tested, after 2 days of non stop regeneration and group update, there is no deadlock

Actions #9

Updated by Alexis Mousset almost 5 years ago

  • Subject changed from Deadlock on compliance on Rudder 5.0 to Deadlock on compliance computing
  • Name check changed from To do to Reviewed
Actions #10

Updated by Vincent MEMBRÉ almost 5 years ago

  • Status changed from Pending release to Released

This bug has been fixed in Rudder 5.0.15 which was released today.

Actions

Also available in: Atom PDF