Actions
Bug #15675
closedLeak in Cache of Node Compliance and NodeInfo and perfs improvement
Status:
Released
Priority:
N/A
Assignee:
Category:
Performance and scalability
Target version:
Pull Request:
Severity:
UX impact:
User visibility:
Effort required:
Priority:
0
Name check:
Reviewed
Fix check:
Checked
Regression:
Description
There's a leak in Node Compliance cache - leaving Rudder alone for a night caused 51 millions of object in the map of CachedReportingServiceImpl, using a wooping 2 Go of RAM (the installation is REALLY large)
There's one also in the LocalNideInfoCache
It seems that mapValues references the initial map, causing it not to be GCable
Updated by Nicolas CHARLES about 5 years ago
- Status changed from New to In progress
- Assignee set to Nicolas CHARLES
Updated by Nicolas CHARLES about 5 years ago
Work in progess here: https://github.com/ncharles/rudder/commit/00a5a85c3b64fe15ffae3c260f01a33434cf268c
Updated by François ARMAND about 5 years ago
The problem is likely with `mapValues`:
object CountMapMemory3 { def bigArray = new Array(10*1024*1024) def smallArray = new Array(1) def printMem(s: String) = { Runtime.getRuntime.gc() println(s"${s}: " + Runtime.getRuntime().freeMemory()/1024) } def main(args: Array[String]): Unit = { def res() = { printMem("in res 0") val m = Map(1 -> bigArray, 2 -> bigArray, 3 -> bigArray) val m2 = m.map { case (k,v) => (k,smallArray) } printMem("in res 1") m2 } printMem("start") val map1 = res() printMem("end") } }
prints:
start: 231530 in res 0: 241548 in res 1: 120031 end: 242940
If we replace the map
line with:
val m2 = m.mapValues(_ => smallArray)
It prints:
start: 231546 in res 0: 241545 in res 1: 120003 end: 120092
Updated by Vincent MEMBRÉ about 5 years ago
- Target version changed from 5.0.14 to 5.0.15
Updated by Nicolas CHARLES almost 5 years ago
Updated by Vincent MEMBRÉ almost 5 years ago
- Target version changed from 5.0.15 to 5.0.16
Updated by Nicolas CHARLES almost 5 years ago
- Related to Bug #16467: A lot of data is computed and stored in Policy object, that is either never used, or used only once added
Updated by Nicolas CHARLES almost 5 years ago
- Related to Bug #16468: Computation of ComplianceLevel generates too many objects added
Updated by Nicolas CHARLES almost 5 years ago
- Related to Bug #16513: JVM GC cannot clean objects in scope in a for { } yield {} even if they are not referenced anymore added
Updated by Alexis Mousset almost 5 years ago
- Target version changed from 5.0.16 to 5.0.17
Updated by Nicolas CHARLES over 4 years ago
- Status changed from In progress to Discussion
all issues have been fixed on related ticket, we can close it
Updated by Nicolas CHARLES over 4 years ago
- Status changed from Discussion to New
Updated by Nicolas CHARLES over 4 years ago
- Status changed from New to Pending release
ok, marking it released as subticket are released
Updated by François ARMAND over 4 years ago
- Fix check changed from To do to Checked
Updated by Alexis Mousset over 4 years ago
- Name check changed from To do to Reviewed
Updated by Vincent MEMBRÉ over 4 years ago
This bug has been fixed in Rudder 5.0.17 and 6.0.5 which were released today.
Updated by Vincent MEMBRÉ over 4 years ago
- Related to Architecture #15678: Remove Map#mapValues because of memory not freed added
Updated by Nicolas CHARLES over 4 years ago
- Status changed from Pending release to Released
Actions