Bug #15858
closed
Computing dynamic groups is very memory intensive, and can lead to FGC or OOM
Description
When dynamic groups are updated, it uses a lot of memory, and can completly prevent a policy generation to finish
We need to find a way to make it less memory consuming
Note: my tests uses a lot of regex and and/or in software, it might be a cause of issue
Files
| Screenshot from 2019-10-04 13-48-05.png (890 KB) Screenshot from 2019-10-04 13-48-05.png | |||
| without.png (1.56 MB) without.png | |||
| with.png (1.55 MB) with.png |
Updated by Nicolas CHARLES over 4 years ago
AgentTypes.scala:256 com.normation.inventory.domain.AgentInfoSerialisation$.parseJson(String, Option) 1585 2021260 847048904 is a strong contender in memory usage
Updated by Nicolas CHARLES over 4 years ago
We could use Circe or Spray-Json or Jsoniter to lower memory footprint (see https://circe.github.io/circe/performance.html , https://github.com/plokhotnyuk/jsoniter-scala)
Updated by Nicolas CHARLES over 4 years ago
- Status changed from New to In progress
- Assignee set to Nicolas CHARLES
Updated by Nicolas CHARLES over 4 years ago
- Status changed from In progress to Pending technical review
- Assignee changed from Nicolas CHARLES to François ARMAND
- Pull Request set to https://github.com/Normation/rudder/pull/2504
Updated by Nicolas CHARLES over 4 years ago
- Category deleted (
Performance and scalability) - Target version deleted (
5.0.15) - Name check deleted (
To do)
We could do much better than that:
process returns a Seq[Nodenfo], by converting the result of the QueryAndCheckNodeId, and converting the NodeEntry, InventoryEntry and Machine to a NodeInfo; but we are throwing it all away to keep only the id in computeDynGroup, so we could make a new method which would return only the NodeIds, and save a lot of hassle
Updated by Nicolas CHARLES over 4 years ago
- Category set to Performance and scalability
Updated by Nicolas CHARLES over 4 years ago
- File without.png without.png added
- File with.png with.png added
ok, i don't have the finished computed part without, because the close tabs is so f***ing close to selecting the tab itself, but screenshot with and without the change
Note how with the change, only ldap uses memory, will without, the parsing takes half of allocated memory
So it tooks 1.5GB to compute dynamic group with the change, and more than 3.5 GB (it wasn't finished) without the change
Updated by Nicolas CHARLES over 4 years ago
- Status changed from Pending technical review to Pending release
Applied in changeset rudder|e1d98277b7d062bb0cf05b917cd5ad23a48a1464.
Updated by Nicolas CHARLES over 4 years ago
Applied in changeset rudder|e1d98277b7d062bb0cf05b917cd5ad23a48a1464.
Updated by Nicolas CHARLES over 4 years ago
Applied in changeset rudder|e455168480f4938c01932cbb37b12c9f5638071d.
Updated by Alexis Mousset over 4 years ago
- Name check changed from To do to Reviewed
Updated by Nicolas CHARLES over 4 years ago
- Fix check changed from To do to Checked
Updated by Vincent MEMBRÉ over 4 years ago
- Status changed from Pending release to Released
This bug has been fixed in Rudder 5.0.15 which was released today.