Project

General

Profile

Actions
Copy link

Bug #13987

closed

Massive performance penalty in policy generation due to invalid usage of StringTemplate

Added by Nicolas CHARLES over 5 years ago. Updated over 5 years ago.

Status:
Released
Priority:
N/A
Assignee:
François ARMAND
Category:
Performance and scalability
Target version:
4.3.8
Pull Request:
https://github.com/Normation/rudder/p...
Severity:
UX impact:
User visibility:
Effort required:
Priority:
0
Name check:
Fix check:
Regression:

Description

We are using String Template in a non efficient way: for each node, for each template, we are re-tokenizing the template
We should instead use a copy of the tokenized version (as sescribed in https://theantlrguy.atlassian.net/wiki/spaces/ST/pages/1409053/ST+condensed+--+Templates+and+groups)

Perf increase is quite impressive:
Without cache of tokenization

[2018-12-17 17:17:42] INFO  policy.generation - Build target configuration    :       8914 ms
[2018-12-17 17:17:42] INFO  policy.generation - Write node configurations     :      84801 ms
[2018-12-17 17:17:42] INFO  policy.generation - Save expected reports         :        785 ms
[2018-12-17 17:17:42] INFO  policy.generation - Run post generation hooks     :         67 ms
[2018-12-17 17:17:42] INFO  policy.generation - Number of nodes updated       :       1597  
[2018-12-17 17:17:42] INFO  policy.generation - StringTemplate related stats: total time of new StringTemplate     :      54574 ms  
[2018-12-17 17:17:42] INFO  policy.generation - StringTemplate related stats: total time of replacement of var     :        467 ms  
[2018-12-17 17:17:42] INFO  policy.generation - StringTemplate related stats: nbTemaplte    :      60110    
[2018-12-17 17:17:42] INFO  policy.generation - Policy generation completed in:     109967 ms

With cache of tokenization

[2018-12-17 17:12:27] INFO  policy.generation - Build target configuration    :       8114 ms
[2018-12-17 17:12:27] INFO  policy.generation - Write node configurations     :      30503 ms
[2018-12-17 17:12:27] INFO  policy.generation - Save expected reports         :        781 ms
[2018-12-17 17:12:27] INFO  policy.generation - Run post generation hooks     :         91 ms
[2018-12-17 17:12:27] INFO  policy.generation - Number of nodes updated       :       1597  
[2018-12-17 17:12:27] INFO  policy.generation - StringTemplate related stats: total time of new StringTemplate     :        293 ms  
[2018-12-17 17:12:27] INFO  policy.generation - StringTemplate related stats: total time of replacement of var     :        430 ms  
[2018-12-17 17:12:27] INFO  policy.generation - StringTemplate related stats: nbTemaplte    :      60110    
[2018-12-17 17:12:27] INFO  policy.generation - Policy generation completed in:      57362 ms

we can contemplate a *2 perf increase

Files

Download all files
with.png (67.9 KB) with.png with the patch Nicolas CHARLES, 2018-12-17 23:15
without.png (82.7 KB) without.png without the patch Nicolas CHARLES, 2018-12-17 23:15

Related issues 1 (0 open — 1 closed)

Related to Rudder - Bug #14322: Directive parameter values are mixed between directivesReleasedFrançois ARMANDActions
Actions
Copy link

Also available in: Atom PDF