Project

General

Profile

Actions
Copy link

Bug #12702

closed

Method copyResourceFile is quite inefficient

Added by Nicolas CHARLES over 6 years ago. Updated over 6 years ago.

Status:
Released
Priority:
N/A
Assignee:
Nicolas CHARLES
Category:
Performance and scalability
Target version:
4.1.13
Pull Request:
https://github.com/Normation/rudder/p...
Severity:
Minor - inconvenience | misleading | easy workaround
UX impact:
User visibility:
Operational - other Techniques | Technique editor | Rudder settings
Effort required:
Priority:
32
Name check:
Fix check:
Regression:

Description

Method copyResourceFile is used to write static files (which don't change from what is on FS)
On my test system, with 1500 nodes, it takes about 10s out of 42s of writing file policy during a generation. Given the low scope of this function, there is probably way to do better

  /**
   * Copy a resource file from a technique to the node promises directory
   */
  private[this] def copyResourceFile

the only ressources I have are

TechniqueFile(TechniqueResourceIdByName(common/1.0,properties.cf),common/1.0/properties.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-stdlib.cf),common/1.0/rudder-stdlib.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,minicurl),common/utilities/minicurl,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-stdlib.cf),common/1.0/rudder-stdlib.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,minicurl),common/utilities/minicurl,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-agent-community-cron),common/cron/rudder-agent-community-cron,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,environment-variables.cf),common/1.0/environment-variables.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-agent-nova-cron),common/cron/rudder-agent-nova-cron,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-stdlib.cf),common/1.0/rudder-stdlib.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,minicurl),common/utilities/minicurl,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-agent-community-cron),common/cron/rudder-agent-community-cron,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,environment-variables.cf),common/1.0/environment-variables.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-agent-nova-cron),common/cron/rudder-agent-nova-cron,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,check-zypper.cf),common/1.0/check-zypper.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,properties.cf),common/1.0/properties.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,minicurl),common/utilities/minicurl,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-agent-community-cron),common/cron/rudder-agent-community-cron,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-agent-nova-cron),common/cron/rudder-agent-nova-cron,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,check-zypper.cf),common/1.0/check-zypper.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,properties.cf),common/1.0/properties.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-stdlib.cf),common/1.0/rudder-stdlib.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,minicurl),common/utilities/minicurl,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-agent-community-cron),common/cron/rudder-agent-community-cron,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,environment-variables.cf),common/1.0/environment-variables.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-agent-nova-cron),common/cron/rudder-agent-nova-cron,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,check-zypper.cf),common/1.0/check-zypper.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,properties.cf),common/1.0/properties.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,environment-variables.cf),common/1.0/environment-variables.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-stdlib.cf),common/1.0/rudder-stdlib.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-agent-nova-cron),common/cron/rudder-agent-nova-cron,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,check-zypper.cf),common/1.0/check-zypper.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,properties.cf),common/1.0/properties.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-agent-community-cron),common/cron/rudder-agent-community-cron,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-stdlib.cf),common/1.0/rudder-stdlib.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,environment-variables.cf),common/1.0/environment-variables.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-agent-community-cron),common/cron/rudder-agent-community-cron,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,environment-variables.cf),common/1.0/environment-variables.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-agent-nova-cron),common/cron/rudder-agent-nova-cron,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,check-zypper.cf),common/1.0/check-zypper.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-stdlib.cf),common/1.0/rudder-stdlib.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,minicurl),common/utilities/minicurl,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-agent-community-cron),common/cron/rudder-agent-community-cron,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,environment-variables.cf),common/1.0/environment-variables.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-agent-nova-cron),common/cron/rudder-agent-nova-cron,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,check-zypper.cf),common/1.0/check-zypper.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,properties.cf),common/1.0/properties.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,environment-variables.cf),common/1.0/environment-variables.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-agent-nova-cron),common/cron/rudder-agent-nova-cron,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,check-zypper.cf),common/1.0/check-zypper.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,properties.cf),common/1.0/properties.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-stdlib.cf),common/1.0/rudder-stdlib.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,properties.cf),common/1.0/properties.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-agent-nova-cron),common/cron/rudder-agent-nova-cron,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,check-zypper.cf),common/1.0/check-zypper.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,properties.cf),common/1.0/properties.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,minicurl),common/utilities/minicurl,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-stdlib.cf),common/1.0/rudder-stdlib.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,check-zypper.cf),common/1.0/check-zypper.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,properties.cf),common/1.0/properties.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,minicurl),common/utilities/minicurl,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-stdlib.cf),common/1.0/rudder-stdlib.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,minicurl),common/utilities/minicurl,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-agent-community-cron),common/cron/rudder-agent-community-cron,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,environment-variables.cf),common/1.0/environment-variables.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-agent-nova-cron),common/cron/rudder-agent-nova-cron,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,check-zypper.cf),common/1.0/check-zypper.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,properties.cf),common/1.0/properties.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,minicurl),common/utilities/minicurl,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-agent-community-cron),common/cron/rudder-agent-community-cron,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-stdlib.cf),common/1.0/rudder-stdlib.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,minicurl),common/utilities/minicurl,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-stdlib.cf),common/1.0/rudder-stdlib.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-stdlib.cf),common/1.0/rudder-stdlib.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,minicurl),common/utilities/minicurl,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-agent-community-cron),common/cron/rudder-agent-community-cron,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,environment-variables.cf),common/1.0/environment-variables.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-agent-nova-cron),common/cron/rudder-agent-nova-cron,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,check-zypper.cf),common/1.0/check-zypper.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,properties.cf),common/1.0/properties.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,minicurl),common/utilities/minicurl,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-agent-community-cron),common/cron/rudder-agent-community-cron,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,environment-variables.cf),common/1.0/environment-variables.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-agent-nova-cron),common/cron/rudder-agent-nova-cron,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,check-zypper.cf),common/1.0/check-zypper.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,properties.cf),common/1.0/properties.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-stdlib.cf),common/1.0/rudder-stdlib.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,minicurl),common/utilities/minicurl,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-agent-community-cron),common/cron/rudder-agent-community-cron,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,environment-variables.cf),common/1.0/environment-variables.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-agent-nova-cron),common/cron/rudder-agent-nova-cron,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,check-zypper.cf),common/1.0/check-zypper.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,properties.cf),common/1.0/properties.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,minicurl),common/utilities/minicurl,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-agent-community-cron),common/cron/rudder-agent-community-cron,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-agent-community-cron),common/cron/rudder-agent-community-cron,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-stdlib.cf),common/1.0/rudder-stdlib.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,minicurl),common/utilities/minicurl,false)

so about 7 or 8 files, of 1500 nodes, so less than 15000 written file in total

Targetting to 4.1 as it is quite impacting for user with large installation

Actions
Copy link

Also available in: Atom PDF