Project

General

Profile

Bug #12702

Method copyResourceFile is quite inefficient

Added by Nicolas CHARLES 7 months ago. Updated 5 months ago.

Status:
Released
Priority:
N/A
Category:
Performance and scalability
Target version:
Severity:
Minor - inconvenience | misleading | easy workaround
User visibility:
Operational - other Techniques | Technique editor | Rudder settings
Effort required:
Priority:
32

Description

Method copyResourceFile is used to write static files (which don't change from what is on FS)
On my test system, with 1500 nodes, it takes about 10s out of 42s of writing file policy during a generation. Given the low scope of this function, there is probably way to do better

  /**
   * Copy a resource file from a technique to the node promises directory
   */
  private[this] def copyResourceFile

the only ressources I have are

TechniqueFile(TechniqueResourceIdByName(common/1.0,properties.cf),common/1.0/properties.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-stdlib.cf),common/1.0/rudder-stdlib.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,minicurl),common/utilities/minicurl,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-stdlib.cf),common/1.0/rudder-stdlib.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,minicurl),common/utilities/minicurl,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-agent-community-cron),common/cron/rudder-agent-community-cron,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,environment-variables.cf),common/1.0/environment-variables.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-agent-nova-cron),common/cron/rudder-agent-nova-cron,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-stdlib.cf),common/1.0/rudder-stdlib.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,minicurl),common/utilities/minicurl,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-agent-community-cron),common/cron/rudder-agent-community-cron,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,environment-variables.cf),common/1.0/environment-variables.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-agent-nova-cron),common/cron/rudder-agent-nova-cron,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,check-zypper.cf),common/1.0/check-zypper.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,properties.cf),common/1.0/properties.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,minicurl),common/utilities/minicurl,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-agent-community-cron),common/cron/rudder-agent-community-cron,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-agent-nova-cron),common/cron/rudder-agent-nova-cron,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,check-zypper.cf),common/1.0/check-zypper.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,properties.cf),common/1.0/properties.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-stdlib.cf),common/1.0/rudder-stdlib.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,minicurl),common/utilities/minicurl,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-agent-community-cron),common/cron/rudder-agent-community-cron,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,environment-variables.cf),common/1.0/environment-variables.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-agent-nova-cron),common/cron/rudder-agent-nova-cron,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,check-zypper.cf),common/1.0/check-zypper.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,properties.cf),common/1.0/properties.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,environment-variables.cf),common/1.0/environment-variables.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-stdlib.cf),common/1.0/rudder-stdlib.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-agent-nova-cron),common/cron/rudder-agent-nova-cron,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,check-zypper.cf),common/1.0/check-zypper.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,properties.cf),common/1.0/properties.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-agent-community-cron),common/cron/rudder-agent-community-cron,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-stdlib.cf),common/1.0/rudder-stdlib.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,environment-variables.cf),common/1.0/environment-variables.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-agent-community-cron),common/cron/rudder-agent-community-cron,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,environment-variables.cf),common/1.0/environment-variables.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-agent-nova-cron),common/cron/rudder-agent-nova-cron,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,check-zypper.cf),common/1.0/check-zypper.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-stdlib.cf),common/1.0/rudder-stdlib.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,minicurl),common/utilities/minicurl,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-agent-community-cron),common/cron/rudder-agent-community-cron,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,environment-variables.cf),common/1.0/environment-variables.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-agent-nova-cron),common/cron/rudder-agent-nova-cron,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,check-zypper.cf),common/1.0/check-zypper.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,properties.cf),common/1.0/properties.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,environment-variables.cf),common/1.0/environment-variables.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-agent-nova-cron),common/cron/rudder-agent-nova-cron,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,check-zypper.cf),common/1.0/check-zypper.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,properties.cf),common/1.0/properties.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-stdlib.cf),common/1.0/rudder-stdlib.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,properties.cf),common/1.0/properties.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-agent-nova-cron),common/cron/rudder-agent-nova-cron,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,check-zypper.cf),common/1.0/check-zypper.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,properties.cf),common/1.0/properties.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,minicurl),common/utilities/minicurl,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-stdlib.cf),common/1.0/rudder-stdlib.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,check-zypper.cf),common/1.0/check-zypper.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,properties.cf),common/1.0/properties.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,minicurl),common/utilities/minicurl,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-stdlib.cf),common/1.0/rudder-stdlib.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,minicurl),common/utilities/minicurl,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-agent-community-cron),common/cron/rudder-agent-community-cron,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,environment-variables.cf),common/1.0/environment-variables.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-agent-nova-cron),common/cron/rudder-agent-nova-cron,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,check-zypper.cf),common/1.0/check-zypper.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,properties.cf),common/1.0/properties.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,minicurl),common/utilities/minicurl,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-agent-community-cron),common/cron/rudder-agent-community-cron,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-stdlib.cf),common/1.0/rudder-stdlib.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,minicurl),common/utilities/minicurl,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-stdlib.cf),common/1.0/rudder-stdlib.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-stdlib.cf),common/1.0/rudder-stdlib.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,minicurl),common/utilities/minicurl,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-agent-community-cron),common/cron/rudder-agent-community-cron,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,environment-variables.cf),common/1.0/environment-variables.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-agent-nova-cron),common/cron/rudder-agent-nova-cron,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,check-zypper.cf),common/1.0/check-zypper.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,properties.cf),common/1.0/properties.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,minicurl),common/utilities/minicurl,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-agent-community-cron),common/cron/rudder-agent-community-cron,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,environment-variables.cf),common/1.0/environment-variables.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-agent-nova-cron),common/cron/rudder-agent-nova-cron,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,check-zypper.cf),common/1.0/check-zypper.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,properties.cf),common/1.0/properties.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-stdlib.cf),common/1.0/rudder-stdlib.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,minicurl),common/utilities/minicurl,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-agent-community-cron),common/cron/rudder-agent-community-cron,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,environment-variables.cf),common/1.0/environment-variables.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-agent-nova-cron),common/cron/rudder-agent-nova-cron,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,check-zypper.cf),common/1.0/check-zypper.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,properties.cf),common/1.0/properties.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,minicurl),common/utilities/minicurl,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-agent-community-cron),common/cron/rudder-agent-community-cron,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-agent-community-cron),common/cron/rudder-agent-community-cron,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-stdlib.cf),common/1.0/rudder-stdlib.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,minicurl),common/utilities/minicurl,false)

so about 7 or 8 files, of 1500 nodes, so less than 15000 written file in total

Targetting to 4.1 as it is quite impacting for user with large installation

Associated revisions

Revision 3ace7e96 (diff)
Added by Nicolas CHARLES 6 months ago

Fixes #12702: Method copyResourceFile is quite inefficient

History

#1 Updated by Nicolas CHARLES 7 months ago

Note:
It's ALWAYS the same files, so we could look it up only once
We could even hardlink or somthing

#2 Updated by Nicolas CHARLES 7 months ago

Ok, it seems the issue is that we read each time the technique content, and that is very expensive:

Tests metrics for resources creations

Creating the resource file, accessing technique repository, no content written in resource file: 12503 ms

    val destination = {
      val out = reportIdToReplace match {
        case None     => file.outPath
        case Some(id) => file.outPath.replaceAll(Policy.TAG_OF_RUDDER_MULTI_POLICY, id.getRudderUniqueId)
      }
      new File(rulePath+"/"+out)
    }

techniqueRepository.getFileContent(file.id) { optStream =>
      optStream match {
        case None => Failure(s"Can not open the technique resource file ${file.id} for reading")
        case Some(s) =>
          try {
            //FileUtils.copyInputStreamToFile(s, destination)
            Full(destination.getAbsolutePath)

Creating the resource file, not accessing the technique repository at all, no content in it: 2757 ms

    val destination = {
      val out = reportIdToReplace match {
        case None     => file.outPath
        case Some(id) => file.outPath.replaceAll(Policy.TAG_OF_RUDDER_MULTI_POLICY, id.getRudderUniqueId)
      }
      new File(rulePath+"/"+out)
    }
    Full(destination.getAbsolutePath)
    /*

[2018-05-25 21:42:43] DEBUG com.normation.rudder.services.policies.PromiseGenerationServiceImpl - Write node configurations : 2757 ms

#4 Updated by Benoît PECCATTE 7 months ago

  • Severity set to Minor - inconvenience | misleading | easy workaround
  • User visibility changed from Infrequent - complex configurations | third party integrations to Operational - other Techniques | Technique editor | Rudder settings
  • Priority changed from 0 to 32

#5 Updated by Nicolas CHARLES 6 months ago

  • Status changed from New to In progress

#6 Updated by Nicolas CHARLES 6 months ago

  • Status changed from In progress to Pending technical review
  • Assignee changed from Nicolas CHARLES to François ARMAND
  • Pull Request set to https://github.com/Normation/rudder/pull/1973

#7 Updated by Nicolas CHARLES 6 months ago

It's roughtly 20-30% performance increase - depending on type of Directives (Directives based on Techniques created by technique editor benefit more from this change)

#8 Updated by Normation Quality Assistant 6 months ago

  • Status changed from Pending technical review to Discussion
  • Assignee changed from François ARMAND to Nicolas CHARLES

#9 Updated by Nicolas CHARLES 6 months ago

  • Status changed from Discussion to Pending technical review
  • Assignee changed from Nicolas CHARLES to François ARMAND

#10 Updated by Normation Quality Assistant 6 months ago

  • Assignee changed from François ARMAND to Nicolas CHARLES

#11 Updated by Nicolas CHARLES 6 months ago

  • Status changed from Pending technical review to Pending release

#12 Updated by Vincent MEMBRÉ 5 months ago

  • Status changed from Pending release to Released

This bug has been fixed in Rudder 4.1.13, 4.2.7 and 4.3.3 which were released today.

Also available in: Atom PDF