Project

General

Profile

Bug #15792

When writing policies, we replace the RudderUniqueId with replaceAll which is twice as slow as StringUtils.replace

Added by Nicolas CHARLES about 2 months ago. Updated about 1 month ago.

Status:
Released
Priority:
N/A
Category:
Performance and scalability
Target version:
Severity:
User visibility:
Effort required:
Priority:
0

Description

Generation measurement shows that 110 s (probably threaded on 10 thread, so in final 10s) are used in replaceAll to replace the RudderUniqueId in template and in path

replaceAll uses internally a regex to replace all, while StringUtils.replace does not ( https://stackoverflow.com/questions/5407592/replace-all-occurrences-of-substring-in-a-string-which-is-more-efficient-in-ja )

Since we are replacing an hardcoded text, we can swith to StringUtils

following tests shows that StringUtils is twice as fast


import org.apache.commons.lang.StringUtils

  val TAG_OF_RUDDER_MULTI_POLICY = "RudderUniqueId" 
  //val p = java.util.regex.Pattern.compile(TAG_OF_RUDDER_MULTI_POLICY)

  def randomString(length: Int) = {
    val r = new scala.util.Random
    val sb = new StringBuilder
    for (i <- 1 to 2*length) {
      sb.append(r.nextPrintableChar)
    }
    sb.toString
  }

  // create 10000 string
  val textToReplace= (1 to 10000).toSeq.map { case x =>
           if (x%2 == 0) {
             randomString(x)
           } else {
             randomString(x).concat(TAG_OF_RUDDER_MULTI_POLICY).concat(randomString(x))
           }
  }

      val start3 = System.currentTimeMillis()
      val replacedText3 = textToReplace.map( x => StringUtils.replace(x, TAG_OF_RUDDER_MULTI_POLICY, "coucou"))
      val end3 = System.currentTimeMillis()
      println(s"Replacing with StringUtil for ${replacedText3.length} element took ${end3-start3}")

      val start = System.currentTimeMillis()
      val replacedText = textToReplace.map( x => x.replaceAll(TAG_OF_RUDDER_MULTI_POLICY, "coucou"))
      val end = System.currentTimeMillis()

      println(s"Replacing with replaceAll for ${replacedText.length} element took ${end-start}")

outputs

Replacing with StringUtil for 10000 element took 639
Replacing with replaceAll for 10000 element took 1146

I tried also with a precompile pattern, but perfs were similar as replaceAll

Associated revisions

Revision 40f23bfe (diff)
Added by Nicolas CHARLES about 2 months ago

Fixes #15792: When writing policies, we replace the RudderUniqueId with replaceAll which is twice as slow as StringUtils.replace

History

#1

Updated by Nicolas CHARLES about 2 months ago

  • Status changed from New to In progress
  • Assignee set to Nicolas CHARLES
#2

Updated by Nicolas CHARLES about 2 months ago

  • Status changed from In progress to Pending technical review
  • Assignee changed from Nicolas CHARLES to François ARMAND
  • Pull Request set to https://github.com/Normation/rudder/pull/2482
#3

Updated by Nicolas CHARLES about 2 months ago

  • Status changed from Pending technical review to Pending release
#6

Updated by Vincent MEMBRÉ about 1 month ago

  • Status changed from Pending release to Released

This bug has been fixed in Rudder 5.0.14 which was released today.

Also available in: Atom PDF