Project

General

Profile

Actions

Bug #15792

closed

When writing policies, we replace the RudderUniqueId with replaceAll which is twice as slow as StringUtils.replace

Added by Nicolas CHARLES over 4 years ago. Updated over 4 years ago.

Status:
Released
Priority:
N/A
Category:
Performance and scalability
Target version:
Severity:
UX impact:
User visibility:
Effort required:
Priority:
0
Name check:
Reviewed
Fix check:
Checked
Regression:

Description

Generation measurement shows that 110 s (probably threaded on 10 thread, so in final 10s) are used in replaceAll to replace the RudderUniqueId in template and in path

replaceAll uses internally a regex to replace all, while StringUtils.replace does not ( https://stackoverflow.com/questions/5407592/replace-all-occurrences-of-substring-in-a-string-which-is-more-efficient-in-ja )

Since we are replacing an hardcoded text, we can swith to StringUtils

following tests shows that StringUtils is twice as fast


import org.apache.commons.lang.StringUtils

  val TAG_OF_RUDDER_MULTI_POLICY = "RudderUniqueId" 
  //val p = java.util.regex.Pattern.compile(TAG_OF_RUDDER_MULTI_POLICY)

  def randomString(length: Int) = {
    val r = new scala.util.Random
    val sb = new StringBuilder
    for (i <- 1 to 2*length) {
      sb.append(r.nextPrintableChar)
    }
    sb.toString
  }

  // create 10000 string
  val textToReplace= (1 to 10000).toSeq.map { case x =>
           if (x%2 == 0) {
             randomString(x)
           } else {
             randomString(x).concat(TAG_OF_RUDDER_MULTI_POLICY).concat(randomString(x))
           }
  }

      val start3 = System.currentTimeMillis()
      val replacedText3 = textToReplace.map( x => StringUtils.replace(x, TAG_OF_RUDDER_MULTI_POLICY, "coucou"))
      val end3 = System.currentTimeMillis()
      println(s"Replacing with StringUtil for ${replacedText3.length} element took ${end3-start3}")

      val start = System.currentTimeMillis()
      val replacedText = textToReplace.map( x => x.replaceAll(TAG_OF_RUDDER_MULTI_POLICY, "coucou"))
      val end = System.currentTimeMillis()

      println(s"Replacing with replaceAll for ${replacedText.length} element took ${end-start}")

outputs

Replacing with StringUtil for 10000 element took 639
Replacing with replaceAll for 10000 element took 1146

I tried also with a precompile pattern, but perfs were similar as replaceAll

Actions #1

Updated by Nicolas CHARLES over 4 years ago

  • Status changed from New to In progress
  • Assignee set to Nicolas CHARLES
Actions #2

Updated by Nicolas CHARLES over 4 years ago

  • Status changed from In progress to Pending technical review
  • Assignee changed from Nicolas CHARLES to François ARMAND
  • Pull Request set to https://github.com/Normation/rudder/pull/2482
Actions #3

Updated by Nicolas CHARLES over 4 years ago

  • Status changed from Pending technical review to Pending release
Actions #4

Updated by François ARMAND over 4 years ago

  • Fix check changed from To do to Checked
Actions #5

Updated by Alexis Mousset over 4 years ago

  • Name check changed from To do to Reviewed
Actions #6

Updated by Vincent MEMBRÉ over 4 years ago

  • Status changed from Pending release to Released

This bug has been fixed in Rudder 5.0.14 which was released today.

Actions

Also available in: Atom PDF