Project

General

Profile

Bug #14991

NullPointerException with hundreds of inventories at once

Added by Nicolas CHARLES over 1 year ago. Updated 7 months ago.

Status:
Released
Priority:
N/A
Category:
Performance and scalability
Target version:
Severity:
Major - prevents use of part of Rudder | no simple workaround
User visibility:
Operational - other Techniques | Rudder settings | Plugins
Effort required:
Small
Priority:
88
Tags:

Description

Workaround

- move inventories from /var/rudder/inventories/accepted-nodes-update or /var/rudder/inventories/incoming to a temp directory;
- restart inventory watcher. From rudder server, do:
- in 5.0, use curl -X POST "http://localhost:8080/endpoint/api/watcher/restart
- in 6.0, use curl -H "X-API-Token: $(cat /var/rudder/run/api-token)" -X POST "https://.../api/latest/inventory/watcher/restart
- move back inventories from temp directory to
/var/rudder/inventories/accepted-nodes-update@ by batch of 100.

Description

With 100s of inventories in incoming folder, i got the following error

java.lang.NullPointerException
       at sun.nio.fs.UnixPath.toUnixPath(UnixPath.java:198)
       at sun.nio.fs.UnixPath.resolve(UnixPath.java:397)
       at sun.nio.fs.UnixPath.resolve(UnixPath.java:43)
       at better.files.FileMonitor.$anonfun$process$1(FileMonitor.scala:33)
       at better.files.FileMonitor.$anonfun$process$1$adapted(FileMonitor.scala:31)
       at scala.collection.Iterator.foreach(Iterator.scala:937)
       at scala.collection.Iterator.foreach$(Iterator.scala:937)
       at scala.collection.AbstractIterator.foreach(Iterator.scala:1425)
       at scala.collection.IterableLike.foreach(IterableLike.scala:70)
       at scala.collection.IterableLike.foreach$(IterableLike.scala:69)
       at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
       at better.files.FileMonitor.process(FileMonitor.scala:31)
       at com.normation.inventory.provisioning.endpoint.Watchers$$anon$1$$anon$2.$anonfun$run$3(InventoryFileWatcher.scala:112)
       at com.normation.inventory.provisioning.endpoint.Watchers$$anon$1$$anon$2.$anonfun$run$3$adapted(InventoryFileWatcher.scala:112)
       at scala.collection.Iterator.foreach(Iterator.scala:937)
       at scala.collection.Iterator.foreach$(Iterator.scala:937)
       at scala.collection.AbstractIterator.foreach(Iterator.scala:1425)
       at com.normation.inventory.provisioning.endpoint.Watchers$$anon$1$$anon$2.$anonfun$run$1(InventoryFileWatcher.scala:112)
       at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
       at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
       at scala.concurrent.package$.blocking(package.scala:142)
       at com.normation.inventory.provisioning.endpoint.Watchers$$anon$1$$anon$2.run(InventoryFileWatcher.scala:112)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       at java.lang.Thread.run(Thread.java:748)

not sure what triggered it - had a .tar.gz in the folder, the inventories were not automatically consumed by the webapp, so only agent was sending them

#1

Updated by Vincent MEMBRÉ about 1 year ago

  • Target version changed from 5.0.12 to 5.0.13
#2

Updated by Vincent MEMBRÉ about 1 year ago

  • Target version changed from 5.0.13 to 5.0.14
#3

Updated by Vincent MEMBRÉ 12 months ago

  • Target version changed from 5.0.14 to 5.0.15
#4

Updated by Vincent MEMBRÉ 10 months ago

  • Target version changed from 5.0.15 to 5.0.16
#5

Updated by François ARMAND 8 months ago

  • Tags set to Sponsored
  • Severity set to Major - prevents use of part of Rudder | no simple workaround
  • User visibility set to Operational - other Techniques | Rudder settings | Plugins
  • Effort required set to Small
  • Priority changed from 0 to 89

This has now been seen in the wild, so I'm making the priority higher.

#6

Updated by François ARMAND 8 months ago

  • Subject changed from Null pointer exception with hundreds of inventories to NullPointerException with hundreds of inventories
  • Description updated (diff)
#7

Updated by François ARMAND 8 months ago

  • Description updated (diff)
#8

Updated by François ARMAND 8 months ago

  • Status changed from New to In progress
  • Assignee set to François ARMAND
#9

Updated by François ARMAND 8 months ago

  • Category deleted (Performance and scalability)
  • Assignee deleted (François ARMAND)
  • Target version deleted (5.0.16)
  • User visibility deleted (Operational - other Techniques | Rudder settings | Plugins)
  • Effort required deleted (Small)
  • Priority changed from 89 to 0

So, the NPE is not the cause that stop processing of inventories. Even if we take care of it, we seems to reach a deadlock after that. A thread dump shows that the two inventory file watchers are lock here:

"inventory-wait-sig-84@7775" daemon prio=5 tid=0x54 nid=NA waiting
  java.lang.Thread.State: WAITING
      at jdk.internal.misc.Unsafe.park(Unsafe.java:-1)
      at java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
      at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
      at java.util.concurrent.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:483)
      at java.util.concurrent.LinkedBlockingDeque.take(LinkedBlockingDeque.java:671)
      at sun.nio.fs.AbstractWatchService.take(AbstractWatchService.java:118)
      at com.normation.inventory.provisioning.endpoint.Watchers$$anon$1$$anon$2.$anonfun$run$2(InventoryFileWatcher.scala:112)
      at com.normation.inventory.provisioning.endpoint.Watchers$$anon$1$$anon$2$$Lambda$266.1405133481.apply(Unknown Source:-1)
      at scala.collection.Iterator$$anon$9.next(Iterator.scala:173)
      at scala.collection.Iterator.foreach(Iterator.scala:941)
      at scala.collection.Iterator.foreach$(Iterator.scala:941)
      at scala.collection.AbstractIterator.foreach(Iterator.scala:1429)
      at com.normation.inventory.provisioning.endpoint.Watchers$$anon$1$$anon$2.$anonfun$run$1(InventoryFileWatcher.scala:112)
      at com.normation.inventory.provisioning.endpoint.Watchers$$anon$1$$anon$2$$Lambda$263.1496893913.apply$mcV$sp(Unknown Source:-1)
      at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
      at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:57)
      at scala.concurrent.package$.blocking(package.scala:146)
      at com.normation.inventory.provisioning.endpoint.Watchers$$anon$1$$anon$2.run(InventoryFileWatcher.scala:112)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
      at java.lang.Thread.run(Thread.java:834)
#10

Updated by François ARMAND 8 months ago

  • Category set to Performance and scalability
  • Assignee set to François ARMAND
  • Target version set to 5.0.16
  • User visibility set to Operational - other Techniques | Rudder settings | Plugins
  • Effort required set to Small
  • Priority changed from 0 to 89
#11

Updated by François ARMAND 8 months ago

  • Status changed from In progress to Pending technical review
  • Assignee changed from François ARMAND to Vincent MEMBRÉ
  • Pull Request set to https://github.com/Normation/rudder/pull/2757
#13

Updated by François ARMAND 8 months ago

  • Status changed from Pending technical review to Pending release
  • Priority changed from 89 to 88
#15

Updated by Alexis MOUSSET 7 months ago

  • Subject changed from NullPointerException with hundreds of inventories to NullPointerException with hundreds of inventories at once
#16

Updated by Vincent MEMBRÉ 7 months ago

  • Status changed from Pending release to Released

This bug has been fixed in Rudder 5.0.16 which was released today.

Also available in: Atom PDF