Bug #14991
closed
NullPointerException with hundreds of inventories at once
Added by Nicolas CHARLES over 5 years ago.
Updated over 2 years ago.
Category:
Performance and scalability
Severity:
Major - prevents use of part of Rudder | no simple workaround
User visibility:
Operational - other Techniques | Rudder settings | Plugins
Description
Workaround¶
- move inventories from /var/rudder/inventories/accepted-nodes-update
or /var/rudder/inventories/incoming
to a temp directory;
- restart inventory watcher. From rudder server, do:
- in 5.0, use curl -X POST "http://localhost:8080/endpoint/api/watcher/restart
- in 6.0, use curl -H "X-API-Token: $(cat /var/rudder/run/api-token)" -X POST "https://.../api/latest/inventory/watcher/restart
- move back inventories from temp directory to
/var/rudder/inventories/accepted-nodes-update@ by batch of 100.
Description¶
With 100s of inventories in incoming folder, i got the following error
java.lang.NullPointerException
at sun.nio.fs.UnixPath.toUnixPath(UnixPath.java:198)
at sun.nio.fs.UnixPath.resolve(UnixPath.java:397)
at sun.nio.fs.UnixPath.resolve(UnixPath.java:43)
at better.files.FileMonitor.$anonfun$process$1(FileMonitor.scala:33)
at better.files.FileMonitor.$anonfun$process$1$adapted(FileMonitor.scala:31)
at scala.collection.Iterator.foreach(Iterator.scala:937)
at scala.collection.Iterator.foreach$(Iterator.scala:937)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1425)
at scala.collection.IterableLike.foreach(IterableLike.scala:70)
at scala.collection.IterableLike.foreach$(IterableLike.scala:69)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at better.files.FileMonitor.process(FileMonitor.scala:31)
at com.normation.inventory.provisioning.endpoint.Watchers$$anon$1$$anon$2.$anonfun$run$3(InventoryFileWatcher.scala:112)
at com.normation.inventory.provisioning.endpoint.Watchers$$anon$1$$anon$2.$anonfun$run$3$adapted(InventoryFileWatcher.scala:112)
at scala.collection.Iterator.foreach(Iterator.scala:937)
at scala.collection.Iterator.foreach$(Iterator.scala:937)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1425)
at com.normation.inventory.provisioning.endpoint.Watchers$$anon$1$$anon$2.$anonfun$run$1(InventoryFileWatcher.scala:112)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
at scala.concurrent.package$.blocking(package.scala:142)
at com.normation.inventory.provisioning.endpoint.Watchers$$anon$1$$anon$2.run(InventoryFileWatcher.scala:112)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
not sure what triggered it - had a .tar.gz in the folder, the inventories were not automatically consumed by the webapp, so only agent was sending them
- Target version changed from 5.0.12 to 5.0.13
- Target version changed from 5.0.13 to 5.0.14
- Target version changed from 5.0.14 to 5.0.15
- Target version changed from 5.0.15 to 5.0.16
- Translation missing: en.field_tag_list set to Sponsored
- Severity set to Major - prevents use of part of Rudder | no simple workaround
- User visibility set to Operational - other Techniques | Rudder settings | Plugins
- Effort required set to Small
- Priority changed from 0 to 89
This has now been seen in the wild, so I'm making the priority higher.
- Subject changed from Null pointer exception with hundreds of inventories to NullPointerException with hundreds of inventories
- Description updated (diff)
- Description updated (diff)
- Status changed from New to In progress
- Assignee set to François ARMAND
- Category deleted (
Performance and scalability)
- Assignee deleted (
François ARMAND)
- Target version deleted (
5.0.16)
- User visibility deleted (
Operational - other Techniques | Rudder settings | Plugins)
- Effort required deleted (
Small)
- Priority changed from 89 to 0
So, the NPE is not the cause that stop processing of inventories. Even if we take care of it, we seems to reach a deadlock after that. A thread dump shows that the two inventory file watchers are lock here:
"inventory-wait-sig-84@7775" daemon prio=5 tid=0x54 nid=NA waiting
java.lang.Thread.State: WAITING
at jdk.internal.misc.Unsafe.park(Unsafe.java:-1)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
at java.util.concurrent.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:483)
at java.util.concurrent.LinkedBlockingDeque.take(LinkedBlockingDeque.java:671)
at sun.nio.fs.AbstractWatchService.take(AbstractWatchService.java:118)
at com.normation.inventory.provisioning.endpoint.Watchers$$anon$1$$anon$2.$anonfun$run$2(InventoryFileWatcher.scala:112)
at com.normation.inventory.provisioning.endpoint.Watchers$$anon$1$$anon$2$$Lambda$266.1405133481.apply(Unknown Source:-1)
at scala.collection.Iterator$$anon$9.next(Iterator.scala:173)
at scala.collection.Iterator.foreach(Iterator.scala:941)
at scala.collection.Iterator.foreach$(Iterator.scala:941)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1429)
at com.normation.inventory.provisioning.endpoint.Watchers$$anon$1$$anon$2.$anonfun$run$1(InventoryFileWatcher.scala:112)
at com.normation.inventory.provisioning.endpoint.Watchers$$anon$1$$anon$2$$Lambda$263.1496893913.apply$mcV$sp(Unknown Source:-1)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:57)
at scala.concurrent.package$.blocking(package.scala:146)
at com.normation.inventory.provisioning.endpoint.Watchers$$anon$1$$anon$2.run(InventoryFileWatcher.scala:112)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.lang.Thread.run(Thread.java:834)
- Category set to Performance and scalability
- Assignee set to François ARMAND
- Target version set to 5.0.16
- User visibility set to Operational - other Techniques | Rudder settings | Plugins
- Effort required set to Small
- Priority changed from 0 to 89
- Status changed from In progress to Pending technical review
- Assignee changed from François ARMAND to Vincent MEMBRÉ
- Pull Request set to https://github.com/Normation/rudder/pull/2757
- Status changed from Pending technical review to Pending release
- Priority changed from 89 to 88
- Subject changed from NullPointerException with hundreds of inventories to NullPointerException with hundreds of inventories at once
- Name check set to Reviewed
- Status changed from Pending release to Released
This bug has been fixed in Rudder 5.0.16 which was released today.
- Priority changed from 88 to 83
Also available in: Atom
PDF