Bug #14991
closedNullPointerException with hundreds of inventories at once
Description
Workaround¶
- move inventories from /var/rudder/inventories/accepted-nodes-update
or /var/rudder/inventories/incoming
to a temp directory;
- restart inventory watcher. From rudder server, do:
- in 5.0, use curl -X POST "http://localhost:8080/endpoint/api/watcher/restart
- in 6.0, use curl -H "X-API-Token: $(cat /var/rudder/run/api-token)" -X POST "https://.../api/latest/inventory/watcher/restart
/var/rudder/inventories/accepted-nodes-update@ by batch of 100.
- move back inventories from temp directory to
Description¶
With 100s of inventories in incoming folder, i got the following error
java.lang.NullPointerException at sun.nio.fs.UnixPath.toUnixPath(UnixPath.java:198) at sun.nio.fs.UnixPath.resolve(UnixPath.java:397) at sun.nio.fs.UnixPath.resolve(UnixPath.java:43) at better.files.FileMonitor.$anonfun$process$1(FileMonitor.scala:33) at better.files.FileMonitor.$anonfun$process$1$adapted(FileMonitor.scala:31) at scala.collection.Iterator.foreach(Iterator.scala:937) at scala.collection.Iterator.foreach$(Iterator.scala:937) at scala.collection.AbstractIterator.foreach(Iterator.scala:1425) at scala.collection.IterableLike.foreach(IterableLike.scala:70) at scala.collection.IterableLike.foreach$(IterableLike.scala:69) at scala.collection.AbstractIterable.foreach(Iterable.scala:54) at better.files.FileMonitor.process(FileMonitor.scala:31) at com.normation.inventory.provisioning.endpoint.Watchers$$anon$1$$anon$2.$anonfun$run$3(InventoryFileWatcher.scala:112) at com.normation.inventory.provisioning.endpoint.Watchers$$anon$1$$anon$2.$anonfun$run$3$adapted(InventoryFileWatcher.scala:112) at scala.collection.Iterator.foreach(Iterator.scala:937) at scala.collection.Iterator.foreach$(Iterator.scala:937) at scala.collection.AbstractIterator.foreach(Iterator.scala:1425) at com.normation.inventory.provisioning.endpoint.Watchers$$anon$1$$anon$2.$anonfun$run$1(InventoryFileWatcher.scala:112) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12) at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) at scala.concurrent.package$.blocking(package.scala:142) at com.normation.inventory.provisioning.endpoint.Watchers$$anon$1$$anon$2.run(InventoryFileWatcher.scala:112) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
not sure what triggered it - had a .tar.gz in the folder, the inventories were not automatically consumed by the webapp, so only agent was sending them
Updated by Vincent MEMBRÉ over 5 years ago
- Target version changed from 5.0.12 to 5.0.13
Updated by Vincent MEMBRÉ over 5 years ago
- Target version changed from 5.0.13 to 5.0.14
Updated by Vincent MEMBRÉ about 5 years ago
- Target version changed from 5.0.14 to 5.0.15
Updated by Vincent MEMBRÉ about 5 years ago
- Target version changed from 5.0.15 to 5.0.16
Updated by François ARMAND almost 5 years ago
- Translation missing: en.field_tag_list set to Sponsored
- Severity set to Major - prevents use of part of Rudder | no simple workaround
- User visibility set to Operational - other Techniques | Rudder settings | Plugins
- Effort required set to Small
- Priority changed from 0 to 89
This has now been seen in the wild, so I'm making the priority higher.
Updated by François ARMAND almost 5 years ago
- Subject changed from Null pointer exception with hundreds of inventories to NullPointerException with hundreds of inventories
- Description updated (diff)
Updated by François ARMAND almost 5 years ago
- Status changed from New to In progress
- Assignee set to François ARMAND
Updated by François ARMAND almost 5 years ago
- Category deleted (
Performance and scalability) - Assignee deleted (
François ARMAND) - Target version deleted (
5.0.16) - User visibility deleted (
Operational - other Techniques | Rudder settings | Plugins) - Effort required deleted (
Small) - Priority changed from 89 to 0
So, the NPE is not the cause that stop processing of inventories. Even if we take care of it, we seems to reach a deadlock after that. A thread dump shows that the two inventory file watchers are lock here:
"inventory-wait-sig-84@7775" daemon prio=5 tid=0x54 nid=NA waiting java.lang.Thread.State: WAITING at jdk.internal.misc.Unsafe.park(Unsafe.java:-1) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081) at java.util.concurrent.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:483) at java.util.concurrent.LinkedBlockingDeque.take(LinkedBlockingDeque.java:671) at sun.nio.fs.AbstractWatchService.take(AbstractWatchService.java:118) at com.normation.inventory.provisioning.endpoint.Watchers$$anon$1$$anon$2.$anonfun$run$2(InventoryFileWatcher.scala:112) at com.normation.inventory.provisioning.endpoint.Watchers$$anon$1$$anon$2$$Lambda$266.1405133481.apply(Unknown Source:-1) at scala.collection.Iterator$$anon$9.next(Iterator.scala:173) at scala.collection.Iterator.foreach(Iterator.scala:941) at scala.collection.Iterator.foreach$(Iterator.scala:941) at scala.collection.AbstractIterator.foreach(Iterator.scala:1429) at com.normation.inventory.provisioning.endpoint.Watchers$$anon$1$$anon$2.$anonfun$run$1(InventoryFileWatcher.scala:112) at com.normation.inventory.provisioning.endpoint.Watchers$$anon$1$$anon$2$$Lambda$263.1496893913.apply$mcV$sp(Unknown Source:-1) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:57) at scala.concurrent.package$.blocking(package.scala:146) at com.normation.inventory.provisioning.endpoint.Watchers$$anon$1$$anon$2.run(InventoryFileWatcher.scala:112) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.lang.Thread.run(Thread.java:834)
Updated by François ARMAND almost 5 years ago
- Category set to Performance and scalability
- Assignee set to François ARMAND
- Target version set to 5.0.16
- User visibility set to Operational - other Techniques | Rudder settings | Plugins
- Effort required set to Small
- Priority changed from 0 to 89
Updated by François ARMAND almost 5 years ago
- Status changed from In progress to Pending technical review
- Assignee changed from François ARMAND to Vincent MEMBRÉ
- Pull Request set to https://github.com/Normation/rudder/pull/2757
Updated by Nicolas CHARLES almost 5 years ago
Updated by François ARMAND almost 5 years ago
- Status changed from Pending technical review to Pending release
- Priority changed from 89 to 88
Applied in changeset rudder|71248a45c0cb2084f1851d002c956ac331fe365d.
Updated by Alexis Mousset almost 5 years ago
- Subject changed from NullPointerException with hundreds of inventories to NullPointerException with hundreds of inventories at once
- Name check set to Reviewed
Updated by Vincent MEMBRÉ almost 5 years ago
- Status changed from Pending release to Released
This bug has been fixed in Rudder 5.0.16 which was released today.