Project

General

Profile

Actions

Bug #10434

closed

Error on one inventory when trying to accept 100 inventories at the same time

Added by Nicolas CHARLES about 7 years ago. Updated over 2 years ago.

Status:
Rejected
Priority:
N/A
Assignee:
-
Category:
Web - Nodes & inventories
Target version:
Severity:
Minor - inconvenience | misleading | easy workaround
UX impact:
User visibility:
Operational - other Techniques | Technique editor | Rudder settings
Effort required:
Priority:
0
Name check:
Fix check:
Regression:

Description

On 4.1, when trying to accept 500 inventories, there is one which is failing with following error message in the web interface
Error when trying to accept node xxxx <- The connection is not established

Error in log is

[2017-03-15 22:28:42] ERROR com.normation.rudder.services.servers.NewNodeManagerImpl - Unit acceptor error for node NodeId(44936c20-2e1a-4673-bb49-eb6fa4faad39) <- Error when trying to accept node 44936c20-2e1a-4673-bb49-eb6fa4faad39 <- The connection is not established.

Don't know which connection - slapd doesn't complain at all


Related issues 1 (1 open0 closed)

Related to Rudder - Architecture #5367: Make node acceptance process asynchronousNewActions
Actions #1

Updated by Nicolas CHARLES about 7 years ago

  • Subject changed from Error on one inventory when trying to accept 500 inventories at the same time to Error on one inventory when trying to accept 100 inventories at the same time
  • Description updated (diff)
Actions #2

Updated by Nicolas CHARLES about 7 years ago

Here is the error
Error when trying to accept node 9c4d80a6-00fb-4a7e-96a2-bda4a27c00ea <- The connection is not established.

2017-03-16 13:25:06] DEBUG com.zaxxer.hikari.pool.PoolBase - HikariPool-1 - Reset (autoCommit) on connection org.postgresql.jdbc4.Jdbc4Connection@25a32fb
[2017-03-16 13:25:06] DEBUG com.normation.rudder.services.servers.NewNodeManagerImpl - Pre accepted phase: accept_or_refuse_new_node:historize_inventory
[2017-03-16 13:25:06] DEBUG com.normation.rudder.services.servers.NewNodeManagerImpl - Pre accepted phase: accept_new_server:ou=node
[2017-03-16 13:25:06] DEBUG com.normation.rudder.services.servers.NewNodeManagerImpl - Pre accepted phase: accept_new_server:inventory
[2017-03-16 13:25:06] DEBUG com.normation.rudder.services.servers.NewNodeManagerImpl - Pre accepted phase: accept_new_server:check_hostname_unicity
[2017-03-16 13:25:06] DEBUG com.normation.rudder.services.servers.NewNodeManagerImpl - Exception in unit acceptor accept_new_server:inventory
com.unboundid.ldap.sdk.LDAPSearchException: The connection is not established.
        at com.unboundid.ldap.sdk.LDAPConnection.search(LDAPConnection.java:3645)
        at com.unboundid.ldap.sdk.LDAPConnection.getEntry(LDAPConnection.java:1712)
        at com.normation.ldap.sdk.RoLDAPConnection.get(LDAPConnection.scala:314)
        at com.normation.ldap.sdk.ReadOnlyEntryLDAPConnection.exists(LDAPConnection.scala:126)
        at com.normation.ldap.sdk.ReadOnlyEntryLDAPConnection.exists$(LDAPConnection.scala:126)
        at com.normation.ldap.sdk.RoLDAPConnection.exists(LDAPConnection.scala:283)
        at com.normation.inventory.ldap.core.FullInventoryRepositoryImpl.$anonfun$getExistingMachineDN$1(FullInventoryRepositoryImpl.scala:98)
        at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:234)
        at scala.collection.immutable.List.foreach(List.scala:378)
        at scala.collection.TraversableLike.map(TraversableLike.scala:234)
        at scala.collection.TraversableLike.map$(TraversableLike.scala:227)
        at scala.collection.immutable.List.map(List.scala:284)
        at com.normation.inventory.ldap.core.FullInventoryRepositoryImpl.getExistingMachineDN(FullInventoryRepositoryImpl.scala:96)
        at com.normation.inventory.ldap.core.FullInventoryRepositoryImpl.$anonfun$move$4(FullInventoryRepositoryImpl.scala:188)
        at net.liftweb.common.Full.flatMap(Box.scala:808)
        at com.normation.inventory.ldap.core.FullInventoryRepositoryImpl.move(FullInventoryRepositoryImpl.scala:181)
        at com.normation.inventory.ldap.core.FullInventoryRepositoryImpl.$anonfun$move$15(FullInventoryRepositoryImpl.scala:408)
        at net.liftweb.common.Full.flatMap(Box.scala:808)
        at net.liftweb.common.Box$WithFilter.flatMap(Box.scala:475)
        at com.normation.inventory.ldap.core.FullInventoryRepositoryImpl.$anonfun$move$13(FullInventoryRepositoryImpl.scala:407)
        at net.liftweb.common.Full.map(Box.scala:806)
        at com.normation.inventory.ldap.core.FullInventoryRepositoryImpl.$anonfun$move$12(FullInventoryRepositoryImpl.scala:400)
        at net.liftweb.common.Full.flatMap(Box.scala:808)
        at com.normation.inventory.ldap.core.FullInventoryRepositoryImpl.move(FullInventoryRepositoryImpl.scala:397)
        at com.normation.rudder.services.servers.AcceptInventory.acceptOne(NewNodeManager.scala:657)
        at com.normation.rudder.services.servers.ComposedNewNodeManager.$anonfun$acceptOne$1(NewNodeManager.scala:425)
        at com.normation.utils.Control$.$anonfun$sequence$1(Control.scala:54)
        at scala.collection.immutable.List.foreach(List.scala:378)
        at com.normation.utils.Control$.sequence(Control.scala:54)
        at com.normation.rudder.services.servers.ComposedNewNodeManager.acceptOne(NewNodeManager.scala:423)
        at com.normation.rudder.services.servers.ComposedNewNodeManager.accept(NewNodeManager.scala:475)
        at com.normation.rudder.services.servers.ComposedNewNodeManager.accept$(NewNodeManager.scala:441)
        at com.normation.rudder.services.servers.NewNodeManagerImpl.accept(NewNodeManager.scala:155)
        at com.normation.rudder.web.snippet.node.AcceptNode.$anonfun$addNodes$1(AcceptNode.scala:164)
        at com.normation.rudder.web.snippet.node.AcceptNode.$anonfun$addNodes$1$adapted(AcceptNode.scala:162)
        at scala.collection.immutable.List.foreach(List.scala:378)
        at com.normation.rudder.web.snippet.node.AcceptNode.addNodes(AcceptNode.scala:162)
        at com.normation.rudder.web.snippet.node.AcceptNode.$anonfun$listNode$6(AcceptNode.scala:313)
        at net.liftweb.http.S$NFuncHolder.$anonfun$apply$3(S.scala:185)
        at scala.collection.immutable.List.map(List.scala:272)
        at net.liftweb.http.S$NFuncHolder.apply(S.scala:185)
        at net.liftweb.http.S$ProxyFuncHolder.apply(S.scala:114)
        at net.liftweb.http.LiftSession.$anonfun$runParams$8(LiftSession.scala:558)
        at net.liftweb.http.LiftSession.$anonfun$runParams$17(LiftSession.scala:571)
       at scala.collection.immutable.List.map(List.scala:272)
        at net.liftweb.http.LiftSession.$anonfun$runParams$13(LiftSession.scala:571)
        at scala.collection.immutable.List.flatMap(List.scala:324)
        at net.liftweb.http.LiftSession.runParams(LiftSession.scala:561)
        at net.liftweb.http.LiftSession.processRequest(LiftSession.scala:1038)
        at net.liftweb.http.LiftServlet.net$liftweb$http$LiftServlet$$dispatchStatefulRequest(LiftServlet.scala:543)
        at net.liftweb.http.LiftServlet$StatefulResponse$.$anonfun$process$3(LiftServlet.scala:385)
        at net.liftweb.http.S.wrapQuery(S.scala:1562)
        at net.liftweb.http.S.$anonfun$_nest2InnerInit$2(S.scala:1729)
        at net.liftweb.http.S.doAround(S.scala:1491)
        at net.liftweb.http.S.$anonfun$_nest2InnerInit$1(S.scala:1717)
        at net.liftweb.util.ThreadGlobal.doWith(ThreadGlobal.scala:71)
        at net.liftweb.http.S._nest2InnerInit(S.scala:1716)
        at net.liftweb.http.S.$anonfun$_innerInit$5(S.scala:1762)
        at net.liftweb.util.ThreadGlobal.doWith(ThreadGlobal.scala:71)
        at net.liftweb.http.S.$anonfun$withReq$2(S.scala:1773)
        at net.liftweb.util.ThreadGlobal.doWith(ThreadGlobal.scala:71)
        at net.liftweb.http.S.withReq(S.scala:1772)
        at net.liftweb.http.S.withReq$(S.scala:1770)
        at net.liftweb.http.S$.withReq(S.scala:47)
(snip)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:745)
Caused by: com.unboundid.ldap.sdk.LDAPException: The connection is not established.
        at com.unboundid.ldap.sdk.LDAPConnection.registerResponseAcceptor(LDAPConnection.java:4476)
        at com.unboundid.ldap.sdk.SearchRequest.processAsync(SearchRequest.java:1373)
        at com.unboundid.ldap.sdk.SearchRequest.process(SearchRequest.java:1135)
        at com.unboundid.ldap.sdk.LDAPConnection.search(LDAPConnection.java:3635)
        ... 199 common frames omitted
[2017-03-16 13:25:06] ERROR com.normation.rudder.services.servers.NewNodeManagerImpl - Error when trying to accept node 9c4d80a6-00fb-4a7e-96a2-bda4a27c00ea. Rollbacking. <- The connection is not established.
[2017-03-16 13:25:06] DEBUG com.normation.rudder.services.servers.NewNodeManagerImpl - 
*****************************************************
Rollbacking
*****************************************************
[2017-03-16 13:25:06] DEBUG com.normation.rudder.services.servers.NewNodeManagerImpl - Rollbacking accept_new_server:check_hostname_unicity for 9c4d80a6-00fb-4a7e-96a2-bda4a27c00ea
[2017-03-16 13:25:06] DEBUG com.normation.rudder.services.servers.NewNodeManagerImpl - Rollbacking accept_new_server:inventory for 9c4d80a6-00fb-4a7e-96a2-bda4a27c00ea
[2017-03-16 13:25:06] DEBUG com.normation.rudder.services.servers.AcceptInventory - Succesfully rollbacked NodeId(9c4d80a6-00fb-4a7e-96a2-bda4a27c00ea) for process 'accept_new_server:inventory'
[2017-03-16 13:25:06] DEBUG com.normation.rudder.services.servers.NewNodeManagerImpl - Rollbacking accept_new_server:ou=node for 9c4d80a6-00fb-4a7e-96a2-bda4a27c00ea
[2017-03-16 13:25:06] DEBUG com.normation.rudder.services.servers.AcceptFullInventoryInNodeOu - Succesfully rollbacked NodeId(9c4d80a6-00fb-4a7e-96a2-bda4a27c00ea) for process 'accept_new_server:ou=node'
[2017-03-16 13:25:06] DEBUG com.normation.rudder.services.servers.NewNodeManagerImpl - Rollbacking accept_or_refuse_new_node:historize_inventory for 9c4d80a6-00fb-4a7e-96a2-bda4a27c00ea
[2017-03-16 13:25:06] ERROR com.normation.rudder.services.servers.NewNodeManagerImpl - Unit acceptor error for node NodeId(9c4d80a6-00fb-4a7e-96a2-bda4a27c00ea) <- Error when trying to accept node 9c4d80a6-00fb-4a7e-96a2-bda4a27c00ea <- The connection is not established.
[2017-03-16 13:25:06] DEBUG com.normation.rudder.services.servers.NewNodeManagerImpl - Pre accepted phase: accept_or_refuse_new_node:historize_inventory
[2017-03-16 13:25:06] DEBUG com.normation.rudder.services.servers.NewNodeManagerImpl - Pre accepted phase: accept_new_server:ou=node
[2017-03-16 13:25:06] DEBUG com.normation.rudder.services.servers.NewNodeManagerImpl - Pre accepted phase: accept_new_server:inventory
[2017-03-16 13:25:06] DEBUG com.normation.rudder.services.servers.NewNodeManagerImpl - Pre accepted phase: accept_new_server:check_hostname_unicity
[2017-03-16 13:25:06] DEBUG com.normation.rudder.services.servers.NewNodeManagerImpl - Unit acceptors ok for NodeId(7bb32578-d893-4f56-8555-bdc2b5ca680c)
[2017-03-16 13:25:06] INFO  com.normation.rudder.services.servers.NewNodeManagerImpl - New node accepted and managed by Rudder: 7bb32578-d893-4f56-8555-bdc2b5ca680c
[2017-03-16 13:25:07] DEBUG com.zaxxer.hikari.pool.PoolBase - HikariPool-1 - Reset (autoCommit) on connection org.postgresql.jdbc4.Jdbc4Connection@152da6f4
[2017-03-16 13:25:07] DEBUG com.zaxxer.hikari.pool.PoolBase - HikariPool-1 - Reset (autoCommit) on connection org.postgresql.jdbc4.Jdbc4Connection@152da6f4
[2017-03-16 13:25:07] DEBUG com.zaxxer.hikari.pool.PoolBase - HikariPool-1 - Reset (autoCommit) on connection org.postgresql.jdbc4.Jdbc4Connection@152da6f4
[2017-03-16 13:25:07] DEBUG com.zaxxer.hikari.pool.PoolBase - HikariPool-1 - Reset (autoCommit) on connection org.postgresql.jdbc4.Jdbc4Connection@152da6f4
[2017-03-16 13:25:07] DEBUG com.normation.rudder.web.snippet.node.AcceptNode - Successfully added node '7bb32578-d893-4f56-8555-bdc2b5ca680c'

Actions #3

Updated by Nicolas CHARLES about 7 years ago

Meanwhile, on slapd:


Mar 16 13:24:59 dhcp240 rudder-slapd[20817]: <= mdb_equality_candidates: (container) not indexed
Mar 16 13:24:59 dhcp240 rudder-slapd[20817]: <= mdb_equality_candidates: (container) not indexed
Mar 16 13:24:59 dhcp240 rudder-slapd[20817]: <= mdb_equality_candidates: (isDynamic) not indexed
Mar 16 13:24:59 dhcp240 rudder-slapd[20817]: <= mdb_equality_candidates: (nodeGroupId) not indexed
Mar 16 13:24:59 dhcp240 rudder-slapd[20817]: <= mdb_equality_candidates: (cn) not indexed
Mar 16 13:24:59 dhcp240 rudder-slapd[20817]: <= mdb_equality_candidates: (directiveId) not indexed
Mar 16 13:24:59 dhcp240 rudder-slapd[20817]: <= mdb_equality_candidates: (nodeHostname) not indexed
Mar 16 13:24:59 dhcp240 rudder-slapd[20817]: <= mdb_equality_candidates: (container) not indexed
Mar 16 13:24:59 dhcp240 rudder-slapd[20817]: <= mdb_equality_candidates: (container) not indexed
Mar 16 13:24:59 dhcp240 rudder-slapd[20817]: <= mdb_equality_candidates: (container) not indexed
Mar 16 13:24:59 dhcp240 rudder-slapd[20817]: <= mdb_equality_candidates: (container) not indexed
Mar 16 13:24:59 dhcp240 rudder-slapd[20817]: <= mdb_equality_candidates: (container) not indexed
Mar 16 13:24:59 dhcp240 rudder-slapd[20817]: <= mdb_equality_candidates: (container) not indexed
Mar 16 13:25:21 dhcp240 rudder-slapd[20817]: <= mdb_equality_candidates: (container) not indexed
Mar 16 13:25:21 dhcp240 rudder-slapd[20817]: <= mdb_equality_candidates: (container) not indexed
Mar 16 13:25:21 dhcp240 rudder-slapd[20817]: <= mdb_equality_candidates: (container) not indexed
Mar 16 13:25:21 dhcp240 rudder-slapd[20817]: <= mdb_equality_candidates: (container) not indexed
Mar 16 13:25:21 dhcp240 rudder-slapd[20817]: <= mdb_equality_candidates: (container) not indexed
Mar 16 13:25:21 dhcp240 rudder-slapd[20817]: <= mdb_equality_candidates: (container) not indexed
Mar 16 13:25:21 dhcp240 rudder-slapd[20817]: <= mdb_equality_candidates: (container) not indexed
Mar 16 13:25:21 dhcp240 rudder-slapd[20817]: <= mdb_equality_candidates: (container) not indexed
Mar 16 13:25:21 dhcp240 rudder-slapd[20817]: <= mdb_equality_candidates: (container) not indexed

Actions #4

Updated by François ARMAND about 7 years ago

  • User visibility changed from Infrequent - complex configurations | third party integrations to Operational - other Techniques | Technique editor | Rudder settings
Actions #5

Updated by Benoît PECCATTE about 7 years ago

  • Priority set to 17
Actions #6

Updated by Vincent MEMBRÉ about 7 years ago

  • Target version changed from 4.1.0 to 4.1.1
Actions #7

Updated by François ARMAND about 7 years ago

Actions #8

Updated by Vincent MEMBRÉ about 7 years ago

  • Target version changed from 4.1.1 to 4.1.2
Actions #9

Updated by Vincent MEMBRÉ about 7 years ago

  • Target version changed from 4.1.2 to 4.1.3
Actions #10

Updated by Vincent MEMBRÉ about 7 years ago

  • Target version changed from 4.1.3 to 4.1.4
  • Priority changed from 17 to 16
Actions #11

Updated by Vincent MEMBRÉ almost 7 years ago

  • Target version changed from 4.1.4 to 4.1.5
Actions #12

Updated by Alexis Mousset almost 7 years ago

  • Target version changed from 4.1.5 to 4.1.6
Actions #13

Updated by Benoît PECCATTE almost 7 years ago

  • Priority changed from 16 to 31
Actions #14

Updated by Vincent MEMBRÉ almost 7 years ago

  • Target version changed from 4.1.6 to 4.1.7
  • Priority changed from 31 to 30
Actions #15

Updated by Vincent MEMBRÉ almost 7 years ago

  • Target version changed from 4.1.7 to 4.1.8
Actions #16

Updated by Vincent MEMBRÉ over 6 years ago

  • Target version changed from 4.1.8 to 4.1.9
  • Priority changed from 30 to 29
Actions #17

Updated by Vincent MEMBRÉ over 6 years ago

  • Target version changed from 4.1.9 to 4.1.10
Actions #18

Updated by Vincent MEMBRÉ over 6 years ago

  • Target version changed from 4.1.10 to 4.1.11
  • Priority changed from 29 to 27
Actions #19

Updated by Vincent MEMBRÉ about 6 years ago

  • Target version changed from 4.1.11 to 4.1.12
Actions #20

Updated by Vincent MEMBRÉ about 6 years ago

  • Target version changed from 4.1.12 to 4.1.13
Actions #21

Updated by Vincent MEMBRÉ almost 6 years ago

  • Target version changed from 4.1.13 to 4.1.14
Actions #22

Updated by Benoît PECCATTE almost 6 years ago

  • Target version changed from 4.1.14 to 4.1.15
Actions #23

Updated by Vincent MEMBRÉ over 5 years ago

  • Target version changed from 4.1.15 to 4.1.16
Actions #24

Updated by Vincent MEMBRÉ over 5 years ago

  • Target version changed from 4.1.16 to 4.1.17
Actions #25

Updated by Vincent MEMBRÉ over 5 years ago

  • Target version changed from 4.1.17 to 4.1.18
  • Priority changed from 27 to 0
Actions #26

Updated by Vincent MEMBRÉ over 5 years ago

  • Target version changed from 4.1.18 to 4.1.19
Actions #27

Updated by Alexis Mousset over 5 years ago

  • Target version changed from 4.1.19 to 4.1.20
Actions #28

Updated by François ARMAND over 5 years ago

  • Target version changed from 4.1.20 to 4.1.21
Actions #29

Updated by Vincent MEMBRÉ about 5 years ago

  • Target version changed from 4.1.21 to 4.1.22
Actions #30

Updated by Vincent MEMBRÉ about 5 years ago

  • Target version changed from 4.1.22 to 4.1.23
Actions #31

Updated by Vincent MEMBRÉ about 5 years ago

  • Target version changed from 4.1.23 to 4.1.24
Actions #32

Updated by Vincent MEMBRÉ almost 5 years ago

  • Target version changed from 4.1.24 to 588
Actions #33

Updated by Nicolas CHARLES almost 5 years ago

  • Target version changed from 588 to 5.0.13

i think i've seen it also in 5.0, retargeting

Actions #34

Updated by Félix DALLIDET almost 5 years ago

I tried to reproduce, but unfortunately, adding 1000 of inventories in one go in the incoming folder lead to NullPointerException so I wasn't able to test.

[2019-07-09 12:18:41] INFO  com.normation.inventory.provisioning.endpoint.config.PropertyPlaceholderConfig - Use configuration file defined by JVM property -Dinventoryweb.configFile : /opt/rudder/etc/inventory-web.properties
[2019-07-09 12:18:43] INFO  inventory-processing - Incoming inventory watcher started - process existing inventories
java.lang.NullPointerException
>-at sun.nio.fs.UnixPath.toUnixPath(UnixPath.java:198)
>-at sun.nio.fs.UnixPath.resolve(UnixPath.java:397)
>-at sun.nio.fs.UnixPath.resolve(UnixPath.java:43)
>-at better.files.FileMonitor.$anonfun$process$1(FileMonitor.scala:33)
>-at better.files.FileMonitor.$anonfun$process$1$adapted(FileMonitor.scala:31)
>-at scala.collection.Iterator.foreach(Iterator.scala:937)
>-at scala.collection.Iterator.foreach$(Iterator.scala:937)
>-at scala.collection.AbstractIterator.foreach(Iterator.scala:1425)
>-at scala.collection.IterableLike.foreach(IterableLike.scala:70)
>-at scala.collection.IterableLike.foreach$(IterableLike.scala:69)
>-at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
>-at better.files.FileMonitor.process(FileMonitor.scala:31)
>-at com.normation.inventory.provisioning.endpoint.Watchers$$anon$1$$anon$2.$anonfun$run$3(InventoryFileWatcher.scala:112)
>-at com.normation.inventory.provisioning.endpoint.Watchers$$anon$1$$anon$2.$anonfun$run$3$adapted(InventoryFileWatcher.scala:112)

Actions #35

Updated by Vincent MEMBRÉ almost 5 years ago

  • Target version changed from 5.0.13 to 5.0.14
Actions #36

Updated by Vincent MEMBRÉ over 4 years ago

  • Target version changed from 5.0.14 to 5.0.15
Actions #37

Updated by Vincent MEMBRÉ over 4 years ago

  • Target version changed from 5.0.15 to 5.0.16
Actions #38

Updated by Alexis Mousset over 4 years ago

  • Target version changed from 5.0.16 to 5.0.17
Actions #39

Updated by Vincent MEMBRÉ about 4 years ago

  • Target version changed from 5.0.17 to 5.0.18
Actions #40

Updated by Vincent MEMBRÉ almost 4 years ago

  • Target version changed from 5.0.18 to 5.0.19
Actions #41

Updated by Vincent MEMBRÉ almost 4 years ago

  • Target version changed from 5.0.19 to 5.0.20
Actions #42

Updated by Vincent MEMBRÉ over 3 years ago

  • Target version changed from 5.0.20 to 797
Actions #43

Updated by Benoît PECCATTE almost 3 years ago

  • Target version changed from 797 to 6.1.14
Actions #44

Updated by Vincent MEMBRÉ almost 3 years ago

  • Target version changed from 6.1.14 to 6.1.15
Actions #45

Updated by Vincent MEMBRÉ almost 3 years ago

  • Target version changed from 6.1.15 to 6.1.16
Actions #46

Updated by Vincent MEMBRÉ almost 3 years ago

  • Target version changed from 6.1.16 to 6.1.17
Actions #47

Updated by Vincent MEMBRÉ over 2 years ago

  • Target version changed from 6.1.17 to 6.1.18
Actions #48

Updated by Vincent MEMBRÉ over 2 years ago

  • Target version changed from 6.1.18 to 6.1.19
Actions #49

Updated by François ARMAND over 2 years ago

  • Status changed from New to Rejected

I'm closing that one because the correction is to implement #5367

Actions

Also available in: Atom PDF