Project

General

Profile

Bug #18955

Updated by François ARMAND over 3 years ago

 
 **RESOLUTION NOTE**: the provided patch use the OpenJDK boot parameter "CrashOnOutOfMemoryError" that is only available since OpenJDK 1.8.0_92. You should really use a more recent OpenJDL than that for performance and security reasons, but if you are stuck with a 6 years old runtime, you can modify the file @/opt/rudder/etc/rudder-jetty.conf@ and remove it. 

 In #14281 we added an handler for fatal exception so that they force rudder to stop (because from that point, rudder is broken in unknown way).  

 But now, rudder doesn't stop on them:  

 <pre> 
 [2021-02-24T15:27:59.413Z] ERROR FATAL Rudder JVM caught an unhandled fatal exception. Rudder will now stop to prevent further inconsistant behavior. This is likely a bug, please contact Rudder developers. You can configure the list of fatal exception in /opt/rudder/etc/rudder-web.properties -> rudder.jvm.fatal.exceptions 
 [2021-02-24T15:28:40.190Z] ERROR FATAL exception in thread 'pool-2-thread-14' (in threadgroup 'main'): 'java.lang.OutOfMemoryError': 'Java heap space' 
 java.lang.OutOfMemoryError: Java heap space 
 [2021-02-24T15:29:01.906Z] ERROR FATAL Rudder JVM caught an unhandled fatal exception. Rudder will now stop to prevent further inconsistant behavior. This is likely a bug, please contact Rudder developers. You can configure the list of fatal exception in /opt/rudder/etc/rudder-web.properties -> rudder.jvm.fatal.exceptions 
 [2021-02-24T15:30:02.031Z] ERROR FATAL exception in thread 'JettyShutdownThread' (in threadgroup 'main'): 'java.lang.OutOfMemoryError': 'Java heap space' 
 java.lang.OutOfMemoryError: Java heap space 
 java.lang.OutOfMemoryError: Java heap space 
 [2021-02-24T15:32:26.929Z] ERROR FATAL Rudder JVM caught an unhandled fatal exception. Rudder will now stop to prevent further inconsistant behavior. This is likely a bug, please contact Rudder developers. You can configure the list of fatal exception in /opt/rudder/etc/rudder-web.properties -> rudder.jvm.fatal.exceptions 
 [2021-02-24T15:34:25.738Z] ERROR FATAL exception in thread 'Health Check Thread for LDAPConnectionPool(serverSet=SingleServerSet(server=localhost:389, includesAuthentication=false, includesPostConnectProcessing=false), maxConnections=2)' (in threadgroup 'main'): 'java.lang.OutOfMemoryError': 'Java heap space' 
 java.lang.OutOfMemoryError: Java heap space 
 java.lang.OutOfMemoryError: Java heap space 
 java.lang.OutOfMemoryError: Java heap space 
 [2021-02-24T15:32:21.339Z] ERROR FATAL Rudder JVM caught an unhandled fatal exception. Rudder will now stop to prevent further inconsistant behavior. This is likely a bug, please contact Rudder developers. You can configure the list of fatal exception in /opt/rudder/etc/rudder-web.properties -> rudder.jvm.fatal.exceptions 
 [2021-02-24T15:34:25.737Z] ERROR FATAL Rudder JVM caught an unhandled fatal exception. Rudder will now stop to prevent further inconsistant behavior. This is likely a bug, please contact Rudder developers. You can configure the list of fatal exception in /opt/rudder/etc/rudder-web.properties -> rudder.jvm.fatal.exceptions 
 [2021-02-24T15:34:25.748Z] ERROR FATAL exception in thread 'zio-default-blocking-42' (in threadgroup 'zio-default-blocking'): 'java.lang.OutOfMemoryError': 'Java heap space' 
 java.lang.OutOfMemoryError: Java heap space 
 [2021-02-24 15:34:25+0000] ERROR net.liftweb.actor.ActorLogger - Actor threw an exception 
 java.lang.OutOfMemoryError: Java heap space 
 java.lang.OutOfMemoryError: Java heap space 
 [2021-02-24T15:34:25.747Z] ERROR FATAL exception in thread 'Connection reader for connection 141 to localhost:389' (in threadgroup 'zio-default-blocking'): 'java.lang.OutOfMemoryError': 'Java heap space' 
 java.lang.OutOfMemoryError: Java heap space 
 [2021-02-24T15:34:25.766Z] ERROR FATAL Rudder JVM caught an unhandled fatal exception. Rudder will now stop to prevent further inconsistant behavior. This is likely a bug, please contact Rudder developers. You can configure the list of fatal exception in /opt/rudder/etc/rudder-web.properties -> rudder.jvm.fatal.exceptions 
 [2021-02-24T15:34:25.766Z] ERROR FATAL exception in thread 'pool-2-thread-12' (in threadgroup 'main'): 'java.lang.OutOfMemoryError': 'Java heap space' 
 java.lang.OutOfMemoryError: Java heap space 
 [2021-02-24 15:32:21+0000] WARN    com.zaxxer.hikari.pool.HikariPool - HikariPool-1 - Thread starvation or clock leap detected (housekeeper delta=1m37s705ms416µs741ns). 
 [2021-02-24 15:34:55+0000] INFO    inventory-processing - Report 'era-afd88dc6-14ef-4782-b50f-e1db3f442449.ocs' for node 'era.ad.cullinanstudio.com' [afd88dc6-14ef-4782-b50f-e1db3f442449] (signature:certified) processed in 11 minutes, 1 second and 965 milliseconds ms 
 [2021-02-24 15:34:55+0000] WARN    com.zaxxer.hikari.pool.HikariPool - HikariPool-1 - Thread starvation or clock leap detected (housekeeper delta=5m37s929ms914µs588ns). 
 [2021-02-24 15:34:56+0000] INFO    inventory-processing - Report 'cs-lxd-843bde7c-7c54-45ea-aafa-545375fa1763.ocs' for node 'cs-lxd.ad.cullinanstudio.com' [843bde7c-7c54-45ea-aafa-545375fa1763] (signature:certified) processed in 1 second and 263 milliseconds ms 
 [2021-02-24 15:39:26+0000] ERROR report.cache - Error when updating compliance cache for nodes: [83df0302-c646-48d4-960c-206eb7d6e76b, 6a7065bd-2857-44a5-a793-299aaf73f877, 64a9fb85-eb95-4e77-b473-bb36d6c755ce, 87257904-cd38-4f26-8bb7-86a95b616b31, 2f8a8a3e-e61e-493f-845b-411efd5e5cc1, eb746532-5c6d-41e9-a2cd-9cb5c359f754, dc4e6ebb-facd-4782-92e7-507c389c1d55, 98524e4d-bbe9-4cc5-a73a-6221ad316000, bc1f48e7-40ed-424f-864e-a36b0f600823, d4632740-3096-4a0f-a960-c58edcfe153f, 3fa39ba8-6134-4d15-a19c-47757443a3e6, ca75c6a1-3dcf-4f1c-a940-458d41955731, b01da0ae-a85e-43c6-90e2-0a97f6b962a9, 431d6b5b-3a63-491b-a545-dcd00149e74a, 0164debd-801f-4dbf-8286-b91f961db385, 012a5fa4-5cae-4a0f-aff0-8cef73605914, 68bfec3d-3148-48da-8936-00d6c6353bb8, 9d7a119e-dae5-401e-bd59-955189442d6e, 801ffa9b-da14-4f3a-9d79-7e25fff83a8b, 68eac959-2a13-48ff-9743-fb189da3751d, dd5da446-4465-4412-898d-d5fad325b057, 29bcf98b-7a98-4fd9-9c65-ca6f558523a1, 72d7d23a-49c3-40a9-b19e-ccca54f977b4, 1c4f8f91-8acf-4041-9182-15b47162c91f, ea1c1a5e-b035-4b30-af11-343d1ca80ac6, 16312315-eaf7-48f2-a738-2a8cc1a18277, 153b1a15-d18a-4382-99c5-cfe28c32040c, e6c5b022-531d-4ac7-ab42-074401be8b29, root, 6abcaf43-a53f-495c-92d9-092ae732f8b7, ebf278c9-18ae-4dc3-a860-0d4db34c0bed, 8651f72c-e34b-4fe9-9fc1-299faeb4f624, a32f001d-1eaf-41ad-bf32-e548e0f796b5, fdd9476f-7b5e-44f1-a0ad-71578250eacc, 900ecf39-cfd5-44e0-9d1d-1d539694fd05, 510bda45-631a-4708-a98e-5db91ee31f49, 7e9d00c1-949c-4666-b102-1bb3d7cb9d87, 193e9378-731b-4246-9a69-251b05f537c8, fff40e4d-694f-485e-b2c0-bacc43b8b482, 020dfd93-295d-408f-b934-34cee86d882f, 3ce75c4f-b041-4c15-877b-e3d441d83bd8, ad063ee1-6a7e-4edc-a028-f36b76cc17ea, bad452a7-36e7-4956-b29b-d2f58d2ad547, e27a8c6e-c5f2-480f-a7c7-05b1e271e078, 4b7b1e60-ee4d-44db-9409-f12b9f9de751, 0f18bc42-e1cc-4ee4-a0ef-b376eef08806, ef503e99-b19d-4a4a-aadc-fd57c64fcfa5, e2f589ae-d1c1-4a99-b67c-57d439f0efed, 2da6b244-019b-4cc4-831e-f0aa0d7c7eba, eac1ddac-df1a-4f18-8760-053a29b3ec57, f80fb2df-107a-4b1c-a1a7-af935b3d6dd2, 93b2bf80-9e3f-47ee-8a45-f4a570f5ea00, 39461107-981b-4e04-aabc-468d9f7cf7f0, afd88dc6-14ef-4782-b50f-e1db3f442449, a820f4cb-5526-4556-826b-d0c9f5f9881e, f6d08522-9cdf-4c40-8845-ef8655db964e, 292d1824-6688-422f-8cff-5cab708ab3f3, 801eafab-0542-4c6a-842b-956adcab89f4, 0176c790-c2f6-4de9-b231-d1b1e23285ac, 00404802-6de4-4920-8cbd-7471670d4392, 1a12816a-b2c4-4d68-a9bd-f371223176d0, 88f6ae7c-a27a-46ee-b410-e0e7fceb0822, 9f439b06-3f86-4828-8161-e5265d201239, acf230f8-914f-4c53-a061-fc3e1994bf92, 6a2f7f67-2910-46ad-9652-ddcc30a3ba67, 8f20e32d-4d4e-4e72-a3fa-184e0a9bfe5c, 31c74e40-5b60-4b79-a341-975a9bb6832b, 106e41c6-2098-4fa0-8305-5f7f2c99245a, 99cc43a8-8b0a-4ffb-844f-65af399097f2, 84b7e2eb-1bf6-4c01-bd9d-e83628fcb372, 6d794d33-9819-4c90-94bd-83e9546c60f6, c6bafff0-fda3-46f2-936e-11a11da8500d, 292e684f-a0f6-4e3c-ab41-860874800576, 0eab4144-d059-43f5-8a96-9779448a4eae, 4e29d7c7-b6bf-4b6b-949b-78a842ab4c26, f87c6d7d-2eec-4ef9-95d2-7a7bc4e80c98, 1a028fe7-ecfd-43c7-b34c-074275e5de7f, 04f67535-18e2-44bc-8343-8bd154bbe195, aa8f328a-f550-4767-b74d-6f47893d3efe, 91a299f2-0723-46a2-ba10-3116878d119e, e9932b2f-16bf-4e35-a7f2-ed04e2764ca6, 79782fa3-28a5-4d6a-bf5c-528adb5bc4e7, 71f11978-ba06-48bb-87bd-ab0c0baf1e1a, 0be487b0-e2dd-4e60-a2e9-86e831a7b8d3, 97a2ada6-6630-4f3a-87ed-694bb00be07d, 23c67ca3-6aae-4e26-90db-07a849490eb7, 232773fc-d6c4-4b95-ae20-fe2d15cbdd7e, 8c422b5d-a571-41be-b9c0-1bf54efb1ca9, 0bc1f85d-026a-4ef2-bb92-a204f95593e6, d93a4b3f-7739-4e3e-a3f4-f0fa1638c99d, f0f7072c-bd1f-48e9-b5ed-27143cad87e3, cbb05021-3ad9-4639-ae2b-cc89d15359f9, 4d2857dd-1b97-4242-ac78-3901d06959d1, 9336448a-3171-4da6-8d7e-8111aaa86b03, 8046ff06-bb9c-4fc2-ad9d-aa89fcab87e1, 283cc90a-7397-49c9-a668-16664f6eb83f, 6906cbf2-832f-4443-8b16-f8fdbdc454d2, 73134710-f1be-44be-8358-791a01cb5cdc, fcf0fa40-d809-4afa-8f54-1163ff33acab, b51dd2a6-a9c3-4c13-b00d-0185acb3cc11, d8d4680f-61b3-4b38-bb2b-f049a27ef39e, 3cfa5756-08a7-4321-9684-8f40f4f33258, 50641570-753c-49a2-83d9-fc29cf06ab4c, adf8aa38-7cdf-4502-a8de-3632bfeadbc7, 8dbf051d-7a95-4e80-9625-1f8df5036b90, b6775c30-8c93-4dd3-a1b2-c3c6a1fae5fc, 774934cf-4014-4ff5-ad80-ab19467720a9, 0e2e1829-e56c-4e63-9580-27e1db0d5992, e908812a-f8d8-45c7-9149-60df0069b772, a57c75c5-7fe7-4550-8d17-1a291430db5e, 890bfdfe-8564-4ec6-9143-3cfe3a7f3bb7, 4ee13920-8ff7-4f0b-ba87-d98cc47b4420, b5cc63b7-4fc6-43da-b7d4-2ae6087d2c92, fdfe3b9d-eb1c-4d45-998a-344e284e0134, 5f10e7a2-c9ce-4410-91c7-577ca9e688d7, 83420577-99d5-4c12-8332-c46c9d2d8cd4, ad4cafd6-29a3-4905-8a46-8cb62c3aac47, f542e332-18e7-45c7-a647-6819341250c4, ea547246-2597-49e8-abc0-b06c426a4271, 45a4be80-0758-4b15-b331-74582b1c6704, ed17924f-3385-4afe-86a7-73ff06640b75, 70027876-60e7-41b3-9f75-c9329153e8d6, 56a99b2e-3020-4d3b-a8bc-ace60b1a4327, 2da1f24b-e5ba-49e0-a405-46208ee2f32d, bbb33619-df86-435d-bda3-8c1d6a6e5e89, 7c9a0647-bc50-4575-80b3-eec19848fff5, e0c854fa-50ec-41ce-9bf3-614b9adc25ec, 88031eca-e2e7-450a-af85-839822971164, 08e0ca64-1e37-41a0-99d1-120c94ae1c91, 7afa0b5e-07e0-4238-9707-e6361414f065, f1dfa384-00bd-4643-be7d-462395f5e3f6, 37eb69d5-2d19-48c9-97d6-29b27dec299c, 843bde7c-7c54-45ea-aafa-545375fa1763, 78d79f85-c368-47a3-b231-5b0783aea27b, b440bb48-e214-4932-a3f9-160a79417ca3, 65266c5d-0108-4fe0-99a1-f824c470e0b5, 95caeed3-cda4-4f4c-b9b9-2e5c7a97ab6b, 79125b40-2f81-42c4-8383-bc8f012db906, 325676e7-dda7-473e-afa2-d6d3d09e7217, 465844a0-6b4e-41d8-bc63-f7780853619a, b0b12eb4-ffa6-4fb7-ab6f-eeb1f085a66e, 37c6ccd0-14b6-4e7b-a1ef-5b5fde24d1be, 8b7128a2-3f50-4e1f-b92b-fb52a081e408, f33caf20-fd02-4e01-b7e7-138e0518d1b6, aafb541c-e0d0-4345-8d5c-54856f22eec1, d585f3ad-297f-4b97-9d56-58920657636a, aca34e01-3e03-46f3-9a4b-ed653af6416a, 04cb40de-0bd5-4d51-bc15-e6f527894409, 537fea5d-58f6-4d69-97ad-eabbdc06e0f9, a2ebface-5ad6-4f08-ace9-332c3a249636, 168d953c-32e7-452a-b654-7f70ae80dca9, e31ae0d9-a864-4ee1-b462-a81a79236445, 501e4d17-e443-478f-ba42-11ae299ed391, 8befd795-d361-4e1f-bef8-69af8167780a, 8d5c514c-c8a0-4a26-b8e0-0cc79fb1e27e, 47759878-2d1b-4f20-b504-6037fb817034, b8d40ac5-2943-4fff-9c2c-241048a5bf05, 18b3ca8c-c3e1-4e48-994a-e52bc5f4ba2f, 12ee5b1a-c0eb-4a7f-b87c-d229b5c631b9, 0750cc88-0cea-4276-bd86-3616dfb8bb04, 68ca441c-18ad-46ca-a46b-8cf0c6373bc4, beb99eda-275c-452c-a53e-2d35c22b7da3, 6e7b9aef-250e-41dc-afb5-c78d61e19f30, 49d4be0c-887d-458f-b97b-3e1d44783ffd, 8cee109d-f58f-4d1d-8666-15d6b03df745, cc90310c-c8b6-4280-8613-29b9ceaab0fa, a8d1cf92-9e51-4253-a848-de542607e524, 7744604b-f46f-4656-8217-090cd6042603, b5d9a984-2077-4430-9d73-85245e4efca4, a53426ce-4dcb-4278-b6c2-c4d1fffe0b74, f9867bcc-2b1f-4e3e-a75d-a638768df723, 843803c8-9916-4709-bc8d-e029cd7d4604, d2868f4e-7a30-4910-af9c-f2c4eccd399b]: Unexpected: BackendException: Error during search ou=Application Properties,cn=rudder-configuration SUB: A client-side timeout was encountered while waiting 300000ms for a response to search request with message ID 288, base DN 'ou=Application Properties,cn=rudder-configuration', scope SUB, and filter '(objectClass=property)' from server localhost:389.; cause was: com.unboundid.ldap.sdk.LDAPSearchException: A client-side timeout was encountered while waiting 300000ms for a response to search request with message ID 288, base DN 'ou=Application Properties,cn=rudder-configuration', scope SUB, and filter '(objectClass=property)' from server localhost:389.  
  -> com.normation.ldap.sdk.RoLDAPConnection.$anonfun$search$1(LDAPConnection.scala:321) 
 [2021-02-24 15:39:26+0000] WARN    explain_compliance.2da6b244-019b-4cc4-831e-f0aa0d7c7eba - Received a run at 2021-02-24T15:20:55.000Z for node '2da6b244-019b-4cc4-831e-f0aa0d7c7eba' configId '20200826-084123-9fa3d540' which is not known by Rudder, and that node should be sending reports for configId 20210224-115717-25e62ed 
 [2021-02-24 15:39:26+0000] WARN    explain_compliance.292d1824-6688-422f-8cff-5cab708ab3f3 - Received a run at 2021-02-24T15:24:19.000Z for node '292d1824-6688-422f-8cff-5cab708ab3f3' configId '20200826-100925-ac5525b9' which is not known by Rudder, and that node should be sending reports for configId 20210224-115717-fc114038 
 .... 
 </pre>

Back