Rudder

I was testing how Rudder webapp reacted to the PostgreSQL server being unavailable.

I was running the webapp and browsing various pages, stopped the SQL server, continued browsing and got several "The server cannot be contacted at this time", restarted the SQL server, and then everything worked fine again (without restarting jetty).

However, during this period, at some point some promise generations were started. One returned an error saying "=> Exception caught during deployment process: Could not get JDBC Connection; nested exception is org.postgresql.util.PSQLException: Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.". Then the next one got stuck (it's been like that for 20 minutes). The log contains this, and I don't know how to stop it:

[2012-07-20 12:18:36] INFO  com.normation.rudder.services.servers.NodeConfigurationChangeDetectServiceImpl - Checking changes in node f13f6726-1fdc-4ddc-9563-186648636ba4
[2012-07-20 12:18:36] INFO  com.normation.rudder.services.servers.NodeConfigurationChangeDetectServiceImpl - Checking changes in node 0876521e-3c81-4775-85c7-5dd7f9d5d3da
[2012-07-20 12:18:36] INFO  com.normation.rudder.services.servers.NodeConfigurationChangeDetectServiceImpl - Checking changes in node 41affc17-e432-420c-8f64-b819e96c67fc
[2012-07-20 12:18:36] INFO  com.normation.rudder.services.servers.NodeConfigurationChangeDetectServiceImpl - Checking changes in node root
[2012-07-20 12:18:36] INFO  com.normation.rudder.services.servers.NodeConfigurationChangeDetectServiceImpl - Checking changes in node 06da3556-5204-4bd7-b3b0-fa5e7bcfbbea
[2012-07-20 12:18:48] INFO  com.normation.rudder.batch.AsyncDeploymentAgent - One automatic deployment process is already pending, ignoring new deployment request
[2012-07-20 12:19:10] ERROR net.liftweb.http.LiftRules - Exception being returned to browser when processing Req(List(F78934657014NTWFQT), Map(F78934657014NTWFQT -> List(06da3556-5204-4bd7-b3b0-fa5e7bcfbbea)), ParsePath(List(ajax_request, F78934657013K4MBPD, index),,true,true), /rudder, PostRequest, Full(application/x-www-form-urlencoded; charset=UTF-8))
org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is org.postgresql.util.PSQLException: Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
    at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80) ~[spring-jdbc-3.1.1.RELEASE.jar:3.1.1.RELEASE]
Caused by: org.postgresql.util.PSQLException: Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
    at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:136) ~[postgresql-8.4-702.jdbc4.jar:na]
Caused by: java.net.ConnectException: Connection refused
    at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.6.0_26]
psql: could not connect to server: Connection refused
    Is the server running on host "localhost" and accepting
    TCP/IP connections on port 5432?
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   366    0     0  100   366      0  56264 --:--:-- --:--:-- --:--:-- 61000
[2012-07-20 12:19:12] ERROR net.liftweb.http.LiftRules - Exception being returned to browser when processing Req(List(F78934657014NTWFQT), Map(F78934657014NTWFQT -> List(06da3556-5204-4bd7-b3b0-fa5e7bcfbbea)), ParsePath(List(ajax_request, F78934657013K4MBPD, index),,true,true), /rudder, PostRequest, Full(application/x-www-form-urlencoded; charset=UTF-8))
org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is org.postgresql.util.PSQLException: Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
    at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80) ~[spring-jdbc-3.1.1.RELEASE.jar:3.1.1.RELEASE]
Caused by: org.postgresql.util.PSQLException: Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
    at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:136) ~[postgresql-8.4-702.jdbc4.jar:na]
Caused by: java.net.ConnectException: Connection refused
    at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.6.0_26]
[2012-07-20 12:19:16] ERROR com.normation.rudder.batch.AsyncDeploymentAgent - Deployment error for process '10' at 2012/07/20 12:19:16: Exception caught during deployment process: Could not get JDBC Connection; nested exception is org.postgresql.util.PSQLException: Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is org.postgresql.util.PSQLException: Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
    at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80) ~[spring-jdbc-3.1.1.RELEASE.jar:3.1.1.RELEASE]
Caused by: org.postgresql.util.PSQLException: Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
    at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:136) ~[postgresql-8.4-702.jdbc4.jar:na]
Caused by: java.net.ConnectException: Connection refused
    at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.6.0_26]
[2012-07-20 12:19:16] ERROR net.liftweb.actor.ActorLogger - Actor threw an exception
org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is org.postgresql.util.PSQLException: Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
    at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80) ~[spring-jdbc-3.1.1.RELEASE.jar:3.1.1.RELEASE]
Caused by: org.postgresql.util.PSQLException: Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
    at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:136) ~[postgresql-8.4-702.jdbc4.jar:na]
Caused by: java.net.ConnectException: Connection refused
    at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.6.0_26]
[2012-07-20 12:19:16] ERROR net.liftweb.http.LiftRules - Exception being returned to browser when processing Req(List(F78934657047Q15ONW), Map(F78934657047Q15ONW -> List(06da3556-5204-4bd7-b3b0-fa5e7bcfbbea)), ParsePath(List(ajax_request, F78934657046J2J2JB, index),,true,true), /rudder, PostRequest, Full(application/x-www-form-urlencoded; charset=UTF-8))
org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is org.postgresql.util.PSQLException: Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
    at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80) ~[spring-jdbc-3.1.1.RELEASE.jar:3.1.1.RELEASE]
Caused by: org.postgresql.util.PSQLException: Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
    at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:136) ~[postgresql-8.4-702.jdbc4.jar:na]
Caused by: java.net.ConnectException: Connection refused
    at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.6.0_26]
[2012-07-20 12:19:18] ERROR net.liftweb.http.LiftRules - Exception being returned to browser when processing Req(List(F78934657047Q15ONW), Map(F78934657047Q15ONW -> List(06da3556-5204-4bd7-b3b0-fa5e7bcfbbea)), ParsePath(List(ajax_request, F78934657046J2J2JB, index),,true,true), /rudder, PostRequest, Full(application/x-www-form-urlencoded; charset=UTF-8))
org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is org.postgresql.util.PSQLException: Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
    at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80) ~[spring-jdbc-3.1.1.RELEASE.jar:3.1.1.RELEASE]
Caused by: org.postgresql.util.PSQLException: Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
    at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:136) ~[postgresql-8.4-702.jdbc4.jar:na]
Caused by: java.net.ConnectException: Connection refused
    at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.6.0_26]
[2012-07-20 12:36:00] INFO  com.normation.cfclerk.services.impl.TechniqueRepositoryImpl - Reloading technique library, found modified technique(s): distributePolicy/1.0
[2012-07-20 12:36:01] INFO  com.normation.rudder.services.policies.DeployOnTechniqueCallback - Technique library reloaded by user
[2012-07-20 12:36:01] INFO  com.normation.rudder.batch.AsyncDeploymentAgent - One automatic deployment process is already pending, ignoring new deployment request