Project

General

Profile

Actions

User story #1988

closed

When the postgresql database is restarted, the application is not usable anymore

Added by Nicolas CHARLES over 12 years ago. Updated almost 12 years ago.

Status:
Released
Priority:
4
Category:
Architecture - Code maintenance
Target version:
UX impact:
Suggestion strength:
User visibility:
Effort required:
Name check:
Fix check:
Regression:

Description

It seems the connctionPool is not trying to invalidate old connection and create new ones

The stacktrace shows :

15:51:04.423 [pool-1-thread-14] ERROR net.liftweb.actor.ActorLogger - Actor threw an exception
org.springframework.dao.DataAccessResourceFailureException: PreparedStatementCallback; SQL []; FATAL: terminating connection due to administrator command; nested exception is org.postgresql.util.PSQLException: FATAL: terminating connection due to administrator command
at org.springframework.jdbc.support.SQLStateSQLExceptionTranslator.doTranslate(SQLStateSQLExceptionTranslator.java:104) ~[spring-jdbc-3.0.5.RELEASE.jar:3.0.5.RELEASE]
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:72) ~[spring-jdbc-3.0.5.RELEASE.jar:3.0.5.RELEASE]
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:80) ~[spring-jdbc-3.0.5.RELEASE.jar:3.0.5.RELEASE]
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:80) ~[spring-jdbc-3.0.5.RELEASE.jar:3.0.5.RELEASE]
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:602) ~[spring-jdbc-3.0.5.RELEASE.jar:3.0.5.RELEASE]
at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:842) ~[spring-jdbc-3.0.5.RELEASE.jar:3.0.5.RELEASE]
at com.normation.rudder.repository.jdbc.EventLogJdbcRepository.saveEventLog(EventLogJdbcRepository.scala:84) ~[EventLogJdbcRepository.class:na]
at com.normation.rudder.services.log.EventLogServiceImpl.saveEventLog(EventLogServiceImpl.scala:52) ~[EventLogServiceImpl.class:na]
at com.normation.rudder.batch.AsyncDeploymentAgent$$anonfun$lowPriority$1.apply(AsyncDeploymentAgent.scala:124) ~[AsyncDeploymentAgent$$anonfun$lowPriority$1.class:na]
at com.normation.rudder.batch.AsyncDeploymentAgent$$anonfun$lowPriority$1.apply(AsyncDeploymentAgent.scala:112) ~[AsyncDeploymentAgent$$anonfun$lowPriority$1.class:na]
at scala.PartialFunction$$anon$1.apply(PartialFunction.scala:46) ~[na:na]
at net.liftweb.actor.LiftActor$class.execTranslate(LiftActor.scala:441) ~[lift-actor_2.9.0-1-2.4-M3.1.jar:2.4-M3.1]
at com.normation.rudder.batch.AsyncDeploymentAgent.execTranslate(AsyncDeploymentAgent.scala:83) [AsyncDeploymentAgent.class:na]
at net.liftweb.actor.SpecializedLiftActor$class.net$liftweb$actor$SpecializedLiftActor$$proc2(LiftActor.scala:289) [lift-actor_2.9.0-1-2.4-M3.1.jar:2.4-M3.1]
at net.liftweb.actor.SpecializedLiftActor$$anonfun$net$liftweb$actor$SpecializedLiftActor$$processMailbox$1.apply(LiftActor.scala:210) [lift-actor_2.9.0-1-2.4-M3.1.jar:2.4-M3.1]
at net.liftweb.actor.SpecializedLiftActor$$anonfun$net$liftweb$actor$SpecializedLiftActor$$processMailbox$1.apply(LiftActor.scala:210) [lift-actor_2.9.0-1-2.4-M3.1.jar:2.4-M3.1]
at net.liftweb.actor.SpecializedLiftActor$class.around(LiftActor.scala:224) [lift-actor_2.9.0-1-2.4-M3.1.jar:2.4-M3.1]
at com.normation.rudder.batch.AsyncDeploymentAgent.around(AsyncDeploymentAgent.scala:83) [AsyncDeploymentAgent.class:na]
at net.liftweb.actor.SpecializedLiftActor$class.net$liftweb$actor$SpecializedLiftActor$$processMailbox(LiftActor.scala:209) [lift-actor_2.9.0-1-2.4-M3.1.jar:2.4-M3.1]
at net.liftweb.actor.SpecializedLiftActor$$anonfun$2$$anonfun$apply$mcV$sp$1.apply$mcV$sp(LiftActor.scala:173) [lift-actor_2.9.0-1-2.4-M3.1.jar:2.4-M3.1]
at net.liftweb.actor.LAScheduler$$anonfun$9$$anon$2$$anon$3.run(LiftActor.scala:64) [lift-actor_2.9.0-1-2.4-M3.1.jar:2.4-M3.1]
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [na:1.6.0_25]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [na:1.6.0_25]
at java.lang.Thread.run(Thread.java:662) [na:1.6.0_25]
Caused by: org.postgresql.util.PSQLException: FATAL: terminating connection due to administrator command
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2077) ~[postgresql-8.4-702.jdbc4.jar:na]
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1810) ~[postgresql-8.4-702.jdbc4.jar:na]
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257) ~[postgresql-8.4-702.jdbc4.jar:na]
at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:498) ~[postgresql-8.4-702.jdbc4.jar:na]
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:386) ~[postgresql-8.4-702.jdbc4.jar:na]
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeUpdate(AbstractJdbc2Statement.java:332) ~[postgresql-8.4-702.jdbc4.jar:na]
at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105) ~[commons-dbcp-1.4.jar:1.4]
at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105) ~[commons-dbcp-1.4.jar:1.4]
at org.springframework.jdbc.core.JdbcTemplate$3.doInPreparedStatement(JdbcTemplate.java:844) ~[spring-jdbc-3.0.5.RELEASE.jar:3.0.5.RELEASE]
at org.springframework.jdbc.core.JdbcTemplate$3.doInPreparedStatement(JdbcTemplate.java:1) ~[spring-jdbc-3.0.5.RELEASE.jar:3.0.5.RELEASE]
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:586) ~[spring-jdbc-3.0.5.RELEASE.jar:3.0.5.RELEASE]
... 19 common frames omitted

Actions #1

Updated by Jonathan CLARKE about 12 years ago

  • Target version set to 2.4.0~alpha3
Actions #2

Updated by Jonathan CLARKE about 12 years ago

  • Target version changed from 2.4.0~alpha3 to 2.4.0~alpha4
Actions #3

Updated by Jonathan CLARKE about 12 years ago

  • Priority changed from N/A to 4
Actions #4

Updated by François ARMAND about 12 years ago

  • Target version changed from 2.4.0~alpha4 to 2.4.0~alpha5
Actions #5

Updated by Jonathan CLARKE about 12 years ago

  • Target version changed from 2.4.0~alpha5 to 2.4.0~alpha6
Actions #6

Updated by Jonathan CLARKE almost 12 years ago

  • Target version changed from 2.4.0~alpha6 to 2.4.0~alpha7
Actions #7

Updated by François ARMAND almost 12 years ago

  • Status changed from New to In progress
  • Assignee set to François ARMAND

The first step is to at least gracefully handle JDBC exception.

Actions #8

Updated by François ARMAND almost 12 years ago

Hum, there is too many jdbcTemplate call to wrap them all now... That will lead to surprises, so that will tracked here: #2438

Actions #10

Updated by François ARMAND almost 12 years ago

Oups,bad pooling driver: http://commons.apache.org/dbcp/configuration.html see validationQuery

Actions #11

Updated by François ARMAND almost 12 years ago

  • Status changed from In progress to Pending technical review
  • % Done changed from 0 to 100
Actions #12

Updated by Nicolas CHARLES almost 12 years ago

  • Status changed from Pending technical review to Released

This looks highly valid, thank you Francois

Actions #13

Updated by Jonathan CLARKE almost 12 years ago

  • Tracker changed from Bug to User story
  • Status changed from Released to 10

I would like to validate this behaviour on a production-like environment, to see how it should be documented and explained - so changing to a user story (it sort of is - "Jim shouldn't have to call Normation to get Rudder to work after restarting his PSQL server" :) ) to add a functional review step.

Actions #14

Updated by Nicolas CHARLES almost 12 years ago

  • Status changed from 10 to In progress
  • % Done changed from 100 to 50

Actually, this modification breaks at the initial startup of the application (when the database is empty), for there are no entries in the eventlog database

Actions #15

Updated by Nicolas CHARLES almost 12 years ago

I suggest this query :
SELECT tables.table_name FROM information_schema.tables WHERE lower(table_name) = 'eventlog';

Actions #16

Updated by Nicolas CHARLES almost 12 years ago

  • Status changed from In progress to Pending technical review
  • % Done changed from 50 to 100
Actions #17

Updated by Nicolas CHARLES almost 12 years ago

  • Assignee changed from François ARMAND to Nicolas CHARLES
Actions #18

Updated by François ARMAND almost 12 years ago

  • Status changed from Pending technical review to 10

The proposed modification is ok

Actions #19

Updated by Jonathan CLARKE almost 12 years ago

OK, this works when the DB is stopped then restarted. However, error handling when the DB is unavailable is still non functional.

Actions #20

Updated by Jonathan CLARKE almost 12 years ago

  • Status changed from 10 to Released
  • Assignee changed from Nicolas CHARLES to François ARMAND

Oops, forgot to actually close this one.

Actions

Also available in: Atom PDF