Project

General

Profile

Actions
Copy link

User story #1988

closed

When the postgresql database is restarted, the application is not usable anymore

Added by Nicolas CHARLES about 13 years ago. Updated over 12 years ago.

Status:
Released
Priority:
4
Assignee:
François ARMAND
Category:
Architecture - Code maintenance
Target version:
2.4.0~alpha7
Pull Request:
UX impact:
Suggestion strength:
User visibility:
Effort required:
Name check:
Fix check:
Regression:

Description

It seems the connctionPool is not trying to invalidate old connection and create new ones

The stacktrace shows :

15:51:04.423 [pool-1-thread-14] ERROR net.liftweb.actor.ActorLogger - Actor threw an exception
org.springframework.dao.DataAccessResourceFailureException: PreparedStatementCallback; SQL []; FATAL: terminating connection due to administrator command; nested exception is org.postgresql.util.PSQLException: FATAL: terminating connection due to administrator command
at org.springframework.jdbc.support.SQLStateSQLExceptionTranslator.doTranslate(SQLStateSQLExceptionTranslator.java:104) ~[spring-jdbc-3.0.5.RELEASE.jar:3.0.5.RELEASE]
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:72) ~[spring-jdbc-3.0.5.RELEASE.jar:3.0.5.RELEASE]
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:80) ~[spring-jdbc-3.0.5.RELEASE.jar:3.0.5.RELEASE]
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:80) ~[spring-jdbc-3.0.5.RELEASE.jar:3.0.5.RELEASE]
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:602) ~[spring-jdbc-3.0.5.RELEASE.jar:3.0.5.RELEASE]
at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:842) ~[spring-jdbc-3.0.5.RELEASE.jar:3.0.5.RELEASE]
at com.normation.rudder.repository.jdbc.EventLogJdbcRepository.saveEventLog(EventLogJdbcRepository.scala:84) ~[EventLogJdbcRepository.class:na]
at com.normation.rudder.services.log.EventLogServiceImpl.saveEventLog(EventLogServiceImpl.scala:52) ~[EventLogServiceImpl.class:na]
at com.normation.rudder.batch.AsyncDeploymentAgent$$anonfun$lowPriority$1.apply(AsyncDeploymentAgent.scala:124) ~[AsyncDeploymentAgent$$anonfun$lowPriority$1.class:na]
at com.normation.rudder.batch.AsyncDeploymentAgent$$anonfun$lowPriority$1.apply(AsyncDeploymentAgent.scala:112) ~[AsyncDeploymentAgent$$anonfun$lowPriority$1.class:na]
at scala.PartialFunction$$anon$1.apply(PartialFunction.scala:46) ~[na:na]
at net.liftweb.actor.LiftActor$class.execTranslate(LiftActor.scala:441) ~[lift-actor_2.9.0-1-2.4-M3.1.jar:2.4-M3.1]
at com.normation.rudder.batch.AsyncDeploymentAgent.execTranslate(AsyncDeploymentAgent.scala:83) [AsyncDeploymentAgent.class:na]
at net.liftweb.actor.SpecializedLiftActor$class.net$liftweb$actor$SpecializedLiftActor$$proc2(LiftActor.scala:289) [lift-actor_2.9.0-1-2.4-M3.1.jar:2.4-M3.1]
at net.liftweb.actor.SpecializedLiftActor$$anonfun$net$liftweb$actor$SpecializedLiftActor$$processMailbox$1.apply(LiftActor.scala:210) [lift-actor_2.9.0-1-2.4-M3.1.jar:2.4-M3.1]
at net.liftweb.actor.SpecializedLiftActor$$anonfun$net$liftweb$actor$SpecializedLiftActor$$processMailbox$1.apply(LiftActor.scala:210) [lift-actor_2.9.0-1-2.4-M3.1.jar:2.4-M3.1]
at net.liftweb.actor.SpecializedLiftActor$class.around(LiftActor.scala:224) [lift-actor_2.9.0-1-2.4-M3.1.jar:2.4-M3.1]
at com.normation.rudder.batch.AsyncDeploymentAgent.around(AsyncDeploymentAgent.scala:83) [AsyncDeploymentAgent.class:na]
at net.liftweb.actor.SpecializedLiftActor$class.net$liftweb$actor$SpecializedLiftActor$$processMailbox(LiftActor.scala:209) [lift-actor_2.9.0-1-2.4-M3.1.jar:2.4-M3.1]
at net.liftweb.actor.SpecializedLiftActor$$anonfun$2$$anonfun$apply$mcV$sp$1.apply$mcV$sp(LiftActor.scala:173) [lift-actor_2.9.0-1-2.4-M3.1.jar:2.4-M3.1]
at net.liftweb.actor.LAScheduler$$anonfun$9$$anon$2$$anon$3.run(LiftActor.scala:64) [lift-actor_2.9.0-1-2.4-M3.1.jar:2.4-M3.1]
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [na:1.6.0_25]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [na:1.6.0_25]
at java.lang.Thread.run(Thread.java:662) [na:1.6.0_25]
Caused by: org.postgresql.util.PSQLException: FATAL: terminating connection due to administrator command
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2077) ~[postgresql-8.4-702.jdbc4.jar:na]
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1810) ~[postgresql-8.4-702.jdbc4.jar:na]
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257) ~[postgresql-8.4-702.jdbc4.jar:na]
at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:498) ~[postgresql-8.4-702.jdbc4.jar:na]
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:386) ~[postgresql-8.4-702.jdbc4.jar:na]
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeUpdate(AbstractJdbc2Statement.java:332) ~[postgresql-8.4-702.jdbc4.jar:na]
at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105) ~[commons-dbcp-1.4.jar:1.4]
at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105) ~[commons-dbcp-1.4.jar:1.4]
at org.springframework.jdbc.core.JdbcTemplate$3.doInPreparedStatement(JdbcTemplate.java:844) ~[spring-jdbc-3.0.5.RELEASE.jar:3.0.5.RELEASE]
at org.springframework.jdbc.core.JdbcTemplate$3.doInPreparedStatement(JdbcTemplate.java:1) ~[spring-jdbc-3.0.5.RELEASE.jar:3.0.5.RELEASE]
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:586) ~[spring-jdbc-3.0.5.RELEASE.jar:3.0.5.RELEASE]
... 19 common frames omitted

Actions
Copy link
 #1

Updated by Jonathan CLARKE almost 13 years ago

  • Target version set to 2.4.0~alpha3
Actions
Copy link
 #2

Updated by Jonathan CLARKE almost 13 years ago

  • Target version changed from 2.4.0~alpha3 to 2.4.0~alpha4
Actions
Copy link
 #3

Updated by Jonathan CLARKE almost 13 years ago

  • Priority changed from N/A to 4
Actions
Copy link
 #4

Updated by François ARMAND almost 13 years ago

  • Target version changed from 2.4.0~alpha4 to 2.4.0~alpha5
Actions
Copy link
 #5

Updated by Jonathan CLARKE almost 13 years ago

  • Target version changed from 2.4.0~alpha5 to 2.4.0~alpha6
Actions
Copy link
 #6

Updated by Jonathan CLARKE over 12 years ago

  • Target version changed from 2.4.0~alpha6 to 2.4.0~alpha7
Actions
Copy link
 #7

Updated by François ARMAND over 12 years ago

  • Status changed from New to In progress
  • Assignee set to François ARMAND

The first step is to at least gracefully handle JDBC exception.

Actions
Copy link
 #8

Updated by François ARMAND over 12 years ago

Hum, there is too many jdbcTemplate call to wrap them all now... That will lead to surprises, so that will tracked here: #2438

Actions
Copy link
 #10

Updated by François ARMAND over 12 years ago

Oups,bad pooling driver: http://commons.apache.org/dbcp/configuration.html see validationQuery

Actions
Copy link
 #11

Updated by François ARMAND over 12 years ago

  • Status changed from In progress to Pending technical review
  • % Done changed from 0 to 100
Actions
Copy link
 #12

Updated by Nicolas CHARLES over 12 years ago

  • Status changed from Pending technical review to Released

This looks highly valid, thank you Francois

Actions
Copy link
 #13

Updated by Jonathan CLARKE over 12 years ago

  • Tracker changed from Bug to User story
  • Status changed from Released to 10

I would like to validate this behaviour on a production-like environment, to see how it should be documented and explained - so changing to a user story (it sort of is - "Jim shouldn't have to call Normation to get Rudder to work after restarting his PSQL server" :) ) to add a functional review step.

Actions
Copy link
 #14

Updated by Nicolas CHARLES over 12 years ago

  • Status changed from 10 to In progress
  • % Done changed from 100 to 50

Actually, this modification breaks at the initial startup of the application (when the database is empty), for there are no entries in the eventlog database

Actions
Copy link
 #15

Updated by Nicolas CHARLES over 12 years ago

I suggest this query :
SELECT tables.table_name FROM information_schema.tables WHERE lower(table_name) = 'eventlog';

Actions
Copy link
 #16

Updated by Nicolas CHARLES over 12 years ago

  • Status changed from In progress to Pending technical review
  • % Done changed from 50 to 100
Actions
Copy link
 #17

Updated by Nicolas CHARLES over 12 years ago

  • Assignee changed from François ARMAND to Nicolas CHARLES
Actions
Copy link
 #18

Updated by François ARMAND over 12 years ago

  • Status changed from Pending technical review to 10

The proposed modification is ok

Actions
Copy link
 #19

Updated by Jonathan CLARKE over 12 years ago

OK, this works when the DB is stopped then restarted. However, error handling when the DB is unavailable is still non functional.

Actions
Copy link
 #20

Updated by Jonathan CLARKE over 12 years ago

  • Status changed from 10 to Released
  • Assignee changed from Nicolas CHARLES to François ARMAND

Oops, forgot to actually close this one.

Actions
Copy link

Also available in: Atom PDF