Project

General

Profile

Actions

Bug #6007

closed

Error in the reports archiving process cause log explosion

Added by François ARMAND over 9 years ago. Updated about 9 years ago.

Status:
Released
Priority:
1
Category:
Performance and scalability
Target version:
Severity:
UX impact:
User visibility:
Effort required:
Priority:
Name check:
Fix check:
Regression:

Description

If there is an error during the archiving of reports, the process is restarted immediatelly, causing log size explosion.

Ex:

12-16 00:19:04] INFO  report - Reports database: Automatic archive started for all reports before 2014-11-16 00:19
[2014-12-16 00:19:04] ERROR com.normation.rudder.repository.jdbc.ReportsJdbcRepository - Could not delete entries in the database, cause is StatementCallback; uncategorized SQLException for SQL [vacuum ruddersysevents]; SQL state [XX001]; error code [0]; ERROR: invalid page header in block 8779 of relation base/16385/24921; nested exception is org.postgresql.util.PSQLException: ERROR: invalid page header in block 8779 of relation base/16385/24921
[2014-12-16 00:19:04] ERROR report - Reports database: Error while processing database deleting, cause is: Failure(Could not delete entries in the database, cause is StatementCallback; uncategorized SQLException for SQL [vacuum ruddersysevents]; SQL state [XX001]; error code [0]; ERROR: invalid page header in block 8779 of relation base/16385/24921; nested exception is org.postgresql.util.PSQLException: ERROR: invalid page header in block 8779 of relation base/16385/24921,Full(org.springframework.jdbc.UncategorizedSQLException: StatementCallback; uncategorized SQLException for SQL [vacuum ruddersysevents]; SQL state [XX001]; error code [0]; ERROR: invalid page header in block 8779 of relation base/16385/24921; nested exception is org.postgresql.util.PSQLException: ERROR: invalid page header in block 8779 of relation base/16385/24921),Empty) 
[2014-12-16 00:19:04] ERROR report - Reports database: Relaunching automatic deleting process for all reports before 2014-09-17 00:19
[2014-12-16 00:19:04] INFO  report - Reports database: Automatic delete started for all reports before 2014-09-17 00:19
[2014-12-16 00:19:04] ERROR com.normation.rudder.repository.jdbc.ReportsJdbcRepository - Could not archive entries in the database, cause is StatementCallback; uncategorized SQLException for SQL [vacuum ruddersysevents]; SQL state [XX001]; error code [0]; ERROR: invalid page header in block 8779 of relation base/16385/24921; nested exception is org.postgresql.util.PSQLException: ERROR: invalid page header in block 8779 of relation base/16385/24921
[2014-12-16 00:19:04] ERROR report - Reports database: Error while processing database archiving, cause is: Failure(Could not archive entries in the database, cause is StatementCallback; uncategorized SQLException for SQL [vacuum ruddersysevents]; SQL state [XX001]; error code [0]; ERROR: invalid page header in block 8779 of relation base/16385/24921; nested exception is org.postgresql.util.PSQLException: ERROR: invalid page header in block 8779 of relation base/16385/24921,Full(org.springframework.jdbc.UncategorizedSQLException: StatementCallback; uncategorized SQLException for SQL [vacuum ruddersysevents]; SQL state [XX001]; error code [0]; ERROR: invalid page header in block 8779 of relation base/16385/24921; nested exception is org.postgresql.util.PSQLException: ERROR: invalid page header in block 8779 of relation base/16385/24921),Empty) 
[2014-12-16 00:19:04] ERROR report - Reports database: Relaunching automatic archiving process for all reports before 2014-11-16 00:19
[2014-12-16 00:19:04] INFO  report - Reports database: Automatic archive started for all reports before 2014-11-16 00:19
[2014-12-16 00:19:04] ERROR com.normation.rudder.repository.jdbc.ReportsJdbcRepository - Could not delete entries in the database, cause is StatementCallback; uncategorized SQLException for SQL [vacuum ruddersysevents]; SQL state [XX001]; error code [0]; ERROR: invalid page header in block 8779 of relation base/16385/24921; nested exception is org.postgresql.util.PSQLException: ERROR: invalid page header in block 8779 of relation base/16385/24921
[2014-12-16 00:19:04] ERROR report - Reports database: Error while processing database deleting, cause is: Failure(Could not delete entries in the database, cause is StatementCallback; uncategorized SQLException for SQL [vacuum ruddersysevents]; SQL state [XX001]; error code [0]; ERROR: invalid page header in block 8779 of relation base/16385/24921; nested exception is org.postgresql.util.PSQLException: ERROR: invalid page header in block 8779 of relation base/16385/24921,Full(org.springframework.jdbc.UncategorizedSQLException: StatementCallback; uncategorized SQLException for SQL [vacuum ruddersysevents]; SQL state [XX001]; error code [0]; ERROR: invalid page header in block 8779 of relation base/16385/24921; nested exception is org.postgresql.util.PSQLException: ERROR: invalid page header in block 8779 of relation base/16385/24921),Empty) 
[2014-12-16 00:19:04] ERROR report - Reports database: Relaunching automatic deleting process for all reports before 2014-09-17 00:19
[2014-12-16 00:19:04] INFO  report - Reports database: Automatic delete started for all reports before 2014-09-17 00:19
[2014-12-16 00:19:04] ERROR com.normation.rudder.repository.jdbc.ReportsJdbcRepository - Could not archive entries in the database, cause is StatementCallback; uncategorized SQLException for SQL [vacuum ruddersysevents]; SQL state [XX001]; error code [0]; ERROR: invalid page header in block 8779 of relation base/16385/24921; nested exception is org.postgresql.util.PSQLException: ERROR: invalid page header in block 8779 of relation base/16385/24921
[2014-12-16 00:19:04] ERROR report - Reports database: Error while processing database archiving, cause is: Failure(Could not archive entries in the database, cause is StatementCallback; uncategorized SQLException for SQL [vacuum ruddersysevents]; SQL state [XX001]; error code [0]; ERROR: invalid page header in block 8779 of relation base/16385/24921; nested exception is org.postgresql.util.PSQLException: ERROR: invalid page header in block 8779 of relation base/16385/24921,Full(org.springframework.jdbc.UncategorizedSQLException: StatementCallback; uncategorized SQLException for SQL [vacuum ruddersysevents]; SQL state [XX001]; error code [0]; ERROR: invalid page header in block 8779 of relation base/16385/24921; nested exception is org.postgresql.util.PSQLException: ERROR: invalid page header in block 8779 of relation base/16385/24921),Empty) 
[2014-12-16 00:19:04] ERROR report - Reports database: Relaunching automatic archiving process for all reports before 2014-11-16 00:19
[2014-12-16 00:19:04] INFO  report - Reports database: Automatic archive started for all reports before 2014-11-16 00:19
[2014-12-16 00:19:04] ERROR com.normation.rudder.repository.jdbc.ReportsJdbcRepository - Could not delete entries in the database, cause is StatementCallback; uncategorized SQLException for SQL [vacuum ruddersysevents]; SQL state [XX001]; error code [0]; ERROR: invalid page header in block 8779 of relation base/16385/24921; nested exception is org.postgresql.util.PSQLException: ERROR: invalid page header in block 8779 of relation base/16385/24921
[2014-12-16 00:19:04] ERROR report - Reports database: Error while processing database deleting, cause is: Failure(Could not delete entries in the database, cause is StatementCallback; uncategorized SQLException for SQL [vacuum ruddersysevents]; SQL state [XX001]; error code [0]; ERROR: invalid page header in block 8779 of relation base/16385/24921; nested exception is org.postgresql.util.PSQLException: ERROR: invalid page header in block 8779 of relation base/16385/24921,Full(org.springframework.jdbc.UncategorizedSQLException: StatementCallback; uncategorized SQLException for SQL [vacuum ruddersysevents]; SQL state [XX001]; error code [0]; ERROR: invalid page header in block 8779 of relation base/16385/24921; nested exception is org.postgresql.util.PSQLException: ERROR: invalid page header in block 8779 of relation base/16385/24921),Empty) 
[2014-12-16 00:19:04] ERROR report - Reports database: Relaunching automatic deleting process for all reports before 2014-09-17 00:19
[2014-12-16 00:19:04] INFO  report - Reports database: Automatic delete started for all reports before 2014-09-17 00:19
[2014-12-16 00:19:04] ERROR com.normation.rudder.repository.jdbc.ReportsJdbcRepository - Could not archive entries in the database, cause is StatementCallback; uncategorized SQLException for SQL [vacuum ruddersysevents]; SQL state [XX001]; error code [0]; ERROR: invalid page header in block 8779 of relation base/16385/24921; nested exception is org.postgresql.util.PSQLException: ERROR: invalid page header in block 8779 of relation base/16385/24921
[2014-12-16 00:19:04] ERROR report - Reports database: Error while processing database archiving, cause is: Failure(Could not archive entries in the database, cause is StatementCallback; uncategorized SQLException for SQL [vacuum ruddersysevents]; SQL state [XX001]; error code [0]; ERROR: invalid page header in block 8779 of relation base/16385/24921; nested exception is org.postgresql.util.PSQLException: ERROR: invalid page header in block 8779 of relation base/16385/24921,Full(org.springframework.jdbc.UncategorizedSQLException: StatementCallback; uncategorized SQLException for SQL [vacuum ruddersysevents]; SQL state [XX001]; error code [0]; ERROR: invalid page header in block 8779 of relation base/16385/24921; nested exception is org.postgresql.util.PSQLException: ERROR: invalid page header in block 8779 of relation base/16385/24921),Empty) 
[2014-12-16 00:19:04] ERROR report - Reports database: Relaunching automatic archiving process for all reports before 2014-11-16 00:19
[2014-12-16 00:19:04] INFO  report - Reports database: Automatic archive started for all reports before 2014-11-16 00:19
[2014-12-16 00:19:04] ERROR com.normation.rudder.repository.jdbc.ReportsJdbcRepository - Could not delete entries in the database, cause is StatementCallback; uncategorized SQLException for SQL [vacuum ruddersysevents]; SQL state [XX001]; error code [0]; ERROR: invalid page header in block 8779 of relation base/16385/24921; nested exception is org.postgresql.util.PSQLException: ERROR: invalid page header in block 8779 of relation base/16385/24921
[2014-12-16 00:19:04] ERROR report - Reports database: Error while processing database deleting, cause is: Failure(Could not delete entries in the database, cause is StatementCallback; uncategorized SQLException for SQL [vacuum ruddersysevents]; SQL state [XX001]; error code [0]; ERROR: invalid page header in block 8779 of relation base/16385/24921; nested exception is org.postgresql.util.PSQLException: ERROR: invalid page header in block 8779 of relation base/16385/24921,Full(org.springframework.jdbc.UncategorizedSQLException: StatementCallback; uncategorized SQLException for SQL [vacuum ruddersysevents]; SQL state [XX001]; error code [0]; ERROR: invalid page header in block 8779 of relation base/16385/24921; nested exception is org.postgresql.util.PSQLException: ERROR: invalid page header in block 8779 of relation base/16385/24921),Empty) 
[2014-12-16 00:19:04] ERROR report - Reports database: Relaunching automatic deleting process for all reports before 2014-09-17 00:19
[2014-12-16 00:19:04] INFO  report - Reports database: Automatic delete started for all reports before 2014-09-17 00:19
[2014-12-16 00:19:04] ERROR com.normation.rudder.repository.jdbc.ReportsJdbcRepository - Could not archive entries in the database, cause is StatementCallback; uncategorized SQLException for SQL [vacuum ruddersysevents]; SQL state [XX001]; error code [0]; ERROR: invalid page header in block 8779 of relation base/16385/24921; nested exception is org.postgresql.util.PSQLException: ERROR: invalid page header in block 8779 of relation base/16385/24921
[2014-12-16 00:19:04] ERROR report - Reports database: Error while processing database archiving, cause is: Failure(Could not archive entries in the database, cause is StatementCallback; uncategorized SQLException for SQL [vacuum ruddersysevents]; SQL state [XX001]; error code [0]; ERROR: invalid page header in block 8779 of relation base/16385/24921; nested exception is org.postgresql.util.PSQLException: ERROR: invalid page header in block 8779 of relation base/16385/24921,Full(org.springframework.jdbc.UncategorizedSQLException: StatementCallback; uncategorized SQLException for SQL [vacuum ruddersysevents]; SQL state [XX001]; error code [0]; ERROR: invalid page header in block 8779 of relation base/16385/24921; nested exception is org.postgresql.util.PSQLException: ERROR: invalid page header in block 8779 of relation base/16385/24921),Empty) 
[2014-12-16 00:19:04] ERROR report - Reports database: Relaunching automatic archiving process for all reports before 2014-11-16 00:19
[2014-12-16 00:19:04] INFO  report - Reports database: Automatic archive started for all reports before 2014-11-16 00:19
....
....
....

I'm not sure restarting the process is a good idea, since it's almost sure that without an human intervention, it will just continue to fails miserably.

I propose to simply log an error and not do anything else.

Actions #1

Updated by François ARMAND over 9 years ago

  • Status changed from In progress to Pending technical review
  • Assignee changed from François ARMAND to Nicolas CHARLES
  • Pull Request set to https://github.com/Normation/rudder/pull/724
Actions #2

Updated by François ARMAND over 9 years ago

  • Status changed from Pending technical review to Pending release
  • % Done changed from 0 to 100
Actions #4

Updated by Vincent MEMBRÉ over 9 years ago

  • Target version changed from 2.10.8 to 2.10.9
Actions #5

Updated by Vincent MEMBRÉ about 9 years ago

  • Target version changed from 2.10.9 to 2.10.10
Actions #6

Updated by Vincent MEMBRÉ about 9 years ago

  • Status changed from Pending release to Released

This bug has been fixed in Rudder 2.10.10 and 2.11.7, which were these days.

Actions

Also available in: Atom PDF