Project

General

Profile

Actions

Bug #6007

closed

Error in the reports archiving process cause log explosion

Added by François ARMAND over 9 years ago. Updated about 9 years ago.

Status:
Released
Priority:
1
Category:
Performance and scalability
Target version:
Severity:
UX impact:
User visibility:
Effort required:
Priority:
Name check:
Fix check:
Regression:

Description

If there is an error during the archiving of reports, the process is restarted immediatelly, causing log size explosion.

Ex:

12-16 00:19:04] INFO  report - Reports database: Automatic archive started for all reports before 2014-11-16 00:19
[2014-12-16 00:19:04] ERROR com.normation.rudder.repository.jdbc.ReportsJdbcRepository - Could not delete entries in the database, cause is StatementCallback; uncategorized SQLException for SQL [vacuum ruddersysevents]; SQL state [XX001]; error code [0]; ERROR: invalid page header in block 8779 of relation base/16385/24921; nested exception is org.postgresql.util.PSQLException: ERROR: invalid page header in block 8779 of relation base/16385/24921
[2014-12-16 00:19:04] ERROR report - Reports database: Error while processing database deleting, cause is: Failure(Could not delete entries in the database, cause is StatementCallback; uncategorized SQLException for SQL [vacuum ruddersysevents]; SQL state [XX001]; error code [0]; ERROR: invalid page header in block 8779 of relation base/16385/24921; nested exception is org.postgresql.util.PSQLException: ERROR: invalid page header in block 8779 of relation base/16385/24921,Full(org.springframework.jdbc.UncategorizedSQLException: StatementCallback; uncategorized SQLException for SQL [vacuum ruddersysevents]; SQL state [XX001]; error code [0]; ERROR: invalid page header in block 8779 of relation base/16385/24921; nested exception is org.postgresql.util.PSQLException: ERROR: invalid page header in block 8779 of relation base/16385/24921),Empty) 
[2014-12-16 00:19:04] ERROR report - Reports database: Relaunching automatic deleting process for all reports before 2014-09-17 00:19
[2014-12-16 00:19:04] INFO  report - Reports database: Automatic delete started for all reports before 2014-09-17 00:19
[2014-12-16 00:19:04] ERROR com.normation.rudder.repository.jdbc.ReportsJdbcRepository - Could not archive entries in the database, cause is StatementCallback; uncategorized SQLException for SQL [vacuum ruddersysevents]; SQL state [XX001]; error code [0]; ERROR: invalid page header in block 8779 of relation base/16385/24921; nested exception is org.postgresql.util.PSQLException: ERROR: invalid page header in block 8779 of relation base/16385/24921
[2014-12-16 00:19:04] ERROR report - Reports database: Error while processing database archiving, cause is: Failure(Could not archive entries in the database, cause is StatementCallback; uncategorized SQLException for SQL [vacuum ruddersysevents]; SQL state [XX001]; error code [0]; ERROR: invalid page header in block 8779 of relation base/16385/24921; nested exception is org.postgresql.util.PSQLException: ERROR: invalid page header in block 8779 of relation base/16385/24921,Full(org.springframework.jdbc.UncategorizedSQLException: StatementCallback; uncategorized SQLException for SQL [vacuum ruddersysevents]; SQL state [XX001]; error code [0]; ERROR: invalid page header in block 8779 of relation base/16385/24921; nested exception is org.postgresql.util.PSQLException: ERROR: invalid page header in block 8779 of relation base/16385/24921),Empty) 
[2014-12-16 00:19:04] ERROR report - Reports database: Relaunching automatic archiving process for all reports before 2014-11-16 00:19
[2014-12-16 00:19:04] INFO  report - Reports database: Automatic archive started for all reports before 2014-11-16 00:19
[2014-12-16 00:19:04] ERROR com.normation.rudder.repository.jdbc.ReportsJdbcRepository - Could not delete entries in the database, cause is StatementCallback; uncategorized SQLException for SQL [vacuum ruddersysevents]; SQL state [XX001]; error code [0]; ERROR: invalid page header in block 8779 of relation base/16385/24921; nested exception is org.postgresql.util.PSQLException: ERROR: invalid page header in block 8779 of relation base/16385/24921
[2014-12-16 00:19:04] ERROR report - Reports database: Error while processing database deleting, cause is: Failure(Could not delete entries in the database, cause is StatementCallback; uncategorized SQLException for SQL [vacuum ruddersysevents]; SQL state [XX001]; error code [0]; ERROR: invalid page header in block 8779 of relation base/16385/24921; nested exception is org.postgresql.util.PSQLException: ERROR: invalid page header in block 8779 of relation base/16385/24921,Full(org.springframework.jdbc.UncategorizedSQLException: StatementCallback; uncategorized SQLException for SQL [vacuum ruddersysevents]; SQL state [XX001]; error code [0]; ERROR: invalid page header in block 8779 of relation base/16385/24921; nested exception is org.postgresql.util.PSQLException: ERROR: invalid page header in block 8779 of relation base/16385/24921),Empty) 
[2014-12-16 00:19:04] ERROR report - Reports database: Relaunching automatic deleting process for all reports before 2014-09-17 00:19
[2014-12-16 00:19:04] INFO  report - Reports database: Automatic delete started for all reports before 2014-09-17 00:19
[2014-12-16 00:19:04] ERROR com.normation.rudder.repository.jdbc.ReportsJdbcRepository - Could not archive entries in the database, cause is StatementCallback; uncategorized SQLException for SQL [vacuum ruddersysevents]; SQL state [XX001]; error code [0]; ERROR: invalid page header in block 8779 of relation base/16385/24921; nested exception is org.postgresql.util.PSQLException: ERROR: invalid page header in block 8779 of relation base/16385/24921
[2014-12-16 00:19:04] ERROR report - Reports database: Error while processing database archiving, cause is: Failure(Could not archive entries in the database, cause is StatementCallback; uncategorized SQLException for SQL [vacuum ruddersysevents]; SQL state [XX001]; error code [0]; ERROR: invalid page header in block 8779 of relation base/16385/24921; nested exception is org.postgresql.util.PSQLException: ERROR: invalid page header in block 8779 of relation base/16385/24921,Full(org.springframework.jdbc.UncategorizedSQLException: StatementCallback; uncategorized SQLException for SQL [vacuum ruddersysevents]; SQL state [XX001]; error code [0]; ERROR: invalid page header in block 8779 of relation base/16385/24921; nested exception is org.postgresql.util.PSQLException: ERROR: invalid page header in block 8779 of relation base/16385/24921),Empty) 
[2014-12-16 00:19:04] ERROR report - Reports database: Relaunching automatic archiving process for all reports before 2014-11-16 00:19
[2014-12-16 00:19:04] INFO  report - Reports database: Automatic archive started for all reports before 2014-11-16 00:19
[2014-12-16 00:19:04] ERROR com.normation.rudder.repository.jdbc.ReportsJdbcRepository - Could not delete entries in the database, cause is StatementCallback; uncategorized SQLException for SQL [vacuum ruddersysevents]; SQL state [XX001]; error code [0]; ERROR: invalid page header in block 8779 of relation base/16385/24921; nested exception is org.postgresql.util.PSQLException: ERROR: invalid page header in block 8779 of relation base/16385/24921
[2014-12-16 00:19:04] ERROR report - Reports database: Error while processing database deleting, cause is: Failure(Could not delete entries in the database, cause is StatementCallback; uncategorized SQLException for SQL [vacuum ruddersysevents]; SQL state [XX001]; error code [0]; ERROR: invalid page header in block 8779 of relation base/16385/24921; nested exception is org.postgresql.util.PSQLException: ERROR: invalid page header in block 8779 of relation base/16385/24921,Full(org.springframework.jdbc.UncategorizedSQLException: StatementCallback; uncategorized SQLException for SQL [vacuum ruddersysevents]; SQL state [XX001]; error code [0]; ERROR: invalid page header in block 8779 of relation base/16385/24921; nested exception is org.postgresql.util.PSQLException: ERROR: invalid page header in block 8779 of relation base/16385/24921),Empty) 
[2014-12-16 00:19:04] ERROR report - Reports database: Relaunching automatic deleting process for all reports before 2014-09-17 00:19
[2014-12-16 00:19:04] INFO  report - Reports database: Automatic delete started for all reports before 2014-09-17 00:19
[2014-12-16 00:19:04] ERROR com.normation.rudder.repository.jdbc.ReportsJdbcRepository - Could not archive entries in the database, cause is StatementCallback; uncategorized SQLException for SQL [vacuum ruddersysevents]; SQL state [XX001]; error code [0]; ERROR: invalid page header in block 8779 of relation base/16385/24921; nested exception is org.postgresql.util.PSQLException: ERROR: invalid page header in block 8779 of relation base/16385/24921
[2014-12-16 00:19:04] ERROR report - Reports database: Error while processing database archiving, cause is: Failure(Could not archive entries in the database, cause is StatementCallback; uncategorized SQLException for SQL [vacuum ruddersysevents]; SQL state [XX001]; error code [0]; ERROR: invalid page header in block 8779 of relation base/16385/24921; nested exception is org.postgresql.util.PSQLException: ERROR: invalid page header in block 8779 of relation base/16385/24921,Full(org.springframework.jdbc.UncategorizedSQLException: StatementCallback; uncategorized SQLException for SQL [vacuum ruddersysevents]; SQL state [XX001]; error code [0]; ERROR: invalid page header in block 8779 of relation base/16385/24921; nested exception is org.postgresql.util.PSQLException: ERROR: invalid page header in block 8779 of relation base/16385/24921),Empty) 
[2014-12-16 00:19:04] ERROR report - Reports database: Relaunching automatic archiving process for all reports before 2014-11-16 00:19
[2014-12-16 00:19:04] INFO  report - Reports database: Automatic archive started for all reports before 2014-11-16 00:19
[2014-12-16 00:19:04] ERROR com.normation.rudder.repository.jdbc.ReportsJdbcRepository - Could not delete entries in the database, cause is StatementCallback; uncategorized SQLException for SQL [vacuum ruddersysevents]; SQL state [XX001]; error code [0]; ERROR: invalid page header in block 8779 of relation base/16385/24921; nested exception is org.postgresql.util.PSQLException: ERROR: invalid page header in block 8779 of relation base/16385/24921
[2014-12-16 00:19:04] ERROR report - Reports database: Error while processing database deleting, cause is: Failure(Could not delete entries in the database, cause is StatementCallback; uncategorized SQLException for SQL [vacuum ruddersysevents]; SQL state [XX001]; error code [0]; ERROR: invalid page header in block 8779 of relation base/16385/24921; nested exception is org.postgresql.util.PSQLException: ERROR: invalid page header in block 8779 of relation base/16385/24921,Full(org.springframework.jdbc.UncategorizedSQLException: StatementCallback; uncategorized SQLException for SQL [vacuum ruddersysevents]; SQL state [XX001]; error code [0]; ERROR: invalid page header in block 8779 of relation base/16385/24921; nested exception is org.postgresql.util.PSQLException: ERROR: invalid page header in block 8779 of relation base/16385/24921),Empty) 
[2014-12-16 00:19:04] ERROR report - Reports database: Relaunching automatic deleting process for all reports before 2014-09-17 00:19
[2014-12-16 00:19:04] INFO  report - Reports database: Automatic delete started for all reports before 2014-09-17 00:19
[2014-12-16 00:19:04] ERROR com.normation.rudder.repository.jdbc.ReportsJdbcRepository - Could not archive entries in the database, cause is StatementCallback; uncategorized SQLException for SQL [vacuum ruddersysevents]; SQL state [XX001]; error code [0]; ERROR: invalid page header in block 8779 of relation base/16385/24921; nested exception is org.postgresql.util.PSQLException: ERROR: invalid page header in block 8779 of relation base/16385/24921
[2014-12-16 00:19:04] ERROR report - Reports database: Error while processing database archiving, cause is: Failure(Could not archive entries in the database, cause is StatementCallback; uncategorized SQLException for SQL [vacuum ruddersysevents]; SQL state [XX001]; error code [0]; ERROR: invalid page header in block 8779 of relation base/16385/24921; nested exception is org.postgresql.util.PSQLException: ERROR: invalid page header in block 8779 of relation base/16385/24921,Full(org.springframework.jdbc.UncategorizedSQLException: StatementCallback; uncategorized SQLException for SQL [vacuum ruddersysevents]; SQL state [XX001]; error code [0]; ERROR: invalid page header in block 8779 of relation base/16385/24921; nested exception is org.postgresql.util.PSQLException: ERROR: invalid page header in block 8779 of relation base/16385/24921),Empty) 
[2014-12-16 00:19:04] ERROR report - Reports database: Relaunching automatic archiving process for all reports before 2014-11-16 00:19
[2014-12-16 00:19:04] INFO  report - Reports database: Automatic archive started for all reports before 2014-11-16 00:19
....
....
....

I'm not sure restarting the process is a good idea, since it's almost sure that without an human intervention, it will just continue to fails miserably.

I propose to simply log an error and not do anything else.

Actions

Also available in: Atom PDF