Bug #6007: Error in the reports archiving process cause log explosion - Rudder - Issue Tracker

Actions

Copy link

Bug #6007

closed

Error in the reports archiving process cause log explosion

Added by François ARMAND about 10 years ago. Updated almost 10 years ago.

Status:

Released

Priority:

1 (highest)

Assignee:

Nicolas CHARLES

Category:

Performance and scalability

Target version:

2.10.10

Pull Request:

https://github.com/Normation/rudder/p...

Severity:

UX impact:

User visibility:

Effort required:

Priority:

Name check:

Fix check:

Regression:

Description

If there is an error during the archiving of reports, the process is restarted immediatelly, causing log size explosion.

Ex:

12-16 00:19:04] INFO  report - Reports database: Automatic archive started for all reports before 2014-11-16 00:19
[2014-12-16 00:19:04] ERROR com.normation.rudder.repository.jdbc.ReportsJdbcRepository - Could not delete entries in the database, cause is StatementCallback; uncategorized SQLException for SQL [vacuum ruddersysevents]; SQL state [XX001]; error code [0]; ERROR: invalid page header in block 8779 of relation base/16385/24921; nested exception is org.postgresql.util.PSQLException: ERROR: invalid page header in block 8779 of relation base/16385/24921
[2014-12-16 00:19:04] ERROR report - Reports database: Error while processing database deleting, cause is: Failure(Could not delete entries in the database, cause is StatementCallback; uncategorized SQLException for SQL [vacuum ruddersysevents]; SQL state [XX001]; error code [0]; ERROR: invalid page header in block 8779 of relation base/16385/24921; nested exception is org.postgresql.util.PSQLException: ERROR: invalid page header in block 8779 of relation base/16385/24921,Full(org.springframework.jdbc.UncategorizedSQLException: StatementCallback; uncategorized SQLException for SQL [vacuum ruddersysevents]; SQL state [XX001]; error code [0]; ERROR: invalid page header in block 8779 of relation base/16385/24921; nested exception is org.postgresql.util.PSQLException: ERROR: invalid page header in block 8779 of relation base/16385/24921),Empty) 
[2014-12-16 00:19:04] ERROR report - Reports database: Relaunching automatic deleting process for all reports before 2014-09-17 00:19
[2014-12-16 00:19:04] INFO  report - Reports database: Automatic delete started for all reports before 2014-09-17 00:19
[2014-12-16 00:19:04] ERROR com.normation.rudder.repository.jdbc.ReportsJdbcRepository - Could not archive entries in the database, cause is StatementCallback; uncategorized SQLException for SQL [vacuum ruddersysevents]; SQL state [XX001]; error code [0]; ERROR: invalid page header in block 8779 of relation base/16385/24921; nested exception is org.postgresql.util.PSQLException: ERROR: invalid page header in block 8779 of relation base/16385/24921
[2014-12-16 00:19:04] ERROR report - Reports database: Error while processing database archiving, cause is: Failure(Could not archive entries in the database, cause is StatementCallback; uncategorized SQLException for SQL [vacuum ruddersysevents]; SQL state [XX001]; error code [0]; ERROR: invalid page header in block 8779 of relation base/16385/24921; nested exception is org.postgresql.util.PSQLException: ERROR: invalid page header in block 8779 of relation base/16385/24921,Full(org.springframework.jdbc.UncategorizedSQLException: StatementCallback; uncategorized SQLException for SQL [vacuum ruddersysevents]; SQL state [XX001]; error code [0]; ERROR: invalid page header in block 8779 of relation base/16385/24921; nested exception is org.postgresql.util.PSQLException: ERROR: invalid page header in block 8779 of relation base/16385/24921),Empty) 
[2014-12-16 00:19:04] ERROR report - Reports database: Relaunching automatic archiving process for all reports before 2014-11-16 00:19
[2014-12-16 00:19:04] INFO  report - Reports database: Automatic archive started for all reports before 2014-11-16 00:19
[2014-12-16 00:19:04] ERROR com.normation.rudder.repository.jdbc.ReportsJdbcRepository - Could not delete entries in the database, cause is StatementCallback; uncategorized SQLException for SQL [vacuum ruddersysevents]; SQL state [XX001]; error code [0]; ERROR: invalid page header in block 8779 of relation base/16385/24921; nested exception is org.postgresql.util.PSQLException: ERROR: invalid page header in block 8779 of relation base/16385/24921
[2014-12-16 00:19:04] ERROR report - Reports database: Error while processing database deleting, cause is: Failure(Could not delete entries in the database, cause is StatementCallback; uncategorized SQLException for SQL [vacuum ruddersysevents]; SQL state [XX001]; error code [0]; ERROR: invalid page header in block 8779 of relation base/16385/24921; nested exception is org.postgresql.util.PSQLException: ERROR: invalid page header in block 8779 of relation base/16385/24921,Full(org.springframework.jdbc.UncategorizedSQLException: StatementCallback; uncategorized SQLException for SQL [vacuum ruddersysevents]; SQL state [XX001]; error code [0]; ERROR: invalid page header in block 8779 of relation base/16385/24921; nested exception is org.postgresql.util.PSQLException: ERROR: invalid page header in block 8779 of relation base/16385/24921),Empty) 
[2014-12-16 00:19:04] ERROR report - Reports database: Relaunching automatic deleting process for all reports before 2014-09-17 00:19
[2014-12-16 00:19:04] INFO  report - Reports database: Automatic delete started for all reports before 2014-09-17 00:19
[2014-12-16 00:19:04] ERROR com.normation.rudder.repository.jdbc.ReportsJdbcRepository - Could not archive entries in the database, cause is StatementCallback; uncategorized SQLException for SQL [vacuum ruddersysevents]; SQL state [XX001]; error code [0]; ERROR: invalid page header in block 8779 of relation base/16385/24921; nested exception is org.postgresql.util.PSQLException: ERROR: invalid page header in block 8779 of relation base/16385/24921
[2014-12-16 00:19:04] ERROR report - Reports database: Error while processing database archiving, cause is: Failure(Could not archive entries in the database, cause is StatementCallback; uncategorized SQLException for SQL [vacuum ruddersysevents]; SQL state [XX001]; error code [0]; ERROR: invalid page header in block 8779 of relation base/16385/24921; nested exception is org.postgresql.util.PSQLException: ERROR: invalid page header in block 8779 of relation base/16385/24921,Full(org.springframework.jdbc.UncategorizedSQLException: StatementCallback; uncategorized SQLException for SQL [vacuum ruddersysevents]; SQL state [XX001]; error code [0]; ERROR: invalid page header in block 8779 of relation base/16385/24921; nested exception is org.postgresql.util.PSQLException: ERROR: invalid page header in block 8779 of relation base/16385/24921),Empty) 
[2014-12-16 00:19:04] ERROR report - Reports database: Relaunching automatic archiving process for all reports before 2014-11-16 00:19
[2014-12-16 00:19:04] INFO  report - Reports database: Automatic archive started for all reports before 2014-11-16 00:19
[2014-12-16 00:19:04] ERROR com.normation.rudder.repository.jdbc.ReportsJdbcRepository - Could not delete entries in the database, cause is StatementCallback; uncategorized SQLException for SQL [vacuum ruddersysevents]; SQL state [XX001]; error code [0]; ERROR: invalid page header in block 8779 of relation base/16385/24921; nested exception is org.postgresql.util.PSQLException: ERROR: invalid page header in block 8779 of relation base/16385/24921
[2014-12-16 00:19:04] ERROR report - Reports database: Error while processing database deleting, cause is: Failure(Could not delete entries in the database, cause is StatementCallback; uncategorized SQLException for SQL [vacuum ruddersysevents]; SQL state [XX001]; error code [0]; ERROR: invalid page header in block 8779 of relation base/16385/24921; nested exception is org.postgresql.util.PSQLException: ERROR: invalid page header in block 8779 of relation base/16385/24921,Full(org.springframework.jdbc.UncategorizedSQLException: StatementCallback; uncategorized SQLException for SQL [vacuum ruddersysevents]; SQL state [XX001]; error code [0]; ERROR: invalid page header in block 8779 of relation base/16385/24921; nested exception is org.postgresql.util.PSQLException: ERROR: invalid page header in block 8779 of relation base/16385/24921),Empty) 
[2014-12-16 00:19:04] ERROR report - Reports database: Relaunching automatic deleting process for all reports before 2014-09-17 00:19
[2014-12-16 00:19:04] INFO  report - Reports database: Automatic delete started for all reports before 2014-09-17 00:19
[2014-12-16 00:19:04] ERROR com.normation.rudder.repository.jdbc.ReportsJdbcRepository - Could not archive entries in the database, cause is StatementCallback; uncategorized SQLException for SQL [vacuum ruddersysevents]; SQL state [XX001]; error code [0]; ERROR: invalid page header in block 8779 of relation base/16385/24921; nested exception is org.postgresql.util.PSQLException: ERROR: invalid page header in block 8779 of relation base/16385/24921
[2014-12-16 00:19:04] ERROR report - Reports database: Error while processing database archiving, cause is: Failure(Could not archive entries in the database, cause is StatementCallback; uncategorized SQLException for SQL [vacuum ruddersysevents]; SQL state [XX001]; error code [0]; ERROR: invalid page header in block 8779 of relation base/16385/24921; nested exception is org.postgresql.util.PSQLException: ERROR: invalid page header in block 8779 of relation base/16385/24921,Full(org.springframework.jdbc.UncategorizedSQLException: StatementCallback; uncategorized SQLException for SQL [vacuum ruddersysevents]; SQL state [XX001]; error code [0]; ERROR: invalid page header in block 8779 of relation base/16385/24921; nested exception is org.postgresql.util.PSQLException: ERROR: invalid page header in block 8779 of relation base/16385/24921),Empty) 
[2014-12-16 00:19:04] ERROR report - Reports database: Relaunching automatic archiving process for all reports before 2014-11-16 00:19
[2014-12-16 00:19:04] INFO  report - Reports database: Automatic archive started for all reports before 2014-11-16 00:19
[2014-12-16 00:19:04] ERROR com.normation.rudder.repository.jdbc.ReportsJdbcRepository - Could not delete entries in the database, cause is StatementCallback; uncategorized SQLException for SQL [vacuum ruddersysevents]; SQL state [XX001]; error code [0]; ERROR: invalid page header in block 8779 of relation base/16385/24921; nested exception is org.postgresql.util.PSQLException: ERROR: invalid page header in block 8779 of relation base/16385/24921
[2014-12-16 00:19:04] ERROR report - Reports database: Error while processing database deleting, cause is: Failure(Could not delete entries in the database, cause is StatementCallback; uncategorized SQLException for SQL [vacuum ruddersysevents]; SQL state [XX001]; error code [0]; ERROR: invalid page header in block 8779 of relation base/16385/24921; nested exception is org.postgresql.util.PSQLException: ERROR: invalid page header in block 8779 of relation base/16385/24921,Full(org.springframework.jdbc.UncategorizedSQLException: StatementCallback; uncategorized SQLException for SQL [vacuum ruddersysevents]; SQL state [XX001]; error code [0]; ERROR: invalid page header in block 8779 of relation base/16385/24921; nested exception is org.postgresql.util.PSQLException: ERROR: invalid page header in block 8779 of relation base/16385/24921),Empty) 
[2014-12-16 00:19:04] ERROR report - Reports database: Relaunching automatic deleting process for all reports before 2014-09-17 00:19
[2014-12-16 00:19:04] INFO  report - Reports database: Automatic delete started for all reports before 2014-09-17 00:19
[2014-12-16 00:19:04] ERROR com.normation.rudder.repository.jdbc.ReportsJdbcRepository - Could not archive entries in the database, cause is StatementCallback; uncategorized SQLException for SQL [vacuum ruddersysevents]; SQL state [XX001]; error code [0]; ERROR: invalid page header in block 8779 of relation base/16385/24921; nested exception is org.postgresql.util.PSQLException: ERROR: invalid page header in block 8779 of relation base/16385/24921
[2014-12-16 00:19:04] ERROR report - Reports database: Error while processing database archiving, cause is: Failure(Could not archive entries in the database, cause is StatementCallback; uncategorized SQLException for SQL [vacuum ruddersysevents]; SQL state [XX001]; error code [0]; ERROR: invalid page header in block 8779 of relation base/16385/24921; nested exception is org.postgresql.util.PSQLException: ERROR: invalid page header in block 8779 of relation base/16385/24921,Full(org.springframework.jdbc.UncategorizedSQLException: StatementCallback; uncategorized SQLException for SQL [vacuum ruddersysevents]; SQL state [XX001]; error code [0]; ERROR: invalid page header in block 8779 of relation base/16385/24921; nested exception is org.postgresql.util.PSQLException: ERROR: invalid page header in block 8779 of relation base/16385/24921),Empty) 
[2014-12-16 00:19:04] ERROR report - Reports database: Relaunching automatic archiving process for all reports before 2014-11-16 00:19
[2014-12-16 00:19:04] INFO  report - Reports database: Automatic archive started for all reports before 2014-11-16 00:19
....
....
....

I'm not sure restarting the process is a good idea, since it's almost sure that without an human intervention, it will just continue to fails miserably.

I propose to simply log an error and not do anything else.