Project

General

Profile

Actions

Bug #15177

closed

Catching up on reports can be extremelly long and prevent compliance to display

Added by Nicolas CHARLES almost 5 years ago. Updated about 4 years ago.

Status:
Rejected
Priority:
N/A
Category:
Web - Config management
Target version:
Severity:
Critical - prevents main use of Rudder | no workaround | data loss | security
UX impact:
User visibility:
Operational - other Techniques | Rudder settings | Plugins
Effort required:
Priority:
67
Name check:
To do
Fix check:
To do
Regression:

Description

When rudder is restarted after some times, it catches back old reports based on configuration parameters.
But the catching can be extremelly long (in our case: 5 days to catch up at a rate of 5 minutes every 5s...).
And during all that time, we don't have compliance (because recent runs with recent reports are not registered yet).

This an extermelly bad source of mileading bugs and it totally brokes Rudder expected behavior ("display compliance") or any app behavior ("restart == clean state").

So, we should change the strategy for getting back old reports (hard because we rely of last ckecked position), or at least make sure the convergence is much quicker (for ex, if we are catching up, don't limit the number of runs, or don't wait between each catch up...).

Edit: the culprit was misidentified as (see comment for why it wasn't):

[2019-07-09 11:54:41] ERROR report - Could not correctly retrieve from were to retrieve report processing, fallbacking to default value, error is is Failure(Could not fetch the highest id before date 2019-07-04T11:54:41.481Z in the databa
se,Empty,Full(Failure(SQL `NULL` read at column 1 (JDBC type BigInt) but mapping is to a non-Option type; use Option here. Note that JDBC column indexing is 1-based.,Full(doobie.util.invariant$NonNullableColumnRead: SQL `NULL` read at col
umn 1 (JDBC type BigInt) but mapping is to a non-Option type; use Option here. Note that JDBC column indexing is 1-based.),Empty)))

Actions

Also available in: Atom PDF