Architecture #9202
closedClean old fileFormat migration and fileFormat numbering policy
Description
We have a lot of old migration that are kept so that old archives can still be read.
We want to only support archive from up to the last LTS, even more now that we have a more or less stable file format.
For example, for 4.0, we want to support only fileFormat of 3.1 and newer, i.e fileFormat=6 and more recent.
We also want to update the rule for file format number bump:
- if we only add new items, or new fields, and these fields are optionnal or a reasonnable default can be provided if they are missing, we DON'T change the fileFormat number,
- we only bump fileFormat in the case of a breaking change, non sortable locally or too higly costly to support either in maintenance cost or performance. Such a case would the renaiming of a lot of business object, or the addition of a new complex, costly I/O field. Changing events from one type to an other is another use case.
With that, we can keep only fileFormat 6 in Rudder.
I propose to keep the migration scheme from fileFormat=5, so it allows to read archive from Rudder 2.10/2.11, which is quite old and let us be sure we won't make user unhappy while keeping the migration logic (which works great, now).