Purging of excess diffs.

[ximon18]

Builds on #804.

The problem

In #550 when a zone is first loaded and is approved it is persisted to disk as a loaded snapshot. Once it is signed and approved a signed snapshot is written to disk (as an AXFR payload in wire format). On process restart the snapshots are restored allowing the published zone to be served as it was before shutdown. Every time thereafter that the zone changes, either due to changes to the loaded zone via reloaded from disk or incoming XFR, or due to changes in the signed zone due to signing loaded record changes and/or (incremental) re-signing to account for DNSKEY changes and/or to avoid signature expiration, that results in loaded and/or signed diffs.

Those diffs are persisted to disk (as IXFR payloads in wire format) in order to be replayed on top of the zone snapshot after process restart, and stored in memory to be served as IXFR out diffs.

The persistence of diffs in #550 is unbounded, both in-memory and on-disk.

Challenges to overcome

Excess IXFR in-memory diffs can just be forgotten, but excess on-disk diffs cannot just be deleted because they form part of the snapshot+diff history of the zone that is replayed on process restart to restore the published zone. Removing a diff would break restoration as the next available diff would not apply correctly to the snapshot.

What this PR provides

This PR sets an upper limit on the number of diffs to keep, adds PersistentDiffManager, Compacter and IxfrZoneDiffs:

• PersistentDiffManager manages the set of paths to persistent diffs on disk. It tracks the next_idx to use to generate a unique filename for the next diff to write (this could perhaps be avoided by using UUIDs or something). It also keeps track of the restore_base_idx which indicates the offset into the set of persisted diffs at which replay during restore should restart.

• Compacter replaces the persisted snapshots for a zone with new versions containing records from the in-memory copy of the currently published zone, as this represents the existing snapshot with all existing diffs applied, and updates restore_base_idx accordingly. Writing out the entire zone to disk is a time consuming operation for a large zone and so we look for zones needing compaction once a minute. I'm not at all sure what strategy should be used here, this is just an initial attempt to avoid persisting to disk on every single change but instead to batch the changes together less frequently.

• IxfrZoneDiffs stores diffs for use by Service when responding to IXFR requests to the publication server. It also provides the logic to look up the diffs needed to answer a query, and "trimming" functionality to keep in-memory diff sizes within configurable limits.

New policy settings

• server.outbound.max-diffs
• server.outbound.max-diffs-size

Features NOT included in this PR

• There is no limit on bytes used in memory or on disk.
• There is no support for RFC 1995 purging of diffs based on total IXFR response size or SOA expire period, though there is something related which is purging based on the ratio of diff size to last published zone size.
• There is no support for RFC 1995 condensation of multiple diffs.

Known issues

• Code quality: All manipulation of the filesystem for persistence zone snapshots/diffs should be done in one place but spread across multiple places. Partly this is because PersistentState and PersistentDiffManager have no reference to Center from which they can gain the zone-state path config value) nor can it be passed in via a constructor as the parent hierarchy uses Default. Likewise the persistent state relates to a single zone but has no concept of which zone that is as when it is created, as well as the Default problem mentioned, it is created as part of the zone so it can't take a reference to something which isn't constructed yet.

Tasks remaining

• What happens if compaction is still running when compaction is attempted again? The lock held on zone state prevents concurrent changes to the set of diff paths or the diffs themselves while compaction, persistence or restoration is in progress.
• Fix Restoring with a rejected signed zone in the history causes loss of the published zone. #825. Ideally the fix would be backported but the code is different in earlier branches so copying the code unmodified not possible.
• Manually test with incremental signing.
• Add support for Add a max perecentage of zone changed requirement for keeping an in-memory diff for IXFR out. #830
• Add unit tests
• Add architectural documentation (see src/persistence/mod.rs)
• Do we maybe need some end user facing documentation about persistence e.g. that zone status may show the zone as restoring or that policy show prints the purging related policy settings, or something in the high level zone transfers page about limiting of diffs kept?

---

• If you are changing Rust code or integration tests (Cargo.*, crates/, etc/, integration-tests/, src/):

  • Did you run the integration tests with act through the act-wrapper (as described in TESTING.md)?

• If you are modifying man pages:

  • Did you commit the updated built man pages?

[tertsdiepraam]

Some comment suggestions to help explain how the restoration works.