Persist zones

[ximon18]

Status

Zone edits (e.g. via file edit and zone reload or due to changes received via XFR) are persisted both for loaded and signed zones. On application restart persisted zones are restored and cascade zone status and dig AXFR appear to work as they did prior to application shutdown.

The code has been initially reviewed and led to some wanted changes:

• @bal-e will extend the zonedata crate interface to enable moving some of the changes to Cascade itself.
• Invoke save_now() instead of mark_dirty(). This is because persisting zone data to disk not enough, as if there was a power outage between calling mark_dirty() and publication of the zone the paths to the persisted files (that are recorded in state) would not have been written to disk, thus save_now() should be invoked instead. We may actually want to invoke save_now() in the publication server itself to ensure all published zone related state is persisted, but that is out of scope for this PR.

tl;dr

• Extend src/persistence/persist.rs to write snapshots and deltas to disk (currently in DNS AXFR/IXFR wire format).
• Extend src/persistence/restore.rs to restore zones by loading the zone snapshot into a zone replacer and passing the zone diffs one at a time to a zone patcher.
• Extend src/persistence/ to store generated/restored diffs for use by IXFR out.
• Extend ZoneState with two vecs of paths to persisted snapshot and delta files.
  • Note: Purging of excess diffs. #657 moves these paths into dedicated persistence related state which I didn't notice existed when I created this PR.
• Extend the CLI to show the zone is restoring in cascade zone status output.

Known issues:

• No support (yet) for condensing/compacting/purging old persisted deltas.
  • See Purging of excess diffs. #657
• Binary snapshot/delta format is not inspectable by operators. We could add a `cascade debug subcommand to render a snapshot/delta in XFR presentation format but that is not included in this PR.
• There's no "scheduling" or spreading out of zone restoration activity, like refreshing it happens as soon as possible.
• There are no metrics for the restoration process.
• Snapshots and deltas are loaded entirely into memory, while they could be processed in a streaming manner instead.
• The persistence location is hard-coded to store files in the same directory as zone-state.

Alternative snapshot/delta storage formats

A choice would need to be driven by requirements:

• Do we need to support IXFR out at the loaded review nameserver? If so, is just one delta enough? Actually do we even need to keep a loaded diff, a review hook would only need it for the next received zone update, not after restart.
• What do we tune for? Memory usage? Persistence speed? Restoration speed?

Some ideas:

• Presentation XFR format: was not done in this PR due to the pain of having to convert new base to old base to render in presentation format, and parsing DNS records in presentation format is only currently properly supported by domain/cascade for entire zones, though one could use the "treat a single RR line as a zonefile" hack that the Stelline code uses.
• (De)serialize cascade DiffData to JSON/CBOR/whatever. Would tie the format to the internal format, changing the internal format would require writing additional code to handle storage format migration or support parsing old as well as new format. A PRO of this approach is being able to storage additional meta data with the snapshot/delta, if needed.

---

• If you are changing Rust code or integration tests (Cargo.*, crates/, etc/, integration-tests/, src/):
  • Did you run the integration tests with act through the act-wrapper (as described in TESTING.md)?

[bal-e]

@ximon18 and I had a discussion while I was writing this review, and decided that I will add commits changing the PR to implement a new approach for the zone data storage state machine. I've left these comments here for us to come back once I'm done; I also didn't review everything, so there will be more to come.