This issue tracks reimplementation of the work from stale PR #1079, which is being closed because it is too out of date to merge directly.
Original PR: https://github.com/clockworklabs/SpacetimeDB/pull/1079
Original author: @kim
Original branch: `kim/commitlog2/direct-io`
Base branch: `master`
## Original PR summary
Introduce page-aligned buffered reads + writes. This is required for direct I/O (O_DIRECT or equivalent), which is now the default.
Direct I/O essentially means to bypass the OS's page cache and operate directly on the device. It does not mean that data is automatically more durable once written, fsync is still required. Optionally, the patch allows to enable O_DSYNC, which blocks a write until the equivalent of fdatasync has occurred.
Both options likely have a performance impact, which needs to be evaluated. There also some performance optimizations of the presented implemention possible, e.g. re-using buffer allocations or scatter/gather ("vectored") I/O.
Stacked on top of #985
Expected complexity level and risk
4
Testing
Describe any testing you've done, and any testing you'd like your reviewers to do,
so that you're confident that all the changes work as expected!
This issue tracks reimplementation of the work from stale PR #1079, which is being closed because it is too out of date to merge directly.
Direct I/O essentially means to bypass the OS's page cache and operate directly on the device. It does not mean that data is automatically more durable once written,
fsyncis still required. Optionally, the patch allows to enable O_DSYNC, which blocks a write until the equivalent offdatasynchas occurred.Both options likely have a performance impact, which needs to be evaluated. There also some performance optimizations of the presented implemention possible, e.g. re-using buffer allocations or scatter/gather ("vectored") I/O.
Stacked on top of #985
Expected complexity level and risk
4
Testing
Describe any testing you've done, and any testing you'd like your reviewers to do,
so that you're confident that all the changes work as expected!
Run test suite under Linux
Check that the code compiles under occult operating systems (macOS / Windows)
Check that the code works under occult operating systems.
Follow-up
master.