WAL

wal is a bounded Go library for durable ordered storage, crash recovery, and acknowledged consumption.

Reliability and recoverability come before benchmark wins. Performance is optimized within—not around—the library's contracts: success, failure, and uncertainty have explicit semantics, locked down by fault-injection, crash, and public-API tests, while the API is shaped to make misuse difficult.

The library provides:

• a segmented filesystem log with atomic transactions and crash-safe compaction;
• a bounded volatile memory log;
• byte-slice and streaming APIs for single records and atomic batches;
• stable-ID retry and conflict detection;
• FIFO and bounded out-of-order consumption with explicit acknowledgment;
• degraded-prefix export for damaged filesystem logs; and
• explicit limits for storage, memory, queues, scans, and callback inputs.

It requires Go 1.25 or newer and uses only the standard library.

Important

Filesystem format version 1 is frozen by independent fixtures. Its atomic multi-group and multi-segment transactions, streaming payloads, recovery, crash, race, and bounded-resource gates passed on the APFS and native Linux/ext4 profiles recorded in QUALIFICATION.md. These results do not qualify a different production filesystem, device, kernel, or workload. Rerun the applicable gates before a critical deployment.

Guarantees

Ordering and atomicity

• Positions increase monotonically and records are observed in FIFO order.
• Append, AppendBatch, AppendFrom, and AppendBatchFrom are logically atomic even when their payloads cross physical commit-group or segment boundaries.
• Readers never observe an incomplete frame, unpublished transaction, or partial logical batch.

Retry and recovery

• Reusing a retained stable ID with identical bytes is idempotent.
• Reusing a retained stable ID with different bytes returns ErrConflict.
• Recovery rejects interior corruption and repairs only an incomplete active tail permitted by the format.
• A failed filesystem synchronization terminalizes that instance for mutation; Synced never silently degrades to Written.

Stable-ID idempotency lasts only while the record remains retained. Applications that need a longer deduplication window must maintain a separate durable idempotency record.

Bounds and ownership

• Every retained resource has an explicit configured bound.
• Scan payloads and consumer batches are borrowed only for their synchronous callback. They must not be retained or modified after it returns.
• CrashPersistent consumption establishes a Synced source watermark and commits acknowledgments with Synced.

Durability levels

Level	Contract
Volatile	Retained for the lifetime of the in-memory instance; no crash-persistence claim.
Written	Writes completed, with no power-loss persistence claim.
Synced	Data and required publication metadata crossed the supported filesystem barriers.

Use Supports before selecting a durability level. A requested level is a minimum: group commit may promote a request to a stronger effective level, and the returned Receipt reports the actual result.

The library provides integrity checks, not encryption or authentication. Record IDs and payloads are opaque plaintext; protect sensitive data before append.

Install

go get github.com/mgurevin/wal

Choose an API

Need	API
Crash-persistent ordered storage	File
Bounded process-lifetime buffering	Memory
Byte-slice append and scan	Log
Bounded-memory streaming append and scan	StreamingLog
Contiguous FIFO acknowledgment	Consumer
Concurrent work with out-of-order acknowledgment	consumer

File and Memory implement both Log and StreamingLog. Their storage behavior differs: File streams large payloads through bounded buffers, while Memory must retain accepted payloads in RAM because memory is its storage medium.

Quick start

Open one filesystem WAL for the owning application or worker lifetime, share it with concurrent appenders when needed, and close it during coordinated shutdown:

package main

import (
	"context"

	"github.com/mgurevin/wal"
)

func writeRecords(
	ctx context.Context,
	directory string,
	records []wal.Record,
) error {
	log, err := wal.OpenFile(ctx, wal.FileConfig{
		Directory: directory,
	})
	if err != nil {
		return err
	}

	for _, record := range records {
		if _, err := log.Append(ctx, record, wal.Synced); err != nil {
			_ = log.Close()

			return err
		}
	}

	return log.Close()
}

OpenFile completes context-aware recovery before returning. A canceled or failed open returns no usable instance. Do not reopen the WAL for each record.

For bounded buffering that does not survive process failure:

log, err := wal.NewMemory(wal.MemoryConfig{
	MaxBytes:   64 << 20,
	MaxRecords: 10_000,
})
if err != nil {
	return err
}
defer log.Close()

_, err = log.Append(ctx, record, wal.Volatile)

Memory owns copies of accepted IDs and payloads and supports only Volatile. Truncation releases retained capacity. Memory.Stats() returns a MemoryStats snapshot; monitor UsedBytes, Records, and FirstPosition to detect a growing backlog before it reaches the configured hard limits.

Producing records

The four append methods make the input and atomicity contract explicit:

Method	Input	Commit unit
Append	Record	one record
AppendBatch	[]Record	the complete ordered batch
AppendFrom	ReaderRecord	one streamed record
AppendBatchFrom	[]ReaderRecord	the complete ordered streamed batch

Concurrent appends are admitted into bounded physical commit groups. Requests selected into the same group share its strongest requested barrier, so a caller that requested Written may receive a receipt with effective durability Synced. Inspect Receipt.Durability to observe promotion; never depend on incidental grouping for correctness.

See PRODUCER.md for admission, group-commit pipelining, receipt semantics, retry outcomes, and concrete producer patterns.

Streaming append

Use AppendFrom when a filesystem WAL should accept a payload without holding the complete value in memory:

receipt, err := log.AppendFrom(ctx, wal.ReaderRecord{
	ID:       object.ID,
	Reader:   object.Reader,
	SizeHint: object.EstimatedBytes,
}, wal.Synced)

SizeHint is optional. Zero means unknown; a positive value is a non-binding setup hint. The WAL discovers the exact size while reading and enforces MaxRecordBytes and MaxAppendBatchBytes against actual bytes. An inaccurate hint cannot weaken a limit or change the persisted record.

AppendBatchFrom applies the same rules to an atomic ordered batch:

• readers are consumed in input order, at most once, and only after admission;
• the WAL never rewinds a reader;
• a failure may occur before a reader reaches EOF or before a later reader is reached; and
• if any reader fails or any actual limit is exceeded, no record in the batch becomes visible.

Retrying an unknown outcome requires fresh readers that produce the same bytes. Retry comparison is also a single forward pass over each fresh reader.

Reading records

Read and Scan expose []byte values and may therefore materialize up to MaxRecordBytes. Prefer the streaming APIs for large payloads.

Use ScanReader for sequential processing:

err := log.ScanReader(ctx, after, limit, func(entry wal.ReaderEntry) error {
	if entry.PayloadBytes > uploadLimit {
		return errPayloadTooLarge
	}

	return upload(entry.ID, entry.PayloadBytes, entry.Reader)
})

PayloadBytes is exact. The reader is borrowed for the synchronous callback only and must not be retained.

Use OpenReader when one exact position must outlive a scan callback:

entry, err := log.OpenReader(ctx, position)
if err != nil {
	return err
}
defer entry.Close()

err = upload(entry.ID(), entry.PayloadBytes(), entry)

EntryReader is a one-shot io.ReadCloser. Reaching EOF releases its storage lease automatically; Close releases it early. Large fragmented records use the same bounded two-pass validation as ScanReader without assembling the logical payload. An unfragmented record remains bounded by its physical commit group. Open readers count against MaxConcurrentScans.

Consuming records

FIFO acknowledgment

Consumer delivers a contiguous prefix in order. It starts no goroutine: call Next or Drain directly, or run Run in an application-owned goroutine.

consumer, err := wal.NewConsumer(wal.ConsumerConfig{
	Log:       log,
	Sender:    sender,
	Guarantee: wal.CrashPersistent,

	MaxBatch:      128,
	MaxBatchBytes: 4 << 20,
})
if err != nil {
	return err
}

go func() {
	if err := consumer.Run(ctx); err != nil &&
		ctx.Err() == nil {
		report(err)
	}
}()

With a nil Acknowledger, successful downstream acceptance is committed by truncating Log. Supply an Acknowledger when acknowledgment must coordinate additional durable state:

type Acknowledger interface {
	Supports(wal.Durability) bool
	Acknowledge(context.Context, wal.Position, wal.Durability) error
}

After downstream acceptance becomes definite, Consumer preserves context values but ignores caller cancellation until acknowledgment returns. This avoids creating an unnecessary redelivery boundary after the remote side has already accepted the prefix.

The sender must acknowledge only a durably accepted contiguous prefix and must handle repeated stable IDs idempotently. A crash after downstream acceptance but before local acknowledgment causes safe redelivery; exactly-once delivery is not claimed.

Permanent rejection stops at the rejected head by default. Applications may explicitly choose a durable idempotent dead-letter handler or lossy discard. Discard waives at-least-once delivery for that record.

See CONSUMER.md for the complete delivery contract, rejection policy, failure semantics, sizing guidance, and integration examples.

Out-of-order acknowledgment

Package consumer reserves source records in order while allowing application workers to acknowledge them in any order. A separate WAL stores the bounded sparse ACK set; contiguous progress still truncates the source through the existing Acknowledger contract.

Use it when independent work should continue past a slow record:

receiver, err := consumer.Open(ctx, consumer.Config{
	Source:             source,
	Journal:            acknowledgmentJournal,
	SourceDurability:   wal.Synced,
	AckDurability:      wal.Synced,
	MaxInFlightRecords: 128,
	MaxInFlightBytes:   64 << 20,
})

The receiver:

• starts no goroutine and owns neither WAL;
• bounds returned, read-ahead, and recoverable sparse state with the same record and byte windows;
• keeps large records streaming; and
• reports startup reconciliation as Operation == wal.Reconcile, separately from each filesystem WAL's recovery.

See consumer/README.md for delivery lifetime, durability combinations, recovery, failure handling, and worker-pool examples.

Configuration

Defaults and hard limits

Zero-valued limits select bounded defaults:

Bound	File default	Memory default
Logical bytes	1 GiB	64 MiB
Records	100,000	10,000
One record	1 MiB	1 MiB
Stable ID	1 KiB	1 KiB
Read entries	128	128
Read chunk	256 KiB	not applicable
Concurrent scans	16	16
Segment	64 MiB or 10,000 records	not applicable
Logical append batch	1,024 records or 8 MiB	1,024 records or 8 MiB
Commit group	1,024 records or 8 MiB	not applicable
Pending append requests	4,096	not applicable
Pending append bytes	64 MiB	not applicable
Index and descriptor memory	256 MiB	covered by record and byte limits

Limits are hard failures, not performance targets. Derive them from:

• the largest legitimate record and atomic batch;
• peak ingress and tolerated admission delay;
• the longest downstream outage to absorb;
• the required recovery deadline; and
• available memory and storage.

Filesystem reopen contract

The filesystem WAL persists limits that affect recovery, admission, capacity, or physical layout:

MaxBytes, MaxDiskBytes, MaxRecords, MaxRecordBytes, MaxIDBytes,
SegmentMaxBytes, SegmentMaxRecords,
MaxAppendBatchRecords, MaxAppendBatchBytes,
MaxCommitGroupRecords, MaxCommitGroupBytes,
MaxIndexBytes, MetadataReserveBytes, Preallocation

Later OpenFile calls must request the same value or a stronger one for every persisted field:

• numeric limits may increase but not decrease;
• preallocation strength is ordered Disabled < BestEffort < Required; and
• a mixed request that raises one field and lowers another is rejected as a whole with ErrConfigDowngrade.

The rejection happens before recovery or tail repair changes a file. It prevents a smaller decoder bound from misclassifying valid committed bytes as a torn tail. Tightening limits requires a separately created WAL and an explicit application migration; draining or truncating the existing WAL does not erase its persisted minimums.

Zero-valued limits are normalized to the current documented defaults on every open; zero does not mean “inherit the persisted value.” Keep the complete accepted configuration in application configuration. If a persisted custom minimum exceeds a default, reopening that field as zero is a downgrade.

A valid upgrade:

1. prepares the larger metadata reserve and standby allocation; then
2. publishes the complete profile through alternating checksummed state slots.

The next open sees either the previous profile or the complete new profile, never a per-field mixture. If an upgrade is canceled, crashes, or reports a storage failure, retry it with the complete requested configuration. OpenDegradedFile enforces the same no-downgrade rule but remains read-only: stronger values bound that inspection and are not persisted.

Interrupted metadata-reserve growth may leave useful allocated bytes before the new profile becomes authoritative. Those bytes remain visible to capacity accounting. If they exceed the old MaxDiskBytes, reopening with the old profile returns ErrCapacity; retry the complete upgrade request whose larger disk bound covers the preparation.

These runtime-tuning fields may change freely on every open:

MaxReadEntries, ReadChunkBytes, MaxConcurrentScans,
GroupCommitDelay, MaxPendingAppendRequests, MaxPendingAppendBytes,
OnProgress, OnInternalError, Logf, ProgressInterval

Progress reporting

OnProgress reports long-running startup and maintenance work. Events identify the operation and phase; profile changes use Operation == Configure with ReservingMetadata, ReservingStandby, and Publishing phases.

Every operation reports:

• its initial state;
• phase transitions; and
• a final Complete event after success.

Intermediate events are time-throttled by ProgressInterval, which defaults to five seconds. Event.Elapsed and Report.Completed allow callers to calculate rates. Report.Total == -1 means the total is unknown; zero means the operation is known to be empty. Only Phase == Complete means the entire operation succeeded.

During configuration work, Report.Scanned counts completed preparation steps and Report.Bytes is their cumulative prepared byte count.

Use ProgressHandlerFunc.For to bind a stable application-owned name when several WALs share one handler. The helper is stateless and does not serialize calls, so independently operating WALs may invoke the shared handler concurrently.

Sizing from a workload

Start with four measured inputs:

• R: peak records appended per second;
• S: average logical bytes per record, including stable ID and payload;
• T: longest downstream outage to absorb, in seconds; and
• H: safety factor for bursts and estimation error, normally at least 1.25.

The minimum retained backlog is:

MaxRecords >= H × R × T
MaxBytes   >= H × R × S × T

Round upward and leave filesystem headroom beyond the configured logical limit. Framing, segment slack, standby files, compaction overlap, and the metadata reserve consume additional disk.

Leaving MaxDiskBytes at zero derives a bounded allowance from MaxBytes, four segments, and the reserve. An explicit value must still cover documented recovery and compaction overlap. MaxDiskBytes is a safety ceiling, not a promise that every logical limit can be reached for every record shape; include encoded per-record overhead in deployment calculations.

Set MaxRecordBytes from the largest legitimate payload, not the average. Set MaxAppendBatchRecords and MaxAppendBatchBytes from the largest atomic request the application must accept. Oversized input is rejected rather than partially committed.

For a tolerated local admission pause of P seconds, start with:

MaxPendingAppendRequests >= peak append requests/second × P
MaxPendingAppendBytes    >= bytes charged by those requests during P

Byte-backed requests charge their actual ID and payload bytes. Reader-backed requests charge their configured worst case because SizeHint is non-binding. A workload dominated by AppendFrom may therefore need more pending-byte capacity or a smaller append-batch ceiling. These limits bound admitted calls; applications still need a bounded queue and full-capacity policy before the WAL.

Estimate recovery time from retained physical bytes and measured scan throughput:

expected recovery time ~= retained physical bytes / measured scan bytes/second

Measure on the production filesystem and device, then add latency and safety margin. If the result exceeds the startup deadline, reduce retention, drain more aggressively, or partition the workload. Do not infer a recovery deadline from the benchmark machine.

Worked example

A service peaks at 2,000 records/second, averages 1 KiB of logical data, and must absorb a 30-minute outage with H = 1.5.

The formulas require at least 5.4 million records and about 5.15 GiB of logical capacity. Reasonable rounded starting values are:

MaxRecords = 5_500_000
MaxBytes   = 6 << 30

At 600 MiB/s measured recovery throughput, scanning 6 GiB takes about 10 seconds before margin; a two-times operational allowance budgets roughly 20 seconds.

If the same service accepts one record per append and tolerates 250 ms of local admission pause, it starts near 500 pending requests. Raise that limit only when measured bursts justify it.

Consumer sizing is independent. MaxBatch controls downstream amortization and the maximum number of records offered at once; MaxBatchBytes bounds borrowed payload memory. Start with the documented defaults, then sweep the deployment workload as described in CONSUMER.md.

Cancellation and admission

All append methods accept cancellation only before admission:

• a canceled request returns ctx.Err() without mutation;
• a canceled batch admits none of its records; and
• after admission, the call waits for and returns the definite storage outcome even if its context is later canceled.

Caller-owned record data and readers remain borrowed until the method returns.

Flush follows the same rule: cancellation is accepted only before its ordered barrier is admitted.

Truncate remains cancellable while it waits and prepares, but not after checkpoint publication begins. Pre-publication cancellation leaves the logical truncated position unchanged, although completed segment barriers may have promoted already-written bytes. Neither method hides a definite post-admission storage result with ctx.Err().

Latency-sensitive applications must place a bounded admission queue with an explicit full-capacity policy before the WAL. Post-admission filesystem I/O is deliberately not abandoned, and one goroutine per append is not bounded admission.

Recovery and maintenance

Use File.Stats to schedule explicit partial-head compaction:

if log.Stats().ReclaimableBytes >= 16<<20 {
	report, err := log.Compact(ctx)
	if err != nil {
		return err
	}

	recordReclaimedBytes(report.ReclaimedBytes)
}

Normal open is fail-stop on corruption. OpenDegradedFile provides read-only access only through the last completely validated prefix and never repairs or mutates the directory. After exporting that prefix, quarantine the complete failed log epoch before resuming writes elsewhere.

See OPERATION.md for startup, recovery, disk-full, synchronization, quarantine, compaction, and shutdown procedures.

Platform support

Filesystem Written is portable. Filesystem Synced is exposed only where regular-file and directory-publication barriers satisfy the documented conformance suite.

The barrier implementation supports qualified Linux filesystem/device profiles and qualified local macOS APFS profiles. Darwin uses F_FULLFSYNC for segment files, WAL-directory publication, and parent-directory publication.

Format changes still require the current crash and durability suite on each supported platform; historical prototype evidence is not a substitute. Other filesystems and device classes require their own qualification. Unsupported or failed barriers return an error instead of weakening durability silently.

Further reading

• PRODUCER.md — admission, group commit, durability, receipts, and retry.
• CONSUMER.md — FIFO delivery, acknowledgment, redelivery, and rejection policy.
• consumer/README.md — bounded out-of-order acknowledgment.
• DESIGN.md — format, concurrency, durability, recovery, compaction, and boundedness contracts.
• OPERATION.md — normal operation and failure recovery.
• BENCHMARK.md — current performance snapshot, reproducible commands, and tuning guidance.
• QUALIFICATION.md — platform evidence, format decisions, resource gates, and historical baselines.