Index file (.idx) for fast offset lookups

[mairinkdev]

Summary

Introduce an index file per segment (e.g. segment_<base_offset>.idx) that maps logical offset to file position. Implement read(offset) that uses the index to seek directly to the record instead of scanning the segment.

Scope

• Index format: document in docs/index.md (or design doc). Simple append-only structure: e.g. each entry is (offset, file_position). Define size and endianness.
• On append: write record to .log; append corresponding (offset, position) to .idx.
• read(offset): look up offset in the current segment’s index (or find correct segment by base_offset); seek to file position; decode and return record or error.
• Tests: append records; call read(offset) for random offsets and assert correct payload; test offset not found (e.g. beyond high watermark) returns clear error.

Acceptance criteria

• Index is updated on every append; format is documented.
• read(offset) is O(1) in the number of records (via index lookup + single seek).
• Tests cover successful lookups and missing-offset behavior.

Suggested commit

feat: add index file for fast offset lookups