Skip to content

feat: sequence fields and monotonic timestamps in the schema DSL#56

Merged
williajm merged 2 commits into
mainfrom
feat/sequence-monotonic-timestamps
Jul 3, 2026
Merged

feat: sequence fields and monotonic timestamps in the schema DSL#56
williajm merged 2 commits into
mainfrom
feat/sequence-monotonic-timestamps

Conversation

@williajm

@williajm williajm commented Jul 3, 2026

Copy link
Copy Markdown
Owner

Implements the first and third items of Phase 1 on the roadmap ("finish the single-table story at scale"): sequence fields and monotonic timestamps. These are the two row-order-aware specs, bundled because they share the same per-call state machinery.

Sequence fields

("sequence", start) — or bare "sequence" for start=1 — generates auto-increment integers in row order:

records(1_000_000, {"id": "sequence"})          # 1, 2, 3, ...
records(1_000_000, {"id": ("sequence", 1000)})  # 1000, 1001, ...

Every seeded table needs a primary key, and this is the only cheap way to get unique integers at very large n — the ("unique", spec) retry loop degrades as the value space saturates. Sequence columns are int64 in Arrow/Parquet and work as derive sources.

Monotonic timestamps

("datetime_seq", start, min_increment, max_increment) — the first row is start, each following row advances by a random whole number of seconds in the given range:

records(1_000_000, {
    "created_at": ("datetime_seq", "2024-01-01T00:00:00", 1, 60),
})

Random in-range datetimes produce unsorted "event logs"; real event data is roughly time-ordered and time-series databases behave very differently on sorted input. start accepts YYYY-MM-DDTHH:MM:SS, YYYY-MM-DD HH:MM:SS, or YYYY-MM-DD (midnight). min_increment >= 1 gives strictly increasing timestamps; 0 allows realistic duplicates.

Semantics

  • Both specs continue across the chunks of records_to_file() and the async methods (a 100M-row streamed file gets one contiguous sequence), and restart on each generation call.
  • Both compose with nullable; null rows don't consume a sequence value or advance the clock.
  • ("unique", ...) wrapping is rejected with an explanatory error — sequences are already unique, and datetime_seq is unique when min_increment >= 1.
  • Overflow (i64 for sequence, chrono's datetime range for datetime_seq) surfaces as ValueError, not a panic (Duration::try_seconds, checked_add).
  • Supported in records(), records_tuples(), records_arrow(), all serialized formats, records_to_file(), and the async methods.

Implementation notes

  • The per-call UniqueState struct is renamed to GenerationState: it now carries sequence counters and last-emitted timestamps alongside unique-value tracking, so cross-chunk continuity reuses the exact mechanism that already threads unique state through the chunked paths. The rename is mechanical (~40 lines of the diff).
  • generate_value() (stateless single-value path) rejects the new specs with a clear error, following the existing derive/custom-provider precedent — reaching it would mean a code path skipped the shared state and would silently repeat values.
  • The Arrow columnar path handles both specs through the same row-wise-with-state branch used by nullable/unique wrappers.

Tests

  • Rust: 22 new unit tests (consecutive values, per-field counters, cross-chunk continuity, per-call restart, nullable interaction, overflow errors, unique rejection, Arrow column types/monotonicity, determinism, flexible datetime parsing).
  • Python: 27 new tests in test_schema_dsl.py, including cross-chunk continuity regressions for records_to_file() and the async methods, output-format coverage (CSV/NDJSON/Arrow), and error cases.
  • Full suites pass locally: cargo test (912), pytest (1568, 100% coverage), clippy -D warnings, mypy --strict, ruff.

Docs

  • README: new "Sequence fields and monotonic timestamps" section with an event-log example.
  • CHANGELOG: Unreleased → Added.
  • ROADMAP: both Phase 1 items checked off.

🤖 Generated with Claude Code

Two new row-order-aware field specs for high-volume generation:

- ("sequence", start) / "sequence": auto-increment integers — cheap
  unique primary keys at any n, with no retry loop or uniqueness
  tracking. Int64 in Arrow/Parquet; usable as a derive source.
- ("datetime_seq", start, min_increment, max_increment): the first row
  is start, each following row advances by a random whole number of
  seconds in [min_increment, max_increment], producing time-ordered
  event logs.

Both continue across the chunks of records_to_file() and the async
methods, restart on each generation call, and compose with nullable
(null rows don't consume a value or advance the clock). unique()
wrapping is rejected with an explanatory error.

The per-call UniqueState struct is renamed to GenerationState since it
now also carries sequence counters and last-emitted timestamps.

Closes the first and third Phase 1 roadmap items.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@codecov

codecov Bot commented Jul 3, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 98.32636% with 8 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
src/providers/records.rs 98.36% 7 Missing ⚠️
src/providers/file_writer.rs 75.00% 1 Missing ⚠️

📢 Thoughts on this report? Let us know!

- Reserve "sequence" in RESERVED_PROVIDER_NAMES: it is a built-in string
  spec, so a custom provider with that name would register successfully
  and then be silently shadowed in schemas.
- Reject a hand-built FieldSpec::Simple("sequence") at validation with a
  pointer to FieldSpec::Sequence, instead of failing at generation with
  "Unknown type: sequence".
- Narrow the README claim that unique accepts any spec except nullable;
  derive, sequence, and datetime_seq are also rejected.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@sonarqubecloud

sonarqubecloud Bot commented Jul 3, 2026

Copy link
Copy Markdown

@williajm
williajm merged commit 2091840 into main Jul 3, 2026
13 checks passed
@williajm
williajm deleted the feat/sequence-monotonic-timestamps branch July 3, 2026 17:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant