feat: sequence fields and monotonic timestamps in the schema DSL#56
Merged
Conversation
Two new row-order-aware field specs for high-volume generation:
- ("sequence", start) / "sequence": auto-increment integers — cheap
unique primary keys at any n, with no retry loop or uniqueness
tracking. Int64 in Arrow/Parquet; usable as a derive source.
- ("datetime_seq", start, min_increment, max_increment): the first row
is start, each following row advances by a random whole number of
seconds in [min_increment, max_increment], producing time-ordered
event logs.
Both continue across the chunks of records_to_file() and the async
methods, restart on each generation call, and compose with nullable
(null rows don't consume a value or advance the clock). unique()
wrapping is rejected with an explanatory error.
The per-call UniqueState struct is renamed to GenerationState since it
now also carries sequence counters and last-emitted timestamps.
Closes the first and third Phase 1 roadmap items.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
- Reserve "sequence" in RESERVED_PROVIDER_NAMES: it is a built-in string
spec, so a custom provider with that name would register successfully
and then be silently shadowed in schemas.
- Reject a hand-built FieldSpec::Simple("sequence") at validation with a
pointer to FieldSpec::Sequence, instead of failing at generation with
"Unknown type: sequence".
- Narrow the README claim that unique accepts any spec except nullable;
derive, sequence, and datetime_seq are also rejected.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



Implements the first and third items of Phase 1 on the roadmap ("finish the single-table story at scale"): sequence fields and monotonic timestamps. These are the two row-order-aware specs, bundled because they share the same per-call state machinery.
Sequence fields
("sequence", start)— or bare"sequence"for start=1 — generates auto-increment integers in row order:Every seeded table needs a primary key, and this is the only cheap way to get unique integers at very large
n— the("unique", spec)retry loop degrades as the value space saturates. Sequence columns areint64in Arrow/Parquet and work as derive sources.Monotonic timestamps
("datetime_seq", start, min_increment, max_increment)— the first row isstart, each following row advances by a random whole number of seconds in the given range:Random in-range datetimes produce unsorted "event logs"; real event data is roughly time-ordered and time-series databases behave very differently on sorted input.
startacceptsYYYY-MM-DDTHH:MM:SS,YYYY-MM-DD HH:MM:SS, orYYYY-MM-DD(midnight).min_increment >= 1gives strictly increasing timestamps;0allows realistic duplicates.Semantics
records_to_file()and the async methods (a 100M-row streamed file gets one contiguous sequence), and restart on each generation call.nullable; null rows don't consume a sequence value or advance the clock.("unique", ...)wrapping is rejected with an explanatory error — sequences are already unique, anddatetime_seqis unique whenmin_increment >= 1.ValueError, not a panic (Duration::try_seconds,checked_add).records(),records_tuples(),records_arrow(), all serialized formats,records_to_file(), and the async methods.Implementation notes
UniqueStatestruct is renamed toGenerationState: it now carries sequence counters and last-emitted timestamps alongside unique-value tracking, so cross-chunk continuity reuses the exact mechanism that already threads unique state through the chunked paths. The rename is mechanical (~40 lines of the diff).generate_value()(stateless single-value path) rejects the new specs with a clear error, following the existingderive/custom-provider precedent — reaching it would mean a code path skipped the shared state and would silently repeat values.nullable/uniquewrappers.Tests
test_schema_dsl.py, including cross-chunk continuity regressions forrecords_to_file()and the async methods, output-format coverage (CSV/NDJSON/Arrow), and error cases.-D warnings, mypy --strict, ruff.Docs
🤖 Generated with Claude Code