Skip to content

Commit d54fb46

Browse files
rustyconoverclaude
andcommitted
Rename vgi/examples → vgi/_test_fixtures; gate fixture deps behind extras
The "examples" directory has grown to ~12,000 lines / 16 files — it's the regression-test fixture catalog used by the pytest suite, the C++ sqllogictest harness, and the conformance suite, not tutorial code. Calling it "examples" misled readers and pulled fixture-only deps (numpy, sqlglot) into the core install. What changed: - Moved 13 fixture files vgi/examples/*.py → vgi/_test_fixtures/*.py and 3 writable-related files into vgi/_test_fixtures/writable/ as a subpackage. vgi/transactor/ stays at top level — it's production code, not a fixture, even though only the writable fixtures consume it today. - Renamed vgi-example-* scripts to vgi-fixture-* (no deprecation shims per request). vgi-transactor keeps its name. vgi-fixture-worker still serves both example and writable catalogs, falling back to example-only when the writable extra is absent. - Gated deps behind new extras: - [test-fixtures] → numpy - [transactor] → sqlglot - [test-fixtures-writable] → vgi[test-fixtures] + vgi[transactor] - [dev] now includes all of the above so contributors don't notice - Core install drops numpy + sqlglot - Friendly errors: vgi-fixture-worker, vgi-fixture-writable-worker, and vgi-transactor each emit a clear "install vgi[X]" message and exit 1 when the relevant extra is missing, instead of a raw ImportError. - Swept all import sites and script-name references across CLAUDE.md, README.md, docs/cli.md, vgi/client/cli.py help/default, conftest (example_worker → fixture_worker), tests, and the C++ companion repo. Verified: - uv run mypy vgi/ → clean - uv run pytest tests/ -n auto → 1526 passed / 71 skipped - Lean-install smoke test (no extras): `from vgi.client import Client` works; vgi-fixture-worker / vgi-fixture-writable-worker / vgi-transactor all print the install hint and exit 1 - C++ aggregate suite passes (12 / 3 skipped) under VGI_TEST_WORKER=vgi-fixture-worker - C++ database_tags.test passes with the renamed worker tag Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent 443637f commit d54fb46

84 files changed

Lines changed: 749 additions & 607 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

CLAUDE.md

Lines changed: 11 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -64,7 +64,7 @@ ensure all tests pass — not just the new/changed test file. Adding a function
6464
**Subprocess transport** (worker spawned as a child process):
6565
```bash
6666
cd ~/Development/vgi
67-
VGI_TEST_WORKER="uv run --project ~/Development/vgi-python vgi-example-worker" \
67+
VGI_TEST_WORKER="uv run --project ~/Development/vgi-python vgi-fixture-worker" \
6868
./build/release/test/unittest "test/sql/integration/*"
6969
```
7070

@@ -110,7 +110,7 @@ with elapsed milliseconds. This helps identify slow queries and bottlenecks.
110110
```bash
111111
# Timing output goes to stderr, grep for the bracket-prefixed lines:
112112
cd ~/Development/vgi
113-
VGI_TEST_WORKER="uv run --project ~/Development/vgi-python vgi-example-worker" \
113+
VGI_TEST_WORKER="uv run --project ~/Development/vgi-python vgi-fixture-worker" \
114114
./build/release/test/unittest "test/sql/integration/table/writable_table*" \
115115
2>&1 | grep "^\[stmt\|^\[query" | sort -t']' -k2 -rn
116116
```
@@ -157,7 +157,7 @@ Subprocess worker wrapper (`/tmp/vgi-coverage-worker.sh`):
157157
```bash
158158
#!/bin/bash
159159
cd /Users/rusty/Development/vgi-python
160-
exec uv run coverage run --parallel-mode -m vgi.examples.worker "$@"
160+
exec uv run coverage run --parallel-mode -m vgi._test_fixtures.worker "$@"
161161
```
162162

163163
HTTP server wrapper (`~/Development/vgi/test/run_http_integration_coverage.sh`):
@@ -189,8 +189,8 @@ external record batch offloading without S3 or any cloud infrastructure.
189189

190190
**Running the example server with demo storage:**
191191
```bash
192-
vgi-example-http --demo-storage
193-
vgi-example-http --demo-storage --externalize-threshold-bytes 4096 --externalize-compression zstd
192+
vgi-fixture-http --demo-storage
193+
vgi-fixture-http --demo-storage --externalize-threshold-bytes 4096 --externalize-compression zstd
194194
```
195195

196196
When `--demo-storage` is enabled:
@@ -230,9 +230,9 @@ FIXME: complete this.
230230
## CLI Commands
231231

232232
```bash
233-
vgi-example-worker # Run example worker
234-
vgi-client --input data.parquet --function echo --worker vgi-example-worker
235-
vgi-client --input data.parquet --function sum_all_columns --worker vgi-example-worker
233+
vgi-fixture-worker # Run example worker
234+
vgi-client --input data.parquet --function echo --worker vgi-fixture-worker
235+
vgi-client --input data.parquet --function sum_all_columns --worker vgi-fixture-worker
236236
```
237237

238238
## Environment Variables
@@ -276,7 +276,7 @@ Set `VGI_WORKER_DEBUG=1` to enable comprehensive debugging for worker failures.
276276
2. **Client side**: Forces `passthrough_stderr=True`, streaming worker logs to the terminal in real-time
277277

278278
```bash
279-
VGI_WORKER_DEBUG=1 vgi-example-worker
279+
VGI_WORKER_DEBUG=1 vgi-fixture-worker
280280
```
281281

282282
When used from the Python client without this env var, errors from worker failures automatically include captured stderr (last 50 lines) in the `ClientError` message. This means integrators using C++ or other clients get the Python traceback in the error message instead of just a generic exit code.
@@ -301,7 +301,7 @@ Accepts `1`, `true`, or `yes` (case-insensitive). Zero overhead when not set.
301301
Enable `VGI_FILTER_DEBUG=1` to trace filter pushdown deserialization, parsing, and evaluation. Useful for debugging filter pushdown issues and understanding how filters are applied.
302302

303303
```bash
304-
VGI_FILTER_DEBUG=1 vgi-example-worker
304+
VGI_FILTER_DEBUG=1 vgi-fixture-worker
305305
```
306306

307307
**Key events logged:**
@@ -415,7 +415,7 @@ during the bind phase, allowing output schema to depend on setting values.
415415
Client passes settings when invoking a method:
416416

417417
```python
418-
with Client("vgi-example-worker") as client:
418+
with Client("vgi-fixture-worker") as client:
419419
for batch in client.table_function(
420420
function_name="settings_aware",
421421
arguments=Arguments(positional=(pa.scalar(10),)),

README.md

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -433,10 +433,10 @@ without S3 or any cloud infrastructure:
433433

434434
```bash
435435
# Start with demo storage (4 KiB threshold for testing)
436-
vgi-example-http --demo-storage --externalize-threshold-bytes 4096
436+
vgi-fixture-http --demo-storage --externalize-threshold-bytes 4096
437437

438438
# With zstd compression
439-
vgi-example-http --demo-storage --externalize-threshold-bytes 4096 --externalize-compression zstd
439+
vgi-fixture-http --demo-storage --externalize-threshold-bytes 4096 --externalize-compression zstd
440440
```
441441

442442
When `--demo-storage` is enabled:
@@ -465,13 +465,13 @@ Workers support `--debug`, `--log-level`, `--log-format`, and `--log-logger` opt
465465

466466
```bash
467467
# Enable debug logging
468-
vgi-example-worker --debug
468+
vgi-fixture-worker --debug
469469

470470
# JSON-formatted logs for structured pipelines
471-
vgi-example-worker --log-format json
471+
vgi-fixture-worker --log-format json
472472

473473
# Target a specific logger
474-
vgi-example-worker --log-level DEBUG --log-logger vgi.worker
474+
vgi-fixture-worker --log-level DEBUG --log-logger vgi.worker
475475
```
476476

477477
You can also use the `VGI_WORKER_DEBUG=1` environment variable, which enables `--debug` on the worker and stderr passthrough on the client without changing any code or CLI flags:

docs/cli.md

Lines changed: 13 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -7,8 +7,7 @@ VGI provides CLI tools for invoking functions and managing catalogs without writ
77
| Command | Description |
88
|---------|-------------|
99
| `vgi-client` | Invoke functions and manage catalogs |
10-
| `vgi-example-worker` | Run the example worker with demo functions |
11-
| `vgi-example-catalog-worker` | Run an in-memory catalog for testing |
10+
| `vgi-fixture-worker` | Run the example worker with demo functions |
1211

1312
---
1413

@@ -31,7 +30,7 @@ vgi-client [OPTIONS]
3130
| `--format FORMAT` | Output format: `json` (default), `csv`, `parquet` |
3231
| `--function NAME` | Function name to invoke |
3332
| `--args JSON` | Function arguments as JSON array (default: `[]`) |
34-
| `--worker PATH` | Worker command (default: `vgi-example-worker`) |
33+
| `--worker PATH` | Worker command (default: `vgi-fixture-worker`) |
3534
| `--type TYPE` | Function type: `auto`, `table`, `table-in-out`, `scalar` |
3635
| `--projection-id N` | Column IDs to project (repeatable) |
3736
| `--max-workers N` | Limit parallel workers |
@@ -492,7 +491,7 @@ vgi-client catalog detach $ATTACH_ID --worker ./worker.py
492491

493492
## Worker Logging
494493

495-
All workers that use `Worker.main()` (including `vgi-example-worker`) support
494+
All workers that use `Worker.main()` (including `vgi-fixture-worker`) support
496495
logging options on the command line. Logs are written to stderr.
497496

498497
### Options
@@ -511,16 +510,16 @@ logging options on the command line. Logs are written to stderr.
511510

512511
```bash
513512
# Enable debug logging
514-
vgi-example-worker --debug
513+
vgi-fixture-worker --debug
515514

516515
# Set WARNING level only
517-
vgi-example-worker --log-level WARNING
516+
vgi-fixture-worker --log-level WARNING
518517

519518
# Target a specific logger at DEBUG
520-
vgi-example-worker --log-level DEBUG --log-logger vgi.worker
519+
vgi-fixture-worker --log-level DEBUG --log-logger vgi.worker
521520

522521
# JSON-formatted logs (for structured log pipelines)
523-
vgi-example-worker --log-format json
522+
vgi-fixture-worker --log-format json
524523
```
525524

526525
### Available Loggers
@@ -594,12 +593,12 @@ app = create_app(
594593

595594
## Example Workers
596595

597-
### vgi-example-worker
596+
### vgi-fixture-worker
598597

599598
Runs the built-in example worker with demo functions.
600599

601600
```bash
602-
vgi-example-worker
601+
vgi-fixture-worker
603602
```
604603

605604
**Available functions:**
@@ -628,12 +627,13 @@ vgi-example-worker
628627
| `bernoulli` | scalar | Generate random booleans (VOLATILE) |
629628
| `random_bytes` | scalar | Generate pseudo-random binary blobs |
630629

631-
### vgi-example-catalog-worker
630+
### In-memory catalog example
632631

633-
Runs an in-memory catalog implementation for testing.
632+
The mutable catalog demo (`vgi/examples/catalog.py`) is no longer installed as a
633+
console script. Run it from a source checkout via:
634634

635635
```bash
636-
vgi-example-catalog-worker
636+
python -m vgi._test_fixtures.catalog
637637
```
638638

639639
---

pyproject.toml

Lines changed: 16 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -7,10 +7,8 @@ requires-python = ">=3.13"
77
dependencies = [
88
"click",
99
"pyarrow",
10-
"sqlglot",
1110
"typer>=0.9",
1211
"platformdirs",
13-
"numpy>=2.4.1",
1412
"vgi-rpc @ file:///Users/rusty/Development/vgi-rpc",
1513
"duckdb>=1.5.0",
1614
]
@@ -23,6 +21,12 @@ azure = [
2321
"pymssql>=2.3.0",
2422
"azure-identity>=1.16.0",
2523
]
24+
test-fixtures = ["numpy>=2.4.1"]
25+
transactor = ["sqlglot"]
26+
test-fixtures-writable = [
27+
"vgi[test-fixtures]",
28+
"vgi[transactor]",
29+
]
2630
dev = [
2731
"mypy",
2832
"pyarrow-stubs",
@@ -32,17 +36,18 @@ dev = [
3236
"pytest-ruff",
3337
"pytest-xdist",
3438
"ruff",
39+
"vgi[test-fixtures,test-fixtures-writable,http,oauth,otel,azure]",
3540
"vgi-rpc[conformance,external,http,oauth,otel]",
3641
]
3742
[project.scripts]
3843
vgi-serve = "vgi.serve:main"
3944
vgi-client = "vgi.client.cli:main"
40-
vgi-example-worker = "vgi.examples.worker:main"
41-
vgi-example-http = "vgi.examples.http_server:main"
42-
vgi-example-versioned-worker = "vgi.examples.versioned:main"
43-
vgi-example-versioned-tables-worker = "vgi.examples.versioned_tables:main"
44-
vgi-example-attach-options-worker = "vgi.examples.attach_options:main"
45-
vgi-example-writable-worker = "vgi.examples.writable_worker:main"
45+
vgi-fixture-worker = "vgi._test_fixtures.worker:main"
46+
vgi-fixture-http = "vgi._test_fixtures.http_server:main"
47+
vgi-fixture-versioned-worker = "vgi._test_fixtures.versioned:main"
48+
vgi-fixture-versioned-tables-worker = "vgi._test_fixtures.versioned_tables:main"
49+
vgi-fixture-attach-options-worker = "vgi._test_fixtures.attach_options:main"
50+
vgi-fixture-writable-worker = "vgi._test_fixtures.writable.worker:main"
4651
vgi-transactor = "vgi.transactor.server:main"
4752

4853
[tool.hatch.metadata]
@@ -85,7 +90,7 @@ ignore_missing_imports = true
8590

8691
[[tool.mypy.overrides]]
8792
# These files have type: ignore comments for ty that mypy doesn't need
88-
module = ["vgi.examples.scalar", "vgi.table_function"]
93+
module = ["vgi._test_fixtures.scalar", "vgi.table_function"]
8994
warn_unused_ignores = false
9095

9196
[tool.ty.environment]
@@ -107,8 +112,8 @@ sigterm = true
107112
branch = true
108113
# What to measure
109114
source = ["vgi"]
110-
# Omit test files and examples from coverage
111-
omit = ["vgi/examples/*", "tests/*"]
115+
# Omit test fixtures and tests from coverage
116+
omit = ["vgi/_test_fixtures/*", "tests/*"]
112117

113118
[tool.coverage.report]
114119
# Fail if coverage drops below this threshold

scripts/measure_startup.py

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -105,7 +105,7 @@ def profile_startup_phases() -> None:
105105
profiler.enable()
106106

107107
# Import the worker module
108-
import vgi.examples.worker # noqa: F401
108+
import vgi._test_fixtures.worker # noqa: F401
109109

110110
profiler.disable()
111111

@@ -138,8 +138,8 @@ def main() -> None:
138138
)
139139
parser.add_argument(
140140
"--worker",
141-
default="vgi-example-worker",
142-
help="Worker command to measure (default: vgi-example-worker)",
141+
default="vgi-fixture-worker",
142+
help="Worker command to measure (default: vgi-fixture-worker)",
143143
)
144144
args = parser.parse_args()
145145

@@ -154,7 +154,7 @@ def main() -> None:
154154

155155
if args.importtime:
156156
print("\n### Import Time Analysis ###\n")
157-
import_times = measure_importtime("vgi.examples.worker")
157+
import_times = measure_importtime("vgi._test_fixtures.worker")
158158

159159
# Sort by cumulative time (descending)
160160
sorted_times = sorted(import_times.items(), key=lambda x: -x[1])
@@ -190,7 +190,7 @@ def main() -> None:
190190

191191
# Also show import time summary
192192
print("\n### Top 10 Slowest Imports ###\n")
193-
import_times = measure_importtime("vgi.examples.worker")
193+
import_times = measure_importtime("vgi._test_fixtures.worker")
194194
sorted_times = sorted(import_times.items(), key=lambda x: -x[1])
195195

196196
print(f"{'Module':<50} {'Time (ms)':>10}")

tests/_http_fixtures.py

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
"""Shared helpers for spawning ``vgi-example-http`` as a subprocess.
1+
"""Shared helpers for spawning ``vgi-fixture-http`` as a subprocess.
22
33
Lifted from ``tests/test_http_demo_storage.py`` so that conformance tests and
44
other HTTP-dependent tests can reuse a single lifecycle.
@@ -29,15 +29,15 @@ def run_example_http_server(
2929
extra_args: Sequence[str] = (),
3030
env: dict[str, str] | None = None,
3131
) -> Iterator[None]:
32-
"""Run ``vgi-example-http`` in a subprocess on ``port``.
32+
"""Run ``vgi-fixture-http`` in a subprocess on ``port``.
3333
3434
``extra_args`` are appended verbatim — e.g. ``("--demo-storage",
3535
"--externalize-threshold-bytes", "4096")``.
3636
"""
3737
cmd = [
3838
sys.executable,
3939
"-m",
40-
"vgi.examples.http_server",
40+
"vgi._test_fixtures.http_server",
4141
"--host",
4242
"127.0.0.1",
4343
"--port",
@@ -75,7 +75,7 @@ def start_http_worker(
7575
extra_args: Sequence[str] = (),
7676
env: dict[str, str] | None = None,
7777
) -> str:
78-
"""Allocate a port, start ``vgi-example-http`` under ``stack``, return base URL."""
78+
"""Allocate a port, start ``vgi-fixture-http`` under ``stack``, return base URL."""
7979
port = free_port()
8080
base_url = f"http://127.0.0.1:{port}"
8181
stack.enter_context(run_example_http_server(port=port, extra_args=tuple(extra_args), env=env))

tests/catalog/test_client_catalog.py

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,10 +8,12 @@
88
see tests/catalog/test_integration.py which exercises all catalog operations.
99
"""
1010

11+
import sys
12+
1113
from vgi.client import Client
1214

1315
# Worker command for catalog tests
14-
CATALOG_WORKER = "vgi-example-catalog-worker"
16+
CATALOG_WORKER = f"{sys.executable} -m vgi._test_fixtures.catalog"
1517

1618

1719
class TestClientCatalogStatelessOperations:

tests/catalog/test_example_worker_catalog.py

Lines changed: 12 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@
66

77
import pyarrow as pa
88

9+
from vgi._test_fixtures.worker import ExampleWorker
910
from vgi.catalog import (
1011
AttachId,
1112
FunctionInfo,
@@ -17,10 +18,9 @@
1718
ViewInfo,
1819
)
1920
from vgi.client import Client
20-
from vgi.examples.worker import ExampleWorker
2121

2222
# Worker command for catalog tests
23-
EXAMPLE_WORKER = "vgi-example-worker"
23+
EXAMPLE_WORKER = "vgi-fixture-worker"
2424

2525

2626
def _get_expected_function_names() -> set[str]:
@@ -132,8 +132,17 @@ def test_all_example_functions_listed(self) -> None:
132132
)
133133
)
134134

135+
# Get aggregate functions
136+
aggregate_funcs = list(
137+
client.schema_contents(
138+
attach_id=attach_result.attach_id,
139+
name="main",
140+
type=SchemaObjectType.AGGREGATE_FUNCTION,
141+
)
142+
)
143+
135144
# Combine all functions
136-
contents = table_funcs + scalar_funcs
145+
contents = table_funcs + scalar_funcs + aggregate_funcs
137146

138147
# Get function names
139148
function_names = {item.name for item in contents}

0 commit comments

Comments
 (0)