Python (feat): pytest dev feature branch

.githooks/pre-push-python/extras.sh

@@ -1,5 +1,8 @@
 # ensure generated pyproject.toml extras are up-to-date
 
+# Clear git env vars set by the parent hook so git commands resolve the work tree normally
+unset GIT_DIR GIT_WORK_TREE GIT_INDEX_FILE GIT_PREFIX
+
 # Store the root directory of the repository
 REPO_ROOT="$(git rev-parse --show-toplevel)"
 PYTHON_DIR="$REPO_ROOT/python"

.githooks/pre-push-python/fmt-lint.sh

@@ -2,6 +2,9 @@
 
 set -e
 
+# Clear git env vars set by the parent hook so git commands resolve the work tree normally
+unset GIT_DIR GIT_WORK_TREE GIT_INDEX_FILE GIT_PREFIX
+
 # Store the root directory of the repository
 REPO_ROOT="$(git rev-parse --show-toplevel)"
 PYTHON_DIR="$REPO_ROOT/python"

.githooks/pre-push-python/stubs.sh

@@ -1,5 +1,8 @@
 # ensure generated python stubs are up-to-date, from sync clients
 
+# Clear git env vars set by the parent hook so git commands resolve the work tree normally
+unset GIT_DIR GIT_WORK_TREE GIT_INDEX_FILE GIT_PREFIX
+
 # Store the root directory of the repository
 REPO_ROOT="$(git rev-parse --show-toplevel)"
 PYTHON_DIR="$REPO_ROOT/python"

python/docs/examples/index.md

@@ -7,6 +7,7 @@ This section contains interactive Jupyter notebook examples demonstrating how to
 - **[Basic Usage](basic.ipynb)** - Introduction to the Sift Python client, covering basic operations and API usage
 - **[Data Ingestion](ingestion.ipynb)** - Learn how to ingest telemetry data into Sift using various methods
 - **[Pytest Plugin](pytest_plugin.md)** - Turn a pytest run into a Sift TestReport with measurements, nested steps, and pass/fail outcomes
+- **[Pytest Plugin Quickstart](pytest_plugin_quickstart.md)** - Guided tour of the runnable demo project under `python/examples/pytest_plugin/`
 
 ## Running Examples Locally
 

python/docs/examples/pytest_plugin_quickstart.md

@@ -0,0 +1,177 @@
+# Pytest Plugin Quickstart
+
+A walkthrough of the runnable demo at
+[`python/examples/pytest_plugin/`](https://github.com/sift-stack/sift/tree/main/python/examples/pytest_plugin).
+The demo is a self-contained pytest project that exercises every layer of the
+plugin's step tree: packages, modules, classes (including nested), parametrize
+axes, manual substeps, and gate markers. It also includes a tests directory
+that uses no Sift APIs at all, to show how the autouse fixtures capture plain
+pytest tests for free.
+
+For a conceptual reference (fixtures, ini flags, status semantics), see
+[Pytest Plugin](pytest_plugin.md).
+
+## Project layout
+
+```
+examples/pytest_plugin/
+├── conftest.py                            # registers the plugin
+├── pytest.ini                             # available ini knobs (all commented at defaults)
+├── .env.example                           # credential template
+└── tests/
+    ├── pytest_only/                       # subpackage step
+    │   ├── __init__.py
+    │   └── test_pytest_only_demo.py       # plain pytest, no Sift APIs
+    └── with_sift/                         # subpackage step
+        ├── __init__.py
+        └── test_with_sift_demo.py         # measurements, substeps, classes, parametrize, gates
+```
+
+Every Python package (directory with `__init__.py`), test file, and test class
+above each test becomes its own parent step in the report tree.
+
+## `conftest.py`
+
+A single `pytest_plugins` declaration loads the plugin; `load_dotenv()` is
+optional and just lets the default `sift_client` fixture pick up
+`SIFT_API_KEY` / `SIFT_GRPC_URI` / `SIFT_REST_URI` from a local `.env`.
+
+```python title="conftest.py"
+--8<-- "examples/pytest_plugin/conftest.py"
+```
+
+## `pytest.ini`
+
+Every knob is commented at its default value. Uncomment any line to opt out of
+a layer of the step tree.
+
+```ini title="pytest.ini"
+--8<-- "examples/pytest_plugin/pytest.ini"
+```
+
+## `.env.example`
+
+```bash title=".env.example"
+--8<-- "examples/pytest_plugin/.env.example"
+```
+
+## The pytest_only module
+
+Plain pytest tests with no `sift_client` imports, no `step` fixture, no
+markers. Each one still becomes a leaf step in the report tree. The plugin's
+autouse fixtures capture pass/fail automatically.
+
+```python title="tests/pytest_only/test_pytest_only_demo.py"
+--8<-- "examples/pytest_plugin/tests/pytest_only/test_pytest_only_demo.py"
+```
+
+## The with_sift module
+
+Exercises the plugin's full surface: numeric / string / bool bounds, nested
+`step.substep`, `@pytest.mark.sift_exclude`, class steps with docstring
+descriptions, nested classes, stacked `@pytest.mark.parametrize`, and
+`step.report_outcome`.
+
+```python title="tests/with_sift/test_with_sift_demo.py"
+--8<-- "examples/pytest_plugin/tests/with_sift/test_with_sift_demo.py"
+```
+
+## Run it
+
+### Without Sift credentials
+
+```bash
+cd python/examples/pytest_plugin
+pytest --sift-disabled -v
+```
+
+`--sift-disabled` makes the plugin a no-op transport: `step.measure(...)`
+still evaluates bounds and returns a real pass/fail boolean, but nothing
+contacts Sift and no log file is written. Useful for previewing the report
+tree or unit-testing measurement logic.
+
+### Against a real Sift org
+
+```bash
+cp .env.example .env
+# Fill in SIFT_API_KEY / SIFT_GRPC_URI / SIFT_REST_URI
+pytest -v
+```
+
+A `TestReport` shows up in Sift once the session finishes.
+
+### Offline (record now, replay later)
+
+```bash
+pytest --sift-offline --sift-log-file=/tmp/sift-demo.jsonl -v
+# Later, from anywhere with credentials:
+import-test-result-log /tmp/sift-demo.jsonl
+```
+
+## Expected report tree
+
+With the plugin's defaults (every layer enabled), the demo produces:
+
+```
+TestReport (FAILED, since failures propagate up from leaves)
+├── pytest_only                         ← package step (FAILED)
+│   └── test_pytest_only_demo.py        ← module step (FAILED)
+│       ├── test_passes                                              PASSED
+│       ├── test_uses_a_pytest_fixture                               PASSED
+│       ├── test_assertion_failure_marks_step_failed                 FAILED
+│       ├── test_skipped                                             SKIPPED
+│       ├── test_unexpected_exception_marks_step_errored             ERROR
+│       ├── test_parametrize_without_step
+│       │   ├── value='v1'                                           PASSED
+│       │   └── value='v2'                                           PASSED
+│       └── TestPytestClass
+│           └── test_method                                          PASSED
+└── with_sift                           ← package step (FAILED)
+    └── test_with_sift_demo.py          ← module step (FAILED)
+        ├── test_measurements                                        PASSED
+        ├── test_substeps                                            PASSED
+        │   ├── phase_1
+        │   └── phase_2
+        │       └── phase_2a
+        │   (test_excluded: @sift_exclude, runs in pytest, NOT in tree)
+        ├── test_measure_series                                      PASSED
+        ├── test_failed_measurement_marks_sift_step_failed           FAILED  (pytest PASSED)
+        ├── test_assert_measurements_passed_at_end                                FAILED  (pytest FAILED)
+        ├── test_report_level_metadata                               PASSED
+        └── TestClassStep
+            ├── test_parametrize
+            │   ├── axis_a='a1'
+            │   │   ├── axis_b='b1'                                  PASSED
+            │   │   └── axis_b='b2'                                  PASSED
+            │   └── axis_a='a2'
+            │       ├── axis_b='b1'                                  PASSED
+            │       └── axis_b='b2'                                  PASSED
+            └── TestNested
+                └── test_report_outcome
+                    └── check                                        PASSED
+```
+
+The `pytest_only` module deliberately includes one failing, one skipped, and
+one erroring test so the demo shows every `TestStatus` mapping (`FAILED` for
+assertions, `SKIPPED` for `pytest.skip`, `ERROR` for any other exception).
+The `with_sift` module shows two patterns for handling measurement results:
+`test_failed_measurement_marks_sift_step_failed` lets the test keep passing
+in pytest while the Sift step is `FAILED` (useful when measurements are
+diagnostic data you want to collect regardless of outcome); and
+`test_assert_measurements_passed_at_end` takes every measurement first and
+then asserts `step.measurements_passed` once at the end, so every
+measurement still lands in the report even when one fails. The end-of-test
+assertion is the recommended pattern: asserting on an individual
+`step.measure(...)` call short-circuits on the first failure and skips
+every measurement that follows. Expected
+pytest output is `16 passed, 3 failed, 1 skipped`.
+
+Flip any of the `sift_*_step` / `sift_parametrize_nesting` flags in
+`pytest.ini` to `false` to collapse a layer.
+
+## Next steps
+
+- [Pytest Plugin](pytest_plugin.md): conceptual reference covering fixtures,
+  ini flags, status semantics, and layout-mapping examples.
+- The demo's [README](https://github.com/sift-stack/sift/blob/main/python/examples/pytest_plugin/README.md)
+  on GitHub mirrors this page and is the canonical source.

python/examples/pytest_plugin/.env.example

@@ -0,0 +1,3 @@
+SIFT_API_KEY=your-api-key
+SIFT_GRPC_URI=your-org.grpc.example.com
+SIFT_REST_URI=https://your-org.rest.example.com

python/examples/pytest_plugin/README.md

@@ -0,0 +1,119 @@
+# Pytest plugin demo
+
+A self-contained pytest project that exercises every feature of
+`sift_client.pytest_plugin`: package / module / class / parametrize step
+nesting, nested classes, manual substeps, `step.measure(...)` against
+numeric / string / bool bounds, gate markers, and the ini opt-outs.
+
+```
+examples/pytest_plugin/
+├── conftest.py                            # registers the plugin
+├── pytest.ini                             # available ini knobs (all commented at defaults)
+├── .env.example                           # credential template (copy to .env for local runs)
+└── tests/
+    ├── pytest_only/                       # subpackage step: `pytest_only` opens a parent step
+    │   ├── __init__.py
+    │   └── test_pytest_only_demo.py       # plain pytest tests with no Sift APIs
+    └── with_sift/                         # subpackage step: `with_sift` opens a parent step
+        ├── __init__.py
+        └── test_with_sift_demo.py         # measurements, substeps, classes, nested classes,
+                                            # stacked parametrize, sift_exclude marker
+```
+
+Every layer of organization shows up in the report tree: Python packages
+(directories with `__init__.py`), modules (test files), classes (including
+nested classes), and parametrize axes each open a parent step. Flip
+`sift_package_step`, `sift_module_step`, `sift_class_step`, or
+`sift_parametrize_nesting` to `false` in `pytest.ini` to disable this behavior.
+
+## Run it
+
+**Against a real Sift org**:
+
+```bash
+cp .env.example .env
+# Fill in SIFT_API_KEY / SIFT_GRPC_URI / SIFT_REST_URI
+pytest -v
+```
+
+A `TestReport` shows up in Sift once the session finishes.
+
+**Offline (record now, replay later - intended for offline environments)**:
+
+```bash
+pytest --sift-offline --sift-log-file=/tmp/sift-demo.jsonl -v
+# Later, from anywhere with credentials:
+import-test-result-log /tmp/sift-demo.jsonl
+```
+
+## What the report tree looks like
+
+With the plugin's defaults (everything in `pytest.ini` left commented), running
+this demo produces a tree like:
+
+```
+TestReport (FAILED, since failures propagate up from leaves)
+├── pytest_only                         ← package step (FAILED)
+│   └── test_pytest_only_demo.py        ← module step (FAILED)
+│       ├── test_passes                                              PASSED
+│       ├── test_uses_a_pytest_fixture                               PASSED
+│       ├── test_assertion_failure_marks_step_failed                 FAILED
+│       ├── test_skipped                                             SKIPPED
+│       ├── test_unexpected_exception_marks_step_errored             ERROR
+│       ├── test_parametrize_without_step
+│       │   ├── value='v1'                                           PASSED
+│       │   └── value='v2'                                           PASSED
+│       └── TestPytestClass
+│           └── test_method                                          PASSED
+└── with_sift                           ← package step (FAILED)
+    └── test_with_sift_demo.py          ← module step (FAILED)
+        ├── test_measurements                                        PASSED
+        ├── test_substeps                                            PASSED
+        │   ├── phase_1
+        │   └── phase_2
+        │       └── phase_2a
+        │   (test_excluded: @sift_exclude, runs in pytest, NOT in tree)
+        ├── test_measure_series                                      PASSED
+        ├── test_failed_measurement_marks_sift_step_failed           FAILED  (pytest PASSED)
+        ├── test_assert_measurements_passed_at_end                                FAILED  (pytest FAILED)
+        ├── test_report_level_metadata                               PASSED
+        └── TestClassStep
+            ├── test_parametrize
+            │   ├── axis_a='a1'
+            │   │   ├── axis_b='b1'                                  PASSED
+            │   │   └── axis_b='b2'                                  PASSED
+            │   └── axis_a='a2'
+            │       ├── axis_b='b1'                                  PASSED
+            │       └── axis_b='b2'                                  PASSED
+            └── TestNested
+                └── test_report_outcome
+                    └── check                                        PASSED
+```
+
+The `pytest_only` module deliberately includes one failing, one skipped, and
+one erroring test so the demo shows every `TestStatus` mapping (`FAILED` for
+assertions, `SKIPPED` for `pytest.skip`, `ERROR` for any other exception).
+The `with_sift` module shows two patterns for handling measurement results:
+`test_failed_measurement_marks_sift_step_failed` lets the test keep passing
+in pytest while the Sift step is `FAILED` (useful when measurements are
+diagnostic data you want to collect regardless of outcome); and
+`test_assert_measurements_passed_at_end` takes every measurement first and
+then asserts `step.measurements_passed` once at the end, so every
+measurement still lands in the report even when one fails. The end-of-test
+assertion is the recommended pattern: asserting on an individual
+`step.measure(...)` call short-circuits on the first failure and skips
+every measurement that follows. Expected
+pytest output is `16 passed, 3 failed, 1 skipped`.
+
+Toggle any of the `sift_*_step` / `sift_parametrize_nesting` flags in
+`pytest.ini` to `false` to collapse a layer.
+
+## What each file demonstrates
+
+| File | Feature |
+|---|---|
+| `conftest.py` | Plugin registration via `pytest_plugins`; optional `load_dotenv()` |
+| `pytest.ini` | The four nesting flags + git metadata flag at their defaults |
+| `tests/pytest_only/test_pytest_only_demo.py` | Plain pytest tests with no Sift APIs. The plugin captures pass/fail automatically; covers functions, fixtures, parametrize, classes, plus one each of `AssertionError` (FAILED), `pytest.skip` (SKIPPED), and a raised `ValueError` (ERROR) |
+| `tests/with_sift/test_with_sift_demo.py` | `step.measure` (numeric/string/bool bounds, units, description, metadata, `channel_names`), `step.measure_avg` and `step.measure_all` for series, an out-of-bounds measurement (pytest PASSED, Sift step FAILED), the recommended `assert step.measurements_passed` end-of-test pattern that fails pytest while still recording every measurement, nested `step.substep` (with step-level `metadata=...`), `@pytest.mark.sift_exclude`, class step + class docstring → description, nested classes, stacked `@pytest.mark.parametrize`, `step.report_outcome`, and session-level metadata via `report_context.report.update({...})` |
+| `tests/{pytest_only,with_sift}/__init__.py` | Each Python package (directory with `__init__.py`) becomes a parent step in the report tree |

python/examples/pytest_plugin/conftest.py

@@ -0,0 +1,15 @@
+"""Project-level conftest for the pytest plugin demo.
+
+A single ``pytest_plugins`` declaration is enough to load the plugin — its
+fixtures, hooks, and CLI options register through standard pytest machinery
+from there. ``load_dotenv()`` is optional; it just lets the default
+``sift_client`` fixture pick up ``SIFT_API_KEY`` / ``SIFT_GRPC_URI`` /
+``SIFT_REST_URI`` from a local ``.env`` when running against a real Sift org.
+These can also be set as environment variables using your preferred method.
+"""
+
+from dotenv import load_dotenv
+
+load_dotenv()
+
+pytest_plugins = ["sift_client.pytest_plugin"]

python/examples/pytest_plugin/pytest.ini

@@ -0,0 +1,11 @@
+[pytest]
+# Defaults give you the full step tree: every package, module, class, and
+# parametrize axis becomes a parent step. These are the available ini options
+# and their defaults.
+#
+# sift_autouse = true              # autouse fixtures (default: true)
+# sift_package_step = true         # Python package (dir with __init__.py) parent step (default: true)
+# sift_module_step = true          # module (test file) parent step (default: true)
+# sift_class_step = true           # class parent step incl. nested (default: true)
+# sift_parametrize_nesting = true  # parametrize parent steps (default: true)
+# sift_git_metadata = true         # git repo/branch/commit included on the report (default: true)

python/examples/pytest_plugin/tests/pytest_only/__init__.py

@@ -0,0 +1,7 @@
+"""Subpackage of plain pytest tests with no Sift awareness.
+
+Demonstrates that the plugin captures any test's pass/fail with no opt-in
+needed — no ``step`` fixture, no markers, no imports from ``sift_client``.
+The package directory itself becomes a parent step in the report tree (via
+``sift_package_step``, on by default).
+"""