Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions python/docs/examples/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ This section contains interactive Jupyter notebook examples demonstrating how to
- **[Basic Usage](basic.ipynb)** - Introduction to the Sift Python client, covering basic operations and API usage
- **[Data Ingestion](ingestion.ipynb)** - Learn how to ingest telemetry data into Sift using various methods
- **[Pytest Plugin](pytest_plugin.md)** - Turn a pytest run into a Sift TestReport with measurements, nested steps, and pass/fail outcomes
- **[Pytest Plugin Quickstart](pytest_plugin_quickstart.md)** - Guided tour of the runnable demo project under `python/examples/pytest_plugin/`

## Running Examples Locally

Expand Down
120 changes: 71 additions & 49 deletions python/docs/examples/pytest_plugin.md
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,8 @@ def sift_client() -> SiftClient:
|---|---|---|---|
| `report_context` | fixture (autouse) | session | The `ReportContext` backing the run's `TestReport`. Use it to attach metadata or open ad-hoc steps. |
| `step` | fixture (autouse) | function | A `NewStep` created for the current test function. Exposes `measure*`, `substep`, `report_outcome`, and `current_step`. |
| `module_substep` | fixture (autouse) | module | One step per test file with each function nested as a substep. |
| `_hierarchy_parents` | internal fixture (autouse) | function | Opens a parent step for each `pytest.Package`, `pytest.Module`, and `pytest.Class` ancestor of the current test. Each layer is gated independently — see [ini options](#ini-options). |
| `_parametrize_parents` | internal fixture (autouse) | function | Opens a parent step for each `@pytest.mark.parametrize` axis (and fixture parametrization), nested inside the hierarchy parents. |
| `client_has_connection` | fixture | session | Calls `sift_client.ping.ping()`; consulted by `report_context` at session start in online mode (the default). Override to skip the ping or use a different reachability signal. |

### CLI options
Expand Down Expand Up @@ -118,6 +119,10 @@ CLI flags, when passed, override the ini values.
| `sift_offline` | bool (default `false`) | `--sift-offline` |
| `sift_disabled` | bool (default `false`) | `--sift-disabled` (also honors `SIFT_DISABLED` env var) |
| `sift_autouse` | bool (default `true`) | _(no CLI flag; controls the marker gate below)_ |
| `sift_package_step` | bool (default `true`) | _(ini-only)_ — open a parent step for each Python package (directory with `__init__.py`) in the test path. |
| `sift_module_step` | bool (default `true`) | _(ini-only)_ — open a parent step for each test module (file). |
| `sift_class_step` | bool (default `true`) | _(ini-only)_ — open a parent step for each test class, including nested classes. |
| `sift_parametrize_nesting` | bool (default `true`) | _(ini-only)_ — cluster parametrized tests under shared parents (`test_x → axis=value`) instead of flat leaves (`test_x[value]`). |

The default `sift_client` fixture reads its two URIs from environment first
and falls back to ini keys when the env vars are unset. `SIFT_API_KEY` is
Expand Down Expand Up @@ -302,8 +307,8 @@ outcomes into `TestStatus`:
| Manual `step.current_step.update({"status": ...})` | Whatever you set; the step exit handler honors a manually-resolved status |

A failure or error at any depth propagates upward: the parent substep, the
function step, the module step (if `module_substep` is active), and the
session report all get marked failed.
function step, the class/module/package steps above it, and the session
report all get marked failed.

## Nested steps

Expand Down Expand Up @@ -339,12 +344,14 @@ Each step gets a hierarchical `step_path` (`1`, `1.1`, `1.1.2`, `2`, …)
assigned by `ReportContext`. Sibling substeps within the same parent
auto-increment; opening a new top-level step starts a new branch.

### One step per file
### Mirroring the test layout

`module_substep` is autouse and module-scoped. When it's active (it's pulled
in by the star-import in `conftest.py`), each file becomes a parent step and
every function in it nests one level down. Its name is the test file's
basename and its description is the module's docstring (if any).
The plugin opens a parent step for each Python package (`__init__.py`
directory), test file, and test class above every test, plus a parent step
for each `@pytest.mark.parametrize` axis. Every layer is on by default and
individually opt-out via ini flags (`sift_package_step`, `sift_module_step`,
`sift_class_step`, `sift_parametrize_nesting`). Class/module/package
docstrings become the matching step's description.

### Linking a Run to the report

Expand Down Expand Up @@ -384,50 +391,43 @@ TestReport
└── test_temperature
```

### One step per file with `module_substep`
### Modules nested under a package

`module_substep` is autouse and module-scoped. Every file becomes a parent
step and every function in it nests one level down.
Two test files under the same Python package (directory with `__init__.py`)
share that package step as their parent.

```python title="test_battery.py"
```python title="suites/__init__.py"
```

```python title="suites/test_battery.py"
def test_voltage(step): ...
def test_current(step): ...
```

```python title="test_thermal.py"
```python title="suites/test_thermal.py"
def test_idle_temp(step): ...
def test_load_temp(step): ...
```

```text title="Sift report"
TestReport
├── test_battery.py
│ ├── test_voltage
│ └── test_current
└── test_thermal.py
├── test_idle_temp
└── test_load_temp
└── suites
├── test_battery.py
│ ├── test_voltage
│ └── test_current
└── test_thermal.py
├── test_idle_temp
└── test_load_temp
```

### Test classes
### Test classes (and nested classes)

Pytest classes (`class TestFoo: ...`) do not create a parent step on their
own. The plugin keys off the test node's `name`, which is just the method
name. To group a class's methods under a class-level step, add a class-scoped
fixture that opens a step with `report_context.new_step(...)`:
`class TestFoo:` and `class TestOuter: class TestInner:` produce class and
nested class steps automatically — no manual fixture needed.

```python title="test_charging.py"
import pytest


class TestCharging:
@pytest.fixture(scope="class", autouse=True)
def class_step(self, report_context):
with report_context.new_step(
name="TestCharging",
description="Charging subsystem",
) as parent:
yield parent
"""Charging subsystem."""

def test_starts_at_zero(self, step): ...
def test_reaches_full(self, step): ...
Expand All @@ -436,23 +436,20 @@ class TestCharging:

```text title="Sift report"
TestReport
└── TestCharging
├── test_starts_at_zero
├── test_reaches_full
└── test_thermal_throttle
└── test_charging.py
└── TestCharging
├── test_starts_at_zero
├── test_reaches_full
└── test_thermal_throttle
```

!!! note "Combining with `module_substep`"
`module_substep` and a class-scoped step both open at module/class scope,
so they each grab the next sibling slot under the report and the inner
one nests under the outer. If you want both layers (file → class →
method), make the class step itself open via the active outer step
rather than the report root.
The class's docstring becomes the step description.

### Parametrized tests

Each parametrize case is a distinct pytest node, so each gets its own step.
The step name includes the parameter id pytest generates.
Parametrized tests cluster under a parent step named after the test function,
with one inner parent per parametrize axis (outer-to-inner in
decorator-on-page order). Stacked parametrize produces nested step levels.

```python
@pytest.mark.parametrize("voltage", [3.3, 5.0, 12.0])
Expand All @@ -462,11 +459,36 @@ def test_rail(step, voltage):

```text title="Sift report"
TestReport
├── test_rail[3.3]
├── test_rail[5.0]
└── test_rail[12.0]
└── test_module.py
└── test_rail
├── voltage=3.3
├── voltage=5.0
└── voltage=12.0
```

Stacked parametrize:

```python
@pytest.mark.parametrize("voltage", ["high", "low"])
@pytest.mark.parametrize("component", ["motor", "valve"])
def test_iso(step, voltage, component): ...
```

```text title="Sift report"
TestReport
└── test_module.py
└── test_iso
├── voltage='high'
│ ├── component='motor'
│ └── component='valve'
└── voltage='low'
├── component='motor'
└── component='valve'
```

Set `sift_parametrize_nesting = false` in `pytest.ini` to fall back to flat
leaf names (`test_rail[3.3]`).

### Helper functions

Helpers called from a test do not auto-create a step. The plugin only sees
Expand Down
177 changes: 177 additions & 0 deletions python/docs/examples/pytest_plugin_quickstart.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,177 @@
# Pytest Plugin Quickstart

A walkthrough of the runnable demo at
[`python/examples/pytest_plugin/`](https://github.com/sift-stack/sift/tree/main/python/examples/pytest_plugin).
The demo is a self-contained pytest project that exercises every layer of the
plugin's step tree: packages, modules, classes (including nested), parametrize
axes, manual substeps, and gate markers. It also includes a tests directory
that uses no Sift APIs at all, to show how the autouse fixtures capture plain
pytest tests for free.

For a conceptual reference (fixtures, ini flags, status semantics), see
[Pytest Plugin](pytest_plugin.md).

## Project layout

```
examples/pytest_plugin/
├── conftest.py # registers the plugin
├── pytest.ini # available ini knobs (all commented at defaults)
├── .env.example # credential template
└── tests/
├── pytest_only/ # subpackage step
│ ├── __init__.py
│ └── test_pytest_only_demo.py # plain pytest, no Sift APIs
└── with_sift/ # subpackage step
├── __init__.py
└── test_with_sift_demo.py # measurements, substeps, classes, parametrize, gates
```

Every Python package (directory with `__init__.py`), test file, and test class
above each test becomes its own parent step in the report tree.

## `conftest.py`

A single `pytest_plugins` declaration loads the plugin; `load_dotenv()` is
optional and just lets the default `sift_client` fixture pick up
`SIFT_API_KEY` / `SIFT_GRPC_URI` / `SIFT_REST_URI` from a local `.env`.

```python title="conftest.py"
--8<-- "examples/pytest_plugin/conftest.py"
```

## `pytest.ini`

Every knob is commented at its default value. Uncomment any line to opt out of
a layer of the step tree.

```ini title="pytest.ini"
--8<-- "examples/pytest_plugin/pytest.ini"
```

## `.env.example`

```bash title=".env.example"
--8<-- "examples/pytest_plugin/.env.example"
```

## The pytest_only module

Plain pytest tests with no `sift_client` imports, no `step` fixture, no
markers. Each one still becomes a leaf step in the report tree. The plugin's
autouse fixtures capture pass/fail automatically.

```python title="tests/pytest_only/test_pytest_only_demo.py"
--8<-- "examples/pytest_plugin/tests/pytest_only/test_pytest_only_demo.py"
```

## The with_sift module

Exercises the plugin's full surface: numeric / string / bool bounds, nested
`step.substep`, `@pytest.mark.sift_exclude`, class steps with docstring
descriptions, nested classes, stacked `@pytest.mark.parametrize`, and
`step.report_outcome`.

```python title="tests/with_sift/test_with_sift_demo.py"
--8<-- "examples/pytest_plugin/tests/with_sift/test_with_sift_demo.py"
```

## Run it

### Without Sift credentials

```bash
cd python/examples/pytest_plugin
pytest --sift-disabled -v
```

`--sift-disabled` makes the plugin a no-op transport: `step.measure(...)`
still evaluates bounds and returns a real pass/fail boolean, but nothing
contacts Sift and no log file is written. Useful for previewing the report
tree or unit-testing measurement logic.

### Against a real Sift org

```bash
cp .env.example .env
# Fill in SIFT_API_KEY / SIFT_GRPC_URI / SIFT_REST_URI
pytest -v
```

A `TestReport` shows up in Sift once the session finishes.

### Offline (record now, replay later)

```bash
pytest --sift-offline --sift-log-file=/tmp/sift-demo.jsonl -v
# Later, from anywhere with credentials:
import-test-result-log /tmp/sift-demo.jsonl
```

## Expected report tree

With the plugin's defaults (every layer enabled), the demo produces:

```
TestReport (FAILED, since failures propagate up from leaves)
├── pytest_only ← package step (FAILED)
│ └── test_pytest_only_demo.py ← module step (FAILED)
│ ├── test_passes PASSED
│ ├── test_uses_a_pytest_fixture PASSED
│ ├── test_assertion_failure_marks_step_failed FAILED
│ ├── test_skipped SKIPPED
│ ├── test_unexpected_exception_marks_step_errored ERROR
│ ├── test_parametrize_without_step
│ │ ├── value='v1' PASSED
│ │ └── value='v2' PASSED
│ └── TestPytestClass
│ └── test_method PASSED
└── with_sift ← package step (FAILED)
└── test_with_sift_demo.py ← module step (FAILED)
├── test_measurements PASSED
├── test_substeps PASSED
│ ├── phase_1
│ └── phase_2
│ └── phase_2a
│ (test_excluded: @sift_exclude, runs in pytest, NOT in tree)
├── test_measure_series PASSED
├── test_failed_measurement_marks_sift_step_failed FAILED (pytest PASSED)
├── test_assert_measurements_passed_at_end FAILED (pytest FAILED)
├── test_report_level_metadata PASSED
└── TestClassStep
├── test_parametrize
│ ├── axis_a='a1'
│ │ ├── axis_b='b1' PASSED
│ │ └── axis_b='b2' PASSED
│ └── axis_a='a2'
│ ├── axis_b='b1' PASSED
│ └── axis_b='b2' PASSED
└── TestNested
└── test_report_outcome
└── check PASSED
```

The `pytest_only` module deliberately includes one failing, one skipped, and
one erroring test so the demo shows every `TestStatus` mapping (`FAILED` for
assertions, `SKIPPED` for `pytest.skip`, `ERROR` for any other exception).
The `with_sift` module shows two patterns for handling measurement results:
`test_failed_measurement_marks_sift_step_failed` lets the test keep passing
in pytest while the Sift step is `FAILED` (useful when measurements are
diagnostic data you want to collect regardless of outcome); and
`test_assert_measurements_passed_at_end` takes every measurement first and
then asserts `step.measurements_passed` once at the end, so every
measurement still lands in the report even when one fails. The end-of-test
assertion is the recommended pattern: asserting on an individual
`step.measure(...)` call short-circuits on the first failure and skips
every measurement that follows. Expected
pytest output is `16 passed, 3 failed, 1 skipped`.

Flip any of the `sift_*_step` / `sift_parametrize_nesting` flags in
`pytest.ini` to `false` to collapse a layer.

## Next steps

- [Pytest Plugin](pytest_plugin.md): conceptual reference covering fixtures,
ini flags, status semantics, and layout-mapping examples.
- The demo's [README](https://github.com/sift-stack/sift/blob/main/python/examples/pytest_plugin/README.md)
on GitHub mirrors this page and is the canonical source.
3 changes: 3 additions & 0 deletions python/examples/pytest_plugin/.env.example
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
SIFT_API_KEY=your-api-key
SIFT_GRPC_URI=your-org.grpc.example.com
SIFT_REST_URI=https://your-org.rest.example.com
Loading
Loading