Skip to content

feat: add local structured substitute benchmark strategy#184

Draft
binaryaaron wants to merge 10 commits into
mainfrom
binaryaaron/perf-structured-substitute
Draft

feat: add local structured substitute benchmark strategy#184
binaryaaron wants to merge 10 commits into
mainfrom
binaryaaron/perf-structured-substitute

Conversation

@binaryaaron

@binaryaaron binaryaaron commented Jun 8, 2026

Copy link
Copy Markdown
Collaborator

Summary

Adds a benchmark-only local structured substitute path that builds deterministic replacement maps for supported structured labels. This lets the benchmark suite compare DataDesigner substitute output against a local, non-LLM substitute baseline without changing Anonymizer defaults.

Stack

What changed

  • Add structured_substitute.py for deterministic replacements on supported structured labels.
  • Add benchmark strategy wiring and replay tooling for structured substitute comparisons.
  • Add focused tests for replacement-map behavior, replay output, benchmark wiring, and analysis fields.
  • Document the strategy as a benchmark probe, not a public product default.

Validation

  • uv run --frozen ruff format ...
  • uv run --frozen ruff check ...
  • uv run pytest tests/engine/test_structured_substitute.py tests/test_measurement.py tests/tools/test_measurement_tools.py tests/tools/test_replacement_strategies.py tests/tools/test_replay_replacement_strategies.py tests/tools/test_benchmark_output_analysis.py tests/tools/test_compare_strategy_pairs.py tests/tools/test_screen_strategy_comparisons.py -q
  • git diff --check

Focused suite result: 150 passed, with existing DataDesigner provider deprecation warnings.

Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
Base automatically changed from binaryaaron/perf-epic to main June 12, 2026 20:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant