Skip to content

reengineering: benchmarking system#73

Draft
iraedeus wants to merge 15 commits intoiraedeus/refactor-metricsfrom
iraedeus/benchmark-refactoring
Draft

reengineering: benchmarking system#73
iraedeus wants to merge 15 commits intoiraedeus/refactor-metricsfrom
iraedeus/benchmark-refactoring

Conversation

@iraedeus
Copy link
Copy Markdown
Contributor

@iraedeus iraedeus commented Apr 7, 2026

This pull request introduces a new benchmark_executor.py module to the pysatl_cpd/benchmark package, providing a robust framework for executing and caching benchmarks for online change-point detection (CPD) algorithms. The module enables running multiple algorithms across datasets and thresholds, with optional disk-based caching to avoid redundant computations.

Key additions and features:

Benchmarking orchestration and caching:

  • Introduces the BenchmarkExecutor generic class, which coordinates the execution of benchmarks for multiple algorithms, datasets, and thresholds, supporting optional result caching via disk dumping.
  • Implements a CSV-based registry and Pickle-based trace serialization to cache and retrieve benchmark results, minimizing redundant computations on subsequent runs.

Benchmark metadata management:

  • Adds the BenchmarkRecord dataclass to uniquely identify each benchmark run and store metadata, including algorithm, configuration, dataset, threshold, and cache file path.

Integration with existing components:

  • Integrates with OnlineAlgorithm, DataProvider, OnlineCpdSolver, and OnlineDetectionTrace from the core modules, ensuring compatibility and reusability within the CPD framework.

@iraedeus iraedeus requested review from Desiment and vkutuev April 7, 2026 02:22
@codacy-production
Copy link
Copy Markdown

codacy-production bot commented Apr 7, 2026

Not up to standards ⛔

TIP This summary will be updated as you push new changes. Give us feedback

@iraedeus iraedeus force-pushed the iraedeus/benchmark-refactoring branch 2 times, most recently from bb999cc to 85e1cdd Compare April 8, 2026 00:56
@iraedeus iraedeus force-pushed the refactor/architecture-rework branch from 32f2c21 to ab266d8 Compare April 8, 2026 00:59
@iraedeus iraedeus force-pushed the iraedeus/benchmark-refactoring branch 3 times, most recently from 662f1c1 to 87ac28d Compare April 10, 2026 19:53
@iraedeus iraedeus changed the base branch from refactor/architecture-rework to dev April 10, 2026 19:54
@iraedeus iraedeus changed the base branch from dev to iraedeus/analysis-metrics April 10, 2026 19:54
@iraedeus iraedeus force-pushed the iraedeus/benchmark-refactoring branch 4 times, most recently from 55d632f to 2979e34 Compare April 11, 2026 09:13
@iraedeus iraedeus self-assigned this Apr 11, 2026
@iraedeus iraedeus changed the base branch from iraedeus/analysis-metrics to iraedeus/multiple-run-metrics April 11, 2026 09:13
@iraedeus iraedeus force-pushed the iraedeus/benchmark-refactoring branch from 2979e34 to b9ce3ab Compare April 12, 2026 22:27
@iraedeus iraedeus force-pushed the iraedeus/benchmark-refactoring branch from b9ce3ab to 39c6903 Compare April 13, 2026 15:31
@iraedeus iraedeus changed the base branch from iraedeus/multiple-run-metrics to iraedeus/refactor-metrics April 14, 2026 00:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant