Skip to content

add persistency#51

Merged
ipmach merged 6 commits intodevelopmentfrom
feat/persistency
Feb 11, 2026
Merged

add persistency#51
ipmach merged 6 commits intodevelopmentfrom
feat/persistency

Conversation

@viktorbeck98
Copy link
Copy Markdown
Collaborator

Summary

  • Introduces a new persistency module providing a flexible, extensible framework for storing and analyzing event data over time
  • Implements multiple storage backends: pandas-based EventDataFrame for simple use cases and Polars-based ChunkedEventDataFrame with built-in retention and lazy materialization for streaming scenarios
  • Adds a stability tracking system that classifies whether event variables are converging to stable values (STATIC, STABLE, UNSTABLE, RANDOM) using segment-based change analysis
  • Includes supporting utilities: RLEList for memory-efficient run-length encoded sequences and preview_helpers for concise collection formatting

Architecture

The module follows a strategy pattern where EventPersistency orchestrates data ingestion and delegates to pluggable EventDataStructure implementations:

  • Dataframes — raw event storage (pandas EventDataFrame, Polars ChunkedEventDataFrame with max-rows retention)
  • Trackers — behavioral analysis via SingleTracker → MultiTracker → EventTracker hierarchy
  • Stability tracking — StabilityClassifier divides a variable's change history into segments and applies increasingly strict thresholds to determine convergence

New dependencies

polars (used by ChunkedEventDataFrame), pandas (used by EventDataFrame)

Test plan

  • test_persistency.py — 463 lines covering EventPersistency, EventDataFrame, ChunkedEventDataFrame, variable filtering, and edge cases
  • test_stability_tracking.py — 554 lines covering RLEList, StabilityClassifier, SingleStabilityTracker, MultiStabilityTracker, and classification scenarios

Solves #15

@viktorbeck98
Copy link
Copy Markdown
Collaborator Author

@ipmach waiting for review

Copy link
Copy Markdown
Contributor

@ipmach ipmach left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!, I move the persistency tests to the test_utils folder because the persistency is inside the utils folder.

@ipmach
Copy link
Copy Markdown
Contributor

ipmach commented Feb 11, 2026

@viktorbeck98 if you are ok with my last commit (moving test to the test_utils folder). Feel free to do the merge!

@ipmach ipmach merged commit 359f98e into development Feb 11, 2026
1 check failed
@ipmach ipmach deleted the feat/persistency branch February 11, 2026 13:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants