Feat/config engine by viktorbeck98 · Pull Request #21 · ait-detectmate/DetectMateLibrary

viktorbeck98 · 2026-01-07T14:58:44Z

First version for the persistency and the configuration engine. The core of the configuration engine is integrated in the persistency. So far I implemented:

A generic persistency class for event-based purposes that incorporates interchangable data structures that are defined at definition of the persistency object.
Multiple data structures that can be injected into the persistency class based on the use case, such as:
- EventDataFrame: best for developers to get an event-type-separated view of the data (based on pandas)
- ChunkedEventDataFrame: similar as above but is fit for online processing (based on polars). Technically production-ready but since it stores all incoming data it is still RAM heavy. Howerver, in the future, for a storage component we could connect this to a database handling the storage.
- EventVariableTracker: is also a data structure just as the two above but only stores series of persitency changes in the most efficient way (run-length encoding - RLE) and the set of unique values per variable. Persistency changes can thereby be seen as a feature extracted from the data. Detectors like the NewValueDetector, ComboDetector, EventSequenceDetector and possibly more can use this as the underlying structure.
A RLU set implementation for handling outdated values (not yet integrated into persitency class)

Some todos:

Integrate persitency class into the existing detectors (I am on it already)
implement the functionality to remove outdated values from persistency

…for __repr__

# Conflicts: # uv.lock

ipmach · 2026-01-22T12:43:08Z

looking good so far!

But we need to discuss about the notebooks, because I dont know if we want them in the main branch. They present multiple issues:

Code traceability is pretty bad. As the files do not have human readable code. (it is written in Jupyter language).
They are not tested in the main pipeline, so they have a higher risk of not working in the next versions.

viktorbeck98 · 2026-01-26T12:56:08Z

@ipmach regarding the notebooks, that would probably be the cleanest solution, but I personally think they are the easiest way to show how smth works. I will remove them for now. Maybe we can think of it at a later point.

Reorganize trackers.py into a proper package with separate modules for event tracking, multi-tracking, stability classification and conversion. Also removes unused demo notebooks and fixes method signatures for consistency. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

+
+    def test_initialization(self):
+        """Test EventVariableTrackerData initialization."""
+        evt = EventTracker(tracker_type=SingleStabilityTracker)


+
+    def test_add_data(self):
+        """Test adding data."""
+        evt = EventTracker(tracker_type=SingleStabilityTracker)


+
+    def test_get_variables(self):
+        """Test retrieving variable names."""
+        evt = EventTracker(tracker_type=SingleStabilityTracker)


+
+    def test_get_stable_variables(self):
+        """Test retrieving stable variables."""
+        evt = EventTracker(tracker_type=SingleStabilityTracker)


+
+    def test_integration_with_stability_tracker(self):
+        """Test full integration with SingleVariableTracker."""
+        evt = EventTracker(tracker_type=SingleStabilityTracker)


+
+    def test_to_data_with_variable_level(self):
+        """Test to_data method with variable level (default)."""
+        evt = EventTracker(tracker_type=SingleStabilityTracker, feature_type="variable")


+
+    def test_to_data_with_variable_combo_level(self):
+        """Test to_data method with variable_combo level."""
+        evt = EventTracker(tracker_type=SingleStabilityTracker, feature_type="variable_combo")


+
+    def test_to_data_integration_with_add_data(self):
+        """Test that to_data and add_data work together correctly."""
+        evt = EventTracker(tracker_type=SingleStabilityTracker, feature_type="variable")


+
+    def test_conversion_function_assignment(self):
+        """Test that converter is correctly assigned based on feature_type."""
+        evt_var = EventTracker(tracker_type=SingleStabilityTracker, feature_type="variable")


+    def test_conversion_function_assignment(self):
+        """Test that converter is correctly assigned based on feature_type."""
+        evt_var = EventTracker(tracker_type=SingleStabilityTracker, feature_type="variable")
+        evt_combo = EventTracker(tracker_type=SingleStabilityTracker, feature_type="variable_combo")


ipmach · 2026-02-09T16:31:57Z

This should be add it in the docs branch once that one is merge. I suggest remove it for now and add it later.

ipmach

The PR is too big, it has many changes and many are not related with the topic. In the future please focus in smaller PR because it breaks the Agile principle and it makes code harder to maintain because it its harder to review and follow.

I added different comments in the files, the main highlights are:

The folder common is not a good place for persistency. It should be part of utils because it is an ad on, not a common component.
I did not review persistency files as is a big review and not a critical component. Is the first version so probably will be future changes so I am not worry. I am ok leaving it like this for now.
Critical the datasets that are not from AIT should not be part of AIT software. They should be remove from the repository. It is ok adding a script to download them, but should not be part of the repository.
Relevant classes like CoreDetector, ComboDetector, NewValueDetector need minor improvements to increase readability.
Parsers modifications are weird that were added. Some functions inside the matcher are terrible for performance and redability. I suggest removing parsers changes from the PR as well as the changes in the reader.

I did not yet test it locally but I think there are many changes that need to be done before approval.

It is clear that you put a lot of work in this and I do not want to discourage you. The configuration engine will be definitely a great feature in DetectMate. But it needs a little of fine-tunning.

Best,
Andre

ipmach · 2026-02-10T09:04:26Z

correct_single_template is a terrible function in readability and efficiency. The overiteration of templates is really inefficient.

Also it does nothing to do with the PR topic

This function was taken from LogBatcher which improves template quality significantly which is also proven in literatue. This was solely for the demo and will come in a separate PR. I had to change TemplateMatcher since it didn't work anymore after your changes ...

ipmach · 2026-02-10T09:05:06Z

Not relevant for the PR

ipmach · 2026-02-10T09:09:59Z

why do we need this class?

It's an abstraction that makes sure that the repr functions of classes have a unified representation

ipmach · 2026-02-10T09:11:12Z

We do not own this dataset. It should be remove from the repository.

ipmach · 2026-02-10T09:11:26Z

We do not own this dataset. It should be remove from the repository.

ipmach · 2026-02-10T09:11:53Z

We do not own this dataset. It should be remove from the repository.

ipmach · 2026-02-10T09:12:51Z

Do we use polaris? Because if not is a strict dependency because it does not have big support in many hardwares-

As we discussed previously, no we dont need it and it should be removed as a strict dependency. We can add it as an optional dependency in pyproject.toml + lazy import in the class

viktorbeck98 · 2026-02-11T08:17:21Z

@ipmach see this PR more as a draft. It took simply too long with all the changes that were necessary and the demos which made it messy but I just went with it. I wanted to have inital feedback from you. I'll closee the PR and create new smaller ones, improved with your feedback

ipmach · 2026-02-11T08:21:57Z

@viktorbeck98 Yes, I get it. It is hard to separate things when we have to develop the tool and do the demos at the same time. Let me know if you any questions from my feedback! :)

viktorbeck98 added 4 commits January 6, 2026 00:38

small changes to DataBuffer

cd1334a

add an RLE list implementation for the persistency + preview helpers …

7f3a130

…for __repr__

create initial version of persistency class

c33f8eb

add a classifier for stability to the persistency

3d0bfe3

viktorbeck98 requested a review from ipmach January 7, 2026 14:58

viktorbeck98 self-assigned this Jan 7, 2026

viktorbeck98 added enhancement New feature or request New feature labels Jan 7, 2026

This was linked to issues Jan 7, 2026

Implement auto-configuration for detectors #14

Closed

Implement persistency class #15

Closed

viktorbeck98 added 9 commits January 8, 2026 17:13

minor change

8dfdea8

Merge remote-tracking branch 'origin/main' into feat/config_engine

cee6ee1

# Conflicts: # uv.lock

turn data structures into dataclasses

a0399ec

implement config-engine in combodetector

907f165

minor change in dataframe persistency

a31978d

add tests for persistency class

3d5a0cd

add notebook for demonstration purposes

7c9d1e7

minor changes

31838dc

add auto configuration capabilities to ComboDetector

e96a0ca

viktorbeck98 added 3 commits January 22, 2026 16:06

adapt templatematcher and json parser

e1997b2

adapt tests

cedefea

adapt trackers

9d996ed

viktorbeck98 and others added 4 commits January 26, 2026 14:40

add a function to get the number of logs in the log file

11c28f1

adapt tests

27553aa

adapt trackers + adapt ComboDetector

d694edc

github-code-quality Bot found potential problems Jan 29, 2026

View reviewed changes

remove file

c3cc898

viktorbeck98 added 2 commits January 29, 2026 16:18

add data for testing

d6dacfa

add data for testing

62094b6

viktorbeck98 changed the base branch from main to development February 2, 2026 13:53

viktorbeck98 added 4 commits February 2, 2026 15:14

adapt NewValueDetector to use persistency class

9a52591

fix persistency tests

ebfcbd7

remove unnecessary functions

b990b75

adapt config + fix detectors

91c325d

viktorbeck98 force-pushed the feat/config_engine branch from 1c46752 to 91c325d Compare February 6, 2026 15:56

viktorbeck98 and others added 2 commits February 6, 2026 17:03

Merge branch 'development' into feat/config_engine

37f6e98

resolve conflicts from merge

2f196a9

viktorbeck98 linked an issue Feb 6, 2026 that may be closed by this pull request

Add inverse of ConfigMethods.process to serialize configs back to nested YAML #49

Closed

add persistency and configuration documentation

e00541c

ipmach reviewed Feb 9, 2026

View reviewed changes

ipmach requested changes Feb 10, 2026

View reviewed changes

viktorbeck98 closed this Feb 11, 2026

ipmach deleted the feat/config_engine branch March 11, 2026 08:39

ipmach restored the feat/config_engine branch March 11, 2026 08:39

ipmach deleted the feat/config_engine branch March 11, 2026 08:39

Conversation

viktorbeck98 commented Jan 7, 2026

Uh oh!

ipmach commented Jan 22, 2026

Uh oh!

viktorbeck98 commented Jan 26, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ipmach left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

viktorbeck98 commented Feb 11, 2026

Uh oh!

ipmach commented Feb 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants