Add FPS benchmark metric to influx by tjb7 · Pull Request #214 · luxonis/modelconverter

tjb7 · 2026-04-02T05:27:31Z

Adds influx db pushing of fps results, is BEST EFFORT, depends on oakctl installed on running system.

tjb7 · 2026-04-02T05:33:35Z

NOTE: it's not really ready for review yet.

Basic idea is just to get FPS metrics into influx so we can watch them across all versions of everything, so nothing slips by

tjb7 · 2026-04-08T11:03:20Z

.github/workflows/fps_benchmark_influx.yaml

@@ -0,0 +1,85 @@
+name: HIL FPS Benchmark Influx


@danilo-pejovic not sure if this is needed, would you rather have a seperate workflow or instrument all the tests directly (in rvc4_test.yaml) ... it is my preference to instrument all tests/gather all data, but up to your choice.

Tests that are running on our hil are bom-test.yaml ones, not rvc4_test - those seem internal tests to ml team. I would just add influx thing to bom-tests and not create a new workflow.

We are still creating a new unnecessary workflow. Can we edit this one that is already part of the bom to also pass info to influx?

https://github.com/luxonis/modelconverter/blob/main/.github/workflows/bom-test.yaml

.github/workflows/rvc4_test.yaml

tjb7 · 2026-04-09T11:23:31Z

Also, here is a screenshot of the final data in grafana:

And the direct link: http://10.12.139.254:3000/d/ad4vt7g/fps-model-converter-tests-v3?orgId=1&from=2026-04-07T10:26:51.553Z&to=2026-04-07T17:48:13.745Z&timezone=browser&var-test_name=luxonis%2Fyolov8-nano-pose-estimation:coco-512x288&var-test_name=luxonis%2Ffastsam-s:512x288&var-test_name=luxonis%2Fyolov6-nano:r2-coco-512x288&var-depthai_version=3.3.0&var-depthai_version=3.4.0&var-depthai_version=3.5.0&var-camera_os_version=$__all&var-run_id=$__all&var-camera_mxid=$__all&var-testbed_name=$__all&var-min_fps_threshold=

danilo-pejovic · 2026-04-10T08:18:42Z

tests/test_benchmark/run_hil_tests.sh

+# Cache device metadata once for the whole run using oakctl. If oakctl is not
+# available, keep the benchmark runnable and record explicit placeholder values
+# for the missing oakctl-derived metadata.
+oakctl_output=$(oakctl list --format json 2>/dev/null || printf '')


Oakctl is not part of these tests. Why are we adding more moving parts that can fail?

Hil framework has built in functions to account get all necessary info from camera:

danilo@danilo:~/Downloads$ camera -t slo316-4pro -n test oak4_pro info -v +----------+----------+------------+------------+------------+------------+------------+--------------+----------+----------------+-------------------------------------------------+----------+ | name | model | platform | revision | mxid | username | password | hostname | adb | state | os_version | adb_id | +==========+==========+============+============+============+============+============+==============+==========+================+=================================================+==========+ | oak4_pro | oak4_pro | rvc4 | 9 | 1583798758 | root | oelinux123 | 10.12.143.89 | 5e66d9e6 | setup-complete | 1.30.0+955791f9014cfe160419deae2f64980f180f4d9c | 5e66d9e6 | +----------+----------+------------+------------+------------+------------+------------+--------------+----------+----------------+-------------------------------------------------+----------+ danilo@danilo:~/Downloads$ camera -t slo316-4pro -n test oak4_pro info -v -j [{"name": "oak4_pro", "model": "oak4_pro", "platform": "rvc4", "revision": 9, "mxid": "1583798758", "username": "root", "password": "oelinux123", "hostname": "10.12.143.89", "adb": "5e66d9e6", "state": "setup-complete", "os_version": "1.30.0+955791f9014cfe160419deae2f64980f180f4d9c", "adb_id": "5e66d9e6"}]

Alternatively - camera object in python can give all the necessary info in even more readable/maintainable way. That is probably the way we should supply that info.

I will sub-in the json version of the above:

(.hil) hil@slo130-fl:~$ camera -t slo316-4pro -n test oak4_pro info -j
[{"name": "oak4_pro", "model": "oak4_pro", "platform": "rvc4", "revision": 9, "mxid": "1583798758", "hostname": "10.12.143.89", "os_version": "1.27.1+9950fe7353d76fec53a99019be9e1d6130e8f69a", "adb_id": "5e66d9e6"}]

Q: how to determine the camera used if there are multiple cameras?

For now I will just use: camera -t ${HIL_TESTBED} -n test all info -j and take the first:

(.hil) hil@slo238-4s:~$ camera -t ${HIL_TESTBED} -n test all info -j use_ssh 2026-04-13 07:21:36,832 [INFO] ssh command failed with the following output: 2026-04-13 07:21:36,832 [INFO] 2026-04-13 07:21:36,832 [INFO] Error message (stderr): 2026-04-13 07:21:36,832 [INFO] 2026-04-13 07:21:36,832 [INFO] Exit code: 5 [{"name": "oak4_s", "model": "oak4_s", "platform": "rvc4", "revision": 9, "mxid": "2671960531", "hostname": "10.12.140.235", "os_version": "1.27.1+9950fe7353d76fec53a99019be9e1d6130e8f69a", "adb_id": "9f42e1d3"}]

As stated in dms via example provided and in first comment here I would prefer camera metadata to be gathered inside the script:

Alternatively - camera object in python can give all the necessary info in even more readable/maintainable way. That is probably the way we should supply that info.

It would be more readable, consistent and maintainable.

tests/test_benchmark/run_hil_tests.sh

danilo-pejovic · 2026-04-10T08:49:05Z

tests/test_benchmark/test_benchmark_regression.py

+    return f'"{escaped}"'
+
+
+def _write_fps_result_to_influx(


Can this be implemented in hil_framework? We can probably hardcode influx variables there and then create simple function like camera/testbed.log_to_influx(arguments) and it will be pretty similar to this implementation and how sanity checks work for hil already. That way it is more maintainable - if we for example change where we host out database, it will be one change instead of changing every repo.

agreed, but let's do it in a follow up ticket, just so I can get this work finished and immediately useful.

I would prefer to do things properly from the start if we know it is something we are going to be building on - it shouldnt take long.

tests/test_benchmark/test_benchmark_regression.py

danilo-pejovic · 2026-04-10T09:12:00Z

.github/workflows/fps_benchmark_influx.yaml

@@ -0,0 +1,85 @@
+name: HIL FPS Benchmark Influx


Tests that are running on our hil are bom-test.yaml ones, not rvc4_test - those seem internal tests to ml team. I would just add influx thing to bom-tests and not create a new workflow.

danilo-pejovic · 2026-04-10T09:12:33Z

tests/test_benchmark/conftest.py

        default="rvc4",
        help="Target platform to benchmark (default: rvc4).",
    )
+    parser.addoption(


Comments above/below adds more info.

Probably should be implemented in more maintainable way to avoid piping info we already have multiple times.

So if I understand correctly, I should just remove this here, and fail hard if the env is not properly configured. I will make the change, please comment again if you mean something else here.

we are passing camera mxid, os versions,... as arguments to the script when there is no need to do that and that info is readily available inside the script via hil framework.

tests/test_benchmark/test_benchmark_regression.py

tests/test_benchmark/run_hil_tests.sh

danilo-pejovic · 2026-04-10T09:33:13Z

Also, here is a screenshot of the final data in grafana:

And the direct link: http://10.12.139.254:3000/d/ad4vt7g/fps-model-converter-tests-v3?orgId=1&from=2026-04-07T10:26:51.553Z&to=2026-04-07T17:48:13.745Z&timezone=browser&var-test_name=luxonis%2Fyolov8-nano-pose-estimation:coco-512x288&var-test_name=luxonis%2Ffastsam-s:512x288&var-test_name=luxonis%2Fyolov6-nano:r2-coco-512x288&var-depthai_version=3.3.0&var-depthai_version=3.4.0&var-depthai_version=3.5.0&var-camera_os_version=$__all&var-run_id=$__all&var-camera_mxid=$__all&var-testbed_name=$__all&var-min_fps_threshold=

In the graph max and min values for models dont seem to be consistent? IIRC they are just some expected value with buffer on both sides so it should be always the same for the same model. Are we maybe mixing up models in tables?

Also as long as those min and max values are set constants we probably dont need to track them for every test. It just adds clutter

tjb7 · 2026-04-13T13:01:17Z

Changes made, ready for review again.

tjb7 requested a review from a team as a code owner April 2, 2026 05:27

tjb7 requested review from conorsim, klemen1999, kozlov721 and tersekmatija and removed request for a team April 2, 2026 05:27

tjb7 marked this pull request as draft April 2, 2026 05:27

tjb7 requested a review from danilo-pejovic April 2, 2026 05:33

tjb7 force-pushed the tjb_fps_graphs branch from e5bc2d2 to 9ae4b83 Compare April 7, 2026 07:40

tjb added 11 commits April 8, 2026 13:00

Add manual FPS benchmark Influx pipeline

cfc995e

Use model-based HIL selection for FPS Influx workflow

ce82fcd

Temporarily repurpose rvc4 workflow for FPS Influx dispatch

f14ff42

Map Influx GitHub variables into FPS workflow env

d0bf4d7

Read Influx workflow settings from GitHub secrets

f65c94d

Add fake Influx CSV fixture for FPS metrics

5fad3b6

Fix annotated CSV format for fake Influx fixture

c95f978

Add fake Influx line protocol fixture

d9308bd

Use correct Influx bucket name in FPS workflows

5e96309

Normalize FPS Influx tags and testbed fallback

e7e23cd

Temporary debugging logging

45768b9

tjb7 force-pushed the tjb_fps_graphs branch from 69f7149 to 45768b9 Compare April 8, 2026 11:00

tjb7 commented Apr 8, 2026

View reviewed changes

Cleanup Commit

22fd18c

tjb7 marked this pull request as ready for review April 8, 2026 11:10

kozlov721 reviewed Apr 9, 2026

View reviewed changes

.github/workflows/rvc4_test.yaml Show resolved Hide resolved

danilo-pejovic reviewed Apr 10, 2026

View reviewed changes

tests/test_benchmark/run_hil_tests.sh Show resolved Hide resolved

tjb added 7 commits April 13, 2026 07:24

Use HIL camera metadata in benchmark tests

1579062

Drop obsolete camera agent metadata

be2ccf9

Use HIL_TESTBED directly in benchmark script

63fdb79

Fail benchmark run on missing camera metadata

a962d3b

Use env metadata in benchmark tests

60a4f54

Select rvc4 camera metadata

ba2e946

Restore RVC4 workflow from main

c24c3ec

Conversation

tjb7 commented Apr 2, 2026

Uh oh!

tjb7 commented Apr 2, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

tjb7 commented Apr 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

danilo-pejovic Apr 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

danilo-pejovic Apr 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

danilo-pejovic commented Apr 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tjb7 commented Apr 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

tjb7 commented Apr 9, 2026 •

edited

Loading

danilo-pejovic Apr 10, 2026 •

edited

Loading

danilo-pejovic Apr 13, 2026 •

edited

Loading

danilo-pejovic commented Apr 10, 2026 •

edited

Loading