Plug-in benchmarking by Lzanamo · Pull Request #86 · CSE498/Spring2026-CompanyB

Lzanamo · 2026-04-18T22:40:25Z

This PR adds a plug-in benchmarking so teams can benchmark the whole simulation or just a specific functionality of their choice. It is a lightweight framework to measure simulation performance and memory behavior across configurable workloads (for example, changes in agent count and tick count).

Core modules location: core
- benchmarking/core/BenchRunner.hpp
- benchmarking/core/BenchRunner.cpp
- benchmarking/core/BenchmarkTypes.hpp
- benchmarking/core/MetricCollector.hpp
- benchmarking/core/MetricCollector.cpp
- benchmarking/core/ReportGenerator.hpp
- benchmarking/core/ReportGenerator.cpp
Core module tests location: tests
- benchmarking/tests/MetricCollectorTest.cpp
- benchmarking/tests/BenchmarkTypesTest.cpp
- benchmarking/tests/BenchRunnerTest.cpp
- benchmarking/tests/ReportGeneratorTest.cpp
Documentation location:
- benchmarking/Benchmarking.md
- benchmarking/BenchmarkResultsInterpretation.md
- benchmarking/groups/README.md
Add benchmark implementations in benchmarking/groups
Benchmark result files are written to benchmarking/groups/results

…nning benchmarking file

… run

Copilot

Pull request overview

Adds a lightweight benchmarking framework (runner + metrics + reporting) and integrates it into the build system so teams can run configurable benchmarks and emit CSV/JSON reports. The PR also refactors parts of the core simulation API (World/Entity/Agent) to support benchmarking-friendly execution (no built-in display delays) and introduces example benchmark mains and documentation.

Changes:

Introduce benchmarking/core modules (BenchRunner, MetricCollector, ReportGenerator, shared types) plus Catch2 tests under benchmarking/tests.
Add Makefile/CMake integration to build/run benchmark entrypoints from benchmarking/groups/.
Refactor core simulation types (WorldBase, Entity, AgentBase, ItemBase) and update/add maze world implementations.

Reviewed changes

Copilot reviewed 28 out of 28 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
source/core/WorldBase.hpp	Refactors world base API (agent container + run loop) to remove display delays and support new `AgentBase` shape.
source/core/ItemBase.hpp	Updates item construction to pass world reference through `Entity`.
source/core/Entity.hpp	Adds `WorldBase` association to all entities via stored reference.
source/core/AgentBase.hpp	Replaces templated agent base with non-templated action-map driven `AgentBase`.
source/Worlds/StepMazeWorld.hpp	Adds a step-based maze world implementation (Step framework).
source/Worlds/MazeWorld.hpp	Updates maze world to the refactored `WorldBase`/`AgentBase` API.
source/Makefile	Adds `make benchmark` target for building/running benchmark mains via CMake.
source/Group12_main.cpp	Switches Group12 main to use `StepMazeWorld`.
source/CMakeLists.txt	Adds benchmarking include dirs, builds benchmarking core sources, and includes benchmarking tests in test target.
benchmarking/core/BenchmarkTypes.hpp	Defines benchmark params, samples, results, and status enums.
benchmarking/core/MetricCollector.hpp	Declares timing/RSS metric collection API plus thread-local helpers.
benchmarking/core/MetricCollector.cpp	Implements timing + RSS capture and thread-local collector binding.
benchmarking/core/ReportGenerator.hpp	Declares CSV/JSON report formats and writer API.
benchmarking/core/ReportGenerator.cpp	Implements CSV/JSON writing and escaping utilities.
benchmarking/core/BenchRunner.hpp	Declares benchmark registry, execution, aggregation, and report-writing API.
benchmarking/core/BenchRunner.cpp	Implements runner logic: registration, warmup, repetition sampling, aggregation, and report writing.
benchmarking/tests/BenchmarkTypesTest.cpp	Adds unit tests for params validation and result defaults.
benchmarking/tests/MetricCollectorTest.cpp	Adds unit tests for metric collector start/stop/sample behavior.
benchmarking/tests/ReportGeneratorTest.cpp	Adds unit tests for CSV/JSON output content and error statuses.
benchmarking/tests/BenchRunnerTest.cpp	Adds unit tests for registration, execution statuses, and report-writing behavior.
benchmarking/groups/example_1.cpp	Adds a minimal example benchmark main writing CSV via `RunBenchmarkAndWriteReport`.
benchmarking/groups/example_2.cpp	Adds an example benchmark main that writes both CSV and JSON reports.
benchmarking/groups/StepMazeWorldBenchmark.cpp	Adds an example benchmark for `StepMazeWorld` simulation stepping.
benchmarking/groups/README.md	Documents where to place group benchmarks and how to run them.
benchmarking/groups/results/StepMazeWorld.csv	Adds an example benchmark output CSV (checked into repo).
benchmarking/Benchmarking.md	Adds benchmarking module documentation (architecture + usage).
benchmarking/BenchmarkResultsInterpretation.md	Adds a guide for interpreting benchmark metrics and statuses.
Makefile	Adds top-level `make benchmark` wrapper target and argument parsing support.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-04-18T22:46:22Z


-      std::chrono::milliseconds frame_delay{200};
-      Clock::time_point last_frame_time{};
+      // Set the agent to its new postion.


Typo in comment: "postion" -> "position".

Suggested change

// Set the agent to its new postion.

// Set the agent to its new position.

Copilot · 2026-04-18T22:46:22Z

+#include <optional>
+#include <unistd.h>


#include <unistd.h> is included unconditionally, which will break builds on Windows (and any non-POSIX platform). Guard this include with #if defined(__linux__)/#if defined(__unix__) (or move the sysconf usage behind the same guard).

Suggested change

#include <optional>

#include <unistd.h>

#include <optional>

#if defined(__linux__)

#include <unistd.h>

#endif

Copilot · 2026-04-18T22:46:23Z

+            return {};
+          }
+
+          if (world.main_grid[step.loc] == world.wall_id) {
+            return {};


WorldErr / VisitRet suggest movement steps can fail, but the visitor always returns success (return {}) even for invalid/wall moves. Either return std::unexpected(WorldErr{...}) on failure (and handle it in the loop) or simplify the visitor to a void/bool result to avoid dead error-handling code.

Suggested change

return {};

}

if (world.main_grid[step.loc] == world.wall_id) {

return {};

return std::unexpected(WorldErr{

WorldErr::Kind::LOC_INVALID,

"Movement target is outside the maze bounds."

});

}

if (world.main_grid[step.loc] == world.wall_id) {

return std::unexpected(WorldErr{

WorldErr::Kind::LOC_INVALID,

"Movement target is blocked by a wall."

});

Copilot · 2026-04-18T22:46:23Z

@@ -0,0 +1,2 @@
+benchmark_name,agent_count,tick_count,warmup_ticks,repetitions,seed,avg_wall_time_ns,min_wall_time_ns,max_wall_time_ns,stdev_wall_time_ns,avg_memory_usage_kb,min_memory_usage_kb,max_memory_usage_kb,stdev_memory_usage_kb,avg_current_rss_at_stop_kb,min_current_rss_at_stop_kb,max_current_rss_at_stop_kb,stdev_current_rss_at_stop_kb,sample_count
+"step_maze_world",100,1000,1,10,42,22064734,12113531,56127281,16540595,0,0,0,0,4480,4480,4480,0,10


This looks like a generated benchmark output artifact being checked in. If results are meant to be produced by running benchmarks, consider removing this file from version control and documenting the expected output format instead (or ensure results directories are gitignored) to avoid noisy diffs and stale data.

Suggested change

"step_maze_world",100,1000,1,10,42,22064734,12113531,56127281,16540595,0,0,0,0,4480,4480,4480,0,10

Copilot · 2026-04-18T22:46:23Z

+  AGENT_T &AddAgent(std::string agent_name = "None") {
+    auto agent_ptr =
+        std::make_unique<AGENT_T>(agent_set.size(), agent_name, *this);
+    AGENT_T &agent_ref = *agent_ptr;
+    ConfigAgent(*agent_ptr);
+    if (agent_ptr->Initialize() == false) {
+      std::cerr << "Failed to initialize agent '" << agent_name << "'."
+                << std::endl;
    }


WorldBase::AddAgent uses std::cerr/std::endl, but this header does not include <iostream>, which will fail to compile in translation units that include WorldBase.hpp without already including <iostream>. Add the missing include (or move the logging out of the header / behind a dedicated logger).

Copilot · 2026-04-18T22:46:23Z

+      case MOVE_UP:      new_position = cur_position.Up(); break;
+      case MOVE_DOWN:    new_position = cur_position.Down(); break;
+      case MOVE_LEFT:    new_position = cur_position.Left(); break;
+      case MOVE_RIGHT:   new_position = cur_position.Right(); break;


new_position is left uninitialized when action_id does not match any of the handled cases (no default:), and is then used in IsValid/indexing. Add a default branch (e.g., treat as REMAIN_STILL or return failure) to avoid undefined behavior for unexpected action IDs.

Suggested change

case MOVE_RIGHT: new_position = cur_position.Right(); break;

case MOVE_RIGHT: new_position = cur_position.Right(); break;

default: return false;

EvilJuiceBox · 2026-05-02T21:01:47Z

Well done on the Honors project, I am quite happy with the results and am glad you get a nice experience building a benchmarking system for our company project. I am going to leave the PR open to keep your project isolated.

I think we should consider forking the repo if we want an integrated product with your project in.

Lzanamo and others added 7 commits April 10, 2026 19:44

Started development on benchmarking feature

460d2d4

Benchmarking modules implemented with make file updated to support ru…

0a33252

…nning benchmarking file

Merge branch 'main' into feat/HP-benchmarking

4a0f757

Added unit tests for benchmarking modules

eaea2a6

Merge branch 'main' into feat/HP-benchmarking

cac2aec

Added Step Maze World benchmarking and random fixes to make sure test…

118261b

… run

Documentation on benchmarking and results

98f630d

Lzanamo requested a review from EvilJuiceBox April 18, 2026 22:40

Lzanamo self-assigned this Apr 18, 2026

Copilot AI review requested due to automatic review settings April 18, 2026 22:40

Copilot started reviewing on behalf of Lzanamo April 18, 2026 22:40 View session

Copilot AI reviewed Apr 18, 2026

View reviewed changes

Lzanamo added 3 commits April 20, 2026 16:04

Merge branch 'main' into feat/HP-benchmarking

f3d64ab

Added traffic world benchmarking

877dcb5

Added final report and some other final tweaks

b501a95

Lzanamo force-pushed the feat/HP-benchmarking branch from d7575eb to b501a95 Compare May 1, 2026 22:54

Merge branch 'main' into feat/HP-benchmarking

483900b

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Plug-in benchmarking#86

Plug-in benchmarking#86
Lzanamo wants to merge 11 commits into
mainfrom
feat/HP-benchmarking

Lzanamo commented Apr 18, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Apr 18, 2026

Uh oh!

Copilot AI Apr 18, 2026

Uh oh!

Copilot AI Apr 18, 2026

Uh oh!

Copilot AI Apr 18, 2026

Uh oh!

Copilot AI Apr 18, 2026

Uh oh!

Copilot AI Apr 18, 2026

Uh oh!

EvilJuiceBox commented May 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

	// Set the agent to its new postion.
	// Set the agent to its new position.

-            return {};
-          }
-          if (world.main_grid[step.loc] == world.wall_id) {
-            return {};
+            return std::unexpected(WorldErr{
+              WorldErr::Kind::LOC_INVALID,
+              "Movement target is outside the maze bounds."
+            });
+          }
+          if (world.main_grid[step.loc] == world.wall_id) {
+            return std::unexpected(WorldErr{
+              WorldErr::Kind::LOC_INVALID,
+              "Movement target is blocked by a wall."
+            });

		@@ -0,0 +1,2 @@
		benchmark_name,agent_count,tick_count,warmup_ticks,repetitions,seed,avg_wall_time_ns,min_wall_time_ns,max_wall_time_ns,stdev_wall_time_ns,avg_memory_usage_kb,min_memory_usage_kb,max_memory_usage_kb,stdev_memory_usage_kb,avg_current_rss_at_stop_kb,min_current_rss_at_stop_kb,max_current_rss_at_stop_kb,stdev_current_rss_at_stop_kb,sample_count
		"step_maze_world",100,1000,1,10,42,22064734,12113531,56127281,16540595,0,0,0,0,4480,4480,4480,0,10

	case MOVE_RIGHT: new_position = cur_position.Right(); break;
	case MOVE_RIGHT: new_position = cur_position.Right(); break;
	default: return false;

Conversation

Lzanamo commented Apr 18, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Apr 18, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 18, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 18, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 18, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 18, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 18, 2026

Choose a reason for hiding this comment

Uh oh!

EvilJuiceBox commented May 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants