Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
120 changes: 120 additions & 0 deletions .github/AGENTS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,120 @@
# robotframework-dashboard — Agent Guide

This file gives AI agents and contributors the context needed to work effectively in this codebase.

---

## Project Purpose

`robotframework-dashboard` is a Python CLI tool that reads Robot Framework `output.xml` execution results, stores them in a SQLite database, and generates a fully self-contained HTML dashboard with interactive charts, tables, and filters. No web server is required to view the output — a single `.html` file contains all data, JS, and CSS.

---

## Core Pipeline: Python CLI → HTML Template → JavaScript

The entire system is this three-stage pipeline:

```
1. PYTHON CLI
output.xml files
└─► OutputProcessor (robot.api ResultVisitor)
└─► SQLite database (runs / suites / tests / keywords tables)

2. HTML TEMPLATE
database.get_data()
└─► DashboardGenerator
├─► DependencyProcessor: merges all JS modules (topological sort) → inline <script>
├─► DependencyProcessor: merges all CSS files → inline <style>
├─► CDN or offline dependency tags
├─► Data encoded as: JSON → zlib compress → base64 → string literal in HTML
└─► templates/dashboard.html (string placeholder replacement) → robot_dashboard.html

3. JAVASCRIPT (runs in the browser)
js/variables/data.js decodes the embedded base64 data back to JS arrays
└─► Chart.js charts, DataTables, filters, layout — all from local data, zero server calls
```

The output is a **single `.html` file** that is entirely self-contained. All Robot Framework data is embedded as compressed strings; all JS and CSS is inlined.

---

## Entry Points

| File | Role |
|---|---|
| `robotframework_dashboard/main.py` | CLI entry point (`robotdashboard` command) |
| `robotframework_dashboard/robotdashboard.py` | `RobotDashboard` class — orchestrates all 5 pipeline steps |
| `robotframework_dashboard/arguments.py` | `ArgumentParser` wrapping `argparse` |
| `robotframework_dashboard/processors.py` | `OutputProcessor` + 4 `ResultVisitor` subclasses |
| `robotframework_dashboard/database.py` | Built-in SQLite implementation |
| `robotframework_dashboard/abstractdb.py` | `AbstractDatabaseProcessor` ABC (custom DB backends) |
| `robotframework_dashboard/queries.py` | All SQL strings as module-level constants |
| `robotframework_dashboard/dashboard.py` | `DashboardGenerator` — template rendering |
| `robotframework_dashboard/dependencies.py` | `DependencyProcessor` — JS/CSS inlining and CDN switching |
| `robotframework_dashboard/server.py` | Optional FastAPI server (`--server` flag) |

---

## JavaScript and CSS

All frontend source lives under `robotframework_dashboard/js/` and `robotframework_dashboard/css/`. **There is no Node.js bundler (no webpack, Vite, or Rollup) for the dashboard.** Bundling is done in Python by `DependencyProcessor` at HTML generation time.

Key JS directories:

| Path | Contents |
|---|---|
| `js/variables/` | Global state, data decoding, settings, graph registry |
| `js/graph_creation/` | Chart.js setup per tab (overview, run, suite, test, keyword, compare, tables) |
| `js/graph_data/` | Data transformation modules that feed Chart.js |
| `js/main.js` | Startup entry — imports and calls all setup functions |
| `js/admin_page/` | Separate JS bundle for the server's `/admin` page only |

See `.github/skills/js-bundling.md` for details on how JS modules are resolved, ordered, and embedded.

---

## HTML Templates

Templates live in `robotframework_dashboard/templates/`. They use simple string placeholder tokens (not Jinja2):

- `templates/dashboard.html` → generates `robot_dashboard.html`
- `templates/admin.html` → generates the server's `/admin` page

Key placeholders: `<!-- placeholder_javascript -->`, `<!-- placeholder_css -->`, `<!-- placeholder_dependencies -->`, `"placeholder_runs"`, `"placeholder_suites"`, `"placeholder_tests"`, `"placeholder_keywords"`.

---

## Database

- Built-in: SQLite via `database.py`. Tables: `runs`, `suites`, `tests`, `keywords`.
- Custom backends: implement `AbstractDatabaseProcessor` from `abstractdb.py`, point to it with `--databaseclass`.
- Run identity: `run_start` timestamp. Duplicate runs are silently skipped.
- Schema migrations are handled inline at DB open time via `ALTER TABLE ADD COLUMN`.

---

## Skills

The `.github/skills/` directory contains domain-specific knowledge files:

| Skill file | When to use |
|---|---|
| `project-architecture.md` | Understanding how components connect and navigating the codebase |
| `dashboard.md` | Dashboard pages, Chart.js graphs, chart types, graph data/creation modules |
| `js-bundling.md` | How JS/CSS is bundled and embedded into the HTML (no Node.js bundler) |
| `conventions-and-gotchas.md` | Edge cases, run identity, offline mode, custom DBs, server auth model |
| `coding-style.md` | Python/JS/CSS style conventions |
| `workflows.md` | CLI usage, running tests, server mode, docs site |
| `testing.md` | Test suite structure, pabot parallelism, how to add tests |
| `server-api.md` | All REST endpoints, authentication, log linking, auto-update behavior |
| `filtering-and-settings.md` | Filter pipeline, settings object, localStorage persistence, layout/GridStack system |

---

## Key Rules for AI Agents

- **Never break the placeholder token names** in templates. Replacement is positional string substitution.
- **When adding a new JS module**, import it from an existing module so `DependencyProcessor` can discover it via the dependency graph. The topological sort handles ordering automatically.
- **Data always flows**: parse → DB → HTML. Do not bypass the pipeline.
- **`package.json` is for the VitePress docs site only.** It has nothing to do with bundling dashboard JS.
- **Offline mode** (`--offlinedependencies`) reads from `robotframework_dashboard/dependencies/`. Keep local copies in sync when upgrading library versions.
4 changes: 3 additions & 1 deletion .github/skills/conventions-and-gotchas.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,11 @@ description: Use when modifying core logic, adding features, or debugging issues
# Project Conventions and Gotchas

- Run identity is `run_start` from output.xml; duplicates are rejected. `run_alias` defaults to file name and may be auto-adjusted to avoid collisions.
- If you add log support, log names must mirror output names (output-XYZ.xml -> log-XYZ.html) for `uselogs` and server log linking.
- If you add log support, log names must mirror output names (output-XYZ.xml -> log-XYZ.html) for `uselogs` and server log linking. The server's `/add-log` endpoint enforces this naming convention at runtime.
- `--projectversion` and `version_` tags are mutually exclusive; version tags are parsed from output tags in `RobotDashboard._process_single_output`.
- Custom DB backends are supported via `--databaseclass`; the module must expose a `DatabaseProcessor` class compatible with `AbstractDatabaseProcessor`.
- Offline mode is handled by embedding dependency content into the HTML; do not assume external CDN availability when `--offlinedependencies` is used.
- Data flow is always: parse outputs -> DB -> HTML. Reuse `RobotDashboard` methods instead of reimplementing this flow.
- Template changes should keep placeholder keys intact (e.g. `placeholder_runs`, `placeholder_css`) because replacements are string-based.
- The server only applies HTTP Basic Auth to `/admin`. All other API endpoints (`/add-outputs`, `/remove-outputs`, `/get-outputs`, etc.) are unauthenticated by design.
- After any mutation on the server (add/remove outputs, add/remove logs), the dashboard HTML is regenerated automatically unless `--no-autoupdate` is set. Use `POST /refresh-dashboard` to trigger it manually.
2 changes: 1 addition & 1 deletion .github/skills/dashboard.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ Raw database data in DataTables for runs, suites, tests, and keywords. Useful fo

## Chart.js Architecture

### Central Config: `graph_config.js`
### Central Config: `js/graph_data/graph_config.js`
`get_graph_config(graphType, graphData, graphTitle, xTitle, yTitle)` is the single factory function that returns a complete Chart.js config object. All graphs route through it.

### Supported Chart Types
Expand Down
157 changes: 157 additions & 0 deletions .github/skills/filtering-and-settings.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,157 @@
---
description: Use when working on dashboard filters, settings, localStorage persistence, the JSON config system, graph layout, or the GridStack drag-and-drop layout editor.
---

# Filtering, Settings, and Layout

## Overview

The dashboard front-end has three tightly coupled systems:
1. **Settings** — a single `settings` object that controls all display preferences, persisted in `localStorage`
2. **Filtering** — a 10-stage pipeline that produces `filteredRuns/Suites/Tests/Keywords` from the raw decoded data
3. **Layout** — GridStack-based drag-and-drop positioning + section ordering, also persisted in `settings`

Key files: `js/variables/settings.js`, `js/variables/globals.js`, `js/filter.js`, `js/localstorage.js`, `js/eventlisteners.js`, `js/layout.js`

---

## Settings Object (`js/variables/settings.js`)

The `settings` object is the single source of truth for all dashboard configuration. Top-level keys:

| Key | Type | Contents |
|---|---|---|
| `switch` | Object | Feature toggles (booleans): run tags, total stats, latest runs, percentage filters, suite paths visibility, etc. |
| `show` | Object | Display settings: date labels, legends, aliases, milliseconds, axis titles, animation, duration, rounding, timezone conversion |
| `theme_colors` | Object | `light`, `dark`, and `custom: {light, dark}` sub-objects with color values |
| `branding` | Object | `title`, `logo` |
| `menu` | Object | Which tabs are visible: `overview`, `dashboard`, `compare`, `tables` |
| `graphTypes` | Object | Per-graph chart type (e.g. `"bar"`, `"line"`, `"percentages"`) |
| `view` | Object | Per-page `sections: {show, hide}` and `graphs: {show, hide}` lists for overview, unified, dashboard, compare, tables |

**localStorage-only keys** (not in the defaults object but always preserved during merge):
- `layouts` — GridStack position data per section
- `libraries` — keyword library show/hide toggles
- `theme` — `"dark"` or `"light"`
- `filterProfiles` — named filter state snapshots

---

## localStorage Persistence (`js/localstorage.js`)

`setup_local_storage()` resolves settings on every page load using this priority:

1. `force_json_config && hasJsonConfig` → use provided JSON config (overrides localStorage)
2. `storedSettings` in localStorage → deserialize and deep-merge with current defaults
3. `!force_json_config && hasJsonConfig` → use JSON config as first-time default only
4. Fallback → use hardcoded `settings` defaults

After resolving, always writes back to `localStorage.setItem('settings', JSON.stringify(settings))`.

`set_local_storage_item(path, value)` — updates the live `settings` object using a dot-separated path (e.g. `"show.legends"`) and persists immediately. Used by all settings toggle handlers.

### Deep Merge Behavior (`merge_deep`)

When merging persisted settings with current defaults:
- `view` → `merge_view()`: preserves `show`/`hide` lists, **purges** entries for graphs/sections that no longer exist
- `layouts` → `merge_layout()`: removes graph IDs that no longer exist in `collect_allowed_graphs()`
- `theme_colors` → `merge_theme_colors()`: preserves the `custom` sub-key
- Keys in localStorage that don't exist in defaults are **dropped** (schema migration) — **except** the four localStorage-only keys above

---

## JSON Config (`--jsonconfig` / `--forcejsonconfig`)

`-j / --jsonconfig path` provides an initial settings JSON file. It is embedded into the generated HTML as `placeholder_json_config` and exposed as the `json_config` JS variable.

`--forcejsonconfig` (boolean, default `False`) — when `True`, the JSON config overrides localStorage on every page load (`force_json_config` JS variable is `"true"`).

In `localstorage.js`: if the placeholder string was not replaced (i.e. the string still contains `"placeholder_"`), `hasJsonConfig` is `false` and the JSON config branch is skipped entirely.

---

## Filtering Pipeline (`js/filter.js`)

`setup_filtered_data_and_filters()` is called when the filter modal closes or on initial load. It runs these 10 stages in order:

| Stage | What it does |
|---|---|
| 1. Remove milliseconds | Strips sub-second precision from `run_start` if `settings.show.milliseconds` is false |
| 2. Convert timezone | Converts `run_start` timestamps to viewer's local timezone if `settings.show.convertTimezone` is true |
| 3. Remove timezone display | Strips `+HH:MM` suffix if `settings.show.timezones` is false |
| 4. `filter_runs()` | Filter by run name, `#runs` dropdown (`"All"` or specific name) |
| 5. `filter_runtags()` | Filter by run tag checkboxes (`#runTag`); AND/OR logic via `#useOrTags`; returns empty if no tags selected |
| 6. `filter_dates()` | Date range via `#fromDate`, `#fromTime`, `#toDate`, `#toTime`; validates range |
| 7. `filter_amount()` | Keep last N runs (`#amount` input); handles edge cases (empty, negative, decimal/comma) |
| 8. `filter_metadata()` | Filter by metadata key:value (`#metadata` dropdown; exact match against `run.metadata`) |
| 9. `filter_project_versions()` | Filter by project version checkboxes (`#projectVersionList`); `"None"` covers runs with null version |
| 10. `filter_data()` | Filter suites/tests/keywords to only include entries whose `run_start` is in `filteredRuns`; applies `settings.libraries` to exclude disabled keyword libraries |

After filtering: updates count headlines (`#runTitle`, etc.), repopulates all dropdown selects (compare run selects, suite/test/keyword selects, test tag selects), then calls `sort_wall_clock()` to re-sort all four arrays chronologically.

---

## Filter Profiles

Named snapshots of the filter modal state, stored in `settings.filterProfiles` (localStorage-only).

- `build_profile_from_checks()` — captures all checkbox/input/select states from the filter modal
- `apply_filter_profile(name)` — restores a saved profile into the modal's UI and triggers filtering
- The active profile name and whether it has been modified since last save is shown in the profile dropdown

---

## Global State (`js/variables/globals.js`)

Key mutable globals used across modules:

| Global | Description |
|---|---|
| `filteredRuns`, `filteredSuites`, `filteredTests`, `filteredKeywords` | Current filtered data arrays (reassigned by `filter.js` after each filter pass) |
| `filteredAmount` | Total count before amount-slicing (shown as "showing X of N runs") |
| `gridUnified`, `gridRun`, `gridSuite`, `gridTest`, `gridKeyword`, `gridCompare` | GridStack instances per section |
| `gridEditMode` | Boolean: layout editor is active |
| `inFullscreen`, `inFullscreenGraph` | Fullscreen graph state; increases data limits (Top 10/30 → Top 50/100) |
| `selectedRunSetting`, `selectedTagSetting` | Applied when navigating to dashboard from overview with a pre-selected filter |
| `projects_by_tag`, `projects_by_name`, `latestRunByProjectTag`, `latestRunByProjectName` | Overview page project groupings (populated by `prepare_overview()`) |

---

## Layout System (`js/layout.js`)

### Section Order

`setup_section_order()` reads `settings.view.[page].sections.show` and `.hide` arrays, then uses jQuery `insertAfter()` to rearrange section `<div>` elements relative to DOM anchor elements (`#topDashboardSection`, `#topOverviewSection`). In `gridEditMode`, hidden sections are also rendered with move-up/move-down and show/hide toggle buttons.

### Graph Layout (GridStack)

One `GridStack` instance per section. Config: `cellHeight: 100`, `float: false`, `resizable: { handles: 'all' }`. Grid is disabled (non-draggable/resizable) outside edit mode.

Each graph is a `grid-stack-item` with attributes: `gs-x`, `gs-y`, `gs-w` (default 4), `gs-h` (default 4), min/max constraints (w: 3–12, h: 3–12), and `data-gs-id`.

**Saved positioning**: `settings.layouts.gridRun` (and `.gridSuite`, `.gridTest`, etc.) holds GridStack layout as a JSON string `[{id, x, y, w, h}, ...]`. On load, saved coordinates are applied if present; otherwise sequential left-to-right placement is computed (4 wide, 3 columns, then wrap).

In normal mode, hidden graphs are rendered to a `#[section]DataHidden` container (no GridStack) so their data computations still run.

### Edit Mode Flow

1. User clicks "Customize Layout" → `gridEditMode = true` → full re-render with drag/resize enabled and show/hide buttons visible
2. User drags/resizes graphs, reorders sections, toggles show/hide buttons
3. User clicks "Save Layout" → `save_layout()`:
- Reads `window[gridId].engine.nodes` for `{x, y, w, h, id}` per widget
- Reads `.shown-graph`/`.hidden-graph` button visibility for show/hide lists
- Reads DOM order of `.move-up-section` for section order
- Persists all to localStorage via `set_local_storage_item`
4. `gridEditMode = false` → full re-render in normal mode

### Layout Merge on Load

`merge_layout()` in `localstorage.js` purges any saved layout entries whose `id` is not in `collect_allowed_graphs()` — prevents stale graph references after settings changes remove graphs.

---

## Event Wiring (`js/eventlisteners.js`)

`setup_filter_modal()` — wires all filter modal controls. The Bootstrap `hidden.bs.modal` event triggers `setup_filtered_data_and_filters()` + `update_dashboard_graphs()` (with a loading overlay and double `requestAnimationFrame` to let the DOM settle before rendering).

`setup_settings_modal()` — data-driven: an array of `{key, elementId, isNumber}` objects creates toggle handlers for all settings checkboxes/inputs. For each: loads stored value on init, attaches change listener that calls `set_local_storage_item(key, value)`. Theme color pickers use `create_theme_color_handler()`, which reads/writes `settings.theme_colors.custom.[light|dark].[colorKey]`.
Loading
Loading