Skip to content
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
# FU-P13-T19 PRD — Add integration coverage for broker-hosted Web UI observability

## Objective

Add an integration test that exercises broker-mode telemetry end-to-end and validates that Web UI APIs expose aggregated metrics and audit visibility for multiple broker-connected clients. The test must simulate realistic concurrent/near-concurrent client traffic over the broker Unix socket, ensure request/response telemetry is persisted through broker transport instrumentation, and verify dashboard API consumers can read that telemetry through `/api/metrics` and `/api/audit` without race-prone assertions.

## Scope and Deliverables

- Add or update integration tests under `tests/integration/webui/`.
- Reuse real broker components (`BrokerDaemon`, `UnixSocketServer`) with a deterministic upstream test script.
- Reuse Web UI API surface (`create_app`, `TestClient`) backed by shared metrics and audit implementations used by broker-hosted mode.
- Validate success-path and error-path telemetry visibility in API responses.
- Produce execution evidence in `SPECS/INPROGRESS/FU-P13-T19_Validation_Report.md`.

## Success Criteria and Acceptance Tests

### Functional Criteria

- Aggregated metrics from broker-routed multi-client tool calls are visible via `/api/metrics`.
- At least one broker-routed error response is reflected in metrics (`total_errors`, tool error counters/error code breakdown) and audit output.
- Audit endpoint returns entries produced by broker telemetry logging (including request IDs and error fields when applicable).

### Determinism / Stability Criteria

- Assertions rely on request completion boundaries (client receives response) rather than arbitrary sleep durations.
- Any waiting for session cleanup or async propagation uses bounded polling with short timeout and clear failure messages.
- Test passes reliably in local runs and CI.

## Test-First Plan

1. Define/extend integration fixtures for:
- temporary short socket path (macOS-safe),
- deterministic upstream script that can emit both success and JSON-RPC error responses,
- broker startup/shutdown lifecycle.

2. Implement failing integration test first:
- send requests from multiple clients through broker socket,
- include one explicit failing tool call,
- query Web UI APIs and assert aggregated visibility.

3. Only adjust support code if tests expose legitimate integration gaps. If no production changes are needed, keep execution test-only.

## Execution Plan

### Phase 1: Build deterministic integration harness

- Inputs: existing broker multi-client helper patterns and Web UI e2e setup patterns.
- Outputs: broker+webui fixture(s) and request helper(s) in integration test module.
- Verification: fixture starts broker, serves API app, and tears down cleanly.

### Phase 2: Implement aggregated observability assertions

- Inputs: multi-client request streams with distinct request IDs and tool names.
- Outputs: assertions for `/api/metrics` and `/api/audit` showing combined visibility across clients.
- Verification: expected totals, tool keys, and non-empty audit entries with matching request IDs.

### Phase 3: Implement error-path observability assertions

- Inputs: one synthetic tool failure from upstream.
- Outputs: assertions for error counters/rates and audit error fields (`error`, `error_code`).
- Verification: API output includes at least one error-classified entry tied to the failing call.

## Decision Notes and Constraints

- Prefer `SharedMetricsStore` over in-memory `MetricsCollector` to mirror broker-hosted deployment behavior.
- Keep upstream behavior inside temporary script fixture to avoid monkeypatching transport internals.
- Keep assertions tolerant to naturally computed latency values (check presence/non-negative rather than exact values).

## Notes (Post-Implementation)

- If test naming or organization changes, ensure discoverability remains under `tests/integration/webui/`.
- If CI runtime increases materially, record rationale and tradeoffs in the validation report.

---
**Archived:** 2026-02-28
**Verdict:** PASS
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
# Validation Report: FU-P13-T19 — Add integration coverage for broker-hosted Web UI observability

**Date:** 2026-02-28
**Task ID:** FU-P13-T19
**Verdict:** PASS

## Scope Implemented

- Added broker-hosted Web UI observability integration coverage:
- `tests/integration/webui/test_broker_observability.py`
- New integration scenario uses real broker components (`BrokerDaemon`, `UnixSocketServer`) with:
- deterministic upstream success and error responses,
- multi-client broker socket traffic,
- Web UI API assertions against `/api/metrics` and `/api/audit`.
- Coverage includes success-path aggregation and error-path telemetry (metrics + audit).

## Quality Gates

- `pytest -q` → FAIL in this environment (`ModuleNotFoundError: mcpbridge_wrapper` due src-layout import path)
- `PYTHONPATH=src pytest tests/integration/webui/test_broker_observability.py -q` → PASS (`1 passed`)
- `PYTHONPATH=src pytest` → PASS (`693 passed, 5 skipped, 2 warnings`)
- `python -m ruff check src/` → PASS (`All checks passed!`)
- `mypy src/` → PASS (`Success: no issues found in 18 source files`)
- `PYTHONPATH=src pytest --cov=src/mcpbridge_wrapper --cov-report=term-missing` → PASS
- Total coverage: **91.72%** (required: >= 90%)

## Acceptance Criteria Check

- [x] Tests demonstrate aggregated metrics visibility for broker-connected clients.
- [x] Tests cover at least one error-path request and verify error reporting in metrics/audit output.
- [x] CI remains stable without flaky timing assumptions.

## Notes

- The integration assertions are response-boundary driven (wait for per-request responses) and avoid arbitrary sleep-based timing assumptions.
- Error telemetry validation covers both metrics error breakdown (`error_counts_by_code`) and audit entry error fields for broker-routed failures.
6 changes: 5 additions & 1 deletion SPECS/ARCHIVE/INDEX.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,12 @@
# mcpbridge-wrapper Tasks Archive

**Last Updated:** 2026-02-28 (FU-P13-T18_Document_unified_single-config_setup_for_broker_+_Web_UI_multi-agent_workflows)
**Last Updated:** 2026-02-28 (FU-P13-T19_Add_integration_coverage_for_broker-hosted_Web_UI_observability)

## Archived Tasks

| Task ID | Folder | Archived | Verdict |
|---------|--------|----------|---------|
| FU-P13-T19 | [FU-P13-T19_Add_integration_coverage_for_broker-hosted_Web_UI_observability/](FU-P13-T19_Add_integration_coverage_for_broker-hosted_Web_UI_observability/) | 2026-02-28 | PASS |
| FU-P13-T18 | [FU-P13-T18_Document_unified_single-config_setup_for_broker_+_Web_UI_multi-agent_workflows/](FU-P13-T18_Document_unified_single-config_setup_for_broker_+_Web_UI_multi-agent_workflows/) | 2026-02-28 | PASS |
| FU-P13-T17 | [FU-P13-T17_Enable_broker-hosted_Web_UI_with_shared_multi-client_telemetry/](FU-P13-T17_Enable_broker-hosted_Web_UI_with_shared_multi-client_telemetry/) | 2026-02-28 | PASS |
| FU-P13-T16 | [FU-P13-T16_Document_multi_agent_MCP_usage_and_single_Web_UI_host/](FU-P13-T16_Document_multi_agent_MCP_usage_and_single_Web_UI_host/) | 2026-02-28 | PASS |
Expand Down Expand Up @@ -165,6 +166,7 @@

| File | Description |
|------|-------------|
| [REVIEW_fu_p13_t19_broker_webui_observability.md](_Historical/REVIEW_fu_p13_t19_broker_webui_observability.md) | Review report for FU-P13-T19 |
| [REVIEW_fu_p13_t18_unified_config_docs.md](_Historical/REVIEW_fu_p13_t18_unified_config_docs.md) | Review report for FU-P13-T18 |
| [REVIEW_fu_p13_t17_broker_hosted_webui.md](_Historical/REVIEW_fu_p13_t17_broker_hosted_webui.md) | Review report for FU-P13-T17 |
| [REVIEW_fu_p13_t16_multi_agent_mcp_docs.md](_Historical/REVIEW_fu_p13_t16_multi_agent_mcp_docs.md) | Review report for FU-P13-T16 |
Expand Down Expand Up @@ -279,6 +281,8 @@

| Date | Task ID | Action |
|------|---------|--------|
| 2026-02-28 | FU-P13-T19 | Archived REVIEW_fu_p13_t19_broker_webui_observability report |
| 2026-02-28 | FU-P13-T19 | Archived Add_integration_coverage_for_broker-hosted_Web_UI_observability (PASS) |
| 2026-02-28 | FU-P13-T18 | Archived REVIEW_fu_p13_t18_unified_config_docs report |
| 2026-02-28 | FU-P13-T18 | Archived Document_unified_single-config_setup_for_broker_+_Web_UI_multi-agent_workflows (PASS) |
| 2026-02-28 | FU-P13-T17 | Archived REVIEW_fu_p13_t17_broker_hosted_webui report |
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
## REVIEW REPORT — FU-P13-T19 Broker Web UI Observability

**Scope:** origin/main..HEAD
**Files:** 6

### Summary Verdict
- [x] Approve
- [ ] Approve with comments
- [ ] Request changes
- [ ] Block

### Critical Issues

- None.

### Secondary Issues

- None.

### Architectural Notes

- The new integration path uses production broker transport + shared Web UI telemetry components, which improves confidence in the broker-hosted observability contract without introducing test-only behavior.
- The test asserts response-boundary outcomes instead of timing sleeps, which reduces flake risk for CI.

### Tests

- `pytest -q` fails in this environment due src-layout import resolution (`ModuleNotFoundError: mcpbridge_wrapper`).
- `PYTHONPATH=src pytest tests/integration/webui/test_broker_observability.py -q` passed.
- `PYTHONPATH=src pytest` passed (`693 passed, 5 skipped`).
- `python -m ruff check src/` passed.
- `mypy src/` passed.
- `PYTHONPATH=src pytest --cov=src/mcpbridge_wrapper --cov-report=term-missing` passed with **91.72%** coverage.

### Next Steps

- FOLLOW-UP skipped: no actionable review findings.
4 changes: 2 additions & 2 deletions SPECS/INPROGRESS/next.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,10 @@

## Recently Archived

- **FU-P13-T19** — Add integration coverage for broker-hosted Web UI observability (2026-02-28, PASS)
- **FU-P13-T18** — Document unified single-config setup for broker + Web UI multi-agent workflows (2026-02-28, PASS)
- **FU-P13-T17** — Enable broker-hosted Web UI with shared multi-client telemetry (2026-02-28, PASS)
- **FU-P13-T16** — Document multi-agent MCP usage and single Web UI host (2026-02-28, PASS)

## Suggested Next Tasks

- **FU-P13-T19** — Add integration coverage for broker-hosted Web UI observability
- **None** — All tasks completed (145/145)
15 changes: 8 additions & 7 deletions SPECS/Workplan.md
Original file line number Diff line number Diff line change
Expand Up @@ -2642,18 +2642,19 @@ Phase 9 Follow-up Backlog

---

#### ⬜️ FU-P13-T19: Add integration coverage for broker-hosted Web UI observability
- **Description:** Add automated test coverage that validates broker-mode telemetry visibility in Web UI APIs under multi-client request load.
#### ✅ FU-P13-T19: Add integration coverage for broker-hosted Web UI observability — Completed (2026-02-28, PASS)
- **Status:** ✅ Completed (2026-02-28, PASS)
- **Description:** Added integration coverage validating broker-hosted Web UI observability by exercising multi-client broker traffic and asserting aggregated/error telemetry via `/api/metrics` and `/api/audit`.
- **Priority:** P1
- **Dependencies:** FU-P13-T17
- **Parallelizable:** yes
- **Outputs/Artifacts:**
- New or updated integration tests under `tests/integration/webui/`
- Validation notes for deterministic multi-client telemetry assertions
- Added `tests/integration/webui/test_broker_observability.py`
- Added `SPECS/ARCHIVE/FU-P13-T19_Add_integration_coverage_for_broker-hosted_Web_UI_observability/FU-P13-T19_Validation_Report.md`
- **Acceptance Criteria:**
- [ ] Tests demonstrate aggregated metrics visibility for broker-connected clients
- [ ] Tests cover at least one error-path request and verify error reporting in metrics/audit output
- [ ] CI remains stable without flaky timing assumptions
- [x] Tests demonstrate aggregated metrics visibility for broker-connected clients
- [x] Tests cover at least one error-path request and verify error reporting in metrics/audit output
- [x] CI remains stable without flaky timing assumptions

---

Expand Down
Loading