Test Coverage Gap: spec use-cases vs unit/integration/e2e (UC-01..UC-19)

[levleontiev]

Context

This issue tracks uncovered and partially covered scenarios found during a spec-vs-tests audit for Fairvisor Edge.

Audit Summary

Covered well:

• UC-01 runaway client bug
• UC-03 noisy-neighbor isolation
• UC-06 kill-switch emergency freeze
• UC-07 shadow mode rollout behavior
• UC-11 cost-budget staged actions

Partially covered / missing end-to-end confidence:

• UC-02 bot abuse policy-flow
• UC-04 tenant tier/plan branch behavior
• UC-05 onboarding handshake
• UC-08 partial config apply semantics
• UC-09 SaaS unavailable + last-known policy continuity
• UC-10 clock skew observability
• UC-12 webhook loop detection on real HTTP flow
• UC-13 loop_detected reason on wire
• UC-14 spend-rate circuit-breaker at e2e level
• UC-15 fixed-cost endpoint semantics at integration/e2e level
• UC-16 OpenAI-compatible TPM/TPD wire contract
• UC-17 streaming mid-stream truncation at e2e level
• UC-18 token-shadow analytics path
• UC-19 AI crawler staged actions

Priority Gaps

P0

• Missing e2e for UC-17 streaming mid-stream truncation (SSE wire behavior)
• Missing e2e for UC-16 OpenAI-compatible token-limit wire contract
• Missing end-to-end check for UC-09 last-known policy during SaaS partition

P1

• Missing integration/e2e for UC-08 partial apply with invalid policy among valid ones
• Missing e2e for UC-13 loop_detected reason at HTTP wire level
• Missing e2e for UC-19 robots staged actions (warn -> throttle -> reject)
• Missing integration/e2e for UC-10 clock-skew observability path

P2

• Missing e2e onboarding handshake coverage for UC-05
• Weak integration/e2e coverage for UC-15 fixed_cost policy semantics
• Missing deterministic tier/plan branch scenario for UC-04

Proposed Test Additions

P0

1. tests/e2e/test_llm_streaming.py

• Scenario: stream=true, completion exceeds max_completion_tokens mid-stream
• Assert graceful close, finish_reason=length, partial usage emitted

2. tests/e2e/test_llm_openai_contract.py

• Scenario A: TPM exceeded -> HTTP 429 + OpenAI-compatible error payload + Retry-After
• Scenario B: TPD exceeded -> same contract expectations

3. spec/integration/saas_client_spec.lua (+ feature scenarios)

• Scenario: after successful initial bundle, SaaS becomes unavailable
• Assert continued enforcement with last-known bundle, circuit open, no policy loss

P1

4. spec/integration/features/bundle_loader.feature

• Scenario: bundle contains 3 policies (1 invalid)
• Assert invalid skipped, valid policies enforce through rule_engine

5. tests/e2e/test_loop_detection.py

• Scenario: repeated identical requests trigger loop_detected
• Assert reason header on wire

6. tests/e2e/test_ai_crawlers.py

• Scenario: GPTBot/ClaudeBot UA + ip key staged actions progress warn -> throttle -> reject

7. spec/integration/features/saas_client.feature

• Scenario: clock skew >10s annotates observability/event payload flags

P2

8. tests/e2e/test_saas_onboarding.py

• Scenario: EDGE_ID/TOKEN registration happy path + config pull/ack

9. spec/integration/features/cost_budget.feature

• Scenario: fixed_cost endpoint consumes expected units regardless of payload

10. spec/integration/features/rule_engine.feature

• Scenario: same route, different jwt:tier branches choose deterministic limits

Acceptance Criteria

• New tests added for all P0 gaps
• P1 and P2 scenarios triaged into follow-up issues or covered here
• Gherkin-first style preserved for unit/integration/e2e narratives
• CI green with new scenarios