diff --git a/FLOWCHART.md b/FLOWCHART.md index d42ee82ebc..e634704e0d 100644 --- a/FLOWCHART.md +++ b/FLOWCHART.md @@ -11,15 +11,22 @@

Owner - Version - Effective Date + Version + Effective Date Review Cycle

-**πŸ“‹ Document Owner:** CEO | **πŸ“„ Version:** 1.2 | **πŸ“… Last Updated:** 2026-04-20 (UTC) -**πŸ”„ Review Cycle:** Quarterly | **⏰ Next Review:** 2026-07-20 +**πŸ“‹ Document Owner:** CEO | **πŸ“„ Version:** 1.3 | **πŸ“… Last Updated:** 2026-05-06 (UTC) +**πŸ”„ Review Cycle:** Quarterly | **⏰ Next Review:** 2026-08-06 **🏒 Owner:** Hack23 AB (Org.nr 5595347807) | **🏷️ Classification:** Public +> **πŸ†• What changed since last review (v1.2 β†’ v1.3, 2026-05-06):** +> - Added **Political Intelligence Generation Flow** (Β§16): documents `generate-political-intelligence.ts` pipeline with catalog, daily-streams, i18n, and render stages. +> - Added **Analysis Gate Validation Flow** (Β§17): documents the 9-check (1–9b) analysis gate in `scripts/agentic/analysis-gate.ts` validating 23 artifacts across Families A–D. +> - Added **Parliamentary Data Download Flow** (Β§18): documents `download-parliamentary-data.ts`, `fetch-voting-records.ts`, and `fetch-calendar.ts` pipelines. +> - Updated Process Inventory tables with political intelligence, analysis gate, and parliamentary data processes. +> - Package version aligned to 0.8.76; 51 workflow files (22 standard + 14 agentic .md + 14 compiled .lock.yml + 1 README). +> > **πŸ†• What changed since last review (v1.1 β†’ v1.2, 2026-04-20):** > - πŸ“ˆ Added **IMF** to the agentic news-pipeline fan-out as a third primary economic data source alongside SCB and World Bank, per [ADR 0001](docs/adr/0001-adopt-imf-data-alongside-world-bank.md). IMF is reached via the **IMF TypeScript client `scripts/imf-client.ts` invoked through the bash tool** β€” *pure-TS, no MCP* β€” so the MCP server count is unchanged. > @@ -345,6 +352,9 @@ flowchart TD | 4 | User Journey | Page visit | < 3s | On demand | | 5 | Security Scanning | Code change | 5-10 min | Per commit | | 6 | Multi-Language | Content creation | 15-30 min | Per article | +| 7 | Political Intelligence | Prebuild chain | 2-4 min | Per build | +| 8 | Analysis Gate | Pre-article | 1-2 min | Per article | +| 9 | Parliamentary Data | Cron daily | 5-10 min | Daily | --- @@ -881,6 +891,122 @@ flowchart TD --- +## 16. 🧠 Political Intelligence Generation Flow + +```mermaid +flowchart TD + TRIGGER[generate-political-intelligence.ts] --> CATALOG[catalog.ts
Load article-types.json registry] + CATALOG --> STREAMS[daily-streams.ts
Identify active streams per horizon] + STREAMS --> HORIZON{Horizon Stratification} + HORIZON --> H72[T+72h Short-term] + HORIZON --> H7D[T+7d Weekly] + HORIZON --> H30D[T+30d Monthly] + HORIZON --> H90D[T+90d Quarterly] + HORIZON --> H365D[T+365d Annual] + HORIZON --> H1460D[T+1460d Election cycle] + + H72 --> I18N[i18n/
14-language translation] + H7D --> I18N + H30D --> I18N + H90D --> I18N + H365D --> I18N + H1460D --> I18N + + I18N --> RENDER[render/
Template-based HTML rendering] + RENDER --> OUTPUT[Output: political-intelligence*.html
14 language variants] + + style TRIGGER fill:#2196f3,color:#ffffff + style OUTPUT fill:#4caf50,color:#000000 + style HORIZON fill:#ff9800,color:#000000 + style CATALOG fill:#9c27b0,color:#ffffff + style STREAMS fill:#9c27b0,color:#ffffff + style RENDER fill:#9c27b0,color:#ffffff +``` + +--- + +## 17. βœ… Analysis Gate Validation Flow + +```mermaid +flowchart TD + INPUT[analysis/daily/DATE/ folder] --> CHECK1{Check 1:
Artifact existence
23 files across Families A-D} + CHECK1 -->|Pass| CHECK2{Check 2:
No stub content} + CHECK1 -->|Fail| FAILED[❌ Gate FAILED
Missing artifacts] + + CHECK2 -->|Pass| CHECK3{Check 3:
Minimum word count} + CHECK2 -->|Fail| FAILED + + CHECK3 -->|Pass| CHECK4{Check 4:
Evidence citations} + CHECK3 -->|Fail| FAILED + + CHECK4 -->|Pass| CHECK5{Check 5:
Mermaid diagrams present} + CHECK4 -->|Fail| FAILED + + CHECK5 -->|Pass| CHECK6{Check 6:
Pass-2 iteration evidence} + CHECK5 -->|Fail| FAILED + + CHECK6 -->|Pass| CHECK7{Check 7:
Cross-references valid} + CHECK6 -->|Fail| FAILED + + CHECK7 -->|Pass| CHECK8{Check 8:
Data-source audit trail} + CHECK7 -->|Fail| FAILED + + CHECK8 -->|Pass| CHECK9A{Check 9a:
Political classification} + CHECK8 -->|Fail| FAILED + + CHECK9A -->|Pass| CHECK9B{Check 9b:
Agency evidence} + CHECK9A -->|Fail| FAILED + + CHECK9B -->|Pass| PASSED[βœ… Gate PASSED
Article generation proceeds] + CHECK9B -->|Fail| FAILED + + FAILED --> BLOCK[Block article pipeline
Return error details] + + style INPUT fill:#2196f3,color:#ffffff + style PASSED fill:#4caf50,color:#000000 + style FAILED fill:#f44336,color:#ffffff + style BLOCK fill:#f44336,color:#ffffff + style CHECK1 fill:#ff9800,color:#000000 + style CHECK9B fill:#ff9800,color:#000000 +``` + +--- + +## 18. πŸ“₯ Parliamentary Data Download Flow + +```mermaid +flowchart TD + TRIGGER[download-parliamentary-data.ts] --> PROPS[Fetch Propositions
from data.riksdagen.se] + TRIGGER --> MOTIONS[Fetch Motions
from data.riksdagen.se] + TRIGGER --> BET[Fetch BetΓ€nkanden
from data.riksdagen.se] + + PROPS --> VALIDATE_P[Validate & transform] + MOTIONS --> VALIDATE_M[Validate & transform] + BET --> VALIDATE_B[Validate & transform] + + VALIDATE_P --> DATA_DIR[Output: data/ directory] + VALIDATE_M --> DATA_DIR + VALIDATE_B --> DATA_DIR + + TRIGGER --> VOTING[fetch-voting-records.ts
Download voting records] + VOTING --> VOTE_DATA[Voting data validated] + VOTE_DATA --> DATA_DIR + + TRIGGER --> CALENDAR[fetch-calendar.ts
Download calendar events] + CALENDAR --> CAL_DATA[Calendar data validated] + CAL_DATA --> DATA_DIR + + DATA_DIR --> READY[Parliamentary data ready
for analysis pipeline] + + style TRIGGER fill:#2196f3,color:#ffffff + style READY fill:#4caf50,color:#000000 + style DATA_DIR fill:#ff9800,color:#000000 + style VOTING fill:#9c27b0,color:#ffffff + style CALENDAR fill:#9c27b0,color:#ffffff +``` + +--- + ## Updated Process Inventory | # | Process | Trigger | Duration | Frequency | Security Controls | @@ -897,6 +1023,9 @@ flowchart TD | 10 | Data Validation | Per data fetch | 1-2 min | Per fetch | 9-stage validation pipeline | | 11 | Content Integrity | Per content | < 1 min | Per article | Git signatures, Sigstore (build artifacts) | | 12 | Runner Hardening | Per job | Continuous | Per job | iptables, egress audit | +| 13 | Political Intelligence | Prebuild chain | 2-4 min | Per build | HTMLHint, schema validation | +| 14 | Analysis Gate | Pre-article | 1-2 min | Per article | 9-check validation (23 artifacts) | +| 15 | Parliamentary Data | Cron daily | 5-10 min | Daily | Data validation, freshness check | --- diff --git a/STATEDIAGRAM.md b/STATEDIAGRAM.md index b2c5d6b30e..e6d6cd4360 100644 --- a/STATEDIAGRAM.md +++ b/STATEDIAGRAM.md @@ -11,20 +11,27 @@

Owner - Version - Effective Date + Version + Effective Date Review Cycle

-**πŸ“‹ Document Owner:** CEO | **πŸ“„ Version:** 1.1 | **πŸ“… Last Updated:** 2026-04-20 (UTC) -**πŸ”„ Review Cycle:** Quarterly | **⏰ Next Review:** 2026-07-20 +**πŸ“‹ Document Owner:** CEO | **πŸ“„ Version:** 1.2 | **πŸ“… Last Updated:** 2026-05-06 (UTC) +**πŸ”„ Review Cycle:** Quarterly | **⏰ Next Review:** 2026-08-06 **🏒 Owner:** Hack23 AB (Org.nr 5595347807) | **🏷️ Classification:** Public +> **πŸ†• What changed since last review (v1.1 β†’ v1.2, 2026-05-06):** +> - Added **analysis artifact lifecycle** state diagram (empty β†’ populated β†’ gate-checked β†’ passed β†’ consumed) documenting the 23-artifact analysis folder lifecycle. +> - Added **analysis gate check** state machine (check-1 through check-9b) per `scripts/agentic/analysis-gate.ts`. +> - Added **political intelligence states** (collection β†’ processing β†’ analysis β†’ dissemination β†’ feedback). +> - Anchored to current toolchain: Node.js β‰₯26, TypeScript 6.0.3, Vite 8.0.10, Vitest 4.1.5, Cypress 15.14.2. +> - Verified all existing state diagrams: news article lifecycle, agentic workflow states, CIA data refresh, release pipeline, incident state machine, npm package lifecycle. +> > **πŸ†• What changed since last review (v1.0 β†’ v1.1, 2026-04-20):** > - Refreshed lifecycle state machines for: (a) **news articles** (draft β†’ safe-output validation β†’ translator fan-out across 14 languages β†’ merged β†’ published β†’ superseded/retired); (b) **agentic workflow runs** (queued β†’ running β†’ safe-output-validation β†’ reviewer-PR β†’ merged/rolled-back); (c) **CIA data refresh** (scheduled β†’ download β†’ schema-validate β†’ diff β†’ auto-PR β†’ merge β†’ deploy); (d) **release pipeline** (tag β†’ build β†’ provenance-attest β†’ npm publish β†’ S3 sync β†’ CloudFront invalidation); (e) **translation states** across 14 locales including RTL (HE, AR). > - Added state diagrams for **npm package lifecycle** (unpublished β†’ published-with-provenance β†’ deprecated β†’ security-advisory β†’ superseded) aligned with `End-of-Life-Strategy.md`. > - Added **incident state machine**: detected β†’ triaged β†’ contained β†’ eradicated β†’ recovered β†’ post-mortem, mapped to [Incident_Response_Plan](https://github.com/Hack23/ISMS-PUBLIC/blob/main/Incident_Response_Plan.md). -> - Anchored to current toolchain: Node.js β‰₯25, TypeScript 6.0.3, Vite 8.0.9, Vitest 4.1.4, Cypress 15.14.0 (optional), Playwright 1.59.1. +> - Anchored to current toolchain: Node.js β‰₯26, TypeScript 6.0.3, Vite 8.0.10, Vitest 4.1.5, Cypress 15.14.2, Playwright 1.52.0. > - Compliance mapping: ISO 27001:2022 A.5.24/A.5.26, NIST CSF 2.0 RS.AN/RS.MI/RC.RP, CIS Controls v8.1 #17, NIS2 Art. 21, EU CRA Annex I Β§2. --- @@ -930,6 +937,155 @@ Action Required: --- +## 6A. πŸ“¦ Analysis Artifact Lifecycle States + +**πŸ“Š Data Focus:** 23 analysis artifacts (Families A-D) produced by every agentic news workflow. + +**πŸ”„ Process Focus:** Shows artifact folder lifecycle from creation through consumption by article rendering. + +**πŸ€– AI Integration:** Artifacts are produced by Copilot agents and validated by `scripts/agentic/analysis-gate.ts`. + +```mermaid +stateDiagram-v2 + [*] --> Empty + + Empty --> Populating: Agentic workflow starts writing artifacts + + Populating --> Populated: All 23 artifacts written + Populating --> Incomplete: Timeout or agent error + + Populated --> GateChecking: analysis-gate.ts invoked + + GateChecking --> GatePassed: All checks 1-9b pass + GateChecking --> GateFailed: One or more checks fail + + GateFailed --> Populating: Agent retries failed artifacts + + GatePassed --> Consumed: render-articles.ts reads artifacts + + Consumed --> Archived: Article published, artifacts retained + Archived --> [*] + + Incomplete --> [*]: Workflow terminates with error + + note right of Empty + analysis/daily/YYYY-MM-DD/type/ folder + 23 expected files (Families A-D) + end note + + note right of GateChecking + Check 1: Artifact existence (23 files) + Check 2: No stub placeholders + Check 3: Minimum word count + Check 4: Evidence citations + Check 5: Mermaid diagrams with colour + Check 6: Pass-2 evidence (revision proof) + Check 7: Cross-references + Check 8: Data-source connectivity audit + Check 9a: Political classification + Check 9b: Agency evidence (Statskontoret) + end note + + note right of Consumed + aggregate-analysis.ts aggregates + render-articles.ts renders HTML + Artifacts become read-only + end note +``` + +### 6A.1 State Definitions + +| State | Description | Entry Conditions | Exit Conditions | Typical Duration | +|-------|-------------|------------------|-----------------|------------------| +| **EMPTY** | Analysis folder created but no artifacts written | Workflow triggered | First artifact written | Seconds | +| **POPULATING** | Agent writing artifacts to folder | First artifact write | All 23 present or timeout | 5-15 minutes | +| **INCOMPLETE** | Timeout or error, fewer than 23 artifacts | Agent timeout/error | Workflow terminates | N/A | +| **POPULATED** | All 23 artifacts present in folder | 23 files written | Gate check invoked | Seconds | +| **GATE_CHECKING** | analysis-gate.ts running checks 1-9b | Populated state | Pass or fail | 5-30 seconds | +| **GATE_PASSED** | All gate checks passed | All checks pass | Render starts | Seconds | +| **GATE_FAILED** | One or more checks failed | Any check fails | Retry or terminate | Variable | +| **CONSUMED** | Artifacts read by render pipeline | render-articles.ts invoked | Article published | 1-5 minutes | +| **ARCHIVED** | Artifacts retained for audit trail | Article published | Permanent | Permanent | + +--- + +## 6B. πŸ•΅οΈ Political Intelligence Assessment States + +**πŸ“Š Data Focus:** Intelligence assessment lifecycle for political analysis products. + +**πŸ”„ Process Focus:** Shows the intelligence cycle from collection through dissemination and feedback. + +**πŸ€– AI Integration:** Horizon stratification (T+72h β†’ T+1460d) drives assessment scope and confidence levels. + +```mermaid +stateDiagram-v2 + [*] --> Collection + + Collection --> Processing: Raw data gathered + + Processing --> Analysis: Data cleaned and structured + + Analysis --> Assessment: Analytical products created + + Assessment --> Dissemination: Confidence labels assigned + + Dissemination --> Feedback: Published to audience + + Feedback --> Collection: New requirements identified + Feedback --> [*]: Cycle complete + + Collection --> Collection: Continuous monitoring + + note right of Collection + Sources: Riksdag MCP (32 tools) + SCB statistics, IMF economic data + Government documents, voting records + end note + + note right of Analysis + 20 methodologies applied + 39 analysis templates used + Horizon-specific framing: + T+72h, T+7d, T+30d, T+90d, T+365d, T+1460d + end note + + note right of Assessment + Confidence levels: High/Medium/Low + WEP language ladder per horizon band + Scenario analysis (β‰₯3 scenarios) + end note + + note right of Dissemination + 14 languages, RTL support + Article rendering pipeline + RSS, sitemap, indexes generated + end note +``` + +### 6B.1 State Definitions + +| State | Description | Entry Conditions | Exit Conditions | Typical Duration | +|-------|-------------|------------------|-----------------|------------------| +| **COLLECTION** | Gathering raw political data from sources | Workflow trigger or schedule | Sufficient data gathered | 2-5 minutes | +| **PROCESSING** | Cleaning, structuring, deduplication | Raw data available | Structured data ready | 1-3 minutes | +| **ANALYSIS** | Applying methodologies and templates | Structured data ready | Analysis artifacts produced | 5-15 minutes | +| **ASSESSMENT** | Assigning confidence, creating judgments | Analysis complete | Assessment products ready | 2-5 minutes | +| **DISSEMINATION** | Publishing to audience in multiple languages | Assessment approved | Content published | 3-10 minutes | +| **FEEDBACK** | Receiving feedback, identifying new requirements | Content published | New PIRs identified | Continuous | + +### 6B.2 Horizon Stratification + +| Horizon | Timeframe | Scope | Confidence Band | Scenario Depth | +|---------|-----------|-------|-----------------|----------------| +| **T+72h** | 3 days | Tactical | High | 2-3 scenarios | +| **T+7d** | 1 week | Tactical/Operational | High-Medium | 3 scenarios | +| **T+30d** | 1 month | Operational | Medium | 3-4 scenarios | +| **T+90d** | Quarter | Strategic | Medium-Low | 4 scenarios | +| **T+365d** | 1 year | Strategic | Low-Medium | 4+ scenarios + wildcards | +| **T+1460d** | Election cycle | Grand strategic | Low | 4Γ—3 coalition branches + wildcards | + +--- + ## 7. πŸ“Š State Metrics and Monitoring ### 7.1 Time in State (Average Duration) diff --git a/WORKFLOWS.md b/WORKFLOWS.md index 5dfa546771..61f5103981 100644 --- a/WORKFLOWS.md +++ b/WORKFLOWS.md @@ -11,15 +11,22 @@

Owner - Version - Last Updated + Version + Last Updated Review Cycle

-**πŸ“‹ Document Owner:** CEO | **πŸ“„ Version:** 7.4 | **πŸ“… Last Updated:** 2026-05-05 (UTC) +**πŸ“‹ Document Owner:** CEO | **πŸ“„ Version:** 7.5 | **πŸ“… Last Updated:** 2026-05-06 (UTC) **πŸ”„ Review Cycle:** Quarterly | **⏰ Next Review:** 2026-08-05 **🏒 Owner:** Hack23 AB (Org.nr 5595347807) | **🏷️ Classification:** Public +> **πŸ†• What changed since last review (v7.4 β†’ v7.5, 2026-05-06):** +> - πŸ›οΈ Reconciled document with **v0.8.76** β€” confirmed 51 workflow files (22 standard `.yml` + 14 agentic `.md` + 14 compiled `.lock.yml` + 1 README). +> - 🧠 Added Β§"Political Intelligence Validation Pipeline" documenting: `generate-political-intelligence.ts`, `validate-methodology-reflection.ts`, `validate-quality-scores.cjs`, `validate-article.ts`, and analysis-gate enforcement (checks 1-9b). +> - πŸ“¦ Documented analysis-gate as a CI/CD concern: every agentic workflow MUST pass 23-artifact gate before article rendering. +> - πŸ“‹ Updated prebuild chain to current 13-step sequence (added `generate-article-types-doc`, `copy-vendor-mermaid`, `aggregate-analysis`, `render-articles`, `normalize-static-html-chrome`, `backfill-translated-chrome`, `strip-legacy-chrome-script-tags`). +> - 🌐 Added parliamentary data download scripts: `download-parliamentary-data.ts`, `fetch-voting-records.ts`, `fetch-calendar.ts`, `fetch-statskontoret.ts`, `fetch-rir-followups.ts`. +> > **πŸ†• What changed since last review (v7.3 β†’ v7.4, 2026-05-05):** > - ♻️ Reconciled this document with the current `package.json` toolchain: TypeScript 6.0.3, Vite 8.0.10, Vitest 4.1.5, Cypress 15.14.2, and the current 3,319-test baseline observed in `npm test`. > - πŸ€– Corrected the agentic-news engine and workflow narrative to **14 workflows** using `claude-sonnet-4.6`, the 23-artifact baseline, nested election-cycle folders, and all-language article rendering via `render-articles.ts --all --lang all`. @@ -1445,6 +1452,60 @@ The inline bash validation logic embedded in `.github/prompts/05-analysis-gate.m --- +## Political Intelligence Validation Pipeline + +The political intelligence generation and validation is a CI/CD concern enforced both in the prebuild chain and within agentic workflows: + +### Prebuild Chain (13 Steps) + +``` +generate-article-types-doc β†’ copy-vendor-mermaid β†’ aggregate-analysis β†’ render-articles β†’ +generate-news-indexes β†’ extract-news-metadata β†’ generate-sitemap-html β†’ +generate-political-intelligence β†’ generate-rss β†’ generate-sitemap β†’ +normalize-static-html-chrome β†’ backfill-translated-chrome β†’ strip-legacy-chrome-script-tags +``` + +### Analysis Gate Enforcement + +Every agentic news workflow MUST produce 23 analysis artifacts (Families A-D) and pass the analysis gate (`scripts/agentic/analysis-gate.ts`) before article rendering: + +| Check | Validation | Script | +|-------|-----------|--------| +| 1 | Artifact existence (23 files) | `analysis-gate.ts` | +| 2 | No stub placeholders (`AI_MUST_REPLACE`, `[REQUIRED]`, `TODO:`) | `analysis-gate.ts` | +| 3 | Minimum word count per artifact | `analysis-gate.ts` | +| 4 | Evidence citations (SWOT + significance) | `analysis-gate.ts` | +| 5 | Mermaid diagrams with colour configuration | `analysis-gate.ts` | +| 6 | Pass-2 evidence (revision proof via mtime/snapshot diff) | `analysis-gate.ts` | +| 7 | Cross-references between artifacts | `analysis-gate.ts` | +| 8 | Data-source connectivity audit | `analysis-gate.ts` | +| 9a | Political classification completeness | `analysis-gate.ts` | +| 9b | Agency evidence (Statskontoret recognised agencies) | `analysis-gate.ts` | + +### Additional Validation Scripts + +| Script | Purpose | Trigger | +|--------|---------|---------| +| `validate-methodology-reflection.ts` | Validates methodology-reflection.md structure and required sections | Agentic workflow | +| `validate-quality-scores.cjs` | Validates quality score thresholds | CI quality checks | +| `validate-article.ts` | Validates article HTML structure and metadata | Prebuild + CI | +| `validate-news-translations.ts` | Detects remaining data-translate markers | Translation validation workflow | +| `validate-translations.ts` | Validates all translation files | CI | + +### Parliamentary Data Scripts + +| Script | Purpose | Schedule | +|--------|---------|----------| +| `download-parliamentary-data.ts` | Download propositions, motions, betΓ€nkanden from Riksdag API | Nightly / on-demand | +| `fetch-voting-records.ts` | Fetch voting records from Riksdag | Nightly | +| `fetch-calendar.ts` | Fetch parliamentary calendar events | Nightly | +| `fetch-statskontoret.ts` | Fetch Statskontoret agency data (headcount, budget) | Weekly | +| `fetch-rir-followups.ts` | Fetch Riksrevisionen follow-up reports | Weekly | +| `scb-fetch.ts` | Fetch SCB statistical data | On-demand | +| `imf-fetch.ts` | Fetch IMF economic data (WEO, FM, IFS, DOTS) | On-demand | + +--- + ## πŸ”’ Workflow Security Architecture ### Supply Chain Security @@ -1614,7 +1675,7 @@ flowchart TB --- -**πŸ“‹ Document Owner:** CEO | **πŸ“„ Version:** 7.4 | **πŸ“… Last Updated:** 2026-05-05 (UTC) +**πŸ“‹ Document Owner:** CEO | **πŸ“„ Version:** 7.5 | **πŸ“… Last Updated:** 2026-05-06 (UTC) **πŸ”„ Review Cycle:** Quarterly | **⏰ Next Review:** 2026-08-05 **🏒 Classification:** Public | **πŸ›οΈ Owner:** Hack23 AB (Org.nr 5595347807)