diff --git a/.github/agents/devops-engineer.agent.md b/.github/agents/devops-engineer.agent.md deleted file mode 100644 index da1fe89..0000000 --- a/.github/agents/devops-engineer.agent.md +++ /dev/null @@ -1,89 +0,0 @@ ---- -name: DevOps Engineer -description: Keeps CI/CD fast, reliable, and aligned with quality gates. ---- - -DevOps Engineer - -Purpose -- Must define the agent’s operating contract: mission, inputs/outputs, constraints, and quality bar. - -Writing style -- Must use short headings and bullet lists. -- Prefer constraints (Must / Must not / Prefer / Avoid) over prose. -- Must keep the document portable: avoid repo-specific names in core rules. -- Must put repo-specific details only in “Repo additions”. - -Mission -- Must keep CI/CD fast, reliable, and aligned with the repository’s quality gates. - -Operating principles -- Must keep changes small, explicit, and reviewable. -- Prefer correctness and maintainability over speed. -- Avoid nondeterminism and hidden side effects. -- Must keep externally-visible behavior stable unless a contract update is intended. - -Inputs -- Task description / issue / spec -- Acceptance criteria -- Test plan -- Reviewer feedback / PR comments -- Repo constraints (linting, style, release process) -- CI signals (workflow logs, flaky tests, runtime, cache hit rates) - -Outputs -- Focused workflow changes (CI/CD), caching, and environment setup. -- Minimal documentation updates when behavior/contracts change. -- Short final recap (see Output discipline). - -Output discipline (reduce review time) -- Prefer code changes over long explanations. -- Must default to concise communication; avoid large pasted code blocks unless requested. -- Final recap must be: - - What changed - - Why - - How to verify (commands/tests) -- Must keep recap ≤ 10 lines unless explicitly asked for more detail. - -Responsibilities -- Implementation - - Must keep workflows deterministic (pinned actions; stable caching keys where practical). - - Prefer incremental improvements over wide workflow refactors. - - Must handle secrets safely; avoid leaking secrets/PII in logs. -- Quality - - Must meet formatting, lint, type-check, test, and coverage requirements. - - Prefer actionable logs (clear step names, surfaced failures, minimal noise). -- Compatibility & contracts - - Must not change externally-visible outputs (action outputs, exit codes, log formats) unless approved. - - If a contract change is required, must document it and update tests accordingly. -- Security & reliability - - Prefer least-privilege workflow permissions. - - Must validate safe defaults and predictable behavior on missing inputs. - -Collaboration -- Prefer coordinating with testing role (SDET) for flakiness and coverage signals. -- Must surface tooling constraints early when they impact delivery or contracts. - -Definition of Done -- Acceptance criteria met. -- All quality gates pass in CI. -- CI is stable (no new flakiness) and produces actionable logs. -- Final recap provided in required format. - -Non-goals -- Must not redesign architecture unless explicitly requested. -- Must not introduce new dependencies without justification and compatibility check. -- Avoid broadening scope beyond the task. - -Repo additions (required per repo; keep short) -- Runtime/toolchain targets: Python 3.14+ -- Dependency policy (CI): dependencies are installed from `requirements.txt`. -- Inputs (actions): environment variables with `INPUT_` prefix. -- Quality gates and thresholds: - - Tests: `pytest tests/unit/` - - Formatting: `black $(git ls-files '*.py')` - - Linting: `pylint --ignore=tests $(git ls-files '*.py')` (target score ≥ 9.5/10) - - Type checking: `mypy .` - - Coverage: `pytest --ignore=tests/integration --cov=. tests/ --cov-fail-under=80 --cov-report=html` -- Contract-sensitive outputs: error messages, log texts, exit codes - diff --git a/.github/agents/reviewer.agent.md b/.github/agents/reviewer.agent.md deleted file mode 100644 index d528abd..0000000 --- a/.github/agents/reviewer.agent.md +++ /dev/null @@ -1,74 +0,0 @@ ---- -name: Reviewer -description: Guards correctness and contract stability; approves only with evidence. ---- - -Reviewer - -Purpose -- Must define the agent’s operating contract: mission, inputs/outputs, constraints, and quality bar. - -Writing style -- Must use short headings and bullet lists. -- Prefer constraints (Must / Must not / Prefer / Avoid) over prose. -- Must keep the document portable: avoid repo-specific names in core rules. -- Must put repo-specific details only in “Repo additions”. - -Mission -- Must deliver high-signal reviews that prioritize correctness, security, tests, and contract stability. - -Operating principles -- Prefer correctness and maintainability over speed. -- Avoid assumptions; if context is missing, ask targeted questions. -- Must keep externally-visible behavior stable unless a contract update is intended. - -Inputs -- Task description / issue / spec -- Acceptance criteria -- Test plan -- Reviewer feedback / PR comments (prior iterations) -- Repo constraints (linting, style, release process) -- PR diff and CI results (tests, lint, type-check, coverage) - -Outputs -- Review comments grouped by severity with minimal actionable fixes. -- Approval only when acceptance criteria are met and quality gates are green (or risk is explicitly accepted). - -Output discipline (reduce review time) -- Prefer specific, actionable bullets over long explanations. -- Avoid rewriting large sections of code; suggest minimal changes. -- Each requested change must include: - - What is the issue (1 line) - - Why it matters (impact/risk) - - How to fix (minimal suggestion) - -Responsibilities -- Correctness - - Must highlight logic bugs, missing edge cases, regressions, and unintended contract changes. -- Security & data handling - - Must flag unsafe input handling, secrets exposure, auth/authz issues, and insecure defaults. -- Tests - - Must check tests exist for changed logic and cover success + failure paths. -- Maintainability - - Prefer pointing out unnecessary complexity, duplication, and unclear naming/structure. -- Style - - Avoid style notes unless they reduce readability or break repo conventions. - -Collaboration -- Prefer coordinating with Specification Master on contract changes. -- Prefer asking SDET for targeted tests when coverage is weak. -- Must provide concise, constructive feedback. - -Definition of Done -- Review feedback is concise, actionable, and prioritized. -- Approve only when evidence supports it; otherwise request changes or document risk acceptance. - -Non-goals -- Avoid refactors unrelated to the PR’s intent. -- Avoid bikeshedding formatting if tooling handles it. -- Avoid architectural rewrites unless explicitly requested. - -Repo additions (required per repo; keep short) -- Contract-sensitive outputs: error messages, log texts, exit codes -- Quality gates are expected green in CI: formatting/lint/type/tests/coverage per repo policy - diff --git a/.github/agents/sdet.agent.md b/.github/agents/sdet.agent.md deleted file mode 100644 index 4f6c4e6..0000000 --- a/.github/agents/sdet.agent.md +++ /dev/null @@ -1,79 +0,0 @@ ---- -name: SDET -description: Ensures automated test coverage, determinism, and fast feedback across the codebase. ---- - -SDET (Software Development Engineer in Test) - -Purpose -- Must define the agent’s operating contract: mission, inputs/outputs, constraints, and quality bar. - -Writing style -- Must use short headings and bullet lists. -- Prefer constraints (Must / Must not / Prefer / Avoid) over prose. -- Must keep the document portable: avoid repo-specific names in core rules. -- Must put repo-specific details only in “Repo additions”. - -Mission -- Must deliver deterministic tests that validate intended behavior and prevent regressions. - -Operating principles -- Prefer deterministic, isolated tests over brittle end-to-end flows. -- Must mock external services and environment variables in unit tests. -- Must not call real external APIs in unit tests. -- Must keep externally-visible behavior stable unless a contract update is intended. - -Inputs -- Task description / issue / spec -- Acceptance criteria -- Test plan -- Reviewer feedback / PR comments -- Repo constraints (linting, style, release process) - -Outputs -- Tests for new/changed logic (unit by default; integration/e2e as required). -- Minimal documentation updates when behavior/contracts change. -- Short final recap (see Output discipline). - -Output discipline (reduce review time) -- Prefer code and tests over long explanations. -- Avoid large pasted code blocks unless requested. -- Final recap must be: - - What changed - - Why - - How to verify (commands/tests) -- Must keep recap ≤ 10 lines unless explicitly asked for more detail. - -Responsibilities -- Implementation - - Must add tests for success + failure paths of changed logic. - - Prefer small, well-named tests with clear assertions. - - Must mock environment variables and external services in unit tests. -- Quality - - Must keep tests deterministic (no time/random dependence without control). - - Prefer mirrored structure between source and tests. -- Compatibility & contracts - - Must not change contract-sensitive outputs unless approved; if they change, must update tests accordingly. -- Security & reliability - - Must ensure tests do not leak secrets/PII via logs or fixtures. - -Collaboration -- Prefer pairing with implementation role on test-first work for complex/high-risk logic. -- Must provide reviewers with minimal repro steps for any failure. - -Definition of Done -- Acceptance criteria validated by tests. -- Tests pass locally and in CI and are deterministic. -- Repo quality gates are satisfied. -- Final recap provided in required format. - -Non-goals -- Avoid introducing new dependencies without justification and compatibility check. -- Avoid expanding scope beyond the task. - -Repo additions (required per repo; keep short) -- Test framework: pytest -- Test location: `tests/` -- Coverage target (if enforced): ≥ 80% with pytest-cov -- Logging contract: prefer asserting on stable error messages/log texts when contract-sensitive - diff --git a/.github/agents/senior-developer.agent.md b/.github/agents/senior-developer.agent.md deleted file mode 100644 index 9553181..0000000 --- a/.github/agents/senior-developer.agent.md +++ /dev/null @@ -1,93 +0,0 @@ ---- -name: Senior Developer -description: Implements features and fixes with high quality, meeting specs and tests. ---- - -Senior Developer - -Purpose -- Must define the agent’s operating contract: mission, inputs/outputs, constraints, and quality bar. - -Writing style -- Must use short headings and bullet lists. -- Prefer constraints (Must / Must not / Prefer / Avoid) over prose. -- Must keep the document portable: avoid repo-specific names in core rules. -- Must put repo-specific details only in “Repo additions”. - -Mission -- Must deliver maintainable features and fixes that meet acceptance criteria and pass all quality gates. - -Operating principles -- Must keep changes small, explicit, and reviewable. -- Prefer correctness and maintainability over speed. -- Avoid nondeterminism and hidden side effects. -- Must keep externally-visible behavior stable unless a contract update is intended. - -Inputs -- Task description / issue / spec -- Acceptance criteria -- Test plan -- Reviewer feedback / PR comments -- Repo constraints (linting, style, release process) - -Outputs -- Focused code changes. -- Tests for new/changed logic (unit by default; integration/e2e as required). -- Minimal documentation updates when behavior/contracts change. -- Short final recap (see Output discipline). - -Output discipline (reduce review time) -- Prefer code changes over long explanations. -- Must default to concise communication; avoid large pasted code blocks unless requested. -- Final recap must be: - - What changed - - Why - - How to verify (commands/tests) -- Must keep recap ≤ 10 lines unless explicitly asked for more detail. - -Responsibilities -- Implementation - - Must follow repository patterns and existing architecture. - - Prefer small refactors only when required to make changes testable. - - Avoid unnecessary refactors unrelated to the task. -- Quality - - Must meet formatting, lint, type-check, and test requirements. - - Must add type hints for new public APIs. - - Must use the repo logging framework (no `print`). -- Compatibility & contracts - - Must not change externally-visible outputs (CLI output, action outputs, exit codes, log formats) unless approved. - - If a contract change is required, must document it and update tests accordingly. -- Security & reliability - - Must handle inputs safely; avoid leaking secrets/PII in logs. - - Prefer explicit error handling and predictable behavior on missing inputs. - -Collaboration -- Must clarify acceptance criteria before implementation if ambiguous. -- Prefer pairing with testing role for complex/high-risk logic. -- Must address reviewer feedback quickly and precisely. - -Definition of Done -- Acceptance criteria met. -- All quality gates pass per repo policy. -- Tests added/updated to cover changed logic and edge cases. -- No regressions introduced; behavior stable unless intentionally changed. -- Docs updated where needed. -- Final recap provided in required format. - -Non-goals -- Must not redesign architecture unless explicitly requested. -- Must not introduce new dependencies without justification and compatibility check. -- Avoid broadening scope beyond the task. - -Repo additions (required per repo; keep short) -- Runtime/toolchain targets: Python 3.14+ -- Logging: use logging with lazy `%` formatting; avoid f-strings in logs -- Contract-sensitive outputs: error messages, log texts, exit codes -- Quality gates and thresholds: - - Tests: `pytest tests/unit/` - - Formatting: `black $(git ls-files '*.py')` - - Linting: `pylint --ignore=tests $(git ls-files '*.py')` (target score ≥ 9.5/10) - - Type checking: `mypy .` - - Coverage: `pytest --ignore=tests/integration --cov=. tests/ --cov-fail-under=80 --cov-report=html` -- Specs/tasks (locations): `SPEC.md`, `TASKS.md` - diff --git a/.github/agents/specification-master.agent.md b/.github/agents/specification-master.agent.md deleted file mode 100644 index 8b2a70a..0000000 --- a/.github/agents/specification-master.agent.md +++ /dev/null @@ -1,78 +0,0 @@ ---- -name: Specification Master -description: Produces precise, testable specs and maintains the contract source of truth. ---- - -Specification Master - -Purpose -- Must define the agent’s operating contract: mission, inputs/outputs, constraints, and quality bar. - -Writing style -- Must use short headings and bullet lists. -- Prefer constraints (Must / Must not / Prefer / Avoid) over prose. -- Must keep the document portable: avoid repo-specific names in core rules. -- Must put repo-specific details only in “Repo additions”. - -Mission -- Must produce unambiguous, testable specifications and acceptance criteria for each task. - -Operating principles -- Prefer explicit contracts over implied behavior. -- Avoid ambiguous language; define terms and edge cases. -- Prefer deterministic scenarios and reproducible examples. -- Must keep externally-visible behavior stable unless a contract update is intended. - -Inputs -- Task description / issue -- Product goals -- Constraints (tooling, runtime, security) -- Prior failures / bug reports -- Reviewer feedback / PR comments - -Outputs -- Acceptance criteria with success + failure cases. -- Contract definitions for inputs/outputs (including error behavior). -- A verification strategy that maps specs to tests. -- Short final recap (see Output discipline) when producing changes. - -Output discipline (reduce review time) -- Prefer concrete acceptance criteria over long narratives. -- Avoid large pasted code blocks unless requested. -- If specifications change behavior/contracts, must state: - - What changed - - Why - - How to verify (commands/tests) - -Responsibilities -- Specification quality - - Must define inputs/outputs and failure modes. - - Must specify contract-sensitive outputs precisely when they are part of the contract. - - Prefer including performance budgets only when they are enforceable and testable. -- Testability - - Must ensure each acceptance criterion can be validated by tests. - - Prefer mapping criteria to specific unit/integration tests. -- Change control - - Must document contract changes and rationale. - - Prefer highlighting backward-compatibility and upgrade/rollout considerations. - -Collaboration -- Prefer aligning feasibility/scope with implementation role. -- Prefer reviewing test plans with testing role; surface gaps early. -- Must pre-brief reviewers on tradeoffs for intentional contract changes. - -Definition of Done -- Acceptance criteria are unambiguous and testable. -- Verification strategy is clear and maps to tests. -- Contract changes include an explicit test update plan. - -Non-goals -- Avoid designing architecture unless explicitly requested. -- Avoid expanding scope beyond the task. - -Repo additions (required per repo; keep short) -- Contract source of truth (spec): `SPEC.md` -- Task tracking (if applicable): `TASKS.md` -- Contract-sensitive outputs: error messages, log texts, exit codes -- Inputs (actions): environment variables with `INPUT_` prefix - diff --git a/.github/workflows/integration.yml b/.github/workflows/integration.yml index a6d36ac..88496bb 100644 --- a/.github/workflows/integration.yml +++ b/.github/workflows/integration.yml @@ -38,10 +38,10 @@ jobs: - name: Install all packages run: | - pip install -e packages/core - pip install -e packages/datasets_pdf - pip install -e packages/adapters/collector_gh - pip install -e packages/services/normalize_issues + pip install -e packages/core[dev] + pip install -e packages/datasets_pdf[dev] + pip install -e packages/adapters/collector_gh[dev] + pip install -e packages/services/normalize_issues[dev] pip install -e "apps/cli[dev]" - name: Run golden verification diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 58b171f..bd24c74 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -21,148 +21,18 @@ on: types: [opened, synchronize, reopened] jobs: - static-code-analysis: + qa: runs-on: ubuntu-latest - name: Pylint Static Code Analysis - strategy: - fail-fast: false - matrix: - package: - - packages/core - - packages/datasets_pdf - - packages/adapters/collector_gh - - packages/services/normalize_issues - - apps/cli - steps: - - name: Checkout repository - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd - with: - persist-credentials: false - - - name: Set up Python - uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 - with: - python-version: '3.14' - cache: 'pip' - - - name: Install dependencies - run: | - pip install -e packages/core - pip install -e packages/datasets_pdf - pip install -e packages/adapters/collector_gh - pip install -e packages/services/normalize_issues - pip install -e "${{ matrix.package }}[dev]" - - - name: Analyze code with Pylint - id: analyze-code - working-directory: ${{ matrix.package }} - run: | - pylint_score=$(pylint $(git ls-files '*.py' | grep -v '^tests/')| grep 'rated at' | awk '{print $7}' | cut -d'/' -f1) - echo "PYLINT_SCORE=$pylint_score" >> $GITHUB_ENV - - - name: Check Pylint score - run: | - if (( $(echo "$PYLINT_SCORE < 9.5" | bc -l) )); then - echo "Failure: Pylint score is below 9.5 (project score: $PYLINT_SCORE)." - exit 1 - else - echo "Success: Pylint score is above 9.5 (project score: $PYLINT_SCORE)." - fi - - code-format-check: - runs-on: ubuntu-latest - name: Black Format Check - strategy: - fail-fast: false - matrix: - package: - - packages/core - - packages/datasets_pdf - - packages/adapters/collector_gh - - packages/services/normalize_issues - - apps/cli - steps: - - name: Checkout repository - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd - with: - persist-credentials: false - - - name: Set up Python - uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 - with: - python-version: '3.14' - cache: 'pip' - - - name: Install dependencies - run: | - pip install -e packages/core - pip install -e packages/datasets_pdf - pip install -e packages/adapters/collector_gh - pip install -e packages/services/normalize_issues - pip install -e "${{ matrix.package }}[dev]" - - - name: Check code format with Black - id: check-format - working-directory: ${{ matrix.package }} - run: | - black --check $(git ls-files '*.py') - - unit-test: - name: Unit Tests - runs-on: ubuntu-latest - strategy: - fail-fast: false - matrix: - package: - - packages/core - - packages/datasets_pdf - - packages/adapters/collector_gh - - packages/services/normalize_issues - - apps/cli - + name: QA (Black, Pylint, mypy, Pytest) defaults: run: shell: bash - steps: - - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd - with: - fetch-depth: 0 - persist-credentials: false - - - uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 - with: - python-version: '3.14' - cache: 'pip' - - - name: Install Python dependencies - run: | - pip install -e packages/core - pip install -e packages/datasets_pdf - pip install -e packages/adapters/collector_gh - pip install -e packages/services/normalize_issues - pip install -e "${{ matrix.package }}[dev]" - - - name: Check code coverage with Pytest - working-directory: ${{ matrix.package }} - run: pytest --cov=src -v tests/ --cov-fail-under=80 - - mypy-check: - runs-on: ubuntu-latest - name: Mypy Type Check - strategy: - fail-fast: false - matrix: - package: - - packages/core - - packages/datasets_pdf - - packages/adapters/collector_gh - - packages/services/normalize_issues - - apps/cli steps: - name: Checkout repository uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd with: + fetch-depth: 0 persist-credentials: false - name: Set up Python @@ -171,16 +41,13 @@ jobs: python-version: '3.14' cache: 'pip' - - name: Install dependencies + - name: Install Python dependencies run: | - pip install -e packages/core - pip install -e packages/datasets_pdf - pip install -e packages/adapters/collector_gh - pip install -e packages/services/normalize_issues - pip install -e "${{ matrix.package }}[dev]" + pip install -e packages/core[dev] + pip install -e packages/datasets_pdf[dev] + pip install -e packages/adapters/collector_gh[dev] + pip install -e packages/services/normalize_issues[dev] + pip install -e "apps/cli[dev]" - - name: Check types with Mypy - id: check-types - working-directory: ${{ matrix.package }} - run: | - mypy . + - name: Run QA checks (Black, Pylint, mypy, Pytest) + run: make py-qa diff --git a/DEVELOPER.md b/DEVELOPER.md index d076751..9d303b7 100644 --- a/DEVELOPER.md +++ b/DEVELOPER.md @@ -2,6 +2,7 @@ - [Get Started](#get-started) - [Monorepo Layout](#monorepo-layout) +- [QA Automation with Make](#qa-automation-with-make) - [Running Static Code Analysis](#running-static-code-analysis) - [Run Black Tool Locally](#run-black-tool-locally) - [Run mypy Tool Locally](#run-mypy-tool-locally) @@ -58,6 +59,69 @@ Each package has its own `pyproject.toml`, `src/` layout, and `tests/` directory All quality-gate commands below are designed to run **per-package** from within the package directory, matching how CI executes them. +## QA Automation with Make + +This project includes a `Makefile` to simplify running quality-assurance checks across all +packages. Use Make targets instead of manual loops. + +### Quick Start + +First, install all packages with dev dependencies: + +```shell +make install +``` + +Then run all QA checks (Black, Pylint, mypy, unit tests) on all packages: + +```shell +make py-qa +``` + +Show all available commands: + +```shell +make help +``` + +### Common Make Targets + +| Target | Purpose | +|--------|---------| +| `make py-qa` | Run all QA gates (Black, Pylint, mypy, tests) on all packages | +| `make black` | Format all packages with Black | +| `make pylint` | Lint all packages (score ≥ 9.5) | +| `make mypy` | Type-check all packages | +| `make pytest-unit` | Run unit tests (coverage ≥ 80%) on all packages | + +### Run QA for a Specific Package + +Run all QA checks on one package: + +```shell +make py-qa-core # packages/core +make py-qa-datasets-pdf # packages/datasets_pdf +make py-qa-collector-gh # packages/adapters/collector_gh +make py-qa-normalize # packages/services/normalize_issues +make py-qa-cli # apps/cli +``` + +Run a specific check on one package: + +```shell +make black-core # Black on packages/core +make pylint-packages/core # Pylint on packages/core +make mypy-packages/core # mypy on packages/core +make pytest-unit-packages/core # Tests on packages/core +``` + +### Quality Gates + +- **Black**: Line length 120 characters (configured in each `pyproject.toml`) +- **Pylint**: Minimum score 9.5 / 10.0 +- **mypy**: Strict type checking (configuration in each `pyproject.toml`) +- **pytest**: Minimum coverage 80% + ## Running Static Code Analysis This project uses [Pylint](https://pylint.readthedocs.io/) for static code analysis. @@ -112,10 +176,7 @@ black $(git ls-files '*.py') From the **repository root**: ```shell -for pkg in packages/core packages/datasets_pdf packages/adapters/collector_gh packages/services/normalize_issues apps/cli; do - echo "=== Black: $pkg ===" - (cd "$pkg" && black $(git ls-files '*.py')) -done +Fixed ``` ### Check-only mode (no changes) diff --git a/Makefile b/Makefile new file mode 100644 index 0000000..bb1ec0c --- /dev/null +++ b/Makefile @@ -0,0 +1,148 @@ +.PHONY: help black pylint mypy pytest-unit py-qa \ + black-all pylint-all mypy-all pytest-unit-all \ + install \ + $(patsubst %,black-%,$(PACKAGES) $(APPS)) \ + $(patsubst %,pylint-%,$(PACKAGES) $(APPS)) \ + $(patsubst %,mypy-%,$(PACKAGES) $(APPS)) \ + $(patsubst %,pytest-unit-%,$(PACKAGES) $(APPS)) + +# Packages and apps +PACKAGES := packages/core \ + packages/datasets_pdf \ + packages/adapters/collector_gh \ + packages/services/normalize_issues +APPS := apps/cli +ALL_TARGETS := $(PACKAGES) $(APPS) + +# Python and tools +PYTHON ?= python +MIN_PYLINT_SCORE := 9.5 +MIN_COVERAGE := 80 + +# Color output +GREEN := \033[0;32m +YELLOW := \033[0;33m +NC := \033[0m # No Color + +install: ## Install all packages with [dev] dependencies + @echo "$(YELLOW)Installing packages in dependency order...$(NC)" + $(PYTHON) -m pip install -e packages/core[dev] + $(PYTHON) -m pip install -e packages/datasets_pdf[dev] + $(PYTHON) -m pip install -e packages/adapters/collector_gh[dev] + $(PYTHON) -m pip install -e packages/services/normalize_issues[dev] + $(PYTHON) -m pip install -e "apps/cli[dev]" + @echo "$(GREEN)✓ All packages installed$(NC)" + +help: ## Show this help message + @echo "$(GREEN)Living Documentation Toolkit - QA Commands$(NC)" + @echo "" + @echo "$(YELLOW)Setup:$(NC)" + @echo " make install Install all packages with [dev] dependencies" + @echo "" + @echo "$(YELLOW)Run all QA checks:$(NC)" + @echo " make py-qa Run all QA gates on all packages" + @echo "" + @echo "$(YELLOW)Run QA by check type:$(NC)" + @echo " make black Run Black formatter on all packages" + @echo " make pylint Run Pylint on all packages" + @echo " make mypy Run mypy on all packages" + @echo " make pytest-unit Run unit tests on all packages" + @echo "" + @echo "$(YELLOW)Run QA by package:$(NC)" + @echo " make py-qa-core Run all QA on packages/core" + @echo " make py-qa-datasets-pdf Run all QA on packages/datasets_pdf" + @echo " make py-qa-collector-gh Run all QA on packages/adapters/collector_gh" + @echo " make py-qa-normalize Run all QA on packages/services/normalize_issues" + @echo " make py-qa-cli Run all QA on apps/cli" + @echo "" + @echo "$(YELLOW)Run individual checks by package:$(NC)" + @echo " make black-core Run Black on packages/core" + @echo " make pylint-core Run Pylint on packages/core" + @echo " make mypy-core Run mypy on packages/core" + @echo " make pytest-unit-core Run tests on packages/core" + @echo "" + @echo "$(YELLOW)Package shortcuts (replace 'core' with any package):$(NC)" + @echo " - datasets_pdf, collector_gh, normalize, cli" + +# ============================================================================ +# ALL PACKAGES: Aggregated QA targets +# ============================================================================ + +black: black-all ## Run Black on all packages + +pylint: pylint-all ## Run Pylint on all packages + +mypy: mypy-all ## Run mypy on all packages + +pytest-unit: pytest-unit-all ## Run unit tests on all packages + +py-qa: black pylint mypy pytest-unit ## Run all QA gates on all packages + @echo "$(GREEN)✓ All QA checks passed!$(NC)" + +# ============================================================================ +# Aggregated targets for all packages +# ============================================================================ + +black-all: $(patsubst %,black-%,$(ALL_TARGETS)) + @echo "$(GREEN)✓ Black formatting complete for all packages$(NC)" + +pylint-all: $(patsubst %,pylint-%,$(ALL_TARGETS)) + @echo "$(GREEN)✓ Pylint checks passed for all packages (score >= $(MIN_PYLINT_SCORE))$(NC)" + +mypy-all: $(patsubst %,mypy-%,$(ALL_TARGETS)) + @echo "$(GREEN)✓ mypy type checks passed for all packages$(NC)" + +pytest-unit-all: $(patsubst %,pytest-unit-%,$(ALL_TARGETS)) + @echo "$(GREEN)✓ Unit tests passed for all packages (coverage >= $(MIN_COVERAGE)%)$(NC)" + +# ============================================================================ +# PER-PACKAGE QA targets (all checks for one package) +# ============================================================================ + +py-qa-core: black-core pylint-core mypy-core pytest-unit-core + @echo "$(GREEN)✓ All QA checks passed for packages/core$(NC)" + +py-qa-datasets-pdf: black-packages/datasets_pdf pylint-packages/datasets_pdf mypy-packages/datasets_pdf pytest-unit-packages/datasets_pdf + @echo "$(GREEN)✓ All QA checks passed for packages/datasets_pdf$(NC)" + +py-qa-collector-gh: black-packages/adapters/collector_gh pylint-packages/adapters/collector_gh mypy-packages/adapters/collector_gh pytest-unit-packages/adapters/collector_gh + @echo "$(GREEN)✓ All QA checks passed for packages/adapters/collector_gh$(NC)" + +py-qa-normalize: black-packages/services/normalize_issues pylint-packages/services/normalize_issues mypy-packages/services/normalize_issues pytest-unit-packages/services/normalize_issues + @echo "$(GREEN)✓ All QA checks passed for packages/services/normalize_issues$(NC)" + +py-qa-cli: black-apps/cli pylint-apps/cli mypy-apps/cli pytest-unit-apps/cli + @echo "$(GREEN)✓ All QA checks passed for apps/cli$(NC)" + +# ============================================================================ +# BLACK FORMATTER +# ============================================================================ + +$(patsubst %,black-%,$(ALL_TARGETS)): black-%: + @echo "$(YELLOW)→ Black: $*$(NC)" + cd $* && $(PYTHON) -m black . + +# ============================================================================ +# PYLINT LINTER +# ============================================================================ + +$(patsubst %,pylint-%,$(ALL_TARGETS)): pylint-%: + @echo "$(YELLOW)→ Pylint: $* (threshold >= $(MIN_PYLINT_SCORE))$(NC)" + cd $* && $(PYTHON) -m pylint --fail-under=$(MIN_PYLINT_SCORE) $$(git ls-files '*.py' | grep -v '^tests/') + +# ============================================================================ +# MYPY TYPE CHECKER +# ============================================================================ + +$(patsubst %,mypy-%,$(ALL_TARGETS)): mypy-%: + @echo "$(YELLOW)→ mypy: $*$(NC)" + cd $* && $(PYTHON) -m mypy . + +# ============================================================================ +# PYTEST UNIT TESTS +# ============================================================================ + +$(patsubst %,pytest-unit-%,$(ALL_TARGETS)): pytest-unit-%: + @echo "$(YELLOW)→ Tests: $* (coverage >= $(MIN_COVERAGE)%)$(NC)" + cd $* && $(PYTHON) -m pytest tests/ --cov=src --cov-fail-under=$(MIN_COVERAGE) + diff --git a/README.md b/README.md index b5e59e4..9c68b8a 100644 --- a/README.md +++ b/README.md @@ -5,6 +5,16 @@ A monorepo hosting multiple independent Python services that transform and enrich machine-readable artifacts produced by upstream collectors (e.g., [living-doc-collector-gh](https://github.com/AbsaOSS/living-doc-collector-gh)) into datasets consumable by downstream actions (e.g., [living-doc-generator-pdf](https://github.com/AbsaOSS/living-doc-generator-pdf)). +--- + +- [Overview](#overview) +- [Quickstart](#quickstart) +- [Documentation](#documentation) +- [Services](#services) +- [License](#license) + +--- + ## Overview The Living Documentation Toolkit is a **generic builder** designed to: @@ -23,8 +33,13 @@ The Living Documentation Toolkit is a **generic builder** designed to: git clone https://github.com/AbsaOSS/living-doc-toolkit.git cd living-doc-toolkit -# Install dependencies (once available) -pip install -e . +# Create and activate a virtual environment +python3 -m venv .venv +source .venv/bin/activate + +# Install all packages +pip install --upgrade pip +pip install -r requirements.txt ``` ### Example CLI Usage @@ -41,15 +56,28 @@ living-doc normalize-issues \ ## Documentation -- **[SPEC.md](SPEC.md)** - System specification and architecture details -- **[TASKS.md](TASKS.md)** - Implementation roadmap and task tracking -- **[CHANGELOG.md](CHANGELOG.md)** - Version history and notable changes +### Understand +- **[Architecture](docs/architecture.md)** — System overview, data flow pipeline, package structure +- **[Contracts & Interfaces](docs/contracts.md)** — CLI reference, input/output schemas, audit envelope, change control + +### Use + +Each service below has a **Cookbook** (explains *how* it works — detection logic, compatibility rules, normalization behavior) and **Recipes** (step-by-step guides to run it in a specific environment). See [Services](#services) for the full list. + +### Maintain +- **[Troubleshooting](docs/troubleshooting.md)** — Exit codes, common errors, FAQ +- **[Developer Guide](DEVELOPER.md)** — Environment setup, testing, linting, branch conventions +- **[Changelog](CHANGELOG.md)** — Version history and notable changes ## Services ### `normalize-issues` Converts collector output (`doc-issues.json`) into PDF-ready canonical JSON (`pdf_ready.json`) compliant with the PDF generator specification. +- [Cookbook](docs/cookbooks/normalize-issues.md) — How detection, compatibility, and normalization work +- [Recipe: Local usage](docs/recipes/local-normalize-issues.md) — Run the CLI on your machine +- [Recipe: GitHub Actions](docs/recipes/github-actions-normalize-issues.md) — CI/CD workflow integration + ## License Licensed under the Apache License, Version 2.0. See [LICENSE](LICENSE) for full details. diff --git a/SPEC.md b/SPEC.md deleted file mode 100644 index 7a42801..0000000 --- a/SPEC.md +++ /dev/null @@ -1,1016 +0,0 @@ -# Living Documentation Toolkit — System Specification - -**Version:** 1.0 -**Last Updated:** 2026-01-23 -**Status:** Active -**Python Runtime:** 3.14 - ---- - -## 1. Overview & Scope - -### 1.1 Purpose - -The Living Documentation Toolkit is a **generic builder** hosting multiple independent Python services that transform and enrich machine-readable artifacts produced by upstream collectors (e.g., `AbsaOSS/living-doc-collector-gh`) into datasets consumable by downstream actions (e.g., `AbsaOSS/living-doc-generator-pdf`). - -It is designed as a **monorepo** containing: -- **Multiple services**: Independent, runnable entrypoints (pipelines/capabilities) -- **Adapters**: Input producer detection and parsing (collector action compatibility) -- **Reusable core utilities**: Shared helpers used by all services -- **Versioned contracts**: JSON Schema + Pydantic models for dataset validation -- **Cookbooks and recipes**: Ready-to-use workflow examples for GitHub Actions - -### 1.2 Scope - -**Current Scope (Service #1):** -- **Service:** `normalize-issues` -- **Goal:** Convert collector output `doc-issues.json` into PDF-ready canonical JSON (`pdf_ready.json`) compliant with `AbsaOSS/living-doc-generator-pdf` SPEC v1.0. - -**In Scope:** -- Adapter-based input detection and version compatibility checking -- Markdown normalization into canonical `sections.*` fields -- Enterprise-grade audit trail preservation and augmentation -- JSON Schema validation and Pydantic model enforcement -- CLI wrapper exposing multiple services via subcommands -- Fixture-based verification and golden file testing -- GitHub Actions CI/CD with static analysis and test coverage - -**Out of Scope:** -- PDF rendering (belongs to `living-doc-generator-pdf`) -- Source collection and labeling (belongs to collectors) -- "Smart" content generation (analytics, coverage matrices) unless it becomes a new service - -### 1.3 Design Principles - -1. **Adapter-Driven Input Detection**: Detect producer action by metadata, not file format -2. **Fail-Safe Compatibility**: Warn and attempt when producer version is outside confirmed range -3. **Audit Transparency**: Preserve enterprise-level audit; augment with builder trace -4. **Schema Stability**: Canonical datasets follow versioned contracts -5. **Deterministic Output**: Same input always produces same output (excluding timestamps) -6. **Modular Architecture**: Clear separation of concerns (services, adapters, core, contracts) - ---- - -## 2. Glossary & Invariants - -### 2.1 Key Terms - -- **Service**: An independent, runnable entrypoint that orchestrates a complete transformation pipeline -- **Adapter**: A module that detects and parses input from a specific producer action (e.g., `collector-gh`) -- **Producer**: An upstream action that generates input artifacts (e.g., `AbsaOSS/living-doc-collector-gh`) -- **Dataset Contract**: A versioned JSON schema + Pydantic model defining canonical output structure -- **Audit Envelope**: Standardized metadata structure preserving provenance and transformation trace -- **Canonical JSON**: Normalized, schema-validated JSON ready for downstream consumption -- **Fixture**: Test input file (e.g., `doc-issues.json`) used for verification and golden file testing -- **Golden File**: Expected output file used for regression testing - -### 2.2 Invariants - -1. **Adapter Independence**: Services depend on adapters via stable `AdapterResult` interface, not implementation details -2. **Schema Version Stability**: Once published, schema version 1.0 field meanings never change -3. **Audit Preservation**: Original producer audit metadata must never be lost or corrupted -4. **Compatibility Transparency**: Version mismatches produce warnings, not failures (unless critically incompatible) -5. **Output Determinism**: Same input JSON produces same output structure (timestamps excluded) - ---- - -## 3. Interfaces & Contracts - -### 3.1 CLI Interface - -#### 3.1.1 CLI Name - -**Command:** `living-doc` - -#### 3.1.2 Service: `normalize-issues` - -**Command:** -```bash -living-doc normalize-issues \ - --input \ - --output \ - [--source auto|collector-gh] \ - [--document-title ] \ - [--document-version ] \ - [--verbose] -``` - -**Arguments:** - -| Argument | Type | Required | Default | Description | -|----------|------|----------|---------|-------------| -| `--input` | path | **Yes** | - | Path to input JSON file (e.g., `doc-issues.json`) | -| `--output` | path | **Yes** | - | Path for output JSON file (e.g., `pdf_ready.json`) | -| `--source` | enum | No | `auto` | Producer adapter selection (`auto`, `collector-gh`) | -| `--document-title` | string | No | _(from input)_ | Override document title in `meta.document_title` | -| `--document-version` | string | No | _(from input)_ | Override document version in `meta.document_version` | -| `--verbose` | flag | No | `false` | Enable verbose logging | - -**Exit Codes:** - -| Exit Code | Condition | Error Message Prefix | -|-----------|-----------|---------------------| -| 0 | Success | - | -| 1 | Invalid input (missing file, malformed JSON) | `Invalid input:` | -| 2 | Adapter detection failed | `Adapter error:` | -| 3 | Schema validation failure (output) | `Schema validation failed:` | -| 4 | Normalization error (parsing failure) | `Normalization failed:` | -| 5 | File I/O error (write failure) | `File I/O error:` | - -**Error Message Format:** -``` -{prefix} {specific_detail}. {actionable_guidance} -``` - -**Examples:** -``` -Invalid input: File 'doc-issues.json' not found. Ensure --input points to a valid file. -Adapter error: No compatible adapter found for input. Check metadata.generator.name field. -Schema validation failed: Missing required field 'schema_version' in output. This is a builder bug. -Normalization failed: Unable to parse markdown in issue #42 body. Check for malformed content. -``` - -#### 3.1.3 Verification Commands (Future) - -**Command:** -```bash -living-doc verify normalize-issues --fixture -``` - -**Purpose:** Run verification against golden fixtures for manual testing and CI validation. - -### 3.2 Input Contract: `doc-issues.json` (Collector-GH) - -#### 3.2.1 Producer Detection - -**Detection Logic:** -```python -if payload.get("metadata", {}).get("generator", {}).get("name") == "AbsaOSS/living-doc-collector-gh": - adapter = CollectorGhAdapter() -``` - -**Required Fields for Detection:** -- `metadata.generator.name` (string, non-empty) -- `metadata.generator.version` (string, semver format) - -#### 3.2.2 Compatibility Policy - -**Adapter:** `collector_gh` - -**Confirmed Compatible Range:** `>=1.0.0,<2.0.0` - -**Behavior:** -- **Within Range:** Proceed silently -- **Outside Range:** Log warning, add to `audit.trace[].warnings[]`, attempt processing with newest logic -- **Critically Incompatible:** Exit with `Adapter error:` if schema is unrecognizable - -**Warning Format:** -```json -{ - "code": "VERSION_MISMATCH", - "message": "Producer version 2.1.0 is outside confirmed range >=1.0.0,<2.0.0", - "context": "metadata.generator.version" -} -``` - -#### 3.2.3 Input Schema (Collector-GH v1.x) - -**Example Structure (Minimal):** -```json -{ - "metadata": { - "generator": { - "name": "AbsaOSS/living-doc-collector-gh", - "version": "1.2.0", - "build": "abc123" - }, - "run": { - "run_id": "123456", - "run_attempt": "1", - "actor": "user@example.com", - "workflow": "collect-docs", - "ref": "refs/heads/main", - "sha": "abc123def456" - }, - "source": { - "systems": ["GitHub"], - "repositories": ["AbsaOSS/project"], - "organization": "AbsaOSS", - "enterprise": null - } - }, - "issues": [ - { - "number": 42, - "title": "User login with SSO", - "state": "open", - "labels": ["documentation", "priority:high"], - "html_url": "https://github.com/AbsaOSS/project/issues/42", - "created_at": "2026-01-10T08:00:00Z", - "updated_at": "2026-01-20T14:30:00Z", - "body": "## Description\nAs a user, I want to..." - } - ] -} -``` - -### 3.3 Output Contract: `pdf_ready.json` (Generator-PDF v1.0) - -**Reference:** [`AbsaOSS/living-doc-generator-pdf` SPEC.md](https://github.com/AbsaOSS/living-doc-generator-pdf/blob/master/SPEC.md) - -#### 3.3.1 Schema Version - -**Field:** `schema_version` -**Type:** String -**Required:** Yes -**Allowed Values:** `"1.0"` - -#### 3.3.2 Metadata Section - -**Field:** `meta` -**Type:** Object -**Required:** Yes - -**Required Fields:** -```json -{ - "document_title": "string (non-empty, 1-200 chars)", - "document_version": "string (non-empty, 1-50 chars, semver recommended)", - "generated_at": "ISO 8601 UTC timestamp", - "source_set": ["array of source identifiers, non-empty"], - "selection_summary": { - "total_items": "integer >= 0", - "included_items": "integer >= 0", - "excluded_items": "integer >= 0" - } -} -``` - -**Optional Fields:** -```json -{ - "run_context": { - "ci_run_id": "string", - "triggered_by": "string", - "branch": "string", - "commit_sha": "string" - }, - "audit": { - "schema_version": "string", - "producer": {}, - "run": {}, - "source": {}, - "trace": [], - "extensions": {} - } -} -``` - -#### 3.3.3 Content Section - -**Field:** `content` -**Type:** Object -**Required:** Yes - -**Required Fields:** -```json -{ - "user_stories": [ - { - "id": "string (canonical stable ID, e.g., 'github:owner/repo#123')", - "title": "string (non-empty, 1-500 chars)", - "state": "string (e.g., 'open', 'closed')", - "tags": ["array of strings"], - "url": "string (valid URL)", - "timestamps": { - "created": "ISO 8601 timestamp", - "updated": "ISO 8601 timestamp" - }, - "sections": { - "description": "string (Markdown)", - "business_value": "string (Markdown, optional)", - "preconditions": "string (Markdown, optional)", - "acceptance_criteria": "string (Markdown, optional)", - "user_guide": "string (Markdown, optional)", - "connections": "string (Markdown, optional)", - "last_edited": "string (Markdown, optional)" - } - } - ] -} -``` - -**Section Mapping Rules:** -- All headings in issue body are considered; mapping must be deterministic -- Known heading synonyms map to canonical keys (case-insensitive): - - `Description`, `Overview`, `Summary` → `description` - - `Business Value`, `Value`, `Why` → `business_value` - - `Preconditions`, `Prerequisites`, `Setup` → `preconditions` - - `Acceptance Criteria`, `AC`, `Done Criteria` → `acceptance_criteria` - - `User Guide`, `How To`, `Instructions` → `user_guide` - - `Connections`, `Related`, `Links` → `connections` - - `Last Edited`, `History`, `Changes` → `last_edited` -- Unknown headings: Append to `description` in structured form (e.g., `### Unknown Heading\n{content}`) -- Missing sections: Represented as `null` or empty string -- Preserve Markdown formatting (tables, links, lists) for template rendering - -#### 3.3.4 Output Example - -```json -{ - "schema_version": "1.0", - "meta": { - "document_title": "Product Requirements - Release 2.1", - "document_version": "2.1.0", - "generated_at": "2026-01-23T12:00:00Z", - "source_set": ["github:AbsaOSS/project"], - "selection_summary": { - "total_items": 15, - "included_items": 12, - "excluded_items": 3 - }, - "audit": { - "schema_version": "1.0", - "producer": { - "name": "AbsaOSS/living-doc-collector-gh", - "version": "1.2.0", - "build": "abc123" - }, - "run": { - "run_id": "123456", - "run_attempt": "1", - "actor": "user@example.com", - "workflow": "collect-docs", - "ref": "refs/heads/main", - "sha": "abc123def456" - }, - "source": { - "systems": ["GitHub"], - "repositories": ["AbsaOSS/project"], - "organization": "AbsaOSS", - "enterprise": null - }, - "trace": [ - { - "step": "collection", - "tool": "living-doc-collector-gh", - "tool_version": "1.2.0", - "started_at": "2026-01-23T11:50:00Z", - "finished_at": "2026-01-23T11:55:00Z", - "warnings": [] - }, - { - "step": "normalization", - "tool": "living-doc-toolkit", - "tool_version": "0.1.0", - "started_at": "2026-01-23T12:00:00Z", - "finished_at": "2026-01-23T12:00:05Z", - "warnings": [] - } - ], - "extensions": { - "collector-gh": { - "original_metadata": {} - } - } - } - }, - "content": { - "user_stories": [ - { - "id": "github:AbsaOSS/project#42", - "title": "User login with SSO", - "state": "open", - "tags": ["authentication", "priority:high"], - "url": "https://github.com/AbsaOSS/project/issues/42", - "timestamps": { - "created": "2026-01-10T08:00:00Z", - "updated": "2026-01-20T14:30:00Z" - }, - "sections": { - "description": "As a user, I want to log in using SSO...", - "business_value": "Reduces friction for enterprise users", - "preconditions": "SSO provider configured", - "acceptance_criteria": "- User can click SSO button\n- Redirect to provider\n- Return with session", - "user_guide": null, - "connections": "Related to #41, #43", - "last_edited": "Updated by alice@example.com on 2026-01-20" - } - } - ] - } -} -``` - -### 3.4 Audit Envelope Standard (v1.0) - -#### 3.4.1 Purpose - -Enterprise-grade audit trail that: -- Preserves collector-level provenance and metadata -- Tracks builder transformation steps -- Records warnings and compatibility issues -- Supports extensibility via namespaced extensions - -#### 3.4.2 Schema Structure - -**Field:** `meta.audit` (optional in PDF-ready contract) -**Type:** Object - -**Required Fields:** -```json -{ - "schema_version": "string (fixed: '1.0')", - "producer": { - "name": "string (required, e.g., 'AbsaOSS/living-doc-collector-gh')", - "version": "string (required, semver)", - "build": "string | null" - }, - "run": { - "run_id": "string | null", - "run_attempt": "string | null", - "actor": "string | null", - "workflow": "string | null", - "ref": "string | null", - "sha": "string | null" - }, - "source": { - "systems": ["array of strings, non-empty"], - "repositories": ["array of strings"], - "organization": "string | null", - "enterprise": "string | null" - }, - "trace": [ - { - "step": "string (e.g., 'collection', 'normalization')", - "tool": "string (e.g., 'living-doc-collector-gh')", - "tool_version": "string (semver)", - "started_at": "ISO 8601 timestamp | null", - "finished_at": "ISO 8601 timestamp | null", - "warnings": [ - { - "code": "string (e.g., 'VERSION_MISMATCH')", - "message": "string (human-readable)", - "context": "string | null (e.g., field path)" - } - ] - } - ], - "extensions": { - "collector-gh": {}, - "builder": {}, - "absa.enterprise": {} - } -} -``` - -**Validation Rules:** -- `schema_version`: Required, fixed `"1.0"` -- `producer.name`: Required, non-empty string -- `source.systems`: Required, non-empty array -- `extensions`: Keys must be namespaced strings; values must be JSON objects - -#### 3.4.3 Mapping Collector-GH Metadata - -**Adapter Responsibility:** -- Map `metadata.generator.*` → `audit.producer.*` -- Map `metadata.run.*` → `audit.run.*` -- Map `metadata.source.*` → `audit.source.*` -- Store full original `metadata` in `audit.extensions["collector-gh"].original_metadata` - -**Service Responsibility:** -- Append trace step for normalization: - ```json - { - "step": "normalization", - "tool": "living-doc-toolkit", - "tool_version": "{version}", - "started_at": "{iso_timestamp}", - "finished_at": "{iso_timestamp}", - "warnings": [] - } - ``` - ---- - -## 4. Data & Storage Schemas - -### 4.1 Repository Layout (Final Structure) - -``` -repo-root/ - README.md - SPEC.md # This document - TASKS.md # Implementation roadmap - LICENSE - pyproject.toml # Shared tooling config (ruff/black/mypy) - - .github/ - workflows/ - ci.yml # Static analysis and tests - integration.yml # Fixture-based verification - release.yml # Release automation - - docs/ - cookbooks/ # Explainers and guides - normalize-issues.md - recipes/ # Ready-to-use examples - local-normalize-issues.md - github-actions-normalize-issues.yml - - packages/ - datasets_pdf/ # Dataset contract for PDF generator - pyproject.toml - src/living_doc_datasets_pdf/ - __init__.py - pdf_ready/ - v1/ - __init__.py - models.py # Pydantic models - schema.py # Schema export helpers - audit/ - v1/ - __init__.py - models.py - schemas/ - pdf_ready_v1.schema.json - audit_envelope_v1.schema.json - tests/ - test_pdf_ready_models.py - test_audit_models.py - - core/ # Shared utilities - pyproject.toml - src/living_doc_core/ - __init__.py - json_utils.py # JSON read/write helpers - logging_config.py # Logging setup - markdown_utils.py # Markdown parsing primitives - errors.py # Common error types - tests/ - test_json_utils.py - test_markdown_utils.py - - adapters/ - collector_gh/ # Collector-GH adapter - pyproject.toml - src/living_doc_adapter_collector_gh/ - __init__.py - detector.py # Producer detection - parser.py # Input parsing - models.py # AdapterResult models - tests/ - test_detector.py - test_parser.py - fixtures/ - collector_v1.0.0/ - input/ - doc-issues.json - - services/ - normalize_issues/ # Normalize-issues service - pyproject.toml - src/living_doc_service_normalize_issues/ - __init__.py - service.py # Main service logic - normalizer.py # Markdown normalization - builder.py # PDF-ready JSON builder - tests/ - test_service.py - test_normalizer.py - test_builder.py - verifications/ # Manual verification scripts - verify_golden.py - verify_compatibility.py - - apps/ - cli/ # CLI wrapper - pyproject.toml - src/living_doc_cli/ - __init__.py - main.py # CLI entrypoint - commands/ - normalize_issues.py - tests/ - test_cli.py - - outputs/ # Gitignored runtime artifacts - pdf_ready.json - reports/ -``` - -### 4.2 Output Artifact Location - -**Directory:** `outputs/` (gitignored) - -**Artifacts:** -- `outputs/pdf_ready.json` - Canonical output -- `outputs/reports/` - Service-specific diagnostic reports (optional) - ---- - -## 5. Algorithms & Rules - -### 5.1 Service Pipeline: `normalize-issues` - -``` -1. Load and parse input JSON -2. Detect producer adapter (auto or explicit) -3. Check version compatibility (warn if outside range) -4. Parse input into AdapterResult -5. Normalize markdown sections -6. Build pdf_ready.json structure -7. Augment audit envelope with builder trace -8. Validate output against schema -9. Write output file -10. Log summary and warnings -``` - -### 5.2 Adapter Selection Logic - -**Mode: `auto`** -```python -for adapter in registered_adapters: - if adapter.can_handle(payload): - return adapter -raise AdapterNotFoundError("No compatible adapter found") -``` - -**Mode: `collector-gh`** -```python -return CollectorGhAdapter() -``` - -### 5.3 Markdown Normalization Rules - -**Input:** Raw markdown string (issue body) -**Output:** Dictionary of section keys → markdown content - -**Algorithm:** -1. Split markdown by `##` headings (H2 level) -2. For each heading: - - Normalize heading text (lowercase, strip whitespace) - - Match against known synonyms (see 3.3.3) - - If match: Assign content to canonical section key - - If no match: Append to `description` as structured content -3. Preserve Markdown formatting (tables, lists, links) -4. Handle edge cases: - - Empty sections: `null` or `""` - - Content before first heading: Assign to `description` - - Multiple occurrences of same heading: Concatenate with separator - -**Determinism:** -- Same markdown input always produces same section mapping -- Order of sections in output dictionary is stable (alphabetical) - -### 5.4 Compatibility Warning Logic - -**Trigger:** Producer version outside confirmed range - -**Action:** -1. Log warning at `WARNING` level -2. Add to `audit.trace[].warnings[]`: - ```json - { - "code": "VERSION_MISMATCH", - "message": "Producer version X.Y.Z is outside confirmed range A.B.C-D.E.F", - "context": "metadata.generator.version" - } - ``` -3. Continue processing with newest adapter logic -4. If parsing fails: Exit with `Adapter error:` - -### 5.5 Performance Budgets - -| Operation | Target | Maximum | -|-----------|--------|---------| -| JSON parsing (10 MB) | < 1 second | 5 seconds | -| Adapter detection | < 0.1 seconds | 1 second | -| Markdown normalization (100 issues) | < 2 seconds | 10 seconds | -| Output validation | < 1 second | 5 seconds | -| Total service runtime (100 issues) | < 10 seconds | 30 seconds | - -**Note:** Budgets assume typical collector-gh output (~500 KB - 10 MB JSON) - ---- - -## 6. Phase-by-Phase Acceptance Criteria - -### Phase 1: Repository Structure & Scaffolding - -**Acceptance Criteria:** -- [ ] Monorepo directory structure created (packages/, apps/, docs/, outputs/) -- [ ] Root `pyproject.toml` configured with black, mypy, pytest settings -- [ ] `.gitignore` configured (outputs/, `__pycache__`, `.pytest_cache`, etc.) -- [ ] `README.md` updated with project overview and quickstart -- [ ] License file (Apache 2.0) present -- [ ] CODEOWNERS file configured - -**Verification:** -- Directory structure matches SPEC section 4.1 -- `pyproject.toml` lint checks pass (black --check, mypy .) - -### Phase 2: Dataset/Contract Package (`datasets_pdf`) - -**Acceptance Criteria:** -- [ ] Pydantic models for `pdf_ready` v1.0 created -- [ ] Pydantic models for `audit` v1.0 created -- [ ] JSON Schema files exported and validated -- [ ] Serialization helpers (deterministic JSON output) -- [ ] Unit tests for model validation (valid/invalid inputs) -- [ ] Test coverage ≥ 80% - -**Verification:** -- `tests/packages/datasets_pdf/test_pdf_ready_models.py` -- `tests/packages/datasets_pdf/test_audit_models.py` -- JSON schemas validate correctly against reference examples - -### Phase 3: Core Utilities Package - -**Acceptance Criteria:** -- [ ] JSON read/write helpers implemented -- [ ] Logging configuration module created -- [ ] Markdown parsing primitives (heading splitter, list parser) -- [ ] Common error types defined -- [ ] Unit tests for all utilities -- [ ] Test coverage ≥ 80% - -**Verification:** -- `tests/packages/core/test_json_utils.py` -- `tests/packages/core/test_markdown_utils.py` -- Utilities are reusable and do not encode service-specific logic - -### Phase 4: Adapter Package (`collector_gh`) - -**Acceptance Criteria:** -- [ ] Producer detection logic implemented -- [ ] Version extraction and compatibility checking -- [ ] Input parsing into `AdapterResult` model -- [ ] Audit metadata mapping (metadata → audit envelope) -- [ ] Unit tests with fixture files (collector v1.0.0, v1.2.0) -- [ ] Test coverage ≥ 80% - -**Verification:** -- `tests/packages/adapters/collector_gh/test_detector.py` -- `tests/packages/adapters/collector_gh/test_parser.py` -- Fixture files in `tests/packages/adapters/collector_gh/fixtures/` - -### Phase 5: Service Package (`normalize_issues`) - -**Acceptance Criteria:** -- [ ] Service orchestration logic implemented -- [ ] Markdown normalization with heading mapping -- [ ] PDF-ready JSON builder with audit augmentation -- [ ] Output schema validation -- [ ] Unit tests for normalization rules -- [ ] Integration tests with golden files -- [ ] Test coverage ≥ 80% - -**Verification:** -- `tests/packages/services/normalize_issues/test_service.py` -- `tests/packages/services/normalize_issues/test_normalizer.py` -- Golden file comparison tests - -### Phase 6: CLI Wrapper - -**Acceptance Criteria:** -- [ ] CLI entrypoint (`living-doc`) implemented -- [ ] `normalize-issues` subcommand with argument parsing -- [ ] Exit codes match specification -- [ ] Error messages follow format (prefix + detail + guidance) -- [ ] Verbose logging option functional -- [ ] Unit tests for CLI argument parsing -- [ ] Integration tests for CLI invocation - -**Verification:** -- `tests/apps/cli/test_cli.py` -- Manual CLI invocation tests -- Help text (`--help`) is clear and accurate - -### Phase 7: Python CI/CD Workflow - -**Acceptance Criteria:** -- [ ] CI workflow configured (`.github/workflows/ci.yml`) -- [ ] Pylint analysis with score threshold ≥ 9.5 -- [ ] Black format check -- [ ] Mypy type checking -- [ ] Pytest with coverage ≥ 80% -- [ ] Python 3.14 pinned in workflows -- [ ] Workflows modeled after `living-doc-collector-gh` - -**Verification:** -- CI passes on all commits -- All quality gates enforced automatically - -### Phase 8: Fixture-Based Verification - -**Acceptance Criteria:** -- [ ] Golden fixtures added (collector v1.0.0, v1.2.0) -- [ ] Manual verification scripts (`verifications/verify_golden.py`) -- [ ] Integration test workflow (`.github/workflows/integration.yml`) -- [ ] Verification scripts runnable in CI -- [ ] Compatibility warnings tested with out-of-range versions - -**Verification:** -- `verifications/verify_golden.py` runs successfully -- Integration tests pass with multiple collector versions - -### Phase 9: Cookbooks & Recipes - -**Acceptance Criteria:** -- [ ] Cookbook: `docs/cookbooks/normalize-issues.md` written -- [ ] Recipe: `docs/recipes/local-normalize-issues.md` created -- [ ] Recipe: `docs/recipes/github-actions-normalize-issues.yml` created -- [ ] All recipes tested and verified -- [ ] Documentation includes examples of: - - Local CLI usage - - GitHub Actions workflow integration - - Chaining collector → builder → generator - -**Verification:** -- Recipes execute successfully in local and CI environments -- Documentation is clear and includes expected outputs - -### Phase 10: Release Preparation - -**Acceptance Criteria:** -- [ ] `CHANGELOG.md` created with v0.1.0 notes -- [ ] Version number set in all `pyproject.toml` files -- [ ] Release workflow configured (`.github/workflows/release.yml`) -- [ ] GitHub Release draft created -- [ ] All tests passing -- [ ] Documentation reviewed and complete - -**Verification:** -- Release checklist completed -- Version tags consistent across repository -- Release notes include breaking changes and migration guide - ---- - -## 7. Change Control - -### 7.1 Versioned Contracts - -**Schema Version 1.0 (Immutable):** -- Field names, types, and meanings are **stable** -- New optional fields may be added (backwards compatible) -- Breaking changes require new schema version (2.0) - -**Adapter Interface (Stable):** -- `AdapterResult` model signature is stable -- New fields may be added as optional -- Removing fields requires major version bump - -**CLI Interface (Stable):** -- Argument names and defaults are stable -- New optional arguments may be added -- Removing arguments requires major version bump - -### 7.2 Approval Requirements - -**Changes Requiring Review:** -- Schema modifications (datasets_pdf contract) -- Adapter interface changes -- CLI argument changes -- Error message text changes -- Exit code changes -- Performance budget changes - -**Changes Not Requiring Special Approval:** -- Internal refactoring (preserving behavior) -- Test additions -- Documentation updates -- Logging improvements - ---- - -## 8. Testing Strategy - -### 8.1 Unit Tests - -**Coverage Target:** ≥ 80% - -**Focus:** -- JSON schema validation -- Markdown normalization rules -- Adapter detection and parsing -- Pydantic model validation -- Error handling and messages - -**Tools:** -- `pytest` (testing framework) -- `pytest-cov` (coverage reporting) - -### 8.2 Integration Tests - -**Focus:** -- End-to-end service execution -- Golden file comparison -- Fixture-based compatibility testing -- CLI invocation and exit codes - -### 8.3 Verification Scripts - -**Purpose:** Manual validation and CI quality gates - -**Scripts:** -- `verifications/verify_golden.py` - Golden file comparison -- `verifications/verify_compatibility.py` - Version compatibility checks -- `verifications/verify_schema.py` - Schema validation against examples - ---- - -## 9. Dependencies - -### 9.1 Runtime Dependencies - -| Package | Version | Purpose | -|---------|---------|---------| -| `pydantic` | `^2.10.0` | Data validation and models | -| `jsonschema` | `^4.20.0` | JSON schema validation | -| `click` | `^8.1.0` | CLI framework | - -### 9.2 Development Dependencies - -| Package | Version | Purpose | -|---------|---------|---------| -| `pytest` | `^9.0.0` | Testing framework | -| `pytest-cov` | `^7.0.0` | Coverage reporting | -| `black` | `^26.0.0` | Code formatting | -| `pylint` | `^4.0.0` | Code quality | -| `mypy` | `^1.19.0` | Type checking | - -**Note:** Version constraints follow patterns from `living-doc-collector-gh` - ---- - -## 10. Security Considerations - -### 10.1 Input Validation - -- **JSON Injection:** All inputs validated against strict schemas -- **Path Traversal:** File paths validated to prevent directory traversal -- **Resource Exhaustion:** File size limits enforced (50 MB soft limit) - -### 10.2 Secrets - -- **No Secrets in JSON:** Input/output files must not contain secrets or credentials -- **No API Calls:** Toolkit operates offline (no network calls) - ---- - -## 11. Monitoring & Observability - -### 11.1 Logging Levels - -| Level | Purpose | Examples | -|-------|---------|----------| -| DEBUG | Detailed diagnostics | "Detected adapter: collector-gh", "Parsed 12 issues" | -| INFO | Normal operation | "Normalization complete", "Output written to pdf_ready.json" | -| WARNING | Non-fatal issues | "Producer version outside confirmed range" | -| ERROR | Fatal failures | "Adapter detection failed", "Schema validation failed" | - -### 11.2 Audit Trail - -All transformations tracked in `meta.audit.trace[]`: -- Step name (e.g., "normalization") -- Tool name and version -- Start/finish timestamps -- Warnings (compatibility, parsing issues) - ---- - -## 12. Future Extensions - -- **Adapter Registry:** Plugin system for new producer adapters -- **Normalizer Plugins:** Custom markdown parsing rules -- **Output Validators:** Pluggable schema validators for different versions - ---- - -## 13. References - -### 13.1 Related Specifications - -- [Living Documentation PDF Generator SPEC v1.0](https://github.com/AbsaOSS/living-doc-generator-pdf/blob/master/SPEC.md) -- Living Documentation Toolkit Issue #1 (Draft Specification) - -### 13.2 External Standards - -- [JSON Schema Specification](https://json-schema.org/) -- [Pydantic Documentation](https://docs.pydantic.dev/) -- [Semantic Versioning](https://semver.org/) -- [ISO 8601 Date/Time Format](https://www.iso.org/iso-8601-date-and-time-format.html) - ---- - -## Document History - -| Version | Date | Author | Changes | -|---------|------|--------|---------| -| 1.0 | 2026-01-23 | Specification Master | Initial specification based on draft and references | - ---- - -**End of Specification** diff --git a/TASKS.md b/TASKS.md deleted file mode 100644 index 1d70acc..0000000 --- a/TASKS.md +++ /dev/null @@ -1,707 +0,0 @@ -# Living Documentation Toolkit — Implementation Roadmap - -**Version:** 1.0 -**Last Updated:** 2026-01-23 -**Status:** Active - ---- - -## Purpose - -This document provides an actionable, phase-by-phase implementation roadmap for the Living Documentation Toolkit. Each task includes checkboxes for tracking progress, and all tasks reference specific tools, tests, and quality gates. - -**Quality Standards:** -- All Python code must pass Pylint (≥ 9.5), Black formatting, and Mypy type checking -- All features must have unit tests with ≥ 80% coverage -- All services must have integration tests and verification scripts -- All changes must be documented in CHANGELOG.md - ---- - -## Phase 1: Repository Structure & Scaffolding - -**Goal:** Set up monorepo directory structure and core configuration files. - -### Tasks - -- [ ] **1.1 Create monorepo directory structure** - - Create `packages/`, `apps/`, `docs/`, `outputs/` directories - - Create subdirectories: `packages/datasets_pdf/`, `packages/core/`, `packages/adapters/collector_gh/`, `packages/services/normalize_issues/`, `apps/cli/` - - Create `docs/cookbooks/`, `docs/recipes/` - - **Verification:** Directory structure matches SPEC.md section 4.1 - -- [ ] **1.2 Configure root `pyproject.toml`** - - Add `[tool.black]` with `line-length = 120`, `target-version = ['py314']` - - Add `[tool.mypy]` with `check_untyped_defs = true`, `python_version = "3.14"` - - Add `[tool.coverage.run]` with `omit = ["tests/*"]` - - Add `[tool.pytest.ini_options]` with coverage thresholds - - **Reference:** `living-doc-collector-gh/pyproject.toml` - - **Verification:** Run `black --check .`, `mypy .` (should pass with no errors) - -- [ ] **1.3 Configure `.gitignore`** - - Add `outputs/`, `__pycache__/`, `.pytest_cache/`, `.mypy_cache/`, `.coverage`, `*.egg-info/`, `dist/`, `build/` - - Add `.venv/`, `.env`, `.DS_Store` - - **Verification:** No ignored files tracked in Git - -- [ ] **1.4 Update `README.md`** - - Add project overview and purpose - - Add quickstart installation instructions - - Add link to SPEC.md and TASKS.md - - Add example CLI usage - - **Verification:** README renders correctly on GitHub - -- [ ] **1.5 Verify Apache 2.0 License** - - Ensure `LICENSE` file is present and correct - - Add copyright headers to all Python files (template) - - **Reference:** `living-doc-collector-gh` license headers - -- [ ] **1.6 Create `CHANGELOG.md`** - - Add template with `## [Unreleased]` section - - Add sections: `Added`, `Changed`, `Fixed`, `Removed` - - Follow [Keep a Changelog](https://keepachangelog.com/) format - ---- - -## Phase 2: Dataset/Contract Package (`packages/datasets_pdf`) - -**Goal:** Create versioned Pydantic models and JSON schemas for `pdf_ready` and `audit` contracts. - -### Tasks - -- [ ] **2.1 Set up package structure** - - Create `packages/datasets_pdf/pyproject.toml` - - Define package as `living-doc-datasets-pdf` - - Add dependencies: `pydantic ^2.10.0`, `jsonschema ^4.20.0` - - Create `src/living_doc_datasets_pdf/__init__.py` - - **Verification:** Package installable with `pip install -e packages/datasets_pdf` - -- [ ] **2.2 Create `pdf_ready` v1.0 Pydantic models** - - Create `src/living_doc_datasets_pdf/pdf_ready/v1/models.py` - - Define `PdfReadyV1`, `Meta`, `Content`, `UserStory`, `Sections`, `SelectionSummary` - - Add field validators (e.g., `schema_version` must be `"1.0"`) - - Add `model_config` for strict validation - - **Reference:** SPEC.md section 3.3 - - **Verification:** Models instantiate correctly with valid data - -- [ ] **2.3 Create `audit` v1.0 Pydantic models** - - Create `src/living_doc_datasets_pdf/audit/v1/models.py` - - Define `AuditEnvelopeV1`, `Producer`, `Run`, `Source`, `TraceStep`, `Warning` - - Add field validators (e.g., `source.systems` must be non-empty) - - **Reference:** SPEC.md section 3.4 - - **Verification:** Models instantiate correctly with valid data - -- [ ] **2.4 Export JSON Schemas** - - Create `src/living_doc_datasets_pdf/pdf_ready/v1/schema.py` - - Add function `export_json_schema()` using `model.model_json_schema()` - - Write schema to `schemas/pdf_ready_v1.schema.json` - - Repeat for `audit` → `schemas/audit_envelope_v1.schema.json` - - **Verification:** Schemas validate against reference examples (SPEC.md 3.3.4) - -- [ ] **2.5 Add serialization helpers** - - Create `src/living_doc_datasets_pdf/pdf_ready/v1/serializer.py` - - Add `to_json(model, indent=2, sort_keys=True)` for deterministic output - - Add `from_json(json_str)` for parsing - - **Verification:** Same input produces byte-identical output - -- [ ] **2.6 Write unit tests** - - Create `tests/test_pdf_ready_models.py` - - Test valid input (example from SPEC.md 3.3.4) - - Test invalid inputs (missing required fields, wrong types) - - Test field validators (e.g., `schema_version` must be `"1.0"`) - - Create `tests/test_audit_models.py` - - Test valid audit envelope - - Test invalid inputs - - Test non-empty constraints (`systems`, `producer.name`) - - **Tools:** `pytest`, `pytest-cov` - - **Verification:** Run `pytest packages/datasets_pdf/tests/ --cov=packages/datasets_pdf/src --cov-report=term` - - **Pass criteria:** Coverage ≥ 80% - ---- - -## Phase 3: Core Utilities Package (`packages/core`) - -**Goal:** Create reusable, service-agnostic helper utilities. - -### Tasks - -- [x] **3.1 Set up package structure** - - Create `packages/core/pyproject.toml` - - Define package as `living-doc-core` - - Add dependencies: `click ^8.1.0` (for potential future use) - - Create `src/living_doc_core/__init__.py` - - **Verification:** Package installable with `pip install -e packages/core` - -- [x] **3.2 Create JSON utilities** - - Create `src/living_doc_core/json_utils.py` - - Add `read_json(filepath)` with error handling - - Add `write_json(filepath, data, indent=2, sort_keys=True)` for deterministic output - - Add validation helpers - - **Verification:** Read/write round-trip produces identical output - -- [x] **3.3 Create logging configuration** - - Create `src/living_doc_core/logging_config.py` - - Add `setup_logging(verbose=False)` function - - Configure log levels: `DEBUG`, `INFO`, `WARNING`, `ERROR` - - Add log formatting: `[{level}] {message}` - - **Verification:** Logging output matches expected format - -- [x] **3.4 Create Markdown utilities** - - Create `src/living_doc_core/markdown_utils.py` - - Add `split_by_headings(markdown_text, level=2)` → dict of heading → content - - Add `normalize_heading(text)` → lowercase, stripped - - Add `extract_lists(markdown_text)` (optional, for future use) - - **Verification:** Headings split correctly with test cases - -- [x] **3.5 Create common error types** - - Create `src/living_doc_core/errors.py` - - Define `ToolkitError` (base exception) - - Define `InvalidInputError`, `AdapterError`, `SchemaValidationError`, `NormalizationError`, `FileIOError` - - Map error types to exit codes (from SPEC.md 3.1.2) - - **Verification:** Error classes raise correctly with messages - -- [x] **3.6 Write unit tests** - - Create `tests/test_json_utils.py` - - Test read/write with valid JSON - - Test read with malformed JSON (should raise error) - - Test write with invalid path (should raise error) - - Create `tests/test_markdown_utils.py` - - Test heading splitting with multiple levels - - Test edge cases (no headings, content before first heading) - - Test heading normalization - - **Tools:** `pytest`, `pytest-cov` - - **Verification:** Run `pytest packages/core/tests/ --cov=packages/core/src --cov-report=term` - - **Pass criteria:** Coverage ≥ 80% - ---- - -## Phase 4: Adapter Package (`packages/adapters/collector_gh`) - -**Goal:** Create collector-gh adapter for input detection and parsing. - -### Tasks - -- [ ] **4.1 Set up package structure** - - Create `packages/adapters/collector_gh/pyproject.toml` - - Define package as `living-doc-adapter-collector-gh` - - Add dependencies: `living-doc-core`, `pydantic ^2.10.0` - - Create `src/living_doc_adapter_collector_gh/__init__.py` - - **Verification:** Package installable with `pip install -e packages/adapters/collector_gh` - -- [ ] **4.2 Create `AdapterResult` model** - - Create `src/living_doc_adapter_collector_gh/models.py` - - Define `AdapterResult` with fields: `items`, `metadata`, `warnings` - - Define `AdapterItem` with fields: `id`, `title`, `state`, `tags`, `url`, `timestamps`, `body` - - **Reference:** SPEC.md section 3.2.3 - - **Verification:** Model validates correctly - -- [ ] **4.3 Create producer detection logic** - - Create `src/living_doc_adapter_collector_gh/detector.py` - - Add `can_handle(payload: dict) -> bool` - - Check `payload["metadata"]["generator"]["name"] == "AbsaOSS/living-doc-collector-gh"` - - Add `extract_version(payload: dict) -> str` - - Return `payload["metadata"]["generator"]["version"]` - - **Reference:** SPEC.md section 3.2.1 - - **Verification:** Detection works with fixture files - -- [ ] **4.4 Create version compatibility checker** - - Create `src/living_doc_adapter_collector_gh/compatibility.py` - - Add `check_compatibility(version: str) -> list[Warning]` - - Define confirmed range: `>=1.0.0,<2.0.0` (using `packaging.version`) - - Return warning if outside range - - **Reference:** SPEC.md section 3.2.2 - - **Verification:** Warnings generated for versions `0.9.0`, `2.0.0` - -- [ ] **4.5 Create input parser** - - Create `src/living_doc_adapter_collector_gh/parser.py` - - Add `parse(payload: dict) -> AdapterResult` - - Map `metadata.generator.*` → audit envelope fields - - Map `issues[]` → `AdapterItem[]` - - Store original `metadata` in result - - **Reference:** SPEC.md section 3.4.3 - - **Verification:** Parser output matches expected structure - -- [ ] **4.6 Add fixture files** - - Create `tests/fixtures/collector_v1.0.0/input/doc-issues.json` - - Create `tests/fixtures/collector_v1.2.0/input/doc-issues.json` - - Use realistic examples (10-15 issues, varied markdown) - - **Reference:** `living-doc-collector-gh` output format - -- [ ] **4.7 Write unit tests** - - Create `tests/test_detector.py` - - Test detection with valid collector-gh payload - - Test rejection with invalid payload - - Test version extraction - - Create `tests/test_parser.py` - - Test parsing with fixture v1.0.0 - - Test parsing with fixture v1.2.0 - - Test audit mapping - - Create `tests/test_compatibility.py` - - Test version in range (no warnings) - - Test version outside range (warnings generated) - - **Tools:** `pytest`, `pytest-cov` - - **Verification:** Run `pytest packages/adapters/collector_gh/tests/ --cov=packages/adapters/collector_gh/src --cov-report=term` - - **Pass criteria:** Coverage ≥ 80% - ---- - -## Phase 5: Service Package (`packages/services/normalize_issues`) - -**Goal:** Implement normalize-issues service with markdown normalization and PDF-ready JSON builder. - -### Tasks - -- [ ] **5.1 Set up package structure** - - Create `packages/services/normalize_issues/pyproject.toml` - - Define package as `living-doc-service-normalize-issues` - - Add dependencies: `living-doc-core`, `living-doc-datasets-pdf`, `living-doc-adapter-collector-gh` - - Create `src/living_doc_service_normalize_issues/__init__.py` - - **Verification:** Package installable with `pip install -e packages/services/normalize_issues` - -- [ ] **5.2 Create markdown normalizer** - - Create `src/living_doc_service_normalize_issues/normalizer.py` - - Add `normalize_sections(markdown: str) -> dict` - - Implement heading synonym mapping (SPEC.md 3.3.3) - - `Description`, `Overview`, `Summary` → `description` - - `Business Value`, `Value`, `Why` → `business_value` - - `Acceptance Criteria`, `AC` → `acceptance_criteria` - - etc. - - Handle unknown headings (append to `description`) - - Handle content before first heading (assign to `description`) - - **Reference:** SPEC.md section 5.3 - - **Verification:** Deterministic output for same input - -- [ ] **5.3 Create PDF-ready JSON builder** - - Create `src/living_doc_service_normalize_issues/builder.py` - - Add `build_pdf_ready(adapter_result, options) -> PdfReadyV1` - - Map `AdapterItem` → `UserStory` with normalized sections - - Generate stable `id` (e.g., `github:owner/repo#123`) - - Populate `meta` fields (document_title, document_version, generated_at, etc.) - - Augment `audit.trace[]` with normalization step - - **Reference:** SPEC.md section 3.3 - - **Verification:** Output validates against `PdfReadyV1` schema - -- [ ] **5.4 Create service orchestration** - - Create `src/living_doc_service_normalize_issues/service.py` - - Add `run_service(input_path, output_path, options) -> None` - - Implement pipeline (SPEC.md 5.1): - 1. Load input JSON - 2. Detect adapter (auto or explicit) - 3. Check compatibility - 4. Parse into AdapterResult - 5. Normalize sections - 6. Build pdf_ready.json - 7. Validate output - 8. Write output file - 9. Log summary - - **Reference:** SPEC.md section 5.1 - - **Verification:** End-to-end execution with fixture - -- [ ] **5.5 Write unit tests** - - Create `tests/test_normalizer.py` - - Test heading synonym mapping - - Test unknown heading handling - - Test edge cases (no headings, empty body) - - Test determinism (same input → same output) - - Create `tests/test_builder.py` - - Test `AdapterItem` → `UserStory` mapping - - Test `meta` field population - - Test audit augmentation - - Create `tests/test_service.py` - - Test full pipeline with valid input - - Test error handling (invalid input, missing file) - - **Tools:** `pytest`, `pytest-cov` - - **Verification:** Run `pytest packages/services/normalize_issues/tests/ --cov=packages/services/normalize_issues/src --cov-report=term` - - **Pass criteria:** Coverage ≥ 80% - -- [ ] **5.6 Add golden file integration tests** - - Create `tests/integration/test_golden_files.py` - - Add fixtures: `tests/fixtures/golden/input.json`, `tests/fixtures/golden/expected_output.json` - - Test: Run service, compare output to expected (ignore timestamps) - - **Verification:** Golden file test passes - -- [ ] **5.7 Create verification scripts** - - Create `verifications/verify_golden.py` - - Load fixture input - - Run normalization - - Compare to expected output - - Print diff if mismatch - - Create `verifications/verify_compatibility.py` - - Test with collector v0.9.0 (should warn) - - Test with collector v1.0.0 (should pass) - - Test with collector v2.0.0 (should warn) - - **Verification:** Scripts runnable with `python verifications/verify_golden.py` - ---- - -## Phase 6: CLI Wrapper (`apps/cli`) - -**Goal:** Create unified CLI with `living-doc` command and subcommands. - -### Tasks - -- [ ] **6.1 Set up package structure** - - Create `apps/cli/pyproject.toml` - - Define package as `living-doc-cli` - - Add dependencies: `living-doc-core`, `living-doc-service-normalize-issues`, `click ^8.1.0` - - Create `src/living_doc_cli/__init__.py` - - **Verification:** Package installable with `pip install -e apps/cli` - -- [ ] **6.2 Create CLI entrypoint** - - Create `src/living_doc_cli/main.py` - - Add `@click.group()` for `living-doc` command - - Add `--version` flag - - Add global `--verbose` flag - - **Reference:** SPEC.md section 3.1.1 - - **Verification:** `living-doc --help` displays usage - -- [ ] **6.3 Create `normalize-issues` subcommand** - - Create `src/living_doc_cli/commands/normalize_issues.py` - - Add `@click.command()` for `normalize-issues` - - Add arguments: `--input`, `--output`, `--source`, `--document-title`, `--document-version`, `--verbose` - - Call `run_service()` from service package - - Handle exceptions and map to exit codes (SPEC.md 3.1.2) - - Format error messages: `{prefix} {detail}. {guidance}` - - **Reference:** SPEC.md section 3.1 - - **Verification:** CLI invocation matches specification - -- [ ] **6.4 Add error handling and exit codes** - - Map exceptions to exit codes: - - `InvalidInputError` → exit 1 - - `AdapterError` → exit 2 - - `SchemaValidationError` → exit 3 - - `NormalizationError` → exit 4 - - `FileIOError` → exit 5 - - Ensure error messages follow format - - **Reference:** SPEC.md section 3.1.2 - - **Verification:** Test each error type manually - -- [ ] **6.5 Write unit tests** - - Create `tests/test_cli.py` - - Test argument parsing - - Test `--help` output - - Test `--version` output - - Create `tests/integration/test_cli_invocation.py` - - Test successful execution - - Test invalid input (file not found) → exit 1 - - Test adapter error → exit 2 - - **Tools:** `pytest`, `click.testing.CliRunner` - - **Verification:** Run `pytest apps/cli/tests/ --cov=apps/cli/src --cov-report=term` - - **Pass criteria:** Coverage ≥ 80% - -- [ ] **6.6 Add console script entry point** - - Update `apps/cli/pyproject.toml` - - Add `[project.scripts]` with `living-doc = "living_doc_cli.main:cli"` - - **Verification:** After install, `living-doc --help` works from command line - ---- - -## Phase 7: Python CI/CD Workflow - -**Goal:** Configure GitHub Actions workflows for static analysis, tests, and coverage. - -### Tasks - -- [ ] **7.1 Create static analysis workflow** - - Create `.github/workflows/ci.yml` - - Add Pylint job: - - Set up Python 3.14 - - Install dependencies - - Run `pylint` on all `*.py` files - - Enforce score ≥ 9.5 - - Add Black format check job: - - Run `black --check $(git ls-files '*.py')` - - Add Mypy type check job: - - Run `mypy .` - - **Reference:** `living-doc-collector-gh/.github/workflows/static_analysis_and_tests.yml` - - **Verification:** Workflow passes on clean code - -- [ ] **7.2 Add unit test workflow** - - Add pytest job to `.github/workflows/ci.yml`: - - Set up Python 3.14 - - Install all packages (`pip install -e packages/*/` and `pip install -e apps/cli`) - - Run `pytest --cov=. --cov-report=term --cov-fail-under=80` - - **Reference:** `living-doc-collector-gh` pytest configuration - - **Verification:** Tests pass with coverage ≥ 80% - -- [ ] **7.3 Create integration test workflow** - - Create `.github/workflows/integration.yml` - - Add fixture verification job: - - Run `verifications/verify_golden.py` - - Run `verifications/verify_compatibility.py` - - **Verification:** Integration tests pass in CI - -- [ ] **7.4 Configure Python version pinning** - - Pin Python 3.14 in all workflows - - Use `actions/setup-python@v5` with `python-version: '3.14'` - - Enable pip caching with `cache: 'pip'` - - **Reference:** `living-doc-collector-gh` workflows - -- [ ] **7.5 Add CI status badges to README** - - Add badges for: - - CI status (static analysis + tests) - - Integration tests status - - Coverage (optional, if using Codecov) - - **Verification:** Badges display correctly on GitHub - ---- - -## Phase 8: Fixture-Based Verification - -**Goal:** Add comprehensive fixtures and manual verification scripts. - -### Tasks - -- [ ] **8.1 Add collector-gh fixtures** - - Create `tests/fixtures/collector_gh/v1.0.0/input/doc-issues.json` - - Create `tests/fixtures/collector_gh/v1.2.0/input/doc-issues.json` - - Create `tests/fixtures/collector_gh/v2.0.0/input/doc-issues.json` (out-of-range) - - Use realistic examples with: - - 10-15 issues - - Varied markdown (headings, lists, tables, links) - - Different states (open, closed) - - Multiple tags - - **Verification:** Files are valid JSON and follow collector-gh format - -- [ ] **8.2 Add golden output files** - - Create `tests/fixtures/golden/v1.0.0/expected_output.json` - - Create `tests/fixtures/golden/v1.2.0/expected_output.json` - - Generate expected outputs manually (verified by SPEC) - - **Verification:** Expected outputs validate against `PdfReadyV1` schema - -- [ ] **8.3 Create golden file verification script** - - Create `verifications/verify_golden.py` - - For each fixture: - - Run `living-doc normalize-issues --input {fixture} --output {temp}` - - Load expected output - - Compare (ignore `generated_at`, `started_at`, `finished_at`) - - Print diff if mismatch - - Exit 0 if all pass, exit 1 if any fail - - **Verification:** Script passes with current implementation - -- [ ] **8.4 Create compatibility verification script** - - Create `verifications/verify_compatibility.py` - - Test with v0.9.0 fixture (expect warnings) - - Test with v1.0.0 fixture (expect success) - - Test with v2.0.0 fixture (expect warnings) - - Verify warnings appear in output and `audit.trace[].warnings[]` - - **Verification:** Script detects version mismatches correctly - -- [ ] **8.5 Add CI integration for verification scripts** - - Update `.github/workflows/integration.yml` - - Add steps to run verification scripts - - Fail workflow if verification fails - - **Verification:** CI fails if golden files don't match - ---- - -## Phase 9: Cookbooks & Recipes - -**Goal:** Write user-facing documentation with examples and ready-to-use recipes. - -### Tasks - -- [ ] **9.1 Write `normalize-issues` cookbook** - - Create `docs/cookbooks/normalize-issues.md` - - Sections: - - What it does (overview) - - How detection works (adapter selection) - - How compatibility checking works (version ranges) - - How audit is preserved and augmented (trace steps) - - How to interpret warnings - - Troubleshooting (common errors) - - **Verification:** Documentation is clear and accurate - -- [ ] **9.2 Create local usage recipe** - - Create `docs/recipes/local-normalize-issues.md` - - Include: - - Prerequisites (Python 3.14, pip install) - - Installation commands - - Example CLI invocation (with real paths) - - Expected output - - How to verify output (schema validation) - - **Reference:** SPEC.md section 3.1.1 for CLI examples - - **Verification:** Recipe tested locally and works as documented - -- [ ] **9.3 Create GitHub Actions recipe** - - Create `docs/recipes/github-actions-normalize-issues.yml` - - Include complete workflow example: - - Checkout code - - Set up Python 3.14 - - Install living-doc-toolkit - - Run `living-doc normalize-issues` - - Upload artifacts (pdf_ready.json) - - Show chaining: collector → builder → generator - - **Verification:** Recipe tested in CI and works as documented - -- [ ] **9.4 Add troubleshooting guide** - - Create `docs/troubleshooting.md` - - Cover common issues: - - "File not found" errors - - "No compatible adapter found" errors - - Version mismatch warnings - - Schema validation failures - - Provide actionable solutions for each - - **Verification:** Guide covers all error codes from SPEC.md 3.1.2 - -- [ ] **9.5 Add architecture diagram** (optional) - - Create `docs/architecture.md` - - Include diagram showing: - - Collector → Adapter → Service → Dataset → Generator - - Monorepo package structure - - Data flow - - **Tools:** Mermaid, PlantUML, or draw.io - - **Verification:** Diagram matches SPEC.md architecture - ---- - -## Phase 10: Release Preparation - -**Goal:** Prepare for first release (v0.1.0). - -### Tasks - -- [ ] **10.1 Set version numbers** - - Update `version` in all `pyproject.toml` files to `0.1.0` - - Ensure consistency across all packages - - **Verification:** All version numbers match - -- [ ] **10.2 Update `CHANGELOG.md`** - - Move `[Unreleased]` items to `[0.1.0] - 2026-01-23` - - Sections: - - **Added:** Initial implementation of normalize-issues service, adapters, core utilities, CLI, etc. - - **Changed:** N/A (initial release) - - **Fixed:** N/A (initial release) - - **Verification:** Changelog follows Keep a Changelog format - -- [ ] **10.3 Write migration guide** (for future releases) - - Create `docs/migration_guide.md` - - Add template for future breaking changes - - **Verification:** Template is ready for v0.2.0 updates - -- [ ] **10.4 Run full test suite** - - Run `pytest` on all packages - - Run all verification scripts - - Run CI workflows locally (using `act` or similar) - - **Pass criteria:** All tests pass, coverage ≥ 80% - -- [ ] **10.5 Review documentation** - - Verify SPEC.md is up-to-date - - Verify README.md is accurate - - Verify all recipes work as documented - - Check for broken links - - **Verification:** Documentation review complete - -- [ ] **10.6 Create release workflow** - - Create `.github/workflows/release.yml` - - Add steps: - - Trigger on tag push (`v*`) - - Run full test suite - - Build packages - - Create GitHub Release draft - - Upload artifacts - - **Reference:** `living-doc-collector-gh/.github/workflows/release_draft.yml` - - **Verification:** Workflow tested with dry-run - -- [ ] **10.7 Create GitHub Release draft** - - Tag: `v0.1.0` - - Title: "Living Documentation Toolkit v0.1.0 - Initial Release" - - Body: - - Summary of features - - Link to SPEC.md - - Link to cookbooks and recipes - - Installation instructions - - Known limitations - - **Verification:** Release notes are complete and accurate - -- [ ] **10.8 Run final pre-release checklist** - - [ ] All CI workflows passing - - [ ] Code coverage ≥ 80% - - [ ] Pylint score ≥ 9.5 - - [ ] Black formatting applied - - [ ] Mypy type checking passing - - [ ] All verification scripts passing - - [ ] Documentation complete and reviewed - - [ ] CHANGELOG.md updated - - [ ] Version numbers consistent - - [ ] License headers present - -- [ ] **10.9 Publish release** - - Push tag `v0.1.0` - - Trigger release workflow - - Publish GitHub Release - - **Verification:** Release is live on GitHub - ---- - -## Post-Release Tasks - -**These tasks are for ongoing maintenance after v0.1.0 release.** - -- [ ] **Monitor issue tracker** - - Triage new issues - - Label bugs, enhancements, questions - - Respond to user feedback - -- [ ] **Plan v0.2.0 features** - - Review SPEC.md "Future Extensions" (section 12.1) - - Prioritize next service or feature - - Update TASKS.md with new phases - -- [ ] **Maintain compatibility** - - Track `living-doc-collector-gh` releases - - Update adapter confirmed range if needed - - Test with new collector versions - -- [ ] **Improve documentation** - - Add more examples based on user feedback - - Expand troubleshooting guide - - Record common questions in FAQ - ---- - -## Appendix: Tool Reference - -### Testing Tools - -- **pytest**: Testing framework - - Run: `pytest` - - Coverage: `pytest --cov=. --cov-report=term` - - Coverage threshold: `--cov-fail-under=80` - -- **pytest-cov**: Coverage plugin - - Config: `[tool.coverage.run]` in `pyproject.toml` - -### Code Quality Tools - -- **Black**: Code formatter - - Check: `black --check $(git ls-files '*.py')` - - Format: `black $(git ls-files '*.py')` - - Config: `[tool.black]` in `pyproject.toml` (line-length=120, py314) - -- **Pylint**: Linter - - Run: `pylint $(git ls-files '*.py')` - - Threshold: Score ≥ 9.5 - - Config: `.pylintrc` or `pyproject.toml` - -- **Mypy**: Type checker - - Run: `mypy .` - - Config: `[tool.mypy]` in `pyproject.toml` (check_untyped_defs, py314) - -### CI/CD Tools - -- **GitHub Actions**: CI/CD platform - - Workflows: `.github/workflows/*.yml` - - Python version: 3.14 (pinned) - - Cache: `cache: 'pip'` - ---- - -## Document History - -| Version | Date | Author | Changes | -|---------|------|--------|---------| -| 1.0 | 2026-01-23 | Specification Master | Initial roadmap based on SPEC.md | - ---- - -**End of Document** diff --git a/apps/cli/pyproject.toml b/apps/cli/pyproject.toml index 58fe9d1..8e603b9 100644 --- a/apps/cli/pyproject.toml +++ b/apps/cli/pyproject.toml @@ -11,7 +11,7 @@ license = {text = "Apache-2.0"} authors = [ {name = "AbsaOSS", email = "miroslav.pojer@absa.africa"} ] -requires-python = ">=3.11" +requires-python = ">=3.14" dependencies = [ "living-doc-core", "living-doc-service-normalize-issues", diff --git a/apps/cli/src/living_doc_cli/commands/normalize_issues.py b/apps/cli/src/living_doc_cli/commands/normalize_issues.py index 2e36741..758b177 100644 --- a/apps/cli/src/living_doc_cli/commands/normalize_issues.py +++ b/apps/cli/src/living_doc_cli/commands/normalize_issues.py @@ -105,7 +105,7 @@ def normalize_issues( # pylint: disable=too-many-arguments,too-many-positional- Normalize collector output into PDF-ready JSON format. Converts collector-gh output (doc-issues.json) into canonical PDF-ready format - (pdf_ready.json) compliant with living-doc-generator-pdf SPEC v1.0. + (pdf_ready.json) compliant with living-doc-generator-pdf. """ # Merge verbose flag: local overrides global global_verbose = ctx.obj.get("verbose", False) if ctx.obj else False diff --git a/apps/cli/tests/integration/test_cli_invocation.py b/apps/cli/tests/integration/test_cli_invocation.py index 890f076..124221c 100644 --- a/apps/cli/tests/integration/test_cli_invocation.py +++ b/apps/cli/tests/integration/test_cli_invocation.py @@ -40,9 +40,9 @@ def valid_input_data(): "enterprise": None, }, }, - "issues": [ - { - "number": 1, + "issues": { + "owner/repo/1": { + "issue_number": 1, "title": "Test Issue", "state": "open", "labels": ["enhancement"], @@ -51,7 +51,7 @@ def valid_input_data(): "updated_at": "2026-01-02T00:00:00Z", "body": "## Description\nThis is a test issue.\n\n## Acceptance Criteria\n- Criterion 1\n- Criterion 2", } - ], + }, } @@ -119,7 +119,7 @@ def test_cli_invocation_bad_metadata(runner, tmp_path): # Create input with bad metadata input_data = { "metadata": {"generator": {"name": "unknown-adapter", "version": "1.0.0"}}, - "issues": [], + "issues": {}, } input_file = tmp_path / "input.json" diff --git a/docs/architecture.md b/docs/architecture.md index 255480e..31b3ecb 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -31,7 +31,7 @@ The Living Documentation Toolkit is a **generic builder** that transforms machin ## Data Flow Pipeline -### High-Level Pipeline (SPEC.md §5.1) +### High-Level Pipeline ```mermaid graph LR @@ -122,7 +122,7 @@ flowchart TD ## Monorepo Package Structure -### Directory Layout (SPEC.md §4.1) +### Directory Layout ```mermaid graph TD @@ -517,7 +517,9 @@ graph TB ## Performance Considerations -### Expected Performance (SPEC.md §5.5) +### Expected Performance + +See [Performance Budgets](contracts.md#performance-budgets) for detailed targets. | Input Size | Target Time | Maximum Time | |------------|-------------|--------------| @@ -577,8 +579,8 @@ flowchart TD ## Additional Resources -- **SPEC.md**: Full system specification -- **TASKS.md**: Implementation roadmap -- **Cookbook**: `docs/cookbooks/normalize-issues.md` -- **Recipes**: `docs/recipes/` -- **Troubleshooting**: `docs/troubleshooting.md` +- **[Contracts & Interfaces](contracts.md)** — CLI reference, schemas, change control +- **[Cookbook: normalize-issues](cookbooks/normalize-issues.md)** — How detection, compatibility, and normalization work +- **[Recipe: Local usage](recipes/local-normalize-issues.md)** — Run the CLI on your machine +- **[Recipe: GitHub Actions](recipes/github-actions-normalize-issues.md)** — CI/CD workflow integration +- **[Troubleshooting](troubleshooting.md)** — Exit codes, common errors, FAQ diff --git a/docs/contracts.md b/docs/contracts.md new file mode 100644 index 0000000..1c87d93 --- /dev/null +++ b/docs/contracts.md @@ -0,0 +1,213 @@ +# Contracts & Interfaces Reference + +Quick reference for all external-facing contracts. Changes to items below require review (see [Change Control](#change-control)). + +--- + +- [CLI Interface](#cli-interface) +- [Input Contract: `doc-issues.json`](#input-contract-doc-issuesjson) +- [Output Contract: `pdf_ready.json`](#output-contract-pdf_readyjson) +- [Audit Envelope (v1.0)](#audit-envelope-v10) +- [JSON Schemas](#json-schemas) +- [Change Control](#change-control) +- [Performance Budgets](#performance-budgets) + +--- + +## CLI Interface + +**Command:** `living-doc normalize-issues` + +| Argument | Type | Required | Default | Description | +|----------|------|----------|---------|-------------| +| `--input` | path | Yes | — | Path to input JSON (e.g., `doc-issues.json`) | +| `--output` | path | Yes | — | Path for output JSON (e.g., `pdf_ready.json`) | +| `--source` | enum | No | `auto` | Adapter selection: `auto`, `collector-gh` | +| `--document-title` | string | No | from input | Override `meta.document_title` | +| `--document-version` | string | No | from input | Override `meta.document_version` | +| `--verbose` | flag | No | `false` | Enable verbose logging | + +### Exit Codes + +| Code | Condition | Error Prefix | +|------|-----------|--------------| +| 0 | Success | — | +| 1 | Invalid input (missing file, malformed JSON) | `Invalid input:` | +| 2 | Adapter detection failed | `Adapter error:` | +| 3 | Schema validation failure | `Schema validation failed:` | +| 4 | Normalization error | `Normalization failed:` | +| 5 | File I/O error | `File I/O error:` | + +Error format: `{prefix} {detail}. {guidance}` + +--- + +## Input Contract: `doc-issues.json` + +Produced by [living-doc-collector-gh](https://github.com/AbsaOSS/living-doc-collector-gh). + +### Producer Detection + +Adapter auto-detection checks: +- `metadata.generator.name` == `"AbsaOSS/living-doc-collector-gh"` +- `metadata.generator.version` — semver format + +### Compatibility Policy + +**Confirmed range:** `>=1.0.0,<2.0.0` + +| Scenario | Behavior | +|----------|----------| +| Within range | Proceed silently | +| Outside range | Log warning, add to `audit.trace[].warnings[]`, attempt processing | +| Unrecognizable schema | Exit with `Adapter error:` | + +Warning format in audit: +```json +{ + "code": "VERSION_MISMATCH", + "message": "Producer version 2.1.0 is outside confirmed range >=1.0.0,<2.0.0", + "context": "metadata.generator.version" +} +``` + +--- + +## Output Contract: `pdf_ready.json` + +Target: [living-doc-generator-pdf](https://github.com/AbsaOSS/living-doc-generator-pdf). + +**Schema version:** `"1.0"` (field `schema_version`) + +### Structure + +``` +pdf_ready.json +├── schema_version: "1.0" +├── meta +│ ├── document_title, document_version, generated_at +│ ├── source_set[] +│ ├── selection_summary { total_items, included_items, excluded_items } +│ └── audit (optional) → see Audit Envelope below +└── content + └── user_stories[] + ├── id, title, state, tags[], url + ├── timestamps { created, updated } + └── sections { description, business_value, … } +``` + +### Section Mapping (Heading Synonyms) + +Issue body `##` headings map to canonical section keys (case-insensitive): + +| Canonical Key | Accepted Synonyms | +|---------------|-------------------| +| `description` | Description, Overview, Summary | +| `business_value` | Business Value, Value, Why | +| `preconditions` | Preconditions, Prerequisites, Setup | +| `acceptance_criteria` | Acceptance Criteria, AC, Done Criteria | +| `user_guide` | User Guide, How To, Instructions | +| `connections` | Connections, Related, Links | +| `last_edited` | Last Edited, History, Changes | + +**Edge cases:** +- Unknown headings → appended to `description` as `### {Heading}\n{content}` +- Content before first heading → assigned to `description` +- Multiple occurrences of same heading → concatenated with separator +- Missing sections → `null` or `""` + +### Stable ID Format + +`github:{owner}/{repo}#{number}` (e.g., `github:AbsaOSS/project#42`) + +--- + +## Audit Envelope (v1.0) + +Lives at `meta.audit`. Preserves upstream provenance and tracks transformation steps. + +### Structure + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `schema_version` | string | Yes | Fixed `"1.0"` | +| `producer.name` | string | Yes | e.g., `"AbsaOSS/living-doc-collector-gh"` | +| `producer.version` | string | Yes | Semver | +| `producer.build` | string \| null | No | Build identifier | +| `run.*` | object | No | CI run context (run_id, actor, workflow, ref, sha) | +| `source.systems` | string[] | Yes | Non-empty (e.g., `["GitHub"]`) | +| `source.repositories` | string[] | No | e.g., `["AbsaOSS/project"]` | +| `trace[]` | array | Yes | Ordered transformation steps | +| `extensions` | object | No | Namespaced extra data | + +### Trace Step + +Each pipeline stage appends a trace entry: +```json +{ + "step": "normalization", + "tool": "living-doc-toolkit", + "tool_version": "0.1.0", + "started_at": "2026-01-23T12:00:00Z", + "finished_at": "2026-01-23T12:00:05Z", + "warnings": [] +} +``` + +### Metadata Mapping (Collector → Audit) + +| Collector field | Audit field | +|-----------------|-------------| +| `metadata.generator.*` | `audit.producer.*` | +| `metadata.run.*` | `audit.run.*` | +| `metadata.source.*` | `audit.source.*` | +| Full original `metadata` | `audit.extensions["collector-gh"].original_metadata` | + +--- + +## JSON Schemas + +Machine-readable schemas are at: +- `packages/datasets_pdf/schemas/pdf_ready_v1.schema.json` +- `packages/datasets_pdf/schemas/audit_envelope_v1.schema.json` + +Pydantic models (source of truth): +- `packages/datasets_pdf/src/living_doc_datasets_pdf/pdf_ready/v1/models.py` +- `packages/datasets_pdf/src/living_doc_datasets_pdf/audit/v1/models.py` + +--- + +## Change Control + +### Stable (breaking changes require major version bump) + +- Schema field names, types, and meanings (v1.0) +- `AdapterResult` model signature +- CLI argument names and defaults +- Exit codes and error message prefixes + +### Safe to change (no version bump needed) + +- Internal refactoring (preserving behavior) +- New optional fields in schemas +- New optional CLI arguments +- Test, doc, and logging improvements + +### Requires review before changing + +- Schema modifications (`datasets_pdf`) +- Adapter interface changes +- Error message text +- Performance budgets + +--- + +## Performance Budgets + +| Operation | Target | Maximum | +|-----------|--------|---------| +| JSON parsing (10 MB) | < 1 s | 5 s | +| Adapter detection | < 0.1 s | 1 s | +| Markdown normalization (100 issues) | < 2 s | 10 s | +| Output validation | < 1 s | 5 s | +| Total (100 issues) | < 10 s | 30 s | diff --git a/docs/cookbooks/normalize-issues.md b/docs/cookbooks/normalize-issues.md index 85e8ab5..3a15811 100644 --- a/docs/cookbooks/normalize-issues.md +++ b/docs/cookbooks/normalize-issues.md @@ -6,11 +6,23 @@ --- +- [What It Does (Overview)](#what-it-does-overview) +- [How Detection Works (Adapter Selection)](#how-detection-works-adapter-selection) +- [How Compatibility Checking Works (Version Ranges)](#how-compatibility-checking-works-version-ranges) +- [How Audit Is Preserved and Augmented (Trace Steps)](#how-audit-is-preserved-and-augmented-trace-steps) +- [How to Interpret Warnings](#how-to-interpret-warnings) +- [Troubleshooting (Common Errors)](#troubleshooting-common-errors) +- [Markdown Normalization Rules](#markdown-normalization-rules) +- [Best Practices](#best-practices) +- [Additional Resources](#additional-resources) + +--- + ## What It Does (Overview) -The `normalize-issues` service transforms machine-readable artifacts produced by upstream collectors (e.g., `AbsaOSS/living-doc-collector-gh`) into a canonical, PDF-ready JSON format compliant with `AbsaOSS/living-doc-generator-pdf` SPEC v1.0. +The `normalize-issues` service transforms machine-readable artifacts produced by upstream collectors (e.g., `AbsaOSS/living-doc-collector-gh`) into a canonical, PDF-ready JSON format compliant with [living-doc-generator-pdf](https://github.com/AbsaOSS/living-doc-generator-pdf). -**Pipeline Overview (SPEC.md §1.2, §5.1):** +**Pipeline Overview:** 1. Load and parse input JSON (`doc-issues.json`) 2. Detect producer adapter (auto-detection or explicit selection) 3. Check version compatibility (warn if outside confirmed range) @@ -29,11 +41,23 @@ The `normalize-issues` service transforms machine-readable artifacts produced by - **Audit trail preservation**: Maintains enterprise-level provenance tracking - **Schema validation**: Ensures output conforms to generator contract +**Quick Example:** +```bash +living-doc normalize-issues \ + --input doc-issues.json \ + --output pdf_ready.json \ + --source auto \ + --document-title "Sprint 42 Report" \ + --document-version "1.0.0" +``` + +See [Contracts & Interfaces](../contracts.md#cli-interface) for the full argument reference. + --- ## How Detection Works (Adapter Selection) -### Auto-Detection (SPEC.md §3.2.1, §5.2) +### Auto-Detection When `--source auto` is used (default), the service automatically detects the producer by examining the `metadata.generator.name` field in the input JSON: @@ -66,7 +90,7 @@ This is useful when: ## How Compatibility Checking Works (Version Ranges) -### Confirmed Compatible Range (SPEC.md §3.2.2, §5.4) +### Confirmed Compatible Range **Adapter:** `collector-gh` **Confirmed Range:** `>=1.0.0,<2.0.0` @@ -93,7 +117,7 @@ This policy ensures: ## How Audit Is Preserved and Augmented (Trace Steps) -### Adapter Mapping (SPEC.md §3.4.3) +### Adapter Mapping The adapter maps collector metadata to the audit envelope: @@ -170,7 +194,7 @@ This creates a complete audit trail from collection → normalization → genera ## How to Interpret Warnings -### VERSION_MISMATCH Warning (SPEC.md §3.2.2, §5.4) +### VERSION_MISMATCH Warning When the producer version is outside the confirmed range, a warning is logged and added to the audit trace: @@ -198,7 +222,7 @@ When the producer version is outside the confirmed range, a warning is logged an ## Troubleshooting (Common Errors) -### Exit Code 1: Invalid Input (SPEC.md §3.1.2) +### Exit Code 1: Invalid Input **Error Prefix:** `Invalid input:` @@ -219,7 +243,7 @@ Invalid input: File 'doc-issues.json' not found. Ensure --input points to a vali --- -### Exit Code 2: Adapter Error (SPEC.md §3.1.2) +### Exit Code 2: Adapter Error **Error Prefix:** `Adapter error:` @@ -239,7 +263,7 @@ Adapter error: No compatible adapter found for input. Check metadata.generator.n --- -### Exit Code 3: Schema Validation Failed (SPEC.md §3.1.2) +### Exit Code 3: Schema Validation Failed **Error Prefix:** `Schema validation failed:` @@ -260,7 +284,7 @@ Schema validation failed: Missing required field 'schema_version' in output. Thi --- -### Exit Code 4: Normalization Failed (SPEC.md §3.1.2) +### Exit Code 4: Normalization Failed **Error Prefix:** `Normalization failed:` @@ -280,7 +304,7 @@ Normalization failed: Unable to parse markdown in issue #42 body. Check for malf --- -### Exit Code 5: File I/O Error (SPEC.md §3.1.2) +### Exit Code 5: File I/O Error **Error Prefix:** `File I/O error:` @@ -303,7 +327,9 @@ File I/O error: Cannot write to 'outputs/pdf_ready.json'. Ensure output director ## Markdown Normalization Rules -### Heading Synonym Mapping (SPEC.md §3.3.3) +### Heading Synonym Mapping + +See also [Contracts & Interfaces](../contracts.md#section-mapping-heading-synonyms) for the full reference. The service normalizes issue body headings to canonical section keys using case-insensitive matching: @@ -439,8 +465,8 @@ Pin collector and toolkit versions in CI/CD to ensure reproducibility: ## Additional Resources -- **SPEC.md**: Full system specification -- **Troubleshooting Guide**: `docs/troubleshooting.md` -- **Local Usage Recipe**: `docs/recipes/local-normalize-issues.md` -- **GitHub Actions Recipe**: `docs/recipes/github-actions-normalize-issues.yml` -- **Architecture Overview**: `docs/architecture.md` +- **[Contracts & Interfaces](../contracts.md)** — CLI reference, schemas, change control +- **[Troubleshooting](../troubleshooting.md)** — Exit codes, common errors, FAQ +- **[Recipe: Local usage](../recipes/local-normalize-issues.md)** — Run the CLI on your machine +- **[Recipe: GitHub Actions](../recipes/github-actions-normalize-issues.md)** — CI/CD workflow integration +- **[Architecture](../architecture.md)** — System overview, data flow pipeline diff --git a/docs/recipes/github-actions-normalize-issues.md b/docs/recipes/github-actions-normalize-issues.md index f1cda39..4951056 100644 --- a/docs/recipes/github-actions-normalize-issues.md +++ b/docs/recipes/github-actions-normalize-issues.md @@ -6,6 +6,19 @@ --- +- [Overview](#overview) +- [Prerequisites](#prerequisites) +- [Basic Workflow Setup](#basic-workflow-setup) +- [Customization Options](#customization-options) +- [Complete Pipeline: Collector → Builder → Generator](#complete-pipeline-collector--builder--generator) +- [Pipeline Orchestration Patterns](#pipeline-orchestration-patterns) +- [Error Handling](#error-handling) +- [Best Practices](#best-practices) +- [Troubleshooting](#troubleshooting) +- [Additional Resources](#additional-resources) + +--- + ## Overview This recipe shows how to run the `normalize-issues` service in GitHub Actions, either as a standalone job or as part of a complete living documentation pipeline (Collector → Builder → Generator). @@ -61,11 +74,8 @@ jobs: - name: Install living-doc-toolkit run: | - pip install -e packages/core - pip install -e packages/datasets_pdf - pip install -e packages/adapters/collector_gh - pip install -e packages/services/normalize_issues - pip install -e apps/cli + pip install --upgrade pip + pip install -r requirements.txt - name: Normalize issues run: | @@ -224,7 +234,8 @@ jobs: - name: Install living-doc-toolkit run: | - pip install living-doc-toolkit + pip install --upgrade pip + pip install -r requirements.txt - name: Normalize issues run: | @@ -444,7 +455,7 @@ Use `--verbose` flag in workflows for detailed logs: - name: Install living-doc-toolkit run: | pip install --upgrade pip - pip install living-doc-toolkit + pip install -r requirements.txt ``` ### Issue: "File not found: doc-issues.json" @@ -480,7 +491,7 @@ Use `--verbose` flag in workflows for detailed logs: ## Additional Resources -- **Workflow YAML**: `docs/recipes/github-actions-normalize-issues.yml` -- **Cookbook**: `docs/cookbooks/normalize-issues.md` -- **Troubleshooting**: `docs/troubleshooting.md` -- **GitHub Actions Documentation**: https://docs.github.com/en/actions +- **[Workflow YAML](github-actions-normalize-issues.yml)** — Ready-to-use workflow file +- **[Cookbook: normalize-issues](../cookbooks/normalize-issues.md)** — How detection, compatibility, and normalization work +- **[Troubleshooting](../troubleshooting.md)** — Exit codes, common errors, FAQ +- **[GitHub Actions Documentation](https://docs.github.com/en/actions)** — Official GitHub Actions docs diff --git a/docs/recipes/local-normalize-issues.md b/docs/recipes/local-normalize-issues.md index d22d6d7..8167f1b 100644 --- a/docs/recipes/local-normalize-issues.md +++ b/docs/recipes/local-normalize-issues.md @@ -6,6 +6,17 @@ --- +- [Prerequisites](#prerequisites) +- [Installation](#installation) +- [Example CLI Invocation](#example-cli-invocation) +- [Expected Output](#expected-output) +- [How to Verify Output](#how-to-verify-output) +- [Common Issues and Solutions](#common-issues-and-solutions) +- [Next Steps](#next-steps) +- [Additional Resources](#additional-resources) + +--- + ## Prerequisites ### Required @@ -39,39 +50,18 @@ git clone https://github.com/AbsaOSS/living-doc-toolkit.git cd living-doc-toolkit ``` -### Step 2: Install Core Package +### Step 2: Create and Activate a Virtual Environment ```bash -# Install core utilities -pip install -e packages/core +python3 -m venv .venv +source .venv/bin/activate ``` -### Step 3: Install Dataset Package +### Step 3: Install All Packages ```bash -# Install PDF dataset models -pip install -e packages/datasets_pdf -``` - -### Step 4: Install Adapter Package - -```bash -# Install collector-gh adapter -pip install -e packages/adapters/collector_gh -``` - -### Step 5: Install Service Package - -```bash -# Install normalize-issues service -pip install -e packages/services/normalize_issues -``` - -### Step 6: Install CLI - -```bash -# Install CLI wrapper -pip install -e apps/cli +pip install --upgrade pip +pip install -r requirements.txt ``` ### Verify Installation @@ -105,7 +95,7 @@ Options: ## Example CLI Invocation -### Basic Usage (SPEC.md §3.1.1) +### Basic Usage Convert `doc-issues.json` to `pdf_ready.json` with auto-detection: @@ -175,7 +165,9 @@ Successfully normalized /path/to/input/doc-issues.json -> /path/to/output/pdf_re **Exit Code:** `0` -### Output File Structure (SPEC.md §3.3.4) +### Output File Structure + +See [Contracts & Interfaces](../contracts.md#output-contract-pdf_readyjson) for the full schema reference. The generated `pdf_ready.json` follows this structure: @@ -443,7 +435,7 @@ living-doc normalize-issues \ ## Additional Resources -- **Cookbook**: `docs/cookbooks/normalize-issues.md` — Detailed service documentation -- **Troubleshooting**: `docs/troubleshooting.md` — Common errors and solutions -- **SPEC.md**: Full system specification -- **GitHub Actions Recipe**: `docs/recipes/github-actions-normalize-issues.yml` +- **[Contracts & Interfaces](../contracts.md)** — CLI reference, schemas, change control +- **[Cookbook: normalize-issues](../cookbooks/normalize-issues.md)** — How detection, compatibility, and normalization work +- **[Troubleshooting](../troubleshooting.md)** — Exit codes, common errors, FAQ +- **[Recipe: GitHub Actions](github-actions-normalize-issues.md)** — CI/CD workflow integration diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index 2d3ba77..501968f 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -16,7 +16,7 @@ ## Exit Codes Reference -All toolkit services follow the exit code convention defined in SPEC.md §3.1.2: +All toolkit services follow the exit code convention defined in [Contracts & Interfaces](contracts.md#exit-codes): | Exit Code | Condition | Error Prefix | Description | |-----------|-----------|--------------|-------------| @@ -310,7 +310,7 @@ Normalization failed: Section mapping failed for issue #42. Unknown heading form - Avoid special characters in headings - Use consistent heading levels (##, not ###) -3. **Check SPEC.md §3.3.3 for supported heading synonyms** +3. **Check [Contracts & Interfaces](contracts.md#section-mapping-heading-synonyms) for supported heading synonyms** --- @@ -497,7 +497,7 @@ Future versions may support multi-source input. ### Q: Can I customize the section mapping? -**A:** Not directly. The section mapping is defined by SPEC.md §3.3.3. However, you can: +**A:** Not directly. The section mapping is defined in [Contracts & Interfaces](contracts.md#section-mapping-heading-synonyms). However, you can: 1. Use standard heading names in issue bodies 2. Unknown headings are appended to `description` @@ -612,7 +612,7 @@ living-doc normalize-issues \ jq '.issues | length' doc-issues.json ``` -2. **Expected performance (SPEC.md §5.5):** +2. **Expected performance (see [Performance Budgets](contracts.md#performance-budgets)):** - 100 issues: < 10 seconds - 1000 issues: < 60 seconds @@ -651,8 +651,8 @@ docker run -v $(pwd):/data living-doc-toolkit \ ## Additional Resources -- **Cookbook**: `docs/cookbooks/normalize-issues.md` — Service documentation -- **Local Recipe**: `docs/recipes/local-normalize-issues.md` — Local usage guide -- **GitHub Actions Recipe**: `docs/recipes/github-actions-normalize-issues.yml` — CI/CD integration -- **SPEC.md**: Full system specification -- **GitHub Issues**: https://github.com/AbsaOSS/living-doc-toolkit/issues +- **[Cookbook: normalize-issues](cookbooks/normalize-issues.md)** — How detection, compatibility, and normalization work +- **[Recipe: Local usage](recipes/local-normalize-issues.md)** — Run the CLI on your machine +- **[Recipe: GitHub Actions](recipes/github-actions-normalize-issues.md)** — CI/CD workflow integration +- **[Contracts & Interfaces](contracts.md)** — CLI reference, schemas, change control +- **[GitHub Issues](https://github.com/AbsaOSS/living-doc-toolkit/issues)** — Report bugs or request features diff --git a/packages/adapters/collector_gh/pyproject.toml b/packages/adapters/collector_gh/pyproject.toml index 638f1be..f13956c 100644 --- a/packages/adapters/collector_gh/pyproject.toml +++ b/packages/adapters/collector_gh/pyproject.toml @@ -11,7 +11,7 @@ license = {text = "Apache-2.0"} authors = [ {name = "AbsaOSS", email = "miroslav.pojer@absa.africa"} ] -requires-python = ">=3.11" +requires-python = ">=3.14" dependencies = [ "pydantic>=2.10.0", "packaging" diff --git a/packages/adapters/collector_gh/src/living_doc_adapter_collector_gh/parser.py b/packages/adapters/collector_gh/src/living_doc_adapter_collector_gh/parser.py index 327e67c..5eda1a4 100644 --- a/packages/adapters/collector_gh/src/living_doc_adapter_collector_gh/parser.py +++ b/packages/adapters/collector_gh/src/living_doc_adapter_collector_gh/parser.py @@ -75,12 +75,20 @@ def parse(payload: dict) -> AdapterResult: ) # Parse issues into adapter items - issues = payload.get("issues", []) + # Support both dict format (keyed by issue ID) and list format + issues_data = payload.get("issues", {}) + if isinstance(issues_data, dict): + issues_list = list(issues_data.values()) + else: + issues_list = issues_data if isinstance(issues_data, list) else [] + items = [] - for issue in issues: + for issue in issues_list: try: + # Support both 'number' and 'issue_number' field names + issue_number = issue.get("number") or issue.get("issue_number") item = AdapterItem( - id=f"github:{source_repo}#{issue['number']}", + id=f"github:{source_repo}#{issue_number}", title=issue["title"], state=issue["state"], tags=issue.get("labels", []), @@ -93,7 +101,8 @@ def parse(payload: dict) -> AdapterResult: ) items.append(item) except (KeyError, TypeError) as e: - raise AdapterError(f"Failed to parse issue {issue.get('number', 'unknown')}: {e}") from e + issue_number = issue.get("number") or issue.get("issue_number", "unknown") + raise AdapterError(f"Failed to parse issue {issue_number}: {e}") from e return AdapterResult(items=items, metadata=metadata, warnings=warnings) diff --git a/packages/adapters/collector_gh/tests/fixtures/collector_v1.0.0/input/doc-issues.json b/packages/adapters/collector_gh/tests/fixtures/collector_v1.0.0/input/doc-issues.json index 9842e02..90425eb 100644 --- a/packages/adapters/collector_gh/tests/fixtures/collector_v1.0.0/input/doc-issues.json +++ b/packages/adapters/collector_gh/tests/fixtures/collector_v1.0.0/input/doc-issues.json @@ -20,9 +20,9 @@ "enterprise": null } }, - "issues": [ - { - "number": 1, + "issues": { + "AbsaOSS/example-project/1": { + "issue_number": 1, "title": "User Authentication with OAuth2", "state": "open", "labels": ["documentation", "priority:high", "authentication"], @@ -31,8 +31,8 @@ "updated_at": "2026-01-20T14:30:00Z", "body": "## Description\nAs a user, I want to authenticate using OAuth2 providers so that I can securely access the application without creating a separate account.\n\n## Business Value\nReduces user friction during onboarding and improves security by leveraging trusted identity providers.\n\n## Acceptance Criteria\n- User can select OAuth2 provider (Google, Microsoft, GitHub)\n- Successful authentication redirects to application dashboard\n- Failed authentication shows appropriate error message\n- User profile is created/updated from OAuth2 data" }, - { - "number": 2, + "AbsaOSS/example-project/2": { + "issue_number": 2, "title": "Export Data to CSV Format", "state": "open", "labels": ["documentation", "feature", "priority:medium"], @@ -41,8 +41,8 @@ "updated_at": "2026-01-22T10:45:00Z", "body": "## Description\nAs a data analyst, I want to export report data to CSV format so that I can perform further analysis in Excel or other tools.\n\n## Preconditions\n- User has at least one report generated\n- User has permission to export data\n\n## Acceptance Criteria\n- Export button is visible on report page\n- CSV file includes all report columns\n- CSV file respects user's date/time format preferences\n- Export completes within 30 seconds for reports up to 10,000 rows\n\n## User Guide\n1. Navigate to the report page\n2. Click the \"Export to CSV\" button in the top-right corner\n3. Wait for download to complete\n4. Open the CSV file in your preferred tool" }, - { - "number": 3, + "AbsaOSS/example-project/3": { + "issue_number": 3, "title": "Real-time Notification System", "state": "closed", "labels": ["documentation", "enhancement", "notifications"], @@ -51,8 +51,8 @@ "updated_at": "2026-01-25T16:00:00Z", "body": "## Overview\nImplement a real-time notification system that alerts users about important events without requiring page refresh.\n\n## Business Value\nImproves user engagement and ensures timely awareness of critical updates.\n\n## Done Criteria\n| Criteria | Status |\n|----------|--------|\n| WebSocket connection established | ✓ |\n| Notifications appear as toast messages | ✓ |\n| Users can dismiss notifications | ✓ |\n| Notification history is accessible | ✓ |\n\n## Connections\n- Related to #5 (Push Notification Support)\n- Depends on #12 (WebSocket Infrastructure)" }, - { - "number": 4, + "AbsaOSS/example-project/4": { + "issue_number": 4, "title": "Multi-language Support for UI", "state": "open", "labels": ["documentation", "i18n", "priority:low"], @@ -61,8 +61,8 @@ "updated_at": "2026-01-23T09:20:00Z", "body": "## Summary\nAdd internationalization support to allow users to switch between English, French, and German languages.\n\n## Prerequisites\n- Translation keys are defined in locale files\n- Language selector is added to user preferences\n\n## AC\n- User can select preferred language from profile settings\n- UI text updates immediately after language change\n- Selected language persists across sessions\n- All user-facing text is translatable (no hardcoded strings)\n\n## Last Edited\nUpdated by alice.smith@example.com on 2026-01-23" }, - { - "number": 5, + "AbsaOSS/example-project/5": { + "issue_number": 5, "title": "Advanced Search with Filters", "state": "open", "labels": ["documentation", "search", "ux"], @@ -71,8 +71,8 @@ "updated_at": "2026-01-24T11:30:00Z", "body": "## Description\nAs a power user, I want to use advanced search filters to quickly find specific records based on multiple criteria.\n\n## Why\nReduces time spent manually browsing through large datasets and improves productivity.\n\n## Acceptance Criteria\n- Search supports text, date range, and category filters\n- Filters can be combined with AND/OR logic\n- Search results update in real-time as filters are applied\n- Search history is saved for quick access\n- Export search results to CSV" }, - { - "number": 6, + "AbsaOSS/example-project/6": { + "issue_number": 6, "title": "Dashboard Customization", "state": "open", "labels": ["documentation", "dashboard", "personalization"], @@ -81,8 +81,8 @@ "updated_at": "2026-01-25T08:15:00Z", "body": "## Description\nAllow users to customize their dashboard by adding, removing, and rearranging widgets.\n\n## Business Value\nEnhances user experience by providing personalized views tailored to individual workflows.\n\n## Preconditions\n- User is logged in\n- At least 3 widget types are available\n\n## Acceptance Criteria\n- Users can drag and drop widgets to reorder them\n- Users can add widgets from a widget library\n- Users can remove widgets they don't need\n- Dashboard layout is saved automatically\n- Layout persists across devices\n\n## Instructions\n### Adding a Widget\n1. Click \"Add Widget\" button\n2. Select widget type from dropdown\n3. Configure widget settings\n4. Click \"Save\"\n\n### Removing a Widget\n1. Hover over widget\n2. Click the X icon in the top-right corner\n3. Confirm deletion" }, - { - "number": 7, + "AbsaOSS/example-project/7": { + "issue_number": 7, "title": "Audit Log for Compliance", "state": "closed", "labels": ["documentation", "compliance", "security"], @@ -91,8 +91,8 @@ "updated_at": "2026-01-26T12:00:00Z", "body": "## Description\nImplement comprehensive audit logging to track all user actions for compliance and security purposes.\n\n## Business Value\nEnsures regulatory compliance (GDPR, SOX) and provides security incident investigation capabilities.\n\n## Acceptance Criteria\n- All CRUD operations are logged with timestamp, user ID, and action details\n- Logs are immutable and stored in a tamper-proof format\n- Logs are retained for 7 years\n- Administrators can search and filter audit logs\n- Failed login attempts are logged\n\n## Related\n- Links to compliance documentation: https://example.com/compliance\n- See also: #15 (Role-Based Access Control)" }, - { - "number": 8, + "AbsaOSS/example-project/8": { + "issue_number": 8, "title": "Email Notification Templates", "state": "open", "labels": ["documentation", "notifications", "email"], @@ -101,8 +101,8 @@ "updated_at": "2026-01-24T15:45:00Z", "body": "## Overview\nCreate customizable email templates for various notification types (welcome, password reset, report ready, etc.).\n\n## Value\nProvides consistent branding and messaging across all email communications.\n\n## Setup\n- Email service (SendGrid/Mailgun) is configured\n- Template variables are documented\n\n## Done Criteria\n- At least 5 email templates are available\n- Templates support variable substitution (user name, links, etc.)\n- Templates are responsive (mobile-friendly)\n- Administrators can preview templates before sending\n- Template changes are version-controlled" }, - { - "number": 9, + "AbsaOSS/example-project/9": { + "issue_number": 9, "title": "Two-Factor Authentication (2FA)", "state": "open", "labels": ["documentation", "security", "priority:high"], @@ -111,8 +111,8 @@ "updated_at": "2026-01-26T10:00:00Z", "body": "## Description\nAs a security-conscious user, I want to enable two-factor authentication to add an extra layer of protection to my account.\n\n## Business Value\nSignificantly reduces the risk of unauthorized account access and data breaches.\n\n## Acceptance Criteria\n- Users can enable 2FA via authenticator apps (Google Authenticator, Authy)\n- Backup codes are generated and displayed once\n- 2FA can be disabled with password confirmation\n- Failed 2FA attempts are rate-limited\n- SMS-based 2FA is supported as fallback\n\n## User Guide\n### Enabling 2FA\n1. Go to Account Settings > Security\n2. Click \"Enable Two-Factor Authentication\"\n3. Scan QR code with authenticator app\n4. Enter verification code to confirm\n5. Save backup codes in a secure location\n\n### Disabling 2FA\n1. Go to Account Settings > Security\n2. Click \"Disable Two-Factor Authentication\"\n3. Enter current password\n4. Enter 2FA code\n5. Click \"Confirm\"\n\n## History\nInitially proposed in Q4 2025, prioritized for Q1 2026 release." }, - { - "number": 10, + "AbsaOSS/example-project/10": { + "issue_number": 10, "title": "API Rate Limiting", "state": "closed", "labels": ["documentation", "api", "performance"], @@ -121,8 +121,8 @@ "updated_at": "2026-01-27T09:30:00Z", "body": "## Description\nImplement rate limiting for all API endpoints to prevent abuse and ensure fair resource allocation.\n\n## Business Value\nProtects system availability and prevents denial-of-service attacks.\n\n## Acceptance Criteria\n- Rate limit is 100 requests per minute per API key\n- Exceeding rate limit returns 429 status code with Retry-After header\n- Rate limit counters reset every minute\n- Premium users have higher rate limits (500 requests/minute)\n- Rate limit status is visible in API response headers\n\n## Connections\n- Related to API documentation: https://api.example.com/docs\n- See also: #18 (API Key Management)" }, - { - "number": 11, + "AbsaOSS/example-project/11": { + "issue_number": 11, "title": "Scheduled Report Generation", "state": "open", "labels": ["documentation", "reports", "automation"], @@ -131,8 +131,8 @@ "updated_at": "2026-01-26T14:20:00Z", "body": "## Description\nAllow users to schedule automatic report generation and delivery via email at specified intervals.\n\n## Business Value\nSaves time by automating repetitive reporting tasks and ensures stakeholders receive timely updates.\n\n## Preconditions\n- User has created at least one report template\n- Email service is configured and operational\n\n## AC\n- Users can schedule reports (daily, weekly, monthly)\n- Reports are generated at the specified time\n- Generated reports are emailed as PDF attachments\n- Users can pause or cancel scheduled reports\n- Failed report generations trigger alerts\n\n## How To\n1. Open the report you want to schedule\n2. Click \"Schedule\" button\n3. Select frequency (daily/weekly/monthly)\n4. Choose delivery time\n5. Enter recipient email addresses\n6. Click \"Save Schedule\"\n\n## Changes\nScheduling logic was refactored in v2.3 to improve reliability." }, - { - "number": 12, + "AbsaOSS/example-project/12": { + "issue_number": 12, "title": "Dark Mode Theme", "state": "open", "labels": ["documentation", "ui", "accessibility"], @@ -141,5 +141,5 @@ "updated_at": "2026-01-27T11:00:00Z", "body": "## Summary\nImplement a dark mode theme option to reduce eye strain and improve accessibility for users in low-light environments.\n\n## Value\nEnhances user comfort and accessibility, especially for users who work extended hours.\n\n## Acceptance Criteria\n- Theme toggle is available in user preferences\n- Dark mode applies to all UI components\n- Theme preference persists across sessions\n- System theme preference is detected and applied automatically\n- Contrast ratios meet WCAG AA standards\n\n## Links\n- Design mockups: https://figma.com/darkmode\n- WCAG guidelines: https://www.w3.org/WAI/WCAG21/quickref/" } - ] + } } diff --git a/packages/adapters/collector_gh/tests/fixtures/collector_v1.2.0/input/doc-issues.json b/packages/adapters/collector_gh/tests/fixtures/collector_v1.2.0/input/doc-issues.json index 7e31668..bdb534a 100644 --- a/packages/adapters/collector_gh/tests/fixtures/collector_v1.2.0/input/doc-issues.json +++ b/packages/adapters/collector_gh/tests/fixtures/collector_v1.2.0/input/doc-issues.json @@ -20,9 +20,9 @@ "enterprise": "AbsaCorp" } }, - "issues": [ - { - "number": 20, + "issues": { + "AbsaOSS/example-project/20": { + "issue_number": 20, "title": "Shopping Cart Persistence", "state": "open", "labels": ["documentation", "e-commerce", "priority:high"], @@ -31,8 +31,8 @@ "updated_at": "2026-01-27T15:00:00Z", "body": "## Description\nAs a customer, I want my shopping cart to persist across sessions so that I don't lose my selected items if I close the browser.\n\n## Business Value\nReduces cart abandonment rates and improves conversion by preserving user shopping intent.\n\n## Acceptance Criteria\n- Cart items are saved to database after each addition/removal\n- Cart is restored when user logs in\n- Cart items expire after 30 days of inactivity\n- Guest carts are converted to user carts upon registration\n- Out-of-stock items show appropriate messaging" }, - { - "number": 21, + "AbsaOSS/example-project/21": { + "issue_number": 21, "title": "Product Recommendation Engine", "state": "open", "labels": ["documentation", "ml", "personalization"], @@ -41,8 +41,8 @@ "updated_at": "2026-01-28T10:45:00Z", "body": "## Overview\nImplement a machine learning-based product recommendation system that suggests relevant items based on browsing history and purchase patterns.\n\n## Why\nIncreases average order value and improves customer satisfaction through personalized shopping experiences.\n\n## Prerequisites\n- User behavior tracking is implemented\n- Product catalog is indexed and searchable\n- ML model training pipeline is established\n\n## Done Criteria\n- Recommendations appear on product pages and homepage\n- Recommendations update in real-time based on browsing behavior\n- Click-through rate on recommendations is measured\n- Recommendation quality improves over time\n- Users can dismiss recommendations" }, - { - "number": 22, + "AbsaOSS/example-project/22": { + "issue_number": 22, "title": "Payment Gateway Integration", "state": "closed", "labels": ["documentation", "payments", "integration"], @@ -51,8 +51,8 @@ "updated_at": "2026-01-29T14:30:00Z", "body": "## Description\nIntegrate Stripe payment gateway to support credit card, debit card, and digital wallet payments.\n\n## Business Value\nEnables secure online transactions and supports multiple payment methods for customer convenience.\n\n## Acceptance Criteria\n| Payment Method | Status | Notes |\n|----------------|--------|-------|\n| Credit Cards | ✓ | Visa, Mastercard, Amex |\n| Debit Cards | ✓ | All major providers |\n| Apple Pay | ✓ | iOS only |\n| Google Pay | ✓ | Android and Web |\n| PayPal | Pending | Integration in progress |\n\n## User Guide\n### Making a Payment\n1. Add items to cart\n2. Proceed to checkout\n3. Select payment method\n4. Enter payment details\n5. Review and confirm order\n6. Receive confirmation email\n\n## Related\n- Stripe API documentation: https://stripe.com/docs\n- PCI compliance: #45" }, - { - "number": 23, + "AbsaOSS/example-project/23": { + "issue_number": 23, "title": "Inventory Management Dashboard", "state": "open", "labels": ["documentation", "inventory", "admin"], @@ -61,8 +61,8 @@ "updated_at": "2026-01-28T09:20:00Z", "body": "## Summary\nCreate an admin dashboard for managing product inventory, tracking stock levels, and receiving low-stock alerts.\n\n## Value\nPrevents stockouts and overstocking by providing real-time inventory visibility.\n\n## AC\n- Dashboard displays current stock levels for all products\n- Low-stock alerts are sent when quantity falls below threshold\n- Inventory can be updated manually or via bulk import\n- Stock movement history is tracked and auditable\n- Dashboard is accessible to users with 'inventory_manager' role" }, - { - "number": 24, + "AbsaOSS/example-project/24": { + "issue_number": 24, "title": "Customer Review and Rating System", "state": "open", "labels": ["documentation", "reviews", "social"], @@ -71,8 +71,8 @@ "updated_at": "2026-01-29T11:30:00Z", "body": "## Description\nAs a customer, I want to read and write product reviews to make informed purchasing decisions and share my experiences.\n\n## Business Value\nBuilds trust and credibility, provides social proof, and increases conversion rates.\n\n## Preconditions\n- User has purchased the product\n- User is logged in\n\n## Acceptance Criteria\n- Customers can rate products from 1 to 5 stars\n- Customers can write text reviews (up to 1000 characters)\n- Reviews can include photos (up to 5 images)\n- Reviews are moderated before publication\n- Average rating is displayed on product page\n- Reviews can be sorted by date, rating, or helpfulness\n\n## Instructions\n### Writing a Review\n1. Navigate to product page\n2. Click \"Write a Review\"\n3. Select star rating\n4. Write review text\n5. Optionally upload photos\n6. Click \"Submit Review\"\n7. Review enters moderation queue\n\n### Flagging Inappropriate Reviews\n1. Click \"Report\" on the review\n2. Select reason (spam, offensive, etc.)\n3. Provide additional details\n4. Submit report" }, - { - "number": 25, + "AbsaOSS/example-project/25": { + "issue_number": 25, "title": "Order Tracking and Notifications", "state": "closed", "labels": ["documentation", "orders", "notifications"], @@ -81,8 +81,8 @@ "updated_at": "2026-01-30T08:00:00Z", "body": "## Description\nProvide customers with real-time order tracking and automated notifications at each stage of the fulfillment process.\n\n## Business Value\nReduces customer support inquiries and improves customer satisfaction through transparency.\n\n## Acceptance Criteria\n- Order status is updated in real-time (processing, shipped, delivered)\n- Customers receive email notifications at each status change\n- Tracking link is provided for shipped orders\n- Estimated delivery date is displayed\n- Customers can view order history\n\n## Connections\n- Integration with shipping providers: #48\n- Email notification templates: #8" }, - { - "number": 26, + "AbsaOSS/example-project/26": { + "issue_number": 26, "title": "Wishlist Functionality", "state": "open", "labels": ["documentation", "wishlist", "feature"], @@ -91,8 +91,8 @@ "updated_at": "2026-01-29T15:45:00Z", "body": "## Overview\nAllow customers to save products to a wishlist for future purchase consideration.\n\n## Why\nEncourages repeat visits and provides insights into customer preferences.\n\n## Setup\n- User authentication is implemented\n- Product catalog is available\n\n## Done Criteria\n- Users can add/remove products to wishlist\n- Wishlist is accessible from user profile\n- Wishlist can be shared via unique URL\n- Price drop notifications for wishlist items\n- Move to cart functionality available" }, - { - "number": 27, + "AbsaOSS/example-project/27": { + "issue_number": 27, "title": "Coupon and Discount System", "state": "open", "labels": ["documentation", "promotions", "pricing"], @@ -101,8 +101,8 @@ "updated_at": "2026-01-30T10:15:00Z", "body": "## Description\nAs a marketing manager, I want to create and manage promotional coupons to drive sales and customer engagement.\n\n## Business Value\nIncreases sales during promotional periods and enables targeted marketing campaigns.\n\n## Acceptance Criteria\n- Coupons can be percentage-based or fixed amount\n- Coupons have start and end dates\n- Coupons can have usage limits (per customer and total)\n- Coupons can be restricted to specific products or categories\n- Invalid or expired coupons show appropriate error messages\n- Coupon usage is tracked and reportable\n\n## User Guide\n### Creating a Coupon\n1. Navigate to Admin > Promotions > Coupons\n2. Click \"Create New Coupon\"\n3. Enter coupon code (e.g., SUMMER2026)\n4. Select discount type and amount\n5. Set validity period\n6. Configure usage limits\n7. Click \"Save\"\n\n### Applying a Coupon (Customer)\n1. Add items to cart\n2. Proceed to checkout\n3. Enter coupon code in the promo field\n4. Click \"Apply\"\n5. Discount is reflected in order total" }, - { - "number": 28, + "AbsaOSS/example-project/28": { + "issue_number": 28, "title": "Gift Card System", "state": "open", "labels": ["documentation", "gift-cards", "payments"], @@ -111,8 +111,8 @@ "updated_at": "2026-01-30T12:00:00Z", "body": "## Summary\nImplement a gift card system allowing customers to purchase, send, and redeem gift cards.\n\n## Value\nGenerates additional revenue and provides gifting options for customers.\n\n## Prerequisites\n- Payment gateway is integrated\n- Email notification system is functional\n\n## AC\n- Customers can purchase gift cards in denominations of $25, $50, $100, or custom amounts\n- Gift cards can be sent via email with personalized messages\n- Gift cards have unique codes and are validated at checkout\n- Gift card balance is tracked and displayed\n- Partial redemptions are supported\n- Gift cards expire after 2 years" }, - { - "number": 29, + "AbsaOSS/example-project/29": { + "issue_number": 29, "title": "Multi-Currency Support", "state": "closed", "labels": ["documentation", "i18n", "pricing"], @@ -121,8 +121,8 @@ "updated_at": "2026-01-31T09:30:00Z", "body": "## Description\nSupport multiple currencies to enable international sales and provide localized pricing.\n\n## Business Value\nExpands market reach and improves conversion rates for international customers.\n\n## Acceptance Criteria\n- Support for USD, EUR, GBP, and JPY\n- Currency is detected based on user location\n- Users can manually switch currency\n- Exchange rates are updated daily\n- Prices are displayed in selected currency throughout the site\n- Orders are processed in customer's selected currency\n\n## Links\n- Currency conversion API: https://exchangerate-api.com\n- Pricing strategy documentation" }, - { - "number": 30, + "AbsaOSS/example-project/30": { + "issue_number": 30, "title": "Customer Loyalty Program", "state": "open", "labels": ["documentation", "loyalty", "gamification"], @@ -131,8 +131,8 @@ "updated_at": "2026-01-31T11:00:00Z", "body": "## Description\nAs a returning customer, I want to earn loyalty points for purchases that can be redeemed for discounts on future orders.\n\n## Business Value\nIncreases customer retention and lifetime value through rewards and incentives.\n\n## Acceptance Criteria\n- Customers earn 1 point per $1 spent\n- Points can be redeemed at a rate of 100 points = $5 discount\n- Points balance is visible in user profile\n- Points transactions are logged (earned, redeemed, expired)\n- Bonus point promotions can be configured\n- Points expire after 12 months of inactivity\n\n## How To\n### Earning Points\n- Complete purchases (automatic)\n- Refer friends (bonus points)\n- Write product reviews (bonus points)\n- Complete profile information (one-time bonus)\n\n### Redeeming Points\n1. Go to checkout\n2. View available points balance\n3. Select points to redeem\n4. Discount is applied to order total\n\n## History\nLoyalty program launched in Q2 2026 with initial tier structure." }, - { - "number": 31, + "AbsaOSS/example-project/31": { + "issue_number": 31, "title": "Product Comparison Feature", "state": "open", "labels": ["documentation", "comparison", "ux"], @@ -141,5 +141,5 @@ "updated_at": "2026-01-31T14:20:00Z", "body": "## Overview\nAllow customers to compare specifications and features of up to 4 products side-by-side.\n\n## Why\nHelps customers make informed decisions and reduces decision fatigue.\n\n## Setup\n- Product specifications are standardized in the database\n- Comparison UI is designed and approved\n\n## Done Criteria\n- Users can add products to comparison list from any page\n- Comparison view shows products in a table format\n- Key specifications are highlighted\n- Differences are visually emphasized\n- Comparison can be shared via URL\n- Comparison list is limited to 4 products" } - ] + } } diff --git a/packages/adapters/collector_gh/tests/test_parser.py b/packages/adapters/collector_gh/tests/test_parser.py index f4e1169..2677e5e 100644 --- a/packages/adapters/collector_gh/tests/test_parser.py +++ b/packages/adapters/collector_gh/tests/test_parser.py @@ -49,9 +49,9 @@ def minimal_payload(self): "enterprise": None, }, }, - "issues": [ - { - "number": 1, + "issues": { + "owner/repo/1": { + "issue_number": 1, "title": "Test Issue", "state": "open", "labels": ["test"], @@ -60,7 +60,7 @@ def minimal_payload(self): "updated_at": "2026-01-02T00:00:00Z", "body": "Test body", } - ], + }, } def test_parse_v1_0_0_fixture(self, fixture_v1_0_0): @@ -172,7 +172,7 @@ def test_parse_minimal_payload(self, minimal_payload): def test_parse_with_missing_labels(self, minimal_payload): """Test parsing when labels are missing from issue.""" - minimal_payload["issues"][0].pop("labels") + minimal_payload["issues"]["owner/repo/1"].pop("labels") result = parse(minimal_payload) assert len(result.items) == 1 @@ -180,7 +180,7 @@ def test_parse_with_missing_labels(self, minimal_payload): def test_parse_with_missing_body(self, minimal_payload): """Test parsing when body is missing from issue.""" - minimal_payload["issues"][0].pop("body") + minimal_payload["issues"]["owner/repo/1"].pop("body") result = parse(minimal_payload) assert len(result.items) == 1 @@ -217,7 +217,7 @@ def test_parse_with_closed_issue(self, fixture_v1_0_0): def test_parse_missing_issue_field_raises_error(self, minimal_payload): """Test that missing required issue field raises AdapterError.""" # Remove required field from issue - del minimal_payload["issues"][0]["title"] + del minimal_payload["issues"]["owner/repo/1"]["title"] with pytest.raises(AdapterError) as exc_info: parse(minimal_payload) @@ -225,14 +225,14 @@ def test_parse_missing_issue_field_raises_error(self, minimal_payload): def test_parse_missing_metadata_raises_error(self): """Test that missing metadata raises AdapterError.""" - payload = {"issues": []} + payload = {"issues": {}} with pytest.raises(AdapterError): parse(payload) - def test_parse_empty_issues_list(self, minimal_payload): - """Test parsing with empty issues list.""" - minimal_payload["issues"] = [] + def test_parse_empty_issues_dict(self, minimal_payload): + """Test parsing with empty issues dict.""" + minimal_payload["issues"] = {} result = parse(minimal_payload) assert len(result.items) == 0 diff --git a/packages/core/pyproject.toml b/packages/core/pyproject.toml index 7513e3d..e018117 100644 --- a/packages/core/pyproject.toml +++ b/packages/core/pyproject.toml @@ -11,7 +11,7 @@ license = {text = "Apache-2.0"} authors = [ {name = "AbsaOSS", email = "miroslav.pojer@absa.africa"} ] -requires-python = ">=3.11" +requires-python = ">=3.14" dependencies = [ "click>=8.1.0,<9.0.0" ] diff --git a/packages/datasets_pdf/pyproject.toml b/packages/datasets_pdf/pyproject.toml index 40992c8..13ca9e8 100644 --- a/packages/datasets_pdf/pyproject.toml +++ b/packages/datasets_pdf/pyproject.toml @@ -11,7 +11,7 @@ license = {text = "Apache-2.0"} authors = [ {name = "AbsaOSS", email = "opensource@absa.africa"} ] -requires-python = ">=3.12" +requires-python = ">=3.14" dependencies = [ "pydantic>=2.10.0,<3.0.0", "jsonschema>=4.20.0,<5.0.0" @@ -31,7 +31,7 @@ where = ["src"] [tool.black] line-length = 120 -target-version = ['py312'] +target-version = ['py314'] extend-exclude = ''' ^( tests/ @@ -41,7 +41,7 @@ extend-exclude = ''' [tool.mypy] check_untyped_defs = true -python_version = "3.12" +python_version = "3.14" exclude = '^(tests/|verifications/)' [tool.pytest.ini_options] diff --git a/packages/services/normalize_issues/pyproject.toml b/packages/services/normalize_issues/pyproject.toml index 0a44967..f73c878 100644 --- a/packages/services/normalize_issues/pyproject.toml +++ b/packages/services/normalize_issues/pyproject.toml @@ -11,7 +11,7 @@ license = {text = "Apache-2.0"} authors = [ {name = "AbsaOSS", email = "opensource@absa.africa"} ] -requires-python = ">=3.11" +requires-python = ">=3.14" dependencies = [ "living-doc-core", "living-doc-datasets-pdf", diff --git a/packages/services/normalize_issues/tests/test_service.py b/packages/services/normalize_issues/tests/test_service.py index 59e3362..ec5d126 100644 --- a/packages/services/normalize_issues/tests/test_service.py +++ b/packages/services/normalize_issues/tests/test_service.py @@ -35,9 +35,9 @@ def test_run_service_valid_input(tmp_path): "enterprise": None, }, }, - "issues": [ - { - "number": 123, + "issues": { + "owner/repo/123": { + "issue_number": 123, "title": "Test Issue", "state": "open", "labels": ["enhancement"], @@ -46,7 +46,7 @@ def test_run_service_valid_input(tmp_path): "updated_at": "2026-01-02T00:00:00Z", "body": "## Description\nThis is a test issue.", } - ], + }, } input_file = tmp_path / "input.json" @@ -108,7 +108,7 @@ def test_run_service_adapter_detection_failure(tmp_path): # Create input with wrong metadata input_data = { "metadata": {"generator": {"name": "unknown-adapter", "version": "1.0.0"}}, - "issues": [], + "issues": {}, } input_file = tmp_path / "input.json" @@ -138,9 +138,9 @@ def test_run_service_explicit_adapter(tmp_path): }, "source": {"systems": ["github"], "repositories": ["owner/repo"], "organization": None, "enterprise": None}, }, - "issues": [ - { - "number": 1, + "issues": { + "owner/repo/1": { + "issue_number": 1, "title": "Test", "state": "open", "labels": [], @@ -149,7 +149,7 @@ def test_run_service_explicit_adapter(tmp_path): "updated_at": "2026-01-01T00:00:00Z", "body": None, } - ], + }, } input_file = tmp_path / "input.json" @@ -166,7 +166,7 @@ def test_run_service_explicit_adapter(tmp_path): def test_run_service_unsupported_adapter(tmp_path): """Test error handling for unsupported adapter.""" - input_data = {"metadata": {"generator": {"name": "some-adapter", "version": "1.0.0"}}, "issues": []} + input_data = {"metadata": {"generator": {"name": "some-adapter", "version": "1.0.0"}}, "issues": {}} input_file = tmp_path / "input.json" output_file = tmp_path / "output.json" @@ -197,7 +197,7 @@ def test_run_service_empty_items(tmp_path): }, "source": {"systems": ["github"], "repositories": ["owner/repo"], "organization": None, "enterprise": None}, }, - "issues": [], + "issues": {}, } input_file = tmp_path / "input.json" @@ -233,9 +233,9 @@ def test_run_service_multiple_items(tmp_path): }, "source": {"systems": ["github"], "repositories": ["owner/repo"], "organization": None, "enterprise": None}, }, - "issues": [ - { - "number": i, + "issues": { + f"owner/repo/{i}": { + "issue_number": i, "title": f"Issue {i}", "state": "open", "labels": [], @@ -245,7 +245,7 @@ def test_run_service_multiple_items(tmp_path): "body": f"## Description\nIssue {i} content.", } for i in range(1, 4) - ], + }, } input_file = tmp_path / "input.json" diff --git a/requirements.txt b/requirements.txt new file mode 100644 index 0000000..e53f7e5 --- /dev/null +++ b/requirements.txt @@ -0,0 +1,5 @@ +-e packages/core +-e packages/datasets_pdf +-e packages/adapters/collector_gh +-e packages/services/normalize_issues +-e apps/cli diff --git a/tests/fixtures/collector_gh/v0.9.0/input/doc-issues.json b/tests/fixtures/collector_gh/v0.9.0/input/doc-issues.json index 193d33f..cf81b30 100644 --- a/tests/fixtures/collector_gh/v0.9.0/input/doc-issues.json +++ b/tests/fixtures/collector_gh/v0.9.0/input/doc-issues.json @@ -14,32 +14,40 @@ "sha": "abc123def456" }, "source": { - "systems": ["GitHub"], - "repositories": ["AbsaOSS/test-project"], + "systems": [ + "GitHub" + ], + "repositories": [ + "AbsaOSS/test-project" + ], "organization": "AbsaOSS", "enterprise": null } }, - "issues": [ - { - "number": 1, + "issues": { + "AbsaOSS/test-project/1": { "title": "Implement basic feature", "state": "open", - "labels": ["enhancement"], + "labels": [ + "enhancement" + ], "html_url": "https://github.com/AbsaOSS/test-project/issues/1", "created_at": "2026-01-01T10:00:00Z", "updated_at": "2026-01-02T12:00:00Z", - "body": "## Description\nImplement a basic feature for testing." + "body": "## Description\nImplement a basic feature for testing.", + "issue_number": 1 }, - { - "number": 2, + "AbsaOSS/test-project/2": { "title": "Fix simple bug", "state": "closed", - "labels": ["bug"], + "labels": [ + "bug" + ], "html_url": "https://github.com/AbsaOSS/test-project/issues/2", "created_at": "2026-01-03T14:00:00Z", "updated_at": "2026-01-04T16:00:00Z", - "body": "## Summary\nFix a simple bug in the system." + "body": "## Summary\nFix a simple bug in the system.", + "issue_number": 2 } - ] -} + } +} \ No newline at end of file diff --git a/tests/fixtures/collector_gh/v1.0.0/input/doc-issues.json b/tests/fixtures/collector_gh/v1.0.0/input/doc-issues.json index 48b2476..468cd72 100644 --- a/tests/fixtures/collector_gh/v1.0.0/input/doc-issues.json +++ b/tests/fixtures/collector_gh/v1.0.0/input/doc-issues.json @@ -14,162 +14,211 @@ "sha": "abc123def456789" }, "source": { - "systems": ["GitHub"], - "repositories": ["AbsaOSS/living-doc-project"], + "systems": [ + "GitHub" + ], + "repositories": [ + "AbsaOSS/living-doc-project" + ], "organization": "AbsaOSS", "enterprise": null } }, - "issues": [ - { - "number": 101, + "issues": { + "AbsaOSS/living-doc-project/101": { "title": "Implement user authentication with SSO", "state": "open", - "labels": ["authentication", "priority:high", "documentation"], + "labels": [ + "authentication", + "priority:high", + "documentation" + ], "html_url": "https://github.com/AbsaOSS/living-doc-project/issues/101", "created_at": "2026-01-10T08:00:00Z", "updated_at": "2026-01-20T14:30:00Z", - "body": "## Description\n\nAs a user, I want to authenticate using Single Sign-On (SSO) so that I can access the application securely using my corporate credentials.\n\n## Business Value\n\n- Reduces password fatigue for users\n- Improves security through centralized authentication\n- Simplifies user onboarding and offboarding\n- Enables enterprise-grade access control\n\n## Preconditions\n\n- SSO provider (Azure AD, Okta, or similar) must be configured\n- OAuth 2.0 client credentials must be obtained\n- Callback URLs must be registered with the provider\n\n## Acceptance Criteria\n\n- [ ] User can click \"Sign in with SSO\" button on login page\n- [ ] User is redirected to SSO provider login page\n- [ ] After successful authentication, user is redirected back with valid session\n- [ ] User profile information is populated from SSO claims\n- [ ] Session expires after 8 hours of inactivity\n- [ ] Failed authentication shows appropriate error message\n\n## User Guide\n\n### For End Users\n\n1. Navigate to the login page\n2. Click the \"Sign in with SSO\" button\n3. Enter your corporate email on the SSO provider page\n4. Complete two-factor authentication if required\n5. You will be automatically redirected to the dashboard\n\n### For Administrators\n\nConfiguration steps:\n```yaml\nsso:\n provider: azure-ad\n client_id: ${SSO_CLIENT_ID}\n client_secret: ${SSO_CLIENT_SECRET}\n tenant_id: ${TENANT_ID}\n```\n\n## Connections\n\n- Related to #102 (Multi-factor authentication)\n- Depends on #103 (User profile management)\n- See also: [Security Architecture](https://docs.example.com/security)\n\n## Last Edited\n\nLast updated by alice@example.com on 2026-01-20" + "body": "## Description\n\nAs a user, I want to authenticate using Single Sign-On (SSO) so that I can access the application securely using my corporate credentials.\n\n## Business Value\n\n- Reduces password fatigue for users\n- Improves security through centralized authentication\n- Simplifies user onboarding and offboarding\n- Enables enterprise-grade access control\n\n## Preconditions\n\n- SSO provider (Azure AD, Okta, or similar) must be configured\n- OAuth 2.0 client credentials must be obtained\n- Callback URLs must be registered with the provider\n\n## Acceptance Criteria\n\n- [ ] User can click \"Sign in with SSO\" button on login page\n- [ ] User is redirected to SSO provider login page\n- [ ] After successful authentication, user is redirected back with valid session\n- [ ] User profile information is populated from SSO claims\n- [ ] Session expires after 8 hours of inactivity\n- [ ] Failed authentication shows appropriate error message\n\n## User Guide\n\n### For End Users\n\n1. Navigate to the login page\n2. Click the \"Sign in with SSO\" button\n3. Enter your corporate email on the SSO provider page\n4. Complete two-factor authentication if required\n5. You will be automatically redirected to the dashboard\n\n### For Administrators\n\nConfiguration steps:\n```yaml\nsso:\n provider: azure-ad\n client_id: ${SSO_CLIENT_ID}\n client_secret: ${SSO_CLIENT_SECRET}\n tenant_id: ${TENANT_ID}\n```\n\n## Connections\n\n- Related to #102 (Multi-factor authentication)\n- Depends on #103 (User profile management)\n- See also: [Security Architecture](https://docs.example.com/security)\n\n## Last Edited\n\nLast updated by alice@example.com on 2026-01-20", + "issue_number": 101 }, - { - "number": 102, + "AbsaOSS/living-doc-project/102": { "title": "Add multi-factor authentication support", "state": "open", - "labels": ["security", "enhancement", "priority:high"], + "labels": [ + "security", + "enhancement", + "priority:high" + ], "html_url": "https://github.com/AbsaOSS/living-doc-project/issues/102", "created_at": "2026-01-11T09:30:00Z", "updated_at": "2026-01-21T10:15:00Z", - "body": "## Overview\n\nEnhance security by implementing multi-factor authentication (MFA) for all user accounts.\n\n## Why\n\nMFA significantly reduces the risk of unauthorized access even if passwords are compromised. Industry best practice and compliance requirement for enterprise applications.\n\n## Setup\n\n| Component | Requirement | Status |\n|-----------|-------------|--------|\n| TOTP Library | pyotp >= 2.8.0 | ✅ Available |\n| QR Code Generator | qrcode >= 7.4.0 | ✅ Available |\n| SMS Gateway | Twilio API | ⚠️ Pending setup |\n\n## AC\n\n1. **TOTP Setup**\n - User can enable MFA from account settings\n - QR code is displayed for authenticator app setup\n - Backup codes are generated (10 codes)\n \n2. **Login Flow**\n - After password validation, MFA code prompt appears\n - User enters 6-digit code from authenticator app\n - Code must be valid within 30-second window\n - 3 failed attempts lock account for 15 minutes\n\n3. **Recovery Options**\n - User can use backup codes when authenticator unavailable\n - Admin can reset MFA for user accounts\n - SMS fallback option for verified phone numbers\n\n## Instructions\n\nEnabling MFA:\n1. Go to Profile → Security Settings\n2. Click \"Enable Two-Factor Authentication\"\n3. Scan QR code with Google Authenticator, Authy, or similar app\n4. Enter verification code to confirm\n5. Save backup codes in secure location\n\n## Related\n\n- #101 - SSO authentication (works together with MFA)\n- #104 - Session management\n- External: [OWASP MFA Guidelines](https://owasp.org/mfa)\n\n## Changes\n\nModified by bob@example.com on 2026-01-21 - Added SMS fallback requirement" + "body": "## Overview\n\nEnhance security by implementing multi-factor authentication (MFA) for all user accounts.\n\n## Why\n\nMFA significantly reduces the risk of unauthorized access even if passwords are compromised. Industry best practice and compliance requirement for enterprise applications.\n\n## Setup\n\n| Component | Requirement | Status |\n|-----------|-------------|--------|\n| TOTP Library | pyotp >= 2.8.0 | \u2705 Available |\n| QR Code Generator | qrcode >= 7.4.0 | \u2705 Available |\n| SMS Gateway | Twilio API | \u26a0\ufe0f Pending setup |\n\n## AC\n\n1. **TOTP Setup**\n - User can enable MFA from account settings\n - QR code is displayed for authenticator app setup\n - Backup codes are generated (10 codes)\n \n2. **Login Flow**\n - After password validation, MFA code prompt appears\n - User enters 6-digit code from authenticator app\n - Code must be valid within 30-second window\n - 3 failed attempts lock account for 15 minutes\n\n3. **Recovery Options**\n - User can use backup codes when authenticator unavailable\n - Admin can reset MFA for user accounts\n - SMS fallback option for verified phone numbers\n\n## Instructions\n\nEnabling MFA:\n1. Go to Profile \u2192 Security Settings\n2. Click \"Enable Two-Factor Authentication\"\n3. Scan QR code with Google Authenticator, Authy, or similar app\n4. Enter verification code to confirm\n5. Save backup codes in secure location\n\n## Related\n\n- #101 - SSO authentication (works together with MFA)\n- #104 - Session management\n- External: [OWASP MFA Guidelines](https://owasp.org/mfa)\n\n## Changes\n\nModified by bob@example.com on 2026-01-21 - Added SMS fallback requirement", + "issue_number": 102 }, - { - "number": 103, + "AbsaOSS/living-doc-project/103": { "title": "User profile management dashboard", "state": "closed", - "labels": ["feature", "ui"], + "labels": [ + "feature", + "ui" + ], "html_url": "https://github.com/AbsaOSS/living-doc-project/issues/103", "created_at": "2026-01-05T14:00:00Z", "updated_at": "2026-01-25T16:00:00Z", - "body": "## Summary\n\nCreate a user profile management dashboard where users can view and edit their account information.\n\n## Value\n\nEmpowers users to manage their own information, reducing support burden and improving user satisfaction.\n\n## Prerequisites\n\n- User authentication must be implemented (#101)\n- Database schema for user profiles must be ready\n- Frontend framework (React) must be set up\n\n## Done Criteria\n\n- User can view current profile information (name, email, avatar)\n- User can edit profile fields with inline validation\n- Changes are saved immediately with visual feedback\n- Profile picture upload supports JPEG, PNG (max 5MB)\n- Audit log tracks all profile changes\n\n## How To\n\n**Accessing Your Profile:**\n- Click your avatar in top-right corner\n- Select \"Profile Settings\" from dropdown menu\n\n**Editing Information:**\n- Click the edit icon next to any field\n- Make your changes\n- Changes save automatically when you click outside the field\n\n**Profile Picture:**\n- Click \"Change Photo\" button\n- Select image file (JPEG or PNG, under 5MB)\n- Crop if desired, then confirm\n\n## Links\n\nRelated issues: #101, #105\nDesign mockups: [Figma Link](https://figma.com/file/xyz)" + "body": "## Summary\n\nCreate a user profile management dashboard where users can view and edit their account information.\n\n## Value\n\nEmpowers users to manage their own information, reducing support burden and improving user satisfaction.\n\n## Prerequisites\n\n- User authentication must be implemented (#101)\n- Database schema for user profiles must be ready\n- Frontend framework (React) must be set up\n\n## Done Criteria\n\n- User can view current profile information (name, email, avatar)\n- User can edit profile fields with inline validation\n- Changes are saved immediately with visual feedback\n- Profile picture upload supports JPEG, PNG (max 5MB)\n- Audit log tracks all profile changes\n\n## How To\n\n**Accessing Your Profile:**\n- Click your avatar in top-right corner\n- Select \"Profile Settings\" from dropdown menu\n\n**Editing Information:**\n- Click the edit icon next to any field\n- Make your changes\n- Changes save automatically when you click outside the field\n\n**Profile Picture:**\n- Click \"Change Photo\" button\n- Select image file (JPEG or PNG, under 5MB)\n- Crop if desired, then confirm\n\n## Links\n\nRelated issues: #101, #105\nDesign mockups: [Figma Link](https://figma.com/file/xyz)", + "issue_number": 103 }, - { - "number": 104, + "AbsaOSS/living-doc-project/104": { "title": "Implement session management and timeout", "state": "open", - "labels": ["security", "backend"], + "labels": [ + "security", + "backend" + ], "html_url": "https://github.com/AbsaOSS/living-doc-project/issues/104", "created_at": "2026-01-12T11:00:00Z", "updated_at": "2026-01-22T13:45:00Z", - "body": "## Description\n\nImplement secure session management with configurable timeout and automatic cleanup.\n\nKey requirements:\n- Session tokens stored securely (httpOnly, secure, sameSite cookies)\n- Configurable session timeout (default: 8 hours)\n- Automatic session extension on user activity\n- Session cleanup job runs every hour\n- Multiple concurrent sessions per user (max 5)\n\n## Business Value\n\nBalances security and user convenience. Automatic timeout prevents unauthorized access from unattended sessions while activity tracking minimizes disruption to active users.\n\n## Acceptance Criteria\n\n- [ ] Session tokens use UUID v4 format\n- [ ] Tokens are stored with expiration timestamp in Redis\n- [ ] Cookie flags: httpOnly=true, secure=true, sameSite=strict\n- [ ] Activity within 30 minutes extends session by configured timeout\n- [ ] Expired sessions return 401 Unauthorized\n- [ ] User can view and revoke active sessions from settings\n- [ ] Session data includes: IP address, user agent, last activity timestamp\n\n## Connections\n\nBlocks: #102 (MFA needs session management)\nRelated: #101 (SSO creates sessions)" + "body": "## Description\n\nImplement secure session management with configurable timeout and automatic cleanup.\n\nKey requirements:\n- Session tokens stored securely (httpOnly, secure, sameSite cookies)\n- Configurable session timeout (default: 8 hours)\n- Automatic session extension on user activity\n- Session cleanup job runs every hour\n- Multiple concurrent sessions per user (max 5)\n\n## Business Value\n\nBalances security and user convenience. Automatic timeout prevents unauthorized access from unattended sessions while activity tracking minimizes disruption to active users.\n\n## Acceptance Criteria\n\n- [ ] Session tokens use UUID v4 format\n- [ ] Tokens are stored with expiration timestamp in Redis\n- [ ] Cookie flags: httpOnly=true, secure=true, sameSite=strict\n- [ ] Activity within 30 minutes extends session by configured timeout\n- [ ] Expired sessions return 401 Unauthorized\n- [ ] User can view and revoke active sessions from settings\n- [ ] Session data includes: IP address, user agent, last activity timestamp\n\n## Connections\n\nBlocks: #102 (MFA needs session management)\nRelated: #101 (SSO creates sessions)", + "issue_number": 104 }, - { - "number": 105, + "AbsaOSS/living-doc-project/105": { "title": "Add notification preferences", "state": "open", - "labels": ["feature", "notifications"], + "labels": [ + "feature", + "notifications" + ], "html_url": "https://github.com/AbsaOSS/living-doc-project/issues/105", "created_at": "2026-01-13T10:30:00Z", "updated_at": "2026-01-23T11:20:00Z", - "body": "## Overview\n\nAllow users to configure their notification preferences for email, in-app, and mobile push notifications.\n\n## Value\n\nUsers can control communication frequency and channels, reducing notification fatigue and improving engagement.\n\n## Done Criteria\n\n- Settings page with notification preferences UI\n- Toggle switches for each notification type:\n - Email notifications (on/off)\n - In-app notifications (on/off) \n - Push notifications (on/off)\n- Granular controls:\n - Security alerts (always on)\n - Account updates (optional)\n - Marketing emails (optional)\n - Weekly digest (optional)\n- Preferences saved per user in database\n- Notification service respects user preferences before sending\n\n## User Guide\n\n1. Navigate to Settings → Notifications\n2. Toggle switches for desired notification types\n3. Expand \"Advanced Options\" for granular controls\n4. Click \"Save Preferences\" (auto-saves enabled by default)" + "body": "## Overview\n\nAllow users to configure their notification preferences for email, in-app, and mobile push notifications.\n\n## Value\n\nUsers can control communication frequency and channels, reducing notification fatigue and improving engagement.\n\n## Done Criteria\n\n- Settings page with notification preferences UI\n- Toggle switches for each notification type:\n - Email notifications (on/off)\n - In-app notifications (on/off) \n - Push notifications (on/off)\n- Granular controls:\n - Security alerts (always on)\n - Account updates (optional)\n - Marketing emails (optional)\n - Weekly digest (optional)\n- Preferences saved per user in database\n- Notification service respects user preferences before sending\n\n## User Guide\n\n1. Navigate to Settings \u2192 Notifications\n2. Toggle switches for desired notification types\n3. Expand \"Advanced Options\" for granular controls\n4. Click \"Save Preferences\" (auto-saves enabled by default)", + "issue_number": 105 }, - { - "number": 106, + "AbsaOSS/living-doc-project/106": { "title": "Fix pagination offset calculation bug", "state": "closed", - "labels": ["bug", "priority:medium"], + "labels": [ + "bug", + "priority:medium" + ], "html_url": "https://github.com/AbsaOSS/living-doc-project/issues/106", "created_at": "2026-01-14T13:15:00Z", "updated_at": "2026-01-26T09:00:00Z", - "body": "## Description\n\nPagination offset calculation is incorrect when page size is not 10. This causes wrong results to be displayed on pages 2+.\n\n**Bug Details:**\n```python\n# Current (wrong):\noffset = page * page_size\n\n# Should be:\noffset = (page - 1) * page_size\n```\n\n## Acceptance Criteria\n\n- [ ] Fix offset calculation in all paginated endpoints\n- [ ] Add unit tests for pagination edge cases\n- [ ] Test with page sizes: 5, 10, 20, 50, 100\n- [ ] Verify page 1 shows items 1-N\n- [ ] Verify page 2 shows items N+1 to 2N\n\n## Last Edited\n\nFixed and deployed by charlie@example.com on 2026-01-26" + "body": "## Description\n\nPagination offset calculation is incorrect when page size is not 10. This causes wrong results to be displayed on pages 2+.\n\n**Bug Details:**\n```python\n# Current (wrong):\noffset = page * page_size\n\n# Should be:\noffset = (page - 1) * page_size\n```\n\n## Acceptance Criteria\n\n- [ ] Fix offset calculation in all paginated endpoints\n- [ ] Add unit tests for pagination edge cases\n- [ ] Test with page sizes: 5, 10, 20, 50, 100\n- [ ] Verify page 1 shows items 1-N\n- [ ] Verify page 2 shows items N+1 to 2N\n\n## Last Edited\n\nFixed and deployed by charlie@example.com on 2026-01-26", + "issue_number": 106 }, - { - "number": 107, + "AbsaOSS/living-doc-project/107": { "title": "API documentation with OpenAPI/Swagger", "state": "open", - "labels": ["documentation", "api"], + "labels": [ + "documentation", + "api" + ], "html_url": "https://github.com/AbsaOSS/living-doc-project/issues/107", "created_at": "2026-01-15T09:00:00Z", "updated_at": "2026-01-24T14:30:00Z", - "body": "## Summary\n\nGenerate comprehensive API documentation using OpenAPI 3.0 specification and Swagger UI.\n\n## Why\n\nAPI documentation improves developer experience and reduces integration time for external teams.\n\n## Preconditions\n\n- API endpoints must be stable and versioned (v1)\n- FastAPI/Flask framework must support OpenAPI decorators\n- Hosting environment for Swagger UI docs\n\n## Acceptance Criteria\n\nOpenAPI Specification:\n- [ ] All endpoints documented with descriptions\n- [ ] Request/response schemas defined\n- [ ] Authentication requirements specified\n- [ ] Example requests and responses included\n- [ ] Error responses documented (4xx, 5xx)\n\nSwagger UI:\n- [ ] Accessible at /api/docs endpoint\n- [ ] Interactive \"Try it out\" functionality works\n- [ ] Supports authentication token input\n- [ ] Auto-generated from code annotations\n\n## Instructions\n\nAccessing API Docs:\n- Navigate to https://api.example.com/docs\n- Click \"Authorize\" button to enter API token\n- Expand any endpoint to see details\n- Click \"Try it out\" to test directly from browser\n\n## Related\n\n- #108 - API versioning strategy\n- #109 - Rate limiting\n- External docs: [OpenAPI Specification](https://swagger.io/specification/)" + "body": "## Summary\n\nGenerate comprehensive API documentation using OpenAPI 3.0 specification and Swagger UI.\n\n## Why\n\nAPI documentation improves developer experience and reduces integration time for external teams.\n\n## Preconditions\n\n- API endpoints must be stable and versioned (v1)\n- FastAPI/Flask framework must support OpenAPI decorators\n- Hosting environment for Swagger UI docs\n\n## Acceptance Criteria\n\nOpenAPI Specification:\n- [ ] All endpoints documented with descriptions\n- [ ] Request/response schemas defined\n- [ ] Authentication requirements specified\n- [ ] Example requests and responses included\n- [ ] Error responses documented (4xx, 5xx)\n\nSwagger UI:\n- [ ] Accessible at /api/docs endpoint\n- [ ] Interactive \"Try it out\" functionality works\n- [ ] Supports authentication token input\n- [ ] Auto-generated from code annotations\n\n## Instructions\n\nAccessing API Docs:\n- Navigate to https://api.example.com/docs\n- Click \"Authorize\" button to enter API token\n- Expand any endpoint to see details\n- Click \"Try it out\" to test directly from browser\n\n## Related\n\n- #108 - API versioning strategy\n- #109 - Rate limiting\n- External docs: [OpenAPI Specification](https://swagger.io/specification/)", + "issue_number": 107 }, - { - "number": 108, + "AbsaOSS/living-doc-project/108": { "title": "Implement API versioning strategy", "state": "open", - "labels": ["api", "architecture"], + "labels": [ + "api", + "architecture" + ], "html_url": "https://github.com/AbsaOSS/living-doc-project/issues/108", "created_at": "2026-01-16T10:45:00Z", "updated_at": "2026-01-25T12:00:00Z", - "body": "## Description\n\nEstablish API versioning strategy to support backward compatibility and smooth migrations.\n\n**Proposed Approach:**\n- URL path versioning: `/api/v1/`, `/api/v2/`\n- Major version in path, minor/patch in header\n- Deprecation notices 3 months before removal\n- Support N-1 versions (current + previous)\n\n## Business Value\n\nEnables API evolution without breaking existing integrations. Reduces coordination overhead for external teams consuming our API.\n\n## Setup\n\n- Update API gateway routing rules\n- Implement version detection middleware\n- Create deprecation warning mechanism\n- Update CI/CD to deploy multiple versions\n\n## AC\n\n1. **Version Detection**\n - Requests to `/api/v1/users` route to v1 handler\n - Requests to `/api/v2/users` route to v2 handler\n - Missing version defaults to latest stable\n \n2. **Deprecation Handling**\n - Deprecated endpoints return `Deprecation` header\n - Header includes sunset date and migration guide URL\n - Deprecated endpoints logged for usage analytics\n\n3. **Documentation**\n - Version-specific documentation at /docs/v1, /docs/v2\n - Migration guide between versions\n - Changelog with breaking changes highlighted\n\n## Connections\n\nRelated to #107 (API documentation)" + "body": "## Description\n\nEstablish API versioning strategy to support backward compatibility and smooth migrations.\n\n**Proposed Approach:**\n- URL path versioning: `/api/v1/`, `/api/v2/`\n- Major version in path, minor/patch in header\n- Deprecation notices 3 months before removal\n- Support N-1 versions (current + previous)\n\n## Business Value\n\nEnables API evolution without breaking existing integrations. Reduces coordination overhead for external teams consuming our API.\n\n## Setup\n\n- Update API gateway routing rules\n- Implement version detection middleware\n- Create deprecation warning mechanism\n- Update CI/CD to deploy multiple versions\n\n## AC\n\n1. **Version Detection**\n - Requests to `/api/v1/users` route to v1 handler\n - Requests to `/api/v2/users` route to v2 handler\n - Missing version defaults to latest stable\n \n2. **Deprecation Handling**\n - Deprecated endpoints return `Deprecation` header\n - Header includes sunset date and migration guide URL\n - Deprecated endpoints logged for usage analytics\n\n3. **Documentation**\n - Version-specific documentation at /docs/v1, /docs/v2\n - Migration guide between versions\n - Changelog with breaking changes highlighted\n\n## Connections\n\nRelated to #107 (API documentation)", + "issue_number": 108 }, - { - "number": 109, + "AbsaOSS/living-doc-project/109": { "title": "Add API rate limiting", "state": "closed", - "labels": ["api", "security"], + "labels": [ + "api", + "security" + ], "html_url": "https://github.com/AbsaOSS/living-doc-project/issues/109", "created_at": "2026-01-17T11:30:00Z", "updated_at": "2026-01-27T10:00:00Z", - "body": "## Description\n\nImplement rate limiting to prevent API abuse and ensure fair resource allocation.\n\n**Rate Limits:**\n- Anonymous: 100 requests/hour\n- Authenticated: 1000 requests/hour\n- Premium tier: 5000 requests/hour\n\n## Value\n\nProtects API availability and performance. Prevents denial-of-service attacks and ensures quality of service for all users.\n\n## Acceptance Criteria\n\n- [ ] Rate limiting based on IP address for anonymous users\n- [ ] Rate limiting based on API token for authenticated users\n- [ ] Redis backend for distributed rate limit tracking\n- [ ] Response headers include rate limit info:\n - `X-RateLimit-Limit`: Total requests allowed\n - `X-RateLimit-Remaining`: Requests remaining\n - `X-RateLimit-Reset`: Unix timestamp when limit resets\n- [ ] 429 Too Many Requests response when limit exceeded\n- [ ] Retry-After header indicates wait time in seconds\n\n## History\n\nImplemented by dave@example.com on 2026-01-27" + "body": "## Description\n\nImplement rate limiting to prevent API abuse and ensure fair resource allocation.\n\n**Rate Limits:**\n- Anonymous: 100 requests/hour\n- Authenticated: 1000 requests/hour\n- Premium tier: 5000 requests/hour\n\n## Value\n\nProtects API availability and performance. Prevents denial-of-service attacks and ensures quality of service for all users.\n\n## Acceptance Criteria\n\n- [ ] Rate limiting based on IP address for anonymous users\n- [ ] Rate limiting based on API token for authenticated users\n- [ ] Redis backend for distributed rate limit tracking\n- [ ] Response headers include rate limit info:\n - `X-RateLimit-Limit`: Total requests allowed\n - `X-RateLimit-Remaining`: Requests remaining\n - `X-RateLimit-Reset`: Unix timestamp when limit resets\n- [ ] 429 Too Many Requests response when limit exceeded\n- [ ] Retry-After header indicates wait time in seconds\n\n## History\n\nImplemented by dave@example.com on 2026-01-27", + "issue_number": 109 }, - { - "number": 110, + "AbsaOSS/living-doc-project/110": { "title": "Database migration to PostgreSQL 15", "state": "open", - "labels": ["database", "infrastructure"], + "labels": [ + "database", + "infrastructure" + ], "html_url": "https://github.com/AbsaOSS/living-doc-project/issues/110", "created_at": "2026-01-18T08:00:00Z", "updated_at": "2026-01-26T16:30:00Z", - "body": "## Overview\n\nMigrate production database from PostgreSQL 12 to PostgreSQL 15 to leverage performance improvements and new features.\n\n## Business Value\n\n- 30% faster query performance (benchmark results)\n- Improved JSON query capabilities\n- Better index management\n- Extended support lifecycle\n\n## Prerequisites\n\n- Complete backup of production database\n- Test migration on staging environment\n- Verify application compatibility with PG 15\n- Plan maintenance window (estimated 4 hours)\n- Rollback procedure documented and tested\n\n## Acceptance Criteria\n\nPre-Migration:\n- [ ] Full database backup completed and verified\n- [ ] Staging migration successful with zero data loss\n- [ ] All application tests pass against PG 15\n- [ ] Migration runbook reviewed by team\n- [ ] Stakeholders notified of maintenance window\n\nMigration:\n- [ ] Enable read-only mode on current database\n- [ ] Dump database using pg_dump\n- [ ] Provision PostgreSQL 15 instance\n- [ ] Restore data to new instance\n- [ ] Run ANALYZE to update statistics\n- [ ] Update application connection strings\n- [ ] Verify data integrity checksums\n\nPost-Migration:\n- [ ] All application endpoints responding correctly\n- [ ] Monitor query performance for 24 hours\n- [ ] Keep PG 12 instance running for 1 week (rollback option)\n- [ ] Update documentation with new connection details\n\n## User Guide\n\n**During Maintenance:**\n- Application will be read-only from 02:00-06:00 UTC on 2026-02-01\n- No action required from end users\n- Service will automatically reconnect after migration\n\n**For Developers:**\n- Update local connection strings to new instance\n- Connection pooling parameters may need adjustment\n- Review [PostgreSQL 15 release notes](https://www.postgresql.org/docs/15/release-15.html)\n\n## Connections\n\n- Blocks #111 (performance optimization depends on PG 15 features)\n- Related: Infrastructure upgrade initiative\n\n## Changes\n\nMigration date updated by sre@example.com on 2026-01-26" + "body": "## Overview\n\nMigrate production database from PostgreSQL 12 to PostgreSQL 15 to leverage performance improvements and new features.\n\n## Business Value\n\n- 30% faster query performance (benchmark results)\n- Improved JSON query capabilities\n- Better index management\n- Extended support lifecycle\n\n## Prerequisites\n\n- Complete backup of production database\n- Test migration on staging environment\n- Verify application compatibility with PG 15\n- Plan maintenance window (estimated 4 hours)\n- Rollback procedure documented and tested\n\n## Acceptance Criteria\n\nPre-Migration:\n- [ ] Full database backup completed and verified\n- [ ] Staging migration successful with zero data loss\n- [ ] All application tests pass against PG 15\n- [ ] Migration runbook reviewed by team\n- [ ] Stakeholders notified of maintenance window\n\nMigration:\n- [ ] Enable read-only mode on current database\n- [ ] Dump database using pg_dump\n- [ ] Provision PostgreSQL 15 instance\n- [ ] Restore data to new instance\n- [ ] Run ANALYZE to update statistics\n- [ ] Update application connection strings\n- [ ] Verify data integrity checksums\n\nPost-Migration:\n- [ ] All application endpoints responding correctly\n- [ ] Monitor query performance for 24 hours\n- [ ] Keep PG 12 instance running for 1 week (rollback option)\n- [ ] Update documentation with new connection details\n\n## User Guide\n\n**During Maintenance:**\n- Application will be read-only from 02:00-06:00 UTC on 2026-02-01\n- No action required from end users\n- Service will automatically reconnect after migration\n\n**For Developers:**\n- Update local connection strings to new instance\n- Connection pooling parameters may need adjustment\n- Review [PostgreSQL 15 release notes](https://www.postgresql.org/docs/15/release-15.html)\n\n## Connections\n\n- Blocks #111 (performance optimization depends on PG 15 features)\n- Related: Infrastructure upgrade initiative\n\n## Changes\n\nMigration date updated by sre@example.com on 2026-01-26", + "issue_number": 110 }, - { - "number": 111, + "AbsaOSS/living-doc-project/111": { "title": "Query performance optimization", "state": "open", - "labels": ["performance", "database"], + "labels": [ + "performance", + "database" + ], "html_url": "https://github.com/AbsaOSS/living-doc-project/issues/111", "created_at": "2026-01-19T09:30:00Z", "updated_at": "2026-01-27T11:15:00Z", - "body": "## Description\n\nOptimize slow database queries identified in performance monitoring.\n\n**Problem Queries:**\n1. User search: 2.5s average response time\n2. Dashboard analytics: 4.1s average response time\n3. Report generation: 8.3s average response time\n\n**Target Performance:**\n1. User search: < 200ms\n2. Dashboard analytics: < 500ms\n3. Report generation: < 2s\n\n## Value\n\nImproved application responsiveness leads to better user experience and higher satisfaction scores.\n\n## Setup\n\n- PostgreSQL 15 migration must be completed (#110)\n- Enable query execution plan logging\n- Set up performance monitoring dashboard\n\n## Done Criteria\n\n- [ ] Add composite index on users(email, active, created_at)\n- [ ] Add materialized view for dashboard analytics\n- [ ] Implement query result caching (Redis, 5 min TTL)\n- [ ] Add database connection pooling (pgbouncer)\n- [ ] All target performance metrics achieved\n- [ ] Performance monitoring alerts configured\n\n## Related\n\nDependency: #110 (PostgreSQL 15 migration)" + "body": "## Description\n\nOptimize slow database queries identified in performance monitoring.\n\n**Problem Queries:**\n1. User search: 2.5s average response time\n2. Dashboard analytics: 4.1s average response time\n3. Report generation: 8.3s average response time\n\n**Target Performance:**\n1. User search: < 200ms\n2. Dashboard analytics: < 500ms\n3. Report generation: < 2s\n\n## Value\n\nImproved application responsiveness leads to better user experience and higher satisfaction scores.\n\n## Setup\n\n- PostgreSQL 15 migration must be completed (#110)\n- Enable query execution plan logging\n- Set up performance monitoring dashboard\n\n## Done Criteria\n\n- [ ] Add composite index on users(email, active, created_at)\n- [ ] Add materialized view for dashboard analytics\n- [ ] Implement query result caching (Redis, 5 min TTL)\n- [ ] Add database connection pooling (pgbouncer)\n- [ ] All target performance metrics achieved\n- [ ] Performance monitoring alerts configured\n\n## Related\n\nDependency: #110 (PostgreSQL 15 migration)", + "issue_number": 111 }, - { - "number": 112, + "AbsaOSS/living-doc-project/112": { "title": "Implement audit logging system", "state": "open", - "labels": ["security", "compliance"], + "labels": [ + "security", + "compliance" + ], "html_url": "https://github.com/AbsaOSS/living-doc-project/issues/112", "created_at": "2026-01-20T10:00:00Z", "updated_at": "2026-01-28T09:00:00Z", - "body": "## Overview\n\nImplement comprehensive audit logging for compliance and security monitoring.\n\n## Why\n\nRegulatory compliance requires detailed audit trails. Security teams need visibility into user actions for threat detection and incident investigation.\n\n## Preconditions\n\n- Log aggregation infrastructure (ELK or similar) must be ready\n- Data retention policy must be approved (7 years for financial data)\n- Log storage capacity planned and provisioned\n\n## AC\n\n**Events to Log:**\n- User authentication (login, logout, failed attempts)\n- Authorization changes (role assignments, permission changes)\n- Data access (view, export of sensitive data)\n- Data modifications (create, update, delete)\n- Administrative actions (user management, configuration changes)\n- Security events (password resets, MFA changes)\n\n**Log Format:**\n```json\n{\n \"timestamp\": \"2026-01-28T09:00:00Z\",\n \"event_type\": \"data_access\",\n \"actor\": \"user@example.com\",\n \"action\": \"export_report\",\n \"resource\": \"financial_report_2025\",\n \"ip_address\": \"192.168.1.100\",\n \"user_agent\": \"Mozilla/5.0...\",\n \"result\": \"success\",\n \"metadata\": {}\n}\n```\n\n**Requirements:**\n- [ ] Logs written to append-only storage\n- [ ] Tamper-proof audit trail (checksums)\n- [ ] Centralized log collection\n- [ ] 7-year retention for compliance logs\n- [ ] 90-day retention for operational logs\n- [ ] Search and reporting UI for security team\n- [ ] Automated alerts for suspicious patterns\n\n## User Guide\n\n**For Security Team:**\n- Access audit logs at /admin/audit\n- Search by user, event type, or date range\n- Export results to CSV for compliance reports\n\n**For Administrators:**\n- Configure retention policies in admin panel\n- Set up email alerts for critical events\n- Review daily summary reports\n\n## Connections\n\n- Related to #101, #102 (authentication/authorization events)\n- Related to compliance initiative\n- External: [OWASP Logging Guide](https://owasp.org/logging)" + "body": "## Overview\n\nImplement comprehensive audit logging for compliance and security monitoring.\n\n## Why\n\nRegulatory compliance requires detailed audit trails. Security teams need visibility into user actions for threat detection and incident investigation.\n\n## Preconditions\n\n- Log aggregation infrastructure (ELK or similar) must be ready\n- Data retention policy must be approved (7 years for financial data)\n- Log storage capacity planned and provisioned\n\n## AC\n\n**Events to Log:**\n- User authentication (login, logout, failed attempts)\n- Authorization changes (role assignments, permission changes)\n- Data access (view, export of sensitive data)\n- Data modifications (create, update, delete)\n- Administrative actions (user management, configuration changes)\n- Security events (password resets, MFA changes)\n\n**Log Format:**\n```json\n{\n \"timestamp\": \"2026-01-28T09:00:00Z\",\n \"event_type\": \"data_access\",\n \"actor\": \"user@example.com\",\n \"action\": \"export_report\",\n \"resource\": \"financial_report_2025\",\n \"ip_address\": \"192.168.1.100\",\n \"user_agent\": \"Mozilla/5.0...\",\n \"result\": \"success\",\n \"metadata\": {}\n}\n```\n\n**Requirements:**\n- [ ] Logs written to append-only storage\n- [ ] Tamper-proof audit trail (checksums)\n- [ ] Centralized log collection\n- [ ] 7-year retention for compliance logs\n- [ ] 90-day retention for operational logs\n- [ ] Search and reporting UI for security team\n- [ ] Automated alerts for suspicious patterns\n\n## User Guide\n\n**For Security Team:**\n- Access audit logs at /admin/audit\n- Search by user, event type, or date range\n- Export results to CSV for compliance reports\n\n**For Administrators:**\n- Configure retention policies in admin panel\n- Set up email alerts for critical events\n- Review daily summary reports\n\n## Connections\n\n- Related to #101, #102 (authentication/authorization events)\n- Related to compliance initiative\n- External: [OWASP Logging Guide](https://owasp.org/logging)", + "issue_number": 112 }, - { - "number": 113, + "AbsaOSS/living-doc-project/113": { "title": "Mobile app: offline mode support", "state": "open", - "labels": ["mobile", "feature"], + "labels": [ + "mobile", + "feature" + ], "html_url": "https://github.com/AbsaOSS/living-doc-project/issues/113", "created_at": "2026-01-21T11:00:00Z", "updated_at": "2026-01-28T14:00:00Z", - "body": "## Summary\n\nEnable offline mode in mobile app to allow users to work without internet connectivity.\n\n## Business Value\n\nUsers can continue working during travel or in areas with poor connectivity. Improves user satisfaction and productivity.\n\n## Prerequisites\n\n- Mobile app architecture supports local storage\n- Conflict resolution strategy defined\n- Background sync mechanism implemented\n\n## Done Criteria\n\nOffline Capabilities:\n- [ ] View cached content (last 7 days)\n- [ ] Create new items (queued for sync)\n- [ ] Edit existing items (queued for sync)\n- [ ] Delete items (queued for sync)\n- [ ] Search within cached content\n\nSync Behavior:\n- [ ] Automatic sync when connectivity restored\n- [ ] Manual sync trigger in app\n- [ ] Conflict detection and resolution UI\n- [ ] Progress indicator during sync\n- [ ] Failed sync items highlighted for user review\n\nStorage:\n- [ ] Max 100MB local storage per user\n- [ ] Automatic cache cleanup (30 days)\n- [ ] User can manually clear cache\n\n## How To\n\n**Using Offline Mode:**\n1. Ensure app is synced while online\n2. App automatically detects loss of connectivity\n3. Offline indicator appears in header\n4. Continue working - changes saved locally\n5. When back online, app syncs automatically\n6. Review \"Sync Status\" to see pending changes\n\n**Handling Conflicts:**\n- If conflict detected, app shows both versions\n- User chooses which version to keep\n- Option to merge changes manually\n\n## Links\n\nDesign: [Offline UX Mockups](https://figma.com/file/offline)\nRelated: #114 (sync optimization)" + "body": "## Summary\n\nEnable offline mode in mobile app to allow users to work without internet connectivity.\n\n## Business Value\n\nUsers can continue working during travel or in areas with poor connectivity. Improves user satisfaction and productivity.\n\n## Prerequisites\n\n- Mobile app architecture supports local storage\n- Conflict resolution strategy defined\n- Background sync mechanism implemented\n\n## Done Criteria\n\nOffline Capabilities:\n- [ ] View cached content (last 7 days)\n- [ ] Create new items (queued for sync)\n- [ ] Edit existing items (queued for sync)\n- [ ] Delete items (queued for sync)\n- [ ] Search within cached content\n\nSync Behavior:\n- [ ] Automatic sync when connectivity restored\n- [ ] Manual sync trigger in app\n- [ ] Conflict detection and resolution UI\n- [ ] Progress indicator during sync\n- [ ] Failed sync items highlighted for user review\n\nStorage:\n- [ ] Max 100MB local storage per user\n- [ ] Automatic cache cleanup (30 days)\n- [ ] User can manually clear cache\n\n## How To\n\n**Using Offline Mode:**\n1. Ensure app is synced while online\n2. App automatically detects loss of connectivity\n3. Offline indicator appears in header\n4. Continue working - changes saved locally\n5. When back online, app syncs automatically\n6. Review \"Sync Status\" to see pending changes\n\n**Handling Conflicts:**\n- If conflict detected, app shows both versions\n- User chooses which version to keep\n- Option to merge changes manually\n\n## Links\n\nDesign: [Offline UX Mockups](https://figma.com/file/offline)\nRelated: #114 (sync optimization)", + "issue_number": 113 }, - { - "number": 114, + "AbsaOSS/living-doc-project/114": { "title": "Empty body test case", "state": "open", - "labels": ["test"], + "labels": [ + "test" + ], "html_url": "https://github.com/AbsaOSS/living-doc-project/issues/114", "created_at": "2026-01-22T12:00:00Z", "updated_at": "2026-01-22T12:00:00Z", - "body": null + "body": null, + "issue_number": 114 }, - { - "number": 115, + "AbsaOSS/living-doc-project/115": { "title": "No headings just plain text", "state": "closed", - "labels": ["documentation"], + "labels": [ + "documentation" + ], "html_url": "https://github.com/AbsaOSS/living-doc-project/issues/115", "created_at": "2026-01-23T13:30:00Z", "updated_at": "2026-01-29T10:00:00Z", - "body": "This issue has no markdown headings, just plain text content. It should be treated as description content. This tests the normalizer's handling of plain text without any markdown structure." + "body": "This issue has no markdown headings, just plain text content. It should be treated as description content. This tests the normalizer's handling of plain text without any markdown structure.", + "issue_number": 115 } - ] -} + } +} \ No newline at end of file diff --git a/tests/fixtures/collector_gh/v1.2.0/input/doc-issues.json b/tests/fixtures/collector_gh/v1.2.0/input/doc-issues.json index 3b0288c..2fee770 100644 --- a/tests/fixtures/collector_gh/v1.2.0/input/doc-issues.json +++ b/tests/fixtures/collector_gh/v1.2.0/input/doc-issues.json @@ -14,152 +14,198 @@ "sha": "def789ghi012345" }, "source": { - "systems": ["GitHub"], - "repositories": ["AbsaOSS/living-doc-v2"], + "systems": [ + "GitHub" + ], + "repositories": [ + "AbsaOSS/living-doc-v2" + ], "organization": "AbsaOSS", "enterprise": null } }, - "issues": [ - { - "number": 201, + "issues": { + "AbsaOSS/living-doc-v2/201": { "title": "Implement real-time notifications", "state": "open", - "labels": ["feature", "websocket", "priority:high"], + "labels": [ + "feature", + "websocket", + "priority:high" + ], "html_url": "https://github.com/AbsaOSS/living-doc-v2/issues/201", "created_at": "2026-02-01T08:00:00Z", "updated_at": "2026-02-10T15:30:00Z", - "body": "## Description\n\nImplement real-time push notifications using WebSocket connections to provide instant updates to users.\n\n## Business Value\n\n- Eliminates need for page refreshing\n- Improves user engagement and responsiveness\n- Reduces server load from polling\n- Enables collaborative features\n\n## Preconditions\n\n- WebSocket infrastructure must be available (Socket.io or similar)\n- Redis pub/sub for message distribution across servers\n- Load balancer must support WebSocket connections\n- Client library for WebSocket management\n\n## Acceptance Criteria\n\n**Connection Management:**\n- [ ] Client establishes WebSocket connection on login\n- [ ] Connection survives network hiccups (auto-reconnect)\n- [ ] Heartbeat ping/pong every 30 seconds\n- [ ] Graceful fallback to polling if WebSocket unavailable\n\n**Notification Types:**\n- [ ] New message received\n- [ ] Task assignment\n- [ ] Status change on watched items\n- [ ] System announcements\n- [ ] Collaboration updates (user joined, user typing)\n\n**Performance:**\n- [ ] Support 10,000 concurrent connections per server\n- [ ] Message delivery latency < 100ms\n- [ ] CPU usage < 20% at max capacity\n\n## User Guide\n\n**For End Users:**\nNotifications appear automatically - no configuration needed!\n- Toast notification appears in bottom-right corner\n- Click notification to navigate to relevant item\n- Bell icon shows notification count\n- Notification center lists all recent notifications\n\n**For Developers:**\n```javascript\n// Subscribe to notifications\nconst socket = io();\nsocket.on('notification', (data) => {\n showToast(data.message, data.type);\n});\n```" + "body": "## Description\n\nImplement real-time push notifications using WebSocket connections to provide instant updates to users.\n\n## Business Value\n\n- Eliminates need for page refreshing\n- Improves user engagement and responsiveness\n- Reduces server load from polling\n- Enables collaborative features\n\n## Preconditions\n\n- WebSocket infrastructure must be available (Socket.io or similar)\n- Redis pub/sub for message distribution across servers\n- Load balancer must support WebSocket connections\n- Client library for WebSocket management\n\n## Acceptance Criteria\n\n**Connection Management:**\n- [ ] Client establishes WebSocket connection on login\n- [ ] Connection survives network hiccups (auto-reconnect)\n- [ ] Heartbeat ping/pong every 30 seconds\n- [ ] Graceful fallback to polling if WebSocket unavailable\n\n**Notification Types:**\n- [ ] New message received\n- [ ] Task assignment\n- [ ] Status change on watched items\n- [ ] System announcements\n- [ ] Collaboration updates (user joined, user typing)\n\n**Performance:**\n- [ ] Support 10,000 concurrent connections per server\n- [ ] Message delivery latency < 100ms\n- [ ] CPU usage < 20% at max capacity\n\n## User Guide\n\n**For End Users:**\nNotifications appear automatically - no configuration needed!\n- Toast notification appears in bottom-right corner\n- Click notification to navigate to relevant item\n- Bell icon shows notification count\n- Notification center lists all recent notifications\n\n**For Developers:**\n```javascript\n// Subscribe to notifications\nconst socket = io();\nsocket.on('notification', (data) => {\n showToast(data.message, data.type);\n});\n```", + "issue_number": 201 }, - { - "number": 202, + "AbsaOSS/living-doc-v2/202": { "title": "Data export to multiple formats", "state": "open", - "labels": ["feature", "export"], + "labels": [ + "feature", + "export" + ], "html_url": "https://github.com/AbsaOSS/living-doc-v2/issues/202", "created_at": "2026-02-02T09:15:00Z", "updated_at": "2026-02-11T10:45:00Z", - "body": "## Overview\n\nEnable users to export data in multiple formats: CSV, Excel, JSON, and PDF.\n\n## Why\n\nUsers need data in different formats for reporting, analysis, and integration with other tools.\n\n## Setup\n\n| Format | Library | Version |\n|--------|---------|--------|\n| CSV | csv | stdlib |\n| Excel | openpyxl | 3.1.0 |\n| JSON | json | stdlib |\n| PDF | reportlab | 4.0.0 |\n\n## AC\n\n**Supported Formats:**\n- [ ] CSV - comma-separated values\n- [ ] XLSX - Excel workbook with formatting\n- [ ] JSON - structured data export\n- [ ] PDF - formatted report with branding\n\n**Export Options:**\n- [ ] Select fields to include/exclude\n- [ ] Date range filter\n- [ ] Sort order selection\n- [ ] Page orientation for PDF (portrait/landscape)\n\n**Implementation:**\n- [ ] Export button in list views\n- [ ] Format selection dropdown\n- [ ] Background job for large exports\n- [ ] Email notification when export ready\n- [ ] Download link expires after 24 hours\n\n## Instructions\n\n1. Navigate to any list view (users, reports, etc.)\n2. Click \"Export\" button in toolbar\n3. Select desired format from dropdown\n4. Configure export options (optional)\n5. Click \"Generate Export\"\n6. For small exports: Download starts immediately\n7. For large exports: Email sent when ready\n\n## Related\n\n- #203 - Report scheduling\n- #204 - Data import" + "body": "## Overview\n\nEnable users to export data in multiple formats: CSV, Excel, JSON, and PDF.\n\n## Why\n\nUsers need data in different formats for reporting, analysis, and integration with other tools.\n\n## Setup\n\n| Format | Library | Version |\n|--------|---------|--------|\n| CSV | csv | stdlib |\n| Excel | openpyxl | 3.1.0 |\n| JSON | json | stdlib |\n| PDF | reportlab | 4.0.0 |\n\n## AC\n\n**Supported Formats:**\n- [ ] CSV - comma-separated values\n- [ ] XLSX - Excel workbook with formatting\n- [ ] JSON - structured data export\n- [ ] PDF - formatted report with branding\n\n**Export Options:**\n- [ ] Select fields to include/exclude\n- [ ] Date range filter\n- [ ] Sort order selection\n- [ ] Page orientation for PDF (portrait/landscape)\n\n**Implementation:**\n- [ ] Export button in list views\n- [ ] Format selection dropdown\n- [ ] Background job for large exports\n- [ ] Email notification when export ready\n- [ ] Download link expires after 24 hours\n\n## Instructions\n\n1. Navigate to any list view (users, reports, etc.)\n2. Click \"Export\" button in toolbar\n3. Select desired format from dropdown\n4. Configure export options (optional)\n5. Click \"Generate Export\"\n6. For small exports: Download starts immediately\n7. For large exports: Email sent when ready\n\n## Related\n\n- #203 - Report scheduling\n- #204 - Data import", + "issue_number": 202 }, - { - "number": 203, + "AbsaOSS/living-doc-v2/203": { "title": "Scheduled report generation", "state": "closed", - "labels": ["feature", "reports"], + "labels": [ + "feature", + "reports" + ], "html_url": "https://github.com/AbsaOSS/living-doc-v2/issues/203", "created_at": "2026-02-03T10:30:00Z", "updated_at": "2026-02-12T14:00:00Z", - "body": "## Summary\n\nAllow users to schedule automatic report generation and delivery.\n\n## Value\n\nSaves time and ensures stakeholders receive regular updates without manual intervention.\n\n## Prerequisites\n\n- Celery or similar job scheduler configured\n- Email delivery system operational\n- Report templates defined\n- Cloud storage for report archives\n\n## Done Criteria\n\n**Scheduling Options:**\n- [ ] Daily, weekly, monthly schedules\n- [ ] Custom cron expressions for advanced users\n- [ ] Timezone selection\n- [ ] Start date and optional end date\n\n**Delivery Methods:**\n- [ ] Email with PDF attachment\n- [ ] Email with download link\n- [ ] Save to cloud storage (S3, GCS)\n- [ ] Webhook notification\n\n**Report Types:**\n- [ ] User activity summary\n- [ ] Performance metrics\n- [ ] Compliance audit report\n- [ ] Custom report builder\n\n## How To\n\n**Creating a Scheduled Report:**\n1. Go to Reports → Scheduled Reports\n2. Click \"New Schedule\"\n3. Select report type and parameters\n4. Choose frequency (daily/weekly/monthly)\n5. Add email recipients\n6. Click \"Create Schedule\"\n\n**Managing Schedules:**\n- View all schedules in dashboard\n- Pause/resume schedules\n- Edit schedule parameters\n- View execution history\n- Download previous reports\n\n## History\n\nImplemented by emma@example.com on 2026-02-12" + "body": "## Summary\n\nAllow users to schedule automatic report generation and delivery.\n\n## Value\n\nSaves time and ensures stakeholders receive regular updates without manual intervention.\n\n## Prerequisites\n\n- Celery or similar job scheduler configured\n- Email delivery system operational\n- Report templates defined\n- Cloud storage for report archives\n\n## Done Criteria\n\n**Scheduling Options:**\n- [ ] Daily, weekly, monthly schedules\n- [ ] Custom cron expressions for advanced users\n- [ ] Timezone selection\n- [ ] Start date and optional end date\n\n**Delivery Methods:**\n- [ ] Email with PDF attachment\n- [ ] Email with download link\n- [ ] Save to cloud storage (S3, GCS)\n- [ ] Webhook notification\n\n**Report Types:**\n- [ ] User activity summary\n- [ ] Performance metrics\n- [ ] Compliance audit report\n- [ ] Custom report builder\n\n## How To\n\n**Creating a Scheduled Report:**\n1. Go to Reports \u2192 Scheduled Reports\n2. Click \"New Schedule\"\n3. Select report type and parameters\n4. Choose frequency (daily/weekly/monthly)\n5. Add email recipients\n6. Click \"Create Schedule\"\n\n**Managing Schedules:**\n- View all schedules in dashboard\n- Pause/resume schedules\n- Edit schedule parameters\n- View execution history\n- Download previous reports\n\n## History\n\nImplemented by emma@example.com on 2026-02-12", + "issue_number": 203 }, - { - "number": 204, + "AbsaOSS/living-doc-v2/204": { "title": "Bulk data import from CSV/Excel", "state": "open", - "labels": ["feature", "import"], + "labels": [ + "feature", + "import" + ], "html_url": "https://github.com/AbsaOSS/living-doc-v2/issues/204", "created_at": "2026-02-04T11:00:00Z", "updated_at": "2026-02-13T09:30:00Z", - "body": "## Description\n\nEnable bulk import of data from CSV and Excel files with validation and error handling.\n\n**Supported Entities:**\n- Users (bulk user provisioning)\n- Products (inventory updates)\n- Transactions (bulk data entry)\n- Configuration settings\n\n## Business Value\n\nAccelerates data migration and initial setup. Reduces manual data entry errors and saves hours of work.\n\n## Acceptance Criteria\n\n**File Upload:**\n- [ ] Drag-and-drop file upload\n- [ ] Support CSV and XLSX formats\n- [ ] Max file size: 10MB\n- [ ] Sample templates available for download\n\n**Validation:**\n- [ ] Column header mapping (automatic + manual)\n- [ ] Data type validation\n- [ ] Required field checking\n- [ ] Duplicate detection\n- [ ] Preview before import (first 10 rows)\n\n**Processing:**\n- [ ] Background processing for large files\n- [ ] Progress bar with percentage complete\n- [ ] Partial import on errors (continue processing valid rows)\n- [ ] Detailed error report for failed rows\n- [ ] Transaction rollback option\n\n**Error Handling:**\n- [ ] Download error report as CSV\n- [ ] Highlight problematic rows and columns\n- [ ] Suggested fixes for common errors\n- [ ] Ability to fix and re-upload\n\n## User Guide\n\n**Import Process:**\n1. Click \"Import\" button\n2. Select entity type (Users, Products, etc.)\n3. Download template file (optional)\n4. Upload your populated CSV/Excel file\n5. Map columns if automatic mapping fails\n6. Review preview of first 10 rows\n7. Click \"Import\" to process\n8. Monitor progress bar\n9. Review completion summary\n10. Download error report if needed\n\n**Template Format:**\nDownload templates include:\n- Required column headers\n- Example data rows\n- Data type comments\n- Validation rules\n\n## Connections\n\nRelated to #202 (data export)" + "body": "## Description\n\nEnable bulk import of data from CSV and Excel files with validation and error handling.\n\n**Supported Entities:**\n- Users (bulk user provisioning)\n- Products (inventory updates)\n- Transactions (bulk data entry)\n- Configuration settings\n\n## Business Value\n\nAccelerates data migration and initial setup. Reduces manual data entry errors and saves hours of work.\n\n## Acceptance Criteria\n\n**File Upload:**\n- [ ] Drag-and-drop file upload\n- [ ] Support CSV and XLSX formats\n- [ ] Max file size: 10MB\n- [ ] Sample templates available for download\n\n**Validation:**\n- [ ] Column header mapping (automatic + manual)\n- [ ] Data type validation\n- [ ] Required field checking\n- [ ] Duplicate detection\n- [ ] Preview before import (first 10 rows)\n\n**Processing:**\n- [ ] Background processing for large files\n- [ ] Progress bar with percentage complete\n- [ ] Partial import on errors (continue processing valid rows)\n- [ ] Detailed error report for failed rows\n- [ ] Transaction rollback option\n\n**Error Handling:**\n- [ ] Download error report as CSV\n- [ ] Highlight problematic rows and columns\n- [ ] Suggested fixes for common errors\n- [ ] Ability to fix and re-upload\n\n## User Guide\n\n**Import Process:**\n1. Click \"Import\" button\n2. Select entity type (Users, Products, etc.)\n3. Download template file (optional)\n4. Upload your populated CSV/Excel file\n5. Map columns if automatic mapping fails\n6. Review preview of first 10 rows\n7. Click \"Import\" to process\n8. Monitor progress bar\n9. Review completion summary\n10. Download error report if needed\n\n**Template Format:**\nDownload templates include:\n- Required column headers\n- Example data rows\n- Data type comments\n- Validation rules\n\n## Connections\n\nRelated to #202 (data export)", + "issue_number": 204 }, - { - "number": 205, + "AbsaOSS/living-doc-v2/205": { "title": "Implement full-text search with Elasticsearch", "state": "open", - "labels": ["feature", "search", "infrastructure"], + "labels": [ + "feature", + "search", + "infrastructure" + ], "html_url": "https://github.com/AbsaOSS/living-doc-v2/issues/205", "created_at": "2026-02-05T08:30:00Z", "updated_at": "2026-02-14T11:00:00Z", - "body": "## Overview\n\nReplace basic SQL LIKE queries with Elasticsearch for powerful full-text search capabilities.\n\n## Why\n\n- Current search is slow on large datasets (3+ seconds)\n- No support for fuzzy matching or typo tolerance\n- Cannot search across multiple fields efficiently\n- No relevance ranking\n\n## Setup\n\n**Infrastructure:**\n- Elasticsearch 8.x cluster (3 nodes)\n- Kibana for monitoring and debugging\n- Logstash for data synchronization\n- 100GB storage allocation\n\n**Data Synchronization:**\n- Full index rebuild: Weekly (Sunday 2 AM)\n- Incremental updates: Real-time via change data capture\n- Fallback: Hourly delta sync job\n\n## AC\n\n**Search Features:**\n- [ ] Full-text search across all text fields\n- [ ] Fuzzy matching (handle typos)\n- [ ] Phrase matching with quotes\n- [ ] Boolean operators (AND, OR, NOT)\n- [ ] Wildcard search (* and ?)\n- [ ] Field-specific search (title:\"example\")\n- [ ] Relevance-based ranking\n- [ ] Search suggestions (autocomplete)\n\n**Performance:**\n- [ ] Search response time < 200ms (p95)\n- [ ] Support 100 concurrent searches\n- [ ] Index size < 5GB\n\n**Indexing:**\n- [ ] Index users, documents, messages, products\n- [ ] Custom analyzers for each language\n- [ ] Configurable field weights for ranking\n\n## Instructions\n\n**Basic Search:**\n- Enter search terms in global search box\n- Press Enter or click search icon\n- Results appear with highlighting\n\n**Advanced Search:**\n- Use quotes for exact phrases: \"product manager\"\n- Combine terms: python AND django\n- Exclude terms: java NOT javascript \n- Field search: author:\"john doe\"\n- Wildcard: test*\n\n## Related\n\n- #206 - Search analytics\n- External: [Elasticsearch Guide](https://elastic.co/guide)" + "body": "## Overview\n\nReplace basic SQL LIKE queries with Elasticsearch for powerful full-text search capabilities.\n\n## Why\n\n- Current search is slow on large datasets (3+ seconds)\n- No support for fuzzy matching or typo tolerance\n- Cannot search across multiple fields efficiently\n- No relevance ranking\n\n## Setup\n\n**Infrastructure:**\n- Elasticsearch 8.x cluster (3 nodes)\n- Kibana for monitoring and debugging\n- Logstash for data synchronization\n- 100GB storage allocation\n\n**Data Synchronization:**\n- Full index rebuild: Weekly (Sunday 2 AM)\n- Incremental updates: Real-time via change data capture\n- Fallback: Hourly delta sync job\n\n## AC\n\n**Search Features:**\n- [ ] Full-text search across all text fields\n- [ ] Fuzzy matching (handle typos)\n- [ ] Phrase matching with quotes\n- [ ] Boolean operators (AND, OR, NOT)\n- [ ] Wildcard search (* and ?)\n- [ ] Field-specific search (title:\"example\")\n- [ ] Relevance-based ranking\n- [ ] Search suggestions (autocomplete)\n\n**Performance:**\n- [ ] Search response time < 200ms (p95)\n- [ ] Support 100 concurrent searches\n- [ ] Index size < 5GB\n\n**Indexing:**\n- [ ] Index users, documents, messages, products\n- [ ] Custom analyzers for each language\n- [ ] Configurable field weights for ranking\n\n## Instructions\n\n**Basic Search:**\n- Enter search terms in global search box\n- Press Enter or click search icon\n- Results appear with highlighting\n\n**Advanced Search:**\n- Use quotes for exact phrases: \"product manager\"\n- Combine terms: python AND django\n- Exclude terms: java NOT javascript \n- Field search: author:\"john doe\"\n- Wildcard: test*\n\n## Related\n\n- #206 - Search analytics\n- External: [Elasticsearch Guide](https://elastic.co/guide)", + "issue_number": 205 }, - { - "number": 206, + "AbsaOSS/living-doc-v2/206": { "title": "Search analytics and insights", "state": "open", - "labels": ["analytics", "search"], + "labels": [ + "analytics", + "search" + ], "html_url": "https://github.com/AbsaOSS/living-doc-v2/issues/206", "created_at": "2026-02-06T09:45:00Z", "updated_at": "2026-02-15T10:15:00Z", - "body": "## Description\n\nTrack and analyze search behavior to improve search relevance and understand user needs.\n\n## Value\n\nInsights from search data help improve content organization, identify gaps, and optimize search algorithms.\n\n## Done Criteria\n\n**Tracked Metrics:**\n- Search queries (text, filters, results count)\n- Click-through rate (which results users clicked)\n- Zero-result searches (queries with no results)\n- Search refinements (query modifications)\n- Time to click (search duration)\n- Abandoned searches\n\n**Analytics Dashboard:**\n- [ ] Top 100 search queries\n- [ ] Most clicked results\n- [ ] Zero-result query report\n- [ ] Search success rate\n- [ ] Average results per query\n- [ ] Search trends over time\n\n**Actionable Insights:**\n- [ ] Recommend synonyms for common searches\n- [ ] Identify content gaps (frequent zero-result queries)\n- [ ] Suggest popular content for homepage\n- [ ] Alert on broken links in search results" + "body": "## Description\n\nTrack and analyze search behavior to improve search relevance and understand user needs.\n\n## Value\n\nInsights from search data help improve content organization, identify gaps, and optimize search algorithms.\n\n## Done Criteria\n\n**Tracked Metrics:**\n- Search queries (text, filters, results count)\n- Click-through rate (which results users clicked)\n- Zero-result searches (queries with no results)\n- Search refinements (query modifications)\n- Time to click (search duration)\n- Abandoned searches\n\n**Analytics Dashboard:**\n- [ ] Top 100 search queries\n- [ ] Most clicked results\n- [ ] Zero-result query report\n- [ ] Search success rate\n- [ ] Average results per query\n- [ ] Search trends over time\n\n**Actionable Insights:**\n- [ ] Recommend synonyms for common searches\n- [ ] Identify content gaps (frequent zero-result queries)\n- [ ] Suggest popular content for homepage\n- [ ] Alert on broken links in search results", + "issue_number": 206 }, - { - "number": 207, + "AbsaOSS/living-doc-v2/207": { "title": "Implement caching strategy with Redis", "state": "closed", - "labels": ["performance", "caching"], + "labels": [ + "performance", + "caching" + ], "html_url": "https://github.com/AbsaOSS/living-doc-v2/issues/207", "created_at": "2026-02-07T10:00:00Z", "updated_at": "2026-02-16T14:30:00Z", - "body": "## Summary\n\nImplement comprehensive caching strategy using Redis to improve application performance.\n\n## Business Value\n\n- 70% reduction in database load\n- 5x faster page load times for cached content\n- Better scalability and reduced infrastructure costs\n\n## Prerequisites\n\n- Redis cluster deployed (3 nodes, master-replica setup)\n- Cache warming strategy defined\n- Cache invalidation events identified\n\n## Acceptance Criteria\n\n**Cache Layers:**\n\n1. **Page Cache** (TTL: 5 minutes)\n - Full HTML responses for public pages\n - Vary by user role for personalized pages\n \n2. **API Response Cache** (TTL: 2 minutes)\n - GET endpoints only\n - Cache key includes query parameters\n - Skip for user-specific data\n \n3. **Database Query Cache** (TTL: 10 minutes)\n - Expensive queries (joins, aggregations)\n - Reference data (countries, categories)\n \n4. **Session Cache** (TTL: 8 hours)\n - User sessions and authentication tokens\n - Shopping cart data\n\n**Cache Invalidation:**\n- [ ] Automatic invalidation on data updates\n- [ ] Manual cache clear for administrators\n- [ ] Tag-based invalidation (e.g., clear all user-related caches)\n- [ ] LRU eviction when memory limit reached\n\n**Monitoring:**\n- [ ] Cache hit/miss rate metrics\n- [ ] Memory usage dashboard\n- [ ] Slow cache operations alerts\n\n## History\n\nDeployed by frank@example.com on 2026-02-16" + "body": "## Summary\n\nImplement comprehensive caching strategy using Redis to improve application performance.\n\n## Business Value\n\n- 70% reduction in database load\n- 5x faster page load times for cached content\n- Better scalability and reduced infrastructure costs\n\n## Prerequisites\n\n- Redis cluster deployed (3 nodes, master-replica setup)\n- Cache warming strategy defined\n- Cache invalidation events identified\n\n## Acceptance Criteria\n\n**Cache Layers:**\n\n1. **Page Cache** (TTL: 5 minutes)\n - Full HTML responses for public pages\n - Vary by user role for personalized pages\n \n2. **API Response Cache** (TTL: 2 minutes)\n - GET endpoints only\n - Cache key includes query parameters\n - Skip for user-specific data\n \n3. **Database Query Cache** (TTL: 10 minutes)\n - Expensive queries (joins, aggregations)\n - Reference data (countries, categories)\n \n4. **Session Cache** (TTL: 8 hours)\n - User sessions and authentication tokens\n - Shopping cart data\n\n**Cache Invalidation:**\n- [ ] Automatic invalidation on data updates\n- [ ] Manual cache clear for administrators\n- [ ] Tag-based invalidation (e.g., clear all user-related caches)\n- [ ] LRU eviction when memory limit reached\n\n**Monitoring:**\n- [ ] Cache hit/miss rate metrics\n- [ ] Memory usage dashboard\n- [ ] Slow cache operations alerts\n\n## History\n\nDeployed by frank@example.com on 2026-02-16", + "issue_number": 207 }, - { - "number": 208, + "AbsaOSS/living-doc-v2/208": { "title": "Add comprehensive input validation", "state": "open", - "labels": ["security", "validation"], + "labels": [ + "security", + "validation" + ], "html_url": "https://github.com/AbsaOSS/living-doc-v2/issues/208", "created_at": "2026-02-08T11:15:00Z", "updated_at": "2026-02-17T09:00:00Z", - "body": "## Overview\n\nImplement comprehensive server-side input validation to prevent injection attacks and data corruption.\n\n## Why\n\nSecurity best practice. Prevents SQL injection, XSS, command injection, and ensures data integrity.\n\n## Setup\n\nValidation library setup:\n- Backend: Pydantic for Python, Joi for Node.js\n- Sanitization: bleach for HTML, validator.js for strings\n- Content Security Policy headers\n\n## AC\n\n**Validation Rules:**\n\nEmail fields:\n- [ ] Valid email format\n- [ ] Domain whitelist (optional)\n- [ ] Max length: 254 characters\n\nText inputs:\n- [ ] HTML tag stripping or escaping\n- [ ] Max length enforcement\n- [ ] No control characters\n- [ ] Unicode normalization\n\nNumeric inputs:\n- [ ] Type validation (int vs float)\n- [ ] Range checking (min/max)\n- [ ] Precision limits\n\nFile uploads:\n- [ ] File extension whitelist\n- [ ] MIME type verification\n- [ ] File size limits\n- [ ] Malware scanning\n\n**Error Messages:**\n- [ ] User-friendly messages (not technical)\n- [ ] Field-specific errors\n- [ ] Multiple errors displayed together\n- [ ] No sensitive information in errors\n\n## User Guide\n\nWhen validation fails:\n- Red border appears around invalid field\n- Error message appears below field\n- Submit button remains disabled\n- Fix all errors to enable submission" + "body": "## Overview\n\nImplement comprehensive server-side input validation to prevent injection attacks and data corruption.\n\n## Why\n\nSecurity best practice. Prevents SQL injection, XSS, command injection, and ensures data integrity.\n\n## Setup\n\nValidation library setup:\n- Backend: Pydantic for Python, Joi for Node.js\n- Sanitization: bleach for HTML, validator.js for strings\n- Content Security Policy headers\n\n## AC\n\n**Validation Rules:**\n\nEmail fields:\n- [ ] Valid email format\n- [ ] Domain whitelist (optional)\n- [ ] Max length: 254 characters\n\nText inputs:\n- [ ] HTML tag stripping or escaping\n- [ ] Max length enforcement\n- [ ] No control characters\n- [ ] Unicode normalization\n\nNumeric inputs:\n- [ ] Type validation (int vs float)\n- [ ] Range checking (min/max)\n- [ ] Precision limits\n\nFile uploads:\n- [ ] File extension whitelist\n- [ ] MIME type verification\n- [ ] File size limits\n- [ ] Malware scanning\n\n**Error Messages:**\n- [ ] User-friendly messages (not technical)\n- [ ] Field-specific errors\n- [ ] Multiple errors displayed together\n- [ ] No sensitive information in errors\n\n## User Guide\n\nWhen validation fails:\n- Red border appears around invalid field\n- Error message appears below field\n- Submit button remains disabled\n- Fix all errors to enable submission", + "issue_number": 208 }, - { - "number": 209, + "AbsaOSS/living-doc-v2/209": { "title": "Implement GDPR compliance features", "state": "open", - "labels": ["compliance", "privacy"], + "labels": [ + "compliance", + "privacy" + ], "html_url": "https://github.com/AbsaOSS/living-doc-v2/issues/209", "created_at": "2026-02-09T08:00:00Z", "updated_at": "2026-02-18T10:30:00Z", - "body": "## Description\n\nImplement features required for GDPR compliance including consent management, data portability, and right to be forgotten.\n\n## Business Value\n\nLegal requirement for EU users. Builds trust and demonstrates commitment to privacy. Avoids hefty fines (up to 4% of revenue).\n\n## Preconditions\n\n- Legal review of privacy policy\n- Data processing agreements with third parties\n- Data classification and inventory complete\n- Privacy impact assessment approved\n\n## Acceptance Criteria\n\n**Consent Management:**\n- [ ] Cookie consent banner on first visit\n- [ ] Granular consent options (necessary, functional, marketing)\n- [ ] Easy withdrawal of consent\n- [ ] Consent records stored with timestamp\n- [ ] Opt-out links in all marketing emails\n\n**Data Subject Rights:**\n\n1. **Right to Access**\n - [ ] User can download all personal data (JSON format)\n - [ ] Data export includes: profile, activity logs, communications\n - [ ] Export generated within 24 hours\n \n2. **Right to Rectification** \n - [ ] User can edit profile information\n - [ ] Request form for data corrections\n \n3. **Right to Erasure (Right to be Forgotten)**\n - [ ] Account deletion request form\n - [ ] 30-day grace period before permanent deletion\n - [ ] Anonymize user data instead of deletion (where legally required to retain)\n - [ ] Email confirmation of deletion\n \n4. **Right to Data Portability**\n - [ ] Export data in machine-readable format (JSON/CSV)\n - [ ] Transfer data directly to another service (if possible)\n\n**Privacy by Design:**\n- [ ] Data minimization (collect only necessary data)\n- [ ] Pseudonymization of personal data in logs\n- [ ] Encryption at rest and in transit\n- [ ] Access controls and audit logging\n- [ ] Automated data retention policies\n\n## User Guide\n\n**Accessing Your Data:**\n1. Go to Settings → Privacy\n2. Click \"Download My Data\"\n3. Email sent when export ready (usually < 1 hour)\n4. Download link valid for 48 hours\n\n**Deleting Your Account:**\n1. Go to Settings → Privacy\n2. Click \"Delete My Account\"\n3. Enter password to confirm\n4. Account enters 30-day deletion queue\n5. Cancel deletion anytime within 30 days\n6. After 30 days, deletion is permanent\n\n## Connections\n\n- Related to #112 (audit logging)\n- Related to legal compliance initiative" + "body": "## Description\n\nImplement features required for GDPR compliance including consent management, data portability, and right to be forgotten.\n\n## Business Value\n\nLegal requirement for EU users. Builds trust and demonstrates commitment to privacy. Avoids hefty fines (up to 4% of revenue).\n\n## Preconditions\n\n- Legal review of privacy policy\n- Data processing agreements with third parties\n- Data classification and inventory complete\n- Privacy impact assessment approved\n\n## Acceptance Criteria\n\n**Consent Management:**\n- [ ] Cookie consent banner on first visit\n- [ ] Granular consent options (necessary, functional, marketing)\n- [ ] Easy withdrawal of consent\n- [ ] Consent records stored with timestamp\n- [ ] Opt-out links in all marketing emails\n\n**Data Subject Rights:**\n\n1. **Right to Access**\n - [ ] User can download all personal data (JSON format)\n - [ ] Data export includes: profile, activity logs, communications\n - [ ] Export generated within 24 hours\n \n2. **Right to Rectification** \n - [ ] User can edit profile information\n - [ ] Request form for data corrections\n \n3. **Right to Erasure (Right to be Forgotten)**\n - [ ] Account deletion request form\n - [ ] 30-day grace period before permanent deletion\n - [ ] Anonymize user data instead of deletion (where legally required to retain)\n - [ ] Email confirmation of deletion\n \n4. **Right to Data Portability**\n - [ ] Export data in machine-readable format (JSON/CSV)\n - [ ] Transfer data directly to another service (if possible)\n\n**Privacy by Design:**\n- [ ] Data minimization (collect only necessary data)\n- [ ] Pseudonymization of personal data in logs\n- [ ] Encryption at rest and in transit\n- [ ] Access controls and audit logging\n- [ ] Automated data retention policies\n\n## User Guide\n\n**Accessing Your Data:**\n1. Go to Settings \u2192 Privacy\n2. Click \"Download My Data\"\n3. Email sent when export ready (usually < 1 hour)\n4. Download link valid for 48 hours\n\n**Deleting Your Account:**\n1. Go to Settings \u2192 Privacy\n2. Click \"Delete My Account\"\n3. Enter password to confirm\n4. Account enters 30-day deletion queue\n5. Cancel deletion anytime within 30 days\n6. After 30 days, deletion is permanent\n\n## Connections\n\n- Related to #112 (audit logging)\n- Related to legal compliance initiative", + "issue_number": 209 }, - { - "number": 210, + "AbsaOSS/living-doc-v2/210": { "title": "Internationalization (i18n) support", "state": "open", - "labels": ["feature", "i18n"], + "labels": [ + "feature", + "i18n" + ], "html_url": "https://github.com/AbsaOSS/living-doc-v2/issues/210", "created_at": "2026-02-10T09:30:00Z", "updated_at": "2026-02-18T11:45:00Z", - "body": "## Overview\n\nAdd internationalization support to enable multi-language user interface.\n\n## Value\n\nExpands market reach. Improves user experience for non-English speakers. Requirement for international expansion.\n\n## Prerequisites\n\n- Translation management system (e.g., Crowdin, Lokalise)\n- Professional translation service engaged\n- Language selection UI/UX designed\n- Right-to-left (RTL) layout considerations\n\n## Done Criteria\n\n**Supported Languages (Phase 1):**\n- [ ] English (default)\n- [ ] Spanish\n- [ ] French\n- [ ] German\n- [ ] Japanese\n\n**Implementation:**\n- [ ] Extract all UI strings to translation files\n- [ ] Use i18n library (react-i18next, Flask-Babel, etc.)\n- [ ] Language selector in header\n- [ ] User language preference saved\n- [ ] Browser language auto-detection\n- [ ] Fallback to English for missing translations\n\n**Content Types:**\n- [ ] UI labels and buttons\n- [ ] Error messages\n- [ ] Email templates\n- [ ] Help documentation\n- [ ] Date/time formatting per locale\n- [ ] Number formatting (decimals, thousands separators)\n- [ ] Currency formatting\n\n**Quality:**\n- [ ] 100% translation coverage for supported languages\n- [ ] Professional translation review\n- [ ] Test all workflows in each language\n- [ ] No hardcoded strings in code\n\n## How To\n\n**Changing Language:**\n1. Click language selector in header (globe icon)\n2. Select desired language from dropdown\n3. Page refreshes with new language\n4. Preference saved for future visits\n\n**For Developers:**\n```javascript\n// Before\n\n\n// After \n\n```\n\n## Related\n\n- #211 - Right-to-left (RTL) language support" + "body": "## Overview\n\nAdd internationalization support to enable multi-language user interface.\n\n## Value\n\nExpands market reach. Improves user experience for non-English speakers. Requirement for international expansion.\n\n## Prerequisites\n\n- Translation management system (e.g., Crowdin, Lokalise)\n- Professional translation service engaged\n- Language selection UI/UX designed\n- Right-to-left (RTL) layout considerations\n\n## Done Criteria\n\n**Supported Languages (Phase 1):**\n- [ ] English (default)\n- [ ] Spanish\n- [ ] French\n- [ ] German\n- [ ] Japanese\n\n**Implementation:**\n- [ ] Extract all UI strings to translation files\n- [ ] Use i18n library (react-i18next, Flask-Babel, etc.)\n- [ ] Language selector in header\n- [ ] User language preference saved\n- [ ] Browser language auto-detection\n- [ ] Fallback to English for missing translations\n\n**Content Types:**\n- [ ] UI labels and buttons\n- [ ] Error messages\n- [ ] Email templates\n- [ ] Help documentation\n- [ ] Date/time formatting per locale\n- [ ] Number formatting (decimals, thousands separators)\n- [ ] Currency formatting\n\n**Quality:**\n- [ ] 100% translation coverage for supported languages\n- [ ] Professional translation review\n- [ ] Test all workflows in each language\n- [ ] No hardcoded strings in code\n\n## How To\n\n**Changing Language:**\n1. Click language selector in header (globe icon)\n2. Select desired language from dropdown\n3. Page refreshes with new language\n4. Preference saved for future visits\n\n**For Developers:**\n```javascript\n// Before\n\n\n// After \n\n```\n\n## Related\n\n- #211 - Right-to-left (RTL) language support", + "issue_number": 210 }, - { - "number": 211, + "AbsaOSS/living-doc-v2/211": { "title": "Support right-to-left (RTL) languages", "state": "open", - "labels": ["i18n", "ui"], + "labels": [ + "i18n", + "ui" + ], "html_url": "https://github.com/AbsaOSS/living-doc-v2/issues/211", "created_at": "2026-02-11T10:00:00Z", "updated_at": "2026-02-18T13:00:00Z", - "body": "## Description\n\nAdd support for right-to-left languages (Arabic, Hebrew) including layout mirroring and text direction.\n\n## Value\n\nEnables product usage in Middle East and Israel markets. Shows cultural sensitivity and attention to detail.\n\n## Setup\n\n- RTL CSS framework or utility classes\n- Bi-directional text handling\n- Icon flipping for directional icons\n\n## Acceptance Criteria\n\n**Layout:**\n- [ ] Entire UI mirrors for RTL languages\n- [ ] Sidebar moves to right side\n- [ ] Text alignment: right for RTL, left for LTR\n- [ ] Directional icons flipped (arrows, chevrons)\n- [ ] Margins and padding reversed\n\n**Text Direction:**\n- [ ] HTML dir attribute set correctly (rtl/ltr)\n- [ ] Bidirectional text handled correctly (mixed RTL/LTR)\n- [ ] Form inputs right-aligned for RTL\n\n**Testing:**\n- [ ] Test with Arabic language\n- [ ] Test with Hebrew language\n- [ ] Test mixed content (English + RTL)\n- [ ] Verify all pages render correctly\n\n## Connections\n\nDepends on #210 (i18n support)" + "body": "## Description\n\nAdd support for right-to-left languages (Arabic, Hebrew) including layout mirroring and text direction.\n\n## Value\n\nEnables product usage in Middle East and Israel markets. Shows cultural sensitivity and attention to detail.\n\n## Setup\n\n- RTL CSS framework or utility classes\n- Bi-directional text handling\n- Icon flipping for directional icons\n\n## Acceptance Criteria\n\n**Layout:**\n- [ ] Entire UI mirrors for RTL languages\n- [ ] Sidebar moves to right side\n- [ ] Text alignment: right for RTL, left for LTR\n- [ ] Directional icons flipped (arrows, chevrons)\n- [ ] Margins and padding reversed\n\n**Text Direction:**\n- [ ] HTML dir attribute set correctly (rtl/ltr)\n- [ ] Bidirectional text handled correctly (mixed RTL/LTR)\n- [ ] Form inputs right-aligned for RTL\n\n**Testing:**\n- [ ] Test with Arabic language\n- [ ] Test with Hebrew language\n- [ ] Test mixed content (English + RTL)\n- [ ] Verify all pages render correctly\n\n## Connections\n\nDepends on #210 (i18n support)", + "issue_number": 211 }, - { - "number": 212, + "AbsaOSS/living-doc-v2/212": { "title": "Implement feature flags system", "state": "closed", - "labels": ["infrastructure", "deployment"], + "labels": [ + "infrastructure", + "deployment" + ], "html_url": "https://github.com/AbsaOSS/living-doc-v2/issues/212", "created_at": "2026-02-12T11:30:00Z", "updated_at": "2026-02-17T15:00:00Z", - "body": "## Summary\n\nImplement feature flags system for gradual rollouts and A/B testing.\n\n## Why\n\nEnables:\n- Safe deployment of new features\n- Gradual rollout to subset of users\n- Quick rollback without code deployment\n- A/B testing for feature validation\n- Different features for different user segments\n\n## Prerequisites\n\n- Feature flag service (LaunchDarkly, Unleash, or custom)\n- Admin UI for flag management\n- SDK integration in application code\n\n## Acceptance Criteria\n\n**Flag Types:**\n- [ ] Boolean flags (on/off)\n- [ ] Percentage rollouts (10%, 25%, 50%, etc.)\n- [ ] User segment targeting (beta users, premium, etc.)\n- [ ] A/B test variants (variant A vs B vs C)\n\n**Flag Management:**\n- [ ] Create/edit/delete flags via admin UI\n- [ ] Flag overrides for testing (per-user, per-session)\n- [ ] Flag change history and audit log\n- [ ] Schedule flag changes (enable at specific time)\n\n**Integration:**\n- [ ] Backend flag evaluation\n- [ ] Frontend flag evaluation\n- [ ] Real-time flag updates (no deployment needed)\n- [ ] Default values when flag service unavailable\n\n**Examples:**\n```python\nif feature_flags.is_enabled('new_dashboard', user):\n return render_new_dashboard()\nelse:\n return render_old_dashboard()\n```\n\n## Changes\n\nRolled out by grace@example.com on 2026-02-17" + "body": "## Summary\n\nImplement feature flags system for gradual rollouts and A/B testing.\n\n## Why\n\nEnables:\n- Safe deployment of new features\n- Gradual rollout to subset of users\n- Quick rollback without code deployment\n- A/B testing for feature validation\n- Different features for different user segments\n\n## Prerequisites\n\n- Feature flag service (LaunchDarkly, Unleash, or custom)\n- Admin UI for flag management\n- SDK integration in application code\n\n## Acceptance Criteria\n\n**Flag Types:**\n- [ ] Boolean flags (on/off)\n- [ ] Percentage rollouts (10%, 25%, 50%, etc.)\n- [ ] User segment targeting (beta users, premium, etc.)\n- [ ] A/B test variants (variant A vs B vs C)\n\n**Flag Management:**\n- [ ] Create/edit/delete flags via admin UI\n- [ ] Flag overrides for testing (per-user, per-session)\n- [ ] Flag change history and audit log\n- [ ] Schedule flag changes (enable at specific time)\n\n**Integration:**\n- [ ] Backend flag evaluation\n- [ ] Frontend flag evaluation\n- [ ] Real-time flag updates (no deployment needed)\n- [ ] Default values when flag service unavailable\n\n**Examples:**\n```python\nif feature_flags.is_enabled('new_dashboard', user):\n return render_new_dashboard()\nelse:\n return render_old_dashboard()\n```\n\n## Changes\n\nRolled out by grace@example.com on 2026-02-17", + "issue_number": 212 }, - { - "number": 213, + "AbsaOSS/living-doc-v2/213": { "title": "Simple issue with markdown table", "state": "open", - "labels": ["test"], + "labels": [ + "test" + ], "html_url": "https://github.com/AbsaOSS/living-doc-v2/issues/213", "created_at": "2026-02-13T12:00:00Z", "updated_at": "2026-02-13T12:00:00Z", - "body": "## Description\n\nTest case with markdown table:\n\n| Column 1 | Column 2 | Column 3 |\n|----------|----------|----------|\n| Value A | Value B | Value C |\n| Value D | Value E | Value F |\n\nThis tests table preservation in normalization." + "body": "## Description\n\nTest case with markdown table:\n\n| Column 1 | Column 2 | Column 3 |\n|----------|----------|----------|\n| Value A | Value B | Value C |\n| Value D | Value E | Value F |\n\nThis tests table preservation in normalization.", + "issue_number": 213 }, - { - "number": 214, + "AbsaOSS/living-doc-v2/214": { "title": "Issue with code blocks and lists", "state": "closed", - "labels": ["test"], + "labels": [ + "test" + ], "html_url": "https://github.com/AbsaOSS/living-doc-v2/issues/214", "created_at": "2026-02-14T13:30:00Z", "updated_at": "2026-02-14T14:00:00Z", - "body": "## Description\n\nTest complex markdown:\n\n```python\ndef hello_world():\n print(\"Hello, World!\")\n return 42\n```\n\n## Acceptance Criteria\n\n1. First criterion\n2. Second criterion \n - Nested item A\n - Nested item B\n3. Third criterion\n\n## User Guide\n\n- **Bold item**: description\n- *Italic item*: description\n- `Code item`: description" + "body": "## Description\n\nTest complex markdown:\n\n```python\ndef hello_world():\n print(\"Hello, World!\")\n return 42\n```\n\n## Acceptance Criteria\n\n1. First criterion\n2. Second criterion \n - Nested item A\n - Nested item B\n3. Third criterion\n\n## User Guide\n\n- **Bold item**: description\n- *Italic item*: description\n- `Code item`: description", + "issue_number": 214 } - ] -} + } +} \ No newline at end of file diff --git a/tests/fixtures/collector_gh/v2.0.0/input/doc-issues.json b/tests/fixtures/collector_gh/v2.0.0/input/doc-issues.json index e0e51f7..ce1e93c 100644 --- a/tests/fixtures/collector_gh/v2.0.0/input/doc-issues.json +++ b/tests/fixtures/collector_gh/v2.0.0/input/doc-issues.json @@ -14,142 +14,185 @@ "sha": "xyz789future123" }, "source": { - "systems": ["GitHub"], - "repositories": ["AbsaOSS/future-project"], + "systems": [ + "GitHub" + ], + "repositories": [ + "AbsaOSS/future-project" + ], "organization": "AbsaOSS", "enterprise": null } }, - "issues": [ - { - "number": 301, + "issues": { + "AbsaOSS/future-project/301": { "title": "AI-powered code review assistant", "state": "open", - "labels": ["ai", "feature", "priority:critical"], + "labels": [ + "ai", + "feature", + "priority:critical" + ], "html_url": "https://github.com/AbsaOSS/future-project/issues/301", "created_at": "2026-03-01T08:00:00Z", "updated_at": "2026-03-10T16:30:00Z", - "body": "## Description\n\nIntegrate AI-powered code review assistant that automatically reviews pull requests and provides intelligent feedback.\n\n## Business Value\n\n- Reduces code review time by 60%\n- Catches common bugs and security issues automatically\n- Improves code quality consistency\n- Frees developers to focus on architectural reviews\n\n## Preconditions\n\n- AI model trained on company codebase\n- API integration with OpenAI or custom model\n- Pull request webhook configuration\n- Review comment posting permissions\n\n## Acceptance Criteria\n\n**Automated Checks:**\n- [ ] Code style and formatting issues\n- [ ] Potential bugs and logic errors\n- [ ] Security vulnerabilities (SQL injection, XSS)\n- [ ] Performance anti-patterns\n- [ ] Missing test coverage\n- [ ] Documentation gaps\n\n**Review Comments:**\n- [ ] Posted as PR comments with line numbers\n- [ ] Severity levels: critical, warning, suggestion\n- [ ] Explanation and suggested fix for each issue\n- [ ] Links to style guide or documentation\n\n**Intelligence:**\n- [ ] Learn from accepted/rejected suggestions\n- [ ] Adapt to team coding style\n- [ ] Context-aware (understands project architecture)\n\n## User Guide\n\n**For Developers:**\n1. Create pull request as usual\n2. AI assistant reviews within 2 minutes\n3. Review comments appear on specific lines\n4. Address critical issues before merging\n5. Accept or dismiss suggestions\n\n**For Team Leads:**\n- Configure AI review rules in settings\n- Set severity thresholds for blocking\n- Review AI performance metrics\n- Provide feedback on AI suggestions" + "body": "## Description\n\nIntegrate AI-powered code review assistant that automatically reviews pull requests and provides intelligent feedback.\n\n## Business Value\n\n- Reduces code review time by 60%\n- Catches common bugs and security issues automatically\n- Improves code quality consistency\n- Frees developers to focus on architectural reviews\n\n## Preconditions\n\n- AI model trained on company codebase\n- API integration with OpenAI or custom model\n- Pull request webhook configuration\n- Review comment posting permissions\n\n## Acceptance Criteria\n\n**Automated Checks:**\n- [ ] Code style and formatting issues\n- [ ] Potential bugs and logic errors\n- [ ] Security vulnerabilities (SQL injection, XSS)\n- [ ] Performance anti-patterns\n- [ ] Missing test coverage\n- [ ] Documentation gaps\n\n**Review Comments:**\n- [ ] Posted as PR comments with line numbers\n- [ ] Severity levels: critical, warning, suggestion\n- [ ] Explanation and suggested fix for each issue\n- [ ] Links to style guide or documentation\n\n**Intelligence:**\n- [ ] Learn from accepted/rejected suggestions\n- [ ] Adapt to team coding style\n- [ ] Context-aware (understands project architecture)\n\n## User Guide\n\n**For Developers:**\n1. Create pull request as usual\n2. AI assistant reviews within 2 minutes\n3. Review comments appear on specific lines\n4. Address critical issues before merging\n5. Accept or dismiss suggestions\n\n**For Team Leads:**\n- Configure AI review rules in settings\n- Set severity thresholds for blocking\n- Review AI performance metrics\n- Provide feedback on AI suggestions", + "issue_number": 301 }, - { - "number": 302, + "AbsaOSS/future-project/302": { "title": "Blockchain-based audit trail", "state": "open", - "labels": ["blockchain", "security"], + "labels": [ + "blockchain", + "security" + ], "html_url": "https://github.com/AbsaOSS/future-project/issues/302", "created_at": "2026-03-02T09:15:00Z", "updated_at": "2026-03-11T10:45:00Z", - "body": "## Overview\n\nImplement immutable blockchain-based audit trail for critical operations.\n\n## Why\n\nProvides cryptographically verifiable audit trail that cannot be tampered with. Meets highest regulatory compliance standards.\n\n## Setup\n\n- Private blockchain network (Hyperledger Fabric)\n- Smart contracts for audit logging\n- Consensus mechanism (RAFT)\n- 5 validator nodes across regions\n\n## AC\n\n**Logged Operations:**\n- [ ] Financial transactions\n- [ ] Data access and modifications\n- [ ] Administrative actions\n- [ ] Security events\n\n**Blockchain Features:**\n- [ ] Immutable append-only ledger\n- [ ] Cryptographic hash chain\n- [ ] Multi-party validation\n- [ ] Timestamp verification\n\n**Verification:**\n- [ ] Audit trail verification API\n- [ ] Cryptographic proof generation\n- [ ] Chain integrity checking\n- [ ] Export for regulatory audits" + "body": "## Overview\n\nImplement immutable blockchain-based audit trail for critical operations.\n\n## Why\n\nProvides cryptographically verifiable audit trail that cannot be tampered with. Meets highest regulatory compliance standards.\n\n## Setup\n\n- Private blockchain network (Hyperledger Fabric)\n- Smart contracts for audit logging\n- Consensus mechanism (RAFT)\n- 5 validator nodes across regions\n\n## AC\n\n**Logged Operations:**\n- [ ] Financial transactions\n- [ ] Data access and modifications\n- [ ] Administrative actions\n- [ ] Security events\n\n**Blockchain Features:**\n- [ ] Immutable append-only ledger\n- [ ] Cryptographic hash chain\n- [ ] Multi-party validation\n- [ ] Timestamp verification\n\n**Verification:**\n- [ ] Audit trail verification API\n- [ ] Cryptographic proof generation\n- [ ] Chain integrity checking\n- [ ] Export for regulatory audits", + "issue_number": 302 }, - { - "number": 303, + "AbsaOSS/future-project/303": { "title": "Quantum-resistant encryption", "state": "open", - "labels": ["security", "cryptography"], + "labels": [ + "security", + "cryptography" + ], "html_url": "https://github.com/AbsaOSS/future-project/issues/303", "created_at": "2026-03-03T10:30:00Z", "updated_at": "2026-03-12T14:00:00Z", - "body": "## Summary\n\nImplement quantum-resistant cryptographic algorithms to future-proof data security.\n\n## Value\n\nProtects against future quantum computing attacks. Ensures long-term data confidentiality.\n\n## Prerequisites\n\n- NIST-approved post-quantum algorithms\n- Library support (liboqs or similar)\n- Performance testing and benchmarks\n- Migration plan for existing encrypted data\n\n## Done Criteria\n\n**Algorithms:**\n- [ ] CRYSTALS-Kyber for key encapsulation\n- [ ] CRYSTALS-Dilithium for digital signatures\n- [ ] Hybrid mode (classical + quantum-resistant)\n\n**Migration:**\n- [ ] Gradual rollout over 6 months\n- [ ] Backward compatibility during transition\n- [ ] Re-encryption of critical data\n\n**Performance:**\n- [ ] Encryption overhead < 20% vs AES\n- [ ] Key generation time < 100ms\n- [ ] Signature verification < 10ms" + "body": "## Summary\n\nImplement quantum-resistant cryptographic algorithms to future-proof data security.\n\n## Value\n\nProtects against future quantum computing attacks. Ensures long-term data confidentiality.\n\n## Prerequisites\n\n- NIST-approved post-quantum algorithms\n- Library support (liboqs or similar)\n- Performance testing and benchmarks\n- Migration plan for existing encrypted data\n\n## Done Criteria\n\n**Algorithms:**\n- [ ] CRYSTALS-Kyber for key encapsulation\n- [ ] CRYSTALS-Dilithium for digital signatures\n- [ ] Hybrid mode (classical + quantum-resistant)\n\n**Migration:**\n- [ ] Gradual rollout over 6 months\n- [ ] Backward compatibility during transition\n- [ ] Re-encryption of critical data\n\n**Performance:**\n- [ ] Encryption overhead < 20% vs AES\n- [ ] Key generation time < 100ms\n- [ ] Signature verification < 10ms", + "issue_number": 303 }, - { - "number": 304, + "AbsaOSS/future-project/304": { "title": "Edge computing integration", "state": "closed", - "labels": ["infrastructure", "edge"], + "labels": [ + "infrastructure", + "edge" + ], "html_url": "https://github.com/AbsaOSS/future-project/issues/304", "created_at": "2026-03-04T11:00:00Z", "updated_at": "2026-03-13T09:30:00Z", - "body": "## Description\n\nDeploy application components to edge locations for ultra-low latency.\n\n## Business Value\n\n- Sub-10ms response times for users\n- Reduced bandwidth costs\n- Improved user experience globally\n- Enables real-time applications\n\n## Acceptance Criteria\n\n**Edge Deployment:**\n- [ ] Static assets on CDN edge\n- [ ] API endpoints on edge functions\n- [ ] Regional data caching\n- [ ] Geographic routing\n\n**Performance:**\n- [ ] p95 latency < 50ms worldwide\n- [ ] Cache hit rate > 90%\n- [ ] Auto-scaling at edge\n\n## History\n\nDeployed by henry@example.com on 2026-03-13" + "body": "## Description\n\nDeploy application components to edge locations for ultra-low latency.\n\n## Business Value\n\n- Sub-10ms response times for users\n- Reduced bandwidth costs\n- Improved user experience globally\n- Enables real-time applications\n\n## Acceptance Criteria\n\n**Edge Deployment:**\n- [ ] Static assets on CDN edge\n- [ ] API endpoints on edge functions\n- [ ] Regional data caching\n- [ ] Geographic routing\n\n**Performance:**\n- [ ] p95 latency < 50ms worldwide\n- [ ] Cache hit rate > 90%\n- [ ] Auto-scaling at edge\n\n## History\n\nDeployed by henry@example.com on 2026-03-13", + "issue_number": 304 }, - { - "number": 305, + "AbsaOSS/future-project/305": { "title": "Neural search with embeddings", "state": "open", - "labels": ["ai", "search"], + "labels": [ + "ai", + "search" + ], "html_url": "https://github.com/AbsaOSS/future-project/issues/305", "created_at": "2026-03-05T08:30:00Z", "updated_at": "2026-03-14T11:00:00Z", - "body": "## Overview\n\nReplace keyword-based search with semantic search using neural embeddings.\n\n## Why\n\n- Understands user intent, not just keywords\n- Finds conceptually similar content\n- Handles multi-language queries\n- Better relevance than traditional search\n\n## Setup\n\n- Embedding model (OpenAI, Sentence-BERT)\n- Vector database (Pinecone, Weaviate)\n- Batch embedding generation pipeline\n- Real-time embedding updates\n\n## AC\n\n**Semantic Search:**\n- [ ] Understands synonyms and related concepts\n- [ ] Cross-language search capability\n- [ ] Handles typos and misspellings\n- [ ] Context-aware results\n\n**Vector Operations:**\n- [ ] Generate embeddings for all content\n- [ ] Similarity search with cosine distance\n- [ ] Hybrid search (keyword + semantic)\n- [ ] Re-ranking with ML model\n\n**Performance:**\n- [ ] Search response time < 100ms\n- [ ] Support millions of vectors\n- [ ] Batch embedding updates in < 1 hour" + "body": "## Overview\n\nReplace keyword-based search with semantic search using neural embeddings.\n\n## Why\n\n- Understands user intent, not just keywords\n- Finds conceptually similar content\n- Handles multi-language queries\n- Better relevance than traditional search\n\n## Setup\n\n- Embedding model (OpenAI, Sentence-BERT)\n- Vector database (Pinecone, Weaviate)\n- Batch embedding generation pipeline\n- Real-time embedding updates\n\n## AC\n\n**Semantic Search:**\n- [ ] Understands synonyms and related concepts\n- [ ] Cross-language search capability\n- [ ] Handles typos and misspellings\n- [ ] Context-aware results\n\n**Vector Operations:**\n- [ ] Generate embeddings for all content\n- [ ] Similarity search with cosine distance\n- [ ] Hybrid search (keyword + semantic)\n- [ ] Re-ranking with ML model\n\n**Performance:**\n- [ ] Search response time < 100ms\n- [ ] Support millions of vectors\n- [ ] Batch embedding updates in < 1 hour", + "issue_number": 305 }, - { - "number": 306, + "AbsaOSS/future-project/306": { "title": "Autonomous system monitoring", "state": "open", - "labels": ["monitoring", "ai"], + "labels": [ + "monitoring", + "ai" + ], "html_url": "https://github.com/AbsaOSS/future-project/issues/306", "created_at": "2026-03-06T09:45:00Z", "updated_at": "2026-03-15T10:15:00Z", - "body": "## Description\n\nAI-powered monitoring system that detects anomalies and auto-remediates issues.\n\n## Value\n\nReduces downtime and mean-time-to-recovery. Prevents issues before they impact users.\n\n## Done Criteria\n\n**Anomaly Detection:**\n- [ ] ML model trained on historical metrics\n- [ ] Real-time anomaly detection\n- [ ] Predictive failure warnings\n- [ ] False positive rate < 5%\n\n**Auto-Remediation:**\n- [ ] Restart unhealthy services\n- [ ] Scale resources automatically\n- [ ] Clear cache when needed\n- [ ] Trigger failover on critical failures\n\n**Alerting:**\n- [ ] Smart alert grouping\n- [ ] Context-rich notifications\n- [ ] Suggested remediation actions\n- [ ] Escalation policies" + "body": "## Description\n\nAI-powered monitoring system that detects anomalies and auto-remediates issues.\n\n## Value\n\nReduces downtime and mean-time-to-recovery. Prevents issues before they impact users.\n\n## Done Criteria\n\n**Anomaly Detection:**\n- [ ] ML model trained on historical metrics\n- [ ] Real-time anomaly detection\n- [ ] Predictive failure warnings\n- [ ] False positive rate < 5%\n\n**Auto-Remediation:**\n- [ ] Restart unhealthy services\n- [ ] Scale resources automatically\n- [ ] Clear cache when needed\n- [ ] Trigger failover on critical failures\n\n**Alerting:**\n- [ ] Smart alert grouping\n- [ ] Context-rich notifications\n- [ ] Suggested remediation actions\n- [ ] Escalation policies", + "issue_number": 306 }, - { - "number": 307, + "AbsaOSS/future-project/307": { "title": "Metaverse integration", "state": "open", - "labels": ["vr", "experimental"], + "labels": [ + "vr", + "experimental" + ], "html_url": "https://github.com/AbsaOSS/future-project/issues/307", "created_at": "2026-03-07T10:00:00Z", "updated_at": "2026-03-16T14:30:00Z", - "body": "## Summary\n\nCreate VR interface for immersive data visualization and collaboration.\n\n## Business Value\n\nEarly adopter advantage in metaverse space. Novel user experience attracts media attention.\n\n## Prerequisites\n\n- WebXR API support\n- 3D rendering engine (Three.js, Babylon.js)\n- VR headset testing (Quest, Vive)\n- Avatar system\n\n## Done Criteria\n\n**VR Features:**\n- [ ] 3D data visualization\n- [ ] Virtual collaboration spaces\n- [ ] Avatar representation\n- [ ] Hand gesture controls\n- [ ] Spatial audio\n\n**Compatibility:**\n- [ ] Works on Meta Quest\n- [ ] Works on HTC Vive\n- [ ] Fallback to desktop view\n\n## How To\n\n1. Put on VR headset\n2. Launch application\n3. Enter virtual workspace\n4. Use hand gestures to interact\n5. Voice commands for navigation" + "body": "## Summary\n\nCreate VR interface for immersive data visualization and collaboration.\n\n## Business Value\n\nEarly adopter advantage in metaverse space. Novel user experience attracts media attention.\n\n## Prerequisites\n\n- WebXR API support\n- 3D rendering engine (Three.js, Babylon.js)\n- VR headset testing (Quest, Vive)\n- Avatar system\n\n## Done Criteria\n\n**VR Features:**\n- [ ] 3D data visualization\n- [ ] Virtual collaboration spaces\n- [ ] Avatar representation\n- [ ] Hand gesture controls\n- [ ] Spatial audio\n\n**Compatibility:**\n- [ ] Works on Meta Quest\n- [ ] Works on HTC Vive\n- [ ] Fallback to desktop view\n\n## How To\n\n1. Put on VR headset\n2. Launch application\n3. Enter virtual workspace\n4. Use hand gestures to interact\n5. Voice commands for navigation", + "issue_number": 307 }, - { - "number": 308, + "AbsaOSS/future-project/308": { "title": "Green computing optimization", "state": "closed", - "labels": ["sustainability", "performance"], + "labels": [ + "sustainability", + "performance" + ], "html_url": "https://github.com/AbsaOSS/future-project/issues/308", "created_at": "2026-03-08T11:15:00Z", "updated_at": "2026-03-17T09:00:00Z", - "body": "## Overview\n\nOptimize application for energy efficiency and carbon footprint reduction.\n\n## Why\n\nCorporate sustainability goals. Reduces operating costs. Positive environmental impact.\n\n## Setup\n\n- Carbon-aware computing library\n- Green energy datacenter migration\n- Energy monitoring tools\n\n## AC\n\n**Optimizations:**\n- [ ] Shift batch jobs to off-peak hours\n- [ ] Use carbon-aware scheduling\n- [ ] Optimize database queries for energy\n- [ ] Compress data transfers\n- [ ] Use renewable energy regions\n\n**Metrics:**\n- [ ] 40% reduction in energy usage\n- [ ] Carbon footprint dashboard\n- [ ] Energy efficiency score\n\n## History\n\nImplemented by sustainability-team@example.com on 2026-03-17" + "body": "## Overview\n\nOptimize application for energy efficiency and carbon footprint reduction.\n\n## Why\n\nCorporate sustainability goals. Reduces operating costs. Positive environmental impact.\n\n## Setup\n\n- Carbon-aware computing library\n- Green energy datacenter migration\n- Energy monitoring tools\n\n## AC\n\n**Optimizations:**\n- [ ] Shift batch jobs to off-peak hours\n- [ ] Use carbon-aware scheduling\n- [ ] Optimize database queries for energy\n- [ ] Compress data transfers\n- [ ] Use renewable energy regions\n\n**Metrics:**\n- [ ] 40% reduction in energy usage\n- [ ] Carbon footprint dashboard\n- [ ] Energy efficiency score\n\n## History\n\nImplemented by sustainability-team@example.com on 2026-03-17", + "issue_number": 308 }, - { - "number": 309, + "AbsaOSS/future-project/309": { "title": "Federated learning implementation", "state": "open", - "labels": ["ai", "privacy"], + "labels": [ + "ai", + "privacy" + ], "html_url": "https://github.com/AbsaOSS/future-project/issues/309", "created_at": "2026-03-09T08:00:00Z", "updated_at": "2026-03-18T10:30:00Z", - "body": "## Description\n\nImplement federated learning for privacy-preserving machine learning.\n\n## Value\n\nTrain ML models without centralizing sensitive data. Meets strict privacy regulations.\n\n## Acceptance Criteria\n\n**Federated Training:**\n- [ ] Client-side model training\n- [ ] Encrypted gradient aggregation\n- [ ] Differential privacy guarantees\n- [ ] Byzantine-robust aggregation\n\n**Infrastructure:**\n- [ ] Coordination server\n- [ ] Client SDK for edge devices\n- [ ] Secure aggregation protocol\n- [ ] Model versioning" + "body": "## Description\n\nImplement federated learning for privacy-preserving machine learning.\n\n## Value\n\nTrain ML models without centralizing sensitive data. Meets strict privacy regulations.\n\n## Acceptance Criteria\n\n**Federated Training:**\n- [ ] Client-side model training\n- [ ] Encrypted gradient aggregation\n- [ ] Differential privacy guarantees\n- [ ] Byzantine-robust aggregation\n\n**Infrastructure:**\n- [ ] Coordination server\n- [ ] Client SDK for edge devices\n- [ ] Secure aggregation protocol\n- [ ] Model versioning", + "issue_number": 309 }, - { - "number": 310, + "AbsaOSS/future-project/310": { "title": "Zero-knowledge proof authentication", "state": "open", - "labels": ["security", "cryptography"], + "labels": [ + "security", + "cryptography" + ], "html_url": "https://github.com/AbsaOSS/future-project/issues/310", "created_at": "2026-03-10T09:30:00Z", "updated_at": "2026-03-18T11:45:00Z", - "body": "## Overview\n\nImplement zero-knowledge proof authentication for privacy-preserving identity verification.\n\n## Value\n\nProves identity without revealing credentials. Ultimate privacy protection.\n\n## Prerequisites\n\n- ZK-SNARK library\n- Trusted setup ceremony\n- Browser extension or SDK\n\n## Done Criteria\n\n**Authentication:**\n- [ ] Generate ZK proof of password knowledge\n- [ ] Verify proof without seeing password\n- [ ] No credentials stored on server\n- [ ] Quantum-resistant variant\n\n**User Experience:**\n- [ ] Seamless login flow\n- [ ] Hardware wallet support\n- [ ] Biometric integration\n\n## How To\n\n1. Install identity wallet\n2. Generate proof locally\n3. Submit proof to server\n4. Server verifies without learning secret\n5. Access granted" + "body": "## Overview\n\nImplement zero-knowledge proof authentication for privacy-preserving identity verification.\n\n## Value\n\nProves identity without revealing credentials. Ultimate privacy protection.\n\n## Prerequisites\n\n- ZK-SNARK library\n- Trusted setup ceremony\n- Browser extension or SDK\n\n## Done Criteria\n\n**Authentication:**\n- [ ] Generate ZK proof of password knowledge\n- [ ] Verify proof without seeing password\n- [ ] No credentials stored on server\n- [ ] Quantum-resistant variant\n\n**User Experience:**\n- [ ] Seamless login flow\n- [ ] Hardware wallet support\n- [ ] Biometric integration\n\n## How To\n\n1. Install identity wallet\n2. Generate proof locally\n3. Submit proof to server\n4. Server verifies without learning secret\n5. Access granted", + "issue_number": 310 }, - { - "number": 311, + "AbsaOSS/future-project/311": { "title": "Holographic display support", "state": "open", - "labels": ["ui", "experimental"], + "labels": [ + "ui", + "experimental" + ], "html_url": "https://github.com/AbsaOSS/future-project/issues/311", "created_at": "2026-03-11T10:00:00Z", "updated_at": "2026-03-18T13:00:00Z", - "body": "## Description\n\nAdd support for holographic displays and 3D interfaces.\n\n## Business Value\n\nFuture-proof interface. Competitive advantage with emerging display technology.\n\n## Setup\n\n- Looking Glass display hardware\n- Holographic rendering library\n- Depth map generation\n\n## Acceptance Criteria\n\n**3D Interface:**\n- [ ] Multi-layer depth rendering\n- [ ] Perspective-correct views\n- [ ] Interactive 3D objects\n- [ ] Gesture-based navigation\n\n**Content:**\n- [ ] 3D charts and graphs\n- [ ] Spatial data visualization\n- [ ] 3D model viewing" + "body": "## Description\n\nAdd support for holographic displays and 3D interfaces.\n\n## Business Value\n\nFuture-proof interface. Competitive advantage with emerging display technology.\n\n## Setup\n\n- Looking Glass display hardware\n- Holographic rendering library\n- Depth map generation\n\n## Acceptance Criteria\n\n**3D Interface:**\n- [ ] Multi-layer depth rendering\n- [ ] Perspective-correct views\n- [ ] Interactive 3D objects\n- [ ] Gesture-based navigation\n\n**Content:**\n- [ ] 3D charts and graphs\n- [ ] Spatial data visualization\n- [ ] 3D model viewing", + "issue_number": 311 }, - { - "number": 312, + "AbsaOSS/future-project/312": { "title": "Satellite communication backup", "state": "closed", - "labels": ["infrastructure", "resilience"], + "labels": [ + "infrastructure", + "resilience" + ], "html_url": "https://github.com/AbsaOSS/future-project/issues/312", "created_at": "2026-03-12T11:30:00Z", "updated_at": "2026-03-17T15:00:00Z", - "body": "## Summary\n\nImplement satellite communication as backup for internet outages.\n\n## Why\n\nEnsures service availability during natural disasters or infrastructure failures.\n\n## Prerequisites\n\n- Starlink or similar satellite service\n- Ground station equipment\n- Automatic failover system\n\n## Acceptance Criteria\n\n**Backup Communication:**\n- [ ] Automatic detection of primary internet failure\n- [ ] Seamless failover to satellite\n- [ ] Critical services remain available\n- [ ] Fallback within 60 seconds\n\n**Monitoring:**\n- [ ] Connection health dashboard\n- [ ] Failover alerts\n- [ ] Bandwidth usage tracking\n\n## Changes\n\nDeployed by infrastructure@example.com on 2026-03-17" + "body": "## Summary\n\nImplement satellite communication as backup for internet outages.\n\n## Why\n\nEnsures service availability during natural disasters or infrastructure failures.\n\n## Prerequisites\n\n- Starlink or similar satellite service\n- Ground station equipment\n- Automatic failover system\n\n## Acceptance Criteria\n\n**Backup Communication:**\n- [ ] Automatic detection of primary internet failure\n- [ ] Seamless failover to satellite\n- [ ] Critical services remain available\n- [ ] Fallback within 60 seconds\n\n**Monitoring:**\n- [ ] Connection health dashboard\n- [ ] Failover alerts\n- [ ] Bandwidth usage tracking\n\n## Changes\n\nDeployed by infrastructure@example.com on 2026-03-17", + "issue_number": 312 }, - { - "number": 313, + "AbsaOSS/future-project/313": { "title": "Test plain text body", "state": "open", - "labels": ["test"], + "labels": [ + "test" + ], "html_url": "https://github.com/AbsaOSS/future-project/issues/313", "created_at": "2026-03-13T12:00:00Z", "updated_at": "2026-03-13T12:00:00Z", - "body": "Just plain text without any headings or markdown formatting." + "body": "Just plain text without any headings or markdown formatting.", + "issue_number": 313 } - ] -} + } +} \ No newline at end of file