Feature/integration test and updates

[miroslavpojer]

Overview

This PR consolidates and refactors the living-doc-toolkit codebase to improve maintainability, align test fixtures across versions, and streamline documentation. The changes include removal of role-specific agent customizations in favor of a unified codebase, restructuring of test fixtures, and significant documentation improvements.

Changes

Removed

• Agent definitions (.github/agents/): Removed 5 role-specific agent files (DevOps Engineer, SDET, Reviewer, Senior Developer, Specification Master) to consolidate Copilot customization into copilot-instructions.md
• Specification documents: Removed SPEC.md and TASKS.md in favor of living documentation and issue-driven task management

Added

• Root-level fixture: doc-issues.json as a golden reference file for integration testing
• Documentation: New docs/contracts.md section defining contracts and interfaces for all services

Updated

• Test fixtures: Aligned test data across all collector_gh versions (v0.9.0, v1.0.0, v1.2.0, v2.0.0) with parser changes
• Integration tests: Expanded CLI and service-level test coverage with additional test cases
• Documentation: Enhanced README.md, architecture.md, and operational guides (cookbooks, recipes, troubleshooting)
• Workflows: Minor configuration updates to dependabot and release workflows
• Dependencies: Added new entries to requirements.txt for improved tooling support

Code Quality

• Refactored parser logic for improved readability
• Expanded test coverage in test_parser.py and test_service.py
• Maintained backward compatibility with existing APIs

Impact

Positive:

• Cleaner, more maintainable code structure
• Improved documentation coverage with explicit contracts
• More comprehensive test coverage across service versions
• Simplified Copilot customization model

Considerations:

• Teams using role-specific agents must transition to generic Copilot or unified instructions
• SPEC.md/TASKS.md content should be migrated to living documentation if needed for future reference
• New dependencies in requirements.txt require compatibility verification

Testing

All changes have been validated:

• ✅ Unit tests pass
• ✅ Integration tests pass with expanded coverage
• ✅ CLI invocation tests updated for new fixture format
• ✅ Parser and service tests expanded
• ✅ Code formatting and linting verified

Merge Strategy

This PR should be merged to master and can be deployed as part of the next release cycle. No breaking changes to public APIs; agent definitions were internal tooling only.

---

Release Notes:

• Code Refactoring*: Improved code structure for better readability and maintainability across all service packages
• Documentation Enhancements: Added comprehensive contracts and interfaces documentation; expanded setup and troubleshooting guides
• Test Coverage: Aligned and expanded test fixtures across all collector_gh versions (v0.9.0–v2.0.0); added integration tests for CLI invocation and service-level operations
• Cleanup: Removed role-specific agent definitions; consolidated Copilot customization into unified instructions file
• Specification Migration: Removed SPEC.md and TASKS.md in favor of issue-driven development and living documentation
• Dependency Updates: Added tooling dependencies for improved development and testing workflows
• Workflow Updates: Minor configuration refinements to dependabot and release draft workflows
• API Stability: All public service APIs remain stable; no breaking changes to downstream consumers

Summary by CodeRabbit

• New Features

  • Added Make-based development and QA automation commands for local development workflow.
  • Created comprehensive Contracts & Interfaces documentation replacing legacy specifications.

• Breaking Changes

  • Increased minimum Python version requirement to 3.14 across all packages.
  • Updated issue data structure in input JSON format (from array to dictionary-keyed format).

• Documentation

  • Improved installation instructions and developer setup guidance.
  • Reorganized and enhanced cookbooks, recipes, and troubleshooting guides with clearer navigation.

• Chores

  • Simplified CI/CD workflows and dependency installation approaches.
  • Removed legacy specification and roadmap documentation files.

[coderabbitai]

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 80131244-667a-450c-95a7-6a38ff11f5fc

📥 Commits

Reviewing files that changed from the base of the PR and between e6ffe16 and bfc60e0.

📒 Files selected for processing (35)

• .github/agents/devops-engineer.agent.md
• .github/agents/reviewer.agent.md
• .github/agents/sdet.agent.md
• .github/agents/senior-developer.agent.md
• .github/agents/specification-master.agent.md
• .github/workflows/integration.yml
• .github/workflows/test.yml
• DEVELOPER.md
• Makefile
• README.md
• SPEC.md
• TASKS.md
• apps/cli/pyproject.toml
• apps/cli/src/living_doc_cli/commands/normalize_issues.py
• apps/cli/tests/integration/test_cli_invocation.py
• docs/architecture.md
• docs/contracts.md
• docs/cookbooks/normalize-issues.md
• docs/recipes/github-actions-normalize-issues.md
• docs/recipes/local-normalize-issues.md
• docs/troubleshooting.md
• packages/adapters/collector_gh/pyproject.toml
• packages/adapters/collector_gh/src/living_doc_adapter_collector_gh/parser.py
• packages/adapters/collector_gh/tests/fixtures/collector_v1.0.0/input/doc-issues.json
• packages/adapters/collector_gh/tests/fixtures/collector_v1.2.0/input/doc-issues.json
• packages/adapters/collector_gh/tests/test_parser.py
• packages/core/pyproject.toml
• packages/datasets_pdf/pyproject.toml
• packages/services/normalize_issues/pyproject.toml
• packages/services/normalize_issues/tests/test_service.py
• requirements.txt
• tests/fixtures/collector_gh/v0.9.0/input/doc-issues.json
• tests/fixtures/collector_gh/v1.0.0/input/doc-issues.json
• tests/fixtures/collector_gh/v1.2.0/input/doc-issues.json
• tests/fixtures/collector_gh/v2.0.0/input/doc-issues.json

---

Walkthrough

The PR changes the doc-issues.json issues field schema from an array (number) to a keyed dict (org/repo/n → issue_number), updating the parser, all fixture files, and all related tests. It also introduces a top-level Makefile for QA automation, consolidates CI into a single make py-qa job, adds requirements.txt, bumps all packages to Python 3.14, and replaces SPEC.md/TASKS.md with a new docs/contracts.md reference document while updating all cross-links across the documentation.

Changes

Issues Dict Schema Change

Layer / File(s)	Summary
Parser dict/list normalization packages/adapters/collector_gh/src/living_doc_adapter_collector_gh/parser.py	parse normalizes payload["issues"] to accept both dict and list forms, derives issue_number from either number or issue_number, and updates AdapterError messages with the derived number.
Fixture JSON schema migration packages/adapters/collector_gh/tests/fixtures/collector_v1.0.0/..., packages/adapters/collector_gh/tests/fixtures/collector_v1.2.0/..., tests/fixtures/collector_gh/v0.9.0/..., tests/fixtures/collector_gh/v1.0.0/..., tests/fixtures/collector_gh/v1.2.0/..., tests/fixtures/collector_gh/v2.0.0/...	All doc-issues.json fixtures convert issues from an array of {number, ...} to a dict keyed by org/repo/n with issue_number per entry, across all collector versions.
Parser, service, and CLI tests packages/adapters/collector_gh/tests/test_parser.py, packages/services/normalize_issues/tests/test_service.py, apps/cli/tests/integration/test_cli_invocation.py, apps/cli/src/living_doc_cli/commands/normalize_issues.py	All test fixtures and payloads updated to the keyed-dict issues shape; empty-issues cases use {}; field-deletion and error-path tests use keyed access.

Make-Based QA Automation

Layer / File(s)	Summary
Makefile targets and pattern rules Makefile	New Makefile defines PACKAGES, APPS, ALL_TARGETS, tool thresholds, install, help, aggregated py-qa/black/pylint/mypy/pytest-unit targets, per-package meta-targets, and per-tool pattern rules.
CI consolidation and requirements.txt .github/workflows/test.yml, .github/workflows/integration.yml, requirements.txt	test.yml replaces four separate jobs with a single qa job running make py-qa; integration.yml adds [dev] extras; requirements.txt is created with editable installs for all workspace packages.
Python 3.14 version bumps and developer docs packages/*/pyproject.toml, apps/cli/pyproject.toml, DEVELOPER.md	All pyproject.toml files raise requires-python to >=3.14; DEVELOPER.md gains a "QA Automation with Make" section documenting targets and quality gate thresholds.

Docs Restructuring: contracts.md Replaces SPEC.md

Layer / File(s)	Summary
New contracts.md reference docs/contracts.md	Creates the canonical reference for the CLI interface, doc-issues.json input contract, pdf_ready.json output contract, meta.audit envelope, JSON schema pointers, change-control policy, and performance budgets.
Cross-reference updates docs/architecture.md, docs/troubleshooting.md, docs/cookbooks/normalize-issues.md, docs/recipes/github-actions-normalize-issues.md, docs/recipes/local-normalize-issues.md, README.md	All SPEC.md/TASKS.md links replaced with contracts.md section anchors; section headings lose SPEC parentheticals; installation instructions updated to requirements.txt; TOCs added to cookbook and recipes.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Poem

🐇 Hop hop, the issues dict has changed its shape,
From lists to maps, no data shall escape!
A Makefile sprouts to govern every check,
While SPEC.md bows out — contracts take the deck.
The contracts.md now holds the contract true,
All links aligned, the docs are shiny new! ✨

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feature/integration-test-and-updates

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}