Add comprehensive CI/CD workflow with Docker services and contract validation

[Copilot]

Overview

This PR implements a complete CI/CD pipeline for the Sherlock Multiagent Data Scientist project, adding automated testing, linting, and validation across both frontend (React/Vite) and backend (Python) components.

What's New

CI/CD Workflow (.github/workflows/ci.yml)

A comprehensive GitHub Actions workflow that includes:

• Frontend Pipeline: Builds the React dashboard with Vite, caches npm dependencies, and uploads build artifacts
• Backend Pipeline: Tests Python code on versions 3.12 and 3.13 using a matrix strategy, runs linting (Ruff, Black) and type checking (MyPy)
• E2E Testing: Spins up a full service stack (MongoDB, Kafka, Zookeeper, Redis) via Docker Compose and runs end-to-end tests
• Contract Validation: Validates API schemas, configuration files, and Python contract tests
• Playwright Support: Infrastructure ready for browser-based E2E tests with 4-way test sharding (disabled until Playwright is added to the project)

Service Orchestration (docker-compose.ci.yml)

Defines the complete service stack for CI testing:

services:
  - Zookeeper (Port 2181)
  - Kafka (Port 9092) 
  - MongoDB (Port 27017)
  - Redis (Port 6379)

All services include health checks with automatic retries to ensure stability before tests run.

Contract Validation (scripts/validate-contracts.js)

A Node.js script that validates system contracts:

• Configuration file validation
• API schema verification
• Python contract test execution
• Secure command execution using spawnSync to prevent path injection vulnerabilities

Comprehensive Documentation

/reports/ci-notes.md (12KB): Complete pipeline documentation including:

• Pipeline architecture and job flow diagrams
• Caching strategies for dependencies
• Branch protection setup guide with required status checks
• Troubleshooting guide for common CI issues
• Future enhancements roadmap

/reports/CI_QUICKSTART.md (4KB): Developer quick reference with:

• Local testing instructions
• Service startup commands
• Common troubleshooting solutions
• Playwright integration guide

Repository Improvements

• CI Badge: Added to README.md showing workflow status at a glance
• .gitignore: Created to exclude build artifacts (node_modules, dist, test results, etc.)

Architecture

The pipeline runs jobs in parallel for efficiency:

┌────────────┐    ┌────────────┐
│  Frontend  │    │  Backend   │
│  Build     │    │  (3.12+13) │
└──────┬─────┘    └──────┬─────┘
       │                 │
       └────────┬────────┘
                ▼
       ┌────────────────┐
       │   E2E Tests    │
       │   + Services   │
       └────────┬───────┘
                │
       ┌────────┴────────┐
       ▼                 ▼
┌─────────────┐   ┌─────────────┐
│ Contract    │   │ Playwright  │
│ Validation  │   │ (Prepared)  │
└──────┬──────┘   └─────────────┘
       │
       ▼
┌─────────────┐
│ CI Success  │
│ Gate Check  │
└─────────────┘

Key Features

• Parallel Execution: Frontend and backend jobs run simultaneously for faster feedback
• Multi-Version Testing: Backend tests run on Python 3.12 and 3.13 to ensure compatibility
• Smart Caching: Dependencies are cached based on lockfile hashes (npm, pip, Playwright browsers)
• Service Health Checks: All Docker services verify they're ready before tests run
• Artifact Retention: Build artifacts retained for 7 days, contract validation results for 30 days
• Security: Secure command execution, no hardcoded secrets, proper error handling

Branch Protection

To enable branch protection, configure these required status checks in GitHub Settings:

1. frontend
2. backend (3.12)
3. backend (3.13)
4. e2e-tests
5. contract-validation
6. ci-success

Detailed setup instructions are in /reports/ci-notes.md.

Testing

All linting and type checking steps use || true to make them non-blocking, allowing incremental adoption without breaking CI. Tests run with appropriate timeouts and error handling.

Future Enhancements

The workflow is prepared for future additions:

• Playwright E2E Tests: Infrastructure ready with 4-way sharding, just needs if: true when Playwright is added
• Code Coverage: Easy to add coverage reporting with Codecov
• Security Scanning: Can integrate Snyk or Dependabot
• Deployment Pipeline: Structure ready for staging/production deployment jobs

Files Changed

• .github/workflows/ci.yml (new) - Main CI workflow
• docker-compose.ci.yml (new) - Service orchestration
• scripts/validate-contracts.js (new) - Contract validation
• reports/ci-notes.md (new) - Comprehensive documentation
• reports/CI_QUICKSTART.md (new) - Quick reference
• .gitignore (new) - Artifact exclusion
• README.md (modified) - CI badge added

---

Total: ~30KB of new configuration and documentation for production-ready CI/CD 🚀

Original prompt

You are A3 CI Scaffolder.

Create .github/workflows/ci.yml that:

• Restores caches (node_modules, Playwright browsers).
• Runs: install, build, lint, typecheck.
• Spins docker-compose.ci.yml services.
• Shards E2E tests via Playwright workers; uploads traces/screenshots on fail.
• Runs scripts/validate-contracts.(ts|js).
• Sets required checks and branch protection docs in /reports/ci-notes.md.

Constraints:

• Use matrix for Node LTS if repo is multi-package.
• Emit badges in README if section exists; otherwise note in /reports.

---

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.