Feature/testing pipelines

.github/workflows/PIPELINE-DESIGN.md

@@ -0,0 +1,230 @@
+# Pipeline Design Philosophy
+
+## The Problem with Inline Tests in CI/CD
+
+### ❌ What Was Wrong
+
+The original pipeline had several anti-patterns:
+
+#### 1. **Inline Test Code in YAML**
+
+```yaml
+# BAD: Embedding test logic in pipeline
+run: |
+  cat > test-claude-detection.js << 'EOF'
+  const { exec } = require('child_process');
+  // ... complex test logic here
+  EOF
+  node test-claude-detection.js
+```
+
+**Problems**:
+
+- Test logic is not version controlled properly
+- No IDE support for the embedded code
+- Hard to debug and maintain
+- Cannot be run locally for development
+- No proper error handling or logging
+- Duplicates test logic across pipeline steps
+
+#### 2. **Redundant Testing**
+
+```yaml
+# BAD: Testing the same thing multiple times
+- name: "Test A"
+- name: "Test B that does the same as A"
+- name: "Test C that also does the same"
+```
+
+**Problems**:
+
+- Wastes CI/CD time and resources
+- Creates confusion about what's actually being tested
+- Makes failures harder to diagnose
+
+#### 3. **Poor Separation of Concerns**
+
+```yaml
+# BAD: Mixing infrastructure and test logic
+run: |
+  # Setup stuff
+  export DISPLAY=:99
+  # Test stuff embedded here
+  # More setup
+  # More test stuff
+```
+
+**Problems**:
+
+- Infrastructure concerns mixed with test logic
+- Hard to understand what each step does
+- Difficult to reuse or modify
+
+### ✅ The Correct Approach
+
+#### 1. **Tests in Codebase, Pipeline Runs Tests**
+
+```yaml
+# GOOD: Pipeline just orchestrates, tests are in codebase
+- name: Run Without Claude CLI tests
+  run: npm run test:ci:without-claude-cli
+```
+
+**Benefits**:
+
+- All test logic is in the codebase
+- Can be run locally for debugging
+- Proper version control and IDE support
+- Clear separation of concerns
+- Reusable across different CI systems
+
+#### 2. **Dedicated Test Scripts**
+
+```javascript
+// GOOD: Proper test file with full functionality
+// scripts/test-claude-detection.js
+class ClaudeDetectionTester {
+  async runAllTests() {
+    // Comprehensive, well-structured test logic
+  }
+}
+```
+
+**Benefits**:
+
+- Full programming language features
+- Proper error handling and logging
+- Can be unit tested itself
+- Clear documentation and comments
+
+#### 3. **Clear Pipeline Responsibilities**
+
+**Pipeline Responsibilities**:
+
+- Environment setup (Docker, dependencies)
+- Artifact management (build, upload, download)
+- Test orchestration (run test commands)
+- Result reporting (success/failure, summaries)
+
+**Test Code Responsibilities**:
+
+- Actual testing logic and assertions
+- Error handling and reporting
+- Test data management
+- Mock setup and teardown
+
+## Our Two-Stage Testing Strategy
+
+### Without Claude CLI: Detection Tests
+
+```bash
+# What it runs
+npm run test:ci:without-claude-cli
+
+# What that includes
+npm run test:unit           # Unit tests
+npm run test:main-window    # VS Code extension test
+npm run test:claude-detection  # CLI detection logic
+```
+
+**Purpose**: Verify the extension handles missing Claude CLI gracefully
+
+### With Claude CLI: Integration Tests
+
+```bash
+# What it runs
+npm run test:ci:with-claude-cli
+
+# What that includes
+npm run test:ci:without-claude-cli      # All Without Claude CLI tests
+npm run test:e2e            # End-to-end workflows
+npm run test:integration    # Integration tests
+```
+
+**Purpose**: Verify full functionality when Claude CLI is available
+
+## Why This Design is Better
+
+### 🏗️ **Maintainability**
+
+- Test logic is in proper source files
+- Can be modified with IDE support
+- Version controlled like other code
+- Can be refactored and improved
+
+### 🧪 **Testability**
+
+- Tests can be run locally during development
+- Easy to debug when they fail
+- Can add more tests without touching pipeline
+- Test the tests themselves
+
+### 🔄 **Reusability**
+
+- Same tests work on different CI systems
+- Developers can run the same tests locally
+- Docker containers can use the same test commands
+- Easy to create new test combinations
+
+### 📊 **Clarity**
+
+- Pipeline shows high-level flow
+- Test details are in appropriate files
+- Clear separation between infrastructure and logic
+- Easy to understand what each phase does
+
+### ⚡ **Performance**
+
+- No redundant testing
+- Tests can be optimized independently
+- Better caching and parallelization
+- Faster feedback loops
+
+## Test Organization
+
+```
+├── .github/workflows/          # CI/CD orchestration only
+│   ├── test-pipeline.yml       # Main 2-stage pipeline
+│   └── docker-e2e.yml         # Docker-based testing
+├── scripts/                   # Utility test scripts
+│   └── test-claude-detection.js
+├── tests/                     # Test suites
+│   ├── e2e/                   # End-to-end tests
+│   └── integration/           # Integration tests
+├── src/test/                  # VS Code extension tests
+│   └── suite/
+└── package.json               # Test command definitions
+```
+
+## Commands and Their Purpose
+
+### Local Development
+
+```bash
+npm run test:claude-detection  # Test CLI detection logic
+npm run test:main-window       # Test VS Code integration
+npm run test:unit             # Test individual functions
+```
+
+### CI Simulation
+
+```bash
+npm run test:ci:without-claude-cli         # Simulate Without Claude CLI
+npm run test:ci:with-claude-cli         # Simulate With Claude CLI
+```
+
+### Individual Categories
+
+```bash
+npm run test:e2e              # End-to-end workflows
+npm run test:integration      # Service integration
+npm run test:all:coverage     # Full coverage report
+```
+
+This design ensures that:
+
+1. **Pipeline focuses on orchestration**, not test implementation
+2. **Tests are proper code** with full language features
+3. **Local development** mirrors CI/CD exactly
+4. **Debugging is easy** when tests fail
+5. **Maintenance is simple** with standard code practices