Add A4 Env & Seed Harness: Complete Testing Infrastructure with Docker Compose, Testcontainers, and Idempotent Seeder

[Copilot]

Overview

This PR delivers a comprehensive testing environment and seed harness for the Sherlock project, providing all infrastructure needed to run tests locally and in CI with minimal setup.

What's Included

🐳 Docker Compose CI Environment (docker-compose.ci.yml)

Provides containerized test dependencies with health checks and optimized configuration:

• PostgreSQL 15-alpine (port 5432) - Test database
• MongoDB 6.0 (port 27017) - Document store
• Redis 7-alpine (port 6379) - Cache and queue
• Kafka 7.4.0 (port 9092) - Message broker
• Zookeeper 7.4.0 (port 2181) - Kafka coordination

All services use tmpfs for fast I/O and include health checks to ensure readiness before tests run.

🌱 Idempotent Test Seeder (scripts/seed-test-env.ts)

TypeScript-based seeder that populates test data safely and repeatably:

// Creates admin user, test user, sample dataset, and workflow
npm run seed:test

Key Features:

• Uses MongoDB upserts for idempotency - can run multiple times without errors
• Seeds admin user (admin/admin123) and test user (testuser/test123)
• Creates sample dataset with 5 rows (id, name, value, category)
• Creates sample workflow with load + analyze steps
• Fully tested and validated

🧪 Testcontainers Configuration (tests/testcontainers.config.ts)

Provides isolated container environments for integration tests:

import { startTestContainers, stopTestContainers, getConnectionUrls } from '../testcontainers.config';

// Tests get fresh, isolated containers
const containers = await startTestContainers();
const urls = getConnectionUrls(containers);
// Use urls.mongodb, urls.redis, urls.postgres

📚 Comprehensive Documentation (docs/testing-env.md)

Complete 6.8KB guide covering:

• Quick start instructions
• Local development setup
• CI/CD integration with example GitHub Actions workflow
• Troubleshooting common issues
• Best practices for test isolation and idempotency

⚙️ npm Scripts

Added convenient commands for test management:

npm run test:setup       # Start services + seed data + install Playwright
npm test                 # Run all tests (integration + E2E)
npm run test:e2e         # Playwright E2E tests
npm run test:integration # Vitest integration tests

Testing & Validation

All components have been thoroughly tested:

• ✅ TypeScript compilation succeeds with no errors
• ✅ All Docker services start and pass health checks
• ✅ Seed script successfully populates data and is idempotent (tested by running twice)
• ✅ Sample integration tests pass (2/2 tests in 7.02s using testcontainers)
• ✅ Testcontainers start and stop cleanly with proper isolation
• ✅ Code review completed with feedback addressed

Design Principles

• Non-invasive: Zero changes to existing production code
• Idempotent: Safe to run setup scripts multiple times
• Isolated: Testcontainers provide clean test environments
• CI-Ready: Health checks, tmpfs, proper timeouts for fast CI execution
• Well-Documented: READMEs in scripts/ and tests/ directories plus comprehensive main guide
• Production-Grade: Following testing best practices

Usage Example

# One-time setup
npm install
npm run test:setup

# Run tests
npm test

# Cleanup
docker compose -f docker-compose.ci.yml down -v

Files Added

• docker-compose.ci.yml - Test services configuration
• tsconfig.json - TypeScript configuration
• vitest.config.ts - Integration test configuration
• playwright.config.ts - E2E test configuration
• scripts/seed-test-env.ts - Idempotent test data seeder
• scripts/README.md - Scripts documentation
• tests/testcontainers.config.ts - Testcontainers utility
• tests/integration/sample.test.ts - Sample integration test
• tests/e2e/sample.spec.ts - Sample E2E test
• tests/README.md - Test structure guide
• docs/testing-env.md - Complete testing documentation
• .gitignore - Test artifacts exclusion

Dependencies Added

• @playwright/test - E2E browser testing
• vitest + @vitest/coverage-v8 - Fast integration testing
• testcontainers - Isolated test containers
• mongodb - MongoDB client for seeding
• tsx - TypeScript execution
• typescript + @types/node - TypeScript support

This infrastructure is ready for immediate use and provides a solid foundation for expanding test coverage. 🚀

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

• 4.0.18.172.in-addr.arpa
  • Triggering command: java -cp /usr/share/java/cp-base-new/* -Dlog4j.configuration=file:/etc/kafka/log4j.properties io.confluent.admin.utils.cli.ZookeeperReadyCommand zookeeper-test:2181 40000 (dns block)
• compass.mongodb.com
  • Triggering command: mongosh mongodb://127.0.0.1:27017/admin?directConne (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

Original prompt

You are A4 Env & Seed Harness.

Deliver:

• docker-compose.ci.yml for DB/cache/services.
• Testcontainers config for integration tests.
• scripts/seed-test-env.ts idempotent seeder (admin + test user + sample object).
• docs at /docs/testing-env.md: how to run tests locally and in CI.

Add npm scripts:

• test:setup -> compose up + seed + playwright install
• test:e2e -> playwright test --reporter=line
• test:integration -> vitest/jest

No invasive refactors; keep changes minimal and documented.

---

💬 Share your feedback on Copilot coding agent for the chance to win a $200 gift card! Click here to start the survey.

The best way to fix this issue is to replace the usage of the generic SHA-256 hash with a secure password hashing algorithm, such as bcrypt. Bcrypt is purpose-built for password hashing, featuring built-in salting and slow, computationally expensive hashing.

To implement the fix:

• Add the required import for the bcrypt library at the top of the file.
• Replace the hashPassword function implementation to use bcrypt.hashSync(password, saltRounds), where a reasonably high saltRounds value is set (such as 10).
• No changes are needed to the rest of the code as the function interface remains the same.
• The rest of the seed script will now generate bcrypt password hashes, which are much more secure.

---