Restructure test suite: service directories, profiles, hybrid fixtures

.github/workflows/code-format.yml

@@ -0,0 +1,50 @@
+name: Format Code
+
+on:
+  pull_request:
+    branches: [main]
+    paths:
+      - "**/*.py"
+
+permissions:
+  contents: write
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  format:
+    runs-on: ubuntu-latest
+
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          ref: ${{ github.head_ref }}
+          fetch-depth: 1
+
+      - name: Set up Python
+        uses: actions/setup-python@v4
+        with:
+          python-version: "3.10"
+
+      - name: Install formatters
+        run: pip install black==24.3.0 isort==5.13.2
+
+      - name: Format Python files (black + isort)
+        run: |
+          isort --profile black .
+          black --line-length 160 .
+
+      - name: Check for changes
+        id: git-check
+        run: git diff --exit-code || echo "changes=true" >> $GITHUB_OUTPUT
+
+      - name: Commit formatting changes
+        if: steps.git-check.outputs.changes == 'true'
+        run: |
+          git config --global user.name 'Code Formatter'
+          git config --global user.email 'noreply@vectara.com'
+          git add .
+          git commit -m "Apply code formatting (black + isort)"
+          git push

.github/workflows/pr-validation.yml

@@ -0,0 +1,35 @@
+name: Validate Tests
+
+on:
+  push:
+    paths:
+      - "**/*.py"
+      - "requirements.txt"
+  pull_request:
+    paths:
+      - "**/*.py"
+      - "requirements.txt"
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  validate:
+    runs-on: ubuntu-latest
+
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v4
+        with:
+          python-version: "3.10"
+
+      - name: Install dependencies
+        run: pip install -r requirements.txt
+
+      - name: Collect tests and validate markers
+        env:
+          VECTARA_API_KEY: dummy-for-collection-only
+        run: pytest tests/services/ --collect-only -q

CLAUDE.md

@@ -0,0 +1,56 @@
+# Development Guidelines
+
+## Build Commands
+- Install deps: `pip install -r requirements.txt`
+- Run all tests: `python run_tests.py --profile full`
+- Run sanity tests: `python run_tests.py --profile sanity`
+- Run core tests: `python run_tests.py --profile core`
+- Run single service: `python run_tests.py --service auth`
+- Run single test: `python -m pytest tests/services/auth/test_api_key_validation.py::TestApiKeyValidation::test_health_check -v`
+- Run by keyword: `python -m pytest tests/services/ -k "test_health_check" -v`
+
+## Environment Variables
+- `VECTARA_API_KEY` — required, Personal API key
+- `VECTARA_BASE_URL` — defaults to `https://api.vectara.io`, use `https://api.vectara.dev` for staging
+
+## Project Structure
+- `tests/services/<service>/` — test files organized by API service (auth, corpus, indexing, query, chat, agents)
+- `tests/workflows/` — cross-service end-to-end flow tests
+- `utils/client.py` — Vectara API client (single class, all HTTP methods)
+- `utils/waiters.py` — polling helpers and SSE reader
+- `utils/config.py` — environment-based configuration
+- `fixtures/sample_data.py` — test data
+- `run_tests.py` — CLI runner with `--profile` and `--service` flags
+
+## Test Markers
+- Every service test must have exactly one depth marker: `@pytest.mark.sanity`, `@pytest.mark.core`, or `@pytest.mark.regression`
+- Workflow tests use `@pytest.mark.workflow`
+- Tests without markers fail collection
+- `@pytest.mark.serial` for tests that must not run in parallel
+
+## Code Style
+- Python: PEP8, type hints, snake_case for variables/functions, CamelCase for classes
+- Imports: Group by standard library, third-party, then local imports
+- Do not add trivial comments. Write self-documenting code with clear naming. Do not delete old explanatory comments though, they are good.
+- Do add docstrings for modules and classes.
+- Fully implement functionality, do not leave stubs "for later".
+- Do not modify tests to make them pass — fix the code under test.
+- Error handling: Use appropriate exceptions, avoid catching generic exceptions.
+- Before creating a new class/type, search for existing types that serve a similar purpose. Extend existing types rather than creating near-duplicates.
+- When modifying a class, modify methods directly rather than adding duplicate methods.
+- Strongly prefer explicit types over `None` sentinels.
+
+## Test Conventions
+- Each test is self-contained via fixtures. No test depends on another test having run.
+- Use `unique_id` fixture for resource names to avoid collisions.
+- Always use explicit UUID keys when creating corpora (`key=f"test_{uuid.uuid4().hex}"`).
+- Never mutate the bootstrap API key used to run the suite.
+- Use `wait_for()` from `utils/waiters.py` instead of `time.sleep()` for async operations.
+- Cleanup resources in `try/finally` blocks.
+- Module-scoped fixtures for shared corpora (read-heavy tests), function-scoped for CRUD tests.
+- **Assertions must verify actual behavior, not just HTTP status.** Always verify response data, field values, and state changes — not just `response.success`.
+
+## General Behavior
+- Treat the user as an expert.
+- Be pithy — use short summaries of actions.
+- When refactoring, spawn sub agents for manual updates rather than using sed/grep/awk.