[feature][testing-harness] Add TestHarness Phase 3: end-to-end run against real Glean

[dishant-gupta-glean]

Problem

Phase 1 and Phase 2 validate connector logic without touching Glean's API. Before shipping a connector, teams need a way to do a bounded end-to-end smoke test — real source data, real Glean indexing — with a safety dial to limit how many items get crawled so they don't flood production.

Solution

Implement `run_end_to_end` and `run_end_to_end_async` as the final phase of the `TestHarness` testing model. Both methods:

1. Apply per-client `max_items` via the same `_patched_clients` context manager used in Phase 2 (bounding the crawl without changing connector logic).
2. Call the connector's real indexing entry-point against whatever `GLEAN_SERVER_URL` / `GLEAN_INDEXING_API_TOKEN` are configured in the environment.
3. Return `None` — results are verified by inspecting Glean's search UI or API, not the mock client.

`run_end_to_end_async` dispatches to `connector.index_data_async()` for `BaseAsyncStreamingDatasourceConnector` subclasses and falls back to `connector.index_data()` for sync connectors.

TestHarness additions

def run_end_to_end(self, *, mode=IndexingMode.FULL, options=None) -> None: ...
async def run_end_to_end_async(self, *, mode=IndexingMode.FULL, options=None) -> None: ...

File layout

tests/unit_tests/testing/harness/test_harness_phase3.py  # 7 unit tests (no live Glean)

Unit tests use `patch.object` to mock `index_data` / `index_data_async` so they can run in CI without credentials. They verify:

• Correct method is called with forwarded `mode` and `options`
• Client attributes are restored after an exception
• `max_items` wrapping is applied before `index_data` is called

Validation

uv run pytest tests/unit_tests/testing/harness/ -v   # 95 tests, all pass
uv run ruff check src/glean/indexing/testing/harness/ tests/unit_tests/testing/harness/

Notes

• Stacked on [feature][testing-harness] Add permission extraction and negative identity assertions #62 (`feature/testing-harness-permissions`). Review that PR first.
• This completes the full stack:
  • PR 1 [feature][testing-harness] Add TestHarness Phase 1: TestConfig and full-mock run #60 — Phase 1: TestConfig + full-mock
  • PR 2 [feature][testing-harness] Add TestHarness Phase 2: NDJSON cache layer and integration test mode #61 — Phase 2: NDJSON cache + integration test
  • PR 3 [feature][testing-harness] Add permission extraction and negative identity assertions #62 — Permission extraction + negative assertions
  • PR 4 [feature][testing-harness] Add TestHarness Phase 3: end-to-end run against real Glean #63 (this) — Phase 3: end-to-end
• To avoid polluting production data, point `GLEAN_SERVER_URL` at a dedicated test instance when running Phase 3.
• The `run_id_prefix` from `TestConfig` is logged for traceability but the connector's own datasource name is used as-is (consistent with how `glean-deploy run` works).

[glean-it]

Issue: The async unit test test_async_connector_calls_index_data_async uses asyncio.coroutine(lambda: None)(), but asyncio.coroutine is deprecated and removed in Python 3.11+, which will cause an AttributeError and break tests on modern Python versions.

Suggested fix: Replace the use of asyncio.coroutine with a real async function or an already-resolved awaitable. For example:

async def _noop():
    return None

mock_async.return_value = _noop()

Alternatively, use unittest.mock.AsyncMock for index_data_async so that the mock is natively awaitable.

_{🔧 Tag @ glean-for-engineering to fix or click here to fix in Glean}

_{💬 Help us improve! Was this comment helpful? React with 👍 or 👎}

[glean-it]

Issue: The Phase 3 documentation for TestHarness.run_end_to_end states that it "Calls connector.index_data() without any mock patching", but the implementation wraps the call in _patched_clients(self._connector, self._clients, self._config), and tests assert that the connector's client is replaced during the run, so the docstring is misleading about patching behavior.

Suggested fix: Update the Phase 3 and run_end_to_end docstrings to clarify that while no network mocking of the Glean server is performed, connector clients are temporarily wrapped/patched (e.g., to enforce max_items). For example, rephrase to: "Calls connector.index_data() against a real Glean instance without mocking the Glean API; connector clients are temporarily wrapped to apply per-client max_items limits."

_{🔧 Tag @ glean-for-engineering to fix or click here to fix in Glean}

_{💬 Help us improve! Was this comment helpful? React with 👍 or 👎}

[Copilot]

Pull request overview

Adds Phase 3 of the TestHarness to run a connector end-to-end against a real Glean instance (using environment-configured credentials), while still applying the existing max_items safety bound via the same client-patching mechanism used in Phase 2.

Changes:

• Implement TestHarness.run_end_to_end() to patch configured clients (for max_items) and invoke connector.index_data() against real Glean.
• Implement TestHarness.run_end_to_end_async() to call index_data_async() for async-streaming connectors and fall back to index_data() for sync connectors, with the same client patching.
• Add new unit tests for Phase 3 behavior and remove Phase 3 “NotImplemented” stub assertions from Phase 1 tests.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 6 comments.

File	Description
src/glean/indexing/testing/harness/harness.py	Implements Phase 3 sync/async entrypoints and updates Phase 3 documentation.
tests/unit_tests/testing/harness/test_harness_phase3.py	Adds unit tests validating Phase 3 method dispatch and client patch/restore behavior.
tests/unit_tests/testing/harness/test_harness_phase1.py	Removes tests asserting Phase 3 methods raise NotImplementedError.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.