Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 25 additions & 13 deletions .github/workflows/postrelease.yml
Original file line number Diff line number Diff line change
@@ -1,27 +1,34 @@
name: Post-release checks

# After a release is published, prove the artifact is actually installable from
# real PyPI and runs end-to-end (creds-free) — the manual step 4 of
# docs/roadmap/local-verification-checklist.md, automated.
# After a release is *published to PyPI*, prove the artifact is actually
# installable from real PyPI and runs end-to-end (creds-free) — the manual
# step 4 of docs/roadmap/local-verification-checklist.md, automated.
#
# Triggers:
# - release: published -> verify the just-released version.
# - workflow_dispatch -> verify an explicit version on demand.
# - workflow_run on "Release" (completed) -> verify the version Release just
# published. Keyed off the Release workflow actually FINISHING, not the
# GitHub `release: published` event — our publish is on-demand OIDC
# (release.yml is dispatched + approved AFTER the GitHub release exists), so
# `release: published` fires long before the artifact is on PyPI and the poll
# would 404 forever (issue #148). `workflow_run` runs only once the Release
# run (build + OIDC publish) completed successfully.
# - workflow_dispatch -> verify an explicit version on demand.
# Polls PyPI (index lag is ~1 min after upload), then installs into a clean venv
# from PyPI and runs the CLI smoke. On the publish trigger it then opens an issue
# if anything fails, so a bad release is loud.
# from PyPI and runs the CLI smoke. On the automatic path it opens an issue if
# anything fails, so a bad release is loud.

on:
release:
types: [published]
workflow_run:
workflows: ["Release"]
types: [completed]
workflow_dispatch:
inputs:
version:
description: "Published version to verify from PyPI (e.g. 0.6.1)"
required: true

concurrency:
group: postrelease-${{ github.ref }}
group: postrelease-${{ github.event.workflow_run.head_branch || inputs.version }}
cancel-in-progress: false

env:
Expand All @@ -34,6 +41,10 @@ jobs:
verify-pypi:
name: install from PyPI + smoke
runs-on: ubuntu-latest
# Only run once Release actually succeeded (build + OIDC publish). A dispatch
# that was never approved leaves Release without a success conclusion, so this
# never fires on a publish that did not happen.
if: ${{ github.event_name == 'workflow_dispatch' || github.event.workflow_run.conclusion == 'success' }}
permissions:
contents: read
# Needed by the github-script step that opens an issue on failure.
Expand All @@ -51,13 +62,14 @@ jobs:

- name: Resolve target version
id: ver
# release event -> the tag (strip a leading v); manual -> the input.
# workflow_run -> the tag Release ran on (head_branch = the vX.Y.Z ref it
# was dispatched with; strip a leading v); manual -> the input.
run: |
set -euo pipefail
if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
v="${{ inputs.version }}"
else
tag="${{ github.event.release.tag_name }}"
tag="${{ github.event.workflow_run.head_branch }}"
v="${tag#v}"
fi
case "$v" in
Expand Down Expand Up @@ -98,7 +110,7 @@ jobs:
- name: Open an issue if the release smoke failed
# Only on the automatic publish path — a manual dispatch failure is
# visible to the person who ran it.
if: ${{ failure() && github.event_name == 'release' }}
if: ${{ failure() && github.event_name == 'workflow_run' }}
uses: actions/github-script@v7
with:
script: |
Expand Down
36 changes: 36 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,42 @@ on a schema mismatch is **rebuild** (ADR-0006).

## [Unreleased]

### Added

- **feat-014 — watch mode + CI-triggered central indexing (freshness / ops).**
feat-004 made re-indexing cheap; this adds the two automatic *triggers* that
keep the graph fresh, one per topology, with a hard wall between them.
- **`ckg watch`** (opt-in, local embedded store only) re-runs the incremental
`refresh()` on a configurable trigger — `on-commit` (default; refreshes on a
commit / branch switch, ignores ordinary saves), `on-idle`, `on-save`,
`interval`, or `manual`. Debounced + single-flight so bursts coalesce; a
watch refresh is **structural-only** by default (`embed_on_watch` /
`enrich_on_watch` off) so it never burns embedding/LLM spend per save. Branch
gating (`watch.branches`) watches feature branches and skips `main` /
`release/*`. `ckg watch --status` reports trigger · store · freshness;
`ckg watch --once` runs one refresh and exits.
- **`ckg watch` refuses a central (`store.central_root`) or read-only store** —
that topology's freshness is CI's job. This local/central split is enforced
in the engine (a developer's watch loop must never race writes into the
shared, authoritative graph).
- **`ckg ci init`** scaffolds a self-contained
`.github/workflows/ckg-index.yml` (managed-marker, idempotent, `--print`,
`--force`) that indexes into the central store on merge-to-`main` + nightly,
single-writer (`concurrency` group). CI is the sole authoritative writer, so
read-only consumers (ENH-018) can trust it.
- New `[watch]` PyPI extra (`watchfiles`, lazy-loaded so the base install stays
lean) and a `watch:` config block. See
[guide 12](docs/guides/12-watch-and-ci-indexing.md).

### Fixed

- **Post-release verification no longer false-alarms (#148).** `postrelease.yml`
now runs off the **Release workflow completing successfully** (`workflow_run`),
not the GitHub `release: published` event. Under the on-demand OIDC publish
model the release is published before the artifact reaches PyPI, so the old
trigger polled PyPI too early, 404'd for ~10 min, and auto-opened a spurious
"post-release smoke failed" issue every release.

## [0.6.2] — 2026-06-27

### Added
Expand Down
25 changes: 21 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,10 +56,14 @@ ckg serve-mcp --repo . # → 10 read-only tools for your agent
- 🔎 **Hybrid retrieval** — vector search **entry** → typed **graph expansion**. Ask
in natural language, get *connected* context: the symbol, its callers, *and* its
governing decision.
- ⚡ **Incremental by default** — re-index only the diff. Edit 3 files in a 5k-file
repo → seconds, not minutes. Embeddings/enrichment recompute only what changed.
- 🤖 **Agent-native** — served read-only over **MCP (10 tools)** or as a native
AgentForge toolset. Every response carries a staleness envelope.
- ⚡ **Incremental & always-fresh** — re-index only the diff (edit 3 files in a
5k-file repo → seconds, not minutes; embeddings/enrichment recompute only what
changed). Keep it fresh automatically: **`ckg watch`** re-indexes your working
copy on a trigger you choose (commit / idle / save), and **`ckg ci init`**
scaffolds a workflow that keeps the shared central index fresh on every merge.
- 🤖 **Agent-native, wired in one command** — served read-only over **MCP (10
tools)** or as a native AgentForge toolset, every response carrying a staleness
envelope. **`ckg setup`** writes your agent's MCP config for you.
- 🧠 **LLM enrichment, budgeted & opt-in** — design-pattern tags (*"this class is a
Repository,"* with confidence + rationale) and bottom-up module summaries, all
`llm`-provenance. **CI needs no model calls or cloud creds.**
Expand Down Expand Up @@ -194,6 +198,19 @@ by default), shows a diff first, and is reversible with `--undo` — see
[guide 11](docs/guides/11-agent-auto-configuration.md). Over HTTP, point any MCP
client at the URL: `{"mcpServers": {"ckg": {"url": "http://127.0.0.1:8765/mcp"}}}`.

### Keep it fresh

```bash
ckg watch # local: re-index on a trigger (commit/idle/save)
ckg ci init # central: scaffold a CI workflow that indexes on merge
```

`ckg watch` (opt-in, `pip install 'agentforge-graph[watch]'`) keeps your working
copy's graph current — `on-commit` by default, so it won't churn on every save —
and refuses a central / read-only store. `ckg ci init` writes a single-writer
`.github/workflows/ckg-index.yml` so the shared index refreshes deterministically
on merge-to-`main`. See [guide 12](docs/guides/12-watch-and-ci-indexing.md).

Or as a native AgentForge toolset:

```python
Expand Down
177 changes: 177 additions & 0 deletions docs/design/design-014-watch-and-ci-indexing.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,177 @@
# design-014: Watch mode + CI-triggered central indexing

Mirrors [feat-014](../features/feat-014-watch-and-ci-indexing.md). The *how*:
file layout, types, resolved decisions.

## Guiding constraints

- **Framework-free** (ADR-0001): the watch package lives under
`ingest.watch`, orchestrating the deterministic `CodeGraph.refresh()`. No
`agentforge` import.
- **The trigger core is pure and clock-injected** so the whole policy matrix is
unit-tested with no filesystem, no sleeps, no `watchfiles`. The fs-watch loop
is a thin adapter over that core.
- **The local/central split is enforced in code**, not documented — `ckg watch`
refuses a central/read-only store before opening anything.

## Package layout

```
src/agentforge_graph/ingest/watch/
__init__.py # exports: Event, EventKind, TriggerPolicy, WatchSettings,
# WatchGuardError, ensure_watchable, branch_active,
# WatchRunner, WatchStatus, run_watch
policy.py # Event, EventKind, WatchSettings, TriggerPolicy (pure core)
gitwatch.py # head_ref(repo) -> str; branch_active(branch, inc, exc)
guard.py # WatchGuardError; ensure_watchable(store_cfg, read_only)
filter.py # WatchFilter: classify a changed path (ignored/git/source)
runner.py # WatchRunner (loop) + WatchStatus + run_watch()
source.py # watchfiles-backed async event source (lazy import)

src/agentforge_graph/ci/
__init__.py # exports: scaffold_workflow, render_workflow, CiInitResult
scaffold.py # provider dispatch + managed-marker write/idempotency
github.py # the GitHub Actions workflow template
```

## The trigger core (`policy.py`)

```python
class EventKind(Enum):
FILE = "file" # a source file changed (already ignore-filtered)
GIT = "git" # .git/HEAD or refs changed (commit / branch switch)

@dataclass(frozen=True)
class Event:
kind: EventKind
path: str = ""

@dataclass(frozen=True)
class WatchSettings: # resolved from WatchConfig + CLI overrides
trigger: str # on-commit | on-idle | on-save | interval | manual
debounce_ms: int
idle_ms: int
interval_ms: int
```

`TriggerPolicy` holds `_pending: bool` and timestamps (`_last_event`,
`_last_git`, `_last_fire`), all in **seconds** (float). Methods, all taking an
injected `now: float`:

- `observe(event, now)` — decide if the event counts for this mode:
- `manual`: ignore everything.
- `on-commit`: ignore FILE; on GIT set pending + `_last_git = now`.
- `on-idle` / `on-save`: FILE and GIT set pending + `_last_event = now`
(a commit is a real change too).
- `interval`: any event sets pending + `_last_event = now`.
- `due(now) -> bool` — should a refresh fire now?
- `on-commit`: pending and `now - _last_git >= debounce_ms` (small debounce
coalesces a branch-switch storm).
- `on-idle`: pending and `now - _last_event >= idle_ms`.
- `on-save`: pending and `now - _last_event >= debounce_ms`.
- `interval`: pending and `now - _last_fire >= interval_ms`.
- `manual`: always False.
- `next_due_in(now) -> float | None` — seconds until `due` flips True given
current pending state, or `None` when nothing is pending (wait indefinitely).
Drives the loop's timeout so the idle/interval window elapses even with no new
events.
- `reset(now)` — clear pending, `_last_fire = now`. Called after each refresh.

This is the entire testable surface: feed `observe`/`due`/`next_due_in` a
scripted clock and assert timing, with no I/O.

## The loop (`runner.py`)

`WatchRunner` is constructed with the resolved settings, a `refresh` coroutine,
a `now` callable (default `time.monotonic`), and a `pull` coroutine
(`async (timeout: float | None) -> Event | None`, `None` = timeout elapsed). The
default `pull` wraps `source.py` (watchfiles); tests inject a scripted `pull`.

```
run():
active = branch_active(current_branch, include, exclude) # may be False → idle-watch for a switch
while not stopped:
timeout = policy.next_due_in(now())
ev = await pull(timeout) # None on timeout
t = now()
if ev and ev.kind is GIT:
# branch switch may flip the gate
active = branch_active(current_branch, include, exclude)
if active and ev:
policy.observe(ev, t)
if active and policy.due(t):
await self._do_refresh() # CodeGraph.refresh() [+ embed if embed_on_watch]
policy.reset(now())
```

**Single-flight** falls out of the sequential `await`: events during a refresh
are buffered by watchfiles / the OS and observed on the next `pull`.

`--once` short-circuits the loop: one `CodeGraph.refresh()` (+ optional embed),
print the report, exit — no watcher constructed. `--status` reads `IndexMeta`
(indexed commit + mtime) + git head to print trigger mode · store · last commit
· dirty?, needing no running process (no IPC).

## Guard (`guard.py`)

```python
def ensure_watchable(store_cfg: StoreConfig, read_only: bool) -> None:
if store_cfg.central_root:
raise WatchGuardError("central store … build it in CI (ckg ci init)")
if read_only or store_cfg.read_only:
raise WatchGuardError("read-only store … watch only a writable embedded index")
```

The CLI handler calls this first and maps `WatchGuardError` to a stderr message
+ exit 2, reusing the ENH-018 read-only wiring already in `cli.py`.

## Ignore filter (`filter.py`)

`WatchFilter.classify(rel_path) -> EventKind | None`:
- path under `.git/` → is it `HEAD` or under `refs/` or `packed-refs`? → `GIT`;
any other `.git/` churn → `None` (ignored).
- matches an exclude glob (`IngestConfig.exclude` ∪ `DEFAULT_EXCLUDES` ∪
`watch.ignore`) via `PurePosixPath(rel).full_match(glob)` → `None`.
- otherwise, does a language pack claim the extension? no → `None`; yes → `FILE`.

Reuses the exact `full_match` matching `RepoSource` uses, so what watch reacts to
== what the indexer would ingest. `watchfiles` is given this as its filter so
ignored churn never even wakes the loop.

## `watchfiles` dependency (`source.py`, `[watch]` extra)

`watchfiles` (Rust `notify`, cross-platform, async `awatch`) is a new optional
dependency behind a `[watch]` extra — base install and CI stay lean, mirroring
`[rerank]`. `source.py` lazy-imports it; a missing dep raises a clear
"`pip install agentforge-graph[watch]`" hint (not a traceback). Where events are
unreliable, `watchfiles` already falls back to polling internally.

## CI scaffolder (`ci/`)

`ckg ci init` renders `.github/workflows/ckg-index.yml` from `github.py`: a
self-contained workflow (`pip install agentforge-graph[<extras>]` + `ckg index`
against `${{ secrets.CKG_CENTRAL_STORE_URL }}`), triggered on push-to-`main`,
nightly cron, and manual dispatch, with a `concurrency.group: ckg-central-index`
(single writer). The file opens with a managed marker comment
(`# managed-by: agentforge-graph`); `scaffold.py` refuses to overwrite a file
lacking the marker unless `--force`, is idempotent when the content matches, and
supports `--print` (render to stdout, write nothing) — the same discipline as
feat-013's `_managed_by` merge. A separately-versioned `ckg-index-action` is
explicitly out of scope (feat-014 §6); the self-contained workflow ships today.

## Config (`WatchConfig`, in `config.py`)

A new `_Block` with `KEY = "watch"` and a nested `BranchGate(include, exclude)`
(pydantic sub-model like `GraphCfg`). Auto-discovered by `block_keys()`. CLI
flags (`--trigger`, `--idle-ms`, `--debounce-ms`, `--interval-ms`, `--once`,
`--status`, `--embed`) override the loaded block into a `WatchSettings`.

## Test plan

`tests/watch/`: `test_policy.py` (the full trigger matrix, injected clock),
`test_filter.py` (git-meta vs source vs ignored), `test_gitwatch.py`
(branch_active globs), `test_guard.py` (central/read-only refusal),
`test_runner.py` (scripted `pull` + fake `refresh`: single-flight coalescing,
branch-gate activation, `--once`), `test_cli_watch.py` (flag→settings, guard
exit 2, `--status`). `tests/ci/test_scaffold.py` (render validity, idempotency,
clobber refusal, `--print`). Fake embedder / `asyncio_mode=auto` throughout.
2 changes: 2 additions & 0 deletions docs/features/TRACKER.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,7 @@ Legend: `proposed` → `accepted` → `in-progress` → `shipped` (also
| [011](feat-011-framework-extractors.md) | Framework-aware extractors | 3 diff | 0.4 | shipped v0.3.0 (7 packs: routes/ORM/DI) | 002 | — | 🟢 |
| [012](feat-012-llm-enrichment.md) | LLM enrichment (summaries, tags) | 3 diff | 0.4 | shipped (tags + summaries) | 006 | — | ✅ |
| [013](feat-013-agent-auto-configuration.md) | Agent auto-configuration & frictionless first run | 4 adoption | 0.6.2 | **shipped (0.6.2)** | 008 | — | ✅ |
| [014](feat-014-watch-and-ci-indexing.md) | Watch mode (local) + CI-triggered central indexing | 4 adoption | 0.6.3 | **shipped (0.6.3)** | 004 | — | ✅ |

---

Expand Down Expand Up @@ -140,6 +141,7 @@ ride alongside, not on, this chain.
| **0.5** ✅ RELEASED | **Org-level central knowledge** — CKG as a shared, org-wide code brain for devs + agents | ENH-018, ENH-019, ENH-020 | Knowledge hostable centrally + consumed read-only; zero-config repo discovery; **one federated MCP endpoint** across services with cross-service tracing — see [theme](../enhancements/THEME-org-central-knowledge.md) |
| **0.6** ✅ RELEASED | **Workspace build** — the build/setup side of org-central knowledge | ENH-021, ENH-022, ENH-023, ENH-024, ENH-026 | Stand up a multi-repo CKG from one `workspace.yaml` + one config + `ckg build --workspace` (members local or git URL); fail-fast `ckg doctor`; config/CLI-controlled logging/tracing |
| **0.6.2** ✅ RELEASED | **Adoption** — shrink "installed" → "an agent is querying my repo" | feat-013 (FA-002 + FA-001 P1) | `ckg setup` auto-writes the agent's MCP config (Claude Code first) + optional `--hooks` nudges toward the graph tools; `uvx`/`pipx` zero-install trial blessed |
| **0.6.3** ✅ RELEASED | **Freshness / ops** — keep the graph current without remembering to re-index | feat-014 (FA-005) | `ckg watch` re-indexes the local working copy on a conditional trigger (commit/idle/save, structural-only by default), refusing central/read-only stores; `ckg ci init` scaffolds a single-writer CI workflow that refreshes the central index on merge |

---

Expand Down
Loading
Loading