Skip to content

perf(topology): bound Sigma work on large graphs#71

Merged
CodeAbra merged 5 commits into
CodeAbra:mainfrom
Marsu6996:pr/bound-sigma-topology
Jul 13, 2026
Merged

perf(topology): bound Sigma work on large graphs#71
CodeAbra merged 5 commits into
CodeAbra:mainfrom
Marsu6996:pr/bound-sigma-topology

Conversation

@Marsu6996

@Marsu6996 Marsu6996 commented Jul 12, 2026

Copy link
Copy Markdown
Contributor

Draft / stacked PR

Do not merge before #67 through #70. This branch is stacked on those fixes, so the GitHub diff includes dependency commits until they land. It will then be rebased onto main to leave only the bounded Sigma topology work.

Problem

The exact native average-shortest-path kernel materializes an N x N float64
distance matrix. A Sigma observation computed it for the observed graph and
three random references, while compute_topology_snapshot also repeated work
already returned by fast_sigma. The snapshot then ran exact betweenness
centrality solely to count the top ten percent of nodes, even though that count
is fixed by definition.

Solution

  • preserve the exact APSL path through 2,500 nodes;
  • above that threshold, sample 512 deterministic source nodes and run SciPy's
    sparse unweighted Dijkstra in batches of 32;
  • reuse fast_sigma's observed Sigma, clustering, and path-length result in
    the topology snapshot;
  • compute the rich-club size ratio directly as ceil(0.10 * N) / N, without a
    discarded exact centrality ranking.

Guarantees and scope

Small and frozen fixtures keep exact native parity. Large-graph APSL becomes a
deterministic estimator; retrieval, embeddings, and HNSW ranking are untouched.
The helper continues to operate on the largest connected component, matching
the existing Sigma contract.

Validation

  • exact-boundary, deterministic sampling, bounded-batch, path-graph accuracy,
    single-pass snapshot, and centrality-cold tests;
  • a large encrypted-store snapshot completed the full topology calculation in
    seconds without an N x N allocation or an exact centrality call;
  • a 512-source run differed from a 2,048-source reference by less than 0.2% on
    that snapshot;
  • exhaustive default-suite result recorded on the final stacked tip.

Designed by Marsu — Refined by Codex.

Public CI result

The cumulative macOS correctness gate is fully green: 5,130 passed, 113 skipped, 107 deselected, and 4 xfailed in 1:16:17. The universal2 wheel and secret scan also pass.

Marsu6996 and others added 5 commits July 13, 2026 01:25
Warm the dispatch surface once, keep a timed-out background warmup alive explicitly, and release async-write and lifecycle resources safely when startup cancellation escapes the inner run loop.

Designed by Marsu — Refined by Codex.

Co-Authored-By: Codex <codex@openai.com>
Remove workstation-only and process-history assumptions, keep version checks source-aware, use the standard library for plist parsing, and preserve correctness signals without unstable global monkeypatches or hard timing gates.

Designed by Marsu — Refined by Codex.

Co-Authored-By: Codex <codex@openai.com>
Recognize exact iai_mcp and iai-mcp path components while preserving Desktop precedence and rejecting substring lookalikes.

Designed by Marsu — Refined by Codex.

Co-Authored-By: Codex <codex@openai.com>
Replay an edge only when both endpoints are active graph nodes, so stale storage edges cannot recreate deleted records or inflate topology work in either full or incremental rebuilds. Bump the graph-cache version once so polluted assignments and rich-club data cannot survive the fix.

Designed by Marsu — Refined by Codex.

Co-Authored-By: Codex <codex@openai.com>
Keep exact APSL through the frozen 2,500-node fixtures, then estimate from 512 deterministic sources in batches of 32 and reuse the observed pass so production topology audits cannot materialize repeated N-by-N matrices.

Designed by Marsu — Refined by Codex.

Co-Authored-By: Codex <codex@openai.com>
@Marsu6996 Marsu6996 force-pushed the pr/bound-sigma-topology branch from 9217f48 to 1af832b Compare July 12, 2026 23:32
@CodeAbra CodeAbra marked this pull request as ready for review July 13, 2026 03:10
@CodeAbra CodeAbra merged commit 6595ffa into CodeAbra:main Jul 13, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants