compdemocracy · jucor · Jun 11, 2026 · Jun 11, 2026 · Jun 11, 2026 · Jun 11, 2026
diff --git a/delphi/CLAUDE.md b/delphi/CLAUDE.md
@@ -16,7 +16,7 @@ this avoids the confusion of having anything called a "cid", the joke was "conve
 
 ## helpful background
 
-this was built in two parts, the pca/kmenas/repness and the umap/narrative, and these are combined in the run_delphi.sh script.
+this was built in two parts, the pca/kmenas/repness and the umap/narrative, and these are combined in the run_delphi.py script.
 
 ## Local Python Environment
 
@@ -175,7 +175,7 @@ AWS_SECRET_ACCESS_KEY=dummy
 AWS_REGION=us-east-1
 ```
 
-These are configured in run_delphi.sh for all DynamoDB operations.
+These are configured in run_delphi.py for all DynamoDB operations.
 
 ### DynamoDB Job Queue System
 
@@ -191,13 +191,13 @@ Delphi now includes a distributed job queue system built on DynamoDB:
 2. **Processing Jobs**: Start the job poller service:
 
    ```bash
-   ./start_poller.sh
+   python start_poller.py
    ```
 
 3. **Table Management**: To reset the job queue:
 
    ```bash
-   aws dynamodb delete-table --table-name DelphiJobQueue --endpoint-url http://localhost:8000 && \
+   aws dynamodb delete-table --table-name Delphi_JobQueue --endpoint-url http://localhost:8000 && \
    docker exec -e PYTHONPATH=/app polis-dev-delphi-1 python /app/create_dynamodb_tables.py --endpoint-url http://host.docker.internal:8000
    ```
 
@@ -210,7 +210,7 @@ Delphi now includes a distributed job queue system built on DynamoDB:
 ### Table Creation
 
 - Primary script: `/create_dynamodb_tables.py` - Creates BOTH Polis math and EVōC tables
-- This script is used in `run_delphi.sh` and now integrated into `umap_narrative/run_pipeline.py`
+- This script is used in `run_delphi.py` and now integrated into `umap_narrative/run_pipeline.py`
 
 ### Schema Definitions
 
@@ -242,7 +242,7 @@ Delphi now includes a distributed job queue system built on DynamoDB:
 - `Delphi_CollectiveStatement` - Collective statements generated for topics
 
 > **Note:** All table names now use the `Delphi_` prefix for consistency.
-> For complete documentation on the table renaming, see `/Users/colinmegill/polis/delphi/docs/DATABASE_NAMING_PROPOSAL.md`
+> Table definitions in `create_dynamodb_tables.py` are the canonical reference for names and schemas.
 
 ## Reset Single Conversation
 
@@ -281,7 +281,7 @@ See [RESET_SINGLE_CONVERSATION.md](docs/RESET_SINGLE_CONVERSATION.md) for detail
 After identifying the correct conversation ZID, run the Delphi pipeline directly with:
 
 ```bash
-./run_delphi.sh --zid=[ZID]
+python run_delphi.py --zid [ZID]
 ```
 
 Additional options include:
@@ -297,7 +297,7 @@ For production environments, use the job queue system:
 1. Start the poller service on your worker machine:
 
    ```bash
-   ./start_poller.sh
+   python start_poller.py
    ```
 
 2. Submit a job from any machine with access to DynamoDB:
@@ -323,13 +323,6 @@ For production environments, use the job queue system:
    docker exec -e PYTHONPATH=/app polis-dev-delphi-1 python /app/create_dynamodb_tables.py --endpoint-url http://host.docker.internal:8000
    ```
 
-   Or use the reset_database.sh script to recreate all tables:
-
-   ```bash
-   # Reset all tables (both Polis math and EVōC tables)
-   ./reset_database.sh
-   ```
-
 2. **Testing specific pipeline stages**:
 
    ```bash

diff --git a/delphi/docs/CLJ-PARITY-FIXES-JOURNAL.md b/delphi/docs/CLJ-PARITY-FIXES-JOURNAL.md
diff --git a/delphi/docs/CLOJURE_COMPARISON.md b/delphi/docs/CLOJURE_COMPARISON.md
@@ -49,10 +49,13 @@ The Clojure reference implementation is in: **`math/src/polismath/math/clusters.
 
 This is the **primary reason** clustering results differ between Python and Clojure:
 
-**Python** (Single-level clustering):
-- `group_clusters`: Direct clustering of participants into k groups
-- Member IDs: Participant IDs
-- Example: {id: 0, members: [ptpt1, ptpt2, ...]}
+**Python** (Two-level clustering, matching Clojure since PR #2431):
+- `base_clusters`: First-level clustering (~100 small clusters of participants)
+  - Member IDs: Participant IDs
+  - Example: 100 base clusters with 3-7 participants each
+- `group_clusters`: Second-level clustering of base clusters into k groups
+  - Members stored as base-cluster IDs internally, unfolded to participant IDs for serialization
+  - Example: {id: 0, members: [0, 1, 5, 8, ...]} where numbers are base cluster IDs (internally)
 
 **Clojure** (Two-level clustering):
 1. `base-clusters`: First-level clustering of participants into ~100 small clusters
@@ -70,7 +73,7 @@ Beyond the architecture, there's also an initialization difference:
 
 | Aspect | Python | Clojure |
 |--------|--------|---------|
-| **Algorithm** | K-means++ (seed 42) | First k distinct points |
+| **Algorithm** | First k distinct points (matching Clojure) | First k distinct points |
 | **Rationale** | Better convergence, industry standard | Simpler implementation |
 | **Result** | Different local optima | Different local optima |
 | **Quality** | Both are valid clustering algorithms | Both are valid clustering algorithms |
@@ -96,12 +99,9 @@ Beyond the architecture, there's also an initialization difference:
 
 ### Why Tests Fail
 
-The clustering test **intentionally fails** because:
-1. Python uses K-means++ initialization → different initial cluster centers
-2. K-means converges to nearest local optimum → different final clusters
-3. Tests use very tight thresholds (95% Jaccard, 5% L1) to detect any difference
-
-This is **expected behavior** until we implement Option A (match Clojure initialization).
+The clustering test **xfails conditionally** on some dataset variants due to incremental-blob
+in-conv divergence / cold-start PCA landscape flatness — NOT initialization mismatch.
+Python now uses first-k-distinct initialization, matching Clojure (since PR #2431).
 
 ## Running Tests
 

diff --git a/delphi/docs/DELPHI_DOCKER.md b/delphi/docs/DELPHI_DOCKER.md
@@ -7,7 +7,7 @@ This document provides information about the Delphi Docker container setup and o
 When the Delphi container starts, it performs the following steps:
 
 1. Initializes DynamoDB tables using `create_dynamodb_tables.py`
-2. Starts the job poller service using `start_poller.sh`
+2. Starts the job poller by running `python scripts/job_poller.py` directly (the Dockerfile CMD invokes the script)
 
 ## Environment Variables
 
@@ -32,8 +32,7 @@ The Delphi container runs the following services:
 If the container exits with code 127, check that:
 
 1. The scripts directory is correctly copied into the container
-2. The `start_poller.sh` script is executable
-3. The DynamoDB endpoint is correct and accessible
+2. The DynamoDB endpoint is correct and accessible
 
 ## Maintaining State
 

diff --git a/delphi/docs/DELPHI_JOB_SYSTEM_TROUBLESHOOTING.md b/delphi/docs/DELPHI_JOB_SYSTEM_TROUBLESHOOTING.md
@@ -325,5 +325,5 @@ print(f"Reset {count} stuck jobs")
 ## Related Documentation
 
 - [JOB_QUEUE_SCHEMA.md](JOB_QUEUE_SCHEMA.md) - Details about the job queue schema
-- [ANTHROPIC_BATCH_API_GUIDE.md](ANTHROPIC_BATCH_API_GUIDE.md) - Guide for working with Anthropic's Batch API
-- [DATABASE_NAMING_PROPOSAL.md](DATABASE_NAMING_PROPOSAL.md) - Information about database naming conventions
+- [DATA_FORMAT_STANDARDS.md](DATA_FORMAT_STANDARDS.md) - DynamoDB key formats and reserved-keyword handling
+- [JOB_STATE_MACHINE_DESIGN.md](JOB_STATE_MACHINE_DESIGN.md) - Job types and state transitions
diff --git a/delphi/docs/DOCKER_BUILD_OPTIMIZATION.md b/delphi/docs/DOCKER_BUILD_OPTIMIZATION.md
@@ -88,6 +88,8 @@ vim pyproject.toml
 
 # 2. Regenerate lock file
 make generate-requirements
+# Note: the Makefile target calls pip-compile directly; project tooling policy is uv
+# (equivalent: uv pip compile pyproject.toml -o requirements.lock)
 
 # 3. Rebuild Docker image
 make docker-build
@@ -145,7 +147,7 @@ make generate-requirements-upgrade
 ```txt
 # requirements.lock (generated by pip-compile)
 #
-# This file is autogenerated by pip-compile with Python 3.13
+# This file is autogenerated by pip-compile with Python 3.12
 # by the following command:
 #
 #    pip-compile --output-file=requirements.lock pyproject.toml

diff --git a/delphi/docs/DOCUMENTATION_DIRECTORY.md b/delphi/docs/DOCUMENTATION_DIRECTORY.md
@@ -1,91 +1,71 @@
 # Delphi Documentation Directory
 
-This document provides an overview of key documentation files in the Delphi system, organized by topic for easy reference.
+Index of the documentation in `delphi/docs/`, organized by topic.
 
-## Core System Documentation
+> **Last cleaned: 2026-06-11.** 33 stale leftover docs (completed fix memos, session
+> logs, unimplemented design proposals, docs describing deleted architecture) were
+> moved to [archive/](archive/) — kept as raw material capturing the original design
+> intent of the 2025 build-out, but **not documentation of the current system** (see
+> [archive/CLAUDE.md](archive/CLAUDE.md) for the per-file index). Surviving docs were
+> spot-checked against the code on that date; still, when a doc and the code disagree,
+> trust the code.
 
-| Document | Description |
-|----------|-------------|
-| [CLAUDE.md](../CLAUDE.md) | Main reference guide with configuration details, database interactions, and system operation |
-| [README.md](../README.md) | Project overview and basic setup instructions |
-| [QUICK_START.md](QUICK_START.md) | Get started quickly with the Delphi system |
-| [RUNNING_THE_SYSTEM.md](RUNNING_THE_SYSTEM.md) | Step-by-step instructions for operating the Delphi system |
-| [architecture_overview.md](architecture_overview.md) | High-level overview of the system architecture |
-| [project_structure.md](project_structure.md) | Explanation of the project's directory and file organization |
-
-## Database and Data Format Documentation
+## Canonical / living documents
 
 | Document | Description |
 |----------|-------------|
-| [DATABASE_NAMING_PROPOSAL.md](DATABASE_NAMING_PROPOSAL.md) | Explanation of table naming conventions and migration plan |
-| [DATA_FORMAT_STANDARDS.md](DATA_FORMAT_STANDARDS.md) | **Critical standards for data formats throughout the system, including DynamoDB key formats** |
-| [JOB_QUEUE_SCHEMA.md](JOB_QUEUE_SCHEMA.md) | Schema documentation for the job queue system |
-| [S3_STORAGE.md](S3_STORAGE.md) | Information about S3 storage configuration and access |
+| [PLAN_DISCREPANCY_FIXES.md](PLAN_DISCREPANCY_FIXES.md) | **Canonical plan** for the Clojure-parity fix campaign (D-fixes), statuses, ordering |
+| [CLJ-PARITY-FIXES-JOURNAL.md](CLJ-PARITY-FIXES-JOURNAL.md) | **Append-only session journal** of the parity work — findings, decisions, test results |
+| [deep-analysis-for-julien/](deep-analysis-for-julien/) | Deep Clojure-vs-Python analysis (architecture, PCA, clustering, repness, routing, discrepancy catalog). Historical reference; see status note in `07-discrepancies.md` |
 
-## Job System Documentation
+## Getting started & operations
 
 | Document | Description |
 |----------|-------------|
-| [JOB_SYSTEM_DESIGN.md](JOB_SYSTEM_DESIGN.md) | Overall job system architecture and design principles |
-| [JOB_STATE_MACHINE_DESIGN.md](JOB_STATE_MACHINE_DESIGN.md) | **Detailed explanation of the job state machine and workflow design** |
-| [DELPHI_JOB_SYSTEM_TROUBLESHOOTING.md](DELPHI_JOB_SYSTEM_TROUBLESHOOTING.md) | **Comprehensive guide to troubleshooting common job system issues** |
-| [JOB_ID_MIGRATION_PLAN.md](JOB_ID_MIGRATION_PLAN.md) | Plan for migrating to the new job ID system |
-
-## API Integration Documentation
+| [QUICK_START.md](QUICK_START.md) | Environment setup (uv/.venv) and standard test invocation |
+| [RUNNING_THE_SYSTEM.md](RUNNING_THE_SYSTEM.md) | Operating the pipeline: run_delphi, CLI, job submission |
+| [DELPHI_DOCKER.md](DELPHI_DOCKER.md) | Delphi container overview |
+| [DOCKER_BUILD_OPTIMIZATION.md](DOCKER_BUILD_OPTIMIZATION.md) | Layered Docker builds, requirements.lock workflow |
+| [DELPHI_AUTOSCALING_SETUP.md](DELPHI_AUTOSCALING_SETUP.md) | Instance-size based worker configuration (`configure_instance.py`, `INSTANCE_SIZE`) |
+| [RESET_SINGLE_CONVERSATION.md](RESET_SINGLE_CONVERSATION.md) | Removing all Delphi data for one conversation |
+| [S3_STORAGE.md](S3_STORAGE.md) | S3/MinIO storage for visualizations |
+| [OLLAMA_MODEL_CONFIG.md](OLLAMA_MODEL_CONFIG.md) | Ollama model configuration for topic naming |
+| [CLI_STATUS_COMMAND.md](CLI_STATUS_COMMAND.md) | `./delphi status <zid>` CLI command |
+
+## Job system
 
 | Document | Description |
 |----------|-------------|
-| [ANTHROPIC_BATCH_API_GUIDE.md](ANTHROPIC_BATCH_API_GUIDE.md) | **Complete guide for working with Anthropic's Batch API, including common issues and solutions** |
-| [OLLAMA_MODEL_CONFIG.md](OLLAMA_MODEL_CONFIG.md) | Configuration guide for Ollama models |
-| [CLI_STATUS_COMMAND.md](CLI_STATUS_COMMAND.md) | Documentation for the CLI status command |
+| [JOB_STATE_MACHINE_DESIGN.md](JOB_STATE_MACHINE_DESIGN.md) | Job types (FULL_PIPELINE, CREATE_NARRATIVE_BATCH, AWAITING_NARRATIVE_BATCH) and transitions |
+| [JOB_QUEUE_SCHEMA.md](JOB_QUEUE_SCHEMA.md) | `Delphi_JobQueue` schema, GSIs, locking patterns |
+| [DELPHI_JOB_SYSTEM_TROUBLESHOOTING.md](DELPHI_JOB_SYSTEM_TROUBLESHOOTING.md) | Diagnosing stuck jobs, DynamoDB gotchas, log locations |
+| [DATA_FORMAT_STANDARDS.md](DATA_FORMAT_STANDARDS.md) | DynamoDB key formats (`#` delimiters), reserved keywords, type conversions |
 
-## Deployment and Infrastructure
+## Math & Clojure parity reference
 
 | Document | Description |
 |----------|-------------|
-| [DELPHI_AUTOSCALING_SETUP.md](DELPHI_AUTOSCALING_SETUP.md) | Configuration for auto-scaling the system |
-| [DISTRIBUTED_SYSTEM_ROADMAP.md](DISTRIBUTED_SYSTEM_ROADMAP.md) | Roadmap for distributed system improvements |
+| [CLOJURE_COMPARISON.md](CLOJURE_COMPARISON.md) | Clojure-vs-Python comparison test infrastructure and known differences |
+| [CLOJURE_TWO_LEVEL_CLUSTERING.md](CLOJURE_TWO_LEVEL_CLUSTERING.md) | Two-level (base→group) clustering architecture, as implemented |
+| [SUBGROUP_CLUSTERING_THIRD_LEVEL.md](SUBGROUP_CLUSTERING_THIRD_LEVEL.md) | Clojure's third clustering level (subgroups) — unported, unused by consumers |
+| [regression_testing.md](regression_testing.md) | Golden-snapshot regression testing: recorder, comparer, datasets |
+| [INVESTIGATION_K_DIVERGENCE.md](INVESTIGATION_K_DIVERGENCE.md) | K-means k divergence investigation (RESOLVED — kept as record) |
+| [SESSION_HANDOFF_KMEANS.md](SESSION_HANDOFF_KMEANS.md) | K-means parity session background (partially historical — see status note) |
+| [HANDOFF_PR14_VECTORIZED_REFACTOR.md](HANDOFF_PR14_VECTORIZED_REFACTOR.md) | Repness refactor handoff (14a in open stack; 14b/14c open) |
+| [HANDOFF_REGRESSION_TEST_PERF.md](HANDOFF_REGRESSION_TEST_PERF.md) | Regression-test performance (mostly resolved — see status note) |
 
-## Algorithm and Analysis Documentation
+## Topic pipeline (umap_narrative)
 
 | Document | Description |
 |----------|-------------|
-| [algorithm_analysis.md](algorithm_analysis.md) | Analysis of the core algorithms used in Delphi |
-| [TOPIC_NAMING.md](TOPIC_NAMING.md) | Topic naming pipeline: exact prompt, deterministic 5‑comment sampling, logging, storage |
-| [usage_examples.md](usage_examples.md) | Examples of system usage and output interpretations |
+| [TOPIC_NAMING.md](TOPIC_NAMING.md) | Topic naming: prompt, sampling, storage in `Delphi_CommentClustersLLMTopicNames` |
+| [topic-moderation-system.md](topic-moderation-system.md) | Topic moderation endpoints and `Delphi_TopicModerationStatus` |
+| [TOPIC_AGENDA_STORAGE_DESIGN.md](TOPIC_AGENDA_STORAGE_DESIGN.md) | Topic agenda storage (`topic_agenda_selections`); Phase 3 never implemented |
+| [VERSIONED_TOPIC_KEYS_IMPLEMENTATION.md](VERSIONED_TOPIC_KEYS_IMPLEMENTATION.md) | Versioned topic/section keys (`report_id#section#model`) |
 
-## Testing and Development
+## Open issues & audits (still unresolved — do not delete until fixed)
 
 | Document | Description |
 |----------|-------------|
-| [SIMPLIFIED_TESTS.md](SIMPLIFIED_TESTS.md) | Simplified testing procedures |
-| [TESTING_LOG.md](TESTING_LOG.md) | Log of testing activities and results |
-| [TEST_RESULTS_SUMMARY.md](TEST_RESULTS_SUMMARY.md) | Summary of test results |
-
-## Recently Added Documentation
-
-The following documentation was recently added to address specific system challenges:
-
-1. **[ANTHROPIC_BATCH_API_GUIDE.md](ANTHROPIC_BATCH_API_GUIDE.md)** - Comprehensive guide for working with Anthropic's Batch API in the Delphi system, including:
-   - Handling JSON Lines (JSONL) responses from the API
-   - Proper error handling for API interactions
-   - Key format requirements for storing results in DynamoDB
-   - Debugging strategies for batch processing issues
-
-2. **[DELPHI_JOB_SYSTEM_TROUBLESHOOTING.md](DELPHI_JOB_SYSTEM_TROUBLESHOOTING.md)** - Detailed guide for troubleshooting job system issues, including:
-   - Strategies for diagnosing stuck jobs
-   - Solutions for common DynamoDB reserved keyword issues
-   - Techniques for tracing end-to-end job execution
-   - Database verification processes
-
-3. **[DATA_FORMAT_STANDARDS.md](DATA_FORMAT_STANDARDS.md)** - Critical standards document focusing on:
-   - Required format for DynamoDB keys (using # as delimiters)
-   - JSON structure standards for reports
-   - Handling of reserved keywords in DynamoDB
-   - Conversion between PostgreSQL and DynamoDB data types
-
-4. **[JOB_STATE_MACHINE_DESIGN.md](JOB_STATE_MACHINE_DESIGN.md)** - Documentation of the state machine design for job processing:
-   - Explicit job types for different processing stages
-   - Clear script mapping between job types and processing scripts
-   - Clean state transition patterns
-   - Error handling best practices
+| [ZID_EXPOSURE_AUDIT.md](ZID_EXPOSURE_AUDIT.md) | **Open security issue**: zid/conversation_id exposed in delphi API responses |
+| TOPIC_LABEL_MISALIGNMENT_ANALYSIS.md | **Open bug**: label/cluster sorting misalignment in `700_datamapplot_for_layer.py` (local-only file, in `.git/info/exclude` — not present on fresh clones) |
diff --git a/delphi/docs/HANDOFF_PR14_VECTORIZED_REFACTOR.md b/delphi/docs/HANDOFF_PR14_VECTORIZED_REFACTOR.md
@@ -1,5 +1,7 @@
 # Handoff: PR 14 — Make Vectorized Code Readable + Blob Injection Tests
 
+> **Status (2026-06-11):** PR 14a (delete dead scalar paths in repness.py) is in the open spr stack as PR #2564. Tasks 14b (vectorized blob-injection tests) and 14c (readability refactor) remain open and are tracked in `PLAN_DISCREPANCY_FIXES.md`. The branch names and stack listing below are from the 2026-03/06 sessions and are stale — do not branch from them.
+
 ## Goal
 
 The scalar functions (`comment_stats`, `add_comparative_stats`, `repness_metric`,

diff --git a/delphi/docs/HANDOFF_REGRESSION_TEST_PERF.md b/delphi/docs/HANDOFF_REGRESSION_TEST_PERF.md
@@ -1,5 +1,7 @@
 # Handoff: Regression Test Performance Investigation
 
+> **Status (2026-06-11):** Bottleneck 1 (`_compute_participant_info_optimized`) was vectorized (conversation.py group-correlation matrix ops). Bottleneck 2 (benchmark 3× runs) is resolved — `benchmark=False` is now the default in `comparer.py` and `test_regression.py` never enables it. `SKIP_GOLDEN` landed via #2515. Only the intermediate-stage redundancy question (Bottleneck 3) remains open.
+
 ## Problem
 
 The `test_regression.py` tests are slow for large private datasets, particularly

diff --git a/delphi/docs/JOB_QUEUE_SCHEMA.md b/delphi/docs/JOB_QUEUE_SCHEMA.md
@@ -7,7 +7,7 @@ This document defines the schema for the Delphi job queue system. The job queue
 ## Table Design
 
 ### Table Name
-`DelphiJobQueue`
+`Delphi_JobQueue`
 
 ### Primary Key Structure
 - **Partition Key**: `job_id` (String) - Unique identifier for each job (UUID v4)
@@ -287,12 +287,14 @@ To manage the growth of the job queue table:
 
 ## Implementation Code
 
+> **Caution:** The sample below predates the final schema — the actual table (see `create_dynamodb_tables.py`) uses `job_id` as the sole hash key, not `status`+`created_at`.
+
 Here's a sample Python code for creating the job queue table:
 
 ```python
 import boto3
 
-def create_job_queue_table(dynamodb=None, table_name='DelphiJobQueue'):
+def create_job_queue_table(dynamodb=None, table_name='Delphi_JobQueue'):
     if not dynamodb:
         dynamodb = boto3.resource('dynamodb')