From 133750eb72f038d4440be1a491c489bda7d2c3b1 Mon Sep 17 00:00:00 2001 From: Lodewijk <32471480+lgelauff@users.noreply.github.com> Date: Thu, 4 Jun 2026 12:54:44 +0200 Subject: [PATCH 1/3] docs(server): add a data-model orientation doc MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Orients a developer to the core participation model (conversations -> comments -> votes/votes_latest_unique -> participants), identity (users, xids), and the math_* output tables, and calls out the conventions that surprise people: per-conversation tid/pid via triggers, epoch-millis timestamps, vis_type gating results, and the raw vote sign (-1 = Agree) vs the export's flipped sign. Content verified by loading all 19 migrations into a throwaway Postgres DB and inspecting the real schema (rules, triggers, defaults, table set), and by grepping server/src for table references — corrected several inventory claims (phantom org tables, the 61-table count, tables still referenced by live code). Matches the house style of the other server/docs; the migrations remain the source of truth. Refs #2554 --- server/docs/data-model.md | 184 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 184 insertions(+) create mode 100644 server/docs/data-model.md diff --git a/server/docs/data-model.md b/server/docs/data-model.md new file mode 100644 index 0000000000..e4dd09eef5 --- /dev/null +++ b/server/docs/data-model.md @@ -0,0 +1,184 @@ +# Polis Data Model + +This document is an orientation to the core PostgreSQL data model — the tables a +developer touches when working on conversations, comments, votes, and participants. It +is not an exhaustive column-by-column reference; the **source of truth is always the SQL +in [`server/postgres/migrations/`](../postgres/migrations/)** (the `000000_initial.sql` +file plus the numbered migrations applied on top of it). The goal here is to make the +schema *legible*: how the pieces relate, and the handful of conventions that surprise +people. + +If you just want to apply migrations, see [docs/migrations.md](../../docs/migrations.md). +For the *shape of the math output* (not the storage schema), see +[docs/pca.md](../../docs/pca.md). + +--- + +## Keys and conventions you need up front + +A few things are true almost everywhere, and knowing them removes most of the confusion: + +- **`zid`** — conversation id. `SERIAL`, globally unique. Internal; never shown to users. +- **`tid`** — comment (statement) id. **Per-conversation**, assigned by the `tid_auto` + trigger — so comment ids start at 0 in every conversation. The real key is `(zid, tid)`. +- **`pid`** — participant id. **Per-conversation**, assigned by the `pid_auto` trigger — + likewise starts at 0 per conversation. The real key is `(zid, pid)`. +- **`uid`** — user (account) id. `SERIAL`, globally unique. A `uid` maps to a `pid` *within* + a conversation via the `participants` table. +- **`zinvite`** — the public, opaque conversation identifier (the string in a conversation + URL). It maps to the internal `zid` via the `zinvites` table. **Outside callers know the + `zinvite`; the database works in `zid`.** +- **Timestamps are `BIGINT` epoch-milliseconds** (`created`, `modified`, …), defaulted via + `now_as_millis()`. Use `to_timestamp(created / 1000.0)` to read one. +- **`-1` does not mean "no" / "negative".** Several columns use small signed integers as + enums where `-1` is a *specific state*, not a sentinel — see the vote sign and `mod` + columns below. Read the column comment in `000000_initial.sql` before assuming. + +--- + +## The core: conversations → comments → votes + +This is the participation loop. Everything else is identity, configuration, or computed +output around it. + +### `conversations` (keyed by `zid`) + +One row per conversation/poll. Holds the content (`topic`, `description`) and a large set +of configuration flags. The ones that matter most operationally: + +- `is_active`, `is_public`, `is_draft` — lifecycle/visibility. +- `strict_moderation` — if true, new comments must be approved before others can vote on them. +- `write_type` — `1` shows the comment form, `0` hides it. +- **`vis_type`** — gates the results visualization. **`0` = results off (the default), `1` + = on.** `GET /api/.../results/` returns nothing until this is non-zero. This is a common + "why are there no results?" cause. +- `owner` → `users(uid)` — who created/owns it. +- Later migrations add `xid_required`, `treevite_enabled`, and `topics_enabled`. + +### `zinvites` (`zid` ↔ `zinvite`) + +Maps the internal `zid` to one or more public `zinvite` strings (unique). A later +migration also adds a `uuid`. This is the indirection that lets the public id be opaque +and stable while internal joins use the integer `zid`. + +### `comments` (keyed by `(zid, tid)`) + +One row per statement participants vote on. `txt` is the statement; `uid`/`pid` is the +author. Key columns: + +- **`mod`** — moderation state, a plain `INTEGER DEFAULT 0` (no DB-level CHECK, but used as an + enum): `-1` = rejected, `0` = unmoderated, `1` = accepted. +- `active` — false hides the comment regardless of `mod`. +- `is_seed` — seeded by the moderator (vs. submitted by a participant). +- `is_meta` — a meta/notice comment, excluded from the math. +- `velocity` — moderation/ranking weight. + +`comments` has a foreign key on `(zid, pid)` into `participants`: a comment must be authored +by a known participant of that conversation. + +### `votes` and `votes_latest_unique` (keyed by `(zid, pid, tid)`) + +Two tables, one concept, and the single most surprising part of the schema. + +- **`votes` is append-only.** Every vote — including a participant *changing* their vote — + inserts a new row. `created` is the epoch-millis timestamp; there is no `modified` column. +- **`votes_latest_unique` is the authoritative current vote** per `(zid, pid, tid)`. A + `RULE` (`on_vote_insert_update_unique_table`) upserts this table on every insert into + `votes`, so it always reflects the latest vote, with a `modified` timestamp. **Query this + table for current state; query `votes` for history.** + +The `vote` value itself: + +``` +vote SMALLINT -- -1 = Agree, 1 = Disagree, 0 = Pass/Unsure +``` + +> ⚠️ **The raw sign is the opposite of the data export.** In these tables (and in the vote +> API) `-1` means **Agree**. The `participants-votes` CSV export *negates* every vote so +> that Agree = `+1`, Disagree = `-1` (see `get-corrected-conversation-votes` in +> [`math/src/polismath/darwin/export.clj`](../../math/src/polismath/darwin/export.clj)). +> So both are "correct" at different layers — but when you read the `votes` table directly, +> **`-1` is Agree.** + +`weight_x_32767` is a separate priority/weight value stored as a fixed-point integer: +divide by 32767 to get a float in `[-1, 1]` (it is unrelated to the `vote` value). + +### `participants` (keyed by `(zid, pid)`, also unique on `(zid, uid)`) + +One row per (user, conversation): a `uid` that has joined conversation `zid`, with their +per-conversation `pid`. Tracks `vote_count`, `last_interaction`, subscription state, and a +`mod` field for the *participant* (`-1` hide-from-vis, `0` none, `1` acknowledge, `2` pin). +The `mod` column is a plain `INTEGER DEFAULT 0` (no DB-level CHECK). +`participants_extended` holds optional extras (referrer, parent_url, a permanent cookie). + +--- + +## Identity: `users` and `xids` + +- **`users` (keyed by `uid`)** — accounts. `email`, `hname`/`username`, and `is_owner` + (can start conversations). Both registered owners and ephemeral participants are `users`. +- **`xids` (cross-identity)** — maps an **external identifier** (`xid`, supplied by an + embedding site) to a `uid`, scoped to an `owner` (the account that owns the integration): + `UNIQUE (owner, xid)`. Carries optional `x_name` / `x_email` / `x_profile_image_url`. This + is how SSO/embedded deployments attach their own user identity to a Polis participant + without Polis accounts. (`xid_whitelist` restricts which xids may participate when a + conversation sets `xid_required`.) +- Invite/token tables sit alongside these: `zinvites` (above), `einvites`/`oinvites`/ + `suzinvites` (email/owner/single-use invites), `auth_tokens`, `pwreset_tokens`, and + `oidc_user_mappings` (OIDC SSO). + +--- + +## Computed output: the `math_*` tables + +The **math** (Clojure) and **delphi** (Python) services poll the vote data, run PCA / +clustering, and write results back for the server and report clients to read. The schema +side is small: + +- **`math_main`** — the headline result: `data jsonb` holds the PCA/clustering object + (consensus, opinion groups, comment stats — the structure documented in + [docs/pca.md](../../docs/pca.md)), keyed by `(zid, math_env)`, with `math_tick` / + `last_vote_timestamp` for cache-invalidation. +- Supporting tables: `math_profile`, `math_ptptstats`, `math_bidtopid`, `math_ticks`, + `math_cache`, `math_exportstatus`, `math_report_correlationmatrix`, and `stats_per_comment`. +- **`reports` / `report_comment_selections`** — saved/curated report views over a conversation. + +The server never computes these; it only reads them. An empty results endpoint usually +means either `vis_type = 0` (above) or that math hasn't produced output yet. + +--- + +## Everything else, at a glance + +After all migrations are applied, the schema has **61 base tables**. Beyond the core above, +they group as follows. Most need no day-to-day attention. + +| Group | Tables | Notes | +|---|---|---| +| Multi-tenant / SaaS | `inviters`, `contexts` | The hosted-product account layer. Note: classic Polis `orgs` / `oadmins` / `omemberships` / `permissions` tables are **not present** in this fork's schema. | +| Participant metadata / surveys | `participant_metadata_questions` / `_answers` / `_choices`, `demographic_data`, `participant_locations` | Optional per-participant survey/demographic data. | +| Features | `treevite_waves` / `treevite_invites` / `treevite_login_codes` (gated entry), `topic_agenda_selections`, `byod_import_jobs`, `notification_tasks`, `worker_tasks`, `event_ptpt_no_more_comments`, `crowd_mod` | Newer or peripheral features. | +| Translations | `comment_translations`, `conversation_translations` | Cached machine translations. | +| Legacy social login | `facebook_users`, `facebook_friends`, `twitter_users`, `social_settings` | Social-login tables. No references in `server/src`, `math`, or `delphi` — dead in this fork. | +| Other dormant tables | `courses`, `beta`, `contributer_agreement_signatures`, and the historically obfuscated-name auth tables `apikeysndvweifu` (api keys) and `jianiuevyew` (password hashes) | No references found in `server/src`, `math`, or `delphi` (this fork authenticates via OIDC, so the password-hash table is unused). Retained for migration history. | + +> ⚠️ **Not "legacy" — still referenced by live server code.** Several tables that look +> vestigial are actually written/read by current `server/src` SQL, so do not assume they are +> safe to drop: `stars`, `upvotes`, `trashes` (comment moderation actions), `page_ids` and +> `permanentcookiezidjoins` (embed / implicit-conversation + legacy permanent-cookie auth), +> `metrics` (client metrics insert), `email_validations`, and `site_domain_whitelist` +> (embed domain allow-list). (There is **no table named `permanent`** — the permanent-cookie +> data lives in `permanentcookiezidjoins` and in `participants_extended.permanent_cookie`.) + +A handful of tables (Slack/Stripe/Canvas/LTI integrations, the waiting list, +geolocation) were dropped by migrations 000004 / 000005 / 000007 and no longer exist. + +--- + +## Where to look next + +- **Exact columns / constraints:** `server/postgres/migrations/000000_initial.sql`, then + the numbered migrations for changes since. +- **Applying migrations:** [docs/migrations.md](../../docs/migrations.md). +- **The math result structure:** [docs/pca.md](../../docs/pca.md). +- **Auth and identity internals:** [server/docs/authentication.md](./authentication.md). From c31473a5c658e6f2d545936444cea670a5b4af36 Mon Sep 17 00:00:00 2001 From: Lodewijk <32471480+lgelauff@users.noreply.github.com> Date: Mon, 8 Jun 2026 21:10:21 +0200 Subject: [PATCH 2/3] =?UTF-8?q?docs(server):=20address=20review=20?= =?UTF-8?q?=E2=80=94=20vis=5Ftype/stats/permanentcookie=20accuracy,=20delp?= =?UTF-8?q?hi+DynamoDB=20caveat,=20SQL<->doc=20backlink?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review fixes on the data-model doc: - vis_type: describe as a client-side UI render flag, not server-side gating of the results endpoint (Copilot). - Drop stats_per_comment from the math inventory — commented out in 000000_initial.sql, not in the current migration set (Copilot). - permanentcookiezidjoins: only present in the schema, no server/src|math|delphi references — move out of the 'still referenced by live code' list and mark its status unverified (Copilot). - Add caveat that the math_* description is the Clojure path; delphi computes much more and stores most of it in DynamoDB, not Postgres (jucor). - velocity / 'known' participant: hedge + clarify (jucor). - zinvite sensitivity note; expand xid_required/treevite_enabled/topics_enabled; NOT-caps (jucor). - Add a KEEP IN SYNC banner to 000000_initial.sql pointing back to data-model.md so schema and doc stay linked (jucor, blocking). --- server/docs/data-model.md | 50 +++++++++++++------ server/postgres/migrations/000000_initial.sql | 5 ++ 2 files changed, 39 insertions(+), 16 deletions(-) diff --git a/server/docs/data-model.md b/server/docs/data-model.md index e4dd09eef5..f582c45bbc 100644 --- a/server/docs/data-model.md +++ b/server/docs/data-model.md @@ -27,10 +27,13 @@ A few things are true almost everywhere, and knowing them removes most of the co a conversation via the `participants` table. - **`zinvite`** — the public, opaque conversation identifier (the string in a conversation URL). It maps to the internal `zid` via the `zinvites` table. **Outside callers know the - `zinvite`; the database works in `zid`.** + `zinvite`; the database works in `zid`.** Treat the `zinvite` as semi-public: it appears in + voting URLs, but when auth/XID isn't enabled anyone holding it can vote, so don't circulate + it beyond the intended voters. `zid` is the internal key; how sensitive it is depends on + your threat model. - **Timestamps are `BIGINT` epoch-milliseconds** (`created`, `modified`, …), defaulted via `now_as_millis()`. Use `to_timestamp(created / 1000.0)` to read one. -- **`-1` does not mean "no" / "negative".** Several columns use small signed integers as +- **`-1` does NOT mean "no" / "negative".** Several columns use small signed integers as enums where `-1` is a *specific state*, not a sentinel — see the vote sign and `mod` columns below. Read the column comment in `000000_initial.sql` before assuming. @@ -49,11 +52,16 @@ of configuration flags. The ones that matter most operationally: - `is_active`, `is_public`, `is_draft` — lifecycle/visibility. - `strict_moderation` — if true, new comments must be approved before others can vote on them. - `write_type` — `1` shows the comment form, `0` hides it. -- **`vis_type`** — gates the results visualization. **`0` = results off (the default), `1` - = on.** `GET /api/.../results/` returns nothing until this is non-zero. This is a common - "why are there no results?" cause. +- **`vis_type`** — UI flag for whether the results visualization is shown. **`0` = off (the + default), `1` = on.** This is a client-side render gate, *not* a server guarantee — the + server does not withhold `GET /api/.../results/` based on it. A common "why don't I see + results?" cause. - `owner` → `users(uid)` — who created/owns it. -- Later migrations add `xid_required`, `treevite_enabled`, and `topics_enabled`. +- Later migrations add: + - `xid_required` — require an XID, the pseudonym mapping the authenticator's user ID to the + polis participant; used for authenticated conversations. + - `treevite_enabled` — `delphi`'s staggered "wave" invites (new/experimental). + - `topics_enabled` — `delphi` embeddings-based topic analysis (new/experimental). ### `zinvites` (`zid` ↔ `zinvite`) @@ -71,10 +79,13 @@ author. Key columns: - `active` — false hides the comment regardless of `mod`. - `is_seed` — seeded by the moderator (vs. submitted by a participant). - `is_meta` — a meta/notice comment, excluded from the math. -- `velocity` — moderation/ranking weight. +- `velocity` — a moderation/ranking weight used by the legacy Clojure comment-routing; the + newer `delphi` routing may not use it. Treat as legacy-leaning. `comments` has a foreign key on `(zid, pid)` into `participants`: a comment must be authored -by a known participant of that conversation. +by a participant that exists in that conversation. "Known" means only that a `participants` +row exists — it says nothing about *who* they are; identifying info exists only if the +organizer enabled auth/XID or demographic collection. ### `votes` and `votes_latest_unique` (keyed by `(zid, pid, tid)`) @@ -135,16 +146,21 @@ The **math** (Clojure) and **delphi** (Python) services poll the vote data, run clustering, and write results back for the server and report clients to read. The schema side is small: +> **Caveat:** this describes the Clojure `math` service's output in Postgres. The newer +> `delphi` (Python) service computes considerably more and — by a deliberate architecture +> decision — stores most of it in **DynamoDB, not Postgres**. Some delphi code still reads +> parts of the Clojure `math` blob, but Postgres is no longer the complete results picture. + - **`math_main`** — the headline result: `data jsonb` holds the PCA/clustering object (consensus, opinion groups, comment stats — the structure documented in [docs/pca.md](../../docs/pca.md)), keyed by `(zid, math_env)`, with `math_tick` / `last_vote_timestamp` for cache-invalidation. - Supporting tables: `math_profile`, `math_ptptstats`, `math_bidtopid`, `math_ticks`, - `math_cache`, `math_exportstatus`, `math_report_correlationmatrix`, and `stats_per_comment`. + `math_cache`, `math_exportstatus`, and `math_report_correlationmatrix`. - **`reports` / `report_comment_selections`** — saved/curated report views over a conversation. -The server never computes these; it only reads them. An empty results endpoint usually -means either `vis_type = 0` (above) or that math hasn't produced output yet. +The server never computes these; it only reads them. If you don't see results in the UI it +usually means math hasn't produced output yet (or, client-side, `vis_type = 0` — above). --- @@ -164,11 +180,13 @@ they group as follows. Most need no day-to-day attention. > ⚠️ **Not "legacy" — still referenced by live server code.** Several tables that look > vestigial are actually written/read by current `server/src` SQL, so do not assume they are -> safe to drop: `stars`, `upvotes`, `trashes` (comment moderation actions), `page_ids` and -> `permanentcookiezidjoins` (embed / implicit-conversation + legacy permanent-cookie auth), -> `metrics` (client metrics insert), `email_validations`, and `site_domain_whitelist` -> (embed domain allow-list). (There is **no table named `permanent`** — the permanent-cookie -> data lives in `permanentcookiezidjoins` and in `participants_extended.permanent_cookie`.) +> safe to drop: `stars`, `upvotes`, `trashes` (comment moderation actions), `page_ids` +> (embed / implicit-conversation), `metrics` (client metrics insert), `email_validations`, +> and `site_domain_whitelist` (embed domain allow-list). (There is **no table named +> `permanent`** — permanent-cookie data lives in `participants_extended.permanent_cookie` +> and `permanentcookiezidjoins`. The latter appears only in the schema — no `server/src`, +> `math`, or `delphi` references found — so treat its status as unverified: don't assume it +> is live, but don't assume it's safe to drop either.) A handful of tables (Slack/Stripe/Canvas/LTI integrations, the waiting list, geolocation) were dropped by migrations 000004 / 000005 / 000007 and no longer exist. diff --git a/server/postgres/migrations/000000_initial.sql b/server/postgres/migrations/000000_initial.sql index 58415bf386..0a7dfa4c04 100644 --- a/server/postgres/migrations/000000_initial.sql +++ b/server/postgres/migrations/000000_initial.sql @@ -1,5 +1,10 @@ -- Copyright (C) 2012-present, The Authors. This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License, version 3, as published by the Free Software Foundation. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details. You should have received a copy of the GNU Affero General Public License along with this program. If not, see . +-- ⚠️ KEEP IN SYNC: server/docs/data-model.md is the developer orientation doc for this +-- schema. If you add/rename/drop a core table or change a documented convention +-- (per-conversation tid/pid triggers, epoch-millis timestamps, the raw vote-sign inversion, +-- vis_type, etc.), update that doc in the SAME PR — code and doc drift apart otherwise. + -- NOTE: use \d TABLENAME to see info, like indexes -- Intersting query examples: From d56138e460bde586b6717fb99f2825766ec3865c Mon Sep 17 00:00:00 2001 From: Lodewijk <32471480+lgelauff@users.noreply.github.com> Date: Mon, 8 Jun 2026 21:31:19 +0200 Subject: [PATCH 3/3] docs(server): correct permanent-cookie attribution MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The 'still referenced by live code' claim for permanentCookieZidJoins came from observing live permanent-cookie auth, but that path actually reads/writes participants_extended.permanent_cookie (server/src/participant.ts, getParticipantByPermanentCookie) — the permanentCookieZidJoins table itself is schema-only with no server/src|math|delphi references (legacy/unused). Make that distinction explicit instead of the earlier 'unverified' hedge. --- server/docs/data-model.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/server/docs/data-model.md b/server/docs/data-model.md index f582c45bbc..d6c92c8c4a 100644 --- a/server/docs/data-model.md +++ b/server/docs/data-model.md @@ -183,10 +183,11 @@ they group as follows. Most need no day-to-day attention. > safe to drop: `stars`, `upvotes`, `trashes` (comment moderation actions), `page_ids` > (embed / implicit-conversation), `metrics` (client metrics insert), `email_validations`, > and `site_domain_whitelist` (embed domain allow-list). (There is **no table named -> `permanent`** — permanent-cookie data lives in `participants_extended.permanent_cookie` -> and `permanentcookiezidjoins`. The latter appears only in the schema — no `server/src`, -> `math`, or `delphi` references found — so treat its status as unverified: don't assume it -> is live, but don't assume it's safe to drop either.) +> `permanent`**. Live permanent-cookie auth reads/writes +> **`participants_extended.permanent_cookie`** (`server/src/participant.ts`, +> `getParticipantByPermanentCookie`) — *not* the `permanentCookieZidJoins` table, which +> appears only in the schema (`000000_initial.sql`) with no `server/src`, `math`, or `delphi` +> references, i.e. legacy/unused.) A handful of tables (Slack/Stripe/Canvas/LTI integrations, the waiting list, geolocation) were dropped by migrations 000004 / 000005 / 000007 and no longer exist.