feat: free-tier analytics fallback, baseline seeding, self-hosted AI Coach, Strava sync

README.md

@@ -33,32 +33,53 @@ When a batch lands, `ingest.ts` does four things in order:
    borrowed from the protocol package) and the IMU-bearing R10 are the ones that carry
    real signal.
 3. **Saves the raw bytes to R2** under `raw/{you}/{device}/{when}-{first}-{last}.txt`,
-   one frame per line. This is the bit I never throw away. Minute rollups get pruned after
-   90 days; the raw frames stay so they can be re-decoded forever.
+   one frame per line — the system of record for re-decoding. Retention is short and
+   layered: raw R2 objects expire at **14 days** (an R2 bucket lifecycle rule, matching
+   `RAW_RETENTION_DAYS` — the re-decode horizon), the per-minute `minute` table and the
+   device `events` table are pruned at **10 days**, and the *derived* tables (`daily`,
+   `sleep`, `baselines`) are **kept permanently** — that's what the long-window trend
+   metrics read from, so nothing needs re-decoding past 10 days.
 4. **Rolls everything into minutes.** `rollup.ts` buckets the decoded samples by
    `floor(ts/60)*60` and writes them to the `minute` table.
 
 The `minute` table is the one clever bit worth understanding before you touch anything.
 It doesn't store an average heart rate, it stores the running pieces: `hr_sum`, `hr_n`,
 `act_sum`, `act_n`, plus min/max. The upsert adds the new pieces onto whatever's already
-there. The reason is that uploads aren't clean. The phone retries, batches overlap, the
-same frame shows up twice. If I stored averages I'd corrupt them on every double-send.
-Storing sums means re-uploading the exact same data converges to the exact same answer.
-Idempotency for free. Don't break this.
+there. The reason is that uploads aren't clean. The phone retries and batches overlap, so
+a day's frames arrive split across many POSTs in no particular order. Storing running sums
+makes the minute totals **order- and partition-independent**: chop the same frames into any
+set of batches, in any order, and the minute rollups come out identical. If I stored
+averages instead, the split would corrupt them.
+
+One honest caveat, because it's easy to misread this as "duplicate-proof": the upsert is
+*additive*, so the **same frame delivered in two separate POSTs is counted twice.** This is
+not idempotent against duplicate sends on its own — it relies on the client deduping frames
+before they ever hit here. The app does exactly that: `raw_records` has the frame hex as a
+primary key (`INSERT OR IGNORE`) and only deletes a record after a 200, so each frame is
+POSTed once. Keep that contract; if you ever ingest from a client without it, dedup frames
+server-side before the rollup. Don't break this.
 
 Once minutes are written, the user gets flagged dirty (or pushed onto a queue if you've
 got the paid plan), and that's where ingest stops. The heavy math is deliberately not on
 the request path.
 
 ## Where the metrics actually get computed
 
-`analytics.ts` is the brain, and it runs on a cron . Two schedules,
-both in `wrangler.toml`:
+`analytics.ts` is the brain, and it runs on a cron. Two schedules, both in
+`wrangler.toml`:
 
-- **Every hour** (`runAnalytics` with a 3-day window): re-derive every dirty user. This
-  is the safety net, it catches anyone the queue missed.
-- **Every night at 3:30** (2-day window): a fuller pass, plus respiratory rate for anyone
-  who slept recently, plus pruning minute rows older than 90 days.
+- **Every 30 minutes** (`*/30`): a light sweep — re-derive every dirty user (their daily
+  numbers, sleep, incremental steps), and the moment a night actually finishes, kick off
+  that night's HRV. Cheap work only; the heavy R2 re-decodes are fanned out onto a queue,
+  one bounded `(user, day)` unit per consumer invocation.
+- **Every night at 3:30** (`30 3`): the backstop — re-decode HRV and respiratory rate
+  only for the recent nights still *missing* them (so a night is never decoded twice, and
+  a night the wake-time run left empty gets retried), true up steps, and prune minute rows
+  past their retention. Anything already computed is skipped.
+
+HRV is real now: the beat-to-beat R-R intervals live in the 1 Hz (V24) records, so
+`biometrics.ts` re-decodes them from the raw bytes in R2 (off the request path) to drive
+recovery, readiness, and HRV-based stress.
 
 `processUser` is where it happens for one person. It reads their minutes, pulls their
 baselines, and calls into the [analytics package](https://github.com/OpenStrap/analytics)
@@ -69,6 +90,12 @@ signal, the coach plan, stress, nocturnal heart. The results land in `daily`, `s
 actually move day to day instead of collapsing into one flat value, which was a real bug
 early on.
 
+The cross-day metrics that need long windows — training load (EWMA acute:chronic over
+7/28 days), Banister fitness/fatigue/form, Foster monotony, sleep regularity — are
+**seeded from the permanent `daily`/`sleep` tables**, not just the few days being
+recomputed. We never re-decode old days (raw is gone after 14 days); we reuse the
+already-derived rows, so a real 28-day ACWR or 14-night regularity actually exists.
+
 Every number comes back wrapped: a value, a unit, a confidence between 0 and 1, a tier,
 and a label. If the inputs aren't there, the value is `null` and the confidence is `0`. I
 would rather show you a dash than make something up. The whole project falls apart the
@@ -84,6 +111,9 @@ moment it starts inventing numbers, so it doesn't.
 | `decode.ts` | Hex frames into decoded samples |
 | `rollup.ts` | Decoded samples into per-minute buckets |
 | `analytics.ts` | The cron brain, `processUser` / `runAnalytics` |
+| `queue.ts` | The queue consumer: one bounded `(user, job, day)` unit per invocation, and the wake-time HRV trigger |
+| `biometrics.ts` | HRV (RMSSD/SDNN/LF-HF), recovery, stress, relative temp/SpO₂ — re-decoded from the R-R intervals in R2 |
+| `steps_imu.ts` | Step counting from the wrist accelerometer (incremental + nightly true-up) |
 | `query.ts` | The read endpoints: today, sleep, strain, trends, chart, history |
 | `daydetail.ts` | Single-day drill-downs (the strain curve, the hypnogram, the stress band) |
 | `history.ts` | Range aggregation and the calendar heatmap |
@@ -119,7 +149,7 @@ Admin stuff for when you run your own: `/admin/run-analytics`, `/admin/run-resp`
 ## The database
 
 Thirteen tables. The ones you'll care about: `minute` (the running-sum rollups, pruned at
-90 days), `daily` and `sleep` and `sessions` (the derived output, mostly JSON columns for
+10 days), `daily` and `sleep` and `sessions` (the derived output, mostly JSON columns for
 the structured bits like coach plans and HR zones), `baselines` (your resting HR, max HR,
 sleep need, the anchors everything else is measured against), and the auth trio (`users`,
 `otps`, `refresh_tokens`). Full DDL is in `src/db/schema.sql`, it's commented, go read it.