feat: free-tier analytics fallback, baseline seeding, self-hosted AI Coach, Strava sync#12
Open
shoshiiiin wants to merge 37 commits into
Open
feat: free-tier analytics fallback, baseline seeding, self-hosted AI Coach, Strava sync#12shoshiiiin wants to merge 37 commits into
shoshiiiin wants to merge 37 commits into
Conversation
ingest: return received/decoded/minutes_written so the client shows an honest count
Metrics hrv explorer
Metrics hrv explorer
Brings main up to the deployed backend: real AN-2554 step counting from R2, admin-controlled app_config (OTA update pointer + home-screen alert banner, public /app/status), and the steps refactor (pure pedometer → analytics, frameAccel → decode.ts, steps_imu.ts as a thin runner).
…rics - Sleep v2: sleep_periods table + /sleep/v2 + /day/v2/sleep (multi-period; additive, v1 untouched) - Step goal: users.step_goal + PATCH/GET /profile + /today - /day/strain now a PURE READ (cron precomputes strain_curve + HR stats) - Queues fan-out: cron enqueues per-(user) sweep + per-(user,day) heavy jobs; consumer runs one bounded unit/invocation (biometrics/resp/steps_full) - Incremental steps (settled-minute cursor) + nightly full true-up - Event-driven biometrics: sweep fires biometrics on freshly-detected sleep - Ingest marks dirty only; the 30-min cron is the single deduped enqueue point - Minute retention 90 -> 10 days - Admin: /admin/enqueue, /admin/run-steps; 14-day caps on R2 re-run routes - Migrations v6-v10
feat: sleep v2, step goal, queue fan-out scaling, event-driven biometrics
… storm (OpenStrap#6) Cost + correctness on the deployed analytics pipeline: - Nightly cron now enqueues biometrics/resp ONLY for (user,day) where hrv_rmssd/resp_rate IS NULL (last 2 days). Already-scored nights are skipped entirely (no redundant R2 re-decode vs the event-driven fire) — the biggest R2 cost cut — and this doubles as the retry for nights the event fire left null, fixing HRV that got permanently stuck pending. - queue.ts: event-driven fire skips if HRV already exists for that night, and stays capped at one heavy decode per night via bio_last_date. - 30-min sweep clears `dirty` optimistically for enqueued users so a persistently-failing sweep isn't re-enqueued every 30 min. - sleep_efficiency: a real 0% is no longer coerced to null ("no data"). Co-authored-by: abdulsaheel <abdulsaheel@users.noreply.github.com>
…inute) events is only read by /day/timeline (single day) and the app only requests the last 7 days, backed by minute data that's pruned at 10 days. So events older than the minute cutoff are never read — age them out together. Also mirrored in /admin/prune.
ACWR / monotony / Banister fitness model / fitness-trend / SRI were fed only the ~3-day recompute window, so 7/28/30-day windows were degenerate (ACWR≈null/1.0). Now seed those series from the permanent daily/sleep tables (fetch 90 days, reuse the rows already loaded for baselines) and offset idx so each day's trailing window includes real history. Intra-span gaps = rest days (strain 0); pre-account days are never fabricated. Forward-only, no re-decode. D1 reads ≈free at scale.
…d permanent) + trailing-window metrics seeded from durable history
- analytics history query now selects hr_max and feeds it as session_hr_max, so calcBaselines can use a real measured max instead of always falling to the Tanaka age formula (the measured-max path was dead). - biometrics: merge drivers via SQL json_patch of only the bio-owned keys, replacing the read-modify-write. Removes the TOCTOU clobber against a concurrent processUser sweep and drops a column from the read. - steps: compare-and-swap on the steps cursor so a concurrent nightly full recompute can't cause the incremental sweep to double-count the day. - trend: relabel sleep regularity "Sleep consistency" (not the Phillips SRI). - README: correct the minute-table idempotency claim (additive upsert is order/partition-independent but relies on the client's hex-PK frame dedup).
Backend should hold no domain knowledge — byte layouts belong in openstrap-protocol, sports-science math in openstrap-analytics, leaving the worker as pure orchestration (bindings, queues, cursors, row->Minute mapping). - Deletes src/decode.ts; its decoders now live in openstrap-protocol/ts/live.ts. - Repoints imports in ingest.ts, rollup.ts, steps_imu.ts to 'openstrap-protocol/ts/live'. steps_imu.ts stays in the backend by design: it is I/O orchestration (R2 list/get, D1 cursor CAS, daily.steps write); the pure pedometer math (calcSteps) already lives in openstrap-analytics. No behavior change.
…trap#8) Two instances of the same data-integrity bug: a job that re-derives a metric from R2 raw (14d retention, per-run object budget) writes the result into the PERMANENT daily row unconditionally — so a null/partial pass (budget exhausted, sparse data, or source expired) overwrites a previously-computed good value. biometrics.ts (HRV/recovery): - Budget-starved days no longer emit a null row (was: pushed null -> clobber). With TOTAL_OBJECTS=24/PER_DAY=12, multi-day runs funded only 2 days and nulled the rest, erasing real HRV. Now: skip (day keeps stored value; per-day onlyDate job recomputes with full budget). - Skip the daily write entirely when rmssd is null (nothing to write that wouldn't risk regressing a good night). - COALESCE(?, col) on the measured indices (hrv_rmssd/sdnn/lfhf/si/cv/temp/spo2) as defense-in-depth, matching the resp_rate guard already present. resp.ts (respiratory rate): - Only persist resp_rate/resp_conf when resp_rate != null. The band emits R21 only during live streaming, so most nights are null -> the unconditional write erased prior good values. COALESCE can't protect resp_conf (it's 0, not null, when absent), so the write is skipped instead. Audit of all D1 writes confirms these are the only two instances: sleep/strain/ steps read the minute table within a window shorter than minute retention, so the source always outlives the write and they can't hit this clobber. Verified: tsc --noEmit clean. Restored the affected dev account's 7 nights of HRV via the per-day path. Co-authored-by: abdulsaheel <abdulsaheel@users.noreply.github.com>
…inute, D1-only loadDayRr) + TTL read-cache + v11/v12 migrations
…alytics), cron isUserAwake ladder (wake_cron), close_day queue job (processUser + HRV-from-minute + cache invalidate), demote nightly to maintenance + retry-net, v13 cursor migration
… per-nap hypnograms on /day/v2/sleep (on-read, no migration)
…) — all minute-reading day endpoints now cached
…acks days >3d into gzipped R2 objects (put-before-delete), day-detail reads fall back to R2; cuts D1 storage + per-row prune-deletes + dodges the 10GB cap. Ingest/processUser paths unchanged (hot window).
…gzipped blob, RMW merge mirroring the old ON CONFLICT) replaces row-per-minute as the hot store; all readers (analytics/daydetail/query/workouts/biometrics/seed) + seal + prune go through minute_store. The D1 write-cost lever (~1,440→~1 write/day). v14 migration.
…ging loadMinutes/stageNight now carry MinuteRec.rr through to stageSleep so the autonomic deep/REM split (RMSSD) runs on /day/sleep and /day/v2/sleep. Pairs with the analytics stageSleep RR change. ops/ + wrangler.v*.toml stay local.
…ake-only detectSessions and autoCloseStaleWorkouts ran only inside the once-a-day close_day / every */10 tick, so an auto-detected workout didn't surface until the next wake and the frequent cron wasn't purely wake-detection. - new workouts.ensureTodayWorkouts(db,user): closes this user's forgotten LIVE workouts + (re)detects TODAY's auto sessions, mirroring processUser Pass-2's session block EXACTLY (delete auto-non-deleted, insert done/auto; manual/ edited/deleted untouched). THROTTLED via read_cache (~120s) so a burst of reads costs one detect, and only for users actively opening the app. - called on-read from listWorkouts, getSessions, and getDayStrain(today). - autoCloseStaleWorkouts now takes an optional userId scope (on-read = one user). - scheduled(): removed autoCloseStaleWorkouts from the every-*/10-tick path; the */10 cron now does ONLY runWakeLadder. Kept autoCloseStaleWorkouts on the nightly 30:3 tick as a safety net for users who never open the app. Validated on v2: /workouts + /sessions 200, detection runs + throttles (wkscan read_cache sentinel set), 0 false workouts on a non-workout night.
… drop steps_imu Steps had three code paths: a per-record r10Motion heuristic (-> minute.steps, fed the chart), the full AN-2554 (calcSteps in ingest_signals -> computed then DISCARDED), and an R2 recompute (steps_imu -> daily.steps at close, lagged all day). Collapse to one: AN-2554, counted once at ingest, live on read. - minute_store.writeBatch: minute.steps += ingest_signals.steps (AN-2554), additive (idempotent via edge hex-dedup). rollup no longer sums the r10Motion s.steps_inc. - processUser folds daily.steps = SUM(this day's minute.steps) into the close (no R2), so past days keep a permanent total after minutes prune. - reads serve TODAY live: getToday + getDayStrain(today) = SUM(minute.steps) from the hot day blob (zero R2); past days use the stored daily.steps. - REMOVED steps_imu.ts entirely + the 'steps_full' queue job + runStepsIncremental from close_day/sweep + the /admin/run-steps endpoint. No step work in any cron now. Validated on v2: /today + /day/strain return live steps (104) from minute.steps; tsc clean.
Ingest aggregates per-minute red/IR/temp ADCs (wrist-on) into the minute blob; biometrics_minute derives a RELATIVE red/IR SpO2 index (confidence- gated) + skin-temp index vs rolling EWMA baselines, feeds temp into illness. Zero extra R2 (reads the D1 minute blob). admin/enqueue forwards onset/wake for close_day verification. Validated end-to-end on v2.
…er + cache /day/sleep Drive the hypnogram, stage breakdown, duration, awake and efficiency from ONE source (analytics stageHypnogram — v1 Cole-Kripke method) so the graph and breakdown can't disagree; pass per-minute RR so the REM tiebreaker reclaims REM mislabeled as awake. Add the fractal sleep cycles (detectSleepCycles) to the response. Wrap getDaySleep in the TTL read-cache: compute the night ONCE, serve from read_cache (past day immutable, today <=60s, close clears it) — no more recomputing the hypnogram on every read.
…creen The RR REM-tiebreaker lived only in the on-read /day/sleep path, so the STORED sleep.duration_min (what /today reads) stayed HR-only — Today showed 4.9h while the Sleep screen showed 5h43m for the same night. processUser now runs the same stageHypnogram (RR tiebreaker) at close and stores its duration/efficiency/stages, so both screens read one consistent value. Verified: /today and /day/sleep both 343min.
…ways-on irregular - Cycle tracking: cycle_log table (migrate_v15) + opt-in users.track_cycle (migrate_v16); GET/POST/DELETE /cycle gated on explicit consent; biometric overlay. - Live HRV spot-check: POST /spotcheck decodes RR from posted live frames (realtimeRr) + timeDomainHrv (stateless). - Skin-temp + SpO2 registered in /trend; skin_temp added to /day/heart. - Irregular-rhythm served for the always-on on-screen card. - Profile GET/PATCH carry track_cycle.
feat: wake-triggered derivation + RR-from-minute HRV + TTL cache + D1-hot/R2-sealed storage
schema.sql: add minute_day + read_cache tables and analytics_cursor wake columns (sleep_phase/phase_since/last_close_date) so a fresh DB bootstraps fully from schema.sql alone; index minute_day(ymd) for the seal/prune scans; mark legacy minute table deprecated. Remove dead code: index.ts cron-sweep helpers (sendChunked/enqueueJobs/ enqueueJobDays/lastNDates) and ingest_signals encodeRr/decodeRr.
Decoders are validated, so raw re-decodability no longer earns its cost. Stop writing the per-POST raw/…txt object (the dominant R2 Class A cost); RAW_BUCKET is kept ONLY for the hot/seal tier (sealOldDays). Every surfaced signal now derives at ingest into the D1 minute blob: - HR/RR/HRV, steps, SpO2/temp already did. - PPG: ingest decodes the R21 green channel to a per-second mean (RIIV proxy) stored as minute green[]; resp is computed from D1 at the wake-close (respFromMinuteGreen), preferred over RSA-from-RR when confident. estimateResp core unchanged. Removed the R2 re-decode subsystem: biometrics.ts (runBiometrics) deleted, resp.ts R2 I/O removed, queue biometrics/resp jobs + maybeTriggerBiometrics gone, admin run-biometrics/run-resp removed, wipe-raw scoped to raw/ prefix. R2 Class A puts: ~1,440/day/user -> ~0 (seal only). Validated on isolated v3.
The D1-backed token bucket cost 1 D1 read + 1 D1 WRITE on every ingest POST to maintain a bucket that, at 60s batching, never throttled legitimate traffic (it fully refilled between posts). Swap to the Workers Rate-Limiting binding (env.RATE_LIMITER.limit) — in-memory, per-colo, no D1 I/O, no extra billing. Limit kept equivalent to the old 30/min (limit 30 / 60s). Guarded so the worker runs unthrottled if the binding isn't configured on an environment. Removes the last removable per-POST D1 write (→ 1 write/POST with the dirty-fix). Binding lives in wrangler.v3.toml (validation); add to v2/v1 tomls at cutover.
markUserDirty did an unconditional upsert per batch (~1,440 D1 row writes/ user/day) to set a flag that stays 1 from first ingest until the wake-close clears it. Add a conditional 'DO UPDATE SET dirty=1 WHERE dirty=0' so a row is written only on the real transition; steady-state batches read the PK and write 0 rows. ~1 dirty-write per wake-cycle instead of per-POST — cuts ingest D1 writes ~1/3 with identical observable state.
…on, incremental cron, calibration ledger; wire restlessness/daytime-HRV/desaturation/cycle-gate - ingest_signals: classify live minutes (Mannini features → classifyActivityWindow) → store one act_class label/minute (no raw samples) - minute_store: MinuteRec.act_class carried through writeBatch + readMinutes - detectSessions persistence (detectStoreDay + processUser): segments/detected_type/type_confidence/type_source; reconciliation skips manual/auto_live overlaps, never clobbers a confirmed/corrected type on re-derive - migrate_v17: sessions adds segments/detected_type/type_confidence/type_source - #D incremental cron (sweepWorkoutDetection): re-derive dirty users every */10 → auto-workouts surface app-closed, backdated to last data minute - POST /workout/:id/type (confirm/correct) + classifier accuracy in /workouts - GET /day/hrv (daytime ultradian HRV) - processUser: cycle-phase gating on calcAnomaly (track_cycle); calcRestlessness folded into sleep_stress JSON - biometrics_minute: resp as illness feature + cycle-phase gating; calcDesaturation into drivers - tsc clean
… break ingest (raw-first)
detectStoreDay and processUser both DELETE auto sessions before re-inserting,
and setWorkoutType leaves source='auto' — so a confirmed/corrected workout was
deleted on every re-derive (the */10 sweep, every close_day) and reset to the
model's guess, erasing the calibration ledger within ~10 min. The INSERT's
ON CONFLICT preserve-corrected clause never fired because the row was already
gone in the same batch transaction.
Exclude type_source IN ('confirmed','corrected') from both DELETEs so the
labelled row survives (ON CONFLICT then preserves type/confidence while
refreshing the model metrics), and add it to the overlap-skip set so a
baseline-drift start_ts shift can't insert a duplicate model row over the
user's label.
feat+perf: workout HAR engine & honest metrics, on the drop-raw-R2 ingest (1 write/POST)
…Coach, Strava sync Four self-contained improvements (details in PR): - Free-tier: derive each day INLINE when no ANALYTICS_Q queue is bound, so self-hosters on the Workers Free plan (queues are paid-only) actually get metrics instead of days that are marked closed but never computed. - Seed profile-prior baselines on a user's first run (uses seedBaselines from the analytics package). - Self-hosted AI Coach: OpenAI-compatible /coach/v1 backed by the Workers AI binding — no external provider/key, runs in the free AI allocation. - Strava bidirectional sync: OAuth + pull (activities incl. bike-computer rides) + push (workouts, deduped so rides are never duplicated), with a daily cron.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Four self-contained improvements (happy to split into separate PRs if you prefer).
1. Free-tier analytics fallback (the important one)
Cloudflare Queues require the Workers Paid plan. On the Free plan there's no
ANALYTICS_Qbinding, so the wake-ladder marked a day closed but theclose_dayjob was never enqueued → self-hosters on the free plan got no derived metrics. Fix: extractcloseDay()inqueue.tsand call it inline fromwake_cron.ts(runWakeLadder+retryStaleCloses) when the queue binding is absent. All D1/R2 work — safely under the free plan's 1000 Cloudflare-subrequest cap for a single user.wrangler.tomlships with the queue blocks commented out + a note on how to re-enable on Paid.2. Profile-prior baseline seeding
processUsernow does anINSERT OR IGNOREofseedBaselines(profile)priors on a user's first run, so strain/HR-zones use a personal resting/max HR from day 1 and the EWMA autonomic baselines warm-start. Depends on OpenStrap/analytics#7 (theseedBaselinesexport). Gated scores (recovery/stress) are untouched.3. Self-hosted AI Coach (
coach.ts)OpenAI-compatible
POST /coach/v1/chat/completions+GET /coach/v1/models, backed by the Workers AI binding — no external provider/key, runs in the free AI allocation. Maps Workers AI{response, tool_calls}→ OpenAI shape; tool-calling verified with@cf/meta/llama-3.1-8b-instruct-fast. Auth via aCOACH_KEYsecret. The app's BYOK AI-Coach just points its base URL at<backend>/coach/v1.4. Strava bidirectional sync (
strava.ts)OAuth (connect/callback/status/disconnect, token refresh), pull (activities incl. bike-computer rides →
strava_activities) and push (completed workouts → manual Strava activities, deduped against pulled rides so a Wahoo/Garmin ride is never duplicated), plus a daily cron sync. New tables inschema.sql. Gated onSTRAVA_CLIENT_ID/STRAVA_CLIENT_SECRETsecrets.Notes
wrangler.tomlkeeps the placeholderdatabase_id.