Reduce GUN sync-loop churn and high-volume writes across frontend sync paths#4
Reduce GUN sync-loop churn and high-volume writes across frontend sync paths#4Copilot wants to merge 3 commits into
Conversation
Agent-Logs-Url: https://github.com/theEndless11/decentralised/sessions/8b490b38-664b-42ac-b504-12397fd0436e Co-authored-by: thegoodduck <163307030+thegoodduck@users.noreply.github.com>
Agent-Logs-Url: https://github.com/theEndless11/decentralised/sessions/8b490b38-664b-42ac-b504-12397fd0436e Co-authored-by: thegoodduck <163307030+thegoodduck@users.noreply.github.com>
Agent-Logs-Url: https://github.com/theEndless11/decentralised/sessions/8b490b38-664b-42ac-b504-12397fd0436e Co-authored-by: thegoodduck <163307030+thegoodduck@users.noreply.github.com>
|
FUCK the test. dont have time for this |
There was a problem hiding this comment.
Pull request overview
This PR reduces GunDB sync churn and write amplification across the frontend by narrowing subscriptions, batching replay writes, and hardening degraded-relay write paths (notably around DM/chat and user/profile updates).
Changes:
- Reduce high-volume Gun write patterns (batching snapshot import + MySQL warmup replay, consolidating comment/poll metadata writes, field-level profile counter writes).
- Reduce subscription churn and listener leakage (root
.on()→map().on(), unsubscribe support, interval cleanup, deduped community emits). - Stabilize DM/chat sync during
chats/*→v3/chats/*migration (merged history/unread state, debounced/batched read markers, duplicate listener prevention).
Reviewed changes
Copilot reviewed 15 out of 15 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| src/views/copilot-views.md | Updates HomePage documentation to reflect merged legacy + v3 DM preview behavior. |
| src/views/HomePage.vue | Merges chat room discovery/listeners across chats/* and v3/chats/*, dedupes message emits, and prevents duplicate room subscriptions. |
| src/stores/copilot-stores.md | Documents batching of MySQL warmup writes back into Gun. |
| src/stores/communityStore.ts | Replays MySQL warmup writes to Gun in small batches with pauses to reduce startup spikes. |
| src/services/userService.ts | Serializes per-user writes, switches counters to field-level writes, adds timeout-bounded put + verification. |
| src/services/snapshotService.ts | Adds batched Gun write replay during snapshot import to avoid sync storms. |
| src/services/postService.ts | Switches global post live subscription from root .on() to map().on() to reduce full-root rescans. |
| src/services/pollService.ts | Embeds signature/encryption metadata into initial poll shell write; removes poll cache warm-up writes. |
| src/services/gunService.ts | Makes GunService.subscribe() return an unsubscribe function. |
| src/services/copilot-services.md | Updates service docs to reflect new subscription cleanup and reduced write amplification behaviors. |
| src/services/communityService.ts | Dedupes repeated community re-emits using a signature of mapped community payload. |
| src/services/commentService.ts | Consolidates comment creation into a single object write and adds timeout + read-back verification before indexing/bumping counts. |
| src/services/chatService.ts | Mirrors messages/read markers across legacy + v3 chat roots; debounces and batches readAt updates. |
| src/composables/useChainSync.ts | Clears the polling interval on unmount to prevent background timer leaks across remounts. |
| src/composables/copilot-composables.md | Documents the useChainSync() interval cleanup behavior. |
| setTimeout(() => { | ||
| const entry = chatList.value.find(c => c.userId === otherUserId); | ||
| if (entry) entry.unreadCount = unread; | ||
| if (entry) entry.unreadCount = unreadIds.size; | ||
| chatList.value = [...chatList.value].sort((a, b) => b.lastMessageTime - a.lastMessageTime); | ||
| }, 300); |
There was a problem hiding this comment.
The inner setTimeout(..., 300) in recomputeUnread isn’t tracked for cleanup on unmount (only the outer debounce timer is). That can run after unmount and do unnecessary work / retain closures. Consider storing these timers too (per-room) and clearing them in onUnmounted, or switching to a single debounced update per room.
| for (let index = 0; index < unreadIds.length; index += batchSize) { | ||
| unreadIds.slice(index, index + batchSize).forEach((msgId) => { | ||
| this.getRoomNodes(roomId).forEach((roomNode) => { |
There was a problem hiding this comment.
writeReadMarkers calls this.getRoomNodes(roomId) inside the per-message loop, recreating the room-node array for every unread message. Pull const roomNodes = this.getRoomNodes(roomId) out of the loop to avoid extra allocations/work when marking many messages as read.
| for (let index = 0; index < unreadIds.length; index += batchSize) { | |
| unreadIds.slice(index, index + batchSize).forEach((msgId) => { | |
| this.getRoomNodes(roomId).forEach((roomNode) => { | |
| const roomNodes = this.getRoomNodes(roomId); | |
| for (let index = 0; index < unreadIds.length; index += batchSize) { | |
| unreadIds.slice(index, index + batchSize).forEach((msgId) => { | |
| roomNodes.forEach((roomNode) => { |
| gunProgress++; | ||
| onProgress?.('gun', gunProgress, totalGun); | ||
| } | ||
| await this.writeGunEntriesInBatches(gunWrites); |
There was a problem hiding this comment.
onProgress and gunProgress are advanced while queuing Gun writes, but the actual .put() calls happen later in writeGunEntriesInBatches(). This can report 100% progress before any Gun writes have been replayed. Consider incrementing progress when each queued write runs (or after each batch flush) so the UI reflects real import progress.
| if (ack?.err) { | ||
| reject(ack.err); | ||
| return; | ||
| } |
There was a problem hiding this comment.
putNodeWithTimeout rejects with ack.err directly, which can be a string/object instead of an Error. This makes downstream error handling inconsistent (e.g., err.message). Wrap ack.err in new Error(...) (or normalize to a string) before rejecting.
| const seenMessageSignatures = new Map<string, string>(); | ||
| const existingEntry = chatList.value.find(c => c.userId === otherUserId); |
There was a problem hiding this comment.
seenMessageSignatures grows by one entry per message ID and is never pruned while HomePage stays mounted. In long-running sessions / large rooms this is unbounded memory growth. Consider capping the map (e.g., keep last N messageIds) or pruning entries for messages older than the newest timestamp seen.
This change targets the frontend patterns that can trigger GUN sync storms and warnings like “syncing more than 1k+ records per second”: broad root subscriptions, repeated listener recreation, and multi-write feedback paths that fan out across the graph. The update narrows subscriptions, reduces write amplification, and adds bounded verification around degraded-relay write paths so realtime behavior remains intact without turning transient lag into sync churn.
Write-path reductions
.put()calls into a single object write.Subscription and listener cleanup
.on()tomap().on()to avoid full-root rescans on unrelated updates.GunService.subscribe()return an unsubscribe function so callers can clean up persistent listeners.useChainSync()so repeated mounts do not leak background polling.CommunityServiceby signature-checking mapped payloads before notifying listeners.Counter and profile update hardening
UserServiceso local counter updates do not race each other.postCount,commentCount, andkarmaas field-level writes instead of rewriting the full profile object.DM / chat sync stabilization
markAsRead()and batch both the read scan and thereadAtwrites instead of issuing immediate per-message updates.chats/*and mirroredv3/chats/*roots to avoid split conversations during migration.readAtactually change.Degraded-relay safety
Example of the main write-amplification change: