Skip to content

Reduce GUN sync-loop churn and high-volume writes across frontend sync paths#4

Closed
Copilot wants to merge 3 commits into
masterfrom
copilot/research-gunjs-sync-loop-issues
Closed

Reduce GUN sync-loop churn and high-volume writes across frontend sync paths#4
Copilot wants to merge 3 commits into
masterfrom
copilot/research-gunjs-sync-loop-issues

Conversation

Copy link
Copy Markdown
Contributor

Copilot AI commented Apr 29, 2026

This change targets the frontend patterns that can trigger GUN sync storms and warnings like “syncing more than 1k+ records per second”: broad root subscriptions, repeated listener recreation, and multi-write feedback paths that fan out across the graph. The update narrows subscriptions, reduces write amplification, and adds bounded verification around degraded-relay write paths so realtime behavior remains intact without turning transient lag into sync churn.

  • Write-path reductions

    • Collapse comment creation from many field-level .put() calls into a single object write.
    • Fold encrypted poll metadata into the initial poll shell write instead of issuing follow-up field writes.
    • Batch snapshot-import Gun writes and MySQL post warmup replay writes to avoid bursty rehydration spikes.
    • Remove the unused poll cache warm-up helper that wrote data back into subscribed poll roots.
  • Subscription and listener cleanup

    • Switch all-post live subscriptions from root .on() to map().on() to avoid full-root rescans on unrelated updates.
    • Make GunService.subscribe() return an unsubscribe function so callers can clean up persistent listeners.
    • Add interval cleanup to useChainSync() so repeated mounts do not leak background polling.
    • Deduplicate repeated community re-emits in CommunityService by signature-checking mapped payloads before notifying listeners.
  • Counter and profile update hardening

    • Serialize per-user writes in UserService so local counter updates do not race each other.
    • Update postCount, commentCount, and karma as field-level writes instead of rewriting the full profile object.
    • Add timeout-bounded write verification for user/profile writes so late ACKs do not silently count as success without read-back confirmation.
  • DM / chat sync stabilization

    • Debounce markAsRead() and batch both the read scan and the readAt writes instead of issuing immediate per-message updates.
    • Merge DM history, unread state, and room discovery across legacy chats/* and mirrored v3/chats/* roots to avoid split conversations during migration.
    • Prevent duplicate Home feed DM listeners while still allowing same-message re-emits when fields like readAt actually change.
  • Degraded-relay safety

    • For comment creation, a timed-out write now requires short read-back verification before indexing the comment under the post or bumping counts, avoiding orphan comment refs from assumed success.

Example of the main write-amplification change:

// Before: many field-level writes
commentNode.get('id').put(commentId)
commentNode.get('postId').put(data.postId)
commentNode.get('authorId').put(data.authorId)
// ...

// After: one consolidated write
commentNode.put({
  id: commentId,
  postId: data.postId,
  communityId: data.communityId,
  authorId: data.authorId,
  authorName: data.authorName,
  content: comment.isEncrypted ? '🔒 Encrypted comment' : data.content,
  encryptedContent: comment.encryptedContent,
  authTag: comment.authTag,
  createdAt: timestamp,
  upvotes: 0,
  downvotes: 0,
  score: 0,
  edited: false,
})

Copilot AI and others added 3 commits April 28, 2026 15:01
Agent-Logs-Url: https://github.com/theEndless11/decentralised/sessions/8b490b38-664b-42ac-b504-12397fd0436e

Co-authored-by: thegoodduck <163307030+thegoodduck@users.noreply.github.com>
Agent-Logs-Url: https://github.com/theEndless11/decentralised/sessions/8b490b38-664b-42ac-b504-12397fd0436e

Co-authored-by: thegoodduck <163307030+thegoodduck@users.noreply.github.com>
Agent-Logs-Url: https://github.com/theEndless11/decentralised/sessions/8b490b38-664b-42ac-b504-12397fd0436e

Co-authored-by: thegoodduck <163307030+thegoodduck@users.noreply.github.com>
@thegoodduck
Copy link
Copy Markdown
Collaborator

FUCK the test. dont have time for this

@thegoodduck thegoodduck marked this pull request as ready for review April 29, 2026 15:07
Copilot AI review requested due to automatic review settings April 29, 2026 15:07
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR reduces GunDB sync churn and write amplification across the frontend by narrowing subscriptions, batching replay writes, and hardening degraded-relay write paths (notably around DM/chat and user/profile updates).

Changes:

  • Reduce high-volume Gun write patterns (batching snapshot import + MySQL warmup replay, consolidating comment/poll metadata writes, field-level profile counter writes).
  • Reduce subscription churn and listener leakage (root .on()map().on(), unsubscribe support, interval cleanup, deduped community emits).
  • Stabilize DM/chat sync during chats/*v3/chats/* migration (merged history/unread state, debounced/batched read markers, duplicate listener prevention).

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
src/views/copilot-views.md Updates HomePage documentation to reflect merged legacy + v3 DM preview behavior.
src/views/HomePage.vue Merges chat room discovery/listeners across chats/* and v3/chats/*, dedupes message emits, and prevents duplicate room subscriptions.
src/stores/copilot-stores.md Documents batching of MySQL warmup writes back into Gun.
src/stores/communityStore.ts Replays MySQL warmup writes to Gun in small batches with pauses to reduce startup spikes.
src/services/userService.ts Serializes per-user writes, switches counters to field-level writes, adds timeout-bounded put + verification.
src/services/snapshotService.ts Adds batched Gun write replay during snapshot import to avoid sync storms.
src/services/postService.ts Switches global post live subscription from root .on() to map().on() to reduce full-root rescans.
src/services/pollService.ts Embeds signature/encryption metadata into initial poll shell write; removes poll cache warm-up writes.
src/services/gunService.ts Makes GunService.subscribe() return an unsubscribe function.
src/services/copilot-services.md Updates service docs to reflect new subscription cleanup and reduced write amplification behaviors.
src/services/communityService.ts Dedupes repeated community re-emits using a signature of mapped community payload.
src/services/commentService.ts Consolidates comment creation into a single object write and adds timeout + read-back verification before indexing/bumping counts.
src/services/chatService.ts Mirrors messages/read markers across legacy + v3 chat roots; debounces and batches readAt updates.
src/composables/useChainSync.ts Clears the polling interval on unmount to prevent background timer leaks across remounts.
src/composables/copilot-composables.md Documents the useChainSync() interval cleanup behavior.

Comment thread src/views/HomePage.vue
Comment on lines 656 to 660
setTimeout(() => {
const entry = chatList.value.find(c => c.userId === otherUserId);
if (entry) entry.unreadCount = unread;
if (entry) entry.unreadCount = unreadIds.size;
chatList.value = [...chatList.value].sort((a, b) => b.lastMessageTime - a.lastMessageTime);
}, 300);
Copy link

Copilot AI Apr 29, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The inner setTimeout(..., 300) in recomputeUnread isn’t tracked for cleanup on unmount (only the outer debounce timer is). That can run after unmount and do unnecessary work / retain closures. Consider storing these timers too (per-room) and clearing them in onUnmounted, or switching to a single debounced update per room.

Copilot uses AI. Check for mistakes.
Comment on lines +429 to +431
for (let index = 0; index < unreadIds.length; index += batchSize) {
unreadIds.slice(index, index + batchSize).forEach((msgId) => {
this.getRoomNodes(roomId).forEach((roomNode) => {
Copy link

Copilot AI Apr 29, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

writeReadMarkers calls this.getRoomNodes(roomId) inside the per-message loop, recreating the room-node array for every unread message. Pull const roomNodes = this.getRoomNodes(roomId) out of the loop to avoid extra allocations/work when marking many messages as read.

Suggested change
for (let index = 0; index < unreadIds.length; index += batchSize) {
unreadIds.slice(index, index + batchSize).forEach((msgId) => {
this.getRoomNodes(roomId).forEach((roomNode) => {
const roomNodes = this.getRoomNodes(roomId);
for (let index = 0; index < unreadIds.length; index += batchSize) {
unreadIds.slice(index, index + batchSize).forEach((msgId) => {
roomNodes.forEach((roomNode) => {

Copilot uses AI. Check for mistakes.
gunProgress++;
onProgress?.('gun', gunProgress, totalGun);
}
await this.writeGunEntriesInBatches(gunWrites);
Copy link

Copilot AI Apr 29, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

onProgress and gunProgress are advanced while queuing Gun writes, but the actual .put() calls happen later in writeGunEntriesInBatches(). This can report 100% progress before any Gun writes have been replayed. Consider incrementing progress when each queued write runs (or after each batch flush) so the UI reflects real import progress.

Copilot uses AI. Check for mistakes.
Comment on lines +46 to +49
if (ack?.err) {
reject(ack.err);
return;
}
Copy link

Copilot AI Apr 29, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

putNodeWithTimeout rejects with ack.err directly, which can be a string/object instead of an Error. This makes downstream error handling inconsistent (e.g., err.message). Wrap ack.err in new Error(...) (or normalize to a string) before rejecting.

Copilot uses AI. Check for mistakes.
Comment thread src/views/HomePage.vue
Comment on lines +666 to +667
const seenMessageSignatures = new Map<string, string>();
const existingEntry = chatList.value.find(c => c.userId === otherUserId);
Copy link

Copilot AI Apr 29, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seenMessageSignatures grows by one entry per message ID and is never pruned while HomePage stays mounted. In long-running sessions / large rooms this is unbounded memory growth. Consider capping the map (e.g., keep last N messageIds) or pruning entries for messages older than the newest timestamp seen.

Copilot uses AI. Check for mistakes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants