feat: DAG Viewer in Rill Cloud #8716
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The OLAPListTables API returns paginated results with a nextPageToken field. Previously only the first page was fetched, causing materialized models and tables on subsequent pages to show as having no size data. This change replaces the derived store pattern with a readable store that: - Fetches all pages sequentially for each connector - Detects nextPageToken in responses and creates follow-up queries - Accumulates all tables before building the final size map - Handles loading/error states correctly This fixes the issue where 5 models were detected but only 2 tables (from the first page) were returned, causing models like "auction_data_model" to show "-" instead of their actual size. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
The issue was that useModelTableSizes was creating a new store on every render when resources changed. If the resources loaded before the queries completed, the component would be unmounted/remounted, cancelling the queries. This change: 1. Caches stores by instanceId:connectorArray to prevent recreation 2. Eagerly subscribes to each store to keep queries alive 3. Uses WeakMap to prevent memory leaks from old subscriptions 4. Ensures queries complete even if component re-renders quickly Now when you refresh the status page, queries stay alive in the background and complete asynchronously, updating the table when ready. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
The store cache was keeping stale subscriptions alive across page refreshes. When you refreshed the page, the old cached store would be returned even if the resources had changed or needed fresh data. This change simplifies the approach: - Create fresh stores on each useModelTableSizes call - Let TanStack Query handle result caching (HTTP level) - Ensure queries always run with latest connector data - No issues with premature garbage collection Now on page refresh, sizes will load correctly immediately. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
The issue was that columns were defined as a static const, so the accessor function captured the initial tableSizes reference and never updated when tableSizes changed. Changes: 1. Made columns reactive with $: so they update when tableSizes changes 2. Added console logging to ProjectResources and ProjectResourcesTable to track when stores and columns are updated 3. Added logging to selectors to see preload and store update timing This should fix the issue where the table shows "-" on initial load but correctly shows sizes after navigating away and back. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
The core issue: columns were defined as a static const, capturing the initial tableSizes reference. When the store updated with new data, the column accessor functions still referenced the old empty Map. Solution: Make columns reactive with $: so they recreate whenever tableSizes changes, ensuring accessor functions get the current data. This allows the UI to update asynchronously as size data arrives. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
The row count fetching was failing with 401 errors because mutations weren't being triggered. Fixed by: - Explicitly calling .mutate() on row count mutations instead of just subscribing - Removing ProjectTablesRowCounts component with manual JWT waiting logic - Leveraging TanStack Query's built-in JWT handling via httpClient interceptor - Using mutation state subscription for proper success/error handling The key insight: createQueryServiceQuery returns a mutation store that requires explicit .mutate() calls to execute—just subscribing to it does nothing. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
createQueryServiceQuery is a query creator, not a mutation creator. It needs: 1. Query parameters (instanceId, queryServiceQueryBody) as first argument 2. Query options with enabled: true as third argument 3. No .mutate() call - queries auto-execute when subscribed The query store will automatically handle JWT auth through httpClient interceptor. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
- Simplified row count state checking to match column count pattern - Added process flag to ensure counts only increment once - Added console logging to diagnose query state issues - Check for state.data?.data directly instead of state.isSuccess 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
- Log all state properties: isLoading, isFetching, isSuccess, isError, data, error, failureReason - Try multiple paths to find row data: direct array, nested .data, or .results - Better error handling to catch failureReason This will help identify where the actual row count data is stored in the query response. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Replace TanStack Query mutation wrapper with direct async httpClient function: - queryServiceQuery(instanceId, queryBody) returns a Promise - httpClient handles JWT auth automatically via interceptor - Simpler async/await pattern instead of store subscriptions - Removed complex state checking logic This approach is cleaner and properly leverages the built-in auth interceptor. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
The 401 errors were happening because queryServiceQuery was running before the JWT token was available in the runtime store. Now we: - Import runtime store and get function - Wait for JWT token to be populated (up to 5 seconds) - Only make the query once JWT is ready This mirrors the pattern from ProjectTablesRowCounts.svelte but keeps the async/await approach integrated in the selector. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
The 401 errors were happening because async row count fetching in the store setup was racing with JWT initialization. Now: 1. Export fetchRowCount(instanceId, tableName) from selectors.ts - Uses httpClient which has built-in JWT waiting via maybeWaitForFreshJWT - Returns Promise<number | 'error'> 2. Create ProjectTablesRowCounts.svelte component - Uses Svelte reactive statements ($:) to wait for JWT - Only fetches when: JWT ready AND tables available - Tracks fetched tables to avoid duplicates 3. Update ProjectTables.svelte - Import and use ProjectTablesRowCounts component - Pass rowCounts from component to ProjectTablesTable - Component-level approach guarantees JWT is ready This mirrors the original working pattern but uses httpClient which has proper JWT sequencing built in. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Instead of relying on httpClient's JWT handling, manually: - Get runtime store to access JWT token and host - Build full URL with host from runtime state - Add Authorization header with Bearer token directly - Add detailed logging for debugging 401 errors This eliminates any abstraction layers and makes JWT handling explicit. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Manual JWT handling was causing 401 errors because the JWT in the store wasn't being properly validated or refreshed. Using httpClient instead leverages the built-in maybeWaitForFreshJWT interceptor which: - Waits for fresh JWT when needed - Handles token refresh automatically - Properly validates tokens with the backend This eliminates manual JWT token management. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
https://www.loom.com/share/f4fb3e2596e1410faa9fea952f3a6875
ideally this would land after:
And still needs e2e.
Checklist: