feat(gesture): real-time gesture recognition + dashboard integration#7
Open
SanskaarUndale21 wants to merge 3 commits into
Open
feat(gesture): real-time gesture recognition + dashboard integration#7SanskaarUndale21 wants to merge 3 commits into
SanskaarUndale21 wants to merge 3 commits into
Conversation
- Add /dashboard page with feature cards for Lip Reading and Gesture Recognition — only accessible when logged in (redirects to /login if not) - Login/signup now redirects to /dashboard instead of home (both useEffect redirect and Google callbackUrl) - Remove gesture button from Hero landing page — it belongs behind auth Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Restore full dashboard: sidebar with profile/history, top bar, 3-card idle screen (Live Webcam · Upload Video · Gesture Detection) - Gesture Detection card navigates to /gesture page - /dashboard page.tsx passes session user to Dashboard component - GestureCapture: back link goes to /dashboard instead of home Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- llm.py: wrap chat() in asyncio.wait_for(timeout=60) — without this the LLM hangs forever and the result message is never sent to the browser - llm.py: move fallback outside both except blocks so TimeoutError doesn't silently return None - llm.py, vsr.py, main.py: try relative import first, fall back to bare — works both as `uvicorn backend.main:app` (package) and direct `python main.py` - backend/__init__.py: ensure package marker exists Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This PR adds a gesture-recognition entry point and integrates it into a restored, session-oriented dashboard UI, while also improving backend reliability (package-relative imports and LLM timeout handling).
Changes:
- Introduces a new authenticated
/dashboardpage with sidebar session history, idle “mode selection” cards, and integrated webcam/upload flows (plus a navigation entry to gesture detection). - Adds new dashboard UI components for webcam capture, upload area, loading screen, and results popup (including TTS).
- Improves backend robustness by switching to package-relative imports (with fallbacks) and adding an LLM request timeout + fallback behavior.
Reviewed changes
Copilot reviewed 15 out of 15 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| frontend/src/components/Hero.tsx | Removes the prior home-page gesture link (gesture entry moves into the dashboard). |
| frontend/src/components/GestureCapture.tsx | Updates the back link to return to /dashboard. |
| frontend/src/components/dashboard/WebcamCapture.tsx | New webcam capture UI that streams frames and controls recording. |
| frontend/src/components/dashboard/UploadArea.tsx | New upload UI for video-file inference. |
| frontend/src/components/dashboard/types.ts | Adds shared types for results + session history. |
| frontend/src/components/dashboard/ResultPopup.tsx | New results modal with N-best list and text-to-speech controls. |
| frontend/src/components/dashboard/LoadingScreen.tsx | New loading/progress animation screen. |
| frontend/src/components/dashboard/HistoryPanel.tsx | New sidebar with user profile, sign-out, and selectable session history. |
| frontend/src/components/dashboard/Dashboard.tsx | New main dashboard orchestrator: WS connection, modes, persistence, and routing to /gesture. |
| frontend/src/components/AuthFlow.tsx | Redirects post-login to /dashboard (and updates Google sign-in callback). |
| frontend/src/app/dashboard/page.tsx | New authenticated dashboard route (redirects unauthenticated users to /login). |
| backend/vsr.py | Switches config import to package-relative with fallback. |
| backend/main.py | Switches backend imports to package-relative with fallback; adjusts gesture-module import handling. |
| backend/llm.py | Adds asyncio.wait_for timeout and consistent fallback for LLM correction. |
| backend/init.py | Adds package marker file so backend can be imported as a package. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+33
to
+41
| useEffect(() => { | ||
| if (show) { | ||
| setTimeout(() => setMounted(true), 10); | ||
| setShowCandidates(false); | ||
| } else { | ||
| setMounted(false); | ||
| if (speaking) { window.speechSynthesis?.cancel(); setSpeaking(false); } | ||
| } | ||
| }, [show, speaking]); |
Comment on lines
+11
to
+13
| export default function LoadingScreen() { | ||
| const [stageIdx, setStageIdx] = useState(0); | ||
| const [bars] = useState(() => Array.from({ length: 32 }, () => Math.random())); |
Comment on lines
+54
to
+57
| # Gesture recogniser — lazy import (mediapipe may be in a separate env) | ||
| import sys as _sys, pathlib as _pl | ||
| _sys.path.insert(0, str(_pl.Path(__file__).parent.parent)) | ||
| try: |
Comment on lines
+39
to
+44
| <img | ||
| src={user.image} | ||
| alt={user.name ?? ''} | ||
| referrerPolicy="no-referrer" | ||
| style={{ width: 34, height: 34, borderRadius: '50%', border: '1px solid var(--fg-4)', flexShrink: 0 }} | ||
| /> |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
/ws/gestureWebSocket endpoint — streams JPEG frames, returns per-frame gesture JSON via MediaPipe Hands (30+ gestures)/gesturepage with live camera overlay showing gesture name, confidence, hand label, and finger-state dotsasyncio.wait_for(timeout=60s)+ fallback on TimeoutErrorimport config→ relative imports so backend works as a package (uvicorn backend.main:app)/dashboardTest plan
/dashboardwith sidebar + 3 cards/gesturepage, camera starts, gestures detected live🤖 Generated with Claude Code