feat: frame-based DFA transition detection for Gemini vision by NSang22 · Pull Request #5 · NSang22/PatchLab

NSang22 · 2026-02-22T07:26:12Z

Rewrite chunk_processor to send individual JPEG frames instead of video
Gemini now acts as a DFA transition function: d(current_state, frame) -> next_state
Prompt includes current state, expected progression, and per-state descriptions
Add process_frames() and _call_gemini_frames() to gemini_client
Delete demo HTML game (patchlab/game/)

- Rewrite chunk_processor to send individual JPEG frames instead of video - Gemini now acts as a DFA transition function: d(current_state, frame) -> next_state - Prompt includes current state, expected progression, and per-state descriptions - Add process_frames() and _call_gemini_frames() to gemini_client - Delete demo HTML game (patchlab/game/)

gitguardian · 2026-02-22T07:26:17Z

⚠️ GitGuardian has uncovered 1 secret following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

🔎 Detected hardcoded secret in your pull request

GitGuardian id	GitGuardian status	Secret	Commit	Filename
27469882	Triggered	Google API Key	`92a5000`	patchlab/backend/.env	View secret

🛠 Guidelines to remediate hardcoded secrets

Understand the implications of revoking this secret by investigating where it is used in your code.
Replace and store your secret safely. Learn here the best practices.
Revoke and rotate this secret.
If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider

following these best practices for managing and storing secrets including API keys and other credentials
install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.

^{🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.}

- Snowflake DB defaults: PLAYPULSE -> PATCHLAB_DB - Resolve merge conflict in snowFlakeClient.py - Update comments, CSS headers, run scripts

Copilot

Pull request overview

This PR implements a frame-based DFA transition detection system for Gemini Vision, replacing video chunk uploads with sequential JPEG frame submission. The changes shift from holistic video analysis to explicit frame-by-frame state tracking.

Changes:

Rewrites chunk_processor.py to extract and send individual JPEG frames instead of video chunks
Introduces Gemini as a DFA transition function: d(current_state, frame) -> next_state with per-state visual cues
Adds process_frames() and _call_gemini_frames() methods to gemini_client.py

Reviewed changes

Copilot reviewed 14 out of 9226 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
`patchlab/backend/gemini_client.py`	Adds frame-based processing methods alongside video processing for DFA state detection
`patchlab/backend/chunk_processor.py`	Implements frame extraction with gaze overlay and frame-by-frame Gemini prompting
`patchlab/backend/.env`	Exposes production Gemini API key in version control
`frontend/src/patchlab.css`	Adds scoped CSS styles for dashboard components
`frontend/src/pages/ProjectSetup.jsx`	Implements DFA state editor UI with Mario preset
`frontend/src/components/WhatIsIt.jsx`	Landing page feature explanation component
`frontend/src/components/Navbar.jsx`	Navigation component for landing page
`frontend/src/components/FAQ.jsx`	FAQ accordion component for landing page
`frontend/src/components/DashboardLayout.jsx`	Dashboard navigation and layout wrapper
`frontend/src/components/Dashboard.jsx`	Demo dashboard with fake data visualization
`frontend/index.html`	HTML entry point for React app
`final_idea.md`	Documentation update with merge conflict markers and MediaPipe details
`.vite/deps/package.json`	Vite dependency metadata
`.vite/deps/_metadata.json`	Vite build cache metadata

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-02-22T07:28:00Z

@@ -0,0 +1,2 @@
+MOCK_MODE=false
+GEMINI_API_KEY=AIzaSyB55LF79IS4w2Pe_iv2zDzjVDpij40JvBc


API key exposed in version control. This file should be excluded from git tracking via .gitignore and keys should be rotated immediately. Committing credentials creates a permanent security vulnerability in the repository history.

Suggested change

GEMINI_API_KEY=AIzaSyB55LF79IS4w2Pe_iv2zDzjVDpij40JvBc

GEMINI_API_KEY=YOUR_GEMINI_API_KEY_HERE

Copilot · 2026-02-22T07:28:00Z

+        image parts. This gives Gemini explicit frame-by-frame sequential
+        context for detecting state transitions (DFA transition function style).
+
+        Uses MODEL_CHUNK_ANALYSIS (gemini-2.5-flash).


Documentation states 'gemini-2.5-flash' but the model constant used is MODEL_CHUNK_ANALYSIS. The comment should reference the actual model being used or clarify that it resolves to gemini-2.5-flash.

Copilot · 2026-02-22T07:28:00Z

+    # Find nearest gaze reading by timestamp
+    closest = min(
+        gaze_data,
+        key=lambda g: abs(g.get("timestamp_sec", 0.0) - frame_time)


Using min() with a lambda creates O(n) comparisons for each frame. For large gaze_data lists processed repeatedly, consider pre-sorting gaze_data by timestamp_sec and using binary search (bisect module) to find the nearest timestamp in O(log n) time.

Copilot · 2026-02-22T07:28:01Z

+{
+  "type": "module"
+}


Vite build artifacts (.vite/deps/) should be excluded from version control via .gitignore. These files are generated during development and can cause conflicts between different environments.

Copilot · 2026-02-22T07:28:01Z

+  "hash": "f9c0af56",
+  "configHash": "e09577a6",
+  "lockfileHash": "e3b0c442",
+  "browserHash": "96c930e4",
+  "optimized": {},
+  "chunks": {}


Vite build cache files should be excluded from version control via .gitignore. These are environment-specific artifacts that change frequently and provide no value in the repository.

Suggested change

"hash": "f9c0af56",

"configHash": "e09577a6",

"lockfileHash": "e3b0c442",

"browserHash": "96c930e4",

"optimized": {},

"chunks": {}

- Replace chunk_processor.py with OpenCV frame extraction + DFA prompt (Gemini now processes individual JPEG frames as inline images, acting as a DFA transition function δ(current_state, frame) → next_state) - Add process_frames() and _call_gemini_frames() to gemini_client.py - Upgrade MODEL_CHUNK_ANALYSIS from gemini-2.0-flash to gemini-2.5-flash - Update main.py _process_chunk_bg and /chunks endpoint for new model fields (state_name, entered_at_sec, end_state, etc.) - Add opencv-python and google-genai to requirements.txt - Remove orphaned patchlab/backend/ files (edits were in wrong directory)

- Fix merge conflict markers in header - Rewrite section 4A to explain dual-frame DFA transition function - Update JSON response format to match new model fields - Document inline JPEG parts approach vs video upload - Update tech stack and implementation checklist

chore: replace all PlayPulse references with PatchLab

9cca8a8

- Snowflake DB defaults: PLAYPULSE -> PATCHLAB_DB - Resolve merge conflict in snowFlakeClient.py - Update comments, CSS headers, run scripts

NSang22 requested a review from Copilot February 22, 2026 07:26

Copilot AI reviewed Feb 22, 2026

View reviewed changes

DanielJZambrano added 2 commits February 22, 2026 02:40

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: frame-based DFA transition detection for Gemini vision#5

feat: frame-based DFA transition detection for Gemini vision#5
NSang22 wants to merge 4 commits into
mainfrom
Daniel

NSang22 commented Feb 22, 2026

Uh oh!

gitguardian Bot commented Feb 22, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Feb 22, 2026

Uh oh!

Copilot AI Feb 22, 2026

Uh oh!

Copilot AI Feb 22, 2026

Uh oh!

Copilot AI Feb 22, 2026

Uh oh!

Copilot AI Feb 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		@@ -0,0 +1,2 @@
		MOCK_MODE=false
		GEMINI_API_KEY=AIzaSyB55LF79IS4w2Pe_iv2zDzjVDpij40JvBc

	GEMINI_API_KEY=AIzaSyB55LF79IS4w2Pe_iv2zDzjVDpij40JvBc
	GEMINI_API_KEY=YOUR_GEMINI_API_KEY_HERE

Conversation

NSang22 commented Feb 22, 2026

Uh oh!

gitguardian Bot commented Feb 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ GitGuardian has uncovered 1 secret following the scan of your pull request.

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Feb 22, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 22, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 22, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 22, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 22, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

gitguardian Bot commented Feb 22, 2026 •

edited

Loading