StudentOS

An AI-powered student command center that unifies schedules, deadlines, email, calendar, and notes to tell college students what to do next, generate study tools, and draft emails.

Built for hackUMBC 2026, targeting the "Main Track" and "Best Use of Gemini API" prize.

Team:

• Dara Adedeji
• Ricky Chen
• Israel Ogwu

Features

• Smart Dashboard - three view modes: urgency-ranked cards, compact list, and an interactive 3D graph visualization
• AI Chat Advisor - conversational assistant with tool calling (searches tasks, courses, Gmail) and WebRTC real-time voice
• Study Tools - upload lecture note photos to generate flashcards and quizzes via AI vision
• Multi-source Task Ingestion - pulls tasks from Gmail, Google Calendar, Blackboard, and screenshot uploads, then deduplicates and ranks by urgency

Demo

Tech Stack

• Frontend: Next.js 15, React 19, Tailwind CSS, shadcn/ui
• Backend: Convex (real-time database + serverless functions)
• AI: Google Gemini 2.5 Flash (primary), OpenAI (fallback + voice), Ollama (local fallback)
• Google Cloud: Vertex AI support via Application Default Credentials - set GOOGLE_CLOUD_PROJECT to route all Gemini calls through Vertex AI instead of AI Studio, with configurable region (GOOGLE_CLOUD_LOCATION, defaults to us-east4). Google OAuth connects Gmail and Calendar APIs for live academic data.
• Auth: NextAuth with Google OAuth
• LMS Sync: Python CLI tool for Blackboard

How It Works

Reverse-Engineering Blackboard

UMBC doesn't expose a public API for Blackboard. To pull real academic data, we built a Python CLI tool that extracts session cookies from the Zen browser's encrypted session storage (Mozilla's LZ4-compressed recovery.jsonlz4), then replays authenticated requests against Blackboard's internal REST endpoints:

• /learn/api/v1/users/me - current user identity
• /learn/api/v1/users/me/memberships - enrolled courses
• /learn/api/public/v1/calendars/items - due dates and deadlines
• /learn/api/v1/courses/{id}/announcements - course announcements
• /learn/api/v1/courses/{id}/contents - course materials

The raw responses are normalized and pushed to Convex DB, which acts as middleware - making course data accessible from anywhere in the app with real-time fetches, reactive subscriptions, and multi-tenant isolation by user.

Gemini with Tool Calling

The AI chat advisor runs on Gemini 2.5 Flash with a function-calling loop (up to 6 rounds per request). Gemini can autonomously invoke tools to answer student questions:

• search_gmail - queries the student's inbox via Gmail API with search operators and date filters
• search_calendar / create_calendar_event - reads upcoming events or creates new ones with reminders via Google Calendar API
• get_academic_snapshot - pulls active tasks, courses, and announcements from Convex
• search_tasks / get_course_context - filters and drills into specific courses or assignments
• generate_study_material - creates flashcards or practice quizzes on the fly

Each tool call result feeds back into the conversation, so Gemini can chain lookups - e.g., check Gmail for a professor's email, cross-reference Calendar for the exam date, then create a study plan.

OpenAI Voice AI Agent

The voice mode uses OpenAI's Realtime API over WebRTC for low-latency, conversational voice interaction. The client establishes a peer connection with server-side Voice Activity Detection (VAD), enabling natural turn-taking and mid-speech interruption.

A state machine manages the full voice lifecycle: idle → starting → ready → user-speaking → processing → assistant-speaking, with support for muting and error recovery. All 7 tools are available during voice sessions - when the student asks about their schedule or tasks, the assistant silently executes tool calls and speaks the results back naturally.

The voice UI renders a reactive glass orb with particle animations driven by real-time audio amplitude, giving visual feedback for both user and assistant speech.

Getting Started

npm install

Run both dev servers simultaneously:

npm run dev           # Next.js (Turbopack)
npx convex dev        # Convex backend (in a second terminal)

Seed Data

Seed the Convex database with demo courses, tasks, and announcements:

npm run seed

Safe to rerun - replaces existing data for the seed user (anonymous by default).

Blackboard Sync

npm run blackboard:sync    # Sync Blackboard data to Convex
npm run blackboard:export  # Export only (skip Convex import)

Environment Variables

GOOGLE_CLOUD_PROJECT      # GCP project ID for Vertex AI (optional)
GOOGLE_CLOUD_LOCATION     # Vertex AI region (default: us-east4)
GEMINI_API_KEY            # Gemini API key - only if not using Vertex AI
OPENAI_API_KEY            # OpenAI (chat fallback + WebRTC voice)
OPENAI_MODEL              # Chat model (default: gpt-4o-mini)
OPENAI_REALTIME_MODEL     # Voice model (default: gpt-4-realtime)
GOOGLE_CLIENT_ID          # Google OAuth
GOOGLE_CLIENT_SECRET      # Google OAuth
NEXTAUTH_SECRET           # NextAuth JWT signing
NEXTAUTH_URL              # Default: http://localhost:3000
NEXT_PUBLIC_CONVEX_URL    # Convex deployment URL
CONVEX_DEPLOYMENT         # Convex deployment name
OLLAMA_BASE_URL           # Default: http://localhost:11434
OLLAMA_TEXT_MODEL          # Default: llama3.2
OLLAMA_VISION_MODEL       # Default: llava

License

MIT