One link, one place: a logo hub linking to the pitch, the full story, and the interactive Qwen Memory Agent demo.
Biomimetic sonar for indoor navigation, designed for visually impaired users. Inspired by bat CF-FM echolocation. Built for the Global AI Hackathon Series with Qwen Cloud — Track: MemoryAgent, persistent adaptive memory across multi-turn, cross-session interactions.
There are millions of people who navigate the world without sight. A white cane tells you what's at arm's length. A guide dog tells you what's ahead — if you're lucky enough to have one, afford one, and live somewhere that trains them. Almost nobody has real-time knowledge of open space, walls, doorways, corners, and corridors before they get there.
Bats solved this problem 50 million years ago. They emit a chirp, listen to the echo, and build a picture of the world from sound alone — in real time, while flying, in total darkness. REBOUND borrows that idea and puts it in a phone: emit a chirp, capture the echo, run it through signal processing and a neural classifier, and tell the person what's around them — out loud, in their language, before they touch it with a cane.
This is not a finished product. It's a hackathon build, made in days, not years. But the idea underneath it — that echolocation plus an adaptive memory agent could give a blind person a few extra seconds of certainty in an unfamiliar space — is real, and worth building further.
rebound-olga.vercel.app — the deployed story page, end to end: from the human problem, to bat-inspired echolocation, to the Qwen Memory Agent that remembers each user.
Interactive demo → /demo — the ML part, live: run a session and watch the Memory Agent learn. The Bayesian priors adapt, episodic memory fills and decays, and semantic patterns ("hesitates in doorways", "confident in open space") consolidate on screen — the real deterministic memory, in the browser, no backend needed.
▶ Watch the walkthrough video — the Memory Agent learning, end to end.
The Memory Agent after a live session: Bayesian priors adapted per space class, 26 episodes stored, and 6 behavioural patterns consolidated — open space and corridors learned as confident, doorways and corners as hesitant. Real deterministic memory, computed in the browser.
 |
 |
 |
 |
 |
 |
 |
 |
- Emits a CF-FM chirp (8 kHz constant tone + downward sweep) through the phone's speaker.
- Captures the returning echo through the microphone, in a latency-aware window that survives Android's variable output delay.
- Deconvolves the echo into a Room Impulse Response (adaptive Wiener filter).
- Extracts acoustic features — mel spectrogram (64×32), MFCCs (13×32), RT60, spectral centroid.
- Classifies the space with a lightweight PyTorch CNN: open space, nearby wall, doorway, corner, corridor, or stairs.
- Runs an independent two-pass staircase detector as DSP reinforcement — stairs are a different kind of danger than a wall, and don't get to rely on the CNN alone.
- A Qwen-powered Memory Agent takes that classification, remembers what it has learned about this user across sessions, and generates a spoken navigation instruction personalized to how this person hesitates, advances, and reacts.
- Adaptive Bayesian priors mean the system gets quieter about things the user has already proven they handle well, and more careful about the things that have tripped them up before.
- Feedback reaches the user through four independent channels — earcons (rhythm and melodic contour, not just pitch), synthesized speech, vibration, and ARIA live regions for VoiceOver/TalkBack — because a navigation aid that only works when the screen is visible has already failed its user.
Mobile (PWA) Alibaba Cloud ECS
┌──────────────┐ ┌──────────────────────────────────┐
│ Web Audio API │── audio buffer ──▶│ POST /predict │
│ getUserMedia │ (f32) + sr │ ├─ Wiener deconvolution (RIR) │
│ speaker emit │ │ ├─ Feature extraction │
│ │ │ ├─ CNN classifier (6 classes) │
│ DeviceOrienta │── gyro pitch ────▶│ ├─ Stair detector (DSP) │
│ tionEvent │ │ └─ predict() result │
│ │◀── JSON ─────────│ │
│ earcons · │ instruction │ POST /process │
│ speech · │ distance_m │ ├─ Qwen-Plus Memory Agent │
│ haptics · │ class_name │ ├─ Episodic/Semantic memory │
│ ARIA live │ stair_message │ └─ User profile update │
└──────────────┘ └──────────────────────────────────┘
│
DashScope API (Qwen-Plus)
One scan end to end:
REBOUND started deployed in Singapore. It worked. It also had almost a full second of round-trip latency for every scan — noticeable, and for a navigation tool where "noticeable" can mean "walked into it before the warning arrived," that latency wasn't a footnote. It was the whole problem.
We didn't guess at the cause. We measured it: a plain curl to /health —
an endpoint that does nothing but return a JSON blob, no model, no Qwen
call — was already taking over a second round-trip. That ruled out the LLM
as the bottleneck and pointed straight at geography. So we did the
unglamorous thing: stood up a second ECS instance from scratch in
US-Virginia, migrated the whole stack — Docker image, model checkpoint,
Qwen credentials, Caddy reverse proxy with a real Let's Encrypt certificate
via DuckDNS — and pointed the domain at it.
The latency dropped hard. Not because the code got smarter, but because we stopped assuming "cloud" means "close enough," and went and checked.
Along the way we also found and fixed a second, subtler problem: /predict
— the single most CPU-expensive endpoint in the system — was running its
signal-processing pipeline synchronously inside the async event loop. Two
people scanning at once weren't just competing for CPU; they were fully
blocking each other. We moved the pipeline to a thread pool behind a bounded
semaphore, so it now degrades gracefully (429, retry) instead of
serializing invisibly. That fix mattered more for concurrent judges testing
the live demo than any latency the region migration alone could have solved.
We treat Alibaba Cloud as more than a place to point a Docker container. Real deployment means real hardening, and it means being honest when something is slow instead of hand-waving it as "cloud latency, nothing we can do."
Not a claim — receipts. The stack ran on Alibaba Cloud ECS, first in Singapore and then migrated to US-Virginia, with the FastAPI backend serving live inference over HTTP.
 |
 |
 |
 |
launch-advisor-20260701): the nginx reverse proxy active (running).2. ECS console — the running instance in US-Virginia (
ecs.c9i.large, 2 vCPU / 4 GiB, public IP 47.85.198.176).3. The REBOUND FastAPI backend live on that instance — Uvicorn on :8000, "Application startup complete", serving real
POST /predict and /process requests.4. A continuous stream of live classifications (open space, nearby wall, corner) at ~10–25 ms each — every request HTTP 200 OK.
The Memory Agent (src/memory/agent.py) calls Qwen-Plus through the
DashScope international endpoint (dashscope-intl.aliyuncs.com/compatible-mode/v1,
OpenAI-compatible chat/completions), both synchronously (QwenMemoryAgent) and
asynchronously (AsyncQwenMemoryAgent via httpx.AsyncClient, non-blocking for the
FastAPI server). Every call is forced to response_format: json_object against a
fixed schema and retried up to 3 times with exponential backoff.
Qwen is the reasoning brain of the memory system, not a chat add-on. Given the CNN's acoustic classification, the current sonar features, the user's episodic/semantic memory, and their behavioral signal (advance / hesitate / retreat / ignore), it decides in a single structured call:
- The navigation instruction to speak to the user
- Confidence adjustments to the Bayesian priors per space class
- Memory operations — what to store, consolidate, or selectively forget across episodic/semantic/procedural memory
- The haptic pattern to trigger
When DASHSCOPE_API_KEY is unset, MockQwenMemoryAgent/MockAsyncQwenMemoryAgent
generate deterministic rule-based responses with the same shape, so the pipeline runs
identically offline for testing and demos.
The backend (src/cloud/api_server.py) is deployed on Alibaba Cloud ECS
(region US-Virginia) via Docker, with a Caddy reverse proxy and a real Let's
Encrypt certificate, exposing endpoints for audio prediction, observation
processing, user profiles, and session management. Every Qwen response is guarded
against hallucination before it touches state: confidence multipliers clamped to
[0.1, 10.0], memory key/value sizes capped, and space-class names checked against
a whitelist before they ever reach a prompt or get applied.
We built and trained the classifier ourselves. ReboundCNN
(src/models/classifier.py) is a custom multi-task CNN written and trained from
scratch on our own NVIDIA RTX 3090 — not a fine-tuned off-the-shelf model. A
single forward pass produces both the space class and the distance estimate, and
the trained checkpoint (models/checkpoints/best_model.pt) ships in this repo.
| Metric | Value |
|---|---|
| Architecture | Custom multi-task CNN — 3 conv blocks (1→16→32→64, Conv2d 3×3 + BatchNorm + ReLU + MaxPool) into a shared FC trunk with two heads |
| Inputs | Mel spectrogram (1×64×32) fused with 2 acoustic scalars — RT60 and spectral centroid |
| Outputs | Class head (6-way softmax) + distance head (regression, Softplus) |
| Parameters | ~296K — small enough to run on CPU, no GPU at inference |
| Loss / optimizer | CrossEntropy + MSE (multi-task), Adam @ lr 1e-3, ReduceLROnPlateau, early stopping on val loss |
| Training run | 50 epochs, batch size 64, on an NVIDIA RTX 3090 |
| CNN classes | 6 (open_space, nearby_wall, doorway, corner, corridor, stairs) |
| val_accuracy (20% stratified split) | 0.953 |
| val_distance_MAE | 0.302 m |
| Tests passing | 88/88 |
| Staircase detector SNR threshold | ~27 dB |
Mel spectrograms extracted from the synthetic dataset — one sample per class, showing the acoustic signature the CNN classifier learns to distinguish:
All figures above are measured on simulated rooms — see Known limitations. Full empirical detail: LIMITATIONS.md.
| Method | Path | Description |
|---|---|---|
POST |
/predict |
Raw audio in → classification + distance + stair detection |
POST |
/process |
Classification result → Memory Agent → navigation instruction |
GET |
/profile/{user_id} |
User profile, episodic stats, semantic memory |
POST |
/session |
Start new session for user |
GET |
/health |
Health check |
All endpoints except /health require an X-API-Token header when
REBOUND_API_TOKEN is set (see Security).
- No spatial direction. REBOUND can tell you there's a wall a meter away. It cannot yet tell you if that wall is to your left, your right, or straight ahead. That needs a stereo microphone array to triangulate direction — a hardware constraint, not a software one. A single phone microphone can't resolve it.
- The training dataset is synthetic, generated from acoustic simulation, not from real rooms with real people walking through them. It works well enough to demo convincingly. It is not yet trained on the messiness of real-world echoes, real hallways, real furniture.
- It still needs a phone in your hand. For someone who already navigates with a cane and possibly a guide dog, adding "also hold a phone up and tap it" is a real usability cost we haven't solved yet.
- Behavioral feedback loop — now demonstrated end to end in the interactive
demo. The full Bayesian personalization pipeline — EpisodicMemory,
SemanticMemory, UserProfile, and the Qwen Memory Agent — is real and tested,
and the interactive web demo (
web/, live at rebound-olga.vercel.app/demo) now runs it in the browser: across a session of realuser_actions the Bayesian priors adapt, episodic memory fills and decays, and semantic patterns consolidate on screen — the same deterministic math assrc/memory/*.py, client-side, with no backend and no API key (Qwen only phrases the already-computed result). This is the ML personalization finally exercised as a loop, not just unit-tested in isolation. What remains is capturing the user's real reaction on the phone: the mobile client (Live.jsx) still hardcodesuser_action: "advance", so the on-device loop isn't closed in the field yet. Integration work remaining, not a capability the system lacks.
None of these are hidden from the judges because none of them are embarrassing. They're the honest next steps for a project built in days, not the finish line for a project that's done. Full empirical constraints, measured rather than assumed: LIMITATIONS.md. The adversarial security review we ran against our own system, findings and fixes both: AUDITORIA_ADVERSARIAL.md and CHANGELOG_2026-07-01.md.
The most exciting fix for all three limitations above is the same one: a dedicated IoT device. A small wearable with a stereo microphone pair (solving spatial direction), an onboard low-power inference chip (cutting round-trip latency further), and no phone required at all (solving the "hands are busy" problem cane users already live with). Once the dataset moves from synthetic to real recordings — ideally collected in partnership with actual blind or low-vision users — the CNN classifier's confidence and the Memory Agent's personalization both improve with it, because the whole system is only as honest as the data it learned from.
- Real-world dataset — field RIR collection to close the sim-to-real gap documented above.
- On-device personalization — Elastic Weight Consolidation fine-tuning
per user (
future/ewc.py, already prototyped, deferred to v2). - Edge inference — 296K parameters is small enough to run offline when there's no signal.
pip install -r requirements.txtSet your DashScope API key:
export DASHSCOPE_API_KEY=your_key_hereRecommended for production — a shared demo token (see Security):
export REBOUND_API_TOKEN=$(openssl rand -hex 16)
export REBOUND_ALLOWED_ORIGINS=https://rebound-olga.duckdns.orgCUDA-capable GPU required for training. CPU is sufficient for inference and for the live demo.
The repo is self-contained. A fresh clone already ships the trained model
(models/checkpoints/best_model.pt) and the processed dataset
(data/processed/), so you can run inference and the API server without
regenerating anything — the training step below is only for retraining from
scratch. The one thing to bring is your own DASHSCOPE_API_KEY for Qwen; the
/demo UI and the deterministic memory run with no key at all.
Generate dataset and train:
python3 -m src.simulation.dataset_builder
python3 -m src.models.train --data data/processed/dataset.npz --epochs 50Run the local demo (mock agent):
python3 -m src.demo.adaptive_demoRun the local demo (real Qwen API):
python3 -m src.demo.adaptive_demo --use-qwenStart the API server locally:
uvicorn src.cloud.api_server:app --host 0.0.0.0 --port 8000Deploy to Alibaba Cloud ECS:
python3 -m src.cloud.deploy --registry <your-acr-registry> --tag v0.1Run tests:
python3 -m pytest tests/ -vsrc/
signal/ chirp, capture, deconvolution, staircase detector
features/ spectral and geometric feature extraction
simulation/ RIR generation (pyroomacoustics + synthetic stairs)
models/ CNN classifier (6 classes), training, inference
memory/ Qwen Memory Agent (sync + async), user profile, episodic/semantic memory
cloud/ FastAPI API server (async) + Alibaba Cloud ECS deploy script
feedback/ haptic pattern definitions
demo/ adaptive real-time visualization (matplotlib)
frontend/ React PWA (Vite) — sonar engine, earcons, speech, live view
future/
ewc.py Elastic Weight Consolidation — deferred to v2
augmentation.py noise augmentation (known bug, see header) — deferred to v2
tests/ 88 unit + integration tests
LIMITATIONS.md empirically measured system constraints
AUDITORIA_ADVERSARIAL.md adversarial security audit (Peircean abductive method)
CHANGELOG_2026-07-01.md bug remediation changelog
REBOUND handles behavioral data about people with disabilities, so we hold it to a real security bar, not a demo-day shrug:
- Token authentication on every data endpoint (
X-API-Token, constant-time comparison) — the server logs loudly on startup if no token is configured, so "silently open" is never silent. - CORS scoped to the deployed frontend origin — no wildcard + credentials in production.
- Payload size caps and bounded concurrency on
/predict: oversized audio is rejected before decoding, and excess concurrent requests get a clean429instead of degrading every request in flight. - LRU eviction with a hard cap on per-user state, so no amount of
invented
user_ids can grow server memory without bound. user_idvalidated against[a-zA-Z0-9_\-]{1,64}(path traversal prevention).class_namevalidated against a fixed whitelist before reaching any prompt (prompt-injection prevention).- LLM-generated multipliers clamped to
[0.1, 10.0](hallucination containment); semantic memory values capped at 512 characters (unbounded growth prevention). - Per-user
asyncio.Lock(race-condition prevention) with heavy DSP/CNN work off the event loop, so concurrent users no longer block each other. .dockerignoreexcludesdata/profiles/anddata/checkpoints/.- Deployed over HTTPS via Caddy with a real Let's Encrypt certificate — not self-signed, because iOS and Android both refuse microphone access otherwise.
We ran an adversarial audit against our own system before polishing the demo and published every finding: see AUDITORIA_ADVERSARIAL.md.
Python, FastAPI, PyTorch, librosa, Qwen Cloud (qwen-plus), Docker, Alibaba Cloud ECS, Caddy, DuckDNS, React (Web Audio API, PWA)
Designed, architected, and developed in full by Olga Vasilieva, with infrastructure and deployment support from the VIGÍA AI Collective.
The GitHub repository is hosted under the account of Anna Tchijova (her daughter) for technical convenience. Anna Tchijova had no participation in the code, design, or technical decisions of this project.
License: Apache License 2.0 — see LICENSE
Qwen Cloud Hackathon · MemoryAgent Track
Every phone is a sonar.
