Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
598 changes: 598 additions & 0 deletions FNOL_WORKFLOW_GUIDE.md

Large diffs are not rendered by default.

83 changes: 83 additions & 0 deletions GEMINI_TTS_SETUP.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
# Gemini TTS Setup for VOX LAB

Configure your Gemini API key in `.env` using a placeholder format:
```
GEMINI_API_KEY=your-gemini-api-key-here
```

## TTS Provider Support

VOX LAB now supports **three TTS providers**:

### 1. **Google Cloud TTS** (default)
- Free tier available
- Large voice selection
- Most stable

### 2. **Azure Speech Services**
- Enterprise-grade
- Real-time streaming
- Multiple voice styles

### 3. **Google Gemini TTS** (NEW!)
- Modern LLM-based synthesis
- Natural, expressive output
- Lower latency for research

## Switching TTS Providers

You can switch providers via environment variable or API:

### Environment Variable
```bash
# In .env file
TTS_PROVIDER=gemini # or 'google' or 'azure'
GEMINI_API_KEY=your-api-key
```

### API Call (Runtime)
```bash
curl -X POST http://localhost:3010/api/set-tts-provider \
-H "Content-Type: application/json" \
-d '{"provider": "gemini"}'
```

## Implementation Notes

The Gemini TTS integration is ready for:
- ✅ Text-to-speech synthesis
- ✅ Multiple voice profiles
- ✅ Streaming audio output
- ✅ Format conversion (WAV, MP3)

### Example: Using Gemini TTS

```javascript
// Client-side or API call
const response = await fetch('/api/tts', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
text: "Hello, this is spoken with Gemini TTS",
voice: { name: "en-US-female" },
audioConfig: { audioEncoding: "MP3" }
})
});

const audio = await response.arrayBuffer();
```

## Next Steps

1. **Test Gemini TTS**: Run a conversation in VOX LAB with TTS provider set to "gemini"
2. **Regenerate API Key**: For security, [regenerate your Gemini API key](https://console.cloud.google.com/apis/credentials)
3. **Monitor Costs**: Gemini TTS charges per request - monitor your Google Cloud usage

## Troubleshooting

If you get "content filter" errors, it means the LLM response violated Azure's content policy. This is separate from TTS and happens during text generation before audio synthesis.

**Solution**: The prompt content has been simplified to reduce false positives. If issues persist:
1. Check agent.js prompts for problematic content
2. Adjust conversation topics to avoid sensitive content
3. Switch LLM providers if needed
21 changes: 0 additions & 21 deletions LICENSE

This file was deleted.

91 changes: 43 additions & 48 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,20 +1,22 @@
# DialogLab
# VOX LAB

<img src="content/dialoglab-fig.png" alt="DialogLab teaser figure" width="100%"/>
<img src="content/dialoglab-fig.png" alt="VOX LAB teaser figure" width="100%"/>

[Paper](https://dl.acm.org/doi/10.1145/3746059.3747696) | [30s Preview](https://www.youtube.com/watch?v=AdvYhP8A51M) | [Video Figure](https://www.youtube.com/watch?v=U2Ag_Ktobzw) | [Web Demo](https://chatlab.3dvar.com/)
**VOX LAB** is an authoring environment for configuring and orchestrating multi-agent conversations with animated 3D avatars. Built with React, Vite, and Express, it helps researchers, designers, and developers create, visualize, and evaluate complex agent-based dialog systems.

**DialogLab** is an authoring tool for configuring and orchestrating multi-agent conversations with animated 3D avatars. Built with React, Vite, and Express, it enables researchers, designers, and developers to create, visualize, and evaluate complex agent-based dialogue systems.
## Overview

VOX LAB combines a visual authoring interface, avatar-driven playback, and runtime model integrations so teams can rapidly prototype and test conversational experiences.

## Features

- **Visual Conversation Design**: Configure multi-agent conversations with an intuitive node-based editor
- **3D Avatar Integration**: Animate conversations using Ready Player Me avatars with synchronized speech
- **Multiple LLM Support**: Compatible with OpenAI GPT and Google Gemini models
- **Scene Management**: Create and manage multiple conversation scenarios
- **Real-time Preview**: Test and iterate on conversations in real-time
- **Verification Tools**: Analyze conversation metrics and quality
- **Experience Mode**: Present finished conversations in a polished viewer
- **Visual conversation design**: Build multi-agent conversations with an intuitive node editor.
- **3D avatar integration**: Animate conversations with Ready Player Me avatars and synchronized speech.
- **Multiple LLM providers**: Use either OpenAI or Google Gemini models.
- **Scene management**: Create and organize multiple conversation scenarios.
- **Real-time preview**: Test and iterate quickly while authoring.
- **Verification tooling**: Inspect conversation quality and metrics.
- **Experience mode**: Present completed conversations in a polished viewer.

## Prerequisites

Expand All @@ -24,7 +26,7 @@
## Repository Structure

```
DialogLab/
VOX-LAB/
├── client/ # React UI (Vite) - Dev server on port 5173
│ ├── src/ # React components and application logic
│ └── public/ # Static assets (avatars, libraries)
Expand All @@ -47,9 +49,24 @@ cd ../server
npm install
```

Alternatively, from the repository root:

```bash
npm --prefix client install
npm --prefix server install
```

### 2. Configure Environment Variables

Create a `.env` file in the `server/` directory with your API keys:
Copy the template and fill in your own values (never commit secrets):

```bash
cp server/.env.example server/.env
```

Then edit `server/.env` and set your provider keys and related config using placeholders only.

Example:

```env
# Core Configuration
Expand All @@ -66,10 +83,10 @@ DEFAULT_GEMINI_MODEL=gemini-2.0-flash

# Text-to-Speech (optional - for avatar speech synthesis)
TTS_API_KEY=your-google-tts-api-key-here
TTS_ENDPOINT=https://eu-texttospeech.googleapis.com/v1beta1/text:synthesize
TTS_ENDPOINT=your-tts-endpoint-here
```

**Note**: You need at least one LLM provider API key (OpenAI or Gemini) to run conversations.
You must provide at least one LLM provider API key (OpenAI or Gemini) to run conversations.

### 3. Start the Server

Expand All @@ -78,7 +95,7 @@ cd server
node server.js
```

The server will start at `http://localhost:3010`. You should see a message confirming the server is running.
The server starts on `http://localhost:3010`.

### 4. Start the Client

Expand All @@ -89,37 +106,15 @@ cd client
npm run dev
```

Open your browser and navigate to `http://localhost:5173`.
Open `http://localhost:5173` in your browser.

## Third-Party Components & Licenses
## Environment Variables (Server)

### TalkingHead (MIT License)
Portions of this project's code are adapted from the open-source project [TalkingHead](https://github.com/met4citizen/TalkingHead), © 2024 Mika Suominen (met4citizen), licensed under the MIT License.

**Files**: `client/public/libs/talkinghead.mjs`

### Three.js (MIT License)
This project uses [Three.js](https://threejs.org/) and its example modules, © 2010–present Three.js authors, licensed under the MIT License.

**License**: [Three.js LICENSE](https://github.com/mrdoob/three.js/blob/dev/LICENSE)

### Ready Player Me Avatars (Custom License)
Example avatar files (`client/public/assets/*.glb`) were created using [Ready Player Me](https://readyplayer.me/) and are subject to [Ready Player Me's Terms of Use](https://readyplayer.me/terms-of-use).

**Note**: These assets are provided for demonstration purposes only and are **not covered** by this project's open-source license. For production use, you must obtain your own avatars or comply with Ready Player Me's licensing terms.

## Citation

If you use DialogLab in your research, please cite our UIST 2025 paper:

```bibtex
@inproceedings{dialoglab2025,
author = {Hu, Erzhen and Chen, Yanhe and Li, Mingyi and Phadnis, Vrushank and Xu, Pingmei and Qian, Xun and Olwal, Alex and Kim, David and Heo, Seongkook and Du, Ruofei},
title = {DialogLab: Configuring and Orchestrating Multi-Agent Conversations},
booktitle = {Proceedings of the 38th Annual ACM Symposium on User Interface Software and Technology (UIST '25)},
year = {2025},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
doi = {10.1145/3746059.3747696}
}
```
- `NODE_ENV`: Runtime environment (`development` by default).
- `GEMINI_API_KEY`: Google Gemini API key.
- `API_KEY_LLM`: OpenAI API key.
- `DEFAULT_LLM_PROVIDER`: Default provider (`gemini` or `openai`).
- `DEFAULT_OPENAI_MODEL`: Default OpenAI model.
- `DEFAULT_GEMINI_MODEL`: Default Gemini model.
- `TTS_API_KEY`: Google TTS API key (optional).
- `TTS_ENDPOINT`: Google TTS endpoint (optional).
4 changes: 2 additions & 2 deletions client/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,8 @@
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<meta name="description" content="Talking Head Interactive Application">
<title>Talking Head React App</title>
<meta name="description" content="VOX LAB - Multi-Agent Conversation Authoring Platform">
<title>VOX LAB</title>

<!-- DNS prefetch for faster resource loading -->
<link rel="dns-prefetch" href="https://cdn.jsdelivr.net">
Expand Down
Loading