Visual Simulation Unity

This repository provides a Visual Scene Simulation module implemented in Unity for generating realistic synthetic fisheye camera images. It is part of a larger audio-visual software architecture used for simulating indoor scenes with cameras and microphones.

Audio-Visual Sensor Fusion Architecture

Features

• Simulates fisheye cameras using real-world intrinsic and extrinsic parameters.

• Supports multiple fisheye projection models:

  • Kannala-Brandt (KB8) Equidistant Projection
  • Brown-Conrady Projection

• Real-time rendering of synthetic fisheye images with GPU-accelerated post-processing.

• Scene simulation with:

  • Virtual cameras positioned according to real-world coordinates.
  • Virtual microphones at predefined 3D positions.
  • Animated 3D human models for realistic movement simulation.

• Data streaming to external applications over TCP/IP.

• Generates scene descriptions in JSON format for use by other software modules.

Repository Structure

VisualSimulationUnity/
├── Assets/                             # Unity project assets
│   ├── RGBD_Camera/                    # Simulate RGB and Depth Camera
│   │   ├── PhysicalCameraSimulator.cs  # Simulate RGB or Depth Camera
│   │   ├── TcpDataSender.cs            # Init TCP Sender
│   │   └── RGBD_Capture_And_Send.cs    # Send RGB + D images via TCP
│   │       └── BarrelDistortionEffect/ # Simulate Fisheye Projection with fragment shader
│   │           ├── KannalaBrandt_KB8_EquidistantProjection.shader
│   │           └── BrownConradyProjection.shader
│   └── Speaker_3D_Assets/               # 3D human models and animations
└── README.md                  

Generated Scenario Folder Structure

Each simulation session generates a structured output folder as follows:

📁 experiment_001/
├── config.json
├── groundtruth_sources.jsonl
├── audio/
│   └── wav/
│       └── multichannel_audio_<timestamp>.wav
├── video/
│   ├── rgb/
│   │   └── RGB_frame_<timestamp>.png
│   └── depth/
│       └── Depth_frame_<timestamp>.png
├── localization/
│   ├── audio_localizations.jsonl
│   └── video_localizations.jsonl
└── tracking/
    ├── audio_tracking.jsonl
    ├── video_tracking.jsonl
    └── audio_video_tracking.jsonl

Dependencies

• Unity 6000.1.6f1
• Optional: 3D human models from RenderPeople

Installation

1. Install Unity

2. Clone the repository:

   git clone https://code.fbi.h-da.de/est/est-workgroup/visualsimulationunity.git

3. Open the project in Unity Hub.

4. Ensure all required assets and packages are imported automatically by Unity.

Usage

1. Open the main scene in Assets/Scenes/.

2. Attach the PhysicalCameraSimulator.cs script to the virtual camera.

3. Configure TCP/IP connection in TcpDataSender.cs if streaming to external applications.

4. Run the scene to start simulation. The simulation will:

   • Render synthetic fisheye camera images.
   • Stream images via TCP/IP.
   • Export the current 3D speaker positions to groundtruth_sources.jsonl.
   • Update at a fixed frequency (default 10 Hz).

Contact

Created by: Laurens Sillekens