The University of Hong Kong & TCL AI Lab
Online Cardiac Monitoring (OCM) emerges as a compelling enhancement for the next-generation video streaming platforms. It enables various applications including remote health, affective computing, and deepfake detection. Yet the physiological information encapsulated in the video streams has been long neglected. In this paper, we present the design and implementation of CardioLive, the first online cardiac monitoring system in video streaming platforms. We leverage the naturally co-existed video and audio streams and devise CardioNet, the first audio-visual network to learn the cardiac series. It incorporates multiple unique designs to extract temporal and spectral features, ensuring robust performance under realistic streaming conditions. To enable the Service-On-Demand OCM, we implement CardioLive as a plug-and-play middleware service and develop systematic solutions to practical issues including changing FPS and unsynchronized streams. Extensive evaluations demonstrate the effectiveness of our system. We achieve a Mean Squared Error of 1.79 BPM error, outperforming the video-only and audio-only solutions by 69.2% and 81.2%, respectively. CardioLive achieves average throughput of 115.97 and 98.16 FPS in Zoom and YouTube. We believe our work opens up new applications for video stream systems. Code is available at https://github.com/aiot-lab/CardioLive.
To get started with CardioLive, follow these steps:
- Clone the GitHub repository:
git clone https://github.com/aiot-lab/CardioLive.git
cd CardioLive- Make sure you have installed uv for virtual environment, and then run
uv sync- Activate the virtual environment
source .venv/bin/activate- You need to specify a config file under
configs/. We have povided some examples. - Then you can run
python train.py -c <configs/your_config.json>.
|-- configs
| |-- cardionet_mmpd_0.8_mmpd_0.2.toml
| |-- cardionet_pure_0.8_pure_0.1_pure_0.1.toml
| |-- cardionet_ubfc_0.8_ubfc_0.2.toml
| |-- cardionet_vah_0.8_vah_0.1_vah_0.1.toml
| |-- config.py
| |-- example\ copy.toml
| |-- example.toml
| `-- __init__.py
|-- evaluate
| |-- evaluate_metric_base.py
| |-- evaluate_metric.py
| |-- __init__.py
| |-- loss.py
| |-- PhysformerLoss.py
| |-- __pycache__
| `-- weighted_loss.py
|-- fig
|-- helpers
| |-- convert4rppg_toolbox.py
| |-- dataloader_fct.py
| |-- __init__.py
| |-- map_device.py
| |-- model_fct.py
| |-- __pycache__
| |-- rppg_toolbox_config.py
| `-- utils.py
|-- model
| |-- cardio_audio_v4.py
| |-- cardionet_v3.py
| |-- cardionet_v4.py
| |-- cardio_video_v50.py
| |-- cardio_video_v5.py
| |-- dtf.py
| |-- __init__.py
| |-- sincnet.py
| |-- time_embeddings.py
| `-- utils.py
|-- pyproject.toml
|-- README.md
|-- trainer
| |-- cardioaudio_trainer.py
| |-- cardionet_trainer.py
| |-- cardionet_trainer_v2.py
| |-- cardiovideov4_trainer.py
| |-- cardiovideov6_trainer.py
| |-- deepphys_trainer.py
| |-- efficientphys_trainer.py
| |-- __init__.py
| |-- lstmaudio_trainer.py
| |-- physformer_trainer.py
| |-- physnet_trainer.py
| |-- rhythmformer_trainer.py
| |-- trainer_fct.py
| `-- tscan_trainer.py
|-- train.py
`-- uv.lock
@inproceedings{lyu2025cardiolive,
author = {Lyu, Sheng and Huang, Ruiming and Ji, Sijie and Rehman, Yasar Abbas Ur and Ma, Lan and Wu, Chenshu},
title = {CardioLive: Empowering Video Streaming with Online Cardiac Monitoring via Audio-Visual Learning},
year = {2025},
isbn = {9798400720352},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3746027.3755588},
booktitle = {Proceedings of the 33rd ACM International Conference on Multimedia},
pages = {8606–8615},
numpages = {10},
keywords = {audio-visual learning, middleware, multimedia applications},
location = {Dublin, Ireland},
series = {MM '25}
}
This project is licensed under the GNU General Public License v3.0 - see the LICENSE file for details.