What
The video path carries frame order (sequential positions) but not absolute time: an 8-frame clip spanning 2 s and one spanning 60 s both map to frame indices 0..7, so the model has no signal for elapsed time (duration, "what happens at 0:30"). Embed each frame's actual presentation time (seconds) and add it to that frame's visual tokens.
Scope
kempnerforge/data/video_io.py, video_dataset.py — carry per-frame timestamps.kempnerforge/model/frame_time.py (new FrameTimeEmbedding), model/vlm.py, distributed/parallel.py.scripts/train.py — pass frame_times into the forward.
Backward compatibility
Yes. Video-only (built when frames_per_clip > 1); image/text paths unchanged. Zero-initialized, so identity at step 0. No config fields added or changed.
What
The video path carries frame order (sequential positions) but not absolute time: an 8-frame clip spanning 2 s and one spanning 60 s both map to frame indices 0..7, so the model has no signal for elapsed time (duration, "what happens at 0:30"). Embed each frame's actual presentation time (seconds) and add it to that frame's visual tokens.
Scope
kempnerforge/data/video_io.py,video_dataset.py— carry per-frame timestamps.kempnerforge/model/frame_time.py(newFrameTimeEmbedding),model/vlm.py,distributed/parallel.py.scripts/train.py— passframe_timesinto the forward.Backward compatibility
Yes. Video-only (built when
frames_per_clip > 1); image/text paths unchanged. Zero-initialized, so identity at step 0. No config fields added or changed.