What
The attentional_pool connector (per-window attention pooling) currently requires the patch grid be divisible by the pool window — it raises on ragged grids (require_divisible=True, adapter.py). That blocks faithful Molmo2 3×3 video pooling: a SigLIP2 @224/patch16 grid is 14×14, and 14 % 3 != 0, so 3×3 attentional pooling is rejected today (you have to fall back to avgpool or a divisible window).
avgpool already handles ragged grids (pad → mask → count-normalize over real patches only). Bring attentional_pool to parity: each partial edge window pools only its real patches via a masked attention (the padded patches are masked out of the per-window K/V). Backward-compatible — divisible grids (e.g. 2×2 on 14×14) are unchanged.
Scope
kempnerforge/model/adapter.py (AttentionalPoolAdapter.forward): for a ragged grid, pad to ceil(grid/w)·w, build a per-window key-padding mask (real vs padded patches), use the masked window-mean as the query, and pass the mask to SDPA so padded patches are excluded from each window's pooling. Remove the ragged-rejection raise. output_num_tokens → ceil(grid/w)². (No NaN guard needed — every ceil-window covers ≥1 real patch.)kempnerforge/model/adapter.py DIVISIBLE_ONLY_POOL_TYPES (consulted by config/adapter.py): drop attentional_pool so config-time + max_seq_len checks accept ragged.- Tests (
tests/unit/test_adapter.py) + docs (docs/how-to/train-on-video.md) + CHANGELOG — drop the "ragged is a follow-up" caveat.
Backward compatibility
- Divisible grids: no padding/mask is built → bit-exact with today.
- Only enables previously-rejected ragged configs; no behavior change for any existing config.
- Pure connector change — no new state-dict keys (same projection weights; window count is sized by the existing
output_num_tokens).
What
The
attentional_poolconnector (per-window attention pooling) currently requires the patch grid be divisible by the pool window — it raises on ragged grids (require_divisible=True,adapter.py). That blocks faithful Molmo2 3×3 video pooling: a SigLIP2 @224/patch16 grid is 14×14, and14 % 3 != 0, so 3×3 attentional pooling is rejected today (you have to fall back toavgpoolor a divisible window).avgpoolalready handles ragged grids (pad → mask → count-normalize over real patches only). Bringattentional_poolto parity: each partial edge window pools only its real patches via a masked attention (the padded patches are masked out of the per-window K/V). Backward-compatible — divisible grids (e.g. 2×2 on 14×14) are unchanged.Scope
kempnerforge/model/adapter.py(AttentionalPoolAdapter.forward): for a ragged grid, pad toceil(grid/w)·w, build a per-window key-padding mask (real vs padded patches), use the masked window-mean as the query, and pass the mask to SDPA so padded patches are excluded from each window's pooling. Remove the ragged-rejection raise.output_num_tokens→ceil(grid/w)². (No NaN guard needed — every ceil-window covers ≥1 real patch.)kempnerforge/model/adapter.pyDIVISIBLE_ONLY_POOL_TYPES(consulted byconfig/adapter.py): dropattentional_poolso config-time +max_seq_lenchecks accept ragged.tests/unit/test_adapter.py) + docs (docs/how-to/train-on-video.md) + CHANGELOG — drop the "ragged is a follow-up" caveat.Backward compatibility
output_num_tokens).