What
Currently, VLM eval is implemented as a bare V1 with room for optimization. At least two pieces can be optimized:
- Tokenizer decoding can be optimized from $O(n^2) \to O(n)$
- Each decode step performs redundant device-host syncs (
cur_max = int(real_len.max().item()) & int(next_tokens[i].item()))
Optimizing this will make eval cheaper.
Scope
Changes are entirely local to kempnerforge/eval/vlm/adapter.py.
Backward compatibility
This will be entirely backwards compatible.
What
Currently, VLM eval is implemented as a bare V1 with room for optimization. At least two pieces can be optimized:
cur_max = int(real_len.max().item())&int(next_tokens[i].item()))Optimizing this will make eval cheaper.
Scope
Changes are entirely local to
kempnerforge/eval/vlm/adapter.py.Backward compatibility
This will be entirely backwards compatible.