Skip to content

Optimize decoding in VLM eval #137

Description

@camilobrownpinilla

What

Currently, VLM eval is implemented as a bare V1 with room for optimization. At least two pieces can be optimized:

  • Tokenizer decoding can be optimized from $O(n^2) \to O(n)$
  • Each decode step performs redundant device-host syncs (cur_max = int(real_len.max().item()) & int(next_tokens[i].item()))

Optimizing this will make eval cheaper.

Scope

Changes are entirely local to kempnerforge/eval/vlm/adapter.py.

Backward compatibility

This will be entirely backwards compatible.

Metadata

Metadata

Labels

enhancementNew feature or request

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions