feat: SoM visual grounding + JSON retry#18

Open

evantakahashi wants to merge 11 commits into

mainfrom

evan/som-and-retry

Commits on Mar 3, 2026

chore: add pyobjc-framework-ApplicationServices for AX element detection
evantakahashi
committed
feat: add SoM config settings
evantakahashi
committed
feat: retry LLM on JSON parse failure instead of defaulting to wait
evantakahashi
committed
feat: SoM element detection, leaf filtering, screenshot annotation
evantakahashi
committed
feat: add LABELED ELEMENTS section to system prompt for SoM
evantakahashi
committed
feat: resolve SoM element labels to coordinates in action executor
evantakahashi
committed
feat: wire SoM into agent loop — detect, annotate, pass to LLM
evantakahashi
committed
fix: SoM AX detection — correct AXValueGetValue calling convention and imports
evantakahashi
committed
feat: SoM prompt rework + multi-process AX detection + benchmark
evantakahashi
committed
chore: remove benchmark and design doc from PR
evantakahashi
committed
feat: SoM dedup, off-screen filter, expanded roles, cap 40→80
evantakahashi
committed