
Thanks for the great work and for releasing the code. I have a question about how the Avg column in the ALFWorld results table (Table 1) is computed from the three per-task-type success rates (Pick&Place / Transform / Examine).
I tried to reconstruct the Avg using ALFWorld game counts per category (Seen: PP=59, Transform=68, Examine=13, total=140; Unseen: PP=41, Transform=75, Examine=18, total=134). For several rows this reconstructs the reported Avg exactly, but for a few it doesn't seem to combine — by any weighting — into the reported value.
For example, RESKILL Seen: the reported Avg is 90.0 (4B) / 90.2 (8B), but a game-count-weighted micro-average of the corresponding Pick&Place / Transform / Examine numbers comes out to ~89.0 for both (the Unseen rows do reconcile). Could you clarify how Avg is defined — is it a micro-average over all games (weighted by per-category counts), a macro-average over the three categories (unweighted), or something else — and are the per-category numbers and the Avg computed over the same set of runs/seeds? Or could you provide the complete, specific numerical values for the different tasks in Table 1?
I'd like to make sure I'm reproducing your metric correctly. Thanks in advance!