Skip to content

how is the "Avg" column in the ALFWorld results table computed? #15

Description

@socialism-redstar
Image Thanks for the great work and for releasing the code. I have a question about how the Avg column in the ALFWorld results table (Table 1) is computed from the three per-task-type success rates (Pick&Place / Transform / Examine). I tried to reconstruct the Avg using ALFWorld game counts per category (Seen: PP=59, Transform=68, Examine=13, total=140; Unseen: PP=41, Transform=75, Examine=18, total=134). For several rows this reconstructs the reported Avg exactly, but for a few it doesn't seem to combine — by any weighting — into the reported value. For example, RESKILL Seen: the reported Avg is 90.0 (4B) / 90.2 (8B), but a game-count-weighted micro-average of the corresponding Pick&Place / Transform / Examine numbers comes out to ~89.0 for both (the Unseen rows do reconcile). Could you clarify how Avg is defined — is it a micro-average over all games (weighted by per-category counts), a macro-average over the three categories (unweighted), or something else — and are the per-category numbers and the Avg computed over the same set of runs/seeds? Or could you provide the complete, specific numerical values ​​for the different tasks in Table 1? I'd like to make sure I'm reproducing your metric correctly. Thanks in advance!

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions