LegacyWorld

LegacyWorld is a desktop benchmark harness for evaluating computer-use agents in legacy software environments. It includes:

• BenchSvc: a Windows-side control service used inside the benchmark VM.
• src/bench_core: Python harness code for VM orchestration, task execution, state checks, reporting, and agent adapters.
• tasks: task definitions and verification logic.
• tests: focused unit and smoke tests for the public harness behavior.

Security Model

BenchSvc is intended to run only inside an isolated benchmark virtual machine. It exposes endpoints that can start processes, inspect desktop state, and fetch files from the configured benchmark working directory. Do not expose it to untrusted networks.

By default, the service binds to 127.0.0.1:5000. If you intentionally bind it to a broader interface, place the VM behind an appropriate network boundary.

Configuration

The harness reads configuration from a .env file at the repo root (loaded via python-dotenv). Copy .env.example to .env and fill in the values you need.

The relevant variables:

• LEGACY_AGENT_HOST, LEGACY_AGENT_KEY, LEGACY_AGENT_TARGET_ID — endpoint, API key, and target identifier for an agent backend that implements the HTTP contract used by LegacyUseAgentHandler (see src/bench_core/agents.py; the handler POSTs to /targets/{target_id}/jobs/ and polls/interrupts/resolves through the same prefix). Leave them unset to run with only the no-op baseline handler.
• BENCHMARK_MODE — custom (default) or generated-prompt.
• QEMU_IMG_PATH, QEMU_SYSTEM_PATH, BASE_IMAGE_PATH — override the defaults in src/bench_core/main.py. Defaults assume Homebrew QEMU on Apple Silicon; on x86 hosts set QEMU_SYSTEM_PATH to the appropriate qemu-system-x86_64 binary.

Development

Install Python dependencies, then run:

make test

Build the Windows service with:

make benchsvc-publish

Generate a local task catalogue with:

make task-catalog

The generated catalogue is written to TASKS.generated.md and is not part of the release source.

License

LegacyWorld is licensed under the Apache License, Version 2.0. See LICENSE.