Oracle solutions for Harbor adaption

[NamburiSrinath]

Hi,

I am trying to port this benchmark to harbor format for wider adaptability.

As part of that, the initial step is to verify oracle solutions (step 3) but I don't see oracle solutions in the repo.

Does the authors plan to release oracle solution?

Thanks

[finyorko]

Hi @NamburiSrinath, thanks for the interest!

We do maintain human-authored oracle solutions internally. They aren't published in the repo for two related reasons: the original solution paths depend on manual operations and exist as scattered records, and we also want to keep them private to prevent contamination in future evaluations. Internally they only served to confirm that each task is completable and evaluable.

For your Harbor adaptation use case, I'd recommend a different approach: run each task inside its Docker container and have an agent (e.g. Claude Code or Codex) interact with it to produce a working solution path. Since each task's test suite already encodes the success criteria, any agent-generated trajectory that passes the F2P tests can serve as a valid oracle for Harbor's Step 3 verification. As agents have become increasingly capable, letting them extract the solution path directly is often a more effective route than relying on manually curated oracles.

Happy to help if you run into specific issues while doing this.

[NamburiSrinath]

Hi @finyorko,

Thanks a lot for this suggestion. I've started using Claude code to iterate on the oracles and I found an interesting issue with one of the task of current benchmark

cs61-fa24-cats: The max f2p.py score was given as 20, but technically, the max score that can be achieved is 19 (as EC.py is not included, following the conventions of cs61-fa24-ants and cs61-fa24-hog). Even if you include that, the score will be 21. So, technically the score will be either 19 or 21 which means there's an issue here.

Any comments on that (tagging @eatingallday as well)? If you agree with the above analysis, can you make a change at your repo as I am pulling from the main branch to construct the adapter.

Something like below inside the f2p.py should suffice!

max_score = 19.0 (instead of 20.0) 

By the way, you can track the progress of this adapter here in this PR - harbor-framework/harbor#1851

I am in the process of converting other tasks and hopefully will come back with few more questions

Thanks for your help

[NamburiSrinath]

Hi @finyorko and @eatingallday

I've another set of questions when I followed the above approach - please refer this comment from the PR for more context

Q1 What's the allowed and acceptable "memory_mb", "cpus", "verifier_timeout_sec"? Because only by increasing these, some questions were able to be solved and this info is not present in the dataset - 61810-lock, 61810-mmap, 61810-fs, cmu15-445-p0, cmu15-445-p1

Q2 For 61810-lock, 61810-mmap, 61810-fs, when the grader time also got bumped up (namely tests/grade-lab-lock, tests/grade-lab-mmap, tests/grade-lab-fs), the oracle (which was produced by agent) got F2P of 1. Is that acceptable? Because this info is directly coming from dataset, if so, this needs to be increased so I can pull from here.

Q3 For cmu15-445-p2, a strong oracle (Opus 4.8) got only 7/18. Even with some changes to dataset (which it shouldn't) - it got 12/18. So, I need help here.

Q4 (just to have everything here) cs61-fa24-cats: The max f2p.py score was given as 20, but technically, the max score that can be achieved is 19 (as EC.py is not included, following the conventions of cs61-fa24-ants and cs61-fa24-hog). Even if you include that, the score will be 21. So, technically the score will be either 19 or 21 which means there's an issue here.

I would really appreciate any comments and help here. Happy to share more details, logs etc; I am debugging at my end to see if some of these can be mitigated.