Released model Performance lower than reported in the paper

[Hyfred]

I used the released HuggingFace checkpoint agentrl/ReSearch-Qwen-7B-Instruct, but the evaluation results are much lower than what is reported in the paper. My results:
Bamboogle: {'em': 0.16, 'f1': 0.20129523809523808, 'acc': 0.176, 'precision': 0.21292307692307694, 'recall': 0.19866666666666669}

ReSearch Paper Table 2 said that:
Bamboogle (testset) EM: 42.40

Here is my runing code (following the guidance in the README):

python run_eval.py \
    --config_path eval_config.yaml \
    --method_name re-call \
    --data_dir /home/a14-hliu/hl542/ReCall/data/ \
    --dataset_name bamboogle \
    --split test \
    --save_dir /home/a14-hliu/hl542/ReCall/eval_results/re-call_qwen3-7b-instruct \
    --save_note re-call_qwen3-7b_ins \
    --sgl_remote_url http://127.0.0.1:8083 \
    --remote_retriever_url http://127.0.0.1:8082 \
    --generator_model /home/a14-hliu/.cache/huggingface/hub/models--agentrl--ReSearch-Qwen-7B-Instruct/snapshots/f0787566dce64b1363746137aca5dd432ac48b9e \
    --sandbox_url http://127.0.0.1:8081

I also followed the README instructions for retrieval materials:

E5-base-v2
wiki18_100w_e5_index.zip
wiki18_100w.zip

Could you please advise what I might be missing? Thank you.

[AnselCmy]

The issue seems to be a mismatch between the model and the evaluation code. You are using the ReSearch model (agentrl/ReSearch-Qwen-7B-Instruct), but you are running the evaluation code designed for ReCall (as indicated by --method_name re-call and the current branch).
To reproduce the results for ReSearch, please switch to the dedicated re-search branch, which contains the correct evaluation pipeline for that model.

You can find the correct branch here: https://github.com/Agent-RL/ReCall/tree/re-search

[Hyfred]

Thank you for the reply. I just tried this branch.

I found that I need to increase the context length to 16,384 when starting the sglang server. Otherwise, the context length is not sufficient for inference the results, showing error below:

Traceback (most recent call last):
  File "/home/a14-hliu/hl542/ReSearch/scripts/evaluation/run_eval.py", line 119, in <module>
    func(args, config_dict)
  File "/home/a14-hliu/hl542/ReSearch/scripts/evaluation/run_eval.py", line 81, in research
    result = pipeline.run(test_data)
  File "/home/a14-hliu/hl542/ReSearch/src/flashrag/pipeline/active_pipeline.py", line 117, in run
    future.result()
  File "/home/a14-hliu/miniconda3/envs/re-search/lib/python3.10/concurrent/futures/_base.py", line 451, in result
    return self.__get_result()
  File "/home/a14-hliu/miniconda3/envs/re-search/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/home/a14-hliu/miniconda3/envs/re-search/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/a14-hliu/hl542/ReSearch/src/flashrag/pipeline/active_pipeline.py", line 59, in run_item
    response = response[0]
TypeError: 'NoneType' object is not subscriptable

After increasing the context length, the performance looks great now:

{'em': 0.432, 'f1': 0.546311111111111, 'acc': 0.456, 'precision': 0.5553333333333333, 'recall': 0.55}

However, in some cases where the reasoning requires more tokens, the same error still occurs. I was wondering if there is a recommended way to avoid rasing this length error?

Thanks @AnselCmy!