Fix accuracy_reward crash when called from non-main thread

[qgallouedec]

Summary

• math_verify's parse() and verify() use signal.alarm() internally, which raises ValueError when called from a non-main thread (required for asynchronous training)
• Detect non-main thread and disable signal-based timeouts (parsing_timeout=None, timeout_seconds=None)
• Suppress the resulting "Timeout is disabled" warnings since the timeout removal is intentional
• Affects both accuracy_reward and reasoning_accuracy_reward

Reproduce

import threading

from trl.rewards import accuracy_reward

solutions = [r"\frac{1}{3}"]
completions = [[{"role": "assistant", "content": r"The answer is \boxed{\frac{1}{3}}"}]]


def run_in_thread():
    result = accuracy_reward(completions, solutions)
    print("Result:", result)


# Works fine in the main thread
print("Main thread:")
run_in_thread()

# Fails in a worker thread
print("\nWorker thread:")
t = threading.Thread(target=run_in_thread)
t.start()
t.join()

Before:

Main thread:
Result: [1.0]

Worker thread:
Exception in thread Thread-1 (run_in_thread):
Traceback (most recent call last):
  File "/fsx/qgallouedec/miniconda3/envs/trl/lib/python3.13/site-packages/math_verify/parser.py", line 702, in parse
    return timeout(timeout_seconds=parsing_timeout)(extract_target_from_pred)(
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        pred,
        ^^^^^
    ...<2 lines>...
        extraction_mode=extraction_mode,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/fsx/qgallouedec/miniconda3/envs/trl/lib/python3.13/site-packages/math_verify/utils.py", line 60, in wrapper
    signal.signal(signal.SIGALRM, handler)
    ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fsx/qgallouedec/miniconda3/envs/trl/lib/python3.13/signal.py", line 58, in signal
    handler = _signal.signal(_enum_to_int(signalnum), _enum_to_int(handler))
ValueError: signal only works in main thread of the main interpreter

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/fsx/qgallouedec/miniconda3/envs/trl/lib/python3.13/threading.py", line 1044, in _bootstrap_inner
    self.run()
    ~~~~~~~~^^
  File "/fsx/qgallouedec/miniconda3/envs/trl/lib/python3.13/threading.py", line 995, in run
    self._target(*self._args, **self._kwargs)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fsx/qgallouedec/trl/reproduce_thread_issue.py", line 12, in run_in_thread
    result = accuracy_reward(completions, solutions)
  File "/fsx/qgallouedec/trl/trl/rewards/accuracy_rewards.py", line 57, in accuracy_reward
    gold_parsed = parse(sol)
  File "/fsx/qgallouedec/miniconda3/envs/trl/lib/python3.13/site-packages/math_verify/parser.py", line 711, in parse
    raise ValueError(
        "Math-Verify 'parse' function doesn't support threaded environment due to usage of signal.alarm() in timeout mechanism. If you need to run in multithreaded environment it's recommended to set the parsing_timeout=None, which will run without timeout (and signal handling). In this case you need to handle the timeouting yourself."
    ) from e
ValueError: Math-Verify 'parse' function doesn't support threaded environment due to usage of signal.alarm() in timeout mechanism. If you need to run in multithreaded environment it's recommended to set the parsing_timeout=None, which will run without timeout (and signal handling). In this case you need to handle the timeouting yourself.

After

Main thread:
Result: [1.0]

Worker thread:
Result: [1.0]

---

Note

Medium Risk
Changes reward evaluation behavior by disabling math_verify timeouts off the main thread, which avoids crashes but can allow parses/verifications to run unbounded in worker threads.

Overview
Fixes accuracy_reward and reasoning_accuracy_reward crashing in non-main threads by detecting thread context and disabling math_verify’s signal-based parse/verify timeouts when not on the main thread.

Also suppresses the expected "timeout disabled" warnings from math_verify in worker threads and adds a regression test that calls accuracy_reward from a threading.Thread.

^{Written by Cursor Bugbot for commit e0e6a76. This will update automatically on new commits. Configure here.}

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

[cursor]

Logger level permanently modified as global side effect

Medium Severity

logging.getLogger() returns a process-wide singleton, so calling setLevel(logging.ERROR) permanently suppresses warnings from math_verify.parser and math_verify.grader for the entire process. After any single call from a non-main thread, all subsequent calls — including from the main thread — will have these warnings silenced. The log levels are never restored after the function returns. A scoped approach (e.g., saving and restoring the original level in a try/finally) would avoid this persistent global side effect.

Additional Locations (1)

• trl/rewards/accuracy_rewards.py#L166-L169