⚡ RLPF

Reinforcement Learning from Performance Feedback for Code Generation

Train code models to produce programs that are both correct and fast.

Overview · Results · Quick Start · Reproduction · Citation

---

RLPF is a reinforcement-learning framework for performance-oriented code generation. Instead of treating every test-passing program as equally good, RLPF executes each generated program and uses correctness, failure stage, and task-normalized runtime feedback to guide policy optimization.

On the 306-task, family-disjoint PerfCodeBench test split, RLPF improves Qwen3-32B from 11.11% to 54.58% CRR and from 8.11% to 38.58% mean CGRE.

📌 This repository contains the official training, reward, evaluation, and analysis code for RLPF. Model checkpoints and the executable benchmark tasks are not stored in this repository.

🎯 Overview

Performance optimization is not captured by a binary pass/fail reward alone. RLPF provides progressively more informative feedback:

• 🧩 Dense stage feedback for extraction, compilation, execution, and correctness.
• ⚙️ Verified execution feedback from the real benchmark harness.
• ⏱️ Task-normalized efficiency feedback relative to baseline and reference programs.

flowchart LR
    A["Generated program"] --> B{"Extracted?"}
    B -- "No" --> S1["Stage feedback"]
    B -- "Yes" --> C{"Compiles?"}
    C -- "No" --> S1
    C -- "Yes" --> D{"Runs?"}
    D -- "No" --> S1
    D -- "Yes" --> E{"Correct?"}
    E -- "No" --> S1
    E -- "Yes" --> F["Task-normalized efficiency<br/>CGRE + FBR + RBR"]
    S1 --> G["GRPO update"]
    F --> G

Loading

For a correct generated program, the default composite reward is

R_correct = max(0.15, CGRE + α_F · FBR + α_R · RBR),

where α_F = 0.16 and α_R = 0.10. Incorrect programs receive the failure-stage reward described below.

Signal	Meaning
CRR	Fraction of all tasks for which the generated program is correct
FBR	Fraction of all tasks for which it is correct and faster than the baseline
RBR	Fraction of all tasks for which it is correct and at least as fast as the reference
CGRE	Correctness-gated relative efficiency, normalized by the baseline-to-reference runtime gap
CGRE≥0.8	Fraction of all tasks with CGRE ≥ 0.8

Failure-stage shaping assigns -0.05 / 0.00 / 0.05 / 0.10 to extraction failure, compilation failure, runtime failure, and wrong output, respectively. The complete implementation is in code/reward.py.

📊 Results

The following results use greedy decoding and three runtime repetitions on the 306-task PerfCodeBench test split. All values are percentages.

Method	CRR ↑	FBR ↑	RBR ↑	CGRE ↑	CGRE≥0.8 ↑
Qwen3-32B	11.11	8.17	5.56	8.11	8.17
RLVR	50.00	31.70	16.34	28.98	28.43
Runtime reward	37.91	31.70	17.65	24.66	22.55
RLPF	54.58	46.41	25.82	38.58	37.25

Component ablations and checkpoint-level summaries are available in logs/eval_summary.md and results/summary/. Exported training curves are provided in results/curves/; each curve is reported on its reward's native scale, so absolute reward magnitudes should not be compared across reward definitions.

🗂️ Repository Structure

RLPF/
├── code/
│   ├── train_grpo.py              # GRPO + LoRA training
│   ├── reward.py                  # RLPF and ablation rewards
│   ├── eval_split_vllm.py         # batched vLLM evaluation
│   ├── executable_benchmark_lib.py
│   └── tb_export_and_plot.py      # training-curve export and plotting
├── configs/                       # Accelerate and DeepSpeed ZeRO-3 configs
├── run_scripts/                   # training, ablation, and evaluation recipes
├── results/
│   ├── curves/                    # exported TensorBoard curves
│   └── summary/                   # test-set metric summaries
├── logs/                          # representative logs and result table
└── requirements.txt

🚀 Quick Start

✅ Requirements

• Linux and CUDA 12.x
• Python 3.12
• A local copy of Qwen3-32B, or its Hugging Face model ID
• A PerfCodeBench-style executable benchmark under executable_tasks/

The reference training configuration uses eight GPUs with DeepSpeed ZeRO-3. Evaluation defaults to vLLM tensor parallelism over eight GPUs.

📦 Installation

git clone https://github.com/HKUST-KnowComp/RLPF.git
cd RLPF

uv venv --python=3.12
source .venv/bin/activate
uv pip install -r requirements.txt

The benchmark layout expected by the reward harness is:

RLPF/
├── executable_tasks/
│   └── <task_id>/
│       ├── instance.json
│       ├── baseline/
│       ├── reference/
│       ├── oracle/
│       └── harness/
└── splits/
    └── split.json

CUDA tasks require nvcc. If it is not discoverable automatically, set PERFCODEBENCH_NVCC to the compiler path. Benchmark timing caches default to results/benchmark_cache/ and can be redirected with PERFCODEBENCH_CACHE_DIR.

🧪 Reproducing the Experiments

🏋️ Training

Run the full RLPF recipe:

MODEL=Qwen/Qwen3-32B bash run_scripts/run_rlpf.sh

The reference configuration uses LoRA rank 32, LoRA alpha 64, GRPO group size 8, learning rate 5e-6, maximum completion length 768, and five epochs.

Run the binary-correctness RLVR baseline:

MODEL=Qwen/Qwen3-32B bash run_scripts/run_rlvr.sh

Run the reward-component ablations:

MODEL=Qwen/Qwen3-32B bash run_scripts/run_reward_ablations.sh

The scripts write checkpoints, TensorBoard events, and logs under runs/. Monitor all experiments with:

tensorboard --logdir runs/

🔎 Evaluation

Evaluate the base model and the two configured RLPF checkpoints with vLLM:

MODEL=Qwen/Qwen3-32B bash run_scripts/eval_rlpf_vllm.sh

For a single model or adapter:

python code/eval_split_vllm.py \
  --model Qwen/Qwen3-32B \
  --adapter runs/full_qwen3_32b_z3/checkpoint-1500 \
  --split test \
  --runs-per-eval 3 \
  --reward-workers 16 \
  --tp 8 \
  --out results/eval/rlpf.json

Each evaluation output contains a summary and per-task rows. The lightweight JSON files committed under results/summary/ retain only aggregate summaries.

🔁 Reproducibility Notes

• Training and evaluation use the same task prompt format from code/prompts.py.
• The default training seed is 20260515.
• Evaluation uses greedy decoding by default.
• The committed result summaries contain aggregate metrics for 306 test tasks; model-generated source files and large intermediate artifacts are excluded.
• Runtime measurements depend on the compiler, hardware, system load, and benchmark cache. Use the same environment when comparing methods.
• The reward curves in results/curves/ are useful for within-run dynamics and stability analysis, not for comparing absolute values across different reward definitions.

📝 Citation

The paper citation and public checkpoint links will be added with the public preprint.

🙏 Acknowledgements

RLPF builds on Qwen3, TRL, DeepSpeed, vLLM, and PerfCodeBench.

📄 License

A standalone license has not yet been added to this repository. Please also review the licenses of the base model, benchmark, and third-party dependencies before reuse.