Skip to content

stair-lab/compel

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

17 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Compel — Curating High Quality Pretraining Data for Language Models via Compression Ratios

TLDR: Paper repo for Compel: filter web-scale pretraining corpora by keeping documents whose LZ4 compression ratio falls in a calibrated quality band ($[0.65, 0.80]$) — a model-free information-density signal. +0.4–1.1 macro-accuracy points on FineWeb / FineWeb-EDU / DCLM at 1.4B–8B scale. This repo holds the paper LaTeX + DMLR submission package; the experiments live in Percy Liang's Marin (git submodule marin/ — Elyas's runs are files/branches there).

Layout

Path What
paper_latex/DMLR_2026_Compel/ DMLR (Journal of Data-centric ML Research) submission: 00_dmlr_compel.tex, compiled 20 pp PDF, dmlr2e style, figures
experiments/01_dmlr_submission/ Submission plan, OpenReview form answers, Mega-QA report, fetch log
marin/ Git submodule → Percy Liang's Marin (marin-community/marin); Compel's pretraining experiments live there as files/branches

Build the paper

cd paper_latex/DMLR_2026_Compel
latexmk -pdf 00_dmlr_compel.tex     # 20 pages; 0 errors / 0 undefined / 0 overfull

Provenance

  • Rebuilt 2026-07-17 from the Dec-2025 double-blind review PDF (ICLR 2026 submission KFafeqE5fe) because no paper .tex repo existed; numbers verified value-for-value against that PDF. De-anonymized for DMLR (single-blind).
  • New for the DMLR version: §4 theoretical grounding (Shannon / Kolmogorov / MDL), contributions list, repaired citations, corrected figure captions.
  • A copy of this package also exists on eobbad/Compel branch dmlr-2026-submission (historical).
  • Overleaf (canonical original source, from Elyas 2026-07-22): https://www.overleaf.com/project/68d45f22cd5b999f572fd8c6 — archived verbatim at experiments/01_dmlr_submission/overleaf_iclr2026_source/ (with zip + PROVENANCE.md). Diffed same day: the DMLR version strictly contains it (no PDF re-upload needed).

Working with the Marin submodule

git submodule update --init --depth 1 marin   # shallow; Marin is large
# Elyas's Compel experiments: search Marin branches/files for compression-ratio filtering runs

About

Paper repo for Compel: curating high-quality LM pretraining data via compression ratios (LZ4 quality band). LaTeX + DMLR submission package + reviewer-response docs. Experiments live in Percy Liang's Marin (git submodule). See paper_latex/DMLR_2026_Compel.

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages