deltagen

Generates TPC-H datasets in Delta Lake format.

Built entirely by Claude (Anthropic).

Usage

# Generate SF=100 and SF=1000 (default)
cargo run --release

# Custom scale factors and output directory
cargo run --release -- --output /data/tpch --scale-factors 1 10 100

# Keep existing tables (disable overwrite)
cargo run --release -- --scale-factors 100 --overwrite false

Output

Each scale factor produces a directory of Delta Lake tables:

output/
  sf100/
    lineitem/
    orders/
    customer/
    part/
    partsupp/
    supplier/
    nation/
    region/
  sf1000/
    ...

All tables use Snappy-compressed Parquet with the standard TPC-H schema. Decimal columns are Decimal128(15,2), dates are Date32, strings are Utf8View.

Scale factor reference

Scale factor	Approx. Parquet output	Lineitem rows
SF=1	~310 MB	~6M
SF=100	~31 GB	~600M
SF=1000	~310 GB	~6B

Note: TPC-H "SF=1000" is conventionally called the "1 TB dataset" — that refers to the uncompressed pipe-delimited format. Parquet with Snappy compresses to roughly 1/3 of that.

Performance

Tables are generated in parallel. On a typical machine:

Scale factor	Approx. time (release build)
SF=1	~8s
SF=100	~15 min
SF=1000	~2–3 hours

Dependencies

• tpchgen — TPC-H data generation
• deltalake — Delta Lake writer (delta-rs)
• arrow — Apache Arrow in-memory format