TinySQL-coder

TinySQL-coder is a learning-first text-to-SQL fine-tuning project using the BIRD mini-dev dataset, Qwen2.5-Coder, LoRA, execution-based evaluation, and a guarded post-generation SQL repair pipeline.

Live portfolio demo: tinysql-lab.streamlit.app

The public Streamlit app lets visitors inspect real frozen evaluation examples, generated SQL, execution results, benchmark progression, and the final unseen test. It is an evaluation explorer rather than live GPU inference.

Important clarification: BIRD does not mean bird images here. BIRD is a database question-answering benchmark. Each example contains a natural-language question, database context, and the SQL query that answers the question.

The project goal is:

database schema + question + evidence -> SQL query

Current Status

Current verified snapshot (2026-07-15):

Best development model pipeline:    Run 013 + values + repair at 35/100
Best execution consensus:           44/100
Best end-to-end system:              gold-free SQL judge at 46/100
Judge SQL executable:                95/100
Final unseen-question pipeline:      Run 013 + values + repair at 43/100
Unseen pipeline SQL executable:      83/100
Automated tests:                     58 passing
Default branch:                   main
Current direction:                experiment complete

Run 013 trained Qwen2.5-Coder-3B with 4-bit QLoRA on the official filtered BIRD rows. Database-value retrieval raised it to 29/100 raw and 35/100 after repair. With result consensus and a gold-free semantic SQL judge, the end-to-end system reached a new project best of 46/100.

The final one-shot evaluation used 100 official BIRD dev questions whose IDs were excluded from the 500-question mini-dev set. The locked Run 013 value- guided pipeline scored 36/100 raw and 43/100 after its unchanged guarded repair pass. These are unseen questions on familiar database schemas. See Eval 019.

The final unseen benchmark and interpretation are documented in Eval 019.

Done so far:

1. Downloaded and inspected the BIRD mini-dev dataset.
2. Added the matching official SQLite database package locally.
3. Extracted compact schema text from each SQLite database.
4. Built processed JSONL training data with schema, question, evidence, and gold SQL.
5. Split data into train and validation sets.
6. Built a fixed 20-example baseline evaluation set.
7. Ran base Qwen baseline evaluation.
8. Added execution-based SQL evaluation against real SQLite databases.
9. Prepared chat-style supervised fine-tuning data.
10. Set up CUDA-enabled PyTorch.
11. Ran a LoRA smoke test.
12. Ran LoRA Training Run 001.
13. Evaluated Qwen + LoRA Run 001.
14. Added a comparison tool for base-vs-LoRA evaluation reports.
15. Added failure-pattern analysis for SQL execution errors.
16. Created SFT V2 with schema guidance and ran LoRA Run 002.
17. Created SFT V3 to oversample join-heavy examples for Run 003.
18. Ran LoRA Run 003 and got the first fine-tuned execution improvement.
19. Inspected Run 003's successful predictions to understand what improved.
20. Created SFT V4 to focus on harder join and subquery examples for Run 004.
21. Ran LoRA Run 004 and reached the best execution score so far.
22. Added hard-failure inspection to guide Run 005 planning.
23. Tried SFT V5 ownership-teacher data for Run 005.
24. Evaluated Run 005 and confirmed that free-form ownership reasoning hurt execution accuracy.
25. Added a conservative SQL alias-repair experiment for Run 004 predictions.
26. Extended SQL repair with conservative join-aware repair using direct foreign keys.
27. Added semantic lookup-table repair for ID columns compared to human-readable labels.
28. Added value canonicalization for safe case-insensitive database value matches.
29. Added guarded leading-join pruning and duplicate-projection DISTINCT repair.
30. Tried guarded table-split repair for wrong-table column references.
31. Created error-aware SFT V6 data for Run 006 planning supervision.
32. Trained and evaluated LoRA Run 006.
33. Improved schema guidance join hints to prefer real foreign keys.
34. Trained and evaluated LoRA Run 007.
35. Created SFT V7 with source-table supervision and evaluated LoRA Run 008.
36. Built a focused repair error set across the strongest repaired runs.
37. Added guarded unqualified-column, undeclared-alias, foreign-key join, and exact syntax repairs.
38. Trained Run 009 for 160 steps on the clean V6 recipe and reached the best raw execution score.
39. Expanded evaluation to all 100 validation examples with per-query SQLite timeouts.
40. Pivoted Run 010 to Qwen2.5-Coder-1.5B and passed a local LoRA smoke run on the 8 GB GPU.
41. Trained and evaluated the 1.5B Run 010 adapter on all 100 held-out examples.
42. Added an execution-fallback cascade and reached the best result at 35/100.
43. Replaced repeated 400-question training with 6,601 curated BIRD training rows.
44. Added database-disjoint validation and exact mini-dev leakage checks.
45. Trained Run 011 on direct SQL targets, then measured join-context failures.
46. Continued Run 012 with aligned foreign-key guidance and reached 27/100 after repair.
47. Added Windows-compatible NF4 QLoRA support and passed a 3B training smoke test.
48. Evaluated the quantized 3B base model at 22/100 raw and 27/100 repaired.
49. Made 3B training length-safe after a step-37 memory failure; 40 tests pass.
50. Trained Run 013 and reached 34/100 with repair and 36/100 by safe cascade.
51. Added leakage-safe SQLite value retrieval; 46 automated tests pass.
52. Evaluated value retrieval and raised the project-best cascade to 39/100.
53. Prepared value-aligned SFT V10 and passed a 3B continuation smoke test.
54. Added execution-result consensus and reached 44/100 without expected answers.
55. Added a gold-free semantic SQL judge and reached the final best of 46/100.
56. Froze a leakage-checked unseen-question set and scored the practical Run 013 pipeline at 43/100 after repair.

Latest high-level result:

Base Qwen execution matches:    1/20
LoRA Run 001 execution matches: 0/20
LoRA Run 002 execution matches: 0/20
LoRA Run 003 execution matches: 2/20
LoRA Run 004 execution matches: 3/20
LoRA Run 005 execution matches: 0/20
LoRA Run 006 execution matches: 3/20
LoRA Run 007 execution matches: 4/20
LoRA Run 008 execution matches: 2/20
LoRA Run 009 execution matches: 5/20
Run 004 + alias repair matches: 3/20
Run 004 + join repair matches:  4/20
Run 004 + semantic repair matches: 5/20
Run 004 + value repair matches:    6/20
Run 004 + distinct repair matches: 7/20
Run 004 + table repair matches:    7/20
Run 006 + repair matches:          3/20
Run 007 + repair matches:          6/20
Run 008 + repair matches:          3/20
Run 009 + repair matches:          6/20

Run 009 on the full 100-example validation benchmark:

Run 009 raw execution matches:     22/100
Run 009 + repair matches:          29/100
Run 009 raw SQL executed:          49/100
Run 009 + repair SQL executed:     75/100
Run 010 raw execution matches:     26/100
Run 010 + repair matches:          28/100
Run 009 -> Run 010 cascade:        35/100
Cascade SQL executed:              92/100
Run 011 raw execution matches:     13/100
Run 011 + repair matches:          19/100
Run 011 + guided repair matches:   24/100
Run 012 raw execution matches:     20/100
Run 012 + repair matches:          27/100
Three-model cascade matches:       35/100
Three-model cascade SQL executed:  96/100
Run 013 raw execution matches:     28/100
Run 013 + repair matches:          34/100
Run 013-led cascade matches:       36/100
Run 013-led cascade SQL executed:  98/100
Run 013 + values raw matches:      29/100
Run 013 + values + repair:         35/100
Value-guided cascade matches:      39/100
Value-guided cascade SQL executed: 99/100
Execution consensus matches:       44/100
Gold-free SQL judge matches:        46/100
Gold-free SQL judge SQL executed:   95/100
Unseen Run 013 + values raw:         36/100
Unseen Run 013 + values + repair:    43/100
Unseen repaired SQL executed:        83/100

The earlier 20-question experiments established the repair pipeline. The fixed 100-question mini-dev benchmark is the authoritative development comparison; the leakage-checked Eval 019 set is the authoritative final test of unseen questions. Because their difficulty mixes differ, their scores should not be treated as a controlled before-and-after comparison. Run 005 was a useful negative result: asking the model to emit ownership notes before SQL made generation less stable. A conservative alias-repair pass made more Run 004 predictions executable, but did not improve execution matches. A join-aware repair pass improved Run 004 to 4/20 execution matches and made 11/20 predictions executable. A semantic lookup repair improved Run 004 to 5/20 execution matches by fixing an ID-column-to-label comparison. Value canonicalization improved Run 004 to 6/20 execution matches by fixing a safe case mismatch. Leading-join pruning plus duplicate-projection DISTINCT repair improved Run 004 to 7/20 execution matches. A guarded table-split repair improved schema handling for quoted columns and incomplete SQLite foreign-key metadata, but did not improve beyond 7/20. The main failure pattern is still schema grounding: the model often puts a real column on the wrong table, skips a needed join, or chooses the wrong fact table.

SFT V6 used short plan-type supervision:

PLAN_TYPE: local_schema_fix | lookup_or_value_fix | fact_table_first | fresh_query_plan
...
FINAL_SQL:
SELECT ...

Run 006 learned the format in training, but scored 3/20 execution matches. Its repair pass increased executable SQL from 7/20 to 10/20, but execution matches stayed 3/20. Cleaning schema guidance join hints improved Run 007 to 4/20 raw execution matches and 6/20 after repair. SFT V7 source-table supervision hurt Run 008, so the next useful work shifted back to small post-generation repair experiments. The focused repair error set found 30 mechanically interesting remaining failures across repaired Runs 004, 007, and 008: 12 wrong-table column references, 7 invented columns, 6 ambiguous or unqualified columns, 4 other execution errors, and 1 hallucinated table.

Four guarded repair experiments improved executability but did not raise the best execution-match score. Returning to the clean V6 training recipe for 160 steps produced Run 009, which improved raw execution matches from Run 007's 4/20 to 5/20 and lowered validation loss from 0.1841 at step 80 to 0.1631 at step 160. Run 009 + repair reached 6/20.

Project Structure

.
+-- data/                         local only, ignored by Git
|   +-- bird_mini_dev/
|       +-- raw/
|       +-- dev_databases/
|       +-- schema/
|       +-- processed/
|       +-- sft/
|       +-- sft_v6/
|       +-- sft_v7/
+-- docs/                         experiment journals and notes
+-- models/                       local only, ignored by Git
|   +-- huggingface/
|   +-- lora-smoke-test/
|   +-- tinysql-coder-lora-run-001/
|   +-- tinysql-coder-lora-run-009/
+-- notebooks/
|   +-- 001-data-exploration.ipynb
|   +-- 002-inspect-qwen-model.ipynb
+-- outputs/                      generated eval outputs, ignored by Git
+-- scripts/
|   +-- download-dataset.py
|   +-- extract_schema_text.py
|   +-- prepare_training_data.py
|   +-- split_training_data.py
|   +-- prepare_sft_data.py
|   +-- prepare_sft_data_v2.py
|   +-- prepare_sft_data_v3.py
|   +-- prepare_sft_data_v4.py
|   +-- prepare_sft_data_v5.py
|   +-- prepare_sft_data_v6.py
|   +-- prepare_sft_data_v7.py
|   +-- create_baseline_eval_set.py
|   +-- run_baseline_model.py
|   +-- evaluate_sql_execution.py
|   +-- compare_eval_runs.py
|   +-- analyze_failure_patterns.py
|   +-- create_focused_error_set.py
|   +-- repair_sql_predictions.py
|   +-- cascade_sql_predictions.py
|   +-- check_training_readiness.py
|   +-- train_lora_smoke_test.py
|   +-- train_lora.py
+-- tests/                        repair, evaluation, and sampling tests
+-- requirements.txt
+-- README.md

The data/, models/, and outputs/ folders are intentionally ignored by Git. They contain downloaded datasets, generated files, model weights, adapters, and evaluation outputs.

Environment

Use the project-local Python 3.12 environment:

.venv312

In VS Code notebooks, select:

Finetuning Model (Python 3.12)

Install dependencies:

.\.venv312\Scripts\python.exe -m pip install -r requirements.txt

The current setup uses CUDA-enabled PyTorch:

torch: 2.11.0+cu128
CUDA available: True
GPU: NVIDIA GeForce RTX 4070 Laptop GPU

Data Pipeline

Download the mini-dev dataset:

.\.venv312\Scripts\python.exe scripts\download-dataset.py

Extract SQLite schemas into compact text:

.\.venv312\Scripts\python.exe scripts\extract_schema_text.py

Prepare processed training data:

.\.venv312\Scripts\python.exe scripts\prepare_training_data.py

Inspect processed examples:

.\.venv312\Scripts\python.exe scripts\inspect_training_data.py

Split train/validation:

.\.venv312\Scripts\python.exe scripts\split_training_data.py

Prepare chat-style SFT data:

.\.venv312\Scripts\python.exe scripts\prepare_sft_data.py

Current data counts:

Raw examples: 500
Train examples: 400
Validation examples: 100
SFT train examples: 400
SFT validation examples: 100

Schema Format

The schema extraction step turns each SQLite database into compact table/column text like:

atom(molecule_id, atom_id, element)
bond(molecule_id, bond_id, bond_type)
connected(atom_id, atom_id2, bond_id)
molecule(molecule_id, label)

Plain English: each line is one database table, and the names inside parentheses are that table's columns.

Baseline Evaluation

Create the fixed baseline evaluation set:

.\.venv312\Scripts\python.exe scripts\create_baseline_eval_set.py

Use --sample-size 100 to build the full held-out validation evaluation set instead of the default balanced 20-question sample.

Run the base Qwen model:

.\.venv312\Scripts\python.exe scripts\run_baseline_model.py

Run only a small smoke test:

.\.venv312\Scripts\python.exe scripts\run_baseline_model.py --limit 2

Evaluate SQL by execution:

.\.venv312\Scripts\python.exe scripts\evaluate_sql_execution.py

Each SQLite query is interrupted after 5 seconds by default so a pathological model prediction cannot freeze a large evaluation. Override this with --query-timeout-seconds when needed.

Latest base model execution result:

Exact matches: 0/20
Execution matches: 1/20
Gold SQL executed successfully: 20/20
Predicted SQL executed successfully: 2/20

Execution match is more useful than exact string match because two different SQL queries can return the same correct result.

LoRA Fine-Tuning

We are fine-tuning Qwen with LoRA adapters.

Base model:

Qwen/Qwen2.5-Coder-0.5B-Instruct

LoRA means we freeze the original Qwen weights and train a small adapter on top. We are still using Qwen; LoRA is the fine-tuning method.

Run the training readiness check:

.\.venv312\Scripts\python.exe scripts\check_training_readiness.py

Run the tiny smoke test:

.\.venv312\Scripts\python.exe scripts\train_lora_smoke_test.py

Run the first real LoRA trainer:

.\.venv312\Scripts\python.exe scripts\train_lora.py --max-steps 20 --gradient-accumulation-steps 4 --eval-every 5 --validation-limit 10 --output-dir models\tinysql-coder-lora-run-001

The trainer can also accept alternate SFT files:

.\.venv312\Scripts\python.exe scripts\train_lora.py --train-path data\bird_mini_dev\sft_v2\train_sft_v2.jsonl --validation-path data\bird_mini_dev\sft_v2\validation_sft_v2.jsonl --max-sequence-length 3072 --output-dir models\tinysql-coder-lora-run-002

LoRA Run 001 result:

trainable params: 4,399,104
all params: 498,431,872
trainable percent: 0.8826
final validation loss: 0.3360

The adapter is saved locally under:

models/tinysql-coder-lora-run-001

Evaluate A LoRA Adapter

Generate predictions with the LoRA adapter:

.\.venv312\Scripts\python.exe scripts\run_baseline_model.py --adapter-path models\tinysql-coder-lora-run-001 --output-path outputs\lora-run-001\predictions.jsonl

Evaluate those predictions by execution:

.\.venv312\Scripts\python.exe scripts\evaluate_sql_execution.py --predictions-path outputs\lora-run-001\predictions.jsonl --output-path outputs\lora-run-001\execution_eval.jsonl --summary-path outputs\lora-run-001\execution_eval_summary.json

LoRA Run 001 execution result:

Exact matches: 0/20
Execution matches: 0/20
Gold SQL executed successfully: 20/20
Predicted SQL executed successfully: 2/20

This means the adapter trained successfully, but did not improve SQL quality in the first short run.

Compare Runs

Generate a local markdown comparison report:

.\.venv312\Scripts\python.exe scripts\compare_eval_runs.py

Default comparison:

outputs/baseline/execution_eval.jsonl
vs
outputs/lora-run-001/execution_eval.jsonl

Default generated report:

outputs/comparisons/base-vs-lora-run-001.md

This output report is ignored by Git.

Analyze Failure Patterns

Run failure analysis on a SQL execution-evaluation file:

.\.venv312\Scripts\python.exe scripts\analyze_failure_patterns.py

Default input:

outputs/lora-run-001/execution_eval.jsonl

Default generated report:

outputs/analysis/lora-run-001-failure-analysis.md

Latest LoRA Run 001 failure pattern:

wrong_table_for_column: 13
ambiguous_or_unqualified_column: 2
executes_wrong_result: 2
hallucinated_table: 2
execution_error_other: 1

Mentor translation: the model is often choosing relevant column names but placing them on the wrong table. Run 002 should teach table-column ownership and join paths more explicitly.

LoRA Run 002 reduced wrong_table_for_column failures from 13 to 11, but still scored 0/20 execution matches. The next likely improvement is not only clearer schema text; it may require more targeted join-focused training examples.

LoRA Run 003 used join-focused oversampling and improved to 2/20 execution matches, with 4/20 predicted SQL queries executing successfully.

The two Run 003 execution matches were a single-table aggregate and a simple two-table join. That suggests join-focused training helped on simpler join structures, while moderate/challenging joins still need stronger supervision.

LoRA Run 004 used hard-join oversampling and improved to 3/20 execution matches. The added success was another simple two-table join, so the next frontier is still moderate/challenging join reasoning.

Hard-failure inspection pointed to this possible training format:

needed columns -> owning tables -> join path -> final SQL

SFT V5 tested that idea directly, but it made generation less stable and scored 0/20 execution matches. A narrower SQL alias-repair experiment improved executability from 5/20 to 8/20 but kept execution matches at 3/20.

Notebooks

Dataset exploration:

notebooks/001-data-exploration.ipynb

Model file and LoRA target-module inspection:

notebooks/002-inspect-qwen-model.ipynb

Use the second notebook to see where Qwen layer names like q_proj, v_proj, gate_proj, and down_proj come from.

Evaluation Journal

Experiment notes are in:

docs/evaluation-journal.md

Each major check or evaluation has its own markdown file under docs/.

Important entries:

docs/eval-001-baseline.md
docs/eval-004-schema-grounding-prompt.md
docs/lora-training-run-001.md
docs/eval-005-lora-run-001.md
docs/eval-comparison-tool.md
docs/failure-pattern-analysis-001.md
docs/sft-v2-schema-guidance.md
docs/eval-009-lora-run-005.md
docs/sql-repair-run-004-001.md
docs/sql-repair-run-004-002.md
docs/sql-repair-run-004-003.md
docs/sql-repair-run-004-004.md
docs/sql-repair-run-004-005.md
docs/sql-repair-run-004-006.md
docs/sft-v6-error-aware.md
docs/lora-training-run-006.md
docs/eval-010-lora-run-006.md
docs/schema-guidance-quality-001.md
docs/lora-training-run-007.md
docs/eval-011-lora-run-007.md
docs/sft-v7-source-table-supervision.md
docs/lora-training-run-008.md
docs/eval-012-lora-run-008.md
docs/sql-repair-unqualified-column-001.md
docs/sql-repair-undeclared-alias-001.md
docs/sql-repair-unqualified-join-001.md
docs/sql-repair-syntax-fragment-001.md
docs/lora-training-run-009.md
docs/eval-013-lora-run-009.md
docs/eval-014-lora-run-009-full-validation.md
docs/model-capacity-decision-001.md
docs/lora-training-run-010.md
docs/eval-015-lora-run-010.md
docs/filtered-bird-training-pivot.md
docs/eval-016-filtered-bird-training.md
docs/qwen-3b-qlora-readiness.md
docs/eval-017-qwen-3b-run-013.md
docs/value-retrieval-v1.md
docs/eval-018-consensus-and-sql-judge.md
docs/eval-019-unseen-bird-dev.md

Final Decision

The final semantic judge improved selection among executable candidates, but the full benchmark still shows that semantic reasoning is the larger bottleneck:

46/100 judge predictions return the correct rows
49/100 execute but return the wrong rows
5/100 fail to execute

Run 013, leakage-safe value retrieval, execution consensus, and the final gold-free SQL judge improved the project best to 46/100. Run 014 and beam search did not improve it. A final one-shot test on unseen questions reached 43/100 with the practical single-model repaired pipeline. The project did not reach the original 50/100 milestone, but it demonstrated real question-level generalization. The experiment series is now stopped. See Eval 018 and Eval 019.