AI for Data Analysis: LLM Benchmark with Real Python Notebooks

This repository publishes a practical benchmark of AI for data analysis using real, step-by-step notebook workflows.

We run the same workflows with multiple LLMs, save each full conversation as a Jupyter notebook (.ipynb), and compare model quality across realistic analytical tasks.

• Benchmark website: https://mljar.com/analysis/
• Product: MLJAR Studio (desktop AI for data analysis): https://mljar.com/studio/
• AI Data Analyst overview: https://mljar.com/ai-data-analyst/

What This Benchmark Covers

We test structured data analysis workflows across multiple domains:

• Exploratory Data Analysis (EDA)
• Time Series Forecasting
• Machine Learning
• Finance
• Statistics
• NLP

Each workflow is a prompt sequence (not a single one-shot prompt). For every model run:

1. We send prompt 1.
2. Wait for the final model response.
3. Send prompt 2.
4. Continue until the workflow is complete.

The resulting conversation, code, outputs, and charts are preserved in a notebook file.

Repository Structure

Each top-level folder is one model source. Inside each folder, notebook filenames represent workflows.

• gpt-5.4/
• gpt-oss:120b/
• glm-5.1/
• gemma4:31b/
• qwen3-coder-next/
• qwen3.5:397b/

Example workflow notebook (present in each model folder):

• iris-feature-analysis.ipynb
• air-passengers-forecast.ipynb
• sentiment-analysis-python.ipynb

This structure lets you compare the same task across different models directly.

How We Score Notebook Quality

Each run is scored on a 0-10 scale using five dimensions:

• Task Completion (0-2): Did the workflow complete required steps?
• Execution Correctness (0-2): Is generated code coherent and likely runnable?
• Output Quality (0-3): Do outputs match expected analytical outcomes?
• Reasoning Quality (0-2): Are explanations and conclusions clear and correct?
• Reliability (0-1): Is behavior robust and low-risk (low hallucination tendency)?

The final score is the sum of all dimensions.

Why This Repo Exists

Most AI benchmark examples are synthetic or too short. This repository focuses on real analytical workflows with reproducible artifacts.

It helps you:

• evaluate model behavior in practical notebook analysis,
• inspect prompt-by-prompt reasoning and code generation,
• choose models for your own data analysis use cases.

Use This Benchmark

You can use these notebooks to:

• study prompt patterns that work in data analysis,
• compare model quality for your preferred domain,
• build your own internal benchmark set.

If you want to run similar workflows on your own data, use MLJAR Studio:

• https://mljar.com/downloads/

Citation / Reference

If you reference this benchmark, please link:

• Repository: https://github.com/pplonski/ai-for-data-analysis
• Analysis hub: https://mljar.com/analysis/

License

See LICENSE.