MLflow GenAI Tutorial Series

Tutorial: Mastering GenAI with MLflow

A step-by-step series on using MLflow's open source platform to build, track, trace, evaluate, and debug GenAI applications, including managing and optimizing prompts along the way.

🎬 Series Overview Video

Watch the full overview: Mastering GenAI Development with MLflow

📚 Tutorial Structure

Notebook	Title	Description	Video Tutorial
1.1	Setup and Introduction	Understanding MLflow for GenAI Installation and configuration First tracked run MLflow UI basics	Watch Tutorial
1.2	Experiment Tracking for LLMs	Tracking LLM parameters and metrics Comparing model configurations Cost tracking and optimization Organizing experiments with tags Parent-child runs for workflows	Watch Tutorial
1.3	Introduction to Tracing	Auto-tracing with MLflow Understanding the trace model Manual instrumentation Viewing traces in UI	Watch Tutorial
1.4	Manual Tracing and Advanced Observability	Custom span decorators Tracing complex workflows Debugging with traces Multi-step agentic patterns	Watch Tutorial
1.5	Prompt Management	Creating prompt templates Versioning prompts Registering in the Prompt Registry Searching Prompt Registry Using prompts from the Prompt Registry Linking prompts to experiments	Watch Tutorial
1.6	Framework Integrations	OpenAI direct API integration LangChain chains and workflows LlamaIndex document indexing and RAG Framework comparison matrix Best practices for each framework	Watch Tutorial
1.7	Evaluating Agents	LLM-as-Judge evaluation patterns MLflow built-in scorers (RelevanceToQuery, Correctness, Guidelines) Custom scorers with @scorer decorator DeepEval integration for conversations Session-level multi-turn evaluation	Watch Tutorial
1.8	Prompt Optimization with GEPA	Automatic prompt optimization with GEPA algorithm MLflow Prompt Registry integration Before/after evaluation comparison	Watch tutorial
1.9	Complete RAG Application	Building a full RAG pipeline End-to-end tracing Performance analysis RAG evaluation with RAGAS metrics	Watch tutorial
1.10	Multi-Agent Supervisor Pattern	Supervisor agent that routes queries to specialized subagents Genie agent for structured data queries (SQL over pandas DataFrame) Knowledge Assistant for document retrieval and policy Q&A Full supervisor-subagent trace hierarchy with MLflow 3-layer evaluation: routing accuracy, response quality, orchestration	Coming Soon...
1.11	LangGraph Deep Agents	Deep Agents with built-in planning (write_todos / read_todos) File system context management (read, write, edit files) Sub-agent delegation via task() tool Evaluating Deep Agent outputs with mlflow.genai.evaluate() Auto-tracing with mlflow.langchain.autolog()	N/A
1.12	CrewAI Multi-Agent Orchestration	Role-based agents with CrewAI (role, goal, backstory) Custom tools: query_disaster_database and search_fema_policies Hierarchical crew with manager agent delegating to specialists Evaluating crew outputs with mlflow.genai.evaluate() Auto-tracing with mlflow.crewai.autolog()	N/A

🎓 What You'll Learn

By the end of the series you'll be able to:

• Set up MLflow for GenAI development and track LLM experiments
• Instrument code with tracing, then use traces to debug and analyze performance
• Track and optimize the cost of LLM calls
• Manage prompts with version control and the Prompt Registry, and optimize them with GEPA
• Integrate MLflow across OpenAI, LangChain, LlamaIndex, and CrewAI
• Evaluate agents with MLflow's built-in judges plus custom, DeepEval, and RAGAS scorers
• Build end-to-end RAG applications
• Build, trace, and evaluate multi-agent orchestration systems

🚀 Getting Started

This project uses UV for dependency management.

1. Install UV (if not already installed)

curl -LsSf https://astral.sh/uv/install.sh | sh

2. Install Dependencies

uv sync

3. Configure API Keys Create a .env file in the tutorials directory:

OPENAI_API_KEY=your-api-key-here
MLFLOW_TRACKING_URI=http://localhost:5000

4. Start Jupyter

uv run jupyter notebook

5. Start MLflow UI (in a separate terminal)

uv run mlflow ui --port 5000

6. Open Browser Navigate to http://localhost:5000

📋 Prerequisites

• Python 3.10+
• UV package manager
• OpenAI API key (or Databricks Workspace)
• Basic understanding of Python and LLMs

📂 Directory Structure

mlflow-genai-tutorial-1/
├── 01_setup_and_introduction.ipynb
├── 02_experiment_tracking.ipynb
├── 03_introduction_to_tracing.ipynb     
├── 04_manual_tracing_advanced.ipynb     
├── 05_prompt_management.ipynb           
├── 06_framework_integrations.ipynb      
├── 07_evaluating_agents.ipynb
├── 08_prompt_optimization.ipynb
├── 09_complete_rag_application.ipynb
├── 10_multi_agent_supervisor.ipynb
├── 11_deep_agents_langgraph.ipynb
├── 12_crewai_multi_agent.ipynb
├── .env                                 (create this yourself and use the template env_template)
└── README.md

🔗 Resources

• MLflow Documentation
• MLflow GenAI Guide
• MLflow GitHub
• MLflow Community

💡 Tips

• Keep the MLflow UI open while working through notebooks
• Experiment with different parameter values
• Compare runs in the UI to understand trade-offs
• Tag runs for easy organization
• Track costs from the beginning

❓ Troubleshooting

Issue: MLflow UI won't start

# Try a different port
uv run mlflow ui --port 5001

Issue: API key not recognized

# Restart Jupyter kernel after adding to .env
# Or set manually:
import os
os.environ["OPENAI_API_KEY"] = "your-key"

Issue: Module not found

# Sync dependencies with UV
uv sync

# Or install specific package
uv add mlflow openai python-dotenv

📝 License

This tutorial series is provided as educational content for learning MLflow's GenAI capabilities.

---

📝 Feedback & Contributions

• Found an issue? Open a GitHub issue
• Have suggestions? Submit a pull request
• Want to share? Tag us on social media
• Questions? Check the MLflow community

---

Authors: Jules (Databricks Developer Relations) + Claude Code Date: February 2026 MLflow Version: 3.14.0