refactor: Unify and modernize agent logging system

[ceberam]

Overview

This PR completely refactors the logging system across all agents and backends to provide unified, clear, and modern agentic logging output. The previous logging was inconsistent, verbose, and made it difficult to understand LLM interactions and multi-turn conversations.

Problem Statement

The original logging had several issues:

• Inconsistent formats across different agents and backends
• Verbose timestamps that cluttered the output
• Unclear LLM interactions - hard to distinguish requests from responses
• No structured context for operations and stages
• Noisy warnings from underlying frameworks (Mellea)
• Poor visibility into multi-turn conversations and retry logic

Solution

1. Enhanced Logging Module (docling_agent/logging.py)

Created a comprehensive logging API with:

• Clean timestamp format: ISO 8601 with milliseconds [YYYY-MM-DD HH:MM:SS.mmm]
• Context managers: agent_context(), operation_context(), timed_operation()
• Structured logging functions:
  • log_agent_start() / log_agent_end() - Agent lifecycle
  • log_stage_start() / log_stage_end() - Processing stages
  • log_llm_request() / log_llm_response() - LLM interactions
  • log_info(), log_warning(), log_error(), log_success(), log_debug()
• Visual indicators: Emoji prefixes for quick scanning (🤖 agent, 🔄 operation, 💬 LLM, ✅ success, etc.)

2. Updated All Backends

• Mellea Backend: Added LLM request/response logging, suppressed noisy template warnings
• Ollama Backend: Enhanced with retry tracking and clear LLM interaction logs
• OpenAI-Compatible Backend: Structured logging for requests, responses, and validation attempts
• LiteLLM Backend: Inherits logging from OpenAI-Compatible
• LM Studio Backend: Inherits logging from OpenAI-Compatible
• Llama-Server Backend: Inherits logging from OpenAI-Compatible (new backend automatically compatible)

3. Updated All Agents

• Writer, Editor, Enricher, Extractor, Orchestrator, RAG, Library: All use new logging API
• Base Functions: Updated error handling with structured logging
• Consistent use of context managers for hierarchical operation tracking

Before & After Examples

Before (Old Logging)

2025-08-15 12:34:56,789 - docling_agent.agent.writer - INFO - Starting writer agent
2025-08-15 12:34:56,790 - docling_agent.backends.mellea_backend - DEBUG - Sending prompt to LLM
2025-08-15 12:34:56,791 - mellea.template - WARNING - Template variable 'context' not found in scope
2025-08-15 12:34:58,123 - docling_agent.backends.mellea_backend - DEBUG - Received response from LLM
2025-08-15 12:34:58,124 - docling_agent.agent.writer - INFO - Processing document
2025-08-15 12:34:58,125 - docling_agent.agent.writer - INFO - Writer agent completed

After (New Logging)

[2025-08-15 12:34:56.789] INFO | 🤖 Starting agent: Writer
[2025-08-15 12:34:56.790] INFO | 🔄 Starting operation: Generate report content
[2025-08-15 12:34:56.791] INFO | 💬 LLM Request [mellea/llama-3.1-8b]
[2025-08-15 12:34:56.791] DEBUG |   Prompt: Write a comprehensive report about...
[2025-08-15 12:34:56.791] DEBUG |   Options: temperature=0.7, max_tokens=2000
[2025-08-15 12:34:58.123] INFO | 💬 LLM Response [mellea/llama-3.1-8b]
[2025-08-15 12:34:58.123] DEBUG |   Response: # Comprehensive Report\n\nThis report...
[2025-08-15 12:34:58.124] INFO | ✅ Operation completed: Generate report content (1.33s)
[2025-08-15 12:34:58.125] INFO | ✅ Agent completed: Writer

Key Improvements in Output

1. Cleaner timestamps: Millisecond precision without microseconds
2. Visual indicators: Emoji prefixes make it easy to scan logs
3. Clear LLM interactions: Request and response clearly marked with model info
4. Hierarchical context: Operations nested within agents
5. Timing information: Automatic duration tracking for operations
6. No noise: Mellea warnings suppressed, only relevant information shown
7. Structured data: Options and metadata clearly formatted

Benefits

1. Unified Experience: All agents and backends use the same logging format
2. Better Debugging: Clear visibility into LLM interactions and multi-turn conversations
3. Modern Agentic Style: Follows best practices for agent logging with visual indicators
4. Performance Tracking: Automatic timing for operations
5. Cleaner Output: Suppressed unnecessary warnings, focused on relevant information
6. Maintainability: Centralized logging logic, easier to extend and modify

Testing

Tested with multiple examples:

• example_01_write_report.py - Writer agent
• example_02_edit_report.py - Editor agent
• example_04_enrich_document.py - Enricher agent

[mergify]

Merge Protections

Your pull request matches the following merge protections and will not be merged until they are valid.

🟢 Enforce conventional commit

Wonderful, this rule succeeded.

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

• title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert)(?:\(.+\))?(!)?:

[github-actions]

✅ DCO Check Passed

Thanks @ceberam, all your commits are properly signed off. 🎉

[codecov]

Codecov Report

❌ Patch coverage is 26.05263% with 281 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
docling_agent/logging.py	42.30%	105 Missing ⚠️
docling_agent/agent/writer.py	2.85%	34 Missing ⚠️
docling_agent/agent/base_functions.py	17.94%	32 Missing ⚠️
docling_agent/agent/orchestrator.py	3.12%	31 Missing ⚠️
docling_agent/agent/editor.py	3.44%	28 Missing ⚠️
docling_agent/agent/enricher.py	10.00%	18 Missing ⚠️
docling_agent/agent/extractor.py	10.00%	9 Missing ⚠️
docling_agent/backends/ollama_backend.py	25.00%	6 Missing ⚠️
docling_agent/backends/openai_compatible.py	25.00%	6 Missing ⚠️
docling_agent/agent/library.py	16.66%	5 Missing ⚠️
... and 2 more

📢 Thoughts on this report? Let us know!

[PeterStaar-IBM]

lgtm!