Release 0.9.0

[jakub-walaszczyk]

Release v0.9.0

Summary

This release adds multilingual support to the RAG optimization pipeline, enabling language-aware prompt generation and automatic benchmark language detection. It also replaces the tiktoken dependency with a lightweight character-based token approximation, simplifies the search space report format (YAML to JSON), and improves embedding reliability with progressive chunk truncation for oversized inputs.

Changes

Added

• Multilingual support — Language dataclass and language parameter on BaseFoundationModel for language-aware prompt template generation
• ai4rag.search_space.prepare.language_detection module for LLM-based benchmark language detection with ISO 639-1 mapping
• CharApproxTokenizer — lightweight, model-agnostic tokenizer approximating token count via character ratio, replacing the tiktoken dependency
• ai4rag.components.assets_generator.prompt_filters module for filtering OGX runtime injection duplicates from HPO prompt templates during Responses API export
• Progressive chunk truncation in OGXEmbeddingModel — oversized chunks are truncated before embedding instead of failing
• max_threads parameter on run_rag_optimization() for controlling concurrent benchmark evaluation threads

Changed

• Breaking: BaseFoundationModel constructor accepts a language parameter; user_message_text and context_template_text are now validated properties instead of RAGPromptTemplateString descriptors
• Breaking: BaseFoundationModel.chat() signature now accepts **kwargs
• Breaking: Search space report format changed from YAML to JSON — search_space_preparation and rag_templates_optimization no longer use pyyaml
• Breaking: OGXEmbeddingModel._embed_text() renamed to _call_embedding_api()
• Replaced tiktoken dependency with character-based token approximation across all chunkers
• Upgraded docling dependency to 2.107.0 and adapted to new API
• Prompt template system refactored — RAGPromptTemplateString descriptor replaced with setter-based validation via validate_prompt_templates_placeholders()
• Responses API payload aligned with previous chat/completion format
• Component functions (text_extraction, search_space_preparation, rag_templates_optimization) made more customizable with additional parameters
• Removed mike dependency and documentation versioning from CI/CD

Fixed

• Chunks exceeding embedding model context length now truncated with progressive margins instead of causing API failures
• random_state parameter properly wired through BaseOptimizer to GAMOptimizer and RandomOptimizer for deterministic optimization runs
• Removed unnecessary docling install cell from indexing notebook template

Removed

• tiktoken dependency — replaced by CharApproxTokenizer
• RAGPromptTemplateString descriptor class from ai4rag.rag.foundation_models.utils
• YAML serialization support for model instances in search space reports

Migration notes

• Search space reports: Existing YAML reports (.yml/.yaml) must be converted to JSON before use with run_rag_optimization(). The prepare_search_space_report() function now writes JSON by default.
• Foundation model subclasses: If you subclass BaseFoundationModel, update chat() to accept **kwargs and note that user_message_text/context_template_text are now properties with validation. The RAGPromptTemplateString descriptor is removed.
• Embedding model: If you call OGXEmbeddingModel._embed_text() directly, rename to _call_embedding_api().
• Language parameter: The BaseFoundationModel constructor now accepts an optional language parameter (defaults to Language(code="", name="auto")). No action required unless you want to specify a language explicitly.

Checklist

• __version__ in ai4rag/__init__.py updated to 0.9.0
• docs/about/changelog.md updated
• All tests pass (pytest)
• Docs build successfully