Release 0.9.0

.claude/commands/generate-release.md

@@ -0,0 +1,97 @@
+Generate a changelog and release notes for the current branch.
+
+## Steps
+
+### 1. Gather context
+
+Run the following in parallel:
+- Find the latest tag matching `vX.Y.Z` pattern: `git tag --sort=-v:refname | grep -E '^v[0-9]+\.[0-9]+\.[0-9]+$' | head -1`
+- Read the current version from `ai4rag/__init__.py`
+- Get commits between that tag and HEAD: `git log <latest-tag>..HEAD --oneline`
+- Get the full diff summary: `git diff <latest-tag>..HEAD --stat`
+
+### 2. Assess the version
+
+Read the current `__version__` from `ai4rag/__init__.py`. Then reason explicitly about whether the version bump already applied is appropriate given the nature of the commits since the latest tag:
+
+- **Patch** (X.Y.Z+1): only bug fixes, doc updates, minor internal changes — no new public API
+- **Minor** (X.Y+1.0): new features or capabilities added, backward compatible
+- **Major** (X+1.0.0): breaking changes to public API or behaviour
+
+State your conclusion clearly:
+- If the version in `__init__.py` already reflects the right bump relative to the latest tag, confirm it and proceed.
+- If the version is insufficient (e.g., new features added but only patch was bumped), suggest the correct version and explain why. Ask the user to confirm before proceeding.
+
+Use the confirmed version (from `__init__.py` or the suggested one) as `RELEASE_VERSION` for all subsequent steps.
+
+### 3. Categorise the changes
+
+Analyse every commit since the latest tag (read relevant changed files if needed for clarity) and categorise changes into:
+- **Added**: new features, new parameters, new modules, new integrations
+- **Changed**: updates to existing behaviour, refactoring, dependency bumps, renamed things
+- **Fixed**: bug fixes
+- **Removed**: deleted features or deprecated items
+- **Deprecated**: features marked for future removal
+
+Omit categories that have no entries. Each bullet should be concrete and user-facing (what changed, not the internal implementation detail).
+
+### 4. Update `docs/about/changelog.md`
+
+Read the existing `docs/about/changelog.md` to understand the format and conventions (Keep a Changelog style, links to GitHub releases). Insert a new section for `RELEASE_VERSION` immediately after the `---` separator that follows the file header (before the current first release entry). Follow the exact same markdown structure used by existing entries:
+
+```markdown
+## [X.Y.Z](https://github.com/IBM/ai4rag/releases/tag/vX.Y.Z)
+
+### Added
+- ...
+
+### Changed
+- ...
+
+### Fixed
+- ...
+
+---
+```
+
+Do not touch the version numbering explanation or the "Release Process" / "Stay Updated" sections at the bottom.
+
+### 5. Generate the release description
+
+Create the file `local/releases/release-X.Y.Z.md` (create `local/releases/` directory if it does not exist). This file is a PR / release description in GitHub-flavoured markdown. It should include:
+
+```markdown
+# Release vX.Y.Z
+
+## Summary
+
+<2-4 sentence plain-English summary of the overall theme of this release — what it enables for users.>
+
+## Changes
+
+### Added
+- ...
+
+### Changed
+- ...
+
+### Fixed
+- ...
+
+## Migration notes
+
+<Any steps a user upgrading from the previous version needs to take. If there are no breaking changes or config changes, write "No breaking changes.">
+
+## Checklist
+
+- [ ] `__version__` in `ai4rag/__init__.py` updated to `X.Y.Z`
+- [ ] `docs/about/changelog.md` updated
+- [ ] All tests pass (`pytest`)
+- [ ] Docs build successfully
+```
+
+After writing the file, tell the user:
+- The release version used and whether the version in `__init__.py` was confirmed or needs updating.
+- The paths of the two files written.
+- If the version in `__init__.py` needs to be changed, remind the user to update it.
+- Remind the user to run `/update-docs` to review and update project documentation before tagging the release.

.claude/commands/update-docs.md

@@ -0,0 +1,85 @@
+Review and update project documentation to reflect code changes since the latest release.
+
+## Steps
+
+### 1. Gather context
+
+Run the following in parallel:
+- Find the latest tag: `git tag --sort=-v:refname | grep -E '^v[0-9]+\.[0-9]+\.[0-9]+$' | head -1`
+- Get the list of changed files since that tag: `git diff <latest-tag>..HEAD --name-only`
+- Get the diff summary: `git diff <latest-tag>..HEAD --stat`
+- Get the commit log: `git log <latest-tag>..HEAD --oneline`
+
+### 2. Map changed source files to documentation pages
+
+Using the changed-files list from Step 1, identify which documentation pages are potentially affected. Use the following mapping — a changed source file implies the corresponding documentation page(s) should be reviewed:
+
+| Source path pattern | Documentation pages to review |
+|---|---|
+| `ai4rag/core/experiment*` | `docs/api-reference/core/experiment.md`, `docs/user-guide/overview.md` |
+| `ai4rag/core/hpo/` | `docs/api-reference/core/hpo.md`, `docs/user-guide/optimizers.md` |
+| `ai4rag/core/pre_selector*` | `docs/api-reference/core/hpo.md`, `docs/architecture/core-components.md` |
+| `ai4rag/search_space/` | `docs/api-reference/search-space/search-space.md`, `docs/user-guide/search-space.md` |
+| `ai4rag/rag/chunking/` | `docs/api-reference/rag/chunking.md` |
+| `ai4rag/rag/embedding/` | `docs/api-reference/rag/embedding.md` |
+| `ai4rag/rag/vector_store/` | `docs/api-reference/rag/vector-stores.md`, `docs/user-guide/hybrid-search.md` |
+| `ai4rag/rag/retrieval/` | `docs/api-reference/rag/retrieval.md` |
+| `ai4rag/rag/foundation_model/` | `docs/api-reference/rag/foundation-models.md` |
+| `ai4rag/rag/template/` | `docs/architecture/rag-components.md`, `docs/architecture/data-flow.md` |
+| `ai4rag/evaluator/` | `docs/api-reference/evaluator/evaluator.md`, `docs/user-guide/evaluation.md` |
+| `ai4rag/utils/event_handler*` | `docs/user-guide/event-handlers.md` |
+| `pyproject.toml` | `docs/getting-started/installation.md`, `README.md` |
+| New or removed modules | `docs/architecture/overview.md`, `docs/architecture/core-components.md`, `docs/architecture/rag-components.md` |
+
+Also always include `README.md` and `docs/getting-started/quick-start.md` in the review when any public API signature changed (new parameters, renamed classes, changed defaults).
+
+If no source files under `ai4rag/` changed, report that no documentation review is needed and stop here.
+
+### 3. Review affected documentation pages
+
+For each documentation page identified in Step 2, read both the documentation page and the corresponding source file(s). Check for each of the following issues:
+
+- **Stale references**: class names, function names, parameter names, or module paths mentioned in the docs that no longer exist in the source code
+- **Undocumented additions**: new public classes, functions, parameters, or configuration options added since the last tag that are not yet covered in docs
+- **Outdated code examples**: snippets in the docs that use old API signatures, removed parameters, or renamed imports
+- **Dependency drift**: version requirements in `pyproject.toml` that changed but are not reflected in `docs/getting-started/installation.md` or `README.md`
+- **Architecture gaps**: structural changes (new modules, moved files, changed inheritance) not reflected in architecture docs
+- **Incorrect descriptions**: behavioural changes that make existing prose inaccurate (e.g., a default value changed, a validation rule added or removed)
+
+### 4. Present findings and apply updates
+
+Produce a summary table of findings before making any edits:
+
+```
+| Documentation page | Issue | Action |
+|---|---|---|
+| docs/api-reference/rag/vector-stores.md | `LSVectorStore` renamed to `OGXVectorStore` | Update all references |
+| docs/getting-started/installation.md | ogx-client bumped to ~=1.0.0 | Update version requirement |
+| ... | ... | ... |
+```
+
+If the table is empty, state "No documentation updates required — all pages are consistent with the current code." and skip to Step 5.
+
+Otherwise, apply the updates to each affected documentation page. When editing:
+- Preserve the existing formatting, heading structure, and admonition style
+- Keep code examples syntactically correct and runnable
+- Use the same terminology conventions as the rest of the docs
+- Do not add or remove sections unless the code changes warrant it
+- For API reference pages that use `mkdocstrings` directives (`::: module.path`), verify the module path is still valid rather than rewriting prose
+
+### 5. Verify documentation build
+
+Run `mkdocs build --strict` to confirm:
+- No broken cross-references or missing pages
+- No warnings about undefined references
+- Build completes successfully
+
+If the build fails, fix the issues and re-run until it passes.
+
+### 6. Report
+
+Tell the user:
+- How many documentation pages were reviewed
+- How many required updates (with the summary table from Step 4)
+- Whether the docs build passed
+- Any pages that may need deeper manual review (e.g., user-guide pages where behavioural nuance is hard to verify mechanically)

.gitignore

@@ -66,7 +66,11 @@ tmp/
 .DS_Store
 
 # AI code tools
-.claude
+.claude/*
+!.claude/commands/
+.claude/commands/*
+!.claude/commands/update-docs.md
+!.claude/commands/generate-release.md
 .cursor
 CLAUDE.md
 

ai4rag/__init__.py

@@ -5,7 +5,7 @@
 import logging
 import os
 
-__version__ = "0.8.2"
+__version__ = "0.9.0"
 
 logger = logging.getLogger("ai4rag")
 logger.setLevel(os.getenv("LOG_LEVEL", "INFO"))

docs/about/changelog.md

@@ -7,6 +7,40 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ---
 
+## [0.9.0](https://github.com/IBM/ai4rag/releases/tag/v0.9.0)
+
+### Added
+- Multilingual support — `Language` dataclass and `language` parameter on `BaseFoundationModel` for language-aware prompt template generation
+- `ai4rag.search_space.prepare.language_detection` module for LLM-based benchmark language detection with ISO 639-1 mapping
+- `CharApproxTokenizer` — lightweight, model-agnostic tokenizer approximating token count via character ratio, replacing the `tiktoken` dependency
+- `ai4rag.components.assets_generator.prompt_filters` module for filtering OGX runtime injection duplicates from HPO prompt templates during Responses API export
+- Progressive chunk truncation in `OGXEmbeddingModel` — oversized chunks are truncated before embedding instead of failing
+- `max_threads` parameter on `run_rag_optimization()` for controlling concurrent benchmark evaluation threads
+
+### Changed
+- **Breaking:** `BaseFoundationModel` constructor accepts a `language` parameter; `user_message_text` and `context_template_text` are now validated properties instead of `RAGPromptTemplateString` descriptors
+- **Breaking:** `BaseFoundationModel.chat()` signature now accepts `**kwargs`
+- **Breaking:** Search space report format changed from YAML to JSON — `search_space_preparation` and `rag_templates_optimization` no longer use `pyyaml`
+- **Breaking:** `OGXEmbeddingModel._embed_text()` renamed to `_call_embedding_api()`
+- Replaced `tiktoken` dependency with character-based token approximation across all chunkers
+- Upgraded `docling` dependency to `2.107.0` and adapted to new API
+- Prompt template system refactored — `RAGPromptTemplateString` descriptor replaced with setter-based validation via `validate_prompt_templates_placeholders()`
+- Responses API payload aligned with previous chat/completion format
+- Component functions (`text_extraction`, `search_space_preparation`, `rag_templates_optimization`) made more customizable with additional parameters
+- Removed `mike` dependency and documentation versioning from CI/CD
+
+### Fixed
+- Chunks exceeding embedding model context length now truncated with progressive margins instead of causing API failures
+- `random_state` parameter properly wired through `BaseOptimizer` to `GAMOptimizer` and `RandomOptimizer` for deterministic optimization runs
+- Removed unnecessary `docling` install cell from indexing notebook template
+
+### Removed
+- `tiktoken` dependency — replaced by `CharApproxTokenizer`
+- `RAGPromptTemplateString` descriptor class from `ai4rag.rag.foundation_models.utils`
+- YAML serialization support for model instances in search space reports
+
+---
+
 ## [0.8.1](https://github.com/IBM/ai4rag/releases/tag/v0.8.1)
 
 ### Changed

docs/architecture/rag-components.md

@@ -13,16 +13,17 @@ classDiagram
         +client: ClientT
         +model_id: str
         +params: ParamsT
+        +language: Language
         +system_message_text: str
         +user_message_text: str
         +context_template_text: str
-        +chat(messages)* list
+        +chat(messages, **kwargs)* list
     }
 
     class OGXFoundationModel {
         +client: OgxClient
         +params: OGXModelParameters
-        +chat(messages) list
+        +chat(messages, **kwargs) list
     }
 
     class BaseEmbeddingModel {
@@ -147,9 +148,14 @@ class BaseFoundationModel(Generic[ClientT, ParamsT], ABC):
         system_message_text: str | None = None,
         user_message_text: str | None = None,
         context_template_text: str | None = None,
+        language: Language | None = None,
     ):
 ```
 
+**Language-Aware Prompt Generation:**
+
+The optional `language` parameter accepts a `Language` dataclass (with `code` and `name` fields) and controls language-aware prompt template generation. When set, `user_message_text` is regenerated to include language-specific instructions. Defaults to `Language(code="", name="auto")`.
+
 **Configurable Prompt Templates:**
 
 Foundation models support three customizable prompt templates:
@@ -201,11 +207,15 @@ foundation_model = OGXFoundationModel(
 )
 ```
 
+**Prompt Template Validation:**
+
+The `user_message_text` and `context_template_text` attributes are validated properties that check for required placeholders (`{question}`, `{reference_documents}` in user message; `{document}` in context template) on assignment. Invalid templates raise a `ValueError`.
+
 **Interface Method:**
 
 ```python
 @abstractmethod
-def chat(self, messages: list[MessageTyped]) -> list[MessageTyped]:
+def chat(self, messages: list[MessageTyped], **kwargs) -> list[MessageTyped]:
     """Chat with the model based on the client capabilities."""
 ```
 
@@ -231,6 +241,7 @@ class OGXFoundationModel(BaseFoundationModel[OgxClient, OGXModelParameters]):
         system_message_text: str | None = None,
         user_message_text: str | None = None,
         context_template_text: str | None = None,
+        language: Language | None = None,
     ):
 ```
 
@@ -246,7 +257,7 @@ class OGXModelParameters:
 **Chat Implementation:**
 
 ```python
-def chat(self, messages: list[MessageTyped]) -> list[MessageTyped]:
+def chat(self, messages: list[MessageTyped], **kwargs) -> list[MessageTyped]:
     response = self.client.chat.completions.create(
         model=self.model_id,
         messages=messages,
@@ -337,6 +348,10 @@ class OGXEmbeddingParams:
 
 When `embedding_dimension` or `context_length` not provided, the model auto-detects them on first use:
 
+**Chunk Truncation:**
+
+When a chunk exceeds the embedding model's context length, `OGXEmbeddingModel` automatically truncates it using a progressive margin strategy (5%, then 10%) before retrying. This prevents embedding failures for oversized chunks while preserving as much content as possible.
+
 **Embedding Dimension Detection:**
 
 ```python
@@ -1229,7 +1244,7 @@ All RAG components are designed for extensibility:
 
 ```python
 class CustomFoundationModel(BaseFoundationModel[MyClient, MyParams]):
-    def chat(self, messages: list[MessageTyped]) -> list[MessageTyped]:
+    def chat(self, messages: list[MessageTyped], **kwargs) -> list[MessageTyped]:
         # Your implementation
         pass
 ```