diff --git a/CHANGELOG.md b/CHANGELOG.md index 259f2f61ee..250d70cc06 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,39 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [0.8.10] - 2026-03-08 + +### Added + +- ๐Ÿ” **Custom OIDC logout endpoint.** Administrators can now configure a custom OpenID Connect logout URL via OPENID_END_SESSION_ENDPOINT, enabling logout functionality for OIDC providers that require custom endpoints like AWS Cognito. [Commit](https://github.com/open-webui/open-webui/commit/3f350f865920daf2844769a758b2d2e6a7ee3efa) +- ๐Ÿ—„๏ธ **MariaDB Vector community support.** Added MariaDB Vector as a new vector database backend, enabling deployments with VECTOR_DB=mariadb-vector; supports cosine and euclidean distance strategies with configurable HNSW indexing. [#21931](https://github.com/open-webui/open-webui/pull/21931) +- ๐Ÿ“ **Task message truncation.** Chat messages sent to task models for title and tag generation can now be truncated using a filter in the prompt template, reducing token usage and processing time for long conversations. [#21499](https://github.com/open-webui/open-webui/issues/21499) +- ๐Ÿ”„ **General improvements.** Various improvements were implemented across the application to enhance performance, stability, and security. +- ๐ŸŒ Translations for Portuguese (Brazil), Spanish, and Malay were enhanced and expanded. + +### Fixed + +- ๐Ÿ”— **Pipeline filter HTTP errors.** Fixed a bug where HTTP errors in pipeline inlet/outlet filters would silently corrupt the user's chat payload; errors are now properly raised before parsing the response. [#22445](https://github.com/open-webui/open-webui/pull/22445) +- ๐Ÿ“š **Knowledge file embedding updates.** Fixed a bug where updating knowledge files left old embeddings in the database, causing search results to include duplicate and stale data. [#20558](https://github.com/open-webui/open-webui/issues/20558) +- ๐Ÿ“ **Files list stability.** Fixed the files list ordering to use created_at with id as secondary sort, ensuring consistent ordering and preventing page crashes when managing many files. [#21879](https://github.com/open-webui/open-webui/issues/21879) +- ๐Ÿ“จ **Teams webhook crash.** Fixed a TypeError crash in the Teams webhook handler when user data is missing from the event payload. [#22444](https://github.com/open-webui/open-webui/pull/22444) +- ๐Ÿ› ๏ธ **Process shutdown handling.** Fixed bare except clauses in the main process that prevented clean shutdown; replaced with proper exception handling. [#22423](https://github.com/open-webui/open-webui/pull/22423) +- ๐Ÿณ **Docker deployment startup.** Docker deployments now start correctly; the missing OpenTelemetry system metrics dependency was added. [#22447](https://github.com/open-webui/open-webui/pull/22447), [#22401](https://github.com/open-webui/open-webui/issues/22401) +- ๐Ÿ› ๏ธ **Tool access for non-admin users.** Fixed a NameError that prevented non-admin users from viewing tools; the missing has_access function is now properly imported. [#22393](https://github.com/open-webui/open-webui/issues/22393) +- ๐Ÿ” **OAuth error handling.** Fixed a bug where bare except clauses silently caught SystemExit and KeyboardInterrupt, preventing clean process shutdown during OAuth authentication. [#22420](https://github.com/open-webui/open-webui/pull/22420) +- ๐Ÿ› ๏ธ **Exception error messages.** Fixed three locations where incorrect exception raising caused confusing TypeError messages instead of proper error descriptions, making debugging much easier. [#22446](https://github.com/open-webui/open-webui/pull/22446) +- ๐Ÿ“„ **YAML file processing.** Fixed an error when uploading YAML files with Docling enabled; YAML and YML files are now properly recognized as text files and processed correctly. [#22399](https://github.com/open-webui/open-webui/pull/22399), [#22263](https://github.com/open-webui/open-webui/issues/22263) +- ๐Ÿ“… **Time range month names.** Fixed month names in time range labels appearing in the wrong language when OS regional settings differ from browser language; month names now consistently display in English. [#22454](https://github.com/open-webui/open-webui/pull/22454) +- ๐Ÿ” **OAuth error URL encoding.** Fixed OAuth error messages with special characters causing malformed redirect URLs; error messages are now properly URL-encoded. [#22415](https://github.com/open-webui/open-webui/pull/22415) +- ๐Ÿ› ๏ธ **Internal tool method filtering.** Tools no longer expose internal methods starting with underscore to the LLM, reducing clutter and improving accuracy. [#22408](https://github.com/open-webui/open-webui/pull/22408) +- ๐Ÿ”Š **Azure TTS locale extraction.** Fixed Azure text-to-speech using incomplete locale codes in SSML; now correctly uses full locale like "en-US" instead of just "en". [#22443](https://github.com/open-webui/open-webui/pull/22443) +- ๐ŸŽค **Azure speech transcription errors.** Improved Azure AI Speech error handling to display user-friendly messages instead of generic connection errors; empty transcripts, no language identified, and other Azure-specific errors now show clear descriptions. [#20485](https://github.com/open-webui/open-webui/issues/20485) +- ๐Ÿ“Š **Analytics group filtering.** Fixed token usage analytics not being filtered by user group; the query now properly respects group filters like other analytics metrics. [#22167](https://github.com/open-webui/open-webui/pull/22167) +- ๐Ÿ” **Web search favicon fallback.** Fixed web search sources showing broken image icons when favicons couldn't be loaded from external sources; now falls back to the default Open WebUI favicon. [#21897](https://github.com/open-webui/open-webui/pull/21897) +- ๐Ÿ”„ **Custom model fallback.** Fixed custom model fallback not working when the base model is unavailable; the base model ID is now correctly retrieved from model info instead of empty params. [#22456](https://github.com/open-webui/open-webui/issues/22456) +- ๐Ÿ–ผ๏ธ **Pending message image display.** Fixed images in queued messages appearing blank; image thumbnails are now properly displayed in the pending message queue. [#22256](https://github.com/open-webui/open-webui/issues/22256) +- ๐Ÿ› ๏ธ **File metadata sanitization.** Fixed file uploads failing with JSON serialization errors when metadata contained non-serializable objects like callable functions; metadata is now sanitized before database insertion. [#20561](https://github.com/open-webui/open-webui/issues/20561) + ## [0.8.9] - 2026-03-07 ### Added diff --git a/CHANGELOG_EXTRA.md b/CHANGELOG_EXTRA.md index be0201145f..cc3bd517c7 100644 --- a/CHANGELOG_EXTRA.md +++ b/CHANGELOG_EXTRA.md @@ -5,6 +5,12 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [0.8.10.1] - 2026.03.09 + +### Changed + +- ๅˆๅนถๅฎ˜ๆ–น 0.8.10 ๆ”นๅŠจ + ## [0.8.9.1] - 2026.03.08 ### Changed diff --git a/Dockerfile b/Dockerfile index 9a10236076..bb9a767fc6 100644 --- a/Dockerfile +++ b/Dockerfile @@ -128,6 +128,7 @@ RUN chown -R $UID:$GID /app $HOME RUN apt-get update && \ apt-get install -y --no-install-recommends \ git build-essential pandoc gcc netcat-openbsd curl jq \ + libmariadb-dev \ python3-dev \ ffmpeg libsm6 libxext6 zstd \ && rm -rf /var/lib/apt/lists/* diff --git a/backend/open_webui/config.py b/backend/open_webui/config.py index b0b51254e4..cbb782fc1d 100644 --- a/backend/open_webui/config.py +++ b/backend/open_webui/config.py @@ -447,6 +447,12 @@ def __getattr__(self, key): os.environ.get("OPENID_PROVIDER_URL", ""), ) +OPENID_END_SESSION_ENDPOINT = PersistentConfig( + "OPENID_END_SESSION_ENDPOINT", + "oauth.oidc.end_session_endpoint", + os.environ.get("OPENID_END_SESSION_ENDPOINT", ""), +) + OPENID_REDIRECT_URI = PersistentConfig( "OPENID_REDIRECT_URI", "oauth.oidc.redirect_uri", @@ -821,13 +827,18 @@ def feishu_oauth_register(oauth: OAuth): if FEISHU_CLIENT_ID.value: configured_providers.append("Feishu") - if configured_providers and not OPENID_PROVIDER_URL.value: + if ( + configured_providers + and not OPENID_PROVIDER_URL.value + and not OPENID_END_SESSION_ENDPOINT.value + ): provider_list = ", ".join(configured_providers) log.warning( f"โš ๏ธ OAuth providers configured ({provider_list}) but OPENID_PROVIDER_URL not set - logout will not work!" ) log.warning( - f"Set OPENID_PROVIDER_URL to your OAuth provider's OpenID Connect discovery endpoint to fix logout functionality." + f"Set OPENID_PROVIDER_URL to your OAuth provider's OpenID Connect discovery endpoint," + f" or set OPENID_END_SESSION_ENDPOINT to a custom logout URL to fix logout functionality." ) @@ -2311,6 +2322,81 @@ class BannerModel(BaseModel): CHROMA_HTTP_SSL = os.environ.get("CHROMA_HTTP_SSL", "false").lower() == "true" # this uses the model defined in the Dockerfile ENV variable. If you dont use docker or docker based deployments such as k8s, the default embedding model will be used (sentence-transformers/all-MiniLM-L6-v2) + +# MariaDB Vector (mariadb-vector) +MARIADB_VECTOR_DB_URL = os.environ.get("MARIADB_VECTOR_DB_URL", "").strip() + +MARIADB_VECTOR_INITIALIZE_MAX_VECTOR_LENGTH = int( + os.environ.get("MARIADB_VECTOR_INITIALIZE_MAX_VECTOR_LENGTH", "1536").strip() + or "1536" +) + +# Distance strategy: +# - cosine => vec_distance_cosine(...) +# - euclidean => vec_distance_euclidean(...) +MARIADB_VECTOR_DISTANCE_STRATEGY = ( + os.environ.get("MARIADB_VECTOR_DISTANCE_STRATEGY", "cosine").strip().lower() +) + +# HNSW M parameter (MariaDB VECTOR INDEX ... M=) +MARIADB_VECTOR_INDEX_M = int( + os.environ.get("MARIADB_VECTOR_INDEX_M", "8").strip() or "8" +) + +# Pooling (MariaDB-Vector) +MARIADB_VECTOR_POOL_SIZE = os.environ.get("MARIADB_VECTOR_POOL_SIZE", None) + +if MARIADB_VECTOR_POOL_SIZE != None: + try: + MARIADB_VECTOR_POOL_SIZE = int(MARIADB_VECTOR_POOL_SIZE) + except Exception: + MARIADB_VECTOR_POOL_SIZE = None + +MARIADB_VECTOR_POOL_MAX_OVERFLOW = os.environ.get("MARIADB_VECTOR_POOL_MAX_OVERFLOW", 0) + +if MARIADB_VECTOR_POOL_MAX_OVERFLOW == "": + MARIADB_VECTOR_POOL_MAX_OVERFLOW = 0 +else: + try: + MARIADB_VECTOR_POOL_MAX_OVERFLOW = int(MARIADB_VECTOR_POOL_MAX_OVERFLOW) + except Exception: + MARIADB_VECTOR_POOL_MAX_OVERFLOW = 0 + +MARIADB_VECTOR_POOL_TIMEOUT = os.environ.get("MARIADB_VECTOR_POOL_TIMEOUT", 30) + +if MARIADB_VECTOR_POOL_TIMEOUT == "": + MARIADB_VECTOR_POOL_TIMEOUT = 30 +else: + try: + MARIADB_VECTOR_POOL_TIMEOUT = int(MARIADB_VECTOR_POOL_TIMEOUT) + except Exception: + MARIADB_VECTOR_POOL_TIMEOUT = 30 + +MARIADB_VECTOR_POOL_RECYCLE = os.environ.get("MARIADB_VECTOR_POOL_RECYCLE", 3600) + +if MARIADB_VECTOR_POOL_RECYCLE == "": + MARIADB_VECTOR_POOL_RECYCLE = 3600 +else: + try: + MARIADB_VECTOR_POOL_RECYCLE = int(MARIADB_VECTOR_POOL_RECYCLE) + except Exception: + MARIADB_VECTOR_POOL_RECYCLE = 3600 + +ENABLE_MARIADB_VECTOR = True +if VECTOR_DB == "mariadb-vector": + if not MARIADB_VECTOR_DB_URL: + ENABLE_MARIADB_VECTOR = False + else: + try: + parsed = urlparse(MARIADB_VECTOR_DB_URL) + scheme = (parsed.scheme or "").lower() + # Require official driver so VECTOR binds as float32 bytes correctly + if scheme != "mariadb+mariadbconnector": + ENABLE_MARIADB_VECTOR = False + except Exception: + ENABLE_MARIADB_VECTOR = False + + # Milvus MILVUS_URI = os.environ.get("MILVUS_URI", f"{DATA_DIR}/vector_db/milvus.db") MILVUS_DB = os.environ.get("MILVUS_DB", "default") diff --git a/backend/open_webui/main.py b/backend/open_webui/main.py index d10e518f4a..b7d96becde 100644 --- a/backend/open_webui/main.py +++ b/backend/open_webui/main.py @@ -1804,8 +1804,8 @@ async def chat_completion( } # Check base model existence for custom models - if model_info_params.get("base_model_id"): - base_model_id = model_info_params.get("base_model_id") + if model_info and model_info.base_model_id: + base_model_id = model_info.base_model_id if base_model_id not in request.app.state.MODELS: if ENABLE_CUSTOM_MODEL_FALLBACK: default_models = ( @@ -1940,7 +1940,7 @@ async def process_chat(request, form_data, user, metadata, model): "model": model_id, }, ) - except: + except Exception: pass ctx = build_chat_response_context( @@ -1987,7 +1987,7 @@ async def process_chat(request, form_data, user, metadata, model): {"type": "chat:tasks:cancel"}, ) - except: + except Exception: pass finally: try: diff --git a/backend/open_webui/models/chat_messages.py b/backend/open_webui/models/chat_messages.py index b707ab358b..00609ce7f0 100644 --- a/backend/open_webui/models/chat_messages.py +++ b/backend/open_webui/models/chat_messages.py @@ -420,11 +420,13 @@ def get_token_usage_by_user( self, start_date: Optional[int] = None, end_date: Optional[int] = None, + group_id: Optional[str] = None, db: Optional[Session] = None, ) -> dict[str, dict]: """Aggregate token usage by user using database-level aggregation.""" with get_db_context(db) as db: from sqlalchemy import func, cast, Integer + from open_webui.models.groups import GroupMember dialect = db.bind.dialect.name @@ -464,6 +466,13 @@ def get_token_usage_by_user( query = query.filter(ChatMessage.created_at >= start_date) if end_date: query = query.filter(ChatMessage.created_at <= end_date) + if group_id: + group_users = ( + db.query(GroupMember.user_id) + .filter(GroupMember.group_id == group_id) + .subquery() + ) + query = query.filter(ChatMessage.user_id.in_(group_users)) results = query.group_by(ChatMessage.user_id).all() diff --git a/backend/open_webui/models/files.py b/backend/open_webui/models/files.py index 09060f9bde..84dd43f5e8 100644 --- a/backend/open_webui/models/files.py +++ b/backend/open_webui/models/files.py @@ -4,6 +4,7 @@ from sqlalchemy.orm import Session from open_webui.internal.db import Base, JSONField, get_db, get_db_context +from open_webui.utils.misc import sanitize_metadata from pydantic import BaseModel, ConfigDict, model_validator from sqlalchemy import BigInteger, Column, String, Text, JSON @@ -127,9 +128,16 @@ def insert_new_file( self, user_id: str, form_data: FileForm, db: Optional[Session] = None ) -> Optional[FileModel]: with get_db_context(db) as db: + file_data = form_data.model_dump() + + # Sanitize meta to remove non-JSON-serializable objects + # (e.g. callable tool functions, MCP client instances from middleware) + if file_data.get("meta"): + file_data["meta"] = sanitize_metadata(file_data["meta"]) + file = FileModel( **{ - **form_data.model_dump(), + **file_data, "user_id": user_id, "created_at": int(time.time()), "updated_at": int(time.time()), @@ -289,7 +297,7 @@ def search_files( db: Optional database session. Returns: - List of matching FileModel objects, ordered by updated_at descending. + List of matching FileModel objects, ordered by created_at descending. """ with get_db_context(db) as db: query = db.query(File) @@ -303,7 +311,7 @@ def search_files( return [ FileModel.model_validate(file) - for file in query.order_by(File.updated_at.desc()) + for file in query.order_by(File.created_at.desc(), File.id.desc()) .offset(skip) .limit(limit) .all() diff --git a/backend/open_webui/retrieval/loaders/main.py b/backend/open_webui/retrieval/loaders/main.py index 83adb8823f..d4cc4391aa 100644 --- a/backend/open_webui/retrieval/loaders/main.py +++ b/backend/open_webui/retrieval/loaders/main.py @@ -86,6 +86,9 @@ "hs", "lhs", "json", + "yaml", + "yml", + "toml", ] diff --git a/backend/open_webui/retrieval/vector/dbs/elasticsearch.py b/backend/open_webui/retrieval/vector/dbs/elasticsearch.py index e209453f5c..dfb02ec029 100644 --- a/backend/open_webui/retrieval/vector/dbs/elasticsearch.py +++ b/backend/open_webui/retrieval/vector/dbs/elasticsearch.py @@ -1,3 +1,7 @@ +""" +NOTE: This vector database integration is community-supported and maintained on a best-effort basis. +""" + from elasticsearch import Elasticsearch, BadRequestError from typing import Optional import ssl diff --git a/backend/open_webui/retrieval/vector/dbs/mariadb_vector.py b/backend/open_webui/retrieval/vector/dbs/mariadb_vector.py new file mode 100644 index 0000000000..dfcfb3da59 --- /dev/null +++ b/backend/open_webui/retrieval/vector/dbs/mariadb_vector.py @@ -0,0 +1,593 @@ +""" +NOTE: This vector database integration is community-supported and maintained on a best-effort basis. +""" + +from __future__ import annotations + +import array +import json +import logging +import math +import re +import sys +from contextlib import contextmanager +from typing import Any, Dict, List, Optional, Tuple + +from sqlalchemy import create_engine +from sqlalchemy.pool import NullPool, QueuePool + +from open_webui.config import ( + MARIADB_VECTOR_DB_URL, + MARIADB_VECTOR_DISTANCE_STRATEGY, + MARIADB_VECTOR_INDEX_M, + MARIADB_VECTOR_INITIALIZE_MAX_VECTOR_LENGTH, + MARIADB_VECTOR_POOL_SIZE, + MARIADB_VECTOR_POOL_MAX_OVERFLOW, + MARIADB_VECTOR_POOL_TIMEOUT, + MARIADB_VECTOR_POOL_RECYCLE, +) +from open_webui.retrieval.vector.main import ( + GetResult, + SearchResult, + VectorDBBase, + VectorItem, +) +from open_webui.retrieval.vector.utils import process_metadata + +log = logging.getLogger(__name__) + +VECTOR_LENGTH = int(MARIADB_VECTOR_INITIALIZE_MAX_VECTOR_LENGTH) + + +def _embedding_to_f32_bytes(vec: List[float]) -> bytes: + """ + Convert a Python float vector into the binary payload expected by MariaDB VECTOR. + + MariaDB Vector expects the vector argument to be bound as a little-endian float32 + byte sequence. We use array('f') to avoid a numpy dependency and byteswap on + big-endian platforms for portability. + """ + a = array.array("f", [float(x) for x in vec]) # float32 + if sys.byteorder != "little": + a.byteswap() + return a.tobytes() + + +def _safe_json(v: Any) -> Dict[str, Any]: + """ + Normalize a potentially JSON-like value into a Python dict. + + Accepts: + - dict: returned as-is + - str / bytes: parsed as JSON if possible + - None / other types: returns {} + """ + if v is None: + return {} + if isinstance(v, dict): + return v + if isinstance(v, (bytes, bytearray)): + try: + v = v.decode("utf-8") + except Exception: + return {} + if isinstance(v, str): + try: + j = json.loads(v) + return j if isinstance(j, dict) else {} + except Exception: + return {} + return {} + + +class MariaDBVectorClient(VectorDBBase): + """ + MariaDB + MariaDB Vector backend using DBAPI cursor parameter binding. + + IMPORTANT: + - Intended for: mariadb+mariadbconnector://... (official MariaDB driver). + - Uses qmark ("?") params and binds vectors as float32 bytes. + - Uses binary binding for BOTH inserts/updates and distance computations. + """ + + def __init__( + self, + db_url: Optional[str] = None, + vector_length: int = VECTOR_LENGTH, + distance_strategy: str = MARIADB_VECTOR_DISTANCE_STRATEGY, + index_m: int = MARIADB_VECTOR_INDEX_M, + ) -> None: + """ + Initialize a MariaDB Vector-backed VectorDBBase implementation. + + Validates URL scheme/driver requirements, ensures schema exists, and guards + against dimension mismatch with an existing VECTOR(n) column. + """ + self.db_url = (db_url or MARIADB_VECTOR_DB_URL).strip() + self.vector_length = int(vector_length) + self.distance_strategy = (distance_strategy or "cosine").strip().lower() + self.index_m = int(index_m) + + if self.distance_strategy not in {"cosine", "euclidean"}: + raise ValueError("distance_strategy must be 'cosine' or 'euclidean'") + + if not self.db_url.lower().startswith("mariadb+mariadbconnector://"): + raise ValueError( + "MariaDBVectorClient requires mariadb+mariadbconnector:// (official MariaDB driver) " + "to ensure qmark paramstyle and correct VECTOR binding." + ) + + if isinstance(MARIADB_VECTOR_POOL_SIZE, int): + if MARIADB_VECTOR_POOL_SIZE > 0: + self.engine = create_engine( + self.db_url, + pool_size=MARIADB_VECTOR_POOL_SIZE, + max_overflow=MARIADB_VECTOR_POOL_MAX_OVERFLOW, + pool_timeout=MARIADB_VECTOR_POOL_TIMEOUT, + pool_recycle=MARIADB_VECTOR_POOL_RECYCLE, + pool_pre_ping=True, + poolclass=QueuePool, + ) + else: + self.engine = create_engine( + self.db_url, pool_pre_ping=True, poolclass=NullPool + ) + else: + self.engine = create_engine(self.db_url, pool_pre_ping=True) + self._init_schema() + self._check_vector_length() + + @contextmanager + def _connect(self): + """ + Yield a context-managed DBAPI connection (SQLAlchemy raw_connection()). + + Callers can use: + with self._connect() as conn: + with conn.cursor() as cur: + ... + """ + conn = self.engine.raw_connection() + try: + yield conn + finally: + try: + conn.close() + except Exception: + pass + + def _init_schema(self) -> None: + """ + Create the backing table and vector index if they do not exist. + + Uses a PK definition compatible with MariaDB Vector's VECTOR INDEX key-size constraints. + """ + with self._connect() as conn: + with conn.cursor() as cur: + try: + dist = self.distance_strategy + cur.execute(f""" + CREATE TABLE IF NOT EXISTS document_chunk ( + -- MariaDB Vector requires the table PRIMARY KEY used with a VECTOR INDEX to be <= 256 bytes. + -- VARCHAR has internal length/metadata overhead, so VARCHAR(255) can exceed the 256-byte limit. + -- We use VARCHAR(254) to stay safely under the limit, and force ASCII (1 byte/char) so the byte + -- size is predictable (avoid utf8mb4 where a "255 char" key could be up to 1020 bytes). + -- ascii_bin gives bytewise, case-sensitive comparisons for stable ID matching. + id VARCHAR(254) CHARACTER SET ascii COLLATE ascii_bin PRIMARY KEY, + embedding VECTOR({self.vector_length}) NOT NULL, + collection_name VARCHAR(255) NOT NULL, + text LONGTEXT NULL, + vmetadata JSON NULL, + VECTOR INDEX (embedding) M={self.index_m} DISTANCE={dist}, + INDEX idx_document_chunk_collection_name (collection_name) + ) ENGINE=InnoDB; + """) + conn.commit() + except Exception as e: + conn.rollback() + log.exception(f"Error during database initialization: {e}") + raise + + def _check_vector_length(self) -> None: + """ + Validate that the existing VECTOR column dimension matches this client's configured dimension. + + Dimension guard: if table already exists with + a different VECTOR(n), refuse to silently mismatch. + """ + with self._connect() as conn: + with conn.cursor() as cur: + cur.execute("SHOW CREATE TABLE document_chunk") + row = cur.fetchone() + if not row or len(row) < 2: + return + ddl = row[1] + m = re.search(r"vector\\((\\d+)\\)", ddl, flags=re.IGNORECASE) + if not m: + return + existing = int(m.group(1)) + if existing != int(self.vector_length): + raise Exception( + f"VECTOR_LENGTH {self.vector_length} does not match existing vector column dimension {existing}. " + "Cannot change vector size after initialization without migrating the data." + ) + + def adjust_vector_length(self, vector: List[float]) -> List[float]: + """ + Pad or truncate a vector to match `self.vector_length`. + """ + n = len(vector) + if n < self.vector_length: + return vector + [0.0] * (self.vector_length - n) + if n > self.vector_length: + return vector[: self.vector_length] + return vector + + def _dist_fn(self) -> str: + """ + Return the MariaDB Vector distance function name for the configured strategy. + """ + return ( + "vec_distance_cosine" + if self.distance_strategy == "cosine" + else "vec_distance_euclidean" + ) + + def _score_from_dist(self, dist: float) -> float: + """ + Convert a DB distance value into a normalized score in (0, 1]. + + - cosine: score ~= 1 - cosine_distance, clamped to [0, 1] + - euclidean: score = 1 / (1 + dist) + """ + if self.distance_strategy == "cosine": + score = 1.0 - dist + if score < 0.0: + score = 0.0 + if score > 1.0: + score = 1.0 + return score + return 1.0 / (1.0 + max(0.0, dist)) + + def _build_filter_sql_qmark(self, expr: Any) -> Tuple[str, List[Any]]: + """ + Build a WHERE-clause fragment and qmark params from a minimal Mongo-like filter. + + Supported forms: + - {"field": "v"} + - {"field": {"$in": ["a","b"]}} + - {"$and": [ ... ]} + - {"$or": [ ... ]} + """ + if not expr or not isinstance(expr, dict): + return "", [] + + if "$and" in expr: + parts: List[str] = [] + params: List[Any] = [] + for e in expr.get("$and") or []: + s, p = self._build_filter_sql_qmark(e) + if s: + parts.append(s) + params.extend(p) + return ("(" + " AND ".join(parts) + ")") if parts else "", params + + if "$or" in expr: + parts: List[str] = [] + params: List[Any] = [] + for e in expr.get("$or") or []: + s, p = self._build_filter_sql_qmark(e) + if s: + parts.append(s) + params.extend(p) + return ("(" + " OR ".join(parts) + ")") if parts else "", params + + clauses: List[str] = [] + params: List[Any] = [] + for key, value in expr.items(): + if key.startswith("$"): + continue + json_expr = f"JSON_UNQUOTE(JSON_EXTRACT(vmetadata, '$.{key}'))" + if isinstance(value, dict) and "$in" in value: + vals = [str(v) for v in (value.get("$in") or [])] + if not vals: + clauses.append("0=1") + continue + ors = [] + for v in vals: + ors.append(f"{json_expr} = ?") + params.append(v) + clauses.append("(" + " OR ".join(ors) + ")") + else: + clauses.append(f"{json_expr} = ?") + params.append(str(value)) + return ("(" + " AND ".join(clauses) + ")") if clauses else "", params + + def insert(self, collection_name: str, items: List[VectorItem]) -> None: + """ + Insert items into the given collection (best-effort, ignores duplicates). + + Uses executemany() with binary VECTOR binding for high-throughput ingestion. + """ + if not items: + return + with self._connect() as conn: + with conn.cursor() as cur: + try: + sql = """ + INSERT IGNORE INTO document_chunk + (id, embedding, collection_name, text, vmetadata) + VALUES + (?, ?, ?, ?, ?) + """ + params: List[Tuple[Any, ...]] = [] + for item in items: + v = self.adjust_vector_length(item["vector"]) + emb = _embedding_to_f32_bytes(v) + meta = process_metadata(item.get("metadata") or {}) + params.append( + ( + item["id"], + emb, + collection_name, + item.get("text"), + json.dumps(meta), + ) + ) + cur.executemany(sql, params) + conn.commit() + except Exception as e: + conn.rollback() + log.exception(f"Error during insert: {e}") + raise + + def upsert(self, collection_name: str, items: List[VectorItem]) -> None: + """ + Insert or update items in the given collection by primary key. + + Uses executemany() and updates embedding/text/metadata on conflicts. + """ + if not items: + return + with self._connect() as conn: + with conn.cursor() as cur: + try: + sql = """ + INSERT INTO document_chunk + (id, embedding, collection_name, text, vmetadata) + VALUES + (?, ?, ?, ?, ?) + ON DUPLICATE KEY UPDATE + embedding = VALUES(embedding), + collection_name = VALUES(collection_name), + text = VALUES(text), + vmetadata = VALUES(vmetadata) + """ + params: List[Tuple[Any, ...]] = [] + for item in items: + v = self.adjust_vector_length(item["vector"]) + emb = _embedding_to_f32_bytes(v) + meta = process_metadata(item.get("metadata") or {}) + params.append( + ( + item["id"], + emb, + collection_name, + item.get("text"), + json.dumps(meta), + ) + ) + cur.executemany(sql, params) + conn.commit() + except Exception as e: + conn.rollback() + log.exception(f"Error during upsert: {e}") + raise + + def search( + self, + collection_name: str, + vectors: List[List[float]], + filter: Optional[Dict[str, Any]] = None, + limit: int = 10, + ) -> Optional[SearchResult]: + """ + Perform a vector similarity search. + + Args: + collection_name: Logical collection partition key. + vectors: One or more query vectors. + filter: Optional metadata filter (Mongo-like subset). + limit: Top-k per query vector. + + Returns a SearchResult where distances are normalized scores (higher is better). + """ + if not vectors: + return None + + dist_fn = self._dist_fn() + ids: List[List[str]] = [[] for _ in vectors] + distances: List[List[float]] = [[] for _ in vectors] + documents: List[List[str]] = [[] for _ in vectors] + metadatas: List[List[Any]] = [[] for _ in vectors] + + try: + with self._connect() as conn: + with conn.cursor() as cur: + fsql, fparams = self._build_filter_sql_qmark(filter or {}) + where = "collection_name = ?" + base_params: List[Any] = [collection_name] + if fsql: + where = where + " AND " + fsql + base_params.extend(fparams) + + sql = f""" + SELECT + id, + text, + vmetadata, + {dist_fn}(embedding, ?) AS dist + FROM document_chunk + WHERE {where} + ORDER BY dist ASC + LIMIT ? + """ + + for q_idx, q in enumerate(vectors): + qv = self.adjust_vector_length(q) + qbin = _embedding_to_f32_bytes(qv) + params = [qbin] + list(base_params) + [int(limit)] + cur.execute(sql, params) + rows = cur.fetchall() + + for r in rows: + rid, rtext, rmeta, rdist = r[0], r[1], r[2], r[3] + ids[q_idx].append(str(rid)) + try: + dist = float(rdist) if rdist is not None else 1.0 + except Exception: + dist = 1.0 + if math.isnan(dist) or math.isinf(dist): + dist = 1.0 + distances[q_idx].append(self._score_from_dist(dist)) + documents[q_idx].append(rtext) + metadatas[q_idx].append(_safe_json(rmeta)) + + return SearchResult( + ids=ids, + distances=distances, + documents=documents, + metadatas=metadatas, + ) + except Exception as e: + log.exception(f"[MARIADB_VECTOR] search() failed: {e}") + return None + + def query( + self, collection_name: str, filter: Dict[str, Any], limit: Optional[int] = None + ) -> Optional[GetResult]: + """ + Retrieve documents by metadata filter (non-vector query). + """ + with self._connect() as conn: + with conn.cursor() as cur: + fsql, fparams = self._build_filter_sql_qmark(filter or {}) + where = "collection_name = ?" + params: List[Any] = [collection_name] + if fsql: + where = where + " AND " + fsql + params.extend(fparams) + sql = f"SELECT id, text, vmetadata FROM document_chunk WHERE {where}" + if limit is not None: + sql += " LIMIT ?" + params.append(int(limit)) + cur.execute(sql, params) + rows = cur.fetchall() + if not rows: + return None + ids = [[str(r[0]) for r in rows]] + documents = [[r[1] for r in rows]] + metadatas = [[_safe_json(r[2]) for r in rows]] + return GetResult(ids=ids, documents=documents, metadatas=metadatas) + + def get( + self, collection_name: str, limit: Optional[int] = None + ) -> Optional[GetResult]: + """ + Retrieve documents in a collection without filtering (optionally limited). + """ + with self._connect() as conn: + with conn.cursor() as cur: + sql = "SELECT id, text, vmetadata FROM document_chunk WHERE collection_name = ?" + params: List[Any] = [collection_name] + if limit is not None: + sql += " LIMIT ?" + params.append(int(limit)) + cur.execute(sql, params) + rows = cur.fetchall() + if not rows: + return None + ids = [[str(r[0]) for r in rows]] + documents = [[r[1] for r in rows]] + metadatas = [[_safe_json(r[2]) for r in rows]] + return GetResult(ids=ids, documents=documents, metadatas=metadatas) + + def delete( + self, + collection_name: str, + ids: Optional[List[str]] = None, + filter: Optional[Dict[str, Any]] = None, + ) -> None: + """ + Delete rows from a collection by id list and/or metadata filter. + + If both are provided, they are combined with AND semantics. + """ + with self._connect() as conn: + with conn.cursor() as cur: + try: + where = ["collection_name = ?"] + params: List[Any] = [collection_name] + + if ids: + ph = ", ".join(["?"] * len(ids)) + where.append(f"id IN ({ph})") + params.extend(ids) + + if filter: + fsql, fparams = self._build_filter_sql_qmark(filter) + if fsql: + where.append(fsql) + params.extend(fparams) + + sql = "DELETE FROM document_chunk WHERE " + " AND ".join(where) + cur.execute(sql, params) + conn.commit() + except Exception as e: + conn.rollback() + log.exception(f"Error during delete: {e}") + raise + + def reset(self) -> None: + """ + Truncate the vector table (drops all collections). + """ + with self._connect() as conn: + with conn.cursor() as cur: + try: + cur.execute("TRUNCATE TABLE document_chunk") + conn.commit() + except Exception as e: + conn.rollback() + log.exception(f"Error during reset: {e}") + raise + + def has_collection(self, collection_name: str) -> bool: + """ + Return True if the collection contains at least one row, else False. + """ + try: + with self._connect() as conn: + with conn.cursor() as cur: + cur.execute( + "SELECT 1 FROM document_chunk WHERE collection_name = ? LIMIT 1", + (collection_name,), + ) + return cur.fetchone() is not None + except Exception: + return False + + def delete_collection(self, collection_name: str) -> None: + """ + Delete all rows in a collection. + """ + self.delete(collection_name) + + def close(self) -> None: + """ + Dispose the underlying SQLAlchemy engine. + """ + try: + self.engine.dispose() + except Exception as e: + log.exception(f"Error during dispose the underlying SQLAlchemy engine: {e}") diff --git a/backend/open_webui/retrieval/vector/dbs/milvus.py b/backend/open_webui/retrieval/vector/dbs/milvus.py index 35cf6b3829..4dcf76c64d 100644 --- a/backend/open_webui/retrieval/vector/dbs/milvus.py +++ b/backend/open_webui/retrieval/vector/dbs/milvus.py @@ -1,3 +1,7 @@ +""" +NOTE: This vector database integration is community-supported and maintained on a best-effort basis. +""" + from pymilvus import MilvusClient as Client from pymilvus import FieldSchema, DataType from pymilvus import connections, Collection diff --git a/backend/open_webui/retrieval/vector/dbs/milvus_multitenancy.py b/backend/open_webui/retrieval/vector/dbs/milvus_multitenancy.py index c58189b2a3..0ecbac15d2 100644 --- a/backend/open_webui/retrieval/vector/dbs/milvus_multitenancy.py +++ b/backend/open_webui/retrieval/vector/dbs/milvus_multitenancy.py @@ -1,3 +1,7 @@ +""" +NOTE: This vector database integration is community-supported and maintained on a best-effort basis. +""" + import logging from typing import Optional, Tuple, List, Dict, Any diff --git a/backend/open_webui/retrieval/vector/dbs/opengauss.py b/backend/open_webui/retrieval/vector/dbs/opengauss.py index 7d4f9ea092..679847a1d4 100644 --- a/backend/open_webui/retrieval/vector/dbs/opengauss.py +++ b/backend/open_webui/retrieval/vector/dbs/opengauss.py @@ -1,3 +1,7 @@ +""" +NOTE: This vector database integration is community-supported and maintained on a best-effort basis. +""" + from typing import Optional, List, Dict, Any import logging import re diff --git a/backend/open_webui/retrieval/vector/dbs/opensearch.py b/backend/open_webui/retrieval/vector/dbs/opensearch.py index ed5a931c68..3ad82d7442 100644 --- a/backend/open_webui/retrieval/vector/dbs/opensearch.py +++ b/backend/open_webui/retrieval/vector/dbs/opensearch.py @@ -1,3 +1,7 @@ +""" +NOTE: This vector database integration is community-supported and maintained on a best-effort basis. +""" + from opensearchpy import OpenSearch from opensearchpy.helpers import bulk from typing import Optional diff --git a/backend/open_webui/retrieval/vector/dbs/oracle23ai.py b/backend/open_webui/retrieval/vector/dbs/oracle23ai.py index f4258c9eff..10428de384 100644 --- a/backend/open_webui/retrieval/vector/dbs/oracle23ai.py +++ b/backend/open_webui/retrieval/vector/dbs/oracle23ai.py @@ -1,4 +1,6 @@ """ +NOTE: This vector database integration is community-supported and maintained on a best-effort basis. + Oracle 23ai Vector Database Client - Fixed Version # .env diff --git a/backend/open_webui/retrieval/vector/dbs/pinecone.py b/backend/open_webui/retrieval/vector/dbs/pinecone.py index 156894bc9e..27bc50b70e 100644 --- a/backend/open_webui/retrieval/vector/dbs/pinecone.py +++ b/backend/open_webui/retrieval/vector/dbs/pinecone.py @@ -1,3 +1,7 @@ +""" +NOTE: This vector database integration is community-supported and maintained on a best-effort basis. +""" + from typing import Optional, List, Dict, Any, Union import logging import time # for measuring elapsed time diff --git a/backend/open_webui/retrieval/vector/dbs/qdrant.py b/backend/open_webui/retrieval/vector/dbs/qdrant.py index d42984e1d6..e774bb875f 100644 --- a/backend/open_webui/retrieval/vector/dbs/qdrant.py +++ b/backend/open_webui/retrieval/vector/dbs/qdrant.py @@ -1,3 +1,7 @@ +""" +NOTE: This vector database integration is community-supported and maintained on a best-effort basis. +""" + from typing import Optional import logging from urllib.parse import urlparse diff --git a/backend/open_webui/retrieval/vector/dbs/qdrant_multitenancy.py b/backend/open_webui/retrieval/vector/dbs/qdrant_multitenancy.py index f87f85a23b..5ad2ac6929 100644 --- a/backend/open_webui/retrieval/vector/dbs/qdrant_multitenancy.py +++ b/backend/open_webui/retrieval/vector/dbs/qdrant_multitenancy.py @@ -1,3 +1,7 @@ +""" +NOTE: This vector database integration is community-supported and maintained on a best-effort basis. +""" + import logging from typing import Optional, Tuple, List, Dict, Any from urllib.parse import urlparse diff --git a/backend/open_webui/retrieval/vector/dbs/s3vector.py b/backend/open_webui/retrieval/vector/dbs/s3vector.py index 96e487f111..1a30e04e55 100644 --- a/backend/open_webui/retrieval/vector/dbs/s3vector.py +++ b/backend/open_webui/retrieval/vector/dbs/s3vector.py @@ -1,3 +1,7 @@ +""" +NOTE: This vector database integration is community-supported and maintained on a best-effort basis. +""" + from open_webui.retrieval.vector.utils import process_metadata from open_webui.retrieval.vector.main import ( VectorDBBase, @@ -61,6 +65,11 @@ def _create_index( dataType=data_type, dimension=dimension, distanceMetric=distance_metric, + metadataConfiguration={ + "nonFilterableMetadataKeys": [ + "text", + ] + }, ) log.info( f"Created S3 index: {index_name} (dim={dimension}, type={data_type}, metric={distance_metric})" diff --git a/backend/open_webui/retrieval/vector/dbs/weaviate.py b/backend/open_webui/retrieval/vector/dbs/weaviate.py index dcc648c788..c9b09ad638 100644 --- a/backend/open_webui/retrieval/vector/dbs/weaviate.py +++ b/backend/open_webui/retrieval/vector/dbs/weaviate.py @@ -1,3 +1,7 @@ +""" +NOTE: This vector database integration is community-supported and maintained on a best-effort basis. +""" + import weaviate import re import uuid diff --git a/backend/open_webui/retrieval/vector/factory.py b/backend/open_webui/retrieval/vector/factory.py index 68595fb595..d92b335864 100644 --- a/backend/open_webui/retrieval/vector/factory.py +++ b/backend/open_webui/retrieval/vector/factory.py @@ -57,6 +57,12 @@ def get_vector(vector_type: str) -> VectorDBBase: from open_webui.retrieval.vector.dbs.opengauss import OpenGaussClient return OpenGaussClient() + case VectorType.MARIADB_VECTOR: + from open_webui.retrieval.vector.dbs.mariadb_vector import ( + MariaDBVectorClient, + ) + + return MariaDBVectorClient() case VectorType.ELASTICSEARCH: from open_webui.retrieval.vector.dbs.elasticsearch import ( ElasticsearchClient, diff --git a/backend/open_webui/retrieval/vector/type.py b/backend/open_webui/retrieval/vector/type.py index de20133fce..df9453aa3e 100644 --- a/backend/open_webui/retrieval/vector/type.py +++ b/backend/open_webui/retrieval/vector/type.py @@ -3,6 +3,7 @@ class VectorType(StrEnum): MILVUS = "milvus" + MARIADB_VECTOR = "mariadb-vector" QDRANT = "qdrant" CHROMA = "chroma" PINECONE = "pinecone" diff --git a/backend/open_webui/routers/analytics.py b/backend/open_webui/routers/analytics.py index 1bd12f7bdb..9579845a49 100644 --- a/backend/open_webui/routers/analytics.py +++ b/backend/open_webui/routers/analytics.py @@ -86,7 +86,7 @@ async def get_user_analytics( start_date=start_date, end_date=end_date, group_id=group_id, db=db ) token_usage = ChatMessages.get_token_usage_by_user( - start_date=start_date, end_date=end_date, db=db + start_date=start_date, end_date=end_date, group_id=group_id, db=db ) # Get user info for top users diff --git a/backend/open_webui/routers/audio.py b/backend/open_webui/routers/audio.py index 3156078326..a1f3ac523f 100644 --- a/backend/open_webui/routers/audio.py +++ b/backend/open_webui/routers/audio.py @@ -492,7 +492,7 @@ async def speech(request: Request, user=Depends(get_verified_user)): region = request.app.state.config.TTS_AZURE_SPEECH_REGION or "eastus" base_url = request.app.state.config.TTS_AZURE_SPEECH_BASE_URL language = request.app.state.config.TTS_VOICE - locale = "-".join(request.app.state.config.TTS_VOICE.split("-")[:1]) + locale = "-".join(request.app.state.config.TTS_VOICE.split("-")[:2]) output_format = request.app.state.config.TTS_AZURE_SPEECH_OUTPUT_FORMAT try: @@ -852,17 +852,34 @@ def transcription_handler(request, file_path, metadata, user=None): except requests.exceptions.RequestException as e: log.exception(e) detail = None + status_code = getattr(r, "status_code", 500) if r else 500 try: if r is not None and r.status_code != 200: res = r.json() - if "error" in res: + # Azure returns {"code": "...", "message": "...", "innerError": {...}} + if "code" in res and "message" in res: + azure_code = res.get("innerError", {}).get("code", res["code"]) + user_facing_codes = { + "EmptyAudioFile", + "AudioLengthLimitExceeded", + "NoLanguageIdentified", + "MultipleLanguagesIdentified", + } + if azure_code in user_facing_codes: + detail = res["message"] + else: + log.error( + f"Azure STT error [{azure_code}]: {res['message']}" + ) + detail = "An error occurred during transcription." + elif "error" in res: detail = f"External: {res['error'].get('message', '')}" except Exception: detail = f"External: {e}" raise HTTPException( - status_code=getattr(r, "status_code", 500) if r else 500, + status_code=status_code, detail=detail if detail else "Open WebUI: Server Connection Error", ) @@ -1087,6 +1104,8 @@ def transcribe( for future in futures: try: results.append(future.result()) + except HTTPException: + raise except Exception as transcribe_exc: raise HTTPException( status_code=status.HTTP_500_INTERNAL_SERVER_ERROR, @@ -1226,6 +1245,8 @@ def transcription( "filename": os.path.basename(file_path), } + except HTTPException: + raise except Exception as e: log.exception(e) @@ -1234,6 +1255,8 @@ def transcription( detail="Transcription failed.", ) + except HTTPException: + raise except Exception as e: log.exception(e) diff --git a/backend/open_webui/routers/auths.py b/backend/open_webui/routers/auths.py index 8bd97db1c5..6227562b8f 100644 --- a/backend/open_webui/routers/auths.py +++ b/backend/open_webui/routers/auths.py @@ -53,6 +53,7 @@ from fastapi.responses import RedirectResponse, Response, JSONResponse from open_webui.config import ( OPENID_PROVIDER_URL, + OPENID_END_SESSION_ENDPOINT, ENABLE_OAUTH_SIGNUP, ENABLE_LDAP, ENABLE_PASSWORD_AUTH, @@ -880,6 +881,19 @@ async def signout( response.delete_cookie("oauth_session_id") session = OAuthSessions.get_session_by_id(oauth_session_id, db=db) + + # If a custom end_session_endpoint is configured (e.g. AWS Cognito), redirect + # there directly instead of attempting OIDC discovery. + if OPENID_END_SESSION_ENDPOINT.value: + return JSONResponse( + status_code=200, + content={ + "status": True, + "redirect_url": OPENID_END_SESSION_ENDPOINT.value, + }, + headers=response.headers, + ) + oauth_server_metadata_url = ( request.app.state.oauth_manager.get_server_metadata_url(session.provider) if session diff --git a/backend/open_webui/routers/files.py b/backend/open_webui/routers/files.py index 6e3271d7ec..5268ea657d 100644 --- a/backend/open_webui/routers/files.py +++ b/backend/open_webui/routers/files.py @@ -583,12 +583,36 @@ def update_file_data_content_by_id( request, ProcessFileForm(file_id=id, content=form_data.content), user=user, + db=db, ) file = Files.get_file_by_id(id=id, db=db) except Exception as e: log.exception(e) log.error(f"Error processing file: {file.id}") + # Propagate content change to all knowledge collections referencing + # this file. Without this the old embeddings remain in the knowledge + # collection and RAG returns both stale and current data (#20558). + knowledges = Knowledges.get_knowledges_by_file_id(id, db=db) + for knowledge in knowledges: + try: + # Remove old embeddings for this file from the KB collection + VECTOR_DB_CLIENT.delete( + collection_name=knowledge.id, filter={"file_id": id} + ) + # Re-add from the now-updated file-{file_id} collection + process_file( + request, + ProcessFileForm(file_id=id, collection_name=knowledge.id), + user=user, + db=db, + ) + except Exception as e: + log.warning( + f"Failed to update knowledge {knowledge.id} after " + f"content change for file {id}: {e}" + ) + return {"content": file.data.get("content", "")} else: raise HTTPException( diff --git a/backend/open_webui/routers/ollama.py b/backend/open_webui/routers/ollama.py index f497c5a35e..fe3f581d9c 100644 --- a/backend/open_webui/routers/ollama.py +++ b/backend/open_webui/routers/ollama.py @@ -1769,7 +1769,9 @@ async def download_file_stream( yield f"data: {json.dumps(res)}\n\n" else: - raise "Ollama: Could not create blob, Please try again." + raise RuntimeError( + "Ollama: Could not create blob, Please try again." + ) # url = "https://huggingface.co/TheBloke/stablelm-zephyr-3b-GGUF/resolve/main/stablelm-zephyr-3b.Q2_K.gguf" diff --git a/backend/open_webui/routers/pipelines.py b/backend/open_webui/routers/pipelines.py index ebedd3027d..fa1b77a09c 100644 --- a/backend/open_webui/routers/pipelines.py +++ b/backend/open_webui/routers/pipelines.py @@ -92,8 +92,8 @@ async def process_pipeline_inlet_filter(request, payload, user, models): json=request_data, ssl=AIOHTTP_CLIENT_SESSION_SSL, ) as response: - payload = await response.json() response.raise_for_status() + payload = await response.json() except aiohttp.ClientResponseError as e: res = ( await response.json() @@ -145,8 +145,8 @@ async def process_pipeline_outlet_filter(request, payload, user, models): json=request_data, ssl=AIOHTTP_CLIENT_SESSION_SSL, ) as response: - payload = await response.json() response.raise_for_status() + payload = await response.json() except aiohttp.ClientResponseError as e: try: res = ( diff --git a/backend/open_webui/routers/tools.py b/backend/open_webui/routers/tools.py index 938e3c3ee8..351c491bdc 100644 --- a/backend/open_webui/routers/tools.py +++ b/backend/open_webui/routers/tools.py @@ -30,7 +30,11 @@ ) from open_webui.utils.tools import get_tool_specs from open_webui.utils.auth import get_admin_user, get_verified_user -from open_webui.utils.access_control import has_permission, filter_allowed_access_grants +from open_webui.utils.access_control import ( + has_permission, + has_access, + filter_allowed_access_grants, +) from open_webui.utils.tools import get_tool_servers from open_webui.config import CACHE_DIR, BYPASS_ADMIN_ACCESS_CONTROL diff --git a/backend/open_webui/utils/misc.py b/backend/open_webui/utils/misc.py index 514616ed0b..75ad086da1 100644 --- a/backend/open_webui/utils/misc.py +++ b/backend/open_webui/utils/misc.py @@ -565,6 +565,53 @@ def sanitize_data_for_db(obj): return obj +def sanitize_metadata(metadata: dict) -> dict: + """ + Return a JSON-safe copy of a metadata dict for database storage. + + The middleware metadata accumulates non-serializable Python objects + (e.g. callable tool functions, MCP client instances) that cause + PostgreSQL JSON inserts to fail. This helper strips those out while + preserving the primitive data needed for file-to-chat linking. + """ + if not isinstance(metadata, dict): + return metadata + + def _sanitize(obj): + if isinstance(obj, (str, int, float, bool, type(None))): + return obj + if isinstance(obj, dict): + return { + k: _sanitize(v) + for k, v in obj.items() + if not callable(v) and _is_serializable(v) + } + if isinstance(obj, list): + return [ + _sanitize(v) for v in obj if not callable(v) and _is_serializable(v) + ] + if callable(obj): + return None + # Last resort: try to see if it's serializable + try: + json.dumps(obj) + return obj + except (TypeError, ValueError): + return None + + def _is_serializable(obj): + """Quick check whether a value can survive JSON serialization.""" + if isinstance(obj, (str, int, float, bool, type(None), dict, list)): + return True + try: + json.dumps(obj) + return True + except (TypeError, ValueError): + return False + + return _sanitize(metadata) + + def extract_folders_after_data_docs(path): # Convert the path to a Path object if it's not already path = Path(path) diff --git a/backend/open_webui/utils/oauth.py b/backend/open_webui/utils/oauth.py index 392ae74f96..636ad957d7 100644 --- a/backend/open_webui/utils/oauth.py +++ b/backend/open_webui/utils/oauth.py @@ -620,7 +620,7 @@ async def _preflight_authorization_url( payload = json.loads(response_text) error = payload.get("error") error_description = payload.get("error_description", "") - except: + except Exception: pass else: error_description = response_text @@ -1706,7 +1706,9 @@ async def handle_callback(self, request, provider, response, db=None): redirect_url = f"{redirect_base_url}/auth" if error_message: - redirect_url = f"{redirect_url}?error={error_message}" + redirect_url = ( + f"{redirect_url}?error={urllib.parse.quote_plus(error_message)}" + ) return RedirectResponse(url=redirect_url, headers=response.headers) response = RedirectResponse(url=redirect_url, headers=response.headers) diff --git a/backend/open_webui/utils/task.py b/backend/open_webui/utils/task.py index abc8920884..0ea525c93e 100644 --- a/backend/open_webui/utils/task.py +++ b/backend/open_webui/utils/task.py @@ -142,41 +142,121 @@ def replacement_function(match): return template +def truncate_content(content: str, max_chars: int, mode: str = "middletruncate") -> str: + """Truncate a string to max_chars using the specified mode. + + Modes: + - middletruncate: keep beginning and end, join with '...' + - start: keep first max_chars characters + - end: keep last max_chars characters + """ + if not content or len(content) <= max_chars: + return content + + if mode == "start": + return content[:max_chars] + elif mode == "end": + return content[-max_chars:] + else: # middletruncate + half = max_chars // 2 + return f"{content[:half]}...{content[-(max_chars - half):]}" + + +def apply_content_filter(messages: list[dict], filter_str: str) -> list[dict]: + """Apply a content filter to each message's content. + + filter_str is like 'middletruncate:500', 'start:200', or 'end:200'. + Returns a new list with truncated content (original messages are not mutated). + """ + parts = filter_str.split(":") + if len(parts) != 2: + return messages + + mode = parts[0].lower() + try: + max_chars = int(parts[1]) + except ValueError: + return messages + + if mode not in ("middletruncate", "start", "end"): + return messages + + result = [] + for msg in messages: + new_msg = dict(msg) + if isinstance(new_msg.get("content"), str): + new_msg["content"] = truncate_content(new_msg["content"], max_chars, mode) + elif isinstance(new_msg.get("content"), list): + new_content = [] + for item in new_msg["content"]: + if isinstance(item, dict) and item.get("type") == "text": + new_item = dict(item) + new_item["text"] = truncate_content( + item.get("text", ""), max_chars, mode + ) + new_content.append(new_item) + else: + new_content.append(item) + new_msg["content"] = new_content + result.append(new_msg) + return result + + def replace_messages_variable( template: str, messages: Optional[list[dict]] = None ) -> str: def replacement_function(match): - full_match = match.group(0) - start_length = match.group(1) - end_length = match.group(2) - middle_length = match.group(3) + # Groups: (1) filter for bare MESSAGES + # (2) START count, (3) filter for START + # (4) END count, (5) filter for END + # (6) MIDDLE count,(7) filter for MIDDLE + bare_filter = match.group(1) + start_length = match.group(2) + start_filter = match.group(3) + end_length = match.group(4) + end_filter = match.group(5) + middle_length = match.group(6) + middle_filter = match.group(7) + # If messages is None, handle it as an empty list if messages is None: return "" - # Process messages based on the number of messages required - if full_match == "{{MESSAGES}}": - return get_messages_content(messages) - elif start_length is not None: - return get_messages_content(messages[: int(start_length)]) + # Select messages based on the variant + if start_length is not None: + selected = messages[: int(start_length)] + content_filter = start_filter elif end_length is not None: - return get_messages_content(messages[-int(end_length) :]) + selected = messages[-int(end_length) :] + content_filter = end_filter elif middle_length is not None: mid = int(middle_length) - if len(messages) <= mid: - return get_messages_content(messages) - # Handle middle truncation: split to get start and end portions of the messages list - half = mid // 2 - start_msgs = messages[:half] - end_msgs = messages[-half:] if mid % 2 == 0 else messages[-(half + 1) :] - formatted_start = get_messages_content(start_msgs) - formatted_end = get_messages_content(end_msgs) - return f"{formatted_start}\n{formatted_end}" - return "" + selected = messages + else: + half = mid // 2 + start_msgs = messages[:half] + end_msgs = messages[-half:] if mid % 2 == 0 else messages[-(half + 1) :] + selected = start_msgs + end_msgs + content_filter = middle_filter + else: + # Bare {{MESSAGES}} or {{MESSAGES|filter}} + selected = messages + content_filter = bare_filter + + # Apply content filter if present + if content_filter: + selected = apply_content_filter(selected, content_filter) + + return get_messages_content(selected) template = re.sub( - r"{{MESSAGES}}|{{MESSAGES:START:(\d+)}}|{{MESSAGES:END:(\d+)}}|{{MESSAGES:MIDDLETRUNCATE:(\d+)}}", + r"(?:" + r"\{\{MESSAGES(?:\|(\w+:\d+))?\}\}" + r"|\{\{MESSAGES:START:(\d+)(?:\|(\w+:\d+))?\}\}" + r"|\{\{MESSAGES:END:(\d+)(?:\|(\w+:\d+))?\}\}" + r"|\{\{MESSAGES:MIDDLETRUNCATE:(\d+)(?:\|(\w+:\d+))?\}\}" + r")", replacement_function, template, ) diff --git a/backend/open_webui/utils/tools.py b/backend/open_webui/utils/tools.py index 081e2f5fab..e525a8284c 100644 --- a/backend/open_webui/utils/tools.py +++ b/backend/open_webui/utils/tools.py @@ -707,8 +707,8 @@ def get_functions_from_tool(tool: object) -> list[Callable]: getattr(tool, func) ) # checks if the attribute is callable (a method or function). and not func.startswith( - "__" - ) # filters out special (dunder) methods like init, str, etc. โ€” these are usually built-in functions of an object that you might not need to use directly. + "_" + ) # filters out internal methods (starting with _) and special (dunder) methods. and not inspect.isclass( getattr(tool, func) ) # ensures that the callable is not a class itself, just a method or function. diff --git a/backend/open_webui/utils/webhook.py b/backend/open_webui/utils/webhook.py index eb2688851f..b3a3c6bcd1 100644 --- a/backend/open_webui/utils/webhook.py +++ b/backend/open_webui/utils/webhook.py @@ -26,9 +26,13 @@ async def post_webhook(name: str, url: str, message: str, event_data: dict) -> b # Microsoft Teams Webhooks elif "webhook.office.com" in url: action = event_data.get("action", "undefined") + user_data = event_data.get("user", "{}") + if isinstance(user_data, dict): + user_dict = user_data + else: + user_dict = json.loads(user_data) facts = [ - {"name": name, "value": value} - for name, value in json.loads(event_data.get("user", {})).items() + {"name": name, "value": value} for name, value in user_dict.items() ] payload = { "@type": "MessageCard", diff --git a/backend/requirements.txt b/backend/requirements.txt index 5c77098dd0..d252420c26 100644 --- a/backend/requirements.txt +++ b/backend/requirements.txt @@ -120,6 +120,7 @@ pgvector==0.4.2 PyMySQL==1.1.2 boto3==1.42.62 +mariadb==1.1.14 pymilvus==2.6.9 qdrant-client==1.17.0 @@ -156,6 +157,7 @@ opentelemetry-instrumentation-requests==0.61b0 opentelemetry-instrumentation-logging==0.61b0 opentelemetry-instrumentation-httpx==0.61b0 opentelemetry-instrumentation-aiohttp-client==0.61b0 +opentelemetry-instrumentation-system-metrics==0.61b0 # Alipay alipay-sdk-python==3.7.796 diff --git a/package-lock.json b/package-lock.json index cabfd5b46e..877562712a 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "open-webui", - "version": "0.8.9.1", + "version": "0.8.10.1", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "open-webui", - "version": "0.8.9.1", + "version": "0.8.10.1", "dependencies": { "@azure/msal-browser": "^4.5.0", "@codemirror/lang-javascript": "^6.2.2", diff --git a/package.json b/package.json index 947db0c9de..e27ebd1639 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "open-webui", - "version": "0.8.9.1", + "version": "0.8.10.1", "private": true, "scripts": { "dev": "npm run pyodide:fetch && vite dev --host", diff --git a/pyproject.toml b/pyproject.toml index 251240ac48..2ec9d033b9 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -137,11 +137,15 @@ postgres = [ "psycopg2-binary==2.9.11", "pgvector==0.4.2", ] +mariadb = [ + "mariadb==1.1.14", +] all = [ "pymongo", "psycopg2-binary==2.9.11", "pgvector==0.4.2", + "mariadb==1.1.14", "moto[s3]>=5.0.26", "gcp-storage-emulator>=2024.8.3", "docker~=7.1.0", diff --git a/src/lib/components/chat/MessageInput.svelte b/src/lib/components/chat/MessageInput.svelte index 9bf3a08139..4664940d3a 100644 --- a/src/lib/components/chat/MessageInput.svelte +++ b/src/lib/components/chat/MessageInput.svelte @@ -1187,6 +1187,7 @@ import { getContext } from 'svelte'; import Tooltip from '$lib/components/common/Tooltip.svelte'; + import Image from '$lib/components/common/Image.svelte'; import GarbageBin from '$lib/components/icons/GarbageBin.svelte'; import EditPencil from '$lib/components/icons/EditPencil.svelte'; import ArrowForward from '$lib/components/icons/ArrowForward.svelte'; + import { WEBUI_API_BASE_URL } from '$lib/constants'; const i18n = getContext('i18n'); export let id: string; export let content: string; + export let files: any[] = []; export let onSendNow: (id: string) => void; export let onEdit: (id: string) => void; export let onDelete: (id: string) => void; @@ -21,8 +24,34 @@ -
-

{content}

+
+ {#if files.length > 0} +
+ {#each files as file} + {#if file.type === 'image' || (file?.content_type ?? '').startsWith('image/')} + {@const fileUrl = + file.url?.startsWith('data') || file.url?.startsWith('http') + ? file.url + : `${WEBUI_API_BASE_URL}/files/${file.url}${file?.content_type ? '/content' : ''}`} + + {:else} +
+ {file.name ?? 'file'} +
+ {/if} + {/each} +
+ {/if} + + {#if content} +

{content}

+ {:else if files.length === 0} +

+ {$i18n.t('Empty message')} +

+ {/if}
diff --git a/src/lib/components/chat/Messages/Citations.svelte b/src/lib/components/chat/Messages/Citations.svelte index 1940448685..8f0d93ce6d 100644 --- a/src/lib/components/chat/Messages/Citations.svelte +++ b/src/lib/components/chat/Messages/Citations.svelte @@ -178,6 +178,9 @@ src="https://www.google.com/s2/favicons?sz=32&domain={citation.source.name}" alt="favicon" class="size-4 rounded-full shrink-0 border border-white dark:border-gray-850 bg-white dark:bg-gray-900" + on:error={(e) => { + e.target.src = '/favicon.png'; + }} /> {/each}
diff --git a/src/lib/components/layout/Sidebar/ChatItem.svelte b/src/lib/components/layout/Sidebar/ChatItem.svelte index 77931c779d..60c04564cd 100644 --- a/src/lib/components/layout/Sidebar/ChatItem.svelte +++ b/src/lib/components/layout/Sidebar/ChatItem.svelte @@ -78,10 +78,6 @@ let chat = null; let mouseOver = false; - let draggable = false; - $: if (mouseOver) { - loadChat(); - } const loadChat = async () => { if (!chat) { @@ -374,7 +370,7 @@ id="sidebar-chat-group" bind:this={itemElement} class=" w-full {className} relative group" - draggable={draggable && !confirmEdit} + draggable={!confirmEdit} > {#if confirmEdit}
{ return results.reverse().join(' '); }; +// Month names used as i18n translation keys โ€” must be English regardless of locale +const MONTH_NAMES = [ + 'January', + 'February', + 'March', + 'April', + 'May', + 'June', + 'July', + 'August', + 'September', + 'October', + 'November', + 'December' +]; + export const getTimeRange = (timestamp) => { const now = new Date(); const date = new Date(timestamp * 1000); // Convert Unix timestamp to milliseconds @@ -1119,7 +1135,7 @@ export const getTimeRange = (timestamp) => { } else if (diffDays <= 30) { return 'Previous 30 days'; } else if (nowYear === dateYear) { - return date.toLocaleString('default', { month: 'long' }); + return MONTH_NAMES[dateMonth]; } else { return date.getFullYear().toString(); }