Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 36 additions & 0 deletions .github/workflows/quality-checks.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
name: Quality Checks PR
concurrency:
group: Quality-Checks-PR-${{ github.workflow }}-${{ github.head_ref || github.event.workflow_run.head_branch || github.run_id }}
cancel-in-progress: true

on:
merge_group:
pull_request: null
push:
branches:
- main

permissions:
contents: read

jobs:
quality-checks:
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # ratchet:actions/checkout@v6
with:
fetch-depth: 0
persist-credentials: false
- uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # ratchet:actions/setup-python@v6
with:
python-version: "3.11"
- uses: j178/prek-action@cbc2f23eb5539cf20d82d1aabd0d0ecbcc56f4e3
with:
prek-version: '0.3.4'
extra-args: ${{ github.event_name == 'pull_request' && format('--from-ref {0} --to-ref {1}', github.event.pull_request.base.sha, github.event.pull_request.head.sha) || github.event_name == 'merge_group' && format('--from-ref {0} --to-ref {1}', github.event.merge_group.base_sha, github.event.merge_group.head_sha) || github.ref_name == 'main' && '--all-files' || '' }}
- name: Check Actions
uses: giner/check-actions@28d366c7cbbe235f9624a88aa31a628167eee28c # ratchet:giner/check-actions@v1.0.1
with:
check_permissions: false
check_versions: false
39 changes: 39 additions & 0 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
default_install_hook_types:
- pre-commit
repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: 3e8a8703264a2f4a69428a0aa4dcb512790b2c8c # frozen: v6.0.0
hooks:
- id: check-added-large-files
name: Check for added large files
args: ["--maxkb=10000"]

- repo: https://github.com/rhysd/actionlint
rev: a443f344ff32813837fa49f7aa6cbc478d770e62 # frozen: v1.7.9
hooks:
- id: actionlint

- repo: https://github.com/pre-commit/mirrors-prettier
rev: ffb6a759a979008c0e6dff86e39f4745a2d9eac4 # frozen: v3.1.0
hooks:
- id: prettier
types_or: [json, css]
language_version: system

- repo: https://github.com/sirwart/ripsecrets
rev: 7d94620933e79b8acaa0cd9e60e9864b07673d86 # frozen: v0.1.11
hooks:
- id: ripsecrets
args:
- --strict-ignore
- --additional-pattern
- ^sk-[A-Za-z0-9_\-]{20,}$

- repo: local
hooks:
- id: format-docs
name: format_docs.py --check
entry: python3 scripts/format_docs.py --check --paths
language: system
types_or: [markdown, mdx]
require_serial: true
14 changes: 7 additions & 7 deletions admins/actions/mcp.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -27,9 +27,9 @@ You can configure Onyx to be an MCP client and allow your Agents to retrieve dat
<img className="rounded-image" src="/assets/overview/core_features/mcp.png" alt="Add MCP Server dialog in Onyx Admin Panel"/>

- **Server Name** — A human-readable name for this MCP server.
Typically the name of the service you're connecting to (e.g. "Slack", "Jira").
Typically the name of the service you're connecting to (e.g. "Slack", "Jira").
- **Description** — Any additional information that helps identify the server.
Useful for distinguishing multiple configured MCP server connections that talk to the same service.
Useful for distinguishing multiple configured MCP server connections that talk to the same service.
- **MCP Server URL** — The URL of the MCP server, which must be reachable from your Onyx instance. Most URLs end in `/mcp`, or `/sse` for older servers.
- **Managed MCP server:** The URL is typically found in the provider's documentation describing how to connect.
- **Self-hosted MCP server:** You'll need to run your server in HTTP transport mode — Onyx doesn't currently support connecting via stdio. If the server is running on the same machine as Onyx, use `http://127.0.0.1:<port>`. Otherwise, use the URL and port the server is exposed at.
Expand All @@ -56,12 +56,12 @@ You can configure Onyx to be an MCP client and allow your Agents to retrieve dat
There are two sub-modes:

- **Shared Key** — The admin configures a single API key used for all user requests.
Similar to No Auth in that users won't need any additional configuration and will all get the same level of access
(whatever is provided by the API key you choose).
Similar to No Auth in that users won't need any additional configuration and will all get the same level of access
(whatever is provided by the API key you choose).
- **Individual Key** — Users authenticate in the chat flow by providing their own API key(s)
to connect to the server. Useful when you want users to have differing access to an external service by identity.
This mode is also the most configurable — Onyx allows the admin to specify arbitrary header patterns to send to
the MCP server.
to connect to the server. Useful when you want users to have differing access to an external service by identity.
This mode is also the most configurable — Onyx allows the admin to specify arbitrary header patterns to send to the
MCP server.

### OAuth

Expand Down
93 changes: 41 additions & 52 deletions admins/advanced_configs/opensearch_document_index_migration.mdx
Original file line number Diff line number Diff line change
@@ -1,87 +1,76 @@
---
title: "OpenSearch Document Index Migration"
description: "Migrating indexed documents from Vespa to OpenSearch"

Check warning on line 3 in admins/advanced_configs/opensearch_document_index_migration.mdx

View check run for this annotation

Mintlify / Mintlify Validation (danswer) - vale-spellcheck

admins/advanced_configs/opensearch_document_index_migration.mdx#L3

Did you really mean 'Vespa'?
icon: "arrow-right-arrow-left"
---

## Overview

Onyx has gradually been working towards switching our backing document index
database from Vespa to OpenSearch. The motivating factors behind this decision
are to reduce the baseline resources required to run Onyx, and to switch to a
widely used and supported documented index provider.
Onyx has gradually been working towards switching our backing document index database from Vespa to OpenSearch.

Check warning on line 9 in admins/advanced_configs/opensearch_document_index_migration.mdx

View check run for this annotation

Mintlify / Mintlify Validation (danswer) - vale-spellcheck

admins/advanced_configs/opensearch_document_index_migration.mdx#L9

Did you really mean 'Vespa'?
The motivating factors behind this decision are to reduce the baseline resources required to run Onyx,
and to switch to a widely used and supported documented index provider.

<Note>
This is a living page and more details will be added over the coming weeks.
This is a living page and more details will be added over the coming weeks.
</Note>

## Migration

As of `v3`, Onyx uses both Vespa and OpenSearch. Any documents which Onyx
indexes while using this version are indexed into both Vespa and OpenSearch, and
there is no migration required.
As of `v3`, Onyx uses both Vespa and OpenSearch.

Check warning on line 19 in admins/advanced_configs/opensearch_document_index_migration.mdx

View check run for this annotation

Mintlify / Mintlify Validation (danswer) - vale-spellcheck

admins/advanced_configs/opensearch_document_index_migration.mdx#L19

Did you really mean 'Vespa'?
Any documents which Onyx indexes while using this version are indexed into both Vespa and OpenSearch,

Check warning on line 20 in admins/advanced_configs/opensearch_document_index_migration.mdx

View check run for this annotation

Mintlify / Mintlify Validation (danswer) - vale-spellcheck

admins/advanced_configs/opensearch_document_index_migration.mdx#L20

Did you really mean 'Vespa'?
and there is no migration required.

Also as of `v3`, Onyx automatically runs a task to migrate your existing
documents from Vespa to OpenSearch. This task makes checkpointed progress so
Onyx does not necessarily need to be running continuously until the migration
completes in order for it to complete successfully. No explicit input is needed
from admins other than upgrading to `v3` in order for this task to run.
Also as of `v3`, Onyx automatically runs a task to migrate your existing documents from Vespa to OpenSearch.

Check warning on line 23 in admins/advanced_configs/opensearch_document_index_migration.mdx

View check run for this annotation

Mintlify / Mintlify Validation (danswer) - vale-spellcheck

admins/advanced_configs/opensearch_document_index_migration.mdx#L23

Did you really mean 'Vespa'?
This task makes checkpointed progress so Onyx does not necessarily need to be running continuously until the migration

Check warning on line 24 in admins/advanced_configs/opensearch_document_index_migration.mdx

View check run for this annotation

Mintlify / Mintlify Validation (danswer) - vale-spellcheck

admins/advanced_configs/opensearch_document_index_migration.mdx#L24

Did you really mean 'checkpointed'?
completes in order for it to complete successfully.
No explicit input is needed from admins other than upgrading to `v3` in order for this task to run.

The migration's progress can be monitored in the Admin Panel, in the Document
Index Migration sidebar tab.
The migration's progress can be monitored in the Admin Panel, in the Document Index Migration sidebar tab.

<Warning>
The migration is best-effort. Although the vast majority of documents are
not expected to have migration issues, some documents (in particular
documents which were indexed a long time ago in versions of Onyx which
enforced weaker/different contracts for indexed content) may have
irreconcilable migration issues. In this event the only solution is a
re-index of connectors for those documents, where they will be processed in
such a way that they can then be indexed into OpenSearch.
The migration is best-effort. Although the vast majority of documents are not expected to have migration issues,
some documents (in particular documents which were indexed a long time ago in versions of Onyx which enforced
weaker/different contracts for indexed content) may have irreconcilable migration issues.
In this event the only solution is a re-index of connectors for those documents,
where they will be processed in such a way that they can then be indexed into OpenSearch.
</Warning>

## Deprecation of Vespa

Check warning on line 38 in admins/advanced_configs/opensearch_document_index_migration.mdx

View check run for this annotation

Mintlify / Mintlify Validation (danswer) - vale-spellcheck

admins/advanced_configs/opensearch_document_index_migration.mdx#L38

Did you really mean 'Vespa'?

`v4` of Onyx will represent the first major version to use OpenSearch entirely
without Vespa.
`v4` of Onyx will represent the first major version to use OpenSearch entirely without Vespa.

Check warning on line 40 in admins/advanced_configs/opensearch_document_index_migration.mdx

View check run for this annotation

Mintlify / Mintlify Validation (danswer) - vale-spellcheck

admins/advanced_configs/opensearch_document_index_migration.mdx#L40

Did you really mean 'Vespa'?

<Warning>
In order to preserve existing indexed content in Vespa without requiring
re-indexing, it will be necessary to upgrade to `v3` of Onyx to allow
migrations to run before upgrading to `v4`, when released.

Although it may be technically feasible without breaking the product for a
determined admin to upgrade Onyx directly from some version < `v3` directly
to some version > `v3` skipping `v3` entirely, this will result in complete
loss of indexed data and will require re-indexing all connectors in order to
be able to search over documents.
In order to preserve existing indexed content in Vespa without requiring re-indexing,

Check warning on line 43 in admins/advanced_configs/opensearch_document_index_migration.mdx

View check run for this annotation

Mintlify / Mintlify Validation (danswer) - vale-spellcheck

admins/advanced_configs/opensearch_document_index_migration.mdx#L43

Did you really mean 'Vespa'?
it will be necessary to upgrade to `v3` of Onyx to allow migrations to run before upgrading to `v4`, when released.

Although it may be technically feasible without breaking the product for a determined admin to upgrade Onyx directly
from some version < `v3` directly to some version > `v3` skipping `v3` entirely,
this will result in complete loss of indexed data and will require re-indexing all connectors in order to be able to
search over documents.
</Warning>

## Retrieval

During `v3`, document retrieval can be done via either Vespa (the default) or
OpenSearch, and which one is used can be toggled in the Document Index Migration
sidebar tab on the Admin Panel. Naturally, if the migration is not yet complete,
retrieval via OpenSearch may not yield every expected document.
During `v3`, document retrieval can be done via either Vespa (the default) or OpenSearch,

Check warning on line 54 in admins/advanced_configs/opensearch_document_index_migration.mdx

View check run for this annotation

Mintlify / Mintlify Validation (danswer) - vale-spellcheck

admins/advanced_configs/opensearch_document_index_migration.mdx#L54

Did you really mean 'Vespa'?
and which one is used can be toggled in the Document Index Migration sidebar tab on the Admin Panel. Naturally,
if the migration is not yet complete, retrieval via OpenSearch may not yield every expected document.

## cloud.onyx.app

The migration for cloud users began in March 2026, and by April 2026 virtually
all tenants' migrations indicated completion.
The migration for cloud users began in March 2026,
and by April 2026 virtually all tenants' migrations indicated completion.

As of `v3.2.0-cloud.7` in April, all tenants in the cloud have been swapped over
to OpenSearch for retrieval, and Vespa is no longer in the loop with Onyx, so
all new documents indexed are indexed only using OpenSearch.
As of `v3.2.0-cloud.7` in April, all tenants in the cloud have been swapped over to OpenSearch for retrieval,
and Vespa is no longer in the loop with Onyx, so all new documents indexed are indexed only using OpenSearch.

Check warning on line 64 in admins/advanced_configs/opensearch_document_index_migration.mdx

View check run for this annotation

Mintlify / Mintlify Validation (danswer) - vale-spellcheck

admins/advanced_configs/opensearch_document_index_migration.mdx#L64

Did you really mean 'Vespa'?

<Warning>
Due to a unique failure mode involving automatic updates in Vespa Cloud,
there may be some migrations which report successful completion despite
there being some existing documents which did not exist from the perspective
of Vespa at the time of migration for that particular document, and which
therefore might not have been migrated into OpenSearch.

Similarly to the best-effort warning above, the recommended solution to this
is a re-index for connectors suspected of having missing documents.
Unfortunately due to the nature of this issue, there is little visibility
into which specific documents may have experienced this.
Due to a unique failure mode involving automatic updates in Vespa Cloud,

Check warning on line 67 in admins/advanced_configs/opensearch_document_index_migration.mdx

View check run for this annotation

Mintlify / Mintlify Validation (danswer) - vale-spellcheck

admins/advanced_configs/opensearch_document_index_migration.mdx#L67

Did you really mean 'Vespa'?
there may be some migrations which report successful completion despite there being some existing documents which did
not exist from the perspective of Vespa at the time of migration for that particular document,

Check warning on line 69 in admins/advanced_configs/opensearch_document_index_migration.mdx

View check run for this annotation

Mintlify / Mintlify Validation (danswer) - vale-spellcheck

admins/advanced_configs/opensearch_document_index_migration.mdx#L69

Did you really mean 'Vespa'?
and which therefore might not have been migrated into OpenSearch.

Similarly to the best-effort warning above,
the recommended solution to this is a re-index for connectors suspected of having missing documents.
Unfortunately due to the nature of this issue,
there is little visibility into which specific documents may have experienced this.
</Warning>
Loading