Theo query service restored#6645
Draft
anna-parker wants to merge 11 commits into
Draft
Conversation
…f LAPIS
Adds a new single-deployment service (query-service/) that proxies
/{organism}/{path} to the corresponding loculus-lapis-service-{organism}.
For now it is a transparent passthrough; the point is to give us a
single hop where we can rewrite LAPIS responses in future iterations
without changing the website or LAPIS.
Wires the website (server-side lapisUrls), the lapis ingress, and the
public lapisUrls (used by the browser and CLI) through the new
service, so all external and internal LAPIS calls now go through it.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The query-service is now the only LAPIS-facing surface. The old
`/{organism}/{path}` passthrough is removed.
API shape (see query-service/README.md):
- Owned verbs under /v1/: aggregated, details, mutations, aaMutations,
insertions, aaInsertions, alignedSequences[/segment],
unalignedSequences[/segment], aaSequences/<protein>, info,
lineageDefinition.
- `?organism=` is required and single-valued.
- Reserved control keywords: organism, format (-> dataFormat),
download (-> downloadAsFile), fields, limit, offset, include,
reference. Anything else is a metadata-column filter.
- Implicit defaults applied centrally: versionStatus=LATEST_VERSION,
isRevocation=false. Override with `include=revoked|older-versions|all`.
Explicit version filters drop the defaults.
Helm:
- Removed `lapisUrls` (per-organism map) from the website runtime config;
replaced with a single `queryServiceUrl` plus an `organisms` list.
- lapis-ingress is now a single rule that routes everything on the
lapis hostname to query-service.
Website:
- Renamed Zodios endpoints to /v1/<verb>; segment / proteinName are
path components (Zodios needs unique paths per endpoint).
- LapisClient now takes (queryServiceUrl, organism, schema) and adds
`?organism=` to every call. Internal name unchanged to limit churn.
- DownloadUrlGenerator builds /v1/... URLs and adds ?organism=.
- Dropped manual versionStatus / isRevocation defaults from server-side
callers (GroupPage, getSeqSetStatistics, getOrganismStatistics) — the
query-service applies them.
- /loculus-info now exposes hosts.queryService instead of hosts.lapis.
CLI:
- Migrated to /v1/ paths with ?organism=.
- get_lapis_url() retained as a thin wrapper that returns the
query-service base URL.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
… POSTs
Adds the missing pieces to keep the prior search-page UX (where users
can clear the version-status / revocation hidden-field defaults to see
all versions) working against the new query-service defaults.
- lapisApi.ts: every /v1/ endpoint now accepts an optional `include`
query parameter.
- serviceHooks.lapisClientHooks: takes an `options.include` and threads
it onto every request as a query string. The search UI passes
`'all'`; autocomplete and detail views leave it unset so query-service
defaults still apply there.
- SearchFullUI / serversideSearch / DownloadUrlGenerator: pass
`include=all`, since the search page manages its own version
defaults via hiddenFieldValues.
- LapisClient.getAllSequenceEntryHistoryForAccession: passes
`include=all` (version history needs every version + revocation).
- query-service:
- reads `include=` from the query string for POSTs too (was only
looking in the body), so the website's `?include=all` is honoured.
- parses `application/x-www-form-urlencoded` bodies for the long-query
download path that submits an HTML form. Uses `getlist` so repeated
keys (`fields=a&fields=b&fields=c`) are preserved as a list, not
collapsed to the first value.
- Adds python-multipart to requirements (Starlette's form parser
dependency).
- Updates DownloadDialog.spec.tsx assertions for the new
`?organism=ebola&include=all&...` URL prefix.
- Fixes one stale `/sample/details` reference in
download.dependent.spec.ts.
Integration suite: 98/98 passing.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Add .github/workflows/query-service-image.yml so the integration tests can pull commit-tagged images of the query-service. Mirrors preprocessing-dummy-image.yml: hash-based caching, multi-tag pushes, ARM build on main. - Run prettier --write across files touched by the migration. - Restore the versionStatus / isRevocation regex segments to the DownloadDialog spec assertions that exercise hiddenFieldValues — those filters _are_ sent in that flow (the test passes them explicitly), and the implicit defaults stay opt-out via include=all. - Update vitest.setup.ts MSW mocks to point at the new sequence paths: /v1/alignedSequences[/segment], /v1/unalignedSequences[/segment]. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…e-all toggle Following up on the previous commit so the website stops shipping the versionStatus / isRevocation hidden defaults that the query-service is already applying: - Remove the `hiddenFieldValues` containing `versionStatus=LATEST_VERSION` and `isRevocation=false` from the search and submission/released pages. The query-service applies those defaults centrally — no need to also send them on the wire from the website. - Drop the blanket `?include=all` from `lapisClientHooks` / `serversideSearch` / `DownloadUrlGenerator`. Default search now relies on query-service's defaults (latest non-revoked). - Add an explicit "Include older versions and revocations" checkbox at the top of the search form. It writes `?include=all` into the URL, which the website then forwards as the `include=` query param. Toggle off and the URL drops the param so defaults reapply. - Update the override-hidden-fields integration test to flip the new toggle instead of clearing the (no longer present) hidden fields. - Replace the "hidden field values are kept in URL params" unit test with one that asserts the new `include=` toggle round-trips through the URL. - Bump query-service CPU request/limit a touch (200m / 2 cores). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
… include=all
- SearchFullUI: stabilise the empty `hiddenFieldValues` default with
`useMemo`. The previous `??= {}` reassigned a fresh object every
render, which destabilised the `useMemo` chain feeding the
search-results `useEffect` and caused it to refire — the user observed
this as an infinite loop of /v1/aggregated requests on the search page.
- /[organism]/submission/[groupId]/released: default `?include=all` so
submitters see every version they've released (including revocations
they've just made). The `mySequencesPage` route helper builds the
link with that param and the page itself injects it into
`initialQueryDict` for direct navigation.
… state The React URL-syncing hook overwrites SSR-injected state with whatever is actually in window.location on hydration, so injecting include=all into initialQueryDict alone gets clobbered. Redirect server-side instead so the URL itself is the source of truth, and hydration sees include=all.
…do /released redirect
- The 'override hidden fields' test was specifically exercising the
removed clear-the-hidden-default mechanism. The replacement coverage
is in search.dependent.spec.ts ('include-all toggle puts include=all
in the URL'); the autocomplete-timeout flake on the deleted test was
not adding signal.
- Reverting the server-side redirect to ?include=all on /released:
some tests (e.g. file-sharing 'bulk revise 2 seqs with files') do
page.goto(page.url() + '?column_submissionId=true') after navigating
to /released, and the redirect made page.url() already carry an
?include=all so the appended ?column_... corrupted the URL into
?include=all?column_submissionId=true and query-service rejected the
malformed include= value with 400 (which then tripped the
console-warnings fixture).
The link helper (routes.mySequencesPage) still adds ?include=all so
the in-page 'released sequences' link gives submitters the
see-everything view they need; tests that navigate directly retain
default behaviour and can append params with ? safely.
Conflicts resolved: - integration-tests/.../override-hidden-fields.spec.ts — kept the branch's deletion (the hidden-default override UX no longer exists; see search.dependent.spec.ts's 'include-all toggle' test). - website/src/components/SearchPage/SearchForm.tsx — main introduced a multi-field-search variant in the Metadata Filters loop; reapplied the branch's organism prop on the SearchField inside it. - website/src/components/SearchPage/fields/HierarchicalField.spec.tsx — main added this new spec; threaded the required `organism` prop into every <HierarchicalField> render.
The new query-service exposes /v1/aggregated rather than
/{organism}/sample/aggregated, so multi-field-search interceptors were
matching nothing and observing undefined status.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
anna-parker
force-pushed
the
theo-query-service-restored
branch
from
2157fb1 to
0986327
Compare
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This adds a proxy in front of the LAPIS APIs so that Loculus can decide on the query structure of its query endpoints, it also creates 1 query service for all organisms. Note we reduce the number of endpoints to those required for loculus and use
organismas a param so that it is easily extendable to new organisms.Loculus query-service
Single-deployment HTTP service that the website and CLI use instead of
talking to LAPIS directly.
v1 API
organismis required and single-valued.PR Checklist
🚀 Preview: Add
previewlabel to enable