feat(indico): scraper, MCP sidecar, and on-demand ingest tooling

[hassan11196]

Summary

Adds Indico as a first-class data source, end-to-end:

1. Indico scraper (bcd4c078, originally @livaage's work — PR feat(data-manager): add Indico scraper integration #550 was not merged upstream, so it ships here) — fetches events/contributions/attachments via the Indico REST API, converts PDF/PPTX/PPT/ODP slides to Markdown via MarkItDown, dedups multi-format uploads, authenticates SSO-protected events via CERNSSOScraper, and prepends per-chunk metadata headers (event/speaker/affiliation/time) to improve BM25 retrieval. Adds markitdown[pdf,pptx] to deps.

2. mcp4indico sidecar (93e994dd) — containerized Indico MCP server using the generic MCP sidecar mechanism from PR Generic MCP sidecar + domain-knowledge skills + tool-use observability #557 (now on main). Ships a Dockerfile pinned to upstream 800c5fc3, an entrypoint wrapper that initializes API client globals from env vars, an Indico skill, and a worked example config block.

3. On-demand ingest agent tools —

   • ingest_url (b48b1554, 76d35e20): wraps POST /document_index/upload_url so the agent can scrape & index any URL it discovers. Dispatches Indico URLs through IndicoScraper (288494e4), falls back to LinkScraper for the rest. Adds configurable routing rules and SSO fallback in the latest revision.
   • ingest_indico_event (5c1f2b9d, a43a819e): ingests Indico event attachments on demand via a shared downloads volume, with ingest_local_path Flask endpoint and LocalFileManager directory-ingestion support.

Suggested agent flow

INDICO_get_files(download_files=false) → ingest_url(event_url) (or ingest_indico_event) → search_vectorstore_hybrid once slides are indexed.

Known limitations (Indico scraper)

• Images/diagrams in slides aren't extracted (text-only conversion).
• <latexit> blocks are stripped; inline LaTeX may still chunk poorly.
• Chunks are per-page, not per-deck — no cross-slide summarization.
• Category URLs (/category/<id>/) are coded but not e2e-tested.
• SSO is CERN-specific (CERNSSOScraper); other Indico instances need a different auth path.
• No rate limiting / incremental scraping.

Example config

The PR ships an example config block in mcp/indico/README.md; below is a working snippet drawn from a live base_config_indico.yaml deployment that exercises both the sidecar and the agent-tool routing.

MCP sidecar

Declare the Indico MCP server under mcp_servers. The build_context points at this PR's mcp/indico/ directory so the sidecar is built locally; tokens come from env (.env / secrets manager). The shared_volume is mounted read-only into the data-manager at the same path, and the template derives INDICO_DOWNLOADS_DIR=/shared/indico-downloads from it so ingest_indico_event can pick up downloaded attachments.

mcp_servers:
  indico:
    transport: streamable_http
    url: http://localhost:8012/mcp        # use http://indico-mcp:8012/mcp on bridge networking
    build_context: ./mcp/indico           # path to this PR's sidecar
    env:
      INDICO_BASE_URL: https://indico.cern.ch
      BEARER_TOKEN: ${INDICO_BEARER_TOKEN}
      API_KEY: ${INDICO_API_KEY}
      API_SECRET: ${INDICO_API_SECRET}
    shared_volume: indico-downloads
    skill: indico

ingest_url routing rules + SSO fallback

ingest_url evaluates rules in declaration order; first match wins.

• action: refuse — return message to the agent so it switches to a better tool (used here to bounce Indico event URLs over to ingest_indico_event, which authenticates via the MCP sidecar instead of storing a login-redirect page).
• action: sso_retry — when sso_fallback_enabled: true, the data-manager retries via CERNSSOScraper if anonymous LinkScraper lands on a Keycloak page. Requires data_manager.sources.sso.enabled: true and SSO_USERNAME / SSO_PASSWORD secrets.

services:
  chat_app:
    tools:
      ingest_url:
        sso_fallback_enabled: true
        routing_rules:
          # 1) Indico event pages -> ingest_indico_event (auth'd via MCP, no Selenium)
          - pattern: '^https?://[^/]*indico\.[^/]*/event/(?P<event_id>\d+)'
            action: refuse
            scraper: indico_mcp
            message: |
              Error: this URL is an Indico event page (event_id={event_id}).
              `ingest_url` cannot authenticate against CERN SSO and would store the
              login redirect page. Call `ingest_indico_event(event_id="{event_id}")`
              instead — it drives the bearer-authenticated Indico MCP server.
          - pattern: '^https?://[^/]*indico\.[^/]*/export/event/(?P<event_id>\d+)\.json'
            action: refuse
            scraper: indico_mcp
            message: |
              Error: Indico API export (event_id={event_id}). Call
              `ingest_indico_event(event_id="{event_id}")` instead.
          # 2) Any other CERN host -> eligible for Selenium-driven SSO fallback
          - pattern: '^https?://[^/]*\.cern\.ch(/|$)'
            action: sso_retry
            scraper: sso

Data-manager SSO source (required for sso_retry)

data_manager:
  sources:
    links:
      selenium_scraper:
        enabled: true
        use_for_scraping: true
        selenium_class: CERNSSOScraper
        selenium_class_map:
          CERNSSOScraper:
            class: CERNSSOScraper
            kwargs:
              headless: true
              max_depth: 5
    sso:
      enabled: true

[hassan11196]

Dedicated skill for indico MCP
https://gitlab.cern.ch/archi/cms-compops/-/blob/hassan-dev-vocms248/config/comp_ops/skills/indico.md?ref_type=heads