[WIP]: Investigate algolia as a search replacement

[jackw]

What this PR does / why we need it:

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

[github-actions]

Hello! 👋 This repository uses Auto for releasing packages using PR labels.

✨ This PR can be merged. It will not be considered when calculating future versions of the npm packages and will not appear in the changelogs.

[Copilot]

Pull request overview

This WIP pull request investigates replacing the current Lunr-based search with Algolia search for the Docusaurus documentation site. The changes include removing the docusaurus-lunr-search plugin, adding Algolia's search SDK, creating crawling scripts to generate search indices, and updating the configuration to use Algolia's DocSearch integration.

Key changes:

• Replaces Lunr search with Algolia DocSearch integration
• Adds crawling infrastructure using Crawlee to generate Algolia search indices
• Updates environment configuration and deployment workflows for testing

Reviewed changes

Copilot reviewed 10 out of 12 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
package-lock.json	Updates dependencies: removes docusaurus-lunr-search, adds algoliasearch 5.45.0 and crawlee 3.15.3
docusaurus/website/package.json	Adds algoliasearch and crawlee dependencies, removes docusaurus-lunr-search, adds generate-index script
docusaurus/website/docusaurus.config.ts	Removes lunr-search plugin config, adds Algolia DocSearch configuration with environment variables
docusaurus/website/.env.production	Adds Algolia environment variables (app ID, API key, index name)
docusaurus/website/.env.development	Adds Algolia environment variables for development
docusaurus/website/scripts/algolia/crawl.ts	New script to crawl documentation site and generate Algolia-formatted records
docusaurus/website/scripts/algolia/uploadToAlgolia.ts	New script to upload records to Algolia index
docusaurus/website/scripts/algolia/types.ts	TypeScript type definitions for Algolia DocSearch records
docusaurus/website/src/theme/tracking/index.ts	Removes search tracking initialization
docusaurus/website/src/css/custom.css	Comments out old search styles, adds Algolia DocSearch dark theme styles
.gitignore	Adds storage directory to gitignore
.github/workflows/deploy-to-developer-portal-dev.yml	Temporarily disables deployment, adds server startup for testing crawl script

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[hugohaggmark]

This could disable the search page completely if we want to. We don't have one in prod today 👍