Docs search crawler is finding 0 records

[jbw976]

The Algolia crawler that powers the search results on the docs site gave an alert today:

Too many missing records
The new index generated by this crawl is missing too many records to replace the production index automatically

SafeReindexingError: [crossplane] Blocking error:
  The difference between the number of records:
    from : 4.5k
    to   : 0
  is too large (100 %), this limit can be modified in the Editor (currently 10 %)

I did expect a drop in number of records as the v2.0-preview docs are being removed in #1105, but that PR hasn't actually been merged yet.

This error is saying that we're about to drop ALL records in the index, so something else is very wrong here 🤓

Looking more closely at https://crawler.algolia.com/admin/crawlers/b1863584-45fc-4117-831f-8d68e47b2e72/monitoring/list?reason=success, it looks like there is only 1 successfully crawled URL and it's the sitemap at https://docs.crossplane.io/sitemap.xml

So something seems wrong with the sitemap where the Algolia crawler can't crawl any further than that sitemap and ends up with 0 records for the search index.

#1104 touches the sitemap but that PR isn't merged either! I'm really not sure what's going on here yet.

[RonaldLePape]

I checked the public docs site:

sitemap.xml is valid XML and contains many URLs
sitemap URLs return HTTP 200
robots.txt is permissive
page content is present in the server-rendered HTML
DocSearch integration is still present in the page

Based on that, I couldn't find an obvious site-side reason why the crawler would only index the sitemap. It may require someone with Algolia crawler access to inspect crawl logs/configuration and see whether URLs are being discovered, filtered, or rejected.

One thing I would still check locally is whether this is a recent docs theme/layout change. The page contains:

<div class="bd-content ps-lg-2 DocSearch-content">

If the crawler configuration was written against an older selector (e.g. .td-content, .content, article), and a recent theme migration happened, then a maintainer with Algolia access could immediately verify that hypothesis.

[jbw976]

Thanks for taking a look also @RonaldLePape!

This may have just been a transient issue - @tr0njavolta ran the crawl manually and it was able to successfully extract all the records it had on the previous successful crawl. I couldn't find more details in the Algolia monitoring/logs about what caused the failure, but nice that it doesn't look persistent 😅