IndexNow integration: instant search engine notification on content changes

[chubes4]

Problem

When Data Machine creates or updates content (events via upsert_event, wire posts via publish_wordpress), search engines don't know about it until they crawl. For Bing/DuckDuckGo/Yandex, this can take days or weeks.

IndexNow is a protocol that lets you ping search engines instantly when content changes. Since Data Machine already knows the exact moment content is created/updated, this is a natural integration point.

Proposed solution

Option A: Post-publish hook (simplest)

Hook into WordPress publish_post / save_post actions and ping IndexNow endpoints automatically. Could be a Data Machine setting to enable/disable.

Option B: IndexNow handler in pipeline flows

A new handler type indexnow that can be added as a pipeline step after upsert_event or publish_wordpress:

fetch → ai → publish_wordpress → indexnow

Option C: Standalone ability

wp datamachine indexnow submit --url=https://events.extrachill.com/events/some-event/
wp datamachine indexnow submit-batch --sitemap=https://extrachill.com/sitemap.xml

IndexNow protocol

• Endpoint: https://api.indexnow.org/IndexNow
• Params: url, key, keyLocation
• Supported by: Bing, Yandex, DuckDuckGo (not Google)
• Free, no rate limits for legitimate use
• Key is a simple text file hosted at site root

Why this matters

• 20% of traffic comes from Bing-indexed search engines
• events.extrachill.com has 182 flows creating/updating events constantly
• wire.extrachill.com auto-publishes from 13 subreddits
• Getting this content indexed immediately instead of waiting for crawl = faster traffic

Recommendation

Option A as the quick win (simple hook, covers all content). Option B for pipeline-level control later.