[Protocol Feedback] Reliability issues on Shared Hosting environments (HTTP 429 errors) #67
Description
Description
Even though this repository focuses on the WordPress implementation, I am reporting a systemic issue with the IndexNow protocol logic that affects millions of users on shared hosting environments (e.g., STRATO, IONOS, Bluehost).
The Problem: IP-based Rate Limiting
Currently, IndexNow seems to rely heavily on IP-based rate limiting. In shared hosting environments, hundreds of independent websites share a single outgoing server IP.
When multiple websites on the same shared server utilize IndexNow, the shared IP quickly hits the HTTP 429 - Too Many Requests threshold. This effectively blocks legitimate updates from smaller websites that do not have the resources for a dedicated IP address.
Proposed Improvement
Since IndexNow already requires a Unique API Key (via host-side text file validation) to verify domain ownership, I propose shifting the rate-limiting logic:
- From: Source IP-based throttling
- To: Domain/API-Key-based throttling
Benefits
- Fairness: Smaller websites won't be penalized for the activity of their "server neighbors."
- Efficiency: Reliable push-updates reduce the need for search engines to perform resource-heavy "discovery crawling."
- Adoption: This change would make IndexNow a viable and stable tool for the global CMS community (Joomla, WordPress, etc.) beyond dedicated server setups.
I would appreciate it if the IndexNow core team could review how rate limits are applied to requests that are already authenticated via a valid key.
Environment:
- Hosting Provider: STRATO (Shared Environment)
- Error received: HTTP 429