Skip to content

detection-methods.md

Latest commit

 

History

History
136 lines (95 loc) · 4.81 KB

detection-methods.md

File metadata and controls

136 lines (95 loc) · 4.81 KB
Raw

TokenAPI Scan — Detection Methodology · 检测方法论

What we test, how we score, and why you can trust the results. 我们检测什么、如何评分、以及为什么结果可信。

This document describes the public methodology behind tokenscanai.com. It does not include implementation details of the detection engine itself, which is proprietary.

1. What is an "AI API relay"?

An AI API relay (中文：中转站) is a third-party service that proxies requests to one or more major LLM provider APIs (OpenAI, Anthropic, Google, Mistral, xAI, etc.) and re-sells access to end users.

Relays exist primarily because:

  • Major LLM providers restrict signups from certain regions
  • Some users prefer pay-as-you-go in local currency / WeChat / Alipay
  • Some teams want a single API key across multiple model providers

The market has hundreds of such services with highly variable reliability. TokenAPI Scan exists to surface the difference.

2. What we measure

For each monitored relay, we continuously collect:

2.1 Reachability

  • HTTP(S) reachability from probes in mainland China and overseas
  • DNS resolution stability
  • TLS certificate validity and expiry
  • Response latency distribution

2.2 Model availability

  • We call the relay's /v1/models (or equivalent) endpoint
  • We cross-reference the returned model list against the relay's marketing claims on its public website
  • We flag relays that list models they do not actually serve

2.3 Transparency

  • Presence of a public pricing page
  • Presence of public ToS / refund policy / contact information
  • Use of a stable domain (not behind frequent domain rotation)
  • ICP filing (for China-based relays) — informational only

2.4 Network profile

  • Which CDN / cloud provider fronts the service
  • Whether the relay is blocked in mainland China at the network level
  • Whether the relay's claimed "China-direct" routing actually works

2.5 Longevity

  • First-seen date in our database
  • Continuous uptime since first probe
  • History of past outages

3. How tiers are assigned

Every relay is placed into one of five tiers, updated automatically:

Tier Meaning
S Long-running, transparent, model claims fully verified, both China & overseas reachable
A Stable, mostly transparent, minor caveats
B Working but with limitations (e.g., overseas only, or model list partially unverified)
C Functional but with red flags (opaque pricing, recent outages, partial model claims)
D Active warnings — frequent outages, fake model lists, or known to have disappeared with user funds

Tier assignments are algorithmic, not editorial. They are recomputed on every probe cycle based on the measurements in §2.

We do not accept payment to change tier assignments. We do not accept payment to remove a site from monitoring.

4. What we do not test

To be transparent about our limits:

  • We do not test output quality of the underlying model. A relay that faithfully forwards GPT-4o requests gets the same model-quality assessment as OpenAI itself — because the model is the same.
  • We do not audit billing accuracy. We report what the pricing page says.
  • We do not verify business identity beyond what the relay publishes.
  • We do not predict the future. A Tier S relay today may disappear tomorrow. The Tier system captures current observable signals, not certainty.

5. Sources of information

For each site, our public detail page (/site/<slug>) shows:

  • ✅ Live probe results (timestamped)
  • ✅ Monthly automated screenshots of the site's homepage
  • ✅ Network reachability matrix (CN / overseas)
  • ✅ Claimed vs. verified model list
  • ✅ LLM-generated qualitative summary (clearly labeled as AI-generated draft)

The LLM summary is generated by ARK code-latest and is reviewed before publication. It cites our own probe data, not external reviews.

6. How to challenge a result

If you operate a relay and believe an assessment is incorrect:

  1. Read the probe log on your site's detail page — it shows exactly what we tested and when
  2. Open an issue at https://github.com/haruki3rd/tokenscanai/issues with the slug of your site and the specific data point you dispute
  3. We will re-probe within 48 hours and post the fresh result publicly

We do not negotiate tier changes. We fix data errors when shown evidence.

7. Updates to this methodology

Material changes to detection methods are logged in changelog.md with a dated entry. Changes that affect tier assignments retroactively are called out explicitly so historical comparisons remain meaningful.

Last reviewed: 2026-05 · Maintained at https://tokenscanai.com