Implement server-side ad slot templates with PBS and APS auction#680
Open
prk-Jr wants to merge 70 commits into
Open
Implement server-side ad slot templates with PBS and APS auction#680prk-Jr wants to merge 70 commits into
prk-Jr wants to merge 70 commits into
Conversation
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Incorporate all review feedback (aram356 + jevansnyc): cache contract, consent/GDPR gating, async restructuring detail, CreativeOpportunityFormat schema, glob pattern fix, XSS escaping, win notifications, APS params, timeout config key, defineSlot fix, gpt.rs ownership, KV migration path, Phase 2 sketch - Fix Prettier formatting (format-docs CI) - Add implementation plan (12 tasks, TDD, ordered by dependency)
- Incorporate all review feedback (aram356 + jevansnyc): cache contract, consent/GDPR gating, async restructuring detail, CreativeOpportunityFormat schema, glob pattern fix, XSS escaping, win notifications, APS params, timeout config key, defineSlot fix, gpt.rs ownership, KV migration path, Phase 2 sketch - Fix Prettier formatting (format-docs CI) - Add implementation plan (12 tasks, TDD, ordered by dependency)
Replace the head-injected __ts_bids design with a server-cached bid delivery model fetched by the client via a new /ts-bids endpoint. The auction never blocks page rendering — </head> flushes immediately, body parses without waiting for bids, and the client fetches bids in parallel with content paint. Key changes: - §2 Goal: bid delivery decoupled from page rendering; FCP unchanged from no-TS baseline - §4.3 Auction Trigger: drop buffered/streaming dichotomy; single mode forces chunked encoding on all origins (WordPress, NextJS, etc.) - §4.4 Head Injection: only __ts_ad_slots and __ts_request_id injected at <head> open; bid results moved to /ts-bids endpoint - §4.6 Client Residual: __tsAdInit defines slots immediately, fetches bids via /ts-bids, applies targeting and fires refresh() after resolve - §4.7 (new) Caching Behavior: explicit cacheability table for HTML, JS, CSS, tsjs bundle, bid results; Fastly edge HTTP cache leveraged for origin HTML - §5 Request-Time Sequence: full mermaid diagram covering content + creative + burl flow with cache-hit and cache-miss branches; separate text sequences for cache-hit (~80ms FCP, ~900ms ad-visible) and cache-miss (~250ms FCP, ~1,050ms ad-visible) - §6 Performance Summary: cache-hit and cache-miss columns; FCP added as a tracked metric - §7 Implementation Scope: add bid_cache.rs, /ts-bids endpoint, force chunked encoding step - §8 Edge Cases: origin-agnostic entries; new entries for /ts-bids 404 and client-never-fetches-/ts-bids Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pivot from the /ts-bids fetch endpoint + in-process bid_cache design to
inline __ts_bids injection before </body>. The earlier design relied on
shared state that doesn't reliably survive Fastly Compute's per-request Wasm
isolate model — body injection achieves the same FCP property in a single
response with no shared-state requirement.
Key changes:
- §4.3: replace /ts-bids long-poll with bounded </body> hold tied to
A_deadline. Body content above </body> paints first; close-tag held
until auction completes or A_deadline fires (graceful __ts_bids = {}
fallback).
- §4.3: add auction-eligibility gating (consent, bot UA, prefetch hints,
HEAD method, slot match) so auctions fire on real first-page-load
impressions only.
- §4.4: replace __ts_request_id + /ts-bids machinery with two inline
<script> blocks — __ts_ad_slots at <head> open, __ts_bids before
</body> via lol_html el.on_end_tag().
- §4.5: move both nurl and burl to client-side firing from
slotRenderEnded after hb_adid match. Server-side firing rejected to
avoid billing inflation on bids that never render.
- §4.6: replace fetch+Promise pattern with synchronous __ts_bids read.
Add lazy slim-Prebid loader (post-window.load) for scroll/refresh
auctions and Phase B identity warm-up. Add ts_initial=1 slot-ownership
sentinel.
- §4.7: switch Cache-Control from private, no-store to private,
max-age=0 to preserve browser BFCache eligibility while still
preventing intermediate-cache leaks.
- §4.8 (new): document the EC/KV identity model as load-bearing auction
input — Phase A retrieval at request time, Phase B post-render
enrichment via slim-Prebid userID modules. Add bare-EC first-impression
caveat and auction_eid_count metric. Note federated-consortium
passphrase property and clickstream-compounding speed win.
- §5: update mermaid + cache-hit/miss timelines for bounded body hold;
ad-visible converges to ~870ms (hit) / ~1,020ms (miss).
- §6: drop /ts-bids RTT row; add DCL row; add clickstream-compounding,
TS-overhead, identity-coverage, and confidence-interval framing.
- §7: drop bid_cache.rs and /ts-bids endpoint from scope; add
auction-eligibility gating and slim-Prebid bundle build target. Add
explicit "Deleted" subsection.
- §8: drop /ts-bids edge cases; add SPA/pushState, bare-EC, bot/prefetch,
HEAD, BFCache restoration cases.
- §9.6: server-side GAM downgraded from "Phase 2 commitment" to
aspirational and contingent on Google agreement. §9.8 (slim-Prebid
bundle composition), §9.9 (Privacy Sandbox), §9.10 (per-bidder consent)
added as follow-ups.
Implementation plan at docs/superpowers/plans/2026-04-30-server-side-ad-templates.md
is now stale relative to this spec; needs regenerating before code lands.
…ities.toml Adds the creative_opportunities field to Settings struct to deserialize configuration for the server-side ad auction feature. Includes build.rs stubs for types required during build-time configuration validation. Creates creative-opportunities.toml with example slot configuration and updates trusted-server.toml with the [creative_opportunities] section defining GAM network ID, auction timeout, and price granularity settings. Tests pass with proper TOML parsing of the creative_opportunities section.
…ared auction state
- Add `ad_slots_script: Option<String>` and `ad_bids_state: Arc<RwLock<Option<String>>>` fields to `HtmlProcessorConfig`
- Update `from_settings` to initialize both new fields with safe defaults
- Prepend `ad_slots_script` inside the existing `<head>` handler before integration inserts
- Add `element!("body", ...)` handler that uses `end_tag_handlers()` to inject `__ts_bids` before `</body>`; falls back to empty `{}` when auction state is `None`
- Add `IntegrationRegistry::empty_for_tests()` test helper
- Add three new tests covering all injection paths
…gibility gates; max-age=0 - Make handle_publisher_request async; add orchestrator and slots_file params - Dispatch origin request with send_async before running auction in parallel - Gate auction on GET, no prefetch, no bot, matched slots, TCF purpose-1 consent - Run server-side auction and write bucketed bids to ad_bids_state Arc<RwLock> - Compute ad_slots_script after response headers; set Cache-Control: private, max-age=0 - Fix Stream arm to thread actual ad_slots_script and ad_bids_state through - Add build_auction_request, build_bid_map, build_bids_script, build_ad_slots_script helpers - Update route_tests.rs to pass empty slots_file to route_request
…m slotRenderEnded
- build_bid_map now returns serde_json::Map with full bid objects (hb_pb,
hb_bidder, hb_adid, nurl, burl) instead of a plain CPM string map
- build_bids_script / build_ad_slots_script now emit full <script> tags
using JSON.parse("…") for safe inline embedding; add html_escape_for_script helper
- build_ad_slots_script uses correct property names (gam_unit_path, div_id,
formats, targeting) matching the client-side TSJS bundle expectations
- Replace map_or(false, …) with is_some_and(…) on lines 546, 549, 567
- Add # Panics doc sections to handle_publisher_request and create_html_processor
…nities.toml at startup
… from slotRenderEnded; slim-Prebid lazy loader
- Enable APS and adserver_mock in auction config; set providers and mediator - Increase auction_timeout_ms from 500ms to 3000ms — 500ms was too tight for HTTPS round-trips to mocktioneer, leaving the mediator zero budget - Fix mediation request: send numeric price instead of opaque encoded_price; mocktioneer requires a decoded price field and does not support encoded_price - Expand creative-opportunities slot page_patterns to include /news/**
Define SlotRenderEndedEvent, SlotRenderEvent, and TestWindow types to eliminate all @typescript-eslint/no-explicit-any violations in gpt/index.ts and gpt/index.test.ts. Extend GptWindow with __tsjs_slim_prebid_url so installSlimPrebidLoader avoids the any cast.
prk-Jr
changed the title
Set gam_network_id to 88059007 (autoblog production network). Update atf_sidebar_ad slot to /88059007/autoblog/news with div_id ad-atf_sidebar-0-_r_2_ (desktop ATF sidebar, 300x250); restrict page_patterns to article paths only (/20**, /news/**) since that div does not exist on the homepage. Add homepage_header_ad slot targeting /88059007/autoblog/homepage with ad-header-0-_R_jpalubtak5lb_ for 970x90/728x90/970x250 leaderboard formats. Reduce auction_timeout_ms from 3000 to 500 to cap TTFB at the spec-recommended ceiling.
… Purpose 1 when gdpr_applies
…patch_auction dispatch_auction was building AuctionContext with a placeholder Request (GET https://placeholder.invalid/) that carried no headers. Prebid's request_bids copies User-Agent, x-forwarded-for, Referer, Accept-Language, and cookies from context.request before sending to Prebid Server, so SSPs received stripped requests and returned empty bids. Fix: dispatch SSP requests before req.send_async(), using the original request directly as AuctionContext.request. DispatchedAuction holds no lifetime reference to Request, so the borrow ends at return and req can be modified (restrict_accept_encoding, Host header) and sent to origin immediately after.
In collect_dispatched_auction, the select loop checked `auction_start.elapsed() >= deadline` after each SSP response and broke early if the 1500ms budget had elapsed. When origin TTFB + body download exceeded the auction budget, the check fired after collecting the first SSP response, abandoning the second SSP's already-buffered response. This left responses with only one (possibly errored) SSP, causing remaining_ms == 0 which skipped the mediator, and select_winning_bids on the partial set returned zero bids. The deadline break is wrong in this context: SSP HTTP connections are already bounded by the backend first_byte_timeout set at dispatch time (1000ms per provider). By the time collect is called at origin EOF, all SSPs have either responded or been errored by Fastly's host. The select() calls drain instantly — no WASM-level deadline enforcement is needed or safe. Also add info-level log statements at dispatch, collect, and write_bids_to_state to make the auction pipeline observable without requiring a dashboard.
In collect_dispatched_auction, the mediator was skipped when
remaining_budget_ms(auction_start, timeout_ms) == 0. In the async-dispatch
path, auction_start is set before pending_origin.wait(), so elapsed time
includes the full origin TTFB and body download. For heavy SSR pages
(autoblog), this exceeds the 1500ms SSP budget, making remaining_ms == 0
at every collection and causing the mediator to be permanently skipped.
The mediator (adserver_mock) is the primary bid source — SSPs alone
return no bids. Skipping it means window.__ts_bids == {} on every full
page load, while handle_page_bids (which uses the sequential run_auction
path) works correctly because it measures remaining time from after SSP
collection.
Fix: give the mediator its own configured timeout (mediator.timeout_ms())
instead of the exhausted SSP budget. This mirrors how run_parallel_mediation
works: the mediator's deadline is independent of SSP round-trip time.
Side effect: mediator backend name is now stable (always t1000 for adserver_mock)
rather than varying per request with remaining_ms.
Resolve conflicts in: - prebid.rs: keep both PBS stored-request tests (branch) and bid-param override rule validation tests (main) - settings.rs: keep both creative_opportunities (branch) and debug (main) fields in Settings struct - trusted-server.toml: keep [creative_opportunities] section from branch Update handle_page_bids to use compat::from_fastly_headers_ref pattern introduced by main's HTTP type migration (PR11), replacing direct use of fastly::Request with the generic Request<EdgeBody> for cookie parsing, request info extraction, and EC ID generation.
`classify_response_route` returns `BufferedProcessed` when HTML has
post-processors registered (e.g. the Next.js integration registers one
via `with_html_post_processor`). Unlike the `Stream` path, which drives
`one_behind_loop` to collect the dispatched auction at origin EOF, the
`BufferedProcessed` branch previously discarded `dispatched_auction`
entirely — so `ad_bids_state` stayed `None` and lol_html injected the
fallback `window.__ts_bids = {}` instead of real bids.
Fix: collect the in-flight dispatched auction in the `BufferedProcessed`
branch before calling `process_response_streaming`, using the same
`collect_dispatched_auction` + `write_bids_to_state` pattern that the
stream path uses. The `debug.auction_html_comment` injection is mirrored
here as well so the comment appears in both code paths when enabled.
html_escape_for_script now unicode-escapes <, >, & and U+2028/2029 in addition to \ and ". These characters allow a crafted bid value to break out of the <script> block or terminate the JS string in some parsers. AuctionOrchestrator::collect_dispatched_auction now computes remaining budget (A_deadline - elapsed) before invoking the mediator. If the budget is already exhausted the mediator is skipped and SSP bids are returned directly; otherwise the mediator timeout is capped at the tighter of its configured value and the remaining budget, preventing it from running past A_deadline.
…erride Pass the real incoming request to AuctionContext in handle_page_bids instead of a placeholder — SSPs now receive browser UA, referer, and cookies on SPA navigation bids. Guard Cache-Control in finalize_response so operator response_headers cannot overwrite the private/no-store directives set for per-user HTML and page-bids responses. Disable auction_html_comment debug flag in trusted-server.toml.
The mock mediator endpoint does not echo nurl/burl/ad_id back in its response. Build a bid index in request_bids keyed by (provider, slot_id, bidder) — where bidder is recovered from the echoed crid field — and restore the fields in parse_mediation_response from the original SSP bids. Fixes the spec requirement: both nurl and burl must travel in __ts_bids for client-side sendBeacon firing on slotRenderEnded (§4.5).
APS reads user agent from request.device — without it, real APS bids arrive with wrong or missing device targeting. Pass the incoming UA from both the page-load and page-bids auction paths.
slotRenderEnded gives a div element id via getSlotElementId(), but __ts_bids is keyed by slot id. Build a divToSlotId map during slot setup (matching the TS implementation) and use it in the event handler. Without this, nurl/burl beacons and hb_adid match checks silently fail in the server-rendered fallback whenever div_id != id.
prk-Jr
requested review from
ChristianPavilonis and
aram356
and removed request for
aram356
PBS bidder credentials (mocktioneer, criteo placeholder params) were being sent directly to PBS on every auction request. Per the design spec, PBS bidder params belong in PBS stored requests keyed by slot ID, not in the edge config file. Removes PbsSlotParams struct, SlotProviders.pbs field, the to_ad_slot wiring block, and the corresponding test. Slots without inline bidder params trigger the existing storedrequest fallback path in the Prebid provider. Closes #697
- Rewrite misleading comment in apply_floor_prices: price=None bids pass through in the parallel-only path because decoding is deferred; in the mediation path the mediator decodes prices before this function runs - Add test: decoded APS bid below slot floor is dropped - Add test: decoded APS bid at or above slot floor is kept Closes #698
- Expand handle_auction doc: inline-params vs stored-request paths, config passthrough and allowed_context_keys, response headers - Document AdRequest, AdUnit, BidConfig with the stored-request contract: absent/empty bids → empty bidders map → PBS stored-request fallback - Add tests for convert_tsjs_to_auction_request: - No bids → empty bidders map (stored-request path) - Inline bids → bidders map populated - Allowed config key passes through; disallowed key dropped - Invalid 3-element banner size returns error Closes #699
- Add debug log at no-match gate in handle_publisher_request and
handle_page_bids so operators can confirm the feature is inactive
on non-article URLs without reading source code
- Add test: empty slots file (kill-switch) returns slots:[] bids:{}
- Add test: URL not matching any slot pattern returns slots:[] bids:{}
Closes #700
- Clarify in handle_auction doc that /auction is for initial render and programmatic callers; scroll/refresh/SPA navigation is slim-Prebid's domain in Phase 1 - Note Phase 2 slot-template-aware refresh API as deferred future work - Add head_inserts doc clarifying __tsAdInit handles initial render only; slotRenderEnded fires win beacons but does not trigger refresh auctions Closes #702
…ie in auction requests
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
creative-opportunities.toml: slots are matched against the request URL at the edge, andwindow.__ts_ad_slotsis injected before<head>closes so the browser never needs a separate fetch.window.__ts_bidswith price-bucketed targeting before</body>, enabling GAM to apply server-won bids synchronously on first render.__tsAdInitinline script that reads__ts_bidssynchronously and drives the GPT/GAM auction flow client-side without a network round-trip for bid data.creative-opportunities.toml.__ts_bidsinjection on pages where the origin HTML contains more than one<body>element (CMS/template pages).Changes
creative-opportunities.tomlcrates/trusted-server-core/src/creative_opportunities.rsPbsSlotParamsremoved — PBS bidder params belong in PBS stored requestscrates/trusted-server-core/src/publisher.rs__ts_ad_slotsand__ts_bidsinjection via shared statecrates/trusted-server-core/src/price_bucket.rscrates/trusted-server-core/src/settings.rsCreativeOpportunitiesConfigand auction config wired intoSettingscrates/trusted-server-core/src/integrations/aps.rscrates/trusted-server-core/src/integrations/adserver_mock.rscrates/trusted-server-core/src/integrations/prebid.rsaps) from PBS imp bidder mapcrates/trusted-server-core/src/auction/orchestrator.rsprice=Nonebids (APS) pass throughapply_floor_pricesin the parallel-only path; in the mediation path the mediator decodes prices first so floor is always enforced on decoded values. Adds tests for decoded APS bids above/below slot floor.crates/trusted-server-core/src/auction/types.rsSiteInfo,AdSlot,AdFormat,Bidextensionscrates/trusted-server-core/src/html_processor.rsAtomicBoolguard prevents double__ts_bidsinjection on pages with multiple<body>elementscrates/trusted-server-core/src/openrtb.rsPrebidImpExtwith optionalstoredrequest; skips empty bidder mapcrates/trusted-server-core/src/integrations/gpt.rs__tsAdInitscript generation and GPT slot wiringcrates/trusted-server-adapter-fastly/src/main.rscreative-opportunities.toml, build orchestrator, pass to publisher handlercrates/js/lib/src/integrations/gpt/index.ts__ts_bidsand__ts_ad_slotstrusted-server.tomlauction_timeout_ms = 3000Closes
Closes #677
Closes #697
Closes #698
Closes #699
Closes #700
Closes #702
Test plan
Automated
cargo test --workspacecargo clippy --workspace --all-targets --all-features -- -D warningscargo fmt --all -- --checkcd crates/js/lib && npx vitest runcd crates/js/lib && npm run formatcd docs && npm run formatcargo build --package trusted-server-adapter-fastly --release --target wasm32-wasip1Manual end-to-end (browser DevTools console)
The steps below build on each other — each step uses data from the previous one so no IDs need to be memorised.
Step 1 — Verify slot config is injected at
<head>openNavigate to a news/article URL (a path matching
/20**or/news/**).Expected: array of slot objects. Each entry has
id,gamUnitPath,divId,formats, andtargeting. Note thedivIdvalue from the matching slot — you will use it in step 3.What this proves: the edge matched the URL against the slot's page patterns and injected the slot config synchronously in
<head>, before any JS ran.Step 2 — Verify server-side auction result is injected before
</body>Expected: object keyed by slot ID (same ID as in step 1), containing
hb_bidderandhb_pbfrom the winning provider.What this proves: the server-side auction (APS + PBS running in parallel) completed, the mediator picked a winner, price was bucketed, and the result was injected into the HTML before the page was sent to the browser.
Step 3 — Verify
__tsAdInitwired the GPT slot with bid targetingUsing the
divIdobserved in step 1:Expected: one entry with the GAM unit path from step 1 and targeting that includes
hb_pb,hb_bidder(matching step 2), plus any slot-level keys (pos,zone) andts_initial: ["1"].Step 4 — Verify slot matching is page-pattern-aware
Navigate to the homepage (
/). Repeat step 2.Expected:
window.__ts_bidscontainshomepage_header_ad(notatf_sidebar_ad).Step 5 — Confirm no duplicate injection
View page source. Search for
__ts_bids.Expected: exactly one occurrence of
window.__ts_bids=immediately before</body>.Pending (GAM line items required)
Step 6 — Creative delivery requires standard Prebid header-bidding line item setup in GAM targeting
hb_pbandhb_bidder(outside this PR's scope).Checklist
unwrap()in production code — useexpect("should ...")logmacros (notprintln!)