feat(speculation): add path scorer extension (#316)

Add the scorer seam from the speculation RFC, as a vendor-agnostic
extension interface under submitqueue/extension/speculation/pathscorer/.

The scorer computes each speculation path's predicted-success score from
the current state: the per-batch scores of the path's base batches
(entity.Batch.Score) and which of those dependencies have resolved
(landed or build-passed), plus optionally other signals. It is a
prediction over live state, so the controller re-runs it on every
respeculate right after reconciling status, and persists the result; the
scorer owns only the formula.

The controller hands it the batch's speculation tree directly — the
subject it scores. Any richer signal an implementation needs (dependency
batch scores, historical pass rates) is injected at its Factory, not put
in the signature. It never writes: its only output is per-path scores
([]entity.PathScore — path ID plus fresh score, an entity-level
seam-output type alongside the path-decision type), which the controller
merges into the tree and persists, staying the single writer of tree
state; structure and status never pass through the scorer. Scores are
probabilities in [0, 1] — the contract every implementation must
satisfy, enforced by the controller on consume.

This is the per-path scorer, distinct from the existing per-batch score
stage (`extension/scorer`) that sets entity.Batch.Score — the path
scorer consumes those to score whole paths.

Follows the repo extension contract: Factory.For(Config) (Scorer, error)
with Config carrying only QueueName. Includes README, gomock package,
and a programmable fake. The speculation RFC's seam descriptions are
updated to match the identity-keyed minimal-output contracts (and gain a
design-decision entry for assigned path identity). Interface only;
concrete impls and controller wiring are deferred.


## Stack
1. #337
1. #315
1. @ #316
1. #317
1. #320
1. #331
1. #332
1. #333

Author: behinddwalls

Makefile

@@ -364,7 +364,7 @@ local-stovepipe-stop: ## Stop the Stovepipe service
 
 mocks: ## Generate mock files using mockgen
 	@echo "Generating mocks..."
-	@$(BAZEL) run @rules_go//go -- generate ./submitqueue/extension/storage/... ./submitqueue/extension/buildrunner/... ./submitqueue/extension/changeprovider/... ./platform/extension/counter/... ./platform/extension/messagequeue/... ./submitqueue/extension/queueconfig/... ./submitqueue/extension/mergechecker/... ./submitqueue/extension/pusher/... ./submitqueue/extension/scorer/... ./submitqueue/extension/conflict/... ./submitqueue/extension/speculation/enumerator/... ./submitqueue/extension/speculation/dependencylimit/... ./submitqueue/extension/validator/... ./platform/consumer/... ./stovepipe/extension/storage/... ./stovepipe/extension/sourcecontrol/...
+	@$(BAZEL) run @rules_go//go -- generate ./submitqueue/extension/storage/... ./submitqueue/extension/buildrunner/... ./submitqueue/extension/changeprovider/... ./platform/extension/counter/... ./platform/extension/messagequeue/... ./submitqueue/extension/queueconfig/... ./submitqueue/extension/mergechecker/... ./submitqueue/extension/pusher/... ./submitqueue/extension/scorer/... ./submitqueue/extension/conflict/... ./submitqueue/extension/speculation/enumerator/... ./submitqueue/extension/speculation/dependencylimit/... ./submitqueue/extension/speculation/pathscorer/... ./submitqueue/extension/validator/... ./platform/consumer/... ./stovepipe/extension/storage/... ./stovepipe/extension/sourcecontrol/...
 	@echo "Mocks generated successfully!"
 
 proto: ## Generate protobuf files from .proto definitions

doc/rfc/submitqueue/speculation.md

@@ -187,13 +187,13 @@ Re-speculation needs no special undo path: the controller refreshes statuses, an
 
 ## Interfaces
 
-The seams are vendor-agnostic extensions, each in its own package; the exact Go signatures live in the source. All are per-queue: the system hands a `Factory` the queue identity, and the factory builds the seam for that queue — so the queue is bound at construction and never re-passed to a method.
+The seams are vendor-agnostic extensions, each in its own package; the exact Go signatures live in the source. All are per-queue: the system hands a `Factory` the queue identity, and the factory builds the seam for that queue — so the queue is bound at construction and never re-passed to a method. Persisted paths carry a controller-assigned identity, and every seam output names a path by that ID rather than restating its Base/Head split — each seam returns only its verdict, and everything else about a path stays controller-owned.
 
 **Decision seams:**
 
-- **Enumerator** (`extension/speculation/enumerator`) — given a batch ID and its ordered active dependencies, returns the batch's speculation tree *structure*: the candidate paths, each a Base/Head split. Pure and deterministic; sets no score and no status.
-- **Scorer** (`extension/speculation/scorer`) — given the speculation tree and the current dependency batches, returns each path's predicted-success score. Called by the controller on every respeculate (during reconciliation) so scores track the live state — dependencies landing, dependency builds passing, siblings failing. Owns the score formula; combines the base batches' `Batch.Score` and their resolved/unresolved state (and optionally other signals).
-- **Selector** (`extension/speculation/selector`) — given a speculation tree (with each path's controller-stamped status and freshly recomputed score), returns a per-path action (`Build` or `Cancel`) for the paths it chooses to act on. It reads status and score and emits actions only; paths it leaves alone are omitted. It is constructed with its **selection limit** and calls it to cap how many paths it builds in parallel.
+- **Enumerator** (`extension/speculation/enumerator`) — given a batch and its ordered active dependencies, returns the batch's candidate paths, each a Base/Head split — structure only. Pure and deterministic; the controller assembles the persisted tree, assigning each path's identity and stamping its status.
+- **Scorer** (`extension/speculation/pathscorer`) — given the speculation tree and the current dependency batches, returns each path's predicted-success score, keyed by path identity. Scores are probabilities in [0, 1]; the controller enforces the range when it merges them into the tree. Called by the controller on every respeculate (during reconciliation) so scores track the live state — dependencies landing, dependency builds passing, siblings failing. Owns the score formula; combines the base batches' `Batch.Score` and their resolved/unresolved state (and optionally other signals).
+- **Selector** (`extension/speculation/selector`) — given a speculation tree (with each path's controller-stamped status and freshly recomputed score), returns a per-path action (`Build` or `Cancel`) for the paths it chooses to act on, each decision naming its path by identity, at most one decision per path. It reads status and score and emits actions only; paths it leaves alone are omitted. It is constructed with its **selection limit** and calls it to cap how many paths it builds in parallel.
 - **Prioritizer** (`extension/speculation/prioritizer`) — given the queue's pending build candidates, returns the subset admitted to run, ranked by score plus any fairness policy. It is constructed with its **prioritization limit** and applies it itself. Operates queue-wide, across all of the queue's in-flight batches.
 
 **Limit policies** — each a signal-driven "how much" seam returning a bound from build-resource and other signals:
@@ -206,6 +206,8 @@ The scorer, prioritizer, and the three limit policies are design-level here —
 
 ## Design decisions
 
+**Persisted paths are referenced by assigned identity.** Each path entry in a persisted tree gets a controller-assigned, immutable, opaque ID when it is first written; scores, decisions, and durable links (such as the path→build mapping) all name paths by it. *Why:* seam outputs stay minimal — an ID plus a verdict — instead of restating structure, and cross-entity links stay valid however the tree is re-derived. *Rejected:* structural reference (restating Base/Head in every output) — couples every consumer to path structure and forces ordered-slice comparison everywhere; an ID derived from the structure — an ID that encodes structure invites parsing, and identity should be free to survive re-enumeration on the controller's terms.
+
 **Two layers: decisions and limits.** Decision seams (enumerator, scorer, selector, prioritizer) — enumeration and scoring *describe* the tree, selection and prioritization *act* on it; limit policies (dependency, selection, prioritization) decide *how much*. *Why:* the "which" is qualitative policy that is stable, while the "how much" must scale with volatile build resources; separating them lets the resource-aware knobs move independently of the decision logic, and lets each be tested in isolation. *Rejected:* baking counts into each decision seam as constants — it hard-codes a policy that needs to breathe with CI capacity.
 
 **Limits are signal-driven, and resources are the primary but not the only signal.** A limit is whatever its policy computes — from available capacity, and optionally historical pass rates, cost, time, or experiment flags. *Why:* speculation aggression should rise and fall with the build system, and the design should not foreclose other inputs. *Rejected:* a single fixed constant, or a static per-queue config value — neither can react to load.

submitqueue/entity/speculation_tree.go

@@ -130,6 +130,19 @@ type SpeculationPathInfo struct {
 	BuildID string
 }
 
+// PathScore is the path scorer's verdict for a single path: the
+// path's identity and its freshly computed predicted-success score. It is the
+// scorer seam's only output — the controller merges scores into the tree by
+// path ID and persists them; tree structure and status never pass through the
+// scorer. Like SpeculationPathDecision, it is ephemeral and never persisted.
+type PathScore struct {
+	// PathID identifies the scored path (SpeculationPathInfo.ID) within the
+	// tree the scorer was handed.
+	PathID string
+	// Score is the path's predicted-success probability, in [0, 1].
+	Score float32
+}
+
 // SpeculationPathDecision is a seam's decision for a single path: the action the
 // controller should take for it. It is the output of both the selector (per
 // batch) and the prioritizer (queue-wide), and is not persisted. A seam returns

submitqueue/extension/speculation/enumerator/enumerator.go

@@ -29,7 +29,7 @@ import (
 // deliberately dumb and purely structural: it mechanically lists candidate
 // Base/Head paths from the dependency batches it is handed and nothing else. It
 // does not score paths — that is the scorer's job (see
-// extension/speculation/scorer), which the controller re-runs on every
+// extension/speculation/pathscorer), which the controller re-runs on every
 // respeculate — it does not decide which paths to build — that is the selector's
 // job (see extension/speculation/selector) — and it does not decide how far
 // back to speculate: the controller gates on the dependency limit and hands

submitqueue/extension/speculation/pathscorer/BUILD.bazel

@@ -0,0 +1,9 @@
+load("@rules_go//go:def.bzl", "go_library")
+
+go_library(
+    name = "go_default_library",
+    srcs = ["pathscorer.go"],
+    importpath = "github.com/uber/submitqueue/submitqueue/extension/speculation/pathscorer",
+    visibility = ["//visibility:public"],
+    deps = ["//submitqueue/entity:go_default_library"],
+)

submitqueue/extension/speculation/pathscorer/README.md

@@ -0,0 +1,19 @@
+# Speculation Path Scorer
+
+Vendor-agnostic interface for scoring the paths in a batch's **speculation tree** — the predicted-success probability of each candidate bet, recomputed as the batch's world changes.
+
+See the [Speculation RFC](/doc/rfc/submitqueue/speculation.md) for the end-to-end design and how scoring fits into the orchestrator pipeline.
+
+## Scorer
+
+A path's score is a **prediction**: *how likely is this bet to pay off, right now?* The scorer answers it from the current state — the per-batch success probabilities of a path's base batches (`entity.Batch.Score`, set by the score stage), which of those dependencies have already landed or had their build pass (resolved assumptions raise confidence), and optionally other signals such as how long the batch has waited or historical pass rates. The score is the common currency the [selector](../selector) and prioritizer both rank on, so keeping it current is what makes both act on the latest reality.
+
+Because it is a prediction over live state, the scorer is **re-run on every respeculate**, right after the controller reconciles path status — so when a dependency lands, its build passes, or a sibling path fails, the surviving paths' scores are recomputed before anything is selected or prioritized. The controller drives *when* to rescore (it is part of reconciliation) and persists the result; the scorer owns the *formula*.
+
+This is the per-**path** scorer, distinct from the per-**batch** [score stage](../../scorer), which sets `entity.Batch.Score`. The path scorer consumes those batch scores to score whole paths. The controller hands it the batch's **speculation tree** directly — the subject it scores — and any richer signal an implementation needs (the dependency batches' scores, historical pass rates) is injected at its factory, not passed in. It never writes: its only output is per-path scores, each naming a path by its ID, and the controller merges them into the tree and persists — the controller stays the single writer of tree state, and everything else about a path (structure, status) never passes through the scorer at all. Paths omitted from the result keep their last persisted score.
+
+Scores are **probabilities in [0, 1]** — 0 is a bet certain to lose, 1 a bet certain to pay off. That is the contract every implementation must satisfy, and the controller enforces the range when it consumes the result. The selector and prioritizer rank on these values, so implementations sharing a queue must agree on this scale.
+
+## Factory
+
+A per-queue factory returns the scorer for a queue, following the repo's extension contract. It is handed only the queue identity; scoring knobs and read access to any extra signals are injected at construction by the integrator in the wiring layer, which resolves per-queue settings through `queueconfig`. Scoring itself stays config-free.

submitqueue/extension/speculation/pathscorer/fake/BUILD.bazel

@@ -0,0 +1,23 @@
+load("@rules_go//go:def.bzl", "go_library", "go_test")
+
+go_library(
+    name = "go_default_library",
+    srcs = ["fake.go"],
+    importpath = "github.com/uber/submitqueue/submitqueue/extension/speculation/pathscorer/fake",
+    visibility = ["//visibility:public"],
+    deps = [
+        "//submitqueue/entity:go_default_library",
+        "//submitqueue/extension/speculation/pathscorer:go_default_library",
+    ],
+)
+
+go_test(
+    name = "go_default_test",
+    srcs = ["fake_test.go"],
+    embed = [":go_default_library"],
+    deps = [
+        "//submitqueue/entity:go_default_library",
+        "@com_github_stretchr_testify//assert:go_default_library",
+        "@com_github_stretchr_testify//require:go_default_library",
+    ],
+)

submitqueue/extension/speculation/pathscorer/fake/fake.go

@@ -0,0 +1,72 @@
+// Copyright (c) 2025 Uber Technologies, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package fake provides a programmable pathscorer.Scorer for tests and
+// examples. By default Score echoes each input path's current score back,
+// keyed by its ID; Returns overrides that with canned path scores, and
+// FailWith injects an error on every call. It is intended for examples and
+// tests only, never production.
+package fake
+
+import (
+	"context"
+
+	"github.com/uber/submitqueue/submitqueue/entity"
+	"github.com/uber/submitqueue/submitqueue/extension/speculation/pathscorer"
+)
+
+// Scorer is a programmable pathscorer.Scorer.
+type Scorer struct {
+	scores    []entity.PathScore
+	hasScores bool
+	err       error
+}
+
+// New returns a fake Scorer that echoes each input path's current score back.
+// Override the returned scores with Returns.
+func New() *Scorer {
+	return &Scorer{}
+}
+
+// Returns makes every Score call return scores instead of echoing its input.
+func (s *Scorer) Returns(scores []entity.PathScore) *Scorer {
+	s.scores = scores
+	s.hasScores = true
+	return s
+}
+
+// FailWith makes every Score call return err.
+func (s *Scorer) FailWith(err error) *Scorer {
+	s.err = err
+	return s
+}
+
+// Score returns the canned scores if set with Returns, otherwise one PathScore
+// per input path echoing the path's current score, unchanged.
+func (s *Scorer) Score(_ context.Context, tree entity.SpeculationTree) ([]entity.PathScore, error) {
+	if s.err != nil {
+		return nil, s.err
+	}
+	if s.hasScores {
+		return s.scores, nil
+	}
+	scores := make([]entity.PathScore, 0, len(tree.Paths))
+	for _, p := range tree.Paths {
+		scores = append(scores, entity.PathScore{PathID: p.ID, Score: p.Score})
+	}
+	return scores, nil
+}
+
+// ensure the fake satisfies the interface.
+var _ pathscorer.Scorer = (*Scorer)(nil)

submitqueue/extension/speculation/pathscorer/fake/fake_test.go

@@ -0,0 +1,51 @@
+// Copyright (c) 2025 Uber Technologies, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package fake
+
+import (
+	"context"
+	"errors"
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+	"github.com/uber/submitqueue/submitqueue/entity"
+)
+
+func TestScore_EchoesInputScoresByDefault(t *testing.T) {
+	tree := entity.SpeculationTree{
+		BatchID: "q/batch/2",
+		Paths: []entity.SpeculationPathInfo{
+			{ID: "q/batch/2/path/0", Path: entity.SpeculationPath{Head: "q/batch/2"}, Score: 0.9},
+		},
+	}
+	got, err := New().Score(context.Background(), tree)
+	require.NoError(t, err)
+	assert.Equal(t, []entity.PathScore{{PathID: "q/batch/2/path/0", Score: 0.9}}, got)
+}
+
+func TestScore_ReturnsCanned(t *testing.T) {
+	in := entity.SpeculationTree{BatchID: "q/batch/2"}
+	scores := []entity.PathScore{{PathID: "q/batch/2/path/0", Score: 0.75}}
+	got, err := New().Returns(scores).Score(context.Background(), in)
+	require.NoError(t, err)
+	assert.Equal(t, scores, got)
+}
+
+func TestScore_FailWith(t *testing.T) {
+	sentinel := errors.New("boom")
+	_, err := New().FailWith(sentinel).Score(context.Background(), entity.SpeculationTree{BatchID: "q/batch/1"})
+	require.ErrorIs(t, err, sentinel)
+}

submitqueue/extension/speculation/pathscorer/mock/BUILD.bazel

@@ -0,0 +1,13 @@
+load("@rules_go//go:def.bzl", "go_library")
+
+go_library(
+    name = "go_default_library",
+    srcs = ["pathscorer_mock.go"],
+    importpath = "github.com/uber/submitqueue/submitqueue/extension/speculation/pathscorer/mock",
+    visibility = ["//visibility:public"],
+    deps = [
+        "//submitqueue/entity:go_default_library",
+        "//submitqueue/extension/speculation/pathscorer:go_default_library",
+        "@org_uber_go_mock//gomock:go_default_library",
+    ],
+)