eval: add TraceMode to omit the value Trace on export

[borisschlosser]

Summary

Object merges set Trace.Base on every produced value, and serialization recurses through the whole chain. A leaf at the end of a D-deep import-merge chain therefore carried ~D partial copies of its ancestors on the wire, so opened payloads grew super-linearly in import depth for the same logical content. Combined with a Value.UnmarshalJSON whose cost was O(size × depth), opened-payload decode came to dominate CPU on import-heavy environments (a large fraction of esc.(*Value).UnmarshalJSON on a production profile).

This PR lets callers drop the Trace on paths that never read it, so those payloads stay bounded by their logical content.

What changed

eval.TraceMode + EvalOptions

Two modes, with the historical behavior as the default:

• TraceModeFull — the zero value. Builds the entire Trace (Def + the full Base chain), so EvalOptions{} is byte-for-byte the prior behavior and no consumer of Trace regresses without opting in.
• TraceModeNone — exports each value with a zero Trace (no Def, no Base chain). Bounds opened-payload size by logical content instead of import-merge depth.

The Trace is built directly during export — (*value).export takes the TraceMode (sourced from the evalContext, which is set once per evaluation and shared with imports, so export memoization stays consistent). Under TraceModeNone the dropped Trace is never allocated; there is no build-then-strip post-processing pass.

The three eval entry points (EvalEnvironment, CheckEnvironment, RotateEnvironment) gain a final EvalOptions argument.

[pgavlin]

Having a TraceMode seems great--in the majority of cases we probably don't need the full trace. Some high-level feedback:

• I don't think we need the Collapsed variant. IMO the value in the trace is being able to walk the complete def chain--walking one level doesn't really help much.
• We should not change the defaults in the public API. The default should remain Full.
• We should not implement by post-processing traces: we should just change (*value).export to not build the trace in the first place.
• We should not roll our own JSON decoder for the value struct. Instead, I think we should add another struct for decoding that hand-rolls only the bits necessary to handle the polymorphic value property. We should also consider using segmentio's JSON decoder, which can do zero-copy decoding from RawMessages. If we want to push performance as far as we can, we might want to investigate easyjson, which does ahead-of-time code generation for JSON serde.

[pgavlin]

Also--wondering if we have profile data backing up these changes? We should take them regardless but I'm curious to know what else is hot

[borisschlosser]

@pgavlin this PR now focuses on not calculating traces when TraceModeNone is set. The majority of callers are env open calls which never read or use the traces at all. So these callers can now opt in to not calculating traces at all which improves overall system performance significantly. Other callers continue to calculate and report traces (TraceModeFull).

If we want to further improve performance during trace calculation we can switch the json library for improved processing. But we should wait until the TraceMode changes land and profile the code again.

Please find an AI research below:

JSON library options for esc.(*Value).UnmarshalJSON — follow-up comparison

The intrinsic cost the stdlib decoder leaves on the table is the double-scan of the value field (RawMessage capture + typed re-unmarshal) plus reflection overhead. The awkward bit is the polymorphic value any field — anything we adopt has to handle null/bool/number/string/[]Value/map[string]Value dispatch.

	segmentio/encoding	mailru/easyjson	encoding/json/v2 (go-json-experiment)
Mechanism	Drop-in reflective decoder, asm-accelerated scanner	AOT codegen per type	Streaming decoder with a shared-decoder hook
Fixes the double-scan?	No — faster scanner, but our UnmarshalJSON still captures+re-parses	Yes — single-pass via jlexer	Yes — UnmarshalJSONFrom(*jsontext.Decoder) decodes the whole tree in one pass
Polymorphic value field	Same custom UnmarshalJSON as today	Hand-write a jlexer decoder for that field	Dispatch via dec.PeekKind(), recurse with json.UnmarshalDecode(dec, &child)
Hand-rolling parsing?	No	Yes (jlexer) — the thing pgavlin objected to, in a lib's API	No — the streaming primitive is library-provided
Dependency status	Already an indirect dep; maintenance-mode	Stable but low-maintenance; adds codegen step + generated files	Pre-stable API; importable as a normal module today; lands in stdlib on a future Go
Behavioral parity risk	Medium — verify json.Number/escaping/unknown-fields parity	Low (you control generated code)	Medium — v2 defaults are stricter (case-sensitive, dup-key rejection); need opts for v1 behavior
Effort	Low (swap import at the decode site)	Medium-high (codegen infra + jlexer hand-code)	Medium (implement UnmarshalJSONFrom)
Expected win	Constant-factor (~1.5–2× scan), double-scan remains	Largest raw speed	Large; eliminates double-scan + per-level decoder allocation

Ranking for this codebase

1. json/v2 — best strategic fit. Its UnmarshalJSONFrom(*jsontext.Decoder) is exactly the primitive our problem needs: single-pass, shared-decoder behavior as a clean library API — so it satisfies "no hand-rolling" and removes the double-scan. It's also the future stdlib direction. Sketch:

   func (v *Value) UnmarshalJSONFrom(dec *jsontext.Decoder) error {
       // dec is shared across the whole subtree: one pass, no per-level decoder alloc.
       // dispatch the "value" field via dec.PeekKind(); recurse children with
       // json.UnmarshalDecode(dec, &child).
   }

   Downside: pre-stable API churn, and a v1↔v2 behavioral-parity review.

2. segmentio — best low-risk quick win. Already an indirect dep, drop-in API, swap is a few lines. But it doesn't fix the double-scan structurally — it just makes each scan faster. Good if a profile shows a modest gap and you want minimal change.

3. easyjson — fastest raw, worst ergonomics. Top speed and true single-pass, but the polymorphic value field forces a hand-written jlexer decoder (more hand-rolling than what was just rejected) plus a codegen step and generated files in the tree. Hard to justify here unless decode is dominant and the other two fall short.