execute-dynamic-code reset does not provide true memory reclamation for loaded dynamic assemblies

[hatayama]

Summary

execute-dynamic-code does not have an unbounded compilation cache, but unique snippets can still accumulate runtime cost across a long Editor session because compiled dynamic assemblies are loaded via Assembly.Load(byte[]) into the default AppDomain.

Today, cache eviction and ResetServerScopedServices() clear references and shut down server-scoped runtime objects, but they do not provide a way to fully unload already loaded dynamic assemblies. This means the current reset path only partially recovers managed memory and does not guarantee a stable memory ceiling for long-running sessions with many unique snippets.

Why this matters

Users expect execute-dynamic-code cache reset to behave like a real memory reset. Right now it does not.

The current implementation can mislead us in two ways:

• the compilation cache is bounded, so dictionary growth is not the problem
• OS-level RSS / Working Set can stay high for reasons unrelated to dynamic code, so we need Memory Profiler based verification

We should make the reset story explicit and ideally provide a reset path that reclaims dynamic-code memory without requiring a domain reload.

Current implementation facts

• Compilation cache is bounded to 32 entries in CompilationCacheManager
• Dynamic assemblies are loaded with Assembly.Load(assemblyBytes) in CompiledAssemblyLoader
• Cache eviction removes cached references but does not unload previously loaded assemblies
• ResetServerScopedServices() tears down server-scoped runtime state and clears caches, but does not unload assemblies from the default AppDomain

Relevant files:

• Packages/src/Editor/Shared/Composition/CompilationCacheManager.cs
• Packages/src/Editor/Compilation/CompiledAssemblyLoader.cs
• Packages/src/Editor/Compilation/DynamicCodeCompiler.cs
• Packages/src/Editor/Composition/DynamicCodeServices.cs

Investigation results

1. Cache is bounded, but that is not enough

The cache is capped at 32 entries, so the cache container itself is not unbounded.

However, assemblies loaded through Assembly.Load(byte[]) are not reclaimed just because the cache entry is evicted.

2. ResetServerScopedServices() only partially recovers managed memory

Using Memory Profiler snapshots around a reset-only flow after generating many unique snippets showed:

• ManagedMemorySummary: about -4.0 MB
• Virtual Machine: about -4.0 MB
• Objects: about -121.9 KB
• AllMemorySummary: still roughly flat to slightly higher because Untracked* moved independently

Interpretation:

• reset does recover some server-scoped managed/runtime state
• reset does not fully reclaim dynamic-code related memory

3. Domain reload recovers the managed growth more convincingly

Snapshot comparison around after_unique_dynamic_code -> after_domain_reload showed:

• ManagedMemorySummary: about -2.8 MB
• Objects: about -3.2 MB
• Virtual Machine: about -2.8 MB

At the same time, remaining growth was mostly in:

• Executables & Mapped
• Untracked*

with notable contributors such as Metal shader cache files and OS malloc / graphics buckets.

Interpretation:

• the managed growth introduced by dynamic code appears to be largely recovered by domain reload
• the remaining process memory noise after reload is mostly native / mapped / OS-side, not evidence that managed dynamic-code memory is still stuck

Problem statement

We do not currently have a reset mechanism that provides strong memory reclamation semantics for dynamic-code execution without relying on domain reload.

Goal

Make dynamic-code reset behavior predictable and actionable.

Minimum acceptable outcome:

• document and measure what reset can and cannot reclaim
• improve reset so it reclaims as much dynamic-code memory as possible
• define and implement a long-term architecture that allows true unloading semantics without domain reload

Proposed plan

Phase 1: short-term mitigation and measurement

1. Add a targeted benchmark / probe for long sessions with many unique snippets.
2. Compare three conditions with Memory Profiler snapshots:

• no reset
• reset only
• domain reload

3. Add a best-effort post-reset cleanup path and measure it:

• GC.Collect()
• GC.WaitForPendingFinalizers()
• GC.Collect()

4. Record whether the cleanup materially improves ManagedMemorySummary, Virtual Machine, and Objects.

Phase 2: design a true unloadable execution model

Preferred direction:

• move dynamic compilation / load / execution into a dedicated worker process
• let reset recycle the worker process
• keep Unity Editor state in the host process, and define a minimal boundary for operations that must touch Unity APIs

Alternative to investigate:

• unloadable load-context style design, if and only if Unity runtime constraints make it viable

Phase 3: product behavior and telemetry

1. Add telemetry for unique snippet count, cache hits, and reset effectiveness.
2. Define a policy for when to recommend or trigger a stronger recycle path.
3. Make reset semantics visible in logs or diagnostics so users understand what happened.

Deliverables

• a reproducible benchmark for unique dynamic snippets
• Memory Profiler based comparison data for reset-only vs domain reload
• a measured verdict on whether explicit GC improves reset behavior enough to matter
• a design doc or implementation plan for worker-process based execution reset
• at least one concrete implementation step toward true unload semantics

Acceptance criteria

• there is a documented benchmark that reproduces long-session dynamic-code memory growth
• reset-only behavior is measured with Memory Profiler, not just RSS / Working Set
• the project has a clear decision on short-term mitigation (GC.Collect path or explicit rejection based on data)
• the project has an approved implementation direction for true unloadability
• a maintainer can read this issue and start implementation work without needing separate context gathering

Notes

This issue is intentionally not phrased as “the cache is unbounded”, because that is not the real problem. The cache is bounded. The real problem is that cache reset and assembly lifetime are different concerns under the current Assembly.Load(byte[]) design.