Asset Remote Streaming

Overview

UntoldEngine supports streaming scene geometry directly from remote HTTP/HTTPS servers (e.g., a CDN). When a scene manifest URL points to a remote server, the engine downloads the manifest, resolves all tile/HLOD/LOD asset URLs relative to that server, and downloads each asset on demand as the camera approaches. Downloaded assets are stored in a persistent disk cache so subsequent sessions never re-download unchanged files.

This layer sits below the tile streaming system but above the local asset loading pipeline. The rest of the engine — GeometryStreamingSystem, TileComponent, UntoldReader — is unaware of whether an asset came from disk or the network; RemoteAssetDownloader resolves a remote URL to a local cached path before the file is opened.

System Components

`RemoteAssetDownloader` (actor)

Sources/UntoldEngine/Systems/RemoteAssetDownloader.swift

The single download actor for all remote assets. Responsibilities:

Downloads assets via URLSession and commits them to AssetDiskCache.
Single-flight deduplication — if two tiles request the same URL concurrently, only one network request is issued. The second caller suspends until the first completes, then returns the cached path.
Exponential backoff retry — up to 3 attempts, with delays of 1 s, 2 s, and 4 s between attempts.
Conditional GET — if a cached ETag sidecar exists for a URL, the next request includes If-None-Match: <etag>. A 304 Not Modified response returns the cached path instantly without re-downloading.
Texture pre-fetch — after downloading a .untold file, immediately fetches all textures referenced in its texture table in the background so they are cache-resident before the tile is parsed.

Public API:

func localURL(for remoteURL: URL) async throws -> URL

Returns a local file:// URL pointing to the cached asset. Throws on permanent failure after all retries are exhausted.

URLSession configuration:

Parameter	Value
`timeoutIntervalForRequest`	30 s
`timeoutIntervalForResource`	300 s
Max retry attempts	3 (delays: 1 s, 2 s, 4 s)
Retry delay formula	`2^attempt` seconds

`AssetDiskCache` (actor)

Sources/UntoldEngine/Systems/AssetDiskCache.swift

A persistent LRU cache that maps remote URLs to local files on disk.

Content-addressed storage — the cache key is SHA256(url.absoluteString). Files are stored at <cacheDir>/<hash>.<ext>.
ETag sidecars — ETags from server responses are stored in <hash>.meta (JSON). Retrieved by RemoteAssetDownloader for conditional GET on subsequent requests.
Atomic writes — each file is written to a temp path first, then renamed. A crash during download leaves an orphaned temp file, not a corrupt cache entry.
LRU eviction — when total cache usage exceeds the budget (default 500 MB), the cache evicts entries by lastAccess timestamp (oldest first) until usage falls to 75% of budget.
Texture sub-paths — textures are stored at relative paths under the cache root via storeAtRelativePath(_:data:), so NativeFormatLoader can resolve texture URIs by the same relative path they have inside the .untold file.

Cache parameters:

Parameter	Default
Cache directory	`Library/Caches/UntoldAssetCache/`
Budget	500 MB
Eviction target	75% of budget
Cache key	`SHA256(url.absoluteString)`
ETag storage	`<hash>.meta`
Write strategy	tmp file → atomic rename

URL Resolution

The engine resolves asset URLs lazily at load time. The helper resolveAssetURL(_:label:) is called by the tile loading path before opening any file:

func resolveAssetURL(_ url: URL, label: String) async -> URL? {
    guard url.scheme == "https" || url.scheme == "http" else {
        return url  // Local path — pass through unchanged
    }
    do {
        return try await RemoteAssetDownloader.shared.localURL(for: url)
    } catch {
        Logger.logError(message: "[TileStreaming] Remote download failed for \(label): \(error)")
        return nil
    }
}

Local file:// paths bypass the downloader entirely. Only HTTP/HTTPS URLs go through the cache.

Tile URL Construction

When loadTiledScene(url:) is called with a remote manifest URL, tile asset URLs are resolved relative to the manifest directory:

Manifest:    https://cdn.example.com/dungeon3/dungeon3.json
Tile path:   tiles/tile_0_0.untold
→ Tile URL:  https://cdn.example.com/dungeon3/tiles/tile_0_0.untold

HLOD path:   hlods/tile_0_0_hlod.untold
→ HLOD URL:  https://cdn.example.com/dungeon3/hlods/tile_0_0_hlod.untold

The same construction applies to HLOD and per-tile LOD URLs. All of these are stored in their respective TileComponent fields as absolute https:// URLs. resolveAssetURL translates them to cached local paths at load time.

End-to-End Flow

Phase 1 — Manifest download

if let manifestURL = URL(string: "https://cdn.example.com/scene/scene.json") {
    loadTiledScene(url: manifestURL)
}

  │
  ├─ URL has https scheme → download manifest
  │   └─ RemoteAssetDownloader.localURL(for: manifestURL)
  │       ├─ AssetDiskCache.localURL(for:) → cache miss
  │       ├─ URLSession GET with optional If-None-Match header
  │       ├─ 200 OK → AssetDiskCache.store(data:for:etag:) atomically
  │       └─ Return local file:// path
  │
  └─ Decode JSON at local path → TileManifest
     └─ registerTiledScene(manifest:baseURL:)
         ├─ Construct tile URLs relative to manifest URL
         ├─ Register lightweight TileComponent stubs (no geometry)
         └─ Each stub stores tileURL as https:// in TileComponent

Phase 2 — Tile download (on demand, per streaming tick)

GeometryStreamingSystem: camera enters prefetchRadius for tile_0_0
  │
  └─ loadTile(entityId:)
      ├─ setEntityMeshAsync(entityId: meshEntity, ...)
      │
      └─ Inside async Task:
          ├─ resolveAssetURL(tileURL)          ← tileURL is https://
          │   └─ RemoteAssetDownloader.localURL(for: tileURL)
          │       ├─ Cache hit? → return local path immediately
          │       └─ Cache miss? → download, store, return local path
          │
          ├─ Parse asset at local path
          │   ├─ .untold → UntoldReader (no ModelIO dependency)
          │   └─ .usdz/.usdc → ModelIO via NativeFormatLoader
          │
          └─ Upload geometry to Metal GPU buffers

Phase 3 — Texture pre-fetch (for `.untold` assets)

After a .untold file is downloaded, RemoteAssetDownloader immediately pre-fetches all referenced textures in the background:

RemoteAssetDownloader downloads tile_0_0.untold
  │
  └─ downloadTextures(in: untoldData, remoteBaseURL: tileDirectoryURL)
      ├─ Parse texture table URIs from .untold header
      └─ For each texture URI (e.g. "Textures/wall_albedo.png"):
          ├─ Check AssetDiskCache for relative path → skip if present
          ├─ Construct: remoteBaseURL + "/" + uri
          ├─ Download → AssetDiskCache.storeAtRelativePath(uri, data:)
          └─ (Failures skipped; geometry still loads, just untextured)

Textures are stored at their relative URI under the cache root. When NativeFormatLoader later resolves texture paths, it finds them at the same relative path without knowing they came from a CDN.

Caching Mechanics

Cache Hit (steady state)

On the second session or after an initial warm-up pass, all tiles are cached locally:

resolveAssetURL(tileURL)
  └─ RemoteAssetDownloader.localURL(for: tileURL)
      └─ AssetDiskCache.localURL(for: tileURL) → hit
          └─ Return file:// path instantly (zero network I/O)

Manifest Revalidation (ETag)

On subsequent sessions the manifest is revalidated cheaply:

GET /scene/scene.json
Headers: If-None-Match: "abc123"

← 304 Not Modified
   → Return cached manifest path, no re-download
   → All tile URLs resolved from unchanged local manifest

If the server returns 200 with a new ETag, the manifest and its ETag sidecar are updated atomically.

LRU Eviction

When accumulated downloads exceed the 500 MB budget:

AssetDiskCache.evictToLimit()
  ├─ Sort entries by lastAccess ascending (oldest first)
  └─ Delete files (and their .meta sidecars) until usage ≤ 375 MB (75%)

Evicted files are re-downloaded transparently on next access. The cache directory persists across app launches; only explicit clearCache() or OS-driven cache purges remove it entirely.

Retry and Error Handling

Network retry (RemoteAssetDownloader)

Attempt	Delay before next attempt
0	— (immediate first try)
1	1 s
2	2 s
3 (final)	4 s

After 3 failed attempts, localURL(for:) throws. The tile load marks state .failed and enters the tile-level exponential backoff (5 s → 10 s → 20 s → 60 s max) before retrying.

HTTP 304 Not Modified

Treated as a cache hit. No file write occurs; localURL(for:) returns the existing cached path.

HTTP error codes (4xx, 5xx)

Counted as a failure, subject to the retry policy above. A 404 is retried like any other error — the manifest may be temporarily unavailable or behind an eventual-consistent CDN.

Texture pre-fetch failures

Individual texture download failures are silently skipped. Geometry still loads and renders, just without that texture (Metal will bind a default 1×1 white fallback). This prevents a single bad texture URL from blocking tile geometry.

Single-flight deduplication

If 10 tiles all reference the same shared texture and all request it simultaneously, only one URLSession task fires. The other 9 suspend and resume with the cached result once the first download completes.

Integration with Tile Streaming

Remote streaming is transparent to GeometryStreamingSystem and TileComponent. From their perspective:

TileComponent.tileURL holds the canonical URL for the tile (may be https:// or file://).
loadTile() calls setEntityMeshAsync, which calls resolveAssetURL to get a local path before parsing.
The streaming state machine (.unloaded → .parsing → .parsed → .unloading) is identical regardless of whether the asset was local or remote.
Tile retry backoff, grace-period unloads, memory budget gates, and prefetch radius behavior all apply equally to remote tiles.

The only difference in behavior is a latency bump on first load when a tile is not yet cached. The prefetch radius absorbs most of this: the tile begins downloading when the camera is effectivePrefetchRadius away, giving the download time to complete before the camera reaches streamingRadius.

For a typical scene with streamingRadius = 80 m and unloadRadius = 120 m, effectivePrefetchRadius auto-computes to 100 m. At walking speed (~1.5 m/s) this is ~13 seconds of headroom — well above the download + parse time for a 15–20 MB tile over a 10 Mbps connection (~12–16 s worst case).

Demo Game Configuration

Sources/DemoGame/DemoState.swift registers two remote scenes:

let remoteScenes: [RemoteSceneOption] = [
    .init(
        id: "dungeon",
        title: "Dungeon",
        manifestURL: URL(string: "https://cdn.example.net/dungeon3/dungeon3.json")!
    ),
    .init(
        id: "city",
        title: "City",
        manifestURL: URL(string: "https://cdn.example.net/city/city.json")!
    ),
]

GameScene.loadTileScene(url:) passes the selected manifest URL directly to loadTiledScene(url:):

func loadTileScene(url: URL, completion: @escaping @Sendable (Bool) -> Void) {
    clearSceneBatches()
    loadedEntity = nil
    GeometryStreamingSystem.shared.enabled = true
    loadTiledScene(url: url) { success in
        completion(success)
    }
}

The HUD (DemoHUD.swift) presents scene options; on selection it calls onLoadTiledScene which routes to loadTileScene(url:).

Threading and Safety

Concern	Mechanism
`RemoteAssetDownloader` actor isolation	Swift actor — safe to call from any `Task` or thread
`AssetDiskCache` actor isolation	Swift actor — all reads and writes are serialised
Single-flight gate	`inFlightDownloads: [URL: Task<URL, Error>]` dictionary, actor-protected
Atomic cache writes	Temp file + `FileManager.moveItem` (atomic on same volume)
ECS mutations	Main thread only, via `withWorldMutationGate`
Tile completion guard	`scene.exists(entityId)` checked before every ECS write in upload completion closures

Key Design Parameters Summary

Parameter	Value	Location
Max download retries	3	`RemoteAssetDownloader`
Retry delay formula	`2^attempt` seconds	`RemoteAssetDownloader`
Request timeout	30 s	`URLSession` configuration
Resource timeout	300 s	`URLSession` configuration
Disk cache budget	500 MB	`AssetDiskCache`
Cache eviction target	75% of budget	`AssetDiskCache`
Cache key	`SHA256(url.absoluteString)`	`AssetDiskCache`
ETag revalidation	Yes (conditional GET)	`RemoteAssetDownloader`
Texture pre-fetch	Yes (post-download, async)	`RemoteAssetDownloader`
Single-flight dedup	Yes (actor-isolated dictionary)	`RemoteAssetDownloader`

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Asset Remote Streaming

Overview

System Components

`RemoteAssetDownloader` (actor)

`AssetDiskCache` (actor)

URL Resolution

Tile URL Construction

End-to-End Flow

Phase 1 — Manifest download

Phase 2 — Tile download (on demand, per streaming tick)

Phase 3 — Texture pre-fetch (for `.untold` assets)

Caching Mechanics

Cache Hit (steady state)

Manifest Revalidation (ETag)

LRU Eviction

Retry and Error Handling

Network retry (RemoteAssetDownloader)

HTTP 304 Not Modified

HTTP error codes (4xx, 5xx)

Texture pre-fetch failures

Single-flight deduplication

Integration with Tile Streaming

Demo Game Configuration

Threading and Safety

Key Design Parameters Summary

See Also

Uh oh!

FilesExpand file tree

asset_remote_streaming.md

Latest commit

History

asset_remote_streaming.md

File metadata and controls

Asset Remote Streaming

Overview

System Components

RemoteAssetDownloader (actor)

AssetDiskCache (actor)

URL Resolution

Tile URL Construction

End-to-End Flow

Phase 1 — Manifest download

Phase 2 — Tile download (on demand, per streaming tick)

Phase 3 — Texture pre-fetch (for .untold assets)

Caching Mechanics

Cache Hit (steady state)

Manifest Revalidation (ETag)

LRU Eviction

Retry and Error Handling

Network retry (RemoteAssetDownloader)

HTTP 304 Not Modified

HTTP error codes (4xx, 5xx)

Texture pre-fetch failures

Single-flight deduplication

Integration with Tile Streaming

Demo Game Configuration

Threading and Safety

Key Design Parameters Summary

See Also

`RemoteAssetDownloader` (actor)

`AssetDiskCache` (actor)

Phase 3 — Texture pre-fetch (for `.untold` assets)