Skip to content

[codex] Re-land remote cold start workflow headers#2224

Open
hansent wants to merge 3 commits intomainfrom
codex/reland-remote-cold-start-headers
Open

[codex] Re-land remote cold start workflow headers#2224
hansent wants to merge 3 commits intomainfrom
codex/reland-remote-cold-start-headers

Conversation

@hansent
Copy link
Copy Markdown
Collaborator

@hansent hansent commented Apr 10, 2026

Summary

  • re-land remote cold start model metadata aggregation in workflow response headers
  • restore the dedicated request_metrics module and its focused tests
  • fix the stale http_api.py middleware copy by using the shared request_metrics implementation

Root Cause

PR #2209 merged into main and was then reverted in #2222 because inference/core/interfaces/http/http_api.py still contained an older local copy of GCPServerlessMiddleware after the shared request_metrics.py extraction. That stale copy referenced symbols that were no longer imported there, which triggered the CI flake8 F821 undefined name failures.

Why

Workflow responses should include remote model IDs, cold start counts, load times, and detailed load metadata when remote execution happens. Without this reland, top-level workflow headers only reflect local execution and under-report remote cold starts.

Validation

  • python3.10 -m py_compile inference/core/interfaces/http/http_api.py inference/core/interfaces/http/request_metrics.py inference_sdk/http/utils/executors.py inference_sdk/config.py inference/core/managers/model_load_collector.py
  • /Users/hansent/.venv/bin/python -m pytest tests/inference_sdk/unit_tests/test_config.py tests/inference_sdk/unit_tests/http/utils/test_remote_processing_time_collection.py tests/inference/unit_tests/core/interfaces/http/test_remote_processing_time_middleware.py tests/inference/unit_tests/core/interfaces/http/test_model_response_headers.py

Context

This PR re-lands the intended behavior from #2209 after the revert in #2222, with the missing import/middleware cleanup included.

@hansent hansent marked this pull request as ready for review April 10, 2026 16:50
@hansent hansent requested a review from rafel-roboflow as a code owner April 14, 2026 14:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant