fix(orchestrator): drop stale pre-init logs by rguliyev · Pull Request #3297 · e2b-dev/infra

rguliyev · 2026-07-16T04:17:31Z

A sandbox resumed from a memory snapshot boots with its clock still set to the snapshot's time. envd's log exporter is restored along with the rest of the VM, so it can send startup logs stamped with that old time before /init corrects the clock.

Each sandbox now records a fresh host timestamp when its lifecycle is created, before it's reachable. The /logs handler rejects any envd log timestamped more than a minute before that anchor. envd's exporter doesn't retry on a rejected request, so those stale records are simply dropped instead of being forwarded to the collector. Logs with a reasonable timestamp go through unchanged.

gemini-code-assist

Code Review

This pull request introduces a mechanism to correct stale log timestamps emitted by a sandbox guest before its clock is synchronized upon resume or creation. It adds a LifecycleStartedAt field to the Sandbox struct, initialized during sandbox creation and resumption, and uses it to clamp stale log timestamps to the host's start time while preserving the original timestamp. The review feedback suggests ensuring timezone consistency by explicitly converting and storing these timestamps in UTC, preventing issues with local timezone offsets in downstream log aggregation systems.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

gemini-code-assist · 2026-07-16T04:18:36Z

+		LifecycleID:        lifecycleID,
+		LifecycleStartedAt: time.Now(),


It is safer to initialize LifecycleStartedAt using time.Now().UTC() to ensure that the anchor timestamp is always stored in UTC, avoiding any local timezone dependencies or discrepancies when comparing or formatting it later.

Suggested change

LifecycleID: lifecycleID,

LifecycleStartedAt: time.Now(),

LifecycleID: lifecycleID,

LifecycleStartedAt: time.Now().UTC(),

Good catch, fixed - forced UTC on both the write side (time.Now().UTC() in CreateSandbox/ResumeSandbox) and the format call in correctStaleTimestamp.

codecov · 2026-07-16T05:11:18Z

❌ 1 Tests Failed:

Tests completed	Failed	Passed	Skipped
3346	1	3345	7

View the top 3 failed test(s) by shortest run time

github.com/e2b-dev/infra/tests/integration/internal/tests/envd::TestListDir/depth_0_lists_only_root_directory

Stack Traces | 0.01s run time

=== RUN   TestListDir/depth_0_lists_only_root_directory
=== PAUSE TestListDir/depth_0_lists_only_root_directory
=== CONT  TestListDir/depth_0_lists_only_root_directory
    filesystem_test.go:96: 
        	Error Trace:	.../tests/envd/filesystem_test.go:96
        	Error:      	Received unexpected error:
        	            	unavailable: 502 Bad Gateway
        	Test:       	TestListDir/depth_0_lists_only_root_directory
--- FAIL: TestListDir/depth_0_lists_only_root_directory (0.01s)

github.com/e2b-dev/infra/tests/integration/internal/tests/envd::TestListDir/depth_2_lists_first_level_of_subdirectories_(in_this_case_the_root_directory)

Stack Traces | 0.01s run time

=== RUN   TestListDir/depth_2_lists_first_level_of_subdirectories_(in_this_case_the_root_directory)
=== PAUSE TestListDir/depth_2_lists_first_level_of_subdirectories_(in_this_case_the_root_directory)
=== CONT  TestListDir/depth_2_lists_first_level_of_subdirectories_(in_this_case_the_root_directory)
    filesystem_test.go:96: 
        	Error Trace:	.../tests/envd/filesystem_test.go:96
        	Error:      	Received unexpected error:
        	            	unavailable: 502 Bad Gateway
        	Test:       	TestListDir/depth_2_lists_first_level_of_subdirectories_(in_this_case_the_root_directory)
--- FAIL: TestListDir/depth_2_lists_first_level_of_subdirectories_(in_this_case_the_root_directory) (0.01s)

github.com/e2b-dev/infra/tests/integration/internal/tests/envd::TestListDir/depth_1_lists_root_directory

Stack Traces | 0.02s run time

=== RUN   TestListDir/depth_1_lists_root_directory
=== PAUSE TestListDir/depth_1_lists_root_directory
=== CONT  TestListDir/depth_1_lists_root_directory
    filesystem_test.go:96: 
        	Error Trace:	.../tests/envd/filesystem_test.go:96
        	Error:      	Received unexpected error:
        	            	unavailable: 502 Bad Gateway
        	Test:       	TestListDir/depth_1_lists_root_directory
--- FAIL: TestListDir/depth_1_lists_root_directory (0.02s)

github.com/e2b-dev/infra/tests/integration/internal/tests/envd::TestListDir

Stack Traces | 1.33s run time

=== RUN   TestListDir
=== PAUSE TestListDir
=== CONT  TestListDir
--- FAIL: TestListDir (1.33s)
Executing command cat in sandbox ip6yqx9gthbyizovx0dyb (user: root)

github.com/e2b-dev/infra/tests/integration/internal/tests/api/templates::TestUpdateTemplateWithInvalidAPIKey

Stack Traces | 168s run time

=== RUN   TestUpdateTemplateWithInvalidAPIKey
=== PAUSE TestUpdateTemplateWithInvalidAPIKey
=== CONT  TestUpdateTemplateWithInvalidAPIKey
    template_update_test.go:107: Build failed: {<nil> An internal error occurred. Please try again or contact support with the build ID. <nil>}
--- FAIL: TestUpdateTemplateWithInvalidAPIKey (168.27s)

View the full list of 1 ❄️ flaky test(s)

github.com/e2b-dev/infra/tests/integration/internal/tests/envd::TestListDir/depth_3_lists_all_directories_and_files

Flake rate in main: 30.77% (Passed 9 times, Failed 4 times)

Stack Traces | 0.01s run time

=== RUN   TestListDir/depth_3_lists_all_directories_and_files
=== PAUSE TestListDir/depth_3_lists_all_directories_and_files
=== CONT  TestListDir/depth_3_lists_all_directories_and_files
    filesystem_test.go:96: 
        	Error Trace:	.../tests/envd/filesystem_test.go:96
        	Error:      	Received unexpected error:
        	            	unavailable: 502 Bad Gateway
        	Test:       	TestListDir/depth_3_lists_all_directories_and_files
--- FAIL: TestListDir/depth_3_lists_all_directories_and_files (0.01s)

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9cdcd7489b

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

matthewlouisbrockman

good enough for now - can we yeet some of the excessive verboseness from the comments please?

one improvement might want to consider later: comparison's happening forever. not sure if there's a way to only do these checks while we're resuming the sandbox? also i think we're dropping logs from before we fix clock while resuming but tradeoff seems fine.

A sandbox resumed from a memory snapshot boots with its clock still set to the snapshot's time, so envd's log exporter (also restored from the snapshot) can emit startup logs stamped with that old time before /init corrects the clock. Record a fresh per-lifecycle timestamp on the host before the sandbox is reachable, and reject any log timestamped more than a minute before it. envd's exporter doesn't retry on rejection, so these stale records are simply dropped instead of forwarded.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9ceb20d55b

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector · 2026-07-16T23:49:17Z

 		return
 	}

+	if hasStaleLogTimestamp(payload, sbx.LifecycleStartedAt) {


Preserve queued pre-pause logs

When the same sandbox is paused with envd log entries still buffered in HTTPExporter.logs, that queue is captured in the memory snapshot and is sent after resume with its original pre-pause timestamp. If the pause lasted more than a minute, this new check returns 400 for those valid same-sandbox entries, and the exporter drops non-2xx sends, so logs produced immediately before pause disappear; the filter needs to distinguish restored pre-init records from buffered pre-pause records.

Useful? React with 👍 / 👎.

rguliyev requested review from ValentaTomas, dobrac and jakubno as code owners July 16, 2026 04:17

cla-bot Bot added the cla-signed label Jul 16, 2026

rguliyev requested a review from matthewlouisbrockman July 16, 2026 04:17

gemini-code-assist Bot reviewed Jul 16, 2026

View reviewed changes

rguliyev requested a review from wj-e2b July 16, 2026 04:19

rguliyev force-pushed the fix/envd-pre-init-log-timestamp branch from 9f99e1b to 308c608 Compare July 16, 2026 04:23

claude Bot reviewed Jul 16, 2026

View reviewed changes

Comment thread packages/orchestrator/pkg/hyperloopserver/handlers/logs.go Outdated

tvi approved these changes Jul 16, 2026

View reviewed changes

rguliyev force-pushed the fix/envd-pre-init-log-timestamp branch 2 times, most recently from 734eb85 to 9cdcd74 Compare July 16, 2026 18:57

chatgpt-codex-connector Bot reviewed Jul 16, 2026

View reviewed changes

Comment thread packages/orchestrator/pkg/hyperloopserver/handlers/logs.go Outdated

rguliyev force-pushed the fix/envd-pre-init-log-timestamp branch from 9cdcd74 to 9a50790 Compare July 16, 2026 19:12

rguliyev changed the title ~~fix(orchestrator): preserve stale pre-init envd logs' true timing~~ fix(orchestrator): drop stale pre-init logs Jul 16, 2026

matthewlouisbrockman approved these changes Jul 16, 2026

View reviewed changes

rguliyev force-pushed the fix/envd-pre-init-log-timestamp branch from 9a50790 to b9d2947 Compare July 16, 2026 23:43

Merge branch 'main' into fix/envd-pre-init-log-timestamp

9ceb20d

chatgpt-codex-connector Bot reviewed Jul 16, 2026

View reviewed changes

rguliyev enabled auto-merge (squash) July 16, 2026 23:51

rguliyev merged commit 8ec4be5 into main Jul 17, 2026
40 checks passed

rguliyev deleted the fix/envd-pre-init-log-timestamp branch July 17, 2026 00:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix(orchestrator): drop stale pre-init logs#3297

fix(orchestrator): drop stale pre-init logs#3297
rguliyev merged 2 commits into
mainfrom
fix/envd-pre-init-log-timestamp

rguliyev commented Jul 16, 2026 •

edited

Loading

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

Uh oh!

gemini-code-assist Bot Jul 16, 2026

Uh oh!

rguliyev Jul 16, 2026

Uh oh!

Uh oh!

Uh oh!

codecov Bot commented Jul 16, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

matthewlouisbrockman left a comment

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot Jul 16, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

rguliyev commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

gemini-code-assist Bot Jul 16, 2026

Choose a reason for hiding this comment

Uh oh!

rguliyev Jul 16, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

codecov Bot commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

❌ 1 Tests Failed:

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

matthewlouisbrockman left a comment

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Jul 16, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

rguliyev commented Jul 16, 2026 •

edited

Loading

codecov Bot commented Jul 16, 2026 •

edited

Loading