Skip to content

Add timing measurement for integration tests#986

Open
xingzihai wants to merge 1 commit into
ForNeVeR:mainfrom
xingzihai:feature/integration-test-timing-985
Open

Add timing measurement for integration tests#986
xingzihai wants to merge 1 commit into
ForNeVeR:mainfrom
xingzihai:feature/integration-test-timing-985

Conversation

@xingzihai
Copy link
Copy Markdown

Summary

This PR addresses issue #985 by adding timing measurement for integration tests.

Changes

  • Added TimingResult.cs and TimingRecorder.cs in Cesium.TestFramework to record test timing information
  • Modified IntegrationTestRunner.cs to measure time for each test stage:
    • Native compiler compile time
    • Native executable run time
    • Cesium compiler compile time
    • Cesium executable run time
  • Timing results are written to a JSON file (integration-test-timing-{OS}-{timestamp}.json) at the end of the test run

Output Format

The JSON file contains:

  • Operating system name (Windows/Linux/macOS)
  • Timestamp (ISO 8601 format)
  • Individual test timing results with:
    • Test name
    • Source file
    • Target architecture (Bit32/Bit64/Wide/Dynamic)
    • Target framework (NetFramework/Net)
    • Native compile time (ms)
    • Native run time (ms)
    • Cesium compile time (ms)
    • Cesium run time (ms)
    • Total test time (ms)
  • Summary statistics including totals for each stage

Purpose

This allows CI teams to download the timing results and analyze which stages of the integration tests are taking the most time, helping to identify optimization opportunities.

Example Output

{
  "os": "Windows",
  "timestamp": "2025-04-01T12:00:00Z",
  "tests": [
    {
      "testName": "Net.hello.c_Bit32",
      "sourceFile": "hello.c",
      "targetArch": "Bit32",
      "targetFramework": "Net",
      "nativeCompileTimeMs": 1234,
      "nativeRunTimeMs": 56,
      "cesiumCompileTimeMs": 789,
      "cesiumRunTimeMs": 23,
      "totalTimeMs": 2102
    }
  ],
  "summary": {
    "totalTests": 100,
    "totalNativeCompileTimeMs": 12345,
    "totalCesiumCompileTimeMs": 67890,
    "totalTimeMs": 80235
  }
}

Fixes #985

- Add TimingResult and TimingRecorder classes in Cesium.TestFramework
- Modify IntegrationTestRunner.DoTestWithTiming to measure time for each stage:
  - Native compiler compile time
  - Native executable run time
  - Cesium compiler compile time
  - Cesium executable run time
- Output timing results to JSON file (integration-test-timing-{OS}-{timestamp}.json)
- This addresses issue ForNeVeR#985: Speed up the integration tests
Copy link
Copy Markdown
Owner

@ForNeVeR ForNeVeR left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sadly, compilation breaks in this PR, would you please take a look?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Speed up the integration tests

2 participants