Benchmark tests are failing

[elliette]

Our dart2wasm benchmark tests are failing with the following error (example run):

Can run web benchmarks with WASM (failed)
  Building Flutter web app (compiler: dart2wasm)...
  Build took 146s to complete.
  Launching Chrome.
  Launching Google Chrome 145.0.7632.76 
  
  Waiting for the benchmark to report benchmark profile.
  [CHROME]: 
  [CHROME]: DevTools listening on ws://127.0.0.1:10000/devtools/browser/3409c080-2f4f-4ae9-947d-b3de97f85223
  Connecting to DevTools: ws://localhost:10000/devtools/page/6A2B907AC5919DAE0A4AC907955D80F8
  Connected to Chrome tab:  (http://localhost:9999/?compiler=wasm)
  Launching benchmark "devtools_navigateThroughOfflineScreens"
  [APP] 2026-02-21 00:37:04.316: TEST STATUS: Warming up.
  [APP] 2026-02-21 00:37:04.428: TEST STATUS: Warm-up finished.
  [APP] 2026-02-21 00:37:04.429: TEST STATUS: Navigate through offline DevTools tabs
  [APP] 2026-02-21 00:37:04.431: TEST STATUS: switching to home screen (icon null, iconAsset: icons/app_bar/devtools.png)
  [APP] 2026-02-21 00:37:07.450: TEST STATUS: switching to performance screen (icon null, iconAsset: icons/app_bar/performance.png)
  [APP] 2026-02-21 00:37:10.485: TEST STATUS: switching to cpu-profiler screen (icon null, iconAsset: icons/app_bar/cpu_profiler.png)
  [APP] 2026-02-21 00:37:13.518: TEST STATUS: switching to memory screen (icon null, iconAsset: icons/app_bar/memory.png)
  [APP] 2026-02-21 00:37:16.537: TEST STATUS: switching to network screen (icon null, iconAsset: icons/app_bar/network.png)
  [APP] 2026-02-21 00:37:19.556: TEST STATUS: switching to app-size screen (icon null, iconAsset: icons/app_bar/app_size.png)
  [APP] 2026-02-21 00:37:22.589: TEST STATUS: switching to deep-links screen (icon null, iconAsset: icons/app_bar/deep_links.png)
  [APP] 2026-02-21 00:37:25.625: TEST STATUS: End navigate through offline DevTools tabs
  Extracted 1255 measured frames.
  Skipped 123 non-measured frames.
  [APP] Client preparing to reload the window to: "http://localhost:9999?compiler=wasm"
  Launching benchmark "devtools_offlineCpuProfilerScreen"
  [APP] 2026-02-21 00:37:31.227: TEST STATUS: Warming up.
  [APP] 2026-02-21 00:37:31.344: TEST STATUS: Warm-up finished.
  [APP] 2026-02-21 00:37:31.344: TEST STATUS: Loading offline CPU profiler data and interacting
  [APP] 2026-02-21 00:38:04.528: TEST STATUS: On Bottom Up tab by default. Scrolling through table.
  [APP] 2026-02-21 00:38:06.061: TEST STATUS: Switching to Call Tree tab.
  [APP] 2026-02-21 00:38:12.098: TEST STATUS: Scrolling through Call Tree table.
  [APP] 2026-02-21 00:38:13.648: TEST STATUS: Switching to Method Table tab.
  [APP] 2026-02-21 00:38:19.664: TEST STATUS: Scrolling through Method Table.
  [APP] 2026-02-21 00:38:21.198: TEST STATUS: Switching to CPU Flame Chart tab.
  Caught browser-side error: Exception: `startMeasureFrame` called twice in a row.
      at $.DomWindow|requestAnimationFrame (http://localhost:9999/main.dart.wasm:wasm-function[5969]:0x21c941)
      at $.EnginePlatformDispatcher.scheduleFrame (http://localhost:9999/main.dart.wasm:wasm-function[5967]:0x21c6a1)
      at $._RecordingWidgetsBinding.handleDrawFrame (http://localhost:9999/main.dart.wasm:wasm-function[29143]:0x50cdcc)
      at $._MixinApplication2&BindingBase&GestureBinding&SchedulerBinding.handleBeginFrame (http://localhost:9999/main.dart.wasm:wasm-function[11963]:0x2d431f)
      at $._RecordingWidgetsBinding.handleBeginFrame (http://localhost:9999/main.dart.wasm:wasm-function[29144]:0x50ceea)
      at $._MixinApplication2&BindingBase&GestureBinding&SchedulerBinding._handleBeginFrame tear-off trampoline (http://localhost:9999/main.dart.wasm:wasm-function[5974]:0x21ca75)
      at $.invoke1 (http://localhost:9999/main.dart.wasm:wasm-function[2066]:0x1aee6c)
      at $.closure wrapper at org-dartlang-sdk:///lib/_engine/engine/frame_service.dart:91:37 trampoline (http://localhost:9999/main.dart.wasm:wasm-function[5968]:0x21c7db)
  dart:async                                     _Completer.completeError
  package:web_benchmarks/src/runner.dart 254:25  BenchmarkServer.run.<fn>
  ===== asynchronous gap ===========================
  dart:async                                     _CustomZone.registerUnaryCallback
  package:web_benchmarks/src/runner.dart 248:27  BenchmarkServer.run.<fn>
  package:shelf/src/cascade.dart 77:53           Cascade.handler.<fn>.<fn>
  ===== asynchronous gap ===========================
  dart:async                                     Future.then
  package:shelf/src/cascade.dart 76:58           Cascade.handler.<fn>
  package:shelf/src/cascade.dart 76:47           Cascade.handler.<fn>.<fn>
  dart:async                                     new Future.sync
  package:shelf/src/cascade.dart 76:21           Cascade.handler.<fn>
  package:shelf/shelf_io.dart 140:29             handleRequest
  package:shelf/shelf_io.dart 93:9               serveRequests.<fn>.<fn>
  ===== asynchronous gap ===========================
  dart:_http                                     _HttpServer.listen
  package:shelf/shelf_io.dart 92:14              serveRequests.<fn>
  package:shelf/src/util.dart 21:20              catchTopLevelErrors
  package:shelf/shelf_io.dart 91:3               serveRequests
  package:web_benchmarks/src/runner.dart 316:16  BenchmarkServer.run
  ===== asynchronous gap ===========================
  dart:async                                     _CustomZone.registerUnaryCallback
  package:web_benchmarks/src/runner.dart 136:42  BenchmarkServer.run
  package:web_benchmarks/server.dart 75:5        serveWebBenchmark
  benchmark/devtools_benchmarks_test.dart 57:28  _runBenchmarks
  ===== asynchronous gap ===========================
  dart:async                                     _CustomZone.registerUnaryCallback
  benchmark/devtools_benchmarks_test.dart 54:25  _runBenchmarks
  benchmark/devtools_benchmarks_test.dart 43:15  main.<fn>
Shell: Starting web benchmark tests ...
Shell: Web benchmark tests finished.
Shell: Verifying devtools_navigateThroughOfflineScreens.js scores against expected thresholds.
Shell: Verifying devtools_offlineCpuProfilerScreen.js scores against expected thresholds.
Shell: Verifying devtools_offlinePerformanceScreen.js scores against expected thresholds.

[elliette]

Re-opening because we are experiencing more benchmark test failures:

• https://github.com/flutter/devtools/actions/runs/22336272555/job/65029549692?pr=9664
• https://github.com/flutter/devtools/actions/runs/22362750485/job/64720745754?pr=9672

[elliette]

Still seeing initial failure as well: https://github.com/flutter/devtools/actions/runs/22461618910/job/65057339304

I'm going to disable the tests for now as I investigate these flakes.

[elliette]

Investigation notes:

failure for job/65057339304

• This failure is due to a pumpAndSettle being missed in Fix failing benchmark tests #9675

failure for job/65029549692

The following error is due to this tabs.single getter in package:web_benchmarks - it expects there to be a single open http://localhost tab but for whatever reason there is none. Filed flutter/flutter#183335 on the web_benchmarks package.

  Connecting to DevTools: ws://localhost:10001/devtools/page/84039657253FC755ED7D526B2FC40AA8
  Bad state: No element
  dart:core                                       Iterable.single
  package:web_benchmarks/src/browser.dart 292:30  _connectToChromeDebugPort
  ===== asynchronous gap ===========================
  dart:async                                      _CustomZone.registerBinaryCallback
  package:web_benchmarks/src/browser.dart 113:38  Chrome.launch
  package:web_benchmarks/src/runner.dart 332:34   BenchmarkServer.run
  ===== asynchronous gap ===========================
  dart:async                                      _CustomZone.registerUnaryCallback
  package:web_benchmarks/src/runner.dart 136:42   BenchmarkServer.run
  package:web_benchmarks/server.dart 75:5         serveWebBenchmark
  benchmark/devtools_benchmarks_test.dart 57:28   _runBenchmarks

failure for job/64720745754

Legitimate failure of CPU profiler not passing benchmarks, this might require adding some retry logic but due to #9397 we will get false passes if we add back in retries on the test case itself.

    Actual: '[devtools_offlineCpuProfilerScreen.wasm] flutter_frame.total_time.p90 was 32860.0 μs, which exceeded the expected threshold, 16666.6 μs.\n'
              ''
  [WASM Benchmarks] The following benchmark scores exceeded their expected thresholds:
  
  [devtools_offlineCpuProfilerScreen.wasm] flutter_frame.total_time.p90 was 32860.0 μs, which exceeded the expected threshold, 16666.6 μs.
  
  
  package:matcher                                expect
  benchmark/devtools_benchmarks_test.dart 120:3  _runBenchmarks