ETW JS stacktrace not symbolicated?

[SuibianP]

https://blogs.igalia.com/dape/2023/03/14/stack-walk-profiling-nodejs-in-windows/ mentioned that --enable-etw-stack-walking can capture JS stack information on Windows to be displayed in WPA. However, in my attempts under both Node 20.18.3 and 22.14.0 (and Electron 34.2.0, for that matter), the JS frames are still displayed as ?!?. I am posting here to see if there are any clues about this.

Run after start tracing with

npx cross-env NODE_OPTIONS="--enable-etw-stack-walking --interpreted-frames-native-stack" node --enable-etw-stack-walking --interpreted-frames-native-stack test.js

where test.js is

function f() {
    let i = 0;
    while ((1 + 2 - 3) != true) {
        i++;
        if (i % 1000 == 0)
            console.log(i);
    }
}

f();

UIforETW log

wevtutil.exe um "C:\Users\jialun.hu\Downloads\etwpackage\bin\etwproviders.man"
wevtutil.exe im "C:\Users\jialun.hu\Downloads\etwpackage\bin\etwproviders.man" /mf:"C:\Users\jialun.hu\AppData\Local\Temp\ETWProviders.dll" /rf:"C:\Users\jialun.hu\AppData\Local\Temp\ETWProviders.dll"

Starting tracing to in-memory circular buffers...
DISK_IO provider is omitted when using circular-buffer tracing. If DISK_IO is needed then manually add it in Settings-> Extra kernel flags.
xperf.exe -start "NT Kernel Logger" -on PROC_THREAD+LOADER+HARD_FAULTS+DPC+INTERRUPT+CSWITCH+PROFILE+POWER+DISPATCHER+DISK_IO_INIT+FILE_IO+FILE_IO_INIT+VIRT_ALLOC+MEMINFO -stackwalk Profile+CSwitch+ReadyThread -buffersize 1024 -minbuffers 300 -maxbuffers 300 -buffering -start UIforETWSession -on Microsoft-Windows-Win32k:0xfdffffffefffffff+Multi-MAIN+Multi-FrameRate+Multi-Input+Multi-Worker+Microsoft-JScript:0x3+Microsoft-Windows-Kernel-Memory:0xE0+Microsoft-Windows-Kernel-Process+Microsoft-Windows-Kernel-Power -buffersize 1024 -minbuffers 50 -maxbuffers 50 -buffering
Tracing is started.
xperf.exe -capturestate UIforETWSession Microsoft-Windows-Win32k:0xfdffffffefffffff+Multi-MAIN+Multi-FrameRate+Multi-Input+Multi-Worker+Microsoft-JScript:0x3+Microsoft-Windows-Kernel-Memory:0xE0+Microsoft-Windows-Kernel-Process+Microsoft-Windows-Kernel-Power

Saving trace to disk...
xperf.exe -flush "NT Kernel Logger" -f "C:\Users\jialun.hu\AppData\Local\Temp\UIForETWkernel.etl" -flush UIforETWSession -f "C:\Users\jialun.hu\AppData\Local\Temp\UIForETWuser.etl"
The trace you have just captured "C:\Users\jialun.hu\AppData\Local\Temp\UIForETWkernel.etl" may contain personally identifiable information, including but not necessarily limited to paths to files accessed, paths to registry accessed and process names. Exact information depends on the events that were logged. Please be aware of this when sharing out this trace with other people.
The trace you have just captured "C:\Users\jialun.hu\AppData\Local\Temp\UIForETWuser.etl" may contain personally identifiable information, including but not necessarily limited to paths to files accessed, paths to registry accessed and process names. Exact information depends on the events that were logged. Please be aware of this when sharing out this trace with other people.
Trace save took 5.0 s
Merging trace...
xperf.exe -merge "C:\Users\jialun.hu\AppData\Local\Temp\UIForETWkernel.etl" "C:\Users\jialun.hu\AppData\Local\Temp\UIForETWuser.etl" "C:\Users\jialun.hu\Documents\etwtraces\2025-02-14_21-10-49_jialun.hu.etl" -compress
Merged Etl: C:\Users\jialun.hu\Documents\etwtraces\2025-02-14_21-10-49_jialun.hu.etl
Trace merge took 1.7 s
Finished recording trace.

Reference: crbug.com/42201026, #46202

Any leads would be much appreciated.

[piotrmaciejbednarski]

I checked and I have a similar situation, except that after loading the symbols, Windows Performance Analyzer cannot download the symbols for node.exe from the Microsoft server. I tried to download and paste the .pdb (https://nodejs.org/download/release/v22.6.0/win-x86/) to the symbol location for node.exe in the C:\ws\out\Release path, but it didn't help and I got a PDB Mismatch error. When I don't upload my own version of .pdb after loading symbols from the context menu and "Trace" I get a Load Status Error, PDB not found:

SYMSRV:  BYINDEX: 0x5
        C:\symbols*https://msdl.microsoft.com/download/symbols
        node.pdb
        373F770E45DE432FAD271DD0471D2CAB1

SYMSRV:  UNC: C:\symbols\node.pdb\373F770E45DE432FAD271DD0471D2CAB1\node.pdb - path not found

SYMSRV:  UNC: C:\symbols\node.pdb\373F770E45DE432FAD271DD0471D2CAB1\node.pd_ - path not found

SYMSRV:  UNC: C:\symbols\node.pdb\373F770E45DE432FAD271DD0471D2CAB1\file.ptr - path not found

SYMSRV:  HTTPGET: /download/symbols/index2.txt

SYMSRV:  HttpQueryInfo: 80190194 - HTTP_STATUS_NOT_FOUND

SYMSRV:  HTTPGET: /download/symbols/node.pdb/373F770E45DE432FAD271DD0471D2CAB1/node.pdb

SYMSRV:  HttpQueryInfo: 80190194 - HTTP_STATUS_NOT_FOUND

SYMSRV:  HTTPGET: /download/symbols/node.pdb/373F770E45DE432FAD271DD0471D2CAB1/node.pd_

SYMSRV:  HttpQueryInfo: 80190194 - HTTP_STATUS_NOT_FOUND

SYMSRV:  HTTPGET: /download/symbols/node.pdb/373F770E45DE432FAD271DD0471D2CAB1/file.ptr

SYMSRV:  HttpQueryInfo: 80190194 - HTTP_STATUS_NOT_FOUND

SYMSRV:  RESULT: 0x80190194

Perhaps this is due to the lack of a .pdb to load the symbols, which is why the frames are not displaying the symbols to you?

[SuibianP]

I tried placing node.pdb under the symbol cache directory, and the node symbols did get picked up but the JS frames still weren't. It was the same case for Electron, which has a PDB server thus less likely to have symbol issues.

I also tried to add this (following https://issues.chromium.org/issues/42201026#comment17, although the two flags should have been fully separated by https://chromium-review.googlesource.com/c/v8/v8/+/3704465)

const v8 = require('node:v8');
v8.setFlagsFromString('--enable-system-instrumentation --enable-etw-stack-walking --interpreted-frames-native-stack');

which also didn't work.

[piotrmaciejbednarski]

The mystery of the "?!?" stack frames. The issue is that these frames are code that WPA cannot decode. The problem is likely that maybe trace logging was not enabled? That's all I was able to find out.

Have you tried running UIForETW with trace compression disabled and CLR tracing enabled? Be careful, the ETL file can be huge.

[SuibianP]

Enabling CLR tracing and disabling compression did not help.

I tested the same config with official Chrome and non-trivial workload, and still encountered the same issue. .\chrome.exe --js-flags="--enable-system-instrumentation --interpreted-frames-native-stack --enable-etw-stack-walking" https://browserbench.org/Speedometer3.0/

xperf.exe -start "NT Kernel Logger" -on PROC_THREAD+LOADER+HARD_FAULTS+DPC+INTERRUPT+CSWITCH+PROFILE+POWER+DISPATCHER+DISK_IO_INIT+FILE_IO+FILE_IO_INIT+VIRT_ALLOC+MEMINFO -stackwalk Profile+CSwitch+ReadyThread -buffersize 1024 -minbuffers 300 -maxbuffers 300 -buffering -start UIforETWSession -on Microsoft-Windows-Win32k:0xfdffffffefffffff+Multi-MAIN+Multi-FrameRate+Multi-Input+Multi-Worker+Microsoft-JScript:0x3+Microsoft-Windows-Kernel-Memory:0xE0+Microsoft-Windows-Kernel-Process+Microsoft-Windows-Kernel-Power+e13c0d23-ccbc-4e12-931b-d9cc2eee27e4:0x1CCBD:0x5 -buffersize 1024 -minbuffers 50 -maxbuffers 50 -buffering
Tracing is started.
xperf.exe -capturestate UIforETWSession Microsoft-Windows-Win32k:0xfdffffffefffffff+Multi-MAIN+Multi-FrameRate+Multi-Input+Multi-Worker+Microsoft-JScript:0x3+Microsoft-Windows-Kernel-Memory:0xE0+Microsoft-Windows-Kernel-Process+Microsoft-Windows-Kernel-Power+e13c0d23-ccbc-4e12-931b-d9cc2eee27e4:0x1CCBD:0x5

I also tried PerfView in case it is a WPA issue, and ended up with the same ?!? stacks.

[SuibianP]

Removing --enable-system-instrumentation and starting tracing before node seems to help by a little. Will do more testing later.