Skip to content

Add fp32 resnet50 model for vitisai ep demo #116

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Json288 wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Add fp32 resnet50 model for vitisai ep demo #116

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
3be4180
add fp32 resnet50 model for vitisai ep deomo
Json288 Apr 4, 2026
5913632
add support for amd fp32 mobilenet v2 model
Json288 Apr 7, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions demos/image-classification/index.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,14 @@ <h1 class="title">
<input type="radio" name="backend" id="webnn_npu" />WebNN NPU
</label>
</div>
<div class="btn-group-toggle dtypes hide" data-toggle="buttons" id="dtypeBtns">
<label class="btn" id="label_dtype_fp16">
<input type="radio" name="dtype" id="dtype_fp16" />FP16
</label>
<label class="btn" id="label_dtype_fp32">
<input type="radio" name="dtype" id="dtype_fp32" />FP32
</label>
</div>
<div class="btn-group-toggle models" data-toggle="buttons" id="modelBtns">
<label class="btn" id="label_mobilenet-v2">
<input type="radio" name="model" id="mobilenet-v2" />MobileNet V2
Expand Down
229 changes: 213 additions & 16 deletions demos/image-classification/index.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,91 @@ if (useRemoteModels) {
log("[Transformer.js] env.allowRemoteModels: " + transformers.env.allowRemoteModels);
log("[Transformer.js] env.allowLocalModels: " + transformers.env.allowLocalModels);

const FP16_MODEL_PATHS = {
Copy link
Copy Markdown
Contributor

@adrastogi adrastogi Apr 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One thing I am still wondering about- I tried using just the production EPs via the following browser command:

PS C:\Users\adityar\AppData\Local\Google\Chrome SxS\Application> .\chrome.exe --no-first-run --no-default-browser-check --enable-features="WebMachineLearningNeuralNetwork,WebNNOnnxRuntime" --webnn-ort-ignore-ep-blocklist --webnn-ort-logging-level=VERBOSE --webnn-ort-library-path-for-testing="C:\Program Files\WindowsApps\Microsoft.WindowsAppRuntime.2-preview2_2.0.0.2_x64__8wekyb3d8bbwe" --webnn-ort-ep-device="VitisAIExecutionProvider,0x1022,0x17F0"

Both the GPU and NPU EP failed to register in this configuration because of a missing DLL dependency, and then the GPU process crashed shortly thereafter.

[12432:33028:0420/092011.118:ERROR:services\webnn\ort\environment.cc:591] : [WebNN] Failed to call ort_api->RegisterExecutionProviderLibrary( env.get(), ep_name.c_str(), package_info->library_path.value().c_str()): [WebNN] ORT status error code: 1 error message: Error loading "C:\Program Files\WindowsApps\MicrosoftCorporationII.WinML.AMD.GPU.EP.1.8_1.8.55.0_x64__8wekyb3d8bbwe\ExecutionProvider\onnxruntime_providers_migraphx.dll" which depends on "migraphx_c.dll" which is missing. (Error 1114: "A dynamic link library (DLL) initialization routine failed.")
[12432:33028:0420/092011.124:ERROR:services\webnn\ort\environment.cc:89] : [ORT] [INFO: onnxruntime, utils.cc:552 onnxruntime::LoadPluginOrProviderBridge] Loading EP library: 0000133C02659FC0 as a plugin
[12432:33028:0420/092011.132:ERROR:services\webnn\ort\environment.cc:591] : [WebNN] Failed to call ort_api->RegisterExecutionProviderLibrary( env.get(), ep_name.c_str(), package_info->library_path.value().c_str()): [WebNN] ORT status error code: 1 error message: Error loading "C:\Program Files\WindowsApps\MicrosoftCorporationII.WinML.AMD.NPU.EP.1.8_1.8.59.0_x64__8wekyb3d8bbwe\ExecutionProvider\onnxruntime_providers_vitisai.dll" which depends on "onnxruntime_providers_shared.dll" which is missing. (Error 126: "The specified module could not be found.")
[12432:33028:0420/092011.132:ERROR:services\webnn\ort\logging.cc:146] : [WebNN] [INFO] Registered OrtEpDevice #0: {ep_name: CPUExecutionProvider, ep_vendor: Microsoft, ep_metadata: {version: 1.24.4}, ep_options: {}}, OrtHardwareDevice: {type: CPU, vendor: AMD, vendor_id: 0x1022, device_id: 0x7, device_metadata: {Description: AMD Ryzen AI 9 365 w/ Radeon 880M              }}
[12432:33028:0420/092011.132:ERROR:services\webnn\ort\logging.cc:146] : [WebNN] [INFO] Registered OrtEpDevice #1: {ep_name: DmlExecutionProvider, ep_vendor: Microsoft, ep_metadata: {version: 1.24.4}, ep_options: {device_id: 0}}, OrtHardwareDevice: {type: GPU, vendor: Advanced Micro Devices, Inc., vendor_id: 0x1002, device_id: 0x150e, device_metadata: {Description: AMD Radeon(TM) 880M Graphics, LUID: 202817, DxgiAdapterNumber: 0, DxgiHighPerformanceIndex: 0, DxgiVideoMemory: 4096 MB, Discrete: 0}}
[12432:33028:0420/092011.132:ERROR:services\webnn\ort\environment.cc:501] : [WebNN] No EP device can be selected due to no matching device for user-specified webnn-ort-ep-device: VitisAIExecutionProvider,0x1022,0x17F0. Please check the registered EP devices in the logs by setting webnn-ort-logging-level to VERBOSE or INFO.
[12432:33028:0420/092011.132:ERROR:services\webnn\ort\environment.cc:501] : [WebNN] No EP device can be selected due to no matching device for user-specified webnn-ort-ep-device: VitisAIExecutionProvider,0x1022,0x17F0. Please check the registered EP devices in the logs by setting webnn-ort-logging-level to VERBOSE or INFO.
[12432:33028:0420/092011.132:ERROR:services\webnn\ort\environment.cc:501] : [WebNN] No EP device can be selected due to no matching device for user-specified webnn-ort-ep-device: VitisAIExecutionProvider,0x1022,0x17F0. Please check the registered EP devices in the logs by setting webnn-ort-logging-level to VERBOSE or INFO.
GpuProcessHost: The GPU process crashed! Exit code: STATUS_BREAKPOINT.

The reason I am interested in this configuration is that it's closer to the production environment (i.e., using the released EPs). I'll dig around more on my end but wanted to check if this looks familiar to you or if you spot something wrong with this configuration.

Copy link
Copy Markdown
Author

@Json288 Json288 Apr 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I noticed that you removed this arg: --webnn-ort-ep-library-path-for-testing=VitisAIExecutionProvider?"C:\Program Files\AMD-EP\onnxruntime_vitisai_ep.dll". Why is that?

Copy link
Copy Markdown
Contributor

@adrastogi adrastogi Apr 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah good question- it goes back to this remark:

The reason I am interested in this configuration is that it's closer to the production environment

Basically, trying to understand and diagnose why the official versions of the EP don't work from the MSIX since that is how AMD customers would use the feature.

Copy link
Copy Markdown
Author

@Json288 Json288 Apr 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For error caused by loading ep dll from msix installation path, we have a pr to fix it that's waiting to be merged.
As for this one, I think it should try to load onnxruntime_vitisai_ep.dll instead of onnxruntime_providers_vitisai.dll. Is this expected?

Copy link
Copy Markdown
Contributor

@adrastogi adrastogi Apr 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it should try to load onnxruntime_vitisai_ep.dll instead of onnxruntime_providers_vitisai.dll. Is this expected?

Yeah, I noticed this as well- something is causing the EP routing to get confused and pick the wrong EP version... not sure what's going on here, trying to debug ORT to get more details.

"mobilenet-v2": "webnn/mobilenet-v2",
"resnet-50": "xenova/resnet-50",
"efficientnet-lite4": "webnn/efficientnet-lite4",
};

/**
* FP32 model IDs. Local AMD repos match xenova (`config.json`, preprocessor, `onnx/`). On the Hub,
* the same repos may use a `webnn/` folder; remote loads rewrite JSON URLs and use `subfolder:
* "webnn/onnx"` for weights.
*/
const FP32_MODEL_PATHS = {
"mobilenet-v2": "amd/MobileNetV2",
"resnet-50": "amd/resnet50",
};

/** AMD Hub repos that store `config.json` / preprocessor under `webnn/` and ONNX under `webnn/onnx/`. */
const AMD_WEBNN_HUB_LAYOUT_MODEL_IDS = ["amd/resnet50", "amd/MobileNetV2"];

const isAmdWebnnHubLayoutModel = modelPath => AMD_WEBNN_HUB_LAYOUT_MODEL_IDS.includes(modelPath);

function isRemoteHubArtifactUrl(urlString) {
if (typeof urlString !== "string" || !/^https?:\/\//i.test(urlString) || !urlString.includes("/resolve/")) {
return false;
}
return urlString.includes("/amd/resnet50/") || urlString.includes("/amd/MobileNetV2/");
}

/**
* Remote Hub only: map `.../resolve/<rev>/config.json` → `.../resolve/<rev>/webnn/config.json`.
* Local file URLs (no `/resolve/`) are unchanged.
*/
function rewriteAmdWebnnHubJsonAssetUrl(urlString) {
if (!isRemoteHubArtifactUrl(urlString)) {
return urlString;
}
if (urlString.includes("/webnn/config.json") || urlString.includes("/webnn/preprocessor_config.json")) {
return urlString;
}
for (const id of AMD_WEBNN_HUB_LAYOUT_MODEL_IDS) {
const escaped = id.replace(/\//g, "\\/");
const re = new RegExp(
`^(https?://[^/]+/${escaped}/resolve/[^/]+/)(config\\.json|preprocessor_config\\.json)(\\?.*)?$`,
"i",
);
const match = urlString.match(re);
if (match) {
return `${match[1]}webnn/${match[2]}${match[3] ?? ""}`;
}
}
return urlString;
}

let amdWebnnHubJsonFetchInstalled = false;

function ensureAmdWebnnHubJsonFetch(env) {
if (amdWebnnHubJsonFetchInstalled) {
return;
}
const inner = env.fetch.bind(env);
env.fetch = async (input, init) => {
if (typeof input === "string") {
return inner(rewriteAmdWebnnHubJsonAssetUrl(input), init);
}
if (typeof Request !== "undefined" && input instanceof Request) {
const next = rewriteAmdWebnnHubJsonAssetUrl(input.url);
if (next !== input.url) {
return inner(new Request(next, input), init);
}
}
return inner(input, init);
};
amdWebnnHubJsonFetchInstalled = true;
}

const resolveModelPath = (id, dtype) => {
if (dtype === "fp32") {
if (id === "efficientnet-lite4") {
return FP32_MODEL_PATHS["resnet-50"];
}
return FP32_MODEL_PATHS[id] ?? FP32_MODEL_PATHS["resnet-50"];
}
return FP16_MODEL_PATHS[id] ?? FP16_MODEL_PATHS["resnet-50"];
};

let provider = "webnn";
let deviceType = "gpu";
let dataType = "fp16";
Expand All @@ -52,6 +137,7 @@ let runs = 1;
let range, rangeValue, runSpan;
let backendLabels, modelLabels;
let label_webgpu, label_webnn_gpu, label_webnn_npu, label_mobilenetV2, label_resnet50, label_efficientnetLite4;
let dtypeLabels, label_dtype_fp16, label_dtype_fp32;
let uploadImage, label_uploadImage;
let imageUrl, image;
let classify;
Expand All @@ -66,6 +152,55 @@ let dataTypeSpan;
let modelIdSpan;
let latency, latencyDiv, indicator;
let title, device, badge;
let dtypeBtnsRow;

const isWebnnNpuFromQuery = () =>
getQueryValue("provider")?.toLowerCase() === "webnn" && getQueryValue("devicetype")?.toLowerCase() === "npu";

const syncDtypeRowVisibility = () => {
if (!dtypeBtnsRow) {
return;
}
if (isWebnnNpuFromQuery()) {
dtypeBtnsRow.classList.remove("hide");
} else {
dtypeBtnsRow.classList.add("hide");
}
};

const syncEfficientnetFp32Visibility = () => {
if (!label_efficientnetLite4) {
return;
}
const hideEfficientnet = isWebnnNpuFromQuery() && getQueryValue("dtype")?.toLowerCase() === "fp32";
if (hideEfficientnet) {
label_efficientnetLite4.classList.add("hide");
} else {
label_efficientnetLite4.classList.remove("hide");
}
};

/**
* FP32 AMD ONNX exports often name the image tensor `input` while the image-classification pipeline
* passes `pixel_values`. Apply to every repo listed in `FP32_MODEL_PATHS`.
*/
const patchAmdClassifierPixelValuesInput = (classifier, modelPath) => {
if (!Object.values(FP32_MODEL_PATHS).includes(modelPath)) {
return;
}
const model = classifier?.model;
if (!model) {
return;
}
const _call = model._call.bind(model);
model._call = async model_inputs => {
let mi = model_inputs;
if (mi?.pixel_values != null && mi.input == null) {
mi = { ...mi, input: mi.pixel_values };
}
return _call(mi);
};
};

const main = async () => {
fullResult.setAttribute("class", "none");
Expand All @@ -77,22 +212,25 @@ const main = async () => {

if (getQueryValue("model")) {
modelId = getQueryValue("model");
switch (modelId) {
case "mobilenet-v2":
modelPath = "webnn/mobilenet-v2";
break;
case "resnet-50":
modelPath = "xenova/resnet-50";
break;
case "efficientnet-lite4":
modelPath = "webnn/efficientnet-lite4";
break;
default:
modelPath = "xenova/resnet-50";
break;
if (!["mobilenet-v2", "resnet-50", "efficientnet-lite4"].includes(modelId)) {
modelId = "resnet-50";
}
}

const dtypeParam = getQueryValue("dtype");
const urlDtype = dtypeParam?.toLowerCase() === "fp32" ? "fp32" : "fp16";
dataType = isWebnnNpuFromQuery() ? urlDtype : "fp16";

if (isWebnnNpuFromQuery() && dataType === "fp32" && modelId === "efficientnet-lite4") {
modelId = "resnet-50";
}

modelPath = resolveModelPath(modelId, dataType);

if (isAmdWebnnHubLayoutModel(modelPath)) {
ensureAmdWebnnHubJsonFetch(transformers.env);
}

await remapHuggingFaceDomainIfNeeded(transformers.env);

let device = "webnn-gpu";
Expand Down Expand Up @@ -120,6 +258,10 @@ const main = async () => {
options.session_options.freeDimensionOverrides = dimensionOverrides[modelId];
}

if (dataType === "fp32" && Object.values(FP32_MODEL_PATHS).includes(modelPath)) {
options.subfolder = useRemoteModels && isAmdWebnnHubLayoutModel(modelPath) ? "webnn/onnx" : "onnx";
}

modelIdSpan.innerHTML = dataType;
dataTypeSpan.innerHTML = modelPath;

Expand All @@ -129,6 +271,7 @@ const main = async () => {
WebNNPerf.configure({ model: modelId, device: deviceType, provider });

const classifier = await transformers.pipeline("image-classification", modelPath, options);
patchAmdClassifierPixelValuesInput(classifier, modelPath);

let [err, output] = await asyncErrorHandling(classifier(imageUrl, { topk: 3 }));

Expand Down Expand Up @@ -240,6 +383,17 @@ const checkWebNN = async () => {
}
};

const initDtypeSelector = () => {
dtypeLabels.forEach(label => {
label.setAttribute("class", "btn");
});
if (dataType === "fp32") {
label_dtype_fp32.setAttribute("class", "btn active");
} else {
label_dtype_fp16.setAttribute("class", "btn active");
}
};

const initModelSelector = () => {
provider = getQueryValue("provider").toLowerCase();
deviceType = getQueryValue("devicetype").toLowerCase();
Expand Down Expand Up @@ -288,6 +442,7 @@ const controls = async () => {

let backendBtns = $("#backendBtns");
let modelBtns = $("#modelBtns");
let dtypeBtns = $("#dtypeBtns");

const updateBackend = e => {
backendLabels.forEach(label => {
Expand Down Expand Up @@ -319,7 +474,7 @@ const controls = async () => {
currentUrl = window.location.href;
updatedUrl = updateQueryStringParameter(currentUrl, "devicetype", "npu");
window.history.pushState({}, "", updatedUrl);
provider = "webgpu";
provider = "webnn";
deviceType = "npu";
}

Expand Down Expand Up @@ -350,8 +505,30 @@ const controls = async () => {
updateUi();
};

const updateDtype = e => {
dtypeLabels.forEach(label => {
label.setAttribute("class", "btn");
});
e.target.parentNode.setAttribute("class", "btn active");
const id = e.target.id.trim();
let currentUrl = window.location.href;
let updatedUrl;
if (id === "dtype_fp16") {
dataType = "fp16";
updatedUrl = updateQueryStringParameter(currentUrl, "dtype", "fp16");
} else if (id === "dtype_fp32") {
dataType = "fp32";
updatedUrl = updateQueryStringParameter(currentUrl, "dtype", "fp32");
}
if (updatedUrl) {
window.history.pushState({}, "", updatedUrl);
}
updateUi();
};

backendBtns.addEventListener("change", updateBackend, false);
modelBtns.addEventListener("change", updateModel, false);
dtypeBtns.addEventListener("change", updateDtype, false);
};

const badgeUpdate = () => {
Expand Down Expand Up @@ -410,9 +587,25 @@ const updateUi = async () => {
modelId = getQueryValue("model");
}

if (getQueryValue("dtype")) {
const d = getQueryValue("dtype").toLowerCase();
dataType = d === "fp32" ? "fp32" : "fp16";
} else {
dataType = "fp16";
}

if (isWebnnNpuFromQuery() && dataType === "fp32" && modelId === "efficientnet-lite4") {
modelId = "resnet-50";
window.history.replaceState({}, "", updateQueryStringParameter(window.location.href, "model", "resnet-50"));
}

initModelSelector();
badgeUpdate();
log(`[Config] Demo config updated · ${modelId} · ${provider} · ${deviceType}`);

initDtypeSelector();
syncDtypeRowVisibility();
syncEfficientnetFp32Visibility();
log(`[Config] Demo config updated · ${modelId} · ${provider} · ${deviceType} · ${dataType}`);
await checkWebNN();
console.log(provider);
console.log(deviceType);
Expand All @@ -432,7 +625,7 @@ const changeImage = async () => {
const ui = async () => {
imageUrl = "./static/tiger.jpg";
if (!(getQueryValue("provider") && getQueryValue("model") && getQueryValue("devicetype") && getQueryValue("run"))) {
let url = "?provider=webnn&devicetype=gpu&model=resnet-50&run=5";
let url = "?provider=webnn&devicetype=gpu&model=resnet-50&run=5&dtype=fp16";
location.replace(url);
}

Expand All @@ -450,6 +643,10 @@ const ui = async () => {
label_mobilenetV2 = $("#label_mobilenet-v2");
label_resnet50 = $("#label_resnet-50");
label_efficientnetLite4 = $("#label_efficientnet-lite4");
dtypeLabels = $$(".dtypes label");
label_dtype_fp16 = $("#label_dtype_fp16");
label_dtype_fp32 = $("#label_dtype_fp32");
dtypeBtnsRow = $("#dtypeBtns");
image = $("#image");
uploadImage = $("#upload-image");
label_uploadImage = $("#label_upload-image");
Expand Down
Loading