Guard save_compiled_model with try-catch to handle serialization failures

[aditya-dl]

Description

Wrap only the save_compiled_model call (in both the initial compile and recompile paths) with a try-catch that logs a warning instead of propagating the exception. All other operations (SerializeToString, parse_onnx_buffer, compile_program) are confirmed to succeed and are left unwrapped.

Motivation and Context

save_compiled_model can throw std::bad_alloc during MXR cache serialization when the compiled model is large. This crashes the entire inference session even though the model compiled and runs correctly; the cache file is purely an optimization for subsequent loads. Inference should not fail because of a cache write failure.

[TedThemistokleous]

@aditya-dl appreciate the patch to this. Curious where are you seeing this blow up?

How large of a file? What limits are you seeing this fail on and is that anywhere we can easily reproduce?

[TedThemistokleous]

Don't do a catch all here as we want to be explicit in what exceptions we're expecting and those that we aren't. If there's an MIGraphX API fail we're hitting since that's what save_compiled_model uses, I don't think we should catch and mask that but instead report it.

[TedThemistokleous]

Few questions/comments on this one. Let me know what error and things you're seeing so we can be explicit in catching certain exceptions. We want to be able to signal API failures if something unexpected happens.

This is a good fix is good to stop a higher level application from crashing but I want to be clear on what cases we're seeing this and dig further if you're hitting and MIGraphX failure and there's a bug somewhere else right now.