fit-params : refactor + add option to output estimated memory per device

[ggerganov]

Overview

cont #16653
ref #19070 (review)

• Refactor the fit param logic. Move from libllama to libcommon
• Add CLI argument -fite, --fit-estimate to the llama-fit-params tool. This is useful for 3rd party applications to estimate the required memory for a model.

Additional information

Example:

llama-fit-params -m ~/models/gemma-3-4b-it/ggml-model-f16.gguf -c 32768 --fit-estimate on

0.00.196.882 I main: printing estimated memory in MiB to stdout (device, model, context, compute) ...
MTL0 7401 814 517 
host 1280 0 154 

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: NO

[JohannesGaessler]

We are making llama-ext.h a C++ only header then?

[ggerganov]

This header is not required to be C-style. It is for staging new API and can be C++ if needed.

Once an API is ready to become public, it has to be become C-style.

[JohannesGaessler]

If the intent is to parse this output programmatically I think it would be preferable to use a well-defined format like JSON.

[ggerganov]

For me, JSON is a huge overhead and in this specific case I don't think it is warranted because the data that we want to output is very simple.

[JohannesGaessler]

I think this descriptions is confusing. It is true that with this option enabled the program will estimate the required memory but that is already what it's doing anyways. Maybe it would be better to call this something like --fit-print (or --fit-print-json depending on my other comment).

[JohannesGaessler]

In principle, if the only goal is to disable the fitting this can already be done by manually setting e.g. -c 0 -ngl 999. The code should then recognize that these have been set manually and will not alter them. The only downside vs. the current approach would be that it's slightly slower than to only retrieve llama_get_device_memory_data once. But if we're already messing around with internal headers anyways we may as well make llama_params_fit_impl return the device memory data instead. Then all that would need to be done is change what is being printed.

[ggerganov]

I took the opportunity to refactor the implementation and move the param fitting logic outside of libllama. It's something that I suggested in the original PR (#16653 (review)) and I think it makes sense to have this logic in libcommon for now. IMO the param fitting API on master is too narrow and does not allow flexibility for the user code. The idea is to eventually expose a much more lightweight and flexible API that would allow the applications to implement more sophisticated logic for param fitting and memory queries.

The prototype of this API is now in llama-ext.h:

llama.cpp/src/llama-ext.h

Lines 61 to 90 in 7681f17

	//
	// device memory querying
	//
	// "memory" as in physical memory for a buffer type, in bytes
	struct llama_memory_breakdown_data {
	size_t model = 0; // memory allocated for the model
	size_t context = 0; // memory allocated for the context
	size_t compute = 0; // memory allocated for temporary compute buffers
	size_t total() const {
	return model + context + compute;
	}
	};
	struct llama_device_memory_data {
	int64_t total;
	int64_t free;
	llama_memory_breakdown_data mb;
	};
	// TODO: convert to C-style data structure
	using llama_memory_breakdown = std::map<ggml_backend_buffer_type_t, llama_memory_breakdown_data>;
	int32_t llama_model_n_expert (const struct llama_model * model);
	int32_t llama_model_n_devices(const struct llama_model * model);
	ggml_backend_dev_t llama_model_get_device(const struct llama_model * model, int i);
	llama_memory_breakdown llama_get_memory_breakdown(const struct llama_context * ctx);

[JohannesGaessler]

Please also add me as a code owner for fit.cpp.