What size of context does this driver assume the model has? How large a prefix/suffix does it send? Some model runners (in particular ollama) do not allocate much context (4096 tokens by default I believe), and if running with a modest GPU one might have to choose between large context and a larger/less quantised model.
If this tool could document how it chooses how much context to send, and how large a window the model should support, that would make planning easier.
If this tool could configure how much context it sent (characters before/after perhaps?), that would help adapt to different models with different context sizes.
I'm not sure how or if a model can signal that it has overflowed its context, but if it were possible for this tool to detect when this occurs, that would also be very helpful - at the least to report it to the user, but ideally to adapt its prompting.
What size of context does this driver assume the model has? How large a prefix/suffix does it send? Some model runners (in particular ollama) do not allocate much context (4096 tokens by default I believe), and if running with a modest GPU one might have to choose between large context and a larger/less quantised model.
If this tool could document how it chooses how much context to send, and how large a window the model should support, that would make planning easier.
If this tool could configure how much context it sent (characters before/after perhaps?), that would help adapt to different models with different context sizes.
I'm not sure how or if a model can signal that it has overflowed its context, but if it were possible for this tool to detect when this occurs, that would also be very helpful - at the least to report it to the user, but ideally to adapt its prompting.