Add support for models via LiteLLM proxy server

[haroon0x]

Is your feature request related to a problem? Please describe.
I’d like to add support for OpenAI models using the LiteLLM simple proxy. This will expand the model compatibility of adk-java to include OpenAI endpoints via a standardized proxy interface.

Describe the solution you'd like
-The plan is to use Java's HTTP client to call the proxy endpoint and integrate the response into the existing ADK pipeline.

[haroon0x]

@cornellgit opened this to track OpenAI model support using LiteLLM.
Let me know if you have any suggestions before I start.

[cornellgit]

I don't think they have any java sdk, therefore

• https://docs.litellm.ai/#litellm-proxy-server-llm-gateway is probably needed in the setup.
• https://litellm-api.up.railway.app/#/ seems like are all REST endpoint that are compatible with the original SDK by overriding the end point location to local, so you might be able to use the Java version https://platform.openai.com/docs/libraries#install-an-official-sdk
• Would be helpful to add a few example for other models too, since that's where LiteLLM is most helpful
• Some documentation to help users to run the setup e2e would be helpufl

For OAI model, it's frequently used, I'd suggest instead of going through LiteLLM route, try branch the code and adjust it to OAI. https://github.com/google/adk-java/blob/490f054abce6b857b5fec9022cd66646c108f213/core/src/main/java/com/google/adk/models/Claude.java

[vorburger]

I don't think they have any java sdk, therefore

BerriAI/litellm#11276

[manojkumarredbus]

One can use this for Ollama, it supports both function call and streaming, and currently being worked upon .
https://github.com/redbus-labs/adk-java/blob/main/core/src/main/java/com/google/adk/models/OllamaBaseLM.java

Also an Azure OAI compatible one available custom built , but one can port it to OAI compatible by removing additional auth code

https://github.com/redbus-labs/adk-java/blob/main/core/src/main/java/com/google/adk/models/RedbusADG.java

[vorburger]

Our LangChain4j connector in contrib/ might help here - have you seen and tried it out?

I'll mark this issue as pending your feedback. We'll close it if we don't hear back from you within ~1 month.

[julienba]

I found this issue when I tried to integrate a Java equivalent of LiteLLM and realized it was supported only for Python.

I managed to get it to work with Mistral using com.google.adk/google-adk-langchain4j. However, I'm losing the tokens used (the equivalent of the Gemini usageMetadata). Is there a way to get the metadata out of LLM integrated with lang4j?

Edit: There is already a PR for it #623!

[daybyter]

I found this issue when I tried to integrate a Java equivalent of LiteLLM and realized it was supported only for Python.

I managed to get it to work with Mistral using com.google.adk/google-adk-langchain4j. However, I'm losing the tokens used (the equivalent of the Gemini usageMetadata). Is there a way to get the metadata out of LLM integrated with lang4j?

Edit: There is already a PR for it #623!

Did you manage to connect the ADK to a local AI provider? I am looking for a download of the

import com.google.adk.models.litellm.LiteLlm;

package, but cannot find anything? It seems, there is a clojure version of LiteLlm, but without that package?

TIA for any response

[julienba]

TIA for any response

No I give up on adk. It does not seems mature enough if you intent to use something else that google api. Instead, I use an AI assistant to implement a client for each model using an HTTP client and malli: 0 dependency and full control.

[daybyter]

TIA for any response

No I give up on adk. It does not seems mature enough if you intent to use something else that google api. Instead, I use an AI assistant to implement a client for each model using an HTTP client and malli: 0 dependency and full control.

Thanks a ton for your response. You seem to be further than I am. I will try for a few more days to use the adk with my local AI server. But I also have to check other frameworks to reduce my code. DeerFlow is on my list to check. Maybe it could be helpful to you, too: https://github.com/bytedance/deer-flow

[daybyter]

TIA for any response

No I give up on adk. It does not seems mature enough if you intent to use something else that google api. Instead, I use an AI assistant to implement a client for each model using an HTTP client and malli: 0 dependency and full control.

So I wrote me a LMStudioLlm mit LlmStudioRequest, but the Response is a bit of a problem, since it seems you cannot simply override the LlmResponse constructor if you are in a different package?
Do you code your own agent framework?