RFC-generic-llm-config

[spichen]

Propose a provider-agnostic GenericLlmConfig component that uses a string-based provider.type discriminator and flexible ProviderConfig, replacing the need for per-provider LlmConfig subclasses.

[cesarebernardis]

Hi @spichen, thank you for the great work.

This is a very good starting point, we have been discussing this design internally, and we agreed on a few points.
As mentioned in #108 (comment), we prefer to make multiple, small, simple changes to the spec, so that it's easier to agree on the design. We would mainly remove the Auth part of it for now, and simplify a bit the current proposal.

Here's what we thought could be a possible solution:

Instead of having a new GenericLlmConfig, we make the LlmConfig not abstract anymore.
We add a few parameters related to providers and apis that are optional strings, type-wise, but in the spec we indicate some known values, similarly to what we do with api_type already in some configs.

# Not abstract anymore
class LlmConfig(Component):
    """A generic, provider-agnostic LLM configuration."""

    model_id: str
    """Primary model identifier"""

    provider: Optional[str] = None
    """Model provider (e.g., meta, openai, anthropic, ...)"""

    api_provider: Optional[str] = None
    """APIs provider (e.g., oci, openai, vertex_ai, aws_bedrock, ...)"""

    api_type: Optional[str] = None
    """API protocol to use (e.g., chat_completions, responses, ...)"""

Existing classes will fix the value of these attributes where needed, and they should be excluded from the serialization for brevity.

class OciGenAiConfig(GenericLlmConfig):
    # Don't know the (model) provider a priori
    api_provider: str = "oci"  # Freeze the value
    ...

class OpenAiConfig(GenericLlmConfig):
    # We know both provider and api_provider, we fix them
    provider: str = "openai"  # Freeze the value 
    api_provider: str = "openai"  # Freeze the value 
    ...

class OpenAiCompatibleConfig(GenericLlmConfig):
    # This limits only the api_type to chat_completions or responses
    # We don't know what is the model, nor the api provider
    ...

The name of the attributes are chosen to fit what is already present in some LlmConfig classes (you can give a look at the OCI ones for example).

The main reason for this choice is that all the attributes are quite independent from each other, and collecting them in a Provider class would require to fix one of the attributes, but in an unclear manner, and the number of subclasses could explode. For example: meta model exposed by AWS bedrock behind responses APIs, cohere model exposed by OCI behind chat_completions, ...
We think it would be a wiser start to have something (almost) free form and not encapsulated in classes, so that users are free to support new models, providers, and APIs without waiting for a new version of Agent Spec, which is I think the biggest value of this proposal.

Let us know what you think about it.

[spichen]

Hi @cesarebernardis , thanks for the thoughtful feedback and for discussing this internally.

I agree with the direction here. Making LlmConfig non-abstract is much cleaner than introducing a separate component. I didn't propose this initially because I didn't want to break the existing abstract contract of LlmConfig, but if the team is on board with that change then it's clearly the better path.

Totally on board with deferring Auth as well. Better to land the provider/API fields as a focused change and layer authentication on top later.

I'll rework the PR to follow this approach. Let me know if there's anything else you'd like adjusted.

[Ahmad-Zaaza]

Strong support for this RFC — concrete use case: OpenRouter + reasoning tokens

I want to share a concrete problem that this RFC would unblock.

The problem today

When using OpenRouter (or similar routing proxies like LiteLLM) with Agent Spec + the LangGraph adapter, the only viable config types are OpenAiCompatibleConfig or VllmConfig. Both map to LangChain's ChatOpenAI inside _llm_convert_to_langgraph (via _create_chat_openai_model).

The issue is that ChatOpenAI explicitly scopes itself to the official OpenAI API spec and silently drops non-standard response fields like reasoning_content and thinking_blocks. LangChain's own docs now state this clearly:

ChatOpenAI targets official OpenAI API specifications only. Non-standard response fields from third-party providers (e.g., reasoning_content, reasoning, reasoning_details) are not extracted or preserved.

This is a known and tracked problem on the LangChain side:

• [langchain-ai/langchain#32981](Reasoning tokens not passing through from OpenRouter to ChatOpenAI langchain-ai/langchain#32981) — reasoning tokens not passing through from OpenRouter to ChatOpenAI
• [langchain-ai/langchain#34328](OpenRouter / LiteLLM / OpenAI API compatibility across providers / proxies langchain-ai/langchain#34328) — umbrella tracking issue for OpenRouter/LiteLLM/vLLM compatibility

LangChain's solution is provider-specific packages. For example, [langchain-openrouter](https://pypi.org/project/langchain-openrouter/) (ChatOpenRouter) properly handles reasoning tokens, content_blocks, and provider-specific extensions. Similarly, ChatDeepSeek handles reasoning_content where ChatOpenAI does not.

But the current Agent Spec type hierarchy has no way to reach these provider-specific LangChain classes. The isinstance dispatch in the LangGraph adapter only knows about VllmConfig → ChatOpenAI, OllamaConfig → ChatOllama, OpenAiConfig → ChatOpenAI, OpenAiCompatibleConfig → ChatOpenAI, and OciGenAiConfig → ChatOCIGenAI. There is no path to ChatOpenRouter.

How this RFC solves it

The api_provider string discriminator is exactly the escape hatch needed. With a bare LlmConfig and api_provider: "openrouter", the LangGraph adapter can dispatch to ChatOpenRouter instead of ChatOpenAI:

component_type: LlmConfig
model_id: "anthropic/claude-sonnet-4"
api_provider: openrouter
api_type: chat_completions
default_generation_parameters:
  temperature: 0.7

And on the adapter side:

# In _llm_convert_to_langgraph, string-based dispatch for bare LlmConfig:
elif llm_config.api_provider == "openrouter":
    from langchain_openrouter import ChatOpenRouter
    return ChatOpenRouter(model=llm_config.model_id, ...)

Without this RFC, the only alternative is adding a full OpenRouterConfig subclass to the spec — plus adapter changes, a new spec version, etc. — which is exactly the scalability problem described in the Motivation section. And the same pattern repeats for LiteLLM, DeepSeek, Groq, Together AI, and every other provider that extends the Chat Completions wire protocol.

Suggestions

1. Add "openrouter" and "litellm" to the well-known values for api_provider — these are routing/proxy providers that are increasingly common in production deployments, and they represent the exact use case where provider (who made the model) ≠ api_provider (who serves the API).

2. Consider whether the adapter dispatch strategy section should mention provider-specific LangChain packages as the expected targets for new api_provider values, since this is how LangChain itself is solving the same "one class can't handle all providers" problem.

3. The provider_extensions / extra kwargs escape hatch would also be valuable here — ChatOpenRouter accepts a reasoning dict ({"effort": "high", "summary": "auto"}) that doesn't map to anything in LlmGenerationConfig today. Being able to pass provider-specific options without waiting for a spec change would complete the story.

4. The adapter converter needs an extension point too — not just the spec. This RFC addresses the spec/config side well, but the api_provider string is only half the story. On the adapter side, AgentSpecToLangGraphConverter._llm_convert_to_langgraph is a closed isinstance chain that ends with raise NotImplementedError. There is no registry, no callback, and no override hook. Even after this RFC lands, a user with a custom or unsupported api_provider value would still hit a hard wall.

   Today the only workaround is subclassing both the converter and the loader, which depends on internal method signatures:

   class MyConverter(AgentSpecToLangGraphConverter):
       def _llm_convert_to_langgraph(self, llm_config, config):
           if getattr(llm_config, 'api_provider', None) == 'my_provider':
               return MyProviderChatModel(model=llm_config.model_id)
           return super()._llm_convert_to_langgraph(llm_config, config)
   
   class MyLoader(AgentSpecLoader):
       @property
       def agentspec_to_runtime_converter(self):
           return MyConverter()

   This works but is fragile and not something most users would discover. A cleaner approach would be for the converter (or the loader) to accept a user-supplied LLM converter registry keyed by api_provider string:

   loader = AgentSpecLoader(
       tool_registry=tools,
       llm_converters={
           "openrouter": lambda cfg, config: ChatOpenRouter(model=cfg.model_id, ...),
           "my_company_proxy": lambda cfg, config: MyCompanyChatModel(model=cfg.model_id, ...),
       }
   )

   The converter would check this registry before falling into the existing isinstance dispatch, and raise a clear error for truly unknown providers. This pairs naturally with the RFC's api_provider discriminator — the spec defines what the provider is, and the adapter lets users register how to convert it without subclassing anything. It would also make Agent Spec more practical for enterprise teams that run internal LLM gateways or custom inference endpoints.

Overall, this is a well-designed proposal. The three-axis separation (provider / api_provider / api_type) maps directly to real deployment patterns, and the backward compatibility with existing subclasses is clean. Suggestion No.4 is the piece I'd most like to see discussed — without an adapter-side extension point, the flexibility this RFC adds at the spec level can't be fully realized by end users. Looking forward to seeing this land.

[cesarebernardis]

Hi @spichen , sorry for the delay in our answer.

We went through the latest version of the proposal internally, and the team approved. You can proceed with the implementation of this design.

Thank you again for your contribution.