Skip to content

feat: added the llama-server backend#28

Merged
ceberam merged 1 commit into
mainfrom
dev/add-llama-server-backend
May 21, 2026
Merged

feat: added the llama-server backend#28
ceberam merged 1 commit into
mainfrom
dev/add-llama-server-backend

Conversation

@PeterStaar-IBM
Copy link
Copy Markdown
Member

@PeterStaar-IBM PeterStaar-IBM commented May 21, 2026

Summary

  • Adds a new LlamaServerBackend for llama.cpp's llama-server OpenAI-compatible HTTP API (default
    http://localhost:8080/v1).
  • Subclasses OpenAICompatibleBackend for consistency with LMStudioBackend and LiteLLMBackend.
  • Wires the new backend into the registry, exports, factory, and the BackendConfig.type literal so it can be
    selected from task YAML via type: llama-server.
  • Updates the CLI task template, README, and the editor.yaml/writer.yaml/enrich.yaml task configs to list the
    new option.

Usage

backend:
type: llama-server # mellea | ollama | lmstudio | litellm | llama-server
# base_url: http://localhost:8080/v1 # default
timeout: 120
models:
reasoning: gpt-oss-20b
writing: gpt-oss-20b

Test plan

  • uv run pytest tests/test_backend_factory.py tests/test_direct_backends.py
    tests/test_task_model_backend_config.py — all 11 tests pass, including:
    • test_create_llama_server_backend_from_config — verifies factory + default base URL.
    • test_create_llama_server_backend_with_custom_base_url — verifies custom base_url is respected.
    • test_llama_server_session_tracks_history — verifies session posts to /chat/completions and maintains chat
      history.
  • Smoke test against a locally running llama-server instance with a GGUF model.

Signed-off-by: Peter Staar <taa@zurich.ibm.com>
@PeterStaar-IBM PeterStaar-IBM requested a review from ceberam May 21, 2026 10:41
@github-actions
Copy link
Copy Markdown
Contributor

DCO Check Passed

Thanks @PeterStaar-IBM, all your commits are properly signed off. 🎉

@mergify
Copy link
Copy Markdown
Contributor

mergify Bot commented May 21, 2026

Merge Protections

Your pull request matches the following merge protections and will not be merged until they are valid.

🟢 Enforce conventional commit

Wonderful, this rule succeeded.

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

  • title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert)(?:\(.+\))?(!)?:

🟢 Require two reviewer for test updates

Wonderful, this rule succeeded.

When test data is updated, we require two reviewers

  • #approved-reviews-by >= 1

@codecov
Copy link
Copy Markdown

codecov Bot commented May 21, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

Copy link
Copy Markdown
Member

@ceberam ceberam left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ceberam ceberam merged commit cb2ad4c into main May 21, 2026
11 checks passed
@ceberam ceberam deleted the dev/add-llama-server-backend branch May 21, 2026 12:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants