-
Notifications
You must be signed in to change notification settings - Fork 0
Add LiteRT-LM MCP Server #5
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
5f17bb2
63e10f1
0fb4ec7
f1715b5
6a7422d
01855d0
362d389
5274568
d99d78d
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,35 @@ | ||
| name: Verify LiteRT-LM MCP | ||
|
|
||
| on: | ||
| push: | ||
| paths: | ||
| - 'mcp-servers/litert-mcp/**' | ||
| - '.github/workflows/verify-litert-mcp.yml' | ||
| pull_request: | ||
| paths: | ||
| - 'mcp-servers/litert-mcp/**' | ||
| - '.github/workflows/verify-litert-mcp.yml' | ||
|
|
||
| jobs: | ||
| verify: | ||
| runs-on: ubuntu-latest | ||
| steps: | ||
| - uses: actions/checkout@v4 | ||
|
|
||
| - name: Set up Python | ||
| uses: actions/setup-python@v5 | ||
| with: | ||
| python-version: '3.12' | ||
|
|
||
| - name: Verify Server Startup (Initialize) | ||
| run: | | ||
| echo '{"jsonrpc": "2.0", "id": 1, "method": "initialize", "params": {}}' | python3 mcp-servers/litert-mcp/server.py | ||
|
|
||
| - name: Verify Tools List | ||
| run: | | ||
| OUTPUT=$(echo '{"jsonrpc": "2.0", "id": 2, "method": "tools/list", "params": {}}' | python3 mcp-servers/litert-mcp/server.py) | ||
| echo "$OUTPUT" | ||
| if [[ "$OUTPUT" != *"run_inference"* ]]; then | ||
| echo "Error: run_inference tool not found in output" | ||
| exit 1 | ||
| fi |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,23 @@ | ||
| # AGENTS.md | ||
|
|
||
| ## Mission | ||
| To enable autonomous AI agents to efficiently reason, plan, and execute tasks within this repository, specifically focusing on the integration and orchestration of Model Context Protocol (MCP) servers and Video Intelligence pipelines. | ||
|
|
||
| ## Scope of "Actionable Room" | ||
| Agents are granted permission and encouraged to: | ||
| 1. **Modify and Expand `mcp-servers/`**: Create new MCP servers, update existing ones, and refactor code to improve modularity and performance. | ||
| 2. **Improve Automation**: Create and edit GitHub Actions workflows (`.github/workflows/`) to add robust testing and verification for new features. | ||
| 3. **Refactor for Clarity**: Improve documentation (READMEs) and code structure to facilitate better "Machine Readability" and "Human Understandability". | ||
|
|
||
| ## Protocols | ||
| 1. **Verify Before Submit**: | ||
| - Always run relevant verification scripts or tests before submitting changes. | ||
| - If no test exists for a new feature, **create one**. | ||
| 2. **CI/CD Alignment**: | ||
| - Ensure all changes pass existing CI checks. | ||
| - When adding a new component (like an MCP server), add a corresponding CI workflow to ensure it remains functional. | ||
| 3. **Cross-Platform Compatibility**: | ||
| - Write code that is compatible with Linux and Windows environments whenever possible (e.g., handling `asyncio` loops correctly). | ||
| 4. **Documentation**: | ||
| - Update `README.md` files when interface changes occur. | ||
| - Document limitations (e.g., "Text-only CLI wrapper") clearly. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,61 @@ | ||
| # LiteRT-LM MCP Server | ||
|
|
||
| This MCP server provides an interface to Google's **LiteRT-LM**, a high-performance runtime for Large Language Models (LLMs) on edge devices (Android, iOS, Linux, MacOS, Windows). | ||
|
|
||
| It allows you to run inference on local models (like Gemma, Phi, Qwen) directly from your MCP ecosystem. | ||
|
|
||
| **Note:** This server currently wraps the `lit` CLI. Multimodal inputs (image/audio) are enabled in the interface but require the C++ API or Python bindings. The CLI wrapper currently supports **Text-only inference** until CLI flags for multimodal are verified. | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| 1. **LiteRT-LM**: You must have LiteRT-LM installed or built. | ||
| * Official Repository: [google-ai-edge/LiteRT-LM](https://github.com/google-ai-edge/LiteRT-LM) | ||
| * Follow the "Build and Run" instructions in the official repo to build the `lit` CLI or `litert_lm_main` binary. | ||
| * Alternatively, download prebuilt binaries if available for your platform. | ||
|
|
||
| 2. **Models**: Download a supported `.litertlm` model. | ||
| * Models are available on Hugging Face: [LiteRT Community](https://huggingface.co/litert-community) | ||
|
|
||
| ## Configuration | ||
|
|
||
| Set the following environment variables: | ||
|
|
||
| * `LIT_BINARY_PATH`: Path to the `lit` CLI executable or `litert_lm_main` binary. Defaults to `lit` (assuming it's in your PATH). | ||
| * `LIT_MODEL_PATH`: Default path to your `.litertlm` model file. (Optional, can be passed per request). | ||
|
|
||
| ## Usage | ||
|
|
||
| ### Tools | ||
|
|
||
| #### `run_inference` | ||
|
|
||
| Runs inference using the configured LiteRT-LM model. | ||
|
|
||
| * **Arguments**: | ||
| * `prompt` (string, required): The input text prompt. | ||
| * `model_path` (string, optional): Path to the `.litertlm` model file. Overrides `LIT_MODEL_PATH` env var. | ||
| * `image_path` (string, optional): Path to an image file for multimodal inference. | ||
| * `audio_path` (string, optional): Path to an audio file for multimodal inference. | ||
| * `backend` (string, optional): Backend to use (`cpu`, `gpu`, `npu`). Defaults to `cpu`. | ||
|
|
||
| ### Example | ||
|
|
||
| ```json | ||
| { | ||
| "name": "run_inference", | ||
| "arguments": { | ||
| "prompt": "What is the capital of France?", | ||
| "backend": "cpu" | ||
| } | ||
| } | ||
| ``` | ||
|
Comment on lines
43
to
51
|
||
|
|
||
| ## Setup for Development | ||
|
|
||
| This server uses a manual JSON-RPC implementation to avoid external dependencies in the base environment. Just run: | ||
|
|
||
| ```bash | ||
| python3 server.py | ||
| ``` | ||
|
|
||
| Ensure `LIT_BINARY_PATH` is set correctly. | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The
README.mdlistsimage_pathandaudio_pathas optional arguments. However, theserver.pyexplicitly returns an error if these are provided, stating that multimodal input is not yet supported via the CLI wrapper. To avoid confusion, please update the README to clearly state this limitation or remove these arguments from the documentation until they are supported.