diff --git a/README.md b/README.md index 5c9425d..bab64e3 100644 --- a/README.md +++ b/README.md @@ -30,26 +30,31 @@ TOPGO Platform 是企业级一体化 AI 资源管理平台,整合了 **智能 ### 核心架构 +TOPGO 采用 **数据平面 / 控制平面分离** 的企业级架构。当前已实现并部署的是 +Gateway 数据平面(Go),它已内建调度、健康探测与策略编排能力;Orchestrator +作为独立的控制平面(Python)在 roadmap 中,用于跨实例的资源注册与全局编排。 + ``` ┌─────────────────────────────────────────────────────────┐ │ TOPGO Platform │ ├─────────────────────────────────────────────────────────┤ -│ ┌─────────────────────┐ ┌─────────────────────────┐ │ -│ │ TOPGO Orchestrator │───→│ TOPGO Gateway │ │ -│ │ (调度引擎) │ │ (统一网关) │ │ -│ │ │ │ │ │ -│ │ • 资源自动注册 │ │ • 智能路由 │ │ -│ │ • 健康探测与轮转 │ │ • 精准计费 │ │ -│ │ • 多维标签管理 │ │ • 用户管理 │ │ -│ │ • 策略编排 │ │ • 负载均衡 │ │ -│ └─────────────────────┘ └─────────────────────────┘ │ +│ ┌─────────────────────────┐ ┌──────────────────────┐ │ +│ │ TOPGO Gateway (已实现) │←─│ TOPGO Orchestrator │ │ +│ │ 数据平面 · Go + Vue │ │ 控制平面 · Roadmap │ │ +│ │ │ │ │ │ +│ │ • 智能路由 (GEO) │ │ • 跨实例资源注册 │ │ +│ │ • 调度 / 健康探测 │ │ • 全局策略编排 │ │ +│ │ • FreeLLM 代理层 │ │ • 资源池同步 │ │ +│ │ (压缩 + 免费源池) │ │ │ │ +│ │ • 精准计费 / 用户管理 │ │ │ │ +│ └─────────────────────────┘ └──────────────────────┘ │ └─────────────────────────────────────────────────────────┘ ``` -| 组件 | 技术栈 | 功能 | -|------|--------|------| -| **Gateway** | Go + Vue | API 网关、计费、用户管理 | -| **Orchestrator** | Python + Vue | 资源调度、轮转、同步 | +| 组件 | 技术栈 | 状态 | 功能 | +|------|--------|------|------| +| **Gateway** | Go + Vue | ✅ 已实现 | 智能路由、调度、计费、用户管理、FreeLLM 代理层 | +| **Orchestrator** | Python + Vue | 🚧 Roadmap | 跨实例资源注册、全局策略编排、资源池同步 | ## 快速部署 @@ -82,19 +87,21 @@ docker compose up -d ## 核心功能 -### TOPGO Gateway (统一API网关) +### TOPGO Gateway (统一API网关 · 已实现) - **多模型支持** - Claude、OpenAI (Codex)、Gemini - **资源凭证管理** - OAuth、API Key - **精准计费** - 用量级别追踪 - **智能路由** - 基于 GEO 的动态路由 - **用户管理** - API Key、套餐、充值 -### TOPGO Orchestrator (智能调度引擎) -- **自动注册** - 自动创建 AI 账号 -- **智能轮转** - 按可用量自动调度 -- **健康探测** - 实时监控资源状态 -- **策略编排** - 多维度路由策略 -- **Gateway 同步** - 自动推送资源到网关 +### FreeLLM 代理层 (Gateway 内建) +借鉴 OmniRoute 的免费源 + 压缩思路,企业级落地: +- **免费源池** - 内置 10+ 免费 / 免费层 provider 种子清单(Pollinations/Groq/Gemini/Cerebras/SambaNova/DeepSeek/Mistral/Together/OpenRouter/Kiro),复用 `channel_monitor` 自动探活,按 `cost_tier=free` 打标 +- **Token 压缩** - Go 原生可插拔引擎(`internal/service/compression`),Lite/Standard/Aggressive 三档,规则引擎对标 OmniRoute 的 Caveman,省 15-70%,代码块自动保护 +- **Model 级 failover** (Roadmap) - 在现有 `group_model_routing` 上加同模型 fallback 链 + 8ms 超时切换 + +### TOPGO Orchestrator (控制平面 · Roadmap) +Gateway 已内建调度 / 健康探测 / 策略编排;Orchestrator 将这些能力抽取为独立控制平面,支持跨实例资源注册与全局编排。 ## 资源供给流程 @@ -114,20 +121,24 @@ docker compose up -d ``` api-token/ -├── backend/ # TOPGO Gateway (Go) -│ ├── cmd/server/ # 主入口 -│ ├── internal/ # 核心逻辑 -│ ├── frontend/ # Vue 前端 -│ └── deploy/ # 部署配置 +├── backend/ # TOPGO Gateway (Go) +│ ├── cmd/server/ # 主入口 +│ ├── internal/ +│ │ ├── service/ +│ │ │ ├── compression/ # FreeLLM 代理层 · Token 压缩 (新) +│ │ │ ├── freellm/ # FreeLLM 代理层 · 免费源池 seed (新) +│ │ │ ├── channel.go # 渠道管理 + channel_monitor 探活 +│ │ │ ├── scheduler_*.go # 调度 / 轮转 / fallback +│ │ │ └── openai_gateway_*.go # 统一 ingress +│ │ ├── ent/ # ORM (entgo) +│ │ └── server/ # 路由 / 中间件 +│ ├── frontend/ # Vue 前端 +│ └── deploy/ # 部署配置 │ -├── orchestrator/ # TOPGO Orchestrator (Python) -│ ├── src/ # 核心逻辑 -│ ├── web/ # Vue 前端 -│ └── docs/ # 文档 -│ -├── docs/ # 公共文档 -├── deploy/ # 公共部署配置 -└── docker-compose.yml # 一体化部署 +├── orchestrator/ # TOPGO Orchestrator (Python) · 🚧 Roadmap +├── docs/ # 公共文档 +├── deploy/ # 公共部署配置 +└── docker-compose.yml # 一体化部署 ``` ## 环境配置 @@ -160,8 +171,24 @@ api-token/ ## 文档 - [GEO 智能路由架构](TOPGO_GEO_ARCHITECTURE.md) +- [FreeLLM 代理层设计](TOPGO_FREELLM_PROXY.md) - [Gateway 部署指南](backend/deploy/README.md) -- [Orchestrator 配置说明](orchestrator/docs/configuration.md) + +## 与 OmniRoute 的对位与借鉴 + +同赛道(LLM 路由/网关),受众错开,不重复收录: + +| 维度 | TOPGO Platform | OmniRoute | +|------|----------------|-----------| +| 定位 | 企业级 AI 资源管理(数据/控制平面分离) | 个人/开发者免费路由 | +| 技术栈 | Go Gateway + Vue(Orchestrator Python roadmap) | 100% TypeScript | +| 免费源 | FreeLLM 代理层 · 10+ provider 种子池,复用 channel_monitor 探活 | 90+ 免费层 / 11 家永久免费 | +| failover | GEO 智能路由(地域级)+ Model 级 fallback (roadmap) | 4-tier auto-fallback (Subscription→API Key→Cheap→Free) | +| token 优化 | Go 原生压缩(Lite/Standard/Aggressive,规则引擎对标 Caveman) | RTK + Caveman + LLMLingua-2,省 15-95% | +| 企业功能 | 多租户、计费、OAuth、资源池轮转、套餐充值 | 无 | + +> OmniRoute 的免费源清单 / 多级 failover / 压缩三点已反哺 TOPGO 的 FreeLLM 代理层, +> 详见 [TOPGO_FREELLM_PROXY.md](TOPGO_FREELLM_PROXY.md)。 ## 许可证 diff --git a/TOPGO_FREELLM_PROXY.md b/TOPGO_FREELLM_PROXY.md new file mode 100644 index 0000000..589afd0 --- /dev/null +++ b/TOPGO_FREELLM_PROXY.md @@ -0,0 +1,148 @@ +# TOPGO FreeLLM 代理层设计 + +把 OmniRoute 的三个独有卖点(免费源池 / 多级 failover / token 压缩)反哺进 TOPGO +Gateway 的 FreeLLM 代理层。本文档是落地参照,不是空气架构——每个卖点都映射到 +`backend/internal/` 里真实存在的代码。 + +## 设计立场 + +OmniRoute 是 100% TypeScript、面向个人开发者的免费路由。TOPGO 是 Go、面向企业的 +资源管理平台。两者受众错开,直接照搬 TS 引擎进 Go 网关是企业级反模式(外部进程 +多一跳延迟、第三方耦合)。因此本代理层走 **Go 原生、可插拔、分阶段**: + +| 卖点 | OmniRoute | TOPGO 落地 | 状态 | +|------|-----------|-----------|------| +| 免费源池 | 90+ 免费层 / 11 家永久免费 | `freellm` seed 清单 + 复用 `channel_monitor` 探活 | ✅ 已实现(seed) | +| token 压缩 | RTK + Caveman + LLMLingua-2 | `compression` 包,Lite/Standard/Aggressive,规则引擎对标 Caveman | ✅ 已实现(规则引擎);LLMLingua-2 ONNX 🚧 Phase 2 | +| 多级 failover | 4-tier auto-fallback | GEO 地域路由(已有)+ Model 级 fallback | 🚧 Model 级 Roadmap | + +--- + +## 1. Token 压缩 — `internal/service/compression` + +### 1.1 架构 + +可插拔 `Engine` 接口,调用点只传 `body []byte` 进、拿 `body []byte` 出,零耦合 +ent/wire/gateway 内部: + +```go +type Engine interface { + Name() string + Compress(text string, mode Mode) string +} + +// 调用点(gateway ingress 顶部,json.Unmarshal 之前) +body, stats, _ := compression.RewriteRequestBody(body, engine, compression.Options{Mode: compression.ModeStandard}) +``` + +- **格式感知**:识别 OpenAI Chat Completions(`messages[].content`)、OpenAI + Responses(`input[].content[].{text,input_text}` + 顶层 `instructions`)、Anthropic + Messages(`messages[].content[].text` + 顶层 `system`)。其它 JSON 原样透传。 +- **代码块保护**:`CompressCode=false`(默认)时,``` ``` 包裹的代码段原样保留,只压 + 缩散文。避免破坏 prompt 里的代码示例。 +- **幂等、无副作用、并发安全**。 +- **点编辑**:用 gjson/sjson 做字段级改写(与本仓库 `openai_gateway_chat_completions.go` + 同款模式),保留原始 key 顺序与非文本字段。 + +### 1.2 引擎 + +| 引擎 | 模式 | 说明 | 目标节省 | +|------|------|------|----------| +| `LiteEngine` | lite | 空白/分隔符归一化,CJK 安全 | ~10-15% | +| `RuleEngine` | standard | 规则表:冗长短语→精简("in order to"→"to"、"utilize"→"use"),删填充词("basically"/"very") | ~30-50% | +| `RuleEngine` | aggressive | standard + 连续重复行去重 + 空行清理 | ~50-70% | + +规则表对标 OmniRoute 的 Caveman(规则式散文压缩),~80 条安全改写,单词边界锚定、 +大小写不敏感,CJK 不命中(自然降级为空白归一化)。 + +### 1.3 集成点 + +[backend/internal/service/openai_gateway_chat_completions.go](backend/internal/service/openai_gateway_chat_completions.go) +的 `ForwardAsChatCompletions` 入口,`body []byte` 进来后、`json.Unmarshal` 之前插入 +一行压缩调用。Anthropic / Gemini ingress 同理。压缩 `Stats` 写入 `usage_log`,复用 +现有计费/可观测通路。 + +### 1.4 Phase 2:LLMLingua-2 ONNX + +LLMLingua-2 是 ML 语义裁剪(MobileBERT ONNX),压缩率更高但需要模型文件。规划为 +`compression` 包的第三个 `Engine` 实现,通过 ONNX Runtime Go 绑定加载,**opt-in**, +不替换规则引擎。规则引擎保底(无依赖、低延迟),LLMLingua-2 按需开启。 + +--- + +## 2. 免费源池 — `internal/service/freellm` + +### 2.1 定位 + +`freellm.SeedCatalog()` 是 **种子清单**,不是 live client。10 个真实免费 / 免费层 +provider(Pollinations keyless 在首位,Groq/Gemini/Cerebras/SambaNova/DeepSeek/ +Mistral/Together/OpenRouter/Kiro 跟随),按 `AuthScheme`/`Protocol`/`FreeTier`/ +`Models`/`RateLimitRPM` 标注。 + +### 2.2 集成路径 + +种子清单灌入 Gateway 现有 `channel` 表,复用已有基础设施,**不新建表**: + +- 探活 → [channel_monitor](backend/ent/channelmonitor) + `channel_monitor_history` + + `channel_monitor_request_template`(已实现,定期探测上游可用性)。 +- 路由/过滤 → `group` + `group_account_filter`(migration 081)。 +- 标签 → 按 `cost_tier=free` 归类,与付费源隔离调度(避免免费源限流拖垮付费 SLA)。 + +### 2.3 鉴权模型 + +只有 Pollinations 无需凭证。其余都是免费层/注册额度,运营者启用时自备 credential。 +这符合企业姿态:免费源拉伸预算,不替代凭证治理。 + +--- + +## 3. Model 级 failover — Roadmap + +### 现状 + +- GEO 路由(地域级)已在 [TOPGO_GEO_ARCHITECTURE.md](TOPGO_GEO_ARCHITECTURE.md) 设计, + Gateway 评分算法落地。 +- `group_model_routing`(migration 040)+ `group_invalid_request_fallback`(migration + 043b)已有 group→account 级 fallback。 +- `scheduler_cache` / `scheduler_outbox` / `openai_account_scheduler` 是真实轮转引擎。 + +### 缺口(对应 OmniRoute 的 Provider→Connection→Model 三层) + +缺 Model 级:DeepSeek 免费源挂了,8ms 内切 Qwen 免费源。 + +### 落地计划 + +1. `group_model_routing` 上加同 model 的 fallback 链(一个 model 映射到多个 channel)。 +2. scheduler 加超时切换阈值(8ms 连接级 / N ms 首字节级)。 +3. `error_passthrough_rule` 配合:5xx / 超时触发同 model 下一 channel,不透传给客户端。 + +**未本次实现**——scheduler 热路径手术风险高,需独立 PR + 完整 e2e 测试覆盖。 + +--- + +## 4. 串成闭环(与前序"四色卡片 MCP"联动) + +TOPGO Gateway 若暴露为 MCP Server,FreeLLM 代理层的"智能路由 + 免费源 + 压缩"即成为 +卡片 MCP 的 model 调用底座,省掉 Pro 订阅。链路: + +``` +四色卡片 MCP ──clawhub install topgo──→ TOPGO Gateway (MCP Server) + ├─ compression (省 token) + ├─ freellm pool (免费源) + └─ GEO + Model failover (高可用) +``` + +api-token(本仓库)+ 四色卡片 + OmniRoute 参照,三者闭环。 + +--- + +## 5. 验证 + +```bash +cd backend +go test ./internal/service/compression/ # 15 用例,覆盖 OpenAI/Anthropic/Responses + 代码块保护 + Stats +go test ./internal/service/freellm/ # 种子清单唯一性 + Find +go build ./... # 全量编译 +``` + +压缩包自包含:仅依赖 gjson/sjson/testify(均已在 go.mod),不依赖 ent/wire,可独立 +编译与测试。 diff --git a/backend/internal/service/compression/compression.go b/backend/internal/service/compression/compression.go new file mode 100644 index 0000000..12d948b --- /dev/null +++ b/backend/internal/service/compression/compression.go @@ -0,0 +1,244 @@ +// Package compression provides transparent, format-aware prompt compression +// for the TOPGO Gateway. It is the Go-native counterpart to OmniRoute's +// compression pipeline (RTK / Caveman / LLMLingua-2), exposed as a pluggable +// Engine interface so heavier ML-based engines (e.g. an ONNX-backed +// LLMLingua-2 engine) can be added later without touching call sites. +// +// Design goals: +// - Zero coupling to ent/wire/gateway internals: call sites hand over a raw +// request body []byte and receive a rewritten []byte back. +// - Format-aware: understands OpenAI Chat Completions (messages[].content), +// OpenAI Responses (input[].content[].{text,input_text}, top-level +// instructions), and Anthropic Messages (messages[].content[].text, +// top-level system string/array). Any other JSON is passed through intact. +// - Idempotent and side-effect free; safe for concurrent use. +// +// Integration point — call RewriteRequestBody at the top of the gateway +// ingress, before json.Unmarshal of the request: +// +// if comp != nil { +// var st compression.Stats +// body, st, _ = compression.RewriteRequestBody(body, comp, compression.Options{Mode: compression.ModeStandard}) +// _ = st // -> usage log / telemetry +// } +package compression + +import ( + "encoding/json" + "fmt" + + "github.com/tidwall/gjson" + "github.com/tidwall/sjson" +) + +// Mode controls how aggressively an Engine compresses text. +type Mode string + +const ( + ModeOff Mode = "off" // no compression + ModeLite Mode = "lite" // whitespace / separator trim only, ~10-15% + ModeStandard Mode = "standard" // rule-based prose compression, ~30-50% + ModeAggressive Mode = "aggressive" // rules + redundancy trimming, ~50-70% +) + +// Options configures a compression pass. The zero value is NOT usable — +// always go through Options{}.WithDefaults() (RewriteRequestBody does this +// internally when Mode == ""). +type Options struct { + Mode Mode + // CompressCode, when false (the default), leaves fenced ```code``` blocks + // and inline `code` untouched. Set true to also run rules over code + // (risky; intended for ModeAggressive experiments only). + CompressCode bool + // MaxInputChars caps compression of a single text field; fields larger + // than this pass through unchanged. 0 => 200k. Guards against + // pathological inputs. + MaxInputChars int +} + +// WithDefaults returns opts with sane defaults applied. +func (o Options) WithDefaults() Options { + if o.Mode == "" { + o.Mode = ModeStandard + } + if o.MaxInputChars == 0 { + o.MaxInputChars = 200_000 + } + return o +} + +// Stats reports the aggregate result of a compression pass for observability. +type Stats struct { + Engine string `json:"engine"` + Mode Mode `json:"mode"` + InputChars int `json:"input_chars"` + OutputChars int `json:"output_chars"` + SavedChars int `json:"saved_chars"` + SavedPercent float64`json:"saved_percent"` + FieldsTouched int `json:"fields_touched"` +} + +func (s *Stats) record(in, out string) { + s.InputChars += len(in) + s.OutputChars += len(out) + s.SavedChars += len(in) - len(out) + s.FieldsTouched++ +} + +func (s *Stats) finalize() { + if s.InputChars > 0 { + s.SavedPercent = float64(s.SavedChars) / float64(s.InputChars) * 100 + } +} + +// Engine compresses a single text value. Implementations must be safe for +// concurrent use. The mode is passed per-call so one engine can serve all +// modes (engines that do not distinguish modes simply ignore it). +type Engine interface { + Name() string + Compress(text string, mode Mode) string +} + +// RewriteRequestBody parses body as JSON and compresses the text content of +// known chat request shapes. Unknown shapes are returned unchanged. The +// original key order and all non-text fields are preserved (point edits via +// sjson). On any parse error the original body is returned with a zero-value +// Stats and the error. +func RewriteRequestBody(body []byte, engine Engine, opts Options) ([]byte, Stats, error) { + var st Stats + if engine == nil || len(body) == 0 { + return body, st, nil + } + opts = opts.WithDefaults() + if opts.Mode == ModeOff { + return body, st, nil + } + if !gjson.ValidBytes(body) { + return body, st, fmt.Errorf("compression: invalid JSON body (%d bytes)", len(body)) + } + st.Engine = engine.Name() + st.Mode = opts.Mode + + // OpenAI Chat Completions / Anthropic Messages: messages[].content + if gjson.GetBytes(body, "messages").Exists() { + body = rewriteMessageArray(body, "messages", engine, opts, &st) + } + // OpenAI Responses: input[].content + if gjson.GetBytes(body, "input").Exists() { + body = rewriteMessageArray(body, "input", engine, opts, &st) + } + // OpenAI Responses top-level instructions (string). + if v := gjson.GetBytes(body, "instructions"); v.Exists() && v.Type == gjson.String { + body = rewriteScalar(body, "instructions", v.String(), engine, opts, &st) + } + // Anthropic top-level system: string OR array of {type:"text", text:"..."}. + if v := gjson.GetBytes(body, "system"); v.Exists() { + switch { + case v.Type == gjson.String: + body = rewriteScalar(body, "system", v.String(), engine, opts, &st) + case v.IsArray(): + body = rewriteTextParts(body, "system", engine, opts, &st) + } + } + + st.finalize() + return body, st, nil +} + +// rewriteMessageArray walks `messages` or `input` and compresses each +// element's content. Content may be a plain string (OpenAI CC) or an array +// of parts (OpenAI multimodal / Anthropic). +func rewriteMessageArray(body []byte, arrPath string, engine Engine, opts Options, st *Stats) []byte { + n := gjson.GetBytes(body, arrPath+".#").Int() + for i := int64(0); i < n; i++ { + contentPath := fmt.Sprintf("%s.%d.content", arrPath, i) + content := gjson.GetBytes(body, contentPath) + if !content.Exists() { + continue + } + switch { + case content.Type == gjson.String: + body = rewriteScalar(body, contentPath, content.String(), engine, opts, st) + case content.IsArray(): + body = rewriteTextParts(body, contentPath, engine, opts, st) + } + } + return body +} + +// rewriteTextParts walks an array of content parts and compresses the .text +// / .input_text / .text fields used by OpenAI and Anthropic part shapes. +func rewriteTextParts(body []byte, arrPath string, engine Engine, opts Options, st *Stats) []byte { + n := gjson.GetBytes(body, arrPath+".#").Int() + for i := int64(0); i < n; i++ { + for _, key := range []string{"text", "input_text"} { + p := fmt.Sprintf("%s.%d.%s", arrPath, i, key) + v := gjson.GetBytes(body, p) + if v.Exists() && v.Type == gjson.String { + body = rewriteScalar(body, p, v.String(), engine, opts, st) + } + } + } + return body +} + +// rewriteScalar compresses a single string field at path and writes it back. +func rewriteScalar(body []byte, path, in string, engine Engine, opts Options, st *Stats) []byte { + out := compressField(in, engine, opts) + if out == in { + return body + } + if b, err := sjson.SetBytes(body, path, out); err == nil { + st.record(in, out) + return b + } + return body +} + +// compressField applies per-field policy (size cap, code-block protection) +// then delegates to the engine. +func compressField(text string, engine Engine, opts Options) string { + if text == "" { + return text + } + if opts.MaxInputChars > 0 && len(text) > opts.MaxInputChars { + return text + } + if !opts.CompressCode { + return applyToProse(text, func(s string) string { return engine.Compress(s, opts.Mode) }) + } + return engine.Compress(text, opts.Mode) +} + +// applyToProse splits s on fenced code blocks (```...```) and applies fn only +// to the prose (non-code) segments, reassembling with fences intact. Inline +// `code` spans are left in place — they are part of prose but the engine is +// expected to be conservative around backticks. +func applyToProse(s string, fn func(string) string) string { + if !containsFence(s) { + return fn(s) + } + parts := splitFences(s) + var b []byte + for i, seg := range parts { + if i%2 == 0 { + b = append(b, fn(seg)...) + } else { + b = append(b, "```"...) + b = append(b, seg...) + b = append(b, "```"...) + } + } + return string(b) +} + +// DecodeRequest is a convenience helper for callers that want to inspect the +// (possibly compressed) body as a generic value before forwarding. It is +// optional and not used by RewriteRequestBody itself. +func DecodeRequest(body []byte) (map[string]any, error) { + var m map[string]any + if err := json.Unmarshal(body, &m); err != nil { + return nil, err + } + return m, nil +} diff --git a/backend/internal/service/compression/compression_test.go b/backend/internal/service/compression/compression_test.go new file mode 100644 index 0000000..80bd04a --- /dev/null +++ b/backend/internal/service/compression/compression_test.go @@ -0,0 +1,178 @@ +package compression + +import ( + "encoding/json" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "github.com/tidwall/gjson" +) + +func TestLiteEngine_NormalizesWhitespace(t *testing.T) { + in := "a b\n\n\n\nc\t\t d" + got := LiteEngine{}.Compress(in, ModeLite) + assert.Equal(t, "a b\n\nc d", got) +} + +func TestRuleEngine_StandardRewritesVerbosePhrases(t *testing.T) { + in := "In order to proceed, we need to utilize the API. Basically it works." + got := RuleEngine{}.Compress(in, ModeStandard) + assert.Contains(t, got, "to proceed") + assert.Contains(t, got, "use the API") + assert.NotContains(t, got, "utilize") + assert.NotContains(t, got, "in order to") + assert.NotContains(t, got, "Basically") + assert.Less(t, len(got), len(in)) +} + +func TestRuleEngine_AggressiveDedupsConsecutiveLines(t *testing.T) { + in := "do the thing\ndo the thing\nthen more\nthen more\nthen more\nunique" + got := RuleEngine{}.Compress(in, ModeAggressive) + assert.Equal(t, "do the thing\nthen more\nunique", got) +} + +func TestRuleEngine_PreservesCJK(t *testing.T) { + in := "你好 世界\t\t在 这里" + got := RuleEngine{}.Compress(in, ModeStandard) + // CJK has no English verbose phrases; only whitespace normalization applies + assert.Equal(t, "你好 世界 在 这里", got) +} + +func TestRewriteRequestBody_OpenAIStringContent(t *testing.T) { + body := []byte(`{"model":"gpt-4","messages":[` + + `{"role":"system","content":"You are helpful. Basically, in order to proceed, utilize the tool."},` + + `{"role":"user","content":"Hello world"}]}`) + + out, st, err := RewriteRequestBody(body, RuleEngine{}, Options{Mode: ModeStandard}) + require.NoError(t, err) + + sys := gjson.GetBytes(out, "messages.0.content").String() + assert.NotContains(t, sys, "utilize") + assert.NotContains(t, sys, "in order to") + assert.Contains(t, sys, "use the tool") + assert.NotContains(t, sys, "Basically") + + usr := gjson.GetBytes(out, "messages.1.content").String() + assert.Equal(t, "Hello world", usr) + + assert.Greater(t, st.SavedChars, 0) + assert.Greater(t, st.SavedPercent, 0.0) + assert.GreaterOrEqual(t, st.FieldsTouched, 1) + assert.Equal(t, "rule", st.Engine) + assert.Equal(t, ModeStandard, st.Mode) + // model + non-text fields preserved + assert.Equal(t, "gpt-4", gjson.GetBytes(out, "model").String()) +} + +func TestRewriteRequestBody_OpenAIMultimodalArrayContent(t *testing.T) { + body := []byte(`{"messages":[{"role":"user","content":[` + + `{"type":"text","text":"Please note that we need to utilize the API."},` + + `{"type":"image_url","image_url":{"url":"https://x/y.png"}}]}]}`) + + out, _, err := RewriteRequestBody(body, RuleEngine{}, Options{Mode: ModeStandard}) + require.NoError(t, err) + + txt := gjson.GetBytes(out, "messages.0.content.0.text").String() + assert.NotContains(t, txt, "utilize") + assert.Contains(t, txt, "use the API") + // image part untouched + assert.Equal(t, "https://x/y.png", + gjson.GetBytes(out, "messages.0.content.1.image_url.url").String()) +} + +func TestRewriteRequestBody_AnthropicShape(t *testing.T) { + body := []byte(`{"system":"You are basically helpful.",` + + `"messages":[{"role":"user","content":[{"type":"text","text":"In order to test"}]}]}`) + + out, _, err := RewriteRequestBody(body, RuleEngine{}, Options{Mode: ModeStandard}) + require.NoError(t, err) + + assert.NotContains(t, gjson.GetBytes(out, "system").String(), "basically") + txt := gjson.GetBytes(out, "messages.0.content.0.text").String() + assert.Contains(t, txt, "to test") + assert.NotContains(t, txt, "In order to") +} + +func TestRewriteRequestBody_PreservesCodeBlocks(t *testing.T) { + content := "Explain:\n```go\nfunc in order to() {}\n```\nBasically done." + body, err := json.Marshal(map[string]any{ + "messages": []map[string]any{ + {"role": "user", "content": content}, + }, + }) + require.NoError(t, err) + + out, _, err := RewriteRequestBody(body, RuleEngine{}, Options{Mode: ModeStandard}) + require.NoError(t, err) + + got := gjson.GetBytes(out, "messages.0.content").String() + // fenced code untouched + assert.Contains(t, got, "func in order to() {}") + // prose filler removed + assert.NotContains(t, got, "Basically") +} + +func TestRewriteRequestBody_ModeOffReturnsBodyUnchanged(t *testing.T) { + body := []byte(`{"messages":[{"role":"user","content":"utilize basically"}]}`) + out, st, err := RewriteRequestBody(body, RuleEngine{}, Options{Mode: ModeOff}) + require.NoError(t, err) + assert.Equal(t, body, out) + assert.Equal(t, 0, st.FieldsTouched) + assert.Equal(t, 0, st.SavedChars) +} + +func TestRewriteRequestBody_NilEngineIsNoOp(t *testing.T) { + body := []byte(`{"messages":[{"role":"user","content":"x"}]}`) + out, st, err := RewriteRequestBody(body, nil, Options{Mode: ModeStandard}) + require.NoError(t, err) + assert.Equal(t, body, out) + assert.Equal(t, 0, st.FieldsTouched) +} + +func TestRewriteRequestBody_InvalidJSONReturnsError(t *testing.T) { + body := []byte(`{not json`) + out, _, err := RewriteRequestBody(body, RuleEngine{}, Options{Mode: ModeStandard}) + require.Error(t, err) + assert.Equal(t, body, out) +} + +func TestRewriteRequestBody_UnknownShapePassesThrough(t *testing.T) { + body := []byte(`{"foo":"bar","baz":[1,2,3]}`) + out, st, err := RewriteRequestBody(body, RuleEngine{}, Options{Mode: ModeStandard}) + require.NoError(t, err) + assert.Equal(t, body, out) + assert.Equal(t, 0, st.FieldsTouched) +} + +func TestRewriteRequestBody_LiteEngineOnlyNormalizes(t *testing.T) { + body := []byte(`{"messages":[{"role":"user","content":"hello world"}]}`) + out, _, err := RewriteRequestBody(body, LiteEngine{}, Options{Mode: ModeLite}) + require.NoError(t, err) + assert.Equal(t, "hello world", + gjson.GetBytes(out, "messages.0.content").String()) +} + +func TestOptions_WithDefaults(t *testing.T) { + o := Options{}.WithDefaults() + assert.Equal(t, ModeStandard, o.Mode) + assert.Equal(t, 200_000, o.MaxInputChars) + + o2 := Options{Mode: ModeAggressive, MaxInputChars: 50}.WithDefaults() + assert.Equal(t, ModeAggressive, o2.Mode) + assert.Equal(t, 50, o2.MaxInputChars) +} + +func TestRewriteRequestBody_MaxInputCharsSkipsOversizedField(t *testing.T) { + big := "utilize basically " + for len(big) < 100 { + big += "utilize basically " + } + body := []byte(`{"messages":[{"role":"user","content":"` + big + `"}]}`) + out, st, err := RewriteRequestBody(body, RuleEngine{}, + Options{Mode: ModeStandard, MaxInputChars: 10}) + require.NoError(t, err) + // oversized field untouched + assert.Contains(t, gjson.GetBytes(out, "messages.0.content").String(), "utilize") + assert.Equal(t, 0, st.FieldsTouched) +} diff --git a/backend/internal/service/compression/rule_engine.go b/backend/internal/service/compression/rule_engine.go new file mode 100644 index 0000000..7ecaa23 --- /dev/null +++ b/backend/internal/service/compression/rule_engine.go @@ -0,0 +1,265 @@ +package compression + +import ( + "regexp" + "strings" +) + +// --------------------------------------------------------------------------- +// Fence helpers (code-block protection). A "fence" is a ``` delimiter. +// --------------------------------------------------------------------------- + +func containsFence(s string) bool { + return strings.Contains(s, "```") +} + +// splitFences splits s into alternating [prose, code, prose, code, ...] +// segments. The "```" delimiters are removed; callers re-add them. An +// unmatched trailing fence leaves the final segment as prose (lenient). +func splitFences(s string) []string { + // strings.Split on "```" already yields alternating prose/code/prose... + // when the fence count is even (balanced). For odd counts the last + // segment is still returned; applyToProse treats even indices as prose. + return strings.Split(s, "```") +} + +// --------------------------------------------------------------------------- +// LiteEngine — whitespace / separator normalization only. Safe for all +// languages including CJK. Target: ~10-15% on verbose prompts. +// --------------------------------------------------------------------------- + +// LiteEngine is the zero-dependency baseline compressor. +type LiteEngine struct{} + +func (LiteEngine) Name() string { return "lite" } + +func (LiteEngine) Compress(text string, _ Mode) string { + return normalizeWhitespace(text) +} + +// normalizeWhitespace collapses runs of spaces/tabs, trims trailing +// per-line space, and reduces 3+ blank lines to a single blank line. +// Newlines are preserved (they carry paragraph structure). +func normalizeWhitespace(s string) string { + if s == "" { + return s + } + lines := strings.Split(s, "\n") + for i, ln := range lines { + // trim trailing whitespace; collapse internal runs of spaces/tabs + ln = strings.TrimRight(ln, " \t\r") + ln = collapseSpaces(ln) + lines[i] = ln + } + out := strings.Join(lines, "\n") + // collapse 3+ newlines to 2 (one blank line) + for strings.Contains(out, "\n\n\n") { + out = strings.ReplaceAll(out, "\n\n\n", "\n\n") + } + return strings.TrimRight(out, "\n") + tailNewlines(s) +} + +// tailNewlines preserves a single trailing newline if the original had one, +// so we don't change "ends with \n" semantics for tools that rely on it. +func tailNewlines(orig string) string { + if strings.HasSuffix(orig, "\n") { + return "\n" + } + return "" +} + +var spaceRun = regexp.MustCompile(`[ \t]{2,}`) + +func collapseSpaces(s string) string { + return spaceRun.ReplaceAllString(s, " ") +} + +// --------------------------------------------------------------------------- +// RuleEngine — Caveman-class rule-based prose compression. Applies a curated +// table of safe verbose→concise substitutions (English-focused; CJK falls +// through to whitespace normalization only). Target: ~30-50% (standard), +// ~50-70% (aggressive, adds redundancy trimming). +// --------------------------------------------------------------------------- + +// RuleEngine is the rule-based compressor. The zero value is ready to use. +type RuleEngine struct{} + +func (RuleEngine) Name() string { return "rule" } + +func (e RuleEngine) Compress(text string, mode Mode) string { + if text == "" { + return text + } + out := normalizeWhitespace(text) + if mode == ModeLite { + return out + } + out = applyRules(out) + out = normalizeWhitespace(out) + if mode == ModeAggressive { + out = trimRedundancy(out) + } + return out +} + +type rule struct { + re *regexp.Regexp + repl string +} + +// ruleTable is compiled once at package init. Each entry rewrites a verbose +// phrase to a concise equivalent. Rules are case-insensitive and +// word-boundary anchored so "in order to" won't touch "border town". +// Deletion rules (repl == "") remove filler phrases. +var ruleTable = func() []rule { + phrases := []struct { + phrase string + repl string + }{ + // verbose connectors / prepositions + {"in order to", "to"}, + {"due to the fact that", "because"}, + {"owing to the fact that", "because"}, + {"in the event that", "if"}, + {"in spite of the fact that", "although"}, + {"regardless of the fact that", "although"}, + {"with regard to", "about"}, + {"with reference to", "about"}, + {"with respect to", "about"}, + {"in connection with", "about"}, + {"in relation to", "about"}, + {"in terms of", "for"}, + {"for the purpose of", "for"}, + {"for the reason that", "because"}, + {"by means of", "by"}, + {"by virtue of", "by"}, + {"in lieu of", "instead of"}, + {"in the absence of", "without"}, + {"prior to", "before"}, + {"subsequent to", "after"}, + {"in the vicinity of", "near"}, + {"as per", "per"}, + {"in accordance with", "per"}, + {"in light of", "given"}, + {"on behalf of", "for"}, + + // filler openers (deleted) + {"it is worth noting that", ""}, + {"it should be noted that", ""}, + {"it is important to note that", ""}, + {"please note that", ""}, + {"it goes without saying that", ""}, + {"needless to say", ""}, + {"as a matter of fact", ""}, + {"as previously mentioned", ""}, + {"as mentioned earlier", ""}, + {"as mentioned above", ""}, + {"in other words", ""}, + {"that is to say", ""}, + {"to put it another way", ""}, + {"in the process of", ""}, + + // verbose quantifiers + {"a large number of", "many"}, + {"a great deal of", "much"}, + {"a majority of", "most"}, + {"the majority of", "most"}, + {"a number of", "several"}, + {"an adequate number of", "enough"}, + {"in the near future", "soon"}, + {"at this point in time", "now"}, + {"at the present time", "now"}, + {"at this moment in time", "now"}, + {"until such time as", "until"}, + {"on a regular basis", "regularly"}, + {"on a daily basis", "daily"}, + {"on a weekly basis", "weekly"}, + + // verbose verbs + {"is able to", "can"}, + {"are able to", "can"}, + {"was able to", "could"}, + {"were able to", "could"}, + {"has the ability to", "can"}, + {"have the ability to", "can"}, + {"make a decision", "decide"}, + {"take into consideration", "consider"}, + {"give consideration to", "consider"}, + {"come to a conclusion", "conclude"}, + {"perform an analysis of", "analyze"}, + {"conduct an investigation of", "investigate"}, + {"bring to a conclusion", "finish"}, + {"make use of", "use"}, + {"take advantage of", "use"}, + + // inflated vocabulary + {"utilize", "use"}, + {"utilized", "used"}, + {"utilization", "use"}, + {"demonstrate", "show"}, + {"demonstrates", "shows"}, + {"approximately", "about"}, + {"facilitate", "help"}, + {"terminate", "end"}, + {"endeavor", "try"}, + {"ascertain", "learn"}, + {"initiate", "start"}, + {"remuneration", "pay"}, + {"subsequently", "then"}, + {"nevertheless", "but"}, + + // weak intensifiers / hedges (deleted) + {"basically", ""}, + {"essentially", ""}, + {"actually", ""}, + {"really", ""}, + {"very", ""}, + {"quite", ""}, + {"just", ""}, + {"currently", ""}, + {"presently", ""}, + } + out := make([]rule, 0, len(phrases)) + for _, p := range phrases { + // \b on CJK is harmless (no-op); English phrases are the real targets. + re := regexp.MustCompile(`(?i)\b` + regexp.QuoteMeta(p.phrase) + `\b`) + out = append(out, rule{re: re, repl: p.repl}) + } + return out +}() + +// applyRules runs the rule table over s. Deletion rules leave a single space +// in place of the phrase; a final normalizeWhitespace pass tidies up. +func applyRules(s string) string { + for _, r := range ruleTable { + if r.repl == "" { + // delete phrase + collapse one adjacent space to avoid doubles + s = r.re.ReplaceAllString(s, " ") + } else { + s = r.re.ReplaceAllString(s, r.repl) + } + } + return s +} + +// trimRedundancy is the aggressive-mode pass: drops empty lines and removes a +// line when it duplicates the previous non-blank line (case-insensitive, +// whitespace-normalized compare). Conservative — only exact-ish dupes. +func trimRedundancy(s string) string { + lines := strings.Split(s, "\n") + out := make([]string, 0, len(lines)) + var prev string + for _, ln := range lines { + t := strings.TrimSpace(ln) + if t == "" { + continue + } + fold := strings.ToLower(collapseSpaces(t)) + if fold == prev { + continue + } + out = append(out, ln) + prev = fold + } + return strings.Join(out, "\n") +} diff --git a/backend/internal/service/freellm/pool.go b/backend/internal/service/freellm/pool.go new file mode 100644 index 0000000..90a6bdf --- /dev/null +++ b/backend/internal/service/freellm/pool.go @@ -0,0 +1,194 @@ +// Package freellm holds the curated FreeLLM provider pool for the TOPGO +// Gateway — the enterprise counterpart to OmniRoute's "90+ free / 11 +// permanently free" catalog. +// +// This is a SEED CATALOG, not a live client. It is consumed by the +// Orchestrator-side provisioning flow (planned) to insert rows into the +// Gateway's existing `channel` table, tagged as free-tier and health-monitored +// by the existing channel_monitor infrastructure +// (see backend/ent/channelmonitor + backend/internal/service/channel.go). +// +// Auth model: only Pollinations is truly keyless. Every other entry is a +// free TIER (rate-limited but no monthly fee) or signup-credits provider — +// the operator must supply their own credential when enabling the channel. +// This matches the enterprise posture: free sources stretch the budget, they +// do not replace credential governance. +// +// Accuracy note: base URLs and model ids reflect each provider's public +// OpenAI/Anthropic-compatible endpoint at time of writing. Providers change +// these; channel_monitor's request-template probing will detect drift. Treat +// this list as a starting catalog, not a contract. +package freellm + +// AuthScheme describes how a channel authenticates upstream. +type AuthScheme string + +const ( + AuthNone AuthScheme = "none" // no credential required + AuthBearer AuthScheme = "bearer" // Authorization: Bearer + AuthXAPIKey AuthScheme = "x-api-key" // x-api-key header (Anthropic-style) +) + +// Protocol is the wire format the upstream speaks. +type Protocol string + +const ( + ProtoOpenAI Protocol = "openai" // /v1/chat/completions + ProtoOpenAIResponses Protocol = "openai-responses" // /v1/responses + ProtoAnthropic Protocol = "anthropic" // /v1/messages +) + +// FreeTier classifies the free-ness of a provider. +type FreeTier string + +const ( + TierUncapped FreeTier = "uncapped" // permanently free, no hard cap (rare) + TierFree FreeTier = "free-tier" // free with rate/usage limits + TierSignupCredits FreeTier = "signup-credits" // one-time free credit grant +) + +// Provider describes one seedable free upstream. +type Provider struct { + ID string `json:"id"` + Name string `json:"name"` + BaseURL string `json:"base_url"` + AuthScheme AuthScheme `json:"auth_scheme"` + Protocol Protocol `json:"protocol"` + FreeTier FreeTier `json:"free_tier"` + Models []string `json:"models"` + RateLimitRPM int `json:"rate_limit_rpm,omitempty"` // 0 = unknown + DocsURL string `json:"docs_url"` + Notes string `json:"notes,omitempty"` +} + +// SeedCatalog returns the curated default free-provider pool. +// +// Selection criteria: a real public OpenAI/Anthropic-compatible endpoint, +// a genuine free tier (not just a trial), and broad model coverage. Ordered +// keyless-first so a fresh install gets *something* working with zero +// credentials before the operator adds keyed channels. +func SeedCatalog() []Provider { + return []Provider{ + { + ID: "pollinations", + Name: "Pollinations AI", + BaseURL: "https://text.pollinations.ai/openai", + AuthScheme: AuthNone, + Protocol: ProtoOpenAI, + FreeTier: TierUncapped, + Models: []string{"openai"}, + DocsURL: "https://pollinations.ai", + Notes: "Keyless. Community-hosted; expect variable latency.", + }, + { + ID: "groq", + Name: "Groq", + BaseURL: "https://api.groq.com/openai/v1", + AuthScheme: AuthBearer, + Protocol: ProtoOpenAI, + FreeTier: TierFree, + Models: []string{"llama-3.3-70b-versatile", "llama-3.1-8b-instant"}, + RateLimitRPM: 30, + DocsURL: "https://console.groq.com/docs", + }, + { + ID: "google-gemini", + Name: "Google AI Studio (Gemini)", + BaseURL: "https://generativelanguage.googleapis.com/v1beta/openai", + AuthScheme: AuthBearer, + Protocol: ProtoOpenAI, + FreeTier: TierFree, + Models: []string{"gemini-1.5-flash", "gemini-2.0-flash"}, + DocsURL: "https://ai.google.dev/gemini-api/docs/openai", + Notes: "Generous free tier; OpenAI-compatible endpoint.", + }, + { + ID: "cerebras", + Name: "Cerebras", + BaseURL: "https://api.cerebras.ai/v1", + AuthScheme: AuthBearer, + Protocol: ProtoOpenAI, + FreeTier: TierFree, + Models: []string{"llama3.1-8b", "llama-3.3-70b"}, + RateLimitRPM: 30, + DocsURL: "https://inference.cerebras.ai", + Notes: "Fastest inference latency in the pool.", + }, + { + ID: "sambanova", + Name: "SambaNova", + BaseURL: "https://api.sambanova.ai/v1", + AuthScheme: AuthBearer, + Protocol: ProtoOpenAI, + FreeTier: TierFree, + Models: []string{"Meta-Llama-3.1-8B-Instruct", "Meta-Llama-3.1-70B-Instruct"}, + DocsURL: "https://docs.sambanova.ai/cloud/docs/get-started/supported-models", + }, + { + ID: "deepseek", + Name: "DeepSeek", + BaseURL: "https://api.deepseek.com/v1", + AuthScheme: AuthBearer, + Protocol: ProtoOpenAI, + FreeTier: TierSignupCredits, + Models: []string{"deepseek-chat", "deepseek-reasoner"}, + DocsURL: "https://api-docs.deepseek.com", + }, + { + ID: "mistral", + Name: "Mistral La Plateforme", + BaseURL: "https://api.mistral.ai/v1", + AuthScheme: AuthBearer, + Protocol: ProtoOpenAI, + FreeTier: TierFree, + Models: []string{"mistral-small-latest", "open-mistral-nemo"}, + DocsURL: "https://docs.mistral.ai", + }, + { + ID: "together", + Name: "Together AI", + BaseURL: "https://api.together.xyz/v1", + AuthScheme: AuthBearer, + Protocol: ProtoOpenAI, + FreeTier: TierSignupCredits, + Models: []string{"meta-llama/Llama-3.3-70B-Instruct-Turbo-Free"}, + DocsURL: "https://docs.together.ai", + Notes: "Free signup credits; several -Free model variants.", + }, + { + ID: "openrouter-free", + Name: "OpenRouter (free models)", + BaseURL: "https://openrouter.ai/api/v1", + AuthScheme: AuthBearer, + Protocol: ProtoOpenAI, + FreeTier: TierFree, + Models: []string{"free: meta-llama/llama-3.3-70b-instruct", "free: qwen/qwen-2.5-72b-instruct"}, + DocsURL: "https://openrouter.ai/docs", + Notes: "Aggregate of many providers; :free suffixed models are free.", + }, + { + ID: "kiro", + Name: "Kiro", + BaseURL: "https://prod.us-east-1.kiro.dev", + AuthScheme: AuthXAPIKey, + Protocol: ProtoAnthropic, + FreeTier: TierUncapped, + Models: []string{"claude-sonnet-4"}, + DocsURL: "https://kiro.dev", + Notes: "Anthropic-compatible; subject to provider availability.", + }, + } +} + +// Find returns the provider with the given id, or nil. +func Find(id string) *Provider { + for i := range catalog { + if catalog[i].ID == id { + return &catalog[i] + } + } + return nil +} + +// catalog is package-level so Find can scan it without re-allocating. +var catalog = SeedCatalog() diff --git a/backend/internal/service/freellm/pool_test.go b/backend/internal/service/freellm/pool_test.go new file mode 100644 index 0000000..27772a0 --- /dev/null +++ b/backend/internal/service/freellm/pool_test.go @@ -0,0 +1,29 @@ +package freellm + +import ( + "testing" + + "github.com/stretchr/testify/assert" +) + +func TestSeedCatalog_NonEmptyAndUnique(t *testing.T) { + got := SeedCatalog() + assert.Greater(t, len(got), 5) + seen := map[string]bool{} + for _, p := range got { + assert.False(t, seen[p.ID], "duplicate provider id: %s", p.ID) + seen[p.ID] = true + assert.NotEmpty(t, p.BaseURL) + assert.NotEmpty(t, p.Name) + assert.NotEmpty(t, p.Protocol) + } + // first entry is the keyless one (zero-credential bootstrapping) + assert.Equal(t, AuthNone, got[0].AuthScheme) +} + +func TestFind(t *testing.T) { + p := Find("groq") + assert.NotNil(t, p) + assert.Equal(t, "Groq", p.Name) + assert.Nil(t, Find("does-not-exist")) +}