Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 2 additions & 22 deletions .env.quickstart.example
Original file line number Diff line number Diff line change
Expand Up @@ -188,23 +188,11 @@ POSTHOG_ENABLED=false
# FIGMA_ACCESS_TOKEN=

# --- Rosetta / KnowledgeBase (project knowledge for codegen agents) ---
# Two modes — choose one:
#
# Mode 1: Bundled plugin (default, open-source quickstart)
# Generic KB, runs locally, no external credentials needed.
# Leave ROSETTA_MCP_ENABLED=false and keep the default ROSETTA_PLUGIN_PATH.
#
# Mode 2: Live MCP (project-tailored KB, Grid Dynamics managed)
# Connects to the Grid Dynamics IMS service for project-specific knowledge.
# Set ROSETTA_MCP_ENABLED=true and fill in the credentials below.
# Live MCP takes precedence when both are configured.
# The Rosetta knowledge base ships as a plugin baked into the Docker image. It is a generic KB,
# runs locally, and needs no external credentials.
#
# Docs: https://github.com/griddynamics/rosetta
#
# ROSETTA_MCP_ENABLED
# What: Toggle between bundled plugin (false) and live IMS-MCP server (true).
ROSETTA_MCP_ENABLED=false
#
# ROSETTA_PLUGIN_PATH
# What: Path to the generic KB plugin baked into the Docker image.
# How: Keep as-is. Only change if you mount a custom plugin.
Expand All @@ -215,14 +203,6 @@ ROSETTA_PLUGIN_PATH=/opt/rosetta-plugin
# the image at BUILD time (build arg, not a runtime setting). Pins the plugin version.
# How: Bump to a newer tag and rebuild the backend image to upgrade the bundled KB plugin.
ROSETTA_MARKETPLACE_REF=v2.0.55
#
# ROSETTA_API_KEY / ROSETTA_USER_EMAIL / ROSETTA_SERVER_URL
# What: Credentials for the live IMS-MCP Rosetta service.
# Only needed when ROSETTA_MCP_ENABLED=true.
# How: Obtain from your Grid Dynamics IMS admin.
# ROSETTA_API_KEY=
# ROSETTA_USER_EMAIL=
# ROSETTA_SERVER_URL=https://ims.evergreen.gcp.griddynamics.net/

# --- Agent MCP servers ---
# MCP_SERVERS_ENABLED
Expand Down
2 changes: 1 addition & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ Spec analysis and implementation planning happen **locally in the user's IDE** v

0. **Local (no backend)** — user calls `check_specification_completeness` and `run_planning` in their IDE; both produce markdown files in the user's project directory. Repeatable, free, no session created.
1. **File upload + contract validation** — `run_generation` uploads the user's `specs/`, optional `src/`, and `outputs_dir/` to the primary workspace. The contract validator (`backend/app/services/contract_validator.py` + `run_contract_validator` in `workflow_steps.py`) fuzzy-matches required files, runs keyword-only MCP prune, converts plans markdown→JSON, and writes Firestore plan data. If any required file is missing or unparseable, `run_generation` fails immediately with a human-readable message. No spec/planning/KB agents in this step (plan conversion agent only).
2. **KB init + Generation** — Rosetta unpacking and KB init run as the first generation step (not before). Then code generation runs across all workspaces in parallel, committing incrementally.
2. **KB init + Generation** — the provisioned Rosetta plugin initializes the knowledge base as the first generation step (not before). Then code generation runs across all workspaces in parallel, committing incrementally.
3. **Deploy & E2E** — deploy loop starts immediately after generation; no user pause in between.

Consequences that must be upheld in code:
Expand Down
19 changes: 13 additions & 6 deletions agents/IMPLEMENTATION.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,15 @@
# Implementation Status

**Updated**: June 23, 2026 | **Branch**: (current) | **Tests**: 1228+ passing (`make unit-tests`)
**Updated**: July 15, 2026 | **Branch**: (current) | **Tests**: 1228+ passing (`make unit-tests`)

## Current Work (July 2026)

- **Rosetta plugin-only cleanup (implementation stage):** Runtime KB initialization now validates
only `ROSETTA_PLUGIN_PATH`, uses the ordinary empty KB-init MCP selection, and supplies the
provisioned plugin's `Skill` / `SlashCommand` tools directly. Prompts and runtime configuration
now describe one direct-to-final-location workflow. Production implementation validation:
changed-file Ruff, Python compilation, residual source/doc searches, and `git diff --check`
pass. Tests remain in the approved workflow.

## Core Systems (Production Ready)

Expand All @@ -11,7 +20,7 @@
- **Database**: `backend/app/database/` — firestore, emulator, in_memory (IDatabase interface)
- **Prompts**: `backend/app/prompts/agents_claude_code.py` — all agent templates
- **Schemas**: `backend/app/schemas/` — estimation_enums (statuses/checkpoints), specification (SpecReadiness)
- **Config**: `backend/app/core/config.py` — env vars including `TOKEN_ENCRYPTION_KEY`, `GITHUB_TOKEN_DEFAULT`, `GIT_USER_NAME_DEFAULT`, K8s secret key name settings; Rosetta `ims-mcp` via `uvx ims-mcp@latest` with `ROSETTA_SERVER_URL`, `ROSETTA_API_KEY`, `ROSETTA_USER_EMAIL`, `ROSETTA_IMS_VERSION` → subprocess `VERSION` (`build_rosetta_mcp_config`); `github_platform_secrets.py` loads Fernet + default PAT (K8s API or env); `workspace_pool_names.py` — pool allowlist; `artifact_subdirs.py` — `ANALYSIS_SUBDIR`, `PLANNING_SUBDIR`, `REPORT_SUBDIR` (SSOT for artifact/workspace output paths); `backend/app/prompts/mcp_workflow_registry.py` — matrix of optional agent MCPs per workflow + prune keyword field map
- **Config**: `backend/app/core/config.py` — env vars including `TOKEN_ENCRYPTION_KEY`, `GITHUB_TOKEN_DEFAULT`, `GIT_USER_NAME_DEFAULT`, K8s secret key name settings; Rosetta KB via the bundled plugin at `ROSETTA_PLUGIN_PATH` (see `app/core/rosetta_kb.py` + `WorkspaceManager.provision_rosetta_plugin`); `github_platform_secrets.py` loads Fernet + default PAT (K8s API or env); `workspace_pool_names.py` — pool allowlist; `artifact_subdirs.py` — `ANALYSIS_SUBDIR`, `PLANNING_SUBDIR`, `REPORT_SUBDIR` (SSOT for artifact/workspace output paths); `backend/app/prompts/mcp_workflow_registry.py` — matrix of optional agent MCPs per workflow + prune keyword field map
- **MCP**: `server.py` — tools: check_specification_completeness, run_planning, run_generation, check_status, retry_generation, download_estimation_outputs; workspace tools take `ctx: Context` and call `apply_mcp_project_root_from_context` so `resolve_path` / session file resolve before `gain_session.json` exists (MCP `list_roots()` → Pydantic `FileUrl` → `_project_root`).

## Key Decisions (PR83)
Expand Down Expand Up @@ -86,8 +95,6 @@
`test_estimation` (`update_completed_estimation_result`), `test_report_generation` (skipped/coverage sections),
`test_workflow_integration`, `test_model_selection`.

- **Rosetta unpack in standalone planning + Rosetta MCP on codegen (Apr 9, ops v0.4.0 #3)**: `planning_workflow` calls `unpack_rosetta_artifacts` on the primary workspace after KB init (or KB_INIT_DONE resume) and before `run_planning_agent`, matching full-estimation behavior from `prepare_parallel_workspaces`. `coding_mcp_servers_and_tools` merges `_rosetta_pair` when `ROSETTA_MCP_ENABLED` so generation/deploy phase agents get KnowledgeBase MCP like planning/KB init. `unpack_rosetta_artifacts` also maps `rosetta/skills/` and `rosetta/commands/` to `.claude/skills/` and `.claude/commands/` (same pattern as `rosetta/agents/`). Tests: `test_agents_claude_code` (coding MCP + rosetta), `test_execute_all_phases_mcps` mocks set `ROSETTA_MCP_ENABLED=False` where isolating Playwright/Figma.

- **Async spec analyze + usage counters + milestone notifications (Apr 3)**: `POST /specification/analyze` returns immediately; background task awaits `archive_analysis` before clearing `spec_analysis_in_progress` and sending email/Slack `notify_spec_check_complete`. `GET /specification/outputs/{ws}` → **410 Gone**; MCP uses short analyze timeout and `download_outputs` only. Cumulative `num_turns` / `total_tokens_used` on estimation docs via transactional `add_agent_query_totals` from successful `agent_query` (TelemetryContext handler from `build_workflow_context`, spec/planning bg tasks, estimation run/retry). `GET .../estimations/{id}/status` + MCP `check_status` expose counters and `total_tokens_used_display` (`format_token_count`). `notify_planning_complete` after `archive_planning` + `PLANNING_DONE` in `planning_workflow`. Edge-case tests: `test_specification_analyze_async_edges.py` (API key lock, `begin_analysis` failure, workflow/archive failures, archive-before-notify ordering, notify failure cleanup), `test_get_estimation_status_includes_usage_and_spec_analysis_fields`. **TODO**: crash recovery could clear stale `spec_analysis_in_progress` if the process dies mid-task.

## Recently Completed (March 2026)
Expand All @@ -104,7 +111,7 @@
flows from MCP `server.py` / sync params → estimation `parameters` → workflows. Backend `SUPPORTED_MCPS`
(`playwright`, `figma`) filters names; `FIGMA_ACCESS_TOKEN` / `FIGMA_API_KEY` + `FIGMA_MCP_*` / `PLAYWRIGHT_MCP_*`
in backend env build stdio MCP configs. Playwright wired for generation + deploy/E2E phase agents; Figma for
spec analysis, planning (with Rosetta), generation, and deploy/E2E. PostHog event `mcp_servers_configuration`
spec analysis, planning, generation, and deploy/E2E. PostHog event `mcp_servers_configuration`
records `mcp_configuration_source` (form / estimation_parameters / backend_settings / workspace_sync_params),
`mcp_configuration_raw`, resolved enablement, and `mcp_supported_ids` at that moment. API form fields + tests in
`test_mcp_config.py`, `test_mcp_configuration_telemetry.py`.
Expand Down Expand Up @@ -132,7 +139,7 @@
- **Background jobs fix**: Removed PENDING/FAILED blocking from `cleanup_workspace()`. Fixed stuck_cleaning_recovery for missing timestamps.
- **Multi-model**: Comma-separated model lists in `LLM_MEDIUM`/`LLM_HIGH`/`LLM_LOW`, round-robin assignment, OpenRouter validation.
- **Integration readiness**: SpecReadiness enum, Part F, dual plans, DEPLOY_AND_E2E_DONE checkpoint, conditional workflow.
- **KB Init**: `ROSETTA_MCP_ENABLED` flag, kb_init workflow step, rosetta artifact unpacking.
- **KB Init**: `kb_init` workflow step (bundled Rosetta plugin — see **Config** above).
- **Post-refactor audit**: State machine enforcement, rogue DB writes removed, CI guards.

## Gaps & Debt
Expand Down
15 changes: 6 additions & 9 deletions backend/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -112,13 +112,11 @@ RUN mkdir -p /root/.claude
COPY claude-settings.json /root/.claude/settings.json

# ---------------------------------------------------------------------------
# Rosetta knowledge-base plugin — bundled so KB init runs WITHOUT the ims-mcp service /
# ROSETTA_API_KEY. Plugin mode is the DEFAULT in every environment (ROSETTA_MCP_ENABLED=false,
# ROSETTA_PLUGIN_PATH=/opt/rosetta-plugin). At workspace prep time the backend copies this
# plugin's agents/skills/commands/hooks into each workspace's .claude/ (see
# WorkspaceManager.provision_rosetta_plugin); the agents discover them via
# setting_sources=["project"]. Set ROSETTA_MCP_ENABLED=true to opt back into the live MCP,
# which then ignores this path.
# Rosetta knowledge-base plugin used by KB initialization.
# The plugin lives at ROSETTA_PLUGIN_PATH=/opt/rosetta-plugin in every environment. At workspace
# prep time the backend copies this plugin's agents/skills/commands/hooks into each workspace's
# .claude/ (see WorkspaceManager.provision_rosetta_plugin); the agents discover them via
# setting_sources=["project"].
#
# Installed from the public griddynamics/rosetta marketplace via the `claude` CLI that ships
# bundled inside claude-agent-sdk (a self-contained native binary at
Expand All @@ -133,8 +131,7 @@ COPY claude-settings.json /root/.claude/settings.json
# (falling back to the default below), so it can be bumped without editing this Dockerfile:
# docker build --build-arg ROSETTA_MARKETPLACE_REF=v2.0.56 ...
#
# The guard asserts the staged plugin actually ships the offline toolset (manifest + skills +
# workflows), failing the build if the marketplace ever serves an empty/MCP-only plugin.
# The guard asserts the installed plugin ships the required manifest, skills, and workflows.
# ---------------------------------------------------------------------------
ENV HOME=/root
ARG ROSETTA_MARKETPLACE_REF=v2.0.55
Expand Down
31 changes: 8 additions & 23 deletions backend/app/core/config.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,6 @@
MCP_PLAYWRIGHT = "playwright"
MCP_FIGMA = "figma"
MCP_FIGMA_SERVER_KEY = "Figma" # figma-developer-mcp registers under this server name
ROSETTA_SERVER_KEY = "KnowledgeBase" # ims-mcp registers under this server name
SUPPORTED_MCPS: FrozenSet[str] = frozenset({MCP_PLAYWRIGHT, MCP_FIGMA})
MCP_SERVERS_ENABLED_ENV = "MCP_SERVERS_ENABLED"
MCP_SERVERS_ENABLED_DEFAULT = MCP_PLAYWRIGHT
Expand Down Expand Up @@ -295,28 +294,14 @@ def _empty_str_to_none_int(cls, v: object) -> object:
FIGMA_ACCESS_TOKEN: Optional[str] = None
FIGMA_API_KEY: Optional[str] = None

# KnowledgeBase/Rosetta MCP — matches `claude mcp add ... -- uvx ims-mcp@latest` env surface.
ROSETTA_MCP_COMMAND: str = "uvx"
ROSETTA_MCP_ARGS: str = "ims-mcp@latest"
ROSETTA_SERVER_URL: Optional[str] = "https://ims.evergreen.gcp.griddynamics.net/"
ROSETTA_API_KEY: Optional[str] = None
ROSETTA_USER_EMAIL: Optional[str] = None
# Passed to ims-mcp subprocess as env VERSION (e.g. r2).
ROSETTA_IMS_VERSION: str = "r2"
# Plugin mode is the DEFAULT for every environment (quickstart AND hosted): KB init
# runs against the bundled Rosetta plugin, no ims-mcp service / ROSETTA_API_KEY needed.
# Set ROSETTA_MCP_ENABLED=true only to opt back into the live ims-mcp server (it then
# wins over the plugin). See app/core/mcp_selection.py:for_kb_init.
ROSETTA_MCP_ENABLED: bool = False
ROSETTA_OUTPUT_DIR: str = "rosetta"
# Path to the Rosetta plugin bundled into the image at build time (backend/Dockerfile
# stages it here). When ROSETTA_MCP_ENABLED is False, WorkspaceManager.provision_rosetta_plugin
# copies this plugin's agents/skills/commands into each workspace's .claude/ and merges its
# hooks into .claude/settings.json so setting_sources=["project"] discovers them (no
# ~/.claude, no SDK plugins= loader). This same path is also exported as CLAUDE_PLUGIN_ROOT
# per agent (claude_code.setup_rosetta_plugin_env) so the merged hooks' ${CLAUDE_PLUGIN_ROOT}
# resolves to the read-only image plugin. Set to None / empty to disable plugin provisioning
# (KB init then no-ops unless MCP is enabled).
# Path to the Rosetta plugin bundled into the image at build time.
# WorkspaceManager.provision_rosetta_plugin copies this plugin's agents/skills/commands into
# each workspace's .claude/ and merges its hooks into .claude/settings.json so
# setting_sources=["project"] discovers them. This same path is exported as
# CLAUDE_PLUGIN_ROOT per agent
# (claude_code.setup_rosetta_plugin_env) so the merged hooks' ${CLAUDE_PLUGIN_ROOT}
# resolves to the read-only image plugin. Set to None / empty to disable plugin
# provisioning (KB init then no-ops).
ROSETTA_PLUGIN_PATH: Optional[str] = "/opt/rosetta-plugin"

# PostHog Telemetry Configuration
Expand Down
51 changes: 4 additions & 47 deletions backend/app/core/mcp_config.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,8 @@
import shlex
from typing import FrozenSet, List, Literal, Optional

from app.core.config import MCP_FIGMA, MCP_FIGMA_SERVER_KEY, MCP_PLAYWRIGHT, ROSETTA_SERVER_KEY, SUPPORTED_MCPS, Settings
from app.core.tool_usage import get_figma_mcp_tools, get_playwright_mcp_tools, get_rosetta_kb_tools
from app.core.config import MCP_FIGMA, MCP_FIGMA_SERVER_KEY, MCP_PLAYWRIGHT, SUPPORTED_MCPS, Settings
from app.core.tool_usage import get_figma_mcp_tools, get_playwright_mcp_tools
from app.prompts.mcp_workflow_registry import (
CODE_GENERATION_AND_DEPLOY_OPTIONAL_MCPS,
PLANNING_OPTIONAL_AGENT_MCPS,
Expand Down Expand Up @@ -205,12 +205,6 @@ def _figma_pair(settings: Settings, enabled_mcps: FrozenSet[str]) -> tuple[dict,
return config, (get_figma_mcp_tools() if config else [])


def _rosetta_pair(settings: Settings) -> tuple[dict, List[str]]:
"""Rosetta MCP config + tools (enablement controlled by settings.ROSETTA_MCP_ENABLED)."""
config = build_rosetta_mcp_config(settings)
return config, (get_rosetta_kb_tools() if config else [])


def _combine_pairs(*pairs: tuple[dict, List[str]]) -> tuple[dict, List[str]]:
"""Merge N (config_dict, tools_list) pairs into one."""
servers: dict = {}
Expand Down Expand Up @@ -246,14 +240,9 @@ def coding_mcp_servers_and_tools(
settings: Settings,
enabled_mcps: FrozenSet[str],
) -> tuple[dict, List[str]]:
"""Playwright + Figma + optional Rosetta — used by generation and deploy/QA phase agents.

Rosetta is appended when ``ROSETTA_MCP_ENABLED`` (same as planning/KB init); not part of
``SUPPORTED_MCPS`` / prune candidates.
"""
"""Playwright + Figma — used by generation and deploy/QA phase agents."""
allowed = enabled_mcps & CODE_GENERATION_AND_DEPLOY_OPTIONAL_MCPS
return _combine_pairs(
_rosetta_pair(settings),
_playwright_pair(settings, allowed),
_figma_pair(settings, allowed),
)
Expand All @@ -263,7 +252,7 @@ def planning_mcp_servers_and_tools(
settings: Settings,
enabled_mcps: FrozenSet[str],
) -> tuple[dict, List[str]]:
"""Figma only — used by planning agents. Rosetta is intentionally excluded from planning."""
"""Figma only — used by planning agents."""
allowed = enabled_mcps & PLANNING_OPTIONAL_AGENT_MCPS
return _combine_pairs(
_figma_pair(settings, allowed),
Expand All @@ -287,35 +276,3 @@ def mcp_prompt_hints(enabled_mcps: FrozenSet[str]) -> str:
if not lines:
return ""
return "\n## Available MCP Tools\n" + "\n".join(f"- {line}" for line in lines) + "\n"


def build_rosetta_mcp_config(settings: Settings) -> dict[str, dict]:
"""Build MCP server config dict for ClaudeAgentOptions.mcp_servers.

Returns an empty dict when the feature is disabled, which signals
callers to skip KB init entirely.
"""
if not settings.ROSETTA_MCP_ENABLED:
return {}

env: dict[str, str] = {}
server_url = (settings.ROSETTA_SERVER_URL or "").strip()
if server_url:
env["ROSETTA_SERVER_URL"] = server_url
rosetta_api_key = (settings.ROSETTA_API_KEY or "").strip()
if rosetta_api_key:
env["ROSETTA_API_KEY"] = rosetta_api_key
user_email = (settings.ROSETTA_USER_EMAIL or "").strip()
if user_email:
env["ROSETTA_USER_EMAIL"] = user_email
ims_version = settings.ROSETTA_IMS_VERSION.strip()
if ims_version:
env["VERSION"] = ims_version

return {
ROSETTA_SERVER_KEY: {
"command": settings.ROSETTA_MCP_COMMAND,
"args": settings.ROSETTA_MCP_ARGS.split(),
"env": env,
}
}
Loading
Loading