[Epic][Security-MCP] - feat: automatic-migration skill#32

Draft

patrykkopycinski wants to merge 42 commits into

elastic:mainfrom

patrykkopycinski:ao/feat-automatic-migration-mcp-app

Commits on May 15, 2026

evals: add types.ts with Dataset, Example, EvalResult and related types

patrykkopycinski
and
claude
committed
ao(create-evals-types-ts-with-typescript-definitions--0): Create `evals/types.ts` with TypeScript definitions for `Dataset`, `Exam
patryks-treadmill
committed
evals: add runner.ts orchestrator, runMcpHostLoop stub, and eval vitest config

patrykkopycinski
and
claude
committed
evals: implement runMcpHostLoop with InMemoryTransport and LLM provider types

patrykkopycinski
and
claude
committed
evals: add OpenAiProvider with LiteLLM proxy support and wire default provider

patrykkopycinski
and
claude
committed
evals: add AnthropicProvider and wire it as the default when ANTHROPIC_API_KEY is set

patrykkopycinski
and
claude
committed
evals: add --reporter=verbose to test:evals script

patrykkopycinski
and
claude
committed
evals: add skill-activation evaluator (binary score)

patrykkopycinski
and
claude
committed
evals: add negative-activation evaluator for distractor examples

patrykkopycinski
and
claude
committed
evals: add tool-selection evaluator (precision/recall F1 against expected.tools)

patrykkopycinski
and
claude
committed
evals: add trajectory evaluator (LCS-based sequence score)

patrykkopycinski
and
claude
committed
evals: add criteria (LLM-as-judge) evaluator

patrykkopycinski
and
claude
committed
evals: add detection-rule-management dataset (4 positives + 4 distractors)

patrykkopycinski
and
claude
committed
evals: add detection-rule-management.eval.test.ts; split dataset from test orchestration

patrykkopycinski
and
claude
committed
ci: add evals.yml GitHub Actions workflow

patrykkopycinski
and
claude
committed
docs: add evals.md — harness design, dataset shape, evaluator catalog, CI gating

patrykkopycinski
and
claude
committed
feat: add MigrationsService wrapping 14 /internal/siem_migrations/* Kibana routes

patrykkopycinski
and
claude
committed
test: add MigrationsService tests covering all 14 route methods and error handling

patrykkopycinski
and
claude
committed
feat: register migration tools (1 model-facing + 10 app-only)

patrykkopycinski
and
claude
committed
test: add migration tool tests (tool registrations + vendor gating)

patrykkopycinski
and
claude
committed
feat: add migration workbench view with WorkbenchState machine

patrykkopycinski
and
claude
committed
feat: tighten vendor-select gate to use opacity-50 cursor-not-allowed

patrykkopycinski
and
claude
committed
feat: implement upload step with file input, drag-and-drop, and start-translation call

patrykkopycinski
and
claude
committed
feat: translating step now polls get-migration instead of get-stats

patrykkopycinski
and
claude
committed
feat: review step renders three-column diff (SPL | generated | editable Monaco)

patrykkopycinski
and
claude
committed
feat: per-rule drawer with ElasticRulePartial form and Re-validate button

patrykkopycinski
and
claude
committed
feat: fix-resources drawer with per-resource inline edit and unresolved highlighting

patrykkopycinski
and
claude
committed
feat: install step and done step with working back navigation

patrykkopycinski
and
claude
committed
feat: build migration view as singlefile HTML bundle (365 kB, < 1 MB)

patrykkopycinski
and
claude
committed
feat: add automatic-migration SKILL.md with lifecycle and gotchas

patrykkopycinski
and
claude
committed
feat: add automatic-migration eval dataset (6 positives + 6 distractors)

patrykkopycinski
and
claude
committed
feat: add automatic-migration eval spec (positives ≥80%, distractors 100%)

patrykkopycinski
and
claude
committed
feat: wire MigrationsService and registerMigrationTools into server

patrykkopycinski
and
claude
committed
chore: bump manifest to 1.1.0 and add migrate-rules tool entry

patrykkopycinski
and
claude
committed
docs: add SIEM Migration to README features table

patrykkopycinski
and
claude
committed
docs: add SIEM Migration section to features.md

patrykkopycinski
and
claude
committed
test(evals): add mock harness tests and fix eval suite robustness

patrykkopycinski
and
claude
committed
feat(evals): allow OPENAI_MODEL override for Ollama / LiteLLM proxies
patrykkopycinski
committed
fix(evals): hide app-only tools from the LLM in runMcpHostLoop
patrykkopycinski
committed
fix(evals): register all 7 model-facing tool groups in createEvalServer
patrykkopycinski
committed

Commits on May 18, 2026