Dev < UI#339
Open
renan-souza wants to merge 51 commits into
Open
Conversation
Add OLCF Frontier example and LMDB/CLI improvements
…rds, and LLM chat
## Backend — FastAPI webservice (src/flowcept/webservice/)
New routers under /api/v1:
- campaigns: GET /campaigns, GET /campaigns/{id} — derived from workflow/task grouping
by campaign_id (no new collection; aggregation pipeline with Python fallback)
- agents: GET /agents, GET /agents/{id}, GET /agents/{id}/tasks — derived from tasks
grouped by agent_id / source_agent_id
- stats: GET /stats/tasks/summary, POST /stats/timeseries, POST /stats/card_data —
shared card-data resolver powering both dashboard cards and the chat make_chart tool
- dashboards: full CRUD for declarative JSON dashboard specs
(DashboardSpec/Card/CardData);
Mongo-backed (new dashboards collection) with FileDashboardStore fallback
- stream: GET /stream/tasks, GET /stream/workflows — SSE via sse-starlette, incremental
DB polling with float/datetime cursor normalization, ping keepalive, configurable batch
- chat: POST /chat — stateless LLM tool-calling loop (bind_tools + stream), emits
token/tool_call/tool_result/card/done SSE events; 503 when LLM deps absent
New webservice internals:
- services/stats.py: task_summary, derive_campaigns, derive_agents, telemetry_timeseries,
resolve_card_data (Mongo aggregation path + pandas/Python fallback)
- services/streaming.py: poll_new_docs with dual float/datetime cursor via $or;
_as_epoch() normalization
- services/chat_service.py: run_chat() generator; _build_langchain_tools() wrapping
prov_tools; fix for ruff F821 (name before assignment)
- services/dashboard_store.py: MongoDashboardStore + FileDashboardStore factory
- services/reports.py: provenance_card_response() shared by workflow/campaign routers
- schemas/dashboards.py: Pydantic DashboardSpec/Card/CardData/VizSpec/LayoutItem
- workflows router: added GET /{id}/provenance_card?format=json|markdown
## Shared provenance tool core (src/flowcept/agents/tools/)
- prov_tools.py: plain-Python tool functions (query_tasks, query_workflows,
get_task_summary, list_campaigns, list_agents, make_chart, get_dashboard,
update_dashboard) guarded by _guarded(); all imports hoisted to module top
- db_prov_tools.py: @mcp_flowcept.tool wrappers registering prov_tools on the MCP
server so external MCP clients (Claude Code, Codex) get real DB-backed querying;
replaces the historical_prov_query stub
- prompts/chat_prompts.py: CHAT_SYSTEM_PROMPT with task/workflow schema context,
operator allowlist, embedded DashboardSpec JSON schema, few-shot card examples
## MongoDBDAO additions (src/flowcept/commons/daos/docdb_dao/mongodb_dao.py)
- raw_pipeline(collection, pipeline): generic aggregation for any collection
- save_dashboard / get_dashboard / list_dashboards / delete_dashboard
- _dashboards_collection property
- Fixed inclusion-projection bug: timestamp:0 exclusion now only added when no
inclusion fields are specified (was causing "Cannot do exclusion on field timestamp
in inclusion projection" with non-empty projection lists)
## Agent / MQ consumer fixes
- flowcept_agent.py: daemon=True server thread; AppStatus.should_exit_event = None
reset before each new uvicorn server to avoid sse-starlette event-loop binding crash
when a second server starts in the same process
- base_agent_context_manager.py: self.start(daemon=True) so the MQ consumer never
blocks process exit in test runners
- agents/__init__.py: re-exports db_prov_tools symbols
## report/service.py
- Extracted public build_provenance_card() from generate_report() (non-breaking;
generate_report still calls it) so routers can return card content without writing
a file
## configs.py / sample_settings.yaml
New config keys: WEBSERVER_UI_ENABLED, WEBSERVER_CORS_ORIGINS,
WEBSERVER_SSE_POLL_INTERVAL, WEBSERVER_SSE_MAX_BATCH, WEBSERVER_DASHBOARDS_DIR,
WEBSERVER_CHAT; web_server section fully documented in sample_settings.yaml
## CLI (src/flowcept/cli.py)
- flowcept --start-webservice [--host --port]
- Fixed argparse duplicate-flag crash (added added_args set; start_agent_gui and
start_webservice both define --port)
## React SPA (ui/)
Stack: React 18 + TypeScript (strict), Vite, Tailwind CSS 4 dark theme, TanStack
Router/Query/Table/Virtual, ECharts (tree-shaken), react-grid-layout v2,
react-markdown, zustand, zod, @microsoft/fetch-event-source, openapi-typescript.
Built assets emit to src/flowcept/webservice/ui_build/ and ship in the wheel.
Key files:
- api/client.ts, types.ts, queries.ts, sse.ts (useEventStream with cursor resume,
backoff, tab-pause)
- lib/format.ts: toEpochSec() normalizes both float epoch-seconds and ISO datetime
strings (handles BSON datetime serialization from DocumentInserter)
- stores/chatStore.ts: zustand chat transcript + panel state
- components/charts/: EChart wrapper, GanttChart (custom series), StatusStrip,
TelemetryChart
- components/tables/DataTable.tsx: TanStack Virtual, div-based
- components/dashboard/: spec.ts (zod mirror of dashboards schema), specToOption.ts
(declarative CardData → ECharts option), CardRenderer.tsx
- components/chat/ChatPanel.tsx: fetchEventSource POST, renders text/tool_call/card
parts inline; card events render as ECharts with "pin to dashboard" action
- routes/: __root (AppShell + sidebar + chat panel slot), index (overview),
campaigns, workflows.$workflowId (tasks/timeline/telemetry/card/raw tabs),
tasks.$taskId, objects, agents, dashboards.$dashboardId (grid editor)
- ui/README.md: stack, code layout, running instructions, chat setup, MCP notes,
test pointers
react-grid-layout v2 API changes: no WidthProvider; use useContainerWidth hook;
props nested in gridConfig/dragConfig/resizeConfig; removed @types/react-grid-layout
(conflicts with v2 built-in types).
## pyproject.toml
- webservice extra: fastapi, uvicorn, pyyaml, sse-starlette
- wheel artifacts: ui_build/ → flowcept/webservice/ui_build
- flowcept[webservice] added to [all]
## Makefile
New targets: ui-install, ui-dev, ui-build, ui-checks, webservice
## CI
- .github/workflows/ui-checks.yml: Node 22, npm ci + tsc + eslint + vite build
(path-filtered on ui/**)
- create-release-n-publish.yml: Node setup + make ui-build before hatch build
## .gitignore
Added: ui/node_modules/, ui/dist/, ui/.vite/, src/flowcept/webservice/ui_build/
## OpenAPI
Regenerated flowcept-openapi.json / .yaml (43 paths, all new endpoints included);
added docs/openapi/scripts/generate_openapi.py
## Tests
- tests/webservice/test_webservice_integration.py: 11 new integration tests covering
campaigns/agents/stats/prov-card, object versioning, SSE tasks and workflows
(real uvicorn via _start_real_server() helper; httpx.stream()), SPA serving,
dashboards CRUD, shared prov_tools core, chat 503 path, real LLM tool round-trip
- tests/agent/agent_tests.py: test_mcp_db_backed_provenance_tools (real agent + real DB)
- tests/webservice/test_webservice_api.py: updated root endpoint assertion to accept
HTML (built UI present) or JSON (API-only)
- _start_real_server() resets AppStatus.should_exit_event = None before each uvicorn
start to prevent sse-starlette event-loop crash in multi-server test runs
Commit co-authored by Claude Fable + Sonnet 4.6
Main <- Dev
- React + TypeScript SPA (Vite) with TanStack Router/Query, ECharts, Tailwind - Views: overview, campaigns, workflows (tasks/graph/timeline/telemetry/card/artifacts/dashboard tabs), objects, agents, dashboards - Dashboard data model: workflow/campaign dashboards with declarative chart specs; configured via web_server.workflow_dashboard / campaign_dashboard in settings.yaml - New webservice routers: /campaigns, /agents, /stats (task summary, timeseries, chart_data), /dashboards CRUD, /stream (SSE), /chat (LLM tool-calling) - DAG view with BFS rank layout; Gantt chart; telemetry normalized overlay chart - Resizable 3-panel shell (sidebar / main / inspector) with bottom chat panel - Workflow Card and provenance card generation for workflows and campaigns - Rename card→chart throughout (dashboards schema, settings, frontend, prompts) - Fix DagView BFS rank bug; fix duration computation for ISO timestamps; filter unnamed/empty workflows - Fix OmegaConf serialization in configs.py (to_container at load time) - Add agent_sandbox/settings.yaml for local UI dev with MongoDB - Update .gitignore: add ui/src/routeTree.gen.ts, ui/docs/ Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
# Conflicts: # docs/openapi/flowcept-openapi.json # docs/openapi/flowcept-openapi.yaml # pyproject.toml # src/flowcept/report/service.py # src/flowcept/webservice/routers/workflows.py
Use workflow-card naming consistently in generated OpenAPI docs, align dashboard routes and schemas around the cards/card_id contract, and make workflow SSE polling use the available workflow timestamp fields.
# Conflicts: # tests/webservice/test_webservice_integration.py
- Fix workflow/campaign list ordering: sort_docs_by_first_date_field now
sorts descending (newest-first) so docs[:limit] returns the most recent
items across all list endpoints (workflows, tasks, objects, agents,
campaigns). Root cause: ascending sort caused today's runs to fall
outside the limit window.
- Auto-hide empty dashboard charts: replace useEffect+useRef pattern
(which broke under React Strict Mode double-mount) with inline
derivation. Charts with zero rows are hidden by default; user pill
toggles stored in userToggles override the auto-hide per chart.
Applied to both WorkflowDashboardTab and CampaignDashboardTab.
- Add collection_sizes chart source: new virtual source in stats.py
runs $bsonSize aggregations on tasks/objects/workflows collections and
returns per-collection byte totals. Added "Data per collection" as a
default workflow chart.
- Rename chart titles: "Data sizes by type" -> "Artifacts size by type";
loss-vs-epochs viz kind scatter -> line.
- Object deletion: DELETE /objects/{id} endpoint (uses delete_object_keys);
trash-bin button in objects list (group-hover pattern) and detail page;
breadcrumb link added to detail page.
- Click chart -> inspector: ChartRenderer pushes kind="chart" entity to
inspectorStore on click; __root.tsx renders a formatted data table in
the right panel.
- Docs: new docs/web_ui.rst Sphinx section (install, start, pages,
dashboards, chat, dev); added to index.rst toctree. Rewrote
ui/README.md with accurate ports, dashboard data model, stack table,
code layout, and two-file sync procedure for default configs.
- AGENTS.md: sections 14 (list ordering rule) and 15 (dashboard config
sync procedure) to prevent recurring mistakes.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Rename chat panel to "Flowcept Agent" with Bot icon; placeholder text is now scoped to the current workflow/campaign id - Add highlight_lineage provenance tool: resolves seed tasks via a generic Mongo-style filter, emits ui:highlight SSE event with task IDs - DataflowView and DagView consume a new highlightStore (Zustand): agent highlights dim all out-of-lineage nodes/edges; clicking a node or empty pane clears the agent highlight - Lineage BFS uses two separate passes (forward/backward) to avoid cross-contamination; frontend is the single source of truth for expansion — backend returns seed IDs only - Fix model_kwargs=null crash in build_llm_model (NoneType.copy) - Coerce LLM-emitted Mongo projection dicts and sort dicts to the list formats expected by query_tasks - Add agent LLM credentials (gpt-oss-120b) to agent_sandbox/settings.yaml - Update docs/agent.rst: document both agent surfaces (web chat vs MCP), streaming vs persisted query paths, and lineage highlighting examples - Update README: add Web UI section, Web UI feature bullet, and expand Querying & Monitoring table row Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… feature - Keep "Flowcept Agent" title + Bot icon from our branch - Keep isMaximized state, Maximize2/Minimize2 buttons, overlay backdrop, and containerClasses from the incoming branch - All useHighlightStore, contextHint, and ui_highlight rendering retained Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
feat: adding agent name and source agent name
…rce_agent_id in ui graphs and inspector
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.