From 7c25a6610b3cc81c5b07234b925378531391cc06 Mon Sep 17 00:00:00 2001 From: "chloe(younghwa)" Date: Mon, 15 Jun 2026 00:25:15 +0900 Subject: [PATCH] [DOC] Split Data Q&A into DATA_QA.md, frame as inner/outer loop MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Extract the Data Q&A section from deep-insight-web/README.md into a dedicated DATA_QA.md, framed as the "inner loop" (fast Text2SQL chat) that complements the analysis "outer loop" (long-form multi-agent report) β€” mirroring the inner/outer loop framing used in the blog. - deep-insight-web/README.md: replace the long Data Q&A section with a short outer/inner loop comparison table + link; move the two Data Q&A-specific troubleshooting items (AccessDenied, Korean font β–‘β–‘β–‘) into DATA_QA.md, keeping shared infra troubleshooting in the README - deep-insight-web/DATA_QA.md (new): how it works, Text2SQL highlights, prompt caching measurements, env vars, local dev, inner-loop troubleshooting, and the Deep Insight Chat demo video - README.md: update the 2026/06 Latest News link to DATA_QA.md, add a Data Q&A demo entry in the Demo section, and note the two workflows under Web UI Deployment --- README.md | 9 ++- deep-insight-web/DATA_QA.md | 155 ++++++++++++++++++++++++++++++++++++ deep-insight-web/README.md | 146 +++++---------------------------- 3 files changed, 183 insertions(+), 127 deletions(-) create mode 100644 deep-insight-web/DATA_QA.md diff --git a/README.md b/README.md index d374315..82ef2eb 100644 --- a/README.md +++ b/README.md @@ -19,7 +19,7 @@ ## *Latest News* πŸ”₯ -- **[2026/06]** Data Q&A β€” natural-language chat over the uploaded CSV with DuckDB Text2SQL, sandboxed in-process (no data leaves your AWS account), Bedrock prompt caching for ~90% input-token savings on multi-turn dialogue, and an inline SQL editor so analysts can inspect, edit, and re-run the generated SQL without another LLM call. ([details](./deep-insight-web/README.md#data-qa)) +- **[2026/06]** Data Q&A β€” the **inner loop** to analysis's outer loop: natural-language chat over the uploaded CSV with DuckDB Text2SQL, sandboxed in-process (no data leaves your AWS account), Bedrock prompt caching for ~90% input-token savings on multi-turn dialogue, and an inline SQL editor so analysts can inspect, edit, and re-run the generated SQL without another LLM call. ([details](./deep-insight-web/DATA_QA.md)) - **[2026/05]** Citation integrity validation β€” every number cited in a report is re-verified and linked to its calculation via `[N]` markers, with an independent Auditor (DeepTRACE A/B/C/D) blocking unverified or hallucinated figures (self-hosted + managed) ([details](./docs/features/data-validation/citation-validation-explained.md)) - **[2026/04]** Auto-generate sample analysis prompts β€” AI generates 3 sample prompts (간단/쀑간/볡작 β€” simple/medium/complex) from your column-definitions JSON, replacing the previous hard-coded fallback chips. Each generated prompt references actual column names so it's immediately runnable. ([details](./docs/features/prompt-generation/README.md)) - **[2026/04]** Auto-generate column definitions β€” AI builds `column_definitions.json` from a CSV header + sample rows, removing the manual JSON-authoring barrier for non-technical users. Includes a Table/JSON review panel with manual edit support. ([details](./docs/features/json-generation/README.md)) @@ -96,6 +96,7 @@ Production deployment using AWS Bedrock AgentCore Runtime with VPC Private Mode: Browser-based interface for non-technical users: - βœ… Upload data, review plans, download reports from the browser +- βœ… Two workflows: **outer loop** (long-form multi-agent analysis) + **inner loop** ([Data Q&A](./deep-insight-web/DATA_QA.md) β€” fast Text2SQL chat on the same CSV) - βœ… Korean/English language support - βœ… Two deployment options: VPN CIDR (internal) or CloudFront + Cognito (external) - βœ… Optional: Ops dashboard for job tracking and email notifications @@ -126,6 +127,12 @@ Browser-based interface for non-technical users: [▢️ Watch Full Demo on YouTube](https://www.youtube.com/watch?v=pn5aPfYSnp0) +### Data Q&A (Inner Loop) + +> Conversational, second-scale Text2SQL exploration on the uploaded CSV β€” the inner loop that complements the long-form analysis report. ([guide](./deep-insight-web/DATA_QA.md)) + +[▢️ Watch the Deep Insight Chat demo on YouTube](https://youtu.be/i5NoAIq7sDk?list=PLrAWXV_UoWzhwubfImuoz6GCHlZnrRdDc) + ### Sample Outputs πŸ“„ [English Report](./self-hosted/assets/report_en.docx) | πŸ“„ [Korean Report](./self-hosted/assets/report.docx) diff --git a/deep-insight-web/DATA_QA.md b/deep-insight-web/DATA_QA.md new file mode 100644 index 0000000..0dc8d6b --- /dev/null +++ b/deep-insight-web/DATA_QA.md @@ -0,0 +1,155 @@ +# Data Q&A (Inner Loop) + +> Conversational, second-scale exploration over the uploaded CSV β€” the **inner +> loop** that complements the long-form analysis report (the **outer loop**). + +Deep Insight runs two distinct workflows on the same uploaded data: + +| | Outer loop β€” Analysis | Inner loop β€” Data Q&A | +|---|---|---| +| **Endpoint** | `POST /analyze` | `POST /chat` | +| **Engine** | Hierarchical multi-agent (Coordinator β†’ Planner β†’ Supervisor β†’ tools) | Single Strands Agent + Text2SQL | +| **Latency** | 15–30 min, runs to completion | Seconds, iterative dialogue | +| **Path** | AgentCore Runtime | In-container DuckDB (**AgentCore not on this path**) | +| **Output** | Complete DOCX report with charts and citations | Inline table / chart per question | +| **User posture** | "Hand it off and wait" | "Poke at it directly" | + +The inner loop is for the questions you have *while* (or *before*) deciding what +the full report should cover: "how many rows?", "what's the top category by +revenue?", "show me the monthly trend". Fast, cheap, and fully transparent. + +Enabled automatically after upload β€” a "Data Q&A" tab appears next to "Analysis" +at the top of the page. + +[▢️ Watch the Deep Insight Chat demo on YouTube](https://youtu.be/i5NoAIq7sDk?list=PLrAWXV_UoWzhwubfImuoz6GCHlZnrRdDc) + +## How it works + +``` +Upload CSV --> DuckDB in-memory table (read_csv_auto) + | +User question --> Strands Agent + Claude Sonnet 4.6 + | + |-- query_sql(sql) -> table (SSE) + |-- create_chart(sql, matplotlib_code) -> PNG (SSE) + |-- describe_schema + v + Answer streamed back with SQL block + table/chart + | +User edits SQL --> POST /sql/execute (re-run without LLM, ms latency) +``` + +## Highlights + +- **Text2SQL with transparency**: Generated SQL is always shown in the chat bubble; collapsible by default +- **Inline SQL editor**: `β–Ά μ‹€ν–‰` re-runs as-is, `✎ νŽΈμ§‘` opens editor; no extra LLM call needed +- **Light-themed charts**: matplotlib with Korean font (NanumGothic), bold titles, click to zoom +- **Dynamic starter questions**: From schema heuristics β€” no LLM call on welcome load +- **Follow-up chips**: LLM inlines `[SUGGESTIONS]q1|q2|q3[/SUGGESTIONS]` at response end +- **Column-definitions aware**: Uses `column_definitions.json` (optional, auto-generated) in the system prompt for domain-accurate SQL +- **Insight-first Response Rules**: System prompt forces concrete numeric headlines and bans filler phrases ("κ²½ν–₯이 μžˆμŠ΅λ‹ˆλ‹€" etc.) + +## Prompt caching + +System prompt + tool specs + conversation history are cached via Bedrock +ephemeral cache using `SystemContentBlock + cachePoint` and +`CacheConfig(strategy="auto")`. TTL is 5 minutes, which is a Bedrock +platform default (not configurable); this matches the typical inter-turn +gap in an interactive Q&A session. + +Measured 5-turn session in production: + +| Turn | total input | cache_read | hit% | +|---|---|---|---| +| 1 | 5,938 | 2,653 | 44.7% | +| 2 | 12,243 | 10,822 | 88.4% | +| 3 | 10,963 | 9,760 | 89.0% | +| 4 | 13,048 | 12,112 | 92.8% | +| 5 | 15,281 | 14,038 | 91.9% | + +Per-turn input cost drops to ~10% on cache hits; session-level savings +typically 50%+ depending on conversation length (write premium grows with +history). + +Toggle via env var: `ENABLE_PROMPT_CACHE=0` disables caching (useful for +local debugging). + +### Observing prompt caching hits + +```bash +# Live tail of per-turn usage metrics +aws logs tail /ecs/deep-insight-web --region us-west-2 --follow | grep "chat usage" +``` + +Each chat turn logs `input=… output=… cache_read=… cache_write=…` (values are +cumulative within a task; diff consecutive lines for per-turn deltas). + +## Environment variables + +| Variable | Default | Purpose | +|----------|---------|---------| +| `CHAT_MODEL_ID` | `global.anthropic.claude-sonnet-4-6` | Chat agent LLM | +| `ENABLE_PROMPT_CACHE` | `1` | Set `0` to disable prompt caching | +| `WEB_UTILITY_MODEL_ID` | `global.anthropic.claude-sonnet-4-6` | Column/prompt auto-generation utility | + +## Local development + +Chat runs without S3 if `S3_BUCKET_NAME` is unset β€” uploaded CSVs go to +`/tmp/deep-insight-uploads/{upload_id}/` instead. Only Bedrock credentials +(`AWS_PROFILE` or env vars) are required. + +```bash +cd deep-insight-web +pip install -r requirements.txt +export AWS_REGION=us-west-2 # CHAT_MODEL_ID uses global. profile +python app.py +# localhost:8080 β†’ upload CSV β†’ "Data Q&A" tab β†’ ask +``` + +## Troubleshooting + +> Deployment/infra issues (image architecture, container startup, task cycling) +> are shared with the outer loop β€” see the +> [README Troubleshooting section](./README.md#troubleshooting). +> The items below are specific to the Data Q&A inner loop. + +### `AccessDeniedException: bedrock:InvokeModel` or `s3:ListBucket` + +The task role's inline policy is out of sync β€” happens when `deploy-cloudfront.sh` +or `deploy.sh` adds new Sids but the role already existed (older script +versions only created policy on role creation). + +Re-run the deploy script; it now always refreshes the inline policy: + +```bash +bash deploy-cloudfront.sh # or deploy.sh +``` + +Verify the policy contains the expected Sids: + +```bash +aws iam get-role-policy \ + --role-name deep-insight-web-task-role \ + --policy-name deep-insight-web-task-policy \ + --query 'PolicyDocument.Statement[].Sid' +# Expected: ["S3Upload", "S3UploadsList", "S3Feedback", +# "S3ArtifactsList", "S3ArtifactsGet", +# "AgentCoreInvoke", "BedrockInvokeClaude"] +``` + +### Chart labels render as β–‘β–‘β–‘ + +The container is missing Korean fonts. The Dockerfile installs `fontconfig + +fonts-nanum` before the `USER appuser` switch. If an older image was pushed +before this change, rebuild and redeploy: + +```bash +bash deploy-cloudfront.sh # or deploy.sh β€” forces new image build + push +``` + +Verify Nanum is present inside the container: + +```bash +docker run --rm 603420654815.dkr.ecr.us-west-2.amazonaws.com/deep-insight-web:latest \ + fc-list | grep -i nanum +``` diff --git a/deep-insight-web/README.md b/deep-insight-web/README.md index 61fe1aa..b1f8ce9 100644 --- a/deep-insight-web/README.md +++ b/deep-insight-web/README.md @@ -127,96 +127,27 @@ Both scripts handle all infrastructure in a single run: --- -## Data Q&A +## Two Workflows: Outer Loop & Inner Loop -Conversational exploration on the uploaded CSV. Complements the long-form -analysis flow (`/analyze`, 15–30 min) with fast, iterative questions. +Deep Insight runs two distinct workflows on the same uploaded data: -Enabled automatically after upload β€” a "Data Q&A" tab appears next to "Analysis" -at the top of the page. +| | Outer loop β€” **Analysis** | Inner loop β€” **Data Q&A** | +|---|---|---| +| **Endpoint** | `POST /analyze` | `POST /chat` | +| **Latency** | 15–30 min, runs to completion | Seconds, iterative dialogue | +| **Path** | AgentCore Runtime (multi-agent) | In-container DuckDB Text2SQL | +| **Output** | Complete DOCX report | Inline table / chart per question | +| **User posture** | "Hand it off and wait" | "Poke at it directly" | -### How it works +The **outer loop** is the long-form multi-agent analysis that produces a full +report. The **inner loop** (Data Q&A) is conversational, second-scale +exploration on the uploaded CSV β€” for the questions you have while (or before) +deciding what the full report should cover. It's enabled automatically after +upload, as a "Data Q&A" tab next to "Analysis". -``` -Upload CSV --> DuckDB in-memory table (read_csv_auto) - | -User question --> Strands Agent + Claude Sonnet 4.6 - | - |-- query_sql(sql) -> table (SSE) - |-- create_chart(sql, matplotlib_code) -> PNG (SSE) - |-- describe_schema - v - Answer streamed back with SQL block + table/chart - | -User edits SQL --> POST /sql/execute (re-run without LLM, ms latency) -``` - -### Highlights - -- **Text2SQL with transparency**: Generated SQL is always shown in the chat bubble; collapsible by default -- **Inline SQL editor**: `β–Ά μ‹€ν–‰` re-runs as-is, `✎ νŽΈμ§‘` opens editor; no extra LLM call needed -- **Light-themed charts**: matplotlib with Korean font (NanumGothic), bold titles, click to zoom -- **Dynamic starter questions**: From schema heuristics β€” no LLM call on welcome load -- **Follow-up chips**: LLM inlines `[SUGGESTIONS]q1|q2|q3[/SUGGESTIONS]` at response end -- **Column-definitions aware**: Uses `column_definitions.json` (optional, auto-generated) in the system prompt for domain-accurate SQL -- **Insight-first Response Rules**: System prompt forces concrete numeric headlines and bans filler phrases ("κ²½ν–₯이 μžˆμŠ΅λ‹ˆλ‹€" etc.) - -### Prompt caching - -System prompt + tool specs + conversation history are cached via Bedrock -ephemeral cache using `SystemContentBlock + cachePoint` and -`CacheConfig(strategy="auto")`. TTL is 5 minutes, which is a Bedrock -platform default (not configurable); this matches the typical inter-turn -gap in an interactive Q&A session. - -Measured 5-turn session in production: - -| Turn | total input | cache_read | hit% | -|---|---|---|---| -| 1 | 5,938 | 2,653 | 44.7% | -| 2 | 12,243 | 10,822 | 88.4% | -| 3 | 10,963 | 9,760 | 89.0% | -| 4 | 13,048 | 12,112 | 92.8% | -| 5 | 15,281 | 14,038 | 91.9% | - -Per-turn input cost drops to ~10% on cache hits; session-level savings -typically 50%+ depending on conversation length (write premium grows with -history). - -Toggle via env var: `ENABLE_PROMPT_CACHE=0` disables caching (useful for -local debugging). - -### Environment variables - -| Variable | Default | Purpose | -|----------|---------|---------| -| `CHAT_MODEL_ID` | `global.anthropic.claude-sonnet-4-6` | Chat agent LLM | -| `ENABLE_PROMPT_CACHE` | `1` | Set `0` to disable prompt caching | -| `WEB_UTILITY_MODEL_ID` | `global.anthropic.claude-sonnet-4-6` | Column/prompt auto-generation utility | - -### Observing prompt caching hits - -```bash -# Live tail of per-turn usage metrics -aws logs tail /ecs/deep-insight-web --region us-west-2 --follow | grep "chat usage" -``` - -Each chat turn logs `input=… output=… cache_read=… cache_write=…` (values are -cumulative within a task; diff consecutive lines for per-turn deltas). - -### Local development - -Chat runs without S3 if `S3_BUCKET_NAME` is unset β€” uploaded CSVs go to -`/tmp/deep-insight-uploads/{upload_id}/` instead. Only Bedrock credentials -(`AWS_PROFILE` or env vars) are required. - -```bash -cd deep-insight-web -pip install -r requirements.txt -export AWS_REGION=us-west-2 # CHAT_MODEL_ID uses global. profile -python app.py -# localhost:8080 β†’ upload CSV β†’ "Data Q&A" tab β†’ ask -``` +> πŸ“– **[Data Q&A (inner loop) β€” full guide β†’](./DATA_QA.md)** How it works, +> Text2SQL transparency, prompt caching measurements, env vars, local +> development, and inner-loop-specific troubleshooting. --- @@ -296,43 +227,6 @@ aws elbv2 describe-target-health \ --region us-west-2 ``` -### Data Q&A: `AccessDeniedException: bedrock:InvokeModel` or `s3:ListBucket` - -The task role's inline policy is out of sync β€” happens when `deploy-cloudfront.sh` -or `deploy.sh` adds new Sids but the role already existed (older script -versions only created policy on role creation). - -Re-run the deploy script; it now always refreshes the inline policy: - -```bash -bash deploy-cloudfront.sh # or deploy.sh -``` - -Verify the policy contains the expected Sids: - -```bash -aws iam get-role-policy \ - --role-name deep-insight-web-task-role \ - --policy-name deep-insight-web-task-policy \ - --query 'PolicyDocument.Statement[].Sid' -# Expected: ["S3Upload", "S3UploadsList", "S3Feedback", -# "S3ArtifactsList", "S3ArtifactsGet", -# "AgentCoreInvoke", "BedrockInvokeClaude"] -``` - -### Data Q&A: chart labels render as β–‘β–‘β–‘ - -The container is missing Korean fonts. The Dockerfile installs `fontconfig + -fonts-nanum` before the `USER appuser` switch. If an older image was pushed -before this change, rebuild and redeploy: - -```bash -bash deploy-cloudfront.sh # or deploy.sh β€” forces new image build + push -``` - -Verify Nanum is present inside the container: - -```bash -docker run --rm 603420654815.dkr.ecr.us-west-2.amazonaws.com/deep-insight-web:latest \ - fc-list | grep -i nanum -``` +> **Data Q&A (inner loop) issues** β€” `AccessDeniedException` on +> `bedrock:InvokeModel`/`s3:ListBucket`, or chart labels rendering as β–‘β–‘β–‘ β€” are +> covered in [DATA_QA.md β†’ Troubleshooting](./DATA_QA.md#troubleshooting).