kabang-knowledge

KakaoBank public-document corpus and manually authored tau3-style banking knowledge tasks.

This repository is an independent research/hackathon project. It is not affiliated with KakaoBank. The source material is based on publicly accessible KakaoBank product, policy, FAQ, and terms documents.

What This Is

The goal is to build a Korean banking knowledge benchmark inspired by the banking_knowledge domain in tau3/tau2:

• collect KakaoBank public source documents
• convert selected product bundles into structured knowledge documents
• manually extract reviewed facts
• manually author task scenarios with hidden state, expected actions, and verifier assertions
• prepare the assets for a future executable tau3/tau2-compatible domain

Current Status

Area	Status
Selected products	27 KakaoBank products
Structured KB docs	207 tau3-style id/title/content JSON documents
Manual tasks	207 task JSON files
tau2 v0 task export	123 DB-delta candidate task JSON files; 4 future user-tool and 80 no-DB-delta tasks skipped
Manual reviews	207 task-level review docs
Product audits	27 product coverage audits
Runtime domain	Partial v0 runtime: empty db.json, deterministic read/write tool replay, DB-only evaluator, and OpenAI-compatible assistant runner for the 123 DB-delta tasks

The current task set is a reviewed design artifact, not yet a fully executable tau3/tau2 domain. Manual tasks now compile into a v0 tau2-style candidate task set with a top-level user_prompt, keeping only cases whose gold replay changes the business DB final hash. The v0 runtime can run assistant-only tool chaining against one OpenAI-compatible chat-completions endpoint and grade success by DB final-state equality. Retrieval supports tau3-style offline bm25, grep, and bm25_grep configs over the 207 exported knowledge documents. There is no user simulator in this v0 path.

Key Paths

data/markdown/selected27_bundles_clean/
  Cleaned markdown bundles for the 27 selected products.

data/structured/kakaobank_fact_db_v0.json
  Reviewed fact database.

data/structured/kakaobank_tau3_documents_v0/
  Exported tau3-style knowledge documents and INDEX.md.

data/tasks/kakaobank_manual_v0/
  Manually authored task JSON files.

data/tau2/domains/kakaobank_knowledge/
  v0 tau2-style candidate task export with user_prompt, per-task
  initialization_data, DB-only reward_basis, and an empty domain db.json.

data/tasks/kakaobank_manual_v0_reviews/
  Human-readable review notes for every task.

data/tasks/kakaobank_manual_v0_product_audits/
  Product-level coverage audits.

data/schema/kakaobank_action_verifier_state_v0.json
  State/action/verifier design schema.

src/tau3_bank_hackathon/kakaobank/
  Current Python models and validators.

Setup

uv sync --extra dev

Useful Commands

Validate the current artifacts:

uv run pytest -q
uv run ruff check .
uv run ruff format --check .

Build the selected product bundle catalog:

uv run tau3-bank-hackathon build-bundle-catalog \
  --bundle-root data/markdown/selected27_bundles_clean \
  --output data/structured/kakaobank_bundle_catalog.json

Build the reviewed fact DB and tau3-style documents:

uv run tau3-bank-hackathon build-fact-db-v0

Export v0 tau2-style tasks:

uv run tau3-bank-hackathon export-kakaobank-tau2-tasks --mode v0

Build the empty runtime DB fixture:

uv run tau3-bank-hackathon build-kakaobank-runtime-db

Replay all exported DB-delta tasks deterministically:

uv run tau3-bank-hackathon replay-kakaobank-tasks

Evaluate captured assistant tool calls for one task:

uv run tau3-bank-hackathon evaluate-kakaobank-actions \
  --task-id kb_manual_demand_deposit_clean_close_success \
  --actions-json candidate_actions.json

Run one task through an OpenAI API, vLLM, SGLang, or other OpenAI-compatible assistant endpoint:

uv run tau3-bank-hackathon run-kakaobank-task \
  --task-id kb_manual_demand_deposit_clean_close_success \
  --model served-model \
  --endpoint http://localhost:8000/v1 \
  --retrieval-config bm25_grep

Run the 123-task v0 set against the same assistant endpoint:

uv run tau3-bank-hackathon run-kakaobank-tasks \
  --model served-model \
  --endpoint http://localhost:8000/v1 \
  --retrieval-config bm25_grep \
  --output-json runs/kakaobank_v0_results.json

--endpoint may be either a full /chat/completions URL or an OpenAI-style /v1 base URL. OPENAI_API_KEY is used when present, but local compatible servers that do not require authentication work without it. --retrieval-config defaults to bm25_grep and also supports bm25 and grep; text embedding and terminal-use variants are intentionally excluded from v0. For fair retrieval comparisons, bm25 and grep search the same per-document markdown surface that terminal use would see: # title plus document content. KB chunk document IDs are kept human-readable without hash-only suffixes after a no-collision check.

Design Notes

This project follows the original tau3 banking_knowledge philosophy:

• knowledge documents and transactional state are separate
• required_documents are oracle metadata, not ordinary user-facing hints
• tasks should require policy retrieval plus hidden state reasoning
• success is evaluated through concrete actions, user tools, verifier assertions, and eventually DB replay, not just a plausible natural-language answer
• unsupported or unverifiable claims are excluded rather than forced into tasks

Next Work

1. Add a kakaobank_knowledge executable domain skeleton.
2. Harden deterministic assistant tools beyond the current v0 replay surface.
3. Run pilot validation with OpenAI API, vLLM, and SGLang endpoints.
4. Add future support for the 4 preserved user-tool tasks.
5. Package the v0 runner as a fuller tau2-compatible executable domain.