diff --git a/CHANGELOG.md b/CHANGELOG.md index 5b9aa18..574d806 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,15 @@ # Changelog +## Unreleased + +Added: +- Multiple email accounts. List mailboxes under `email_accounts` in config.yml: Google-hosted addresses on any domain (via gog) and any other mailbox via IMAP. One run searches them all, and one failing account no longer stops the rest. IMAP passwords come from environment variables, never from config. +- Per-account setup checks. Onboarding and health_check verify gog auth for each Google account, warn when a gmail-configured domain is not actually Google-hosted, note the "External" OAuth client requirement for multi-domain setups, and login-test IMAP accounts. + +Changed: +- Message, thread, and attachment ids are account-scoped. The mailbox MCP tools (search_inbox, get_message, download_attachment, archive_thread, draft_reply) take an optional `account` parameter; it is only needed when several accounts are configured. +- Staged attachment downloads are namespaced by account as well as message, so same-named PDFs from different mailboxes cannot overwrite each other. + ## 0.1.1 Fixes and docs. No breaking changes. diff --git a/CLAUDE.md b/CLAUDE.md index 70662c5..9c72e80 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -23,7 +23,13 @@ pyproject.toml # dependencies + ruff config ./fetch invoices generate-payments ``` -## Gmail +## Email access + +Mailboxes are configured under `email_accounts` in `config.yml`. Each is served +by a `MailProvider` (`providers.py`): `GmailProvider` (`gmail.py`, gog CLI) for +Google-hosted addresses, `ImapProvider` (`imap.py`, imap-tools) for everything +else. Message/thread/attachment IDs are account-scoped — always go through the +provider the message came from (records carry an `account` field). When working **on this codebase**, route Gmail through the `gog gmail` CLI, not a Gmail MCP connector, so calls go through the audited `helpers.async_run` wrapper @@ -32,12 +38,14 @@ end users: a host agent (e.g. Cowork) may use its own Gmail connector for search / read / draft / archive. The one job that still needs `gog` is landing an attachment's bytes on disk (`gog gmail attachment`), since connectors return an attachment id but not its content, and the rest of the pipeline reads real files. +Pass `-a
` to select an account. ```bash -gog gmail search "is:unread" --json +gog -a you@example.com gmail search "is:unread" --json gog gmail get --json gog gmail thread get --full --json gog gmail attachment --out --name +gog auth list --json gog gmail -h ``` @@ -58,8 +66,9 @@ uvx ruff format src/ ## Key conventions -- Config values (companies, destination folders, debtor accounts) live in `config.yml`, not hard-coded +- Config values (companies, destination folders, debtor accounts, email accounts) live in `config.yml`, not hard-coded - `DEFAULT_SLUG` in `config.yml` is the fallback company slug +- IMAP passwords come from env vars (`imap.password_env`), never from config.yml - Classification uses `claude` CLI with haiku model (`classify.py`) - Models are plain dataclasses in `models.py` - All async subprocess calls go through `helpers.py` (`async_run`, `async_run_json`) diff --git a/README.md b/README.md index 5b2446c..32bb4c2 100644 --- a/README.md +++ b/README.md @@ -11,14 +11,14 @@ Small companies (and their agents) that get a periodic "please send the invoices ## What it does 1. Reads the accountant's list, in any format (pasted text, a table, an email). -2. Finds each invoice. First in your Gmail inbox. For anything not there, it hands your agent a clear task to fetch it from the vendor's portal or e-invoice platform. +2. Finds each invoice. First in your email (one mailbox or several, Gmail or IMAP). For anything not there, it hands your agent a clear task to fetch it from the vendor's portal or e-invoice platform. 3. Drafts a reply to the accountant with the found PDFs attached, plus a note on anything still missing. You review the draft and send it. It can also classify and file invoices, copy them to a folder of your choice (Dropbox, a shared drive, or any local folder), and build a SEPA payment file. See the tool list below. ## How it stays safe -- No stored credentials. It uses your local `gog` (Gmail) and `claude` CLIs. +- No stored credentials. It uses your local `gog` (Gmail) and `claude` CLIs. IMAP passwords come from environment variables, never from config. - It drafts, it does not send. Outgoing replies are Gmail drafts you review first. - Local audit log. Every fetch, download, and draft is recorded in `output/audit.log`. @@ -93,12 +93,12 @@ uv sync |------|--------------| | `health_check` | Check gog, claude, Gmail auth, and config.yml | | `parse_missing_list` | Turn a free-form missing list into a checklist | -| `search_inbox`, `get_message`, `download_attachment` | Find and pull invoice PDFs from Gmail | +| `search_inbox`, `get_message`, `download_attachment` | Find and pull invoice PDFs from your mailboxes | | `plan_retrieval` | Build tasks to fetch invoices that are not in the inbox | | `draft_reply` | Draft a reply to the accountant with files attached (never sends) | | `read_audit` | Read the local audit log | | `reconcile`, `build_report` | Optional: match the list to what you found and write a summary | -| `fetch_invoices`, `classify_invoice`, `list_invoices` | Fetch and classify invoices from Gmail | +| `fetch_invoices`, `classify_invoice`, `list_invoices` | Fetch and classify invoices from all your mailboxes | | `copy_to_dropbox`, `generate_payments`, `archive_thread` | File to a folder (Dropbox or any path), build SEPA payments, archive threads | Full reference and the step-by-step workflow: [SKILL.md](SKILL.md). @@ -136,6 +136,36 @@ token without your own Cloud project is Google's OAuth Playground. Either way, `health_check` tells you whether Gmail is connected. +## Multiple email accounts + +By default Fetch searches the Gmail account `gog` is signed in to. To search several mailboxes in one run, list them under `email_accounts` in `config.yml`: + +```yaml +email_accounts: + # Google-hosted mailboxes (gmail.com or any Workspace domain) + - address: you@example.com + - address: info@other-domain.com + + # Any other mailbox via IMAP + - address: user@elsewhere.com + provider: imap + imap: + host: imap.elsewhere.com + port: 993 # optional, default 993 + username: user@elsewhere.com # optional, defaults to address + password_env: ELSEWHERE_IMAP_PASSWORD + folder: INBOX # optional + archive_folder: Archive # optional, used when archiving +``` + +Google accounts need `gog auth add
` once each. The domain does not matter, only that the mailbox is hosted on Google. When accounts span more than one Workspace domain, create the OAuth client as "External" and add each address as a test user while the app is unverified. + +IMAP passwords are never stored in `config.yml`. Set `password_env` to the name of an environment variable and export it before running. For providers with two-factor auth, use an app password. + +Searches cover all accounts at once, and one failing account does not stop the others. Message and thread ids are scoped to the account they came from, so the MCP tools take an optional `account` parameter (only needed when several accounts are configured). Draft replies always go through a Google account. + +`./fetch onboarding` (or the `health_check` tool) checks all of this: gog auth for each Google account, whether each Gmail-configured domain is actually Google-hosted, and IMAP connectivity. + ## Use it inside Cowork (or any agent that already has Gmail) If your agent already has its own Gmail connector (for example Claude Cowork's @@ -198,6 +228,7 @@ debtor_accounts: bic: "LHVBEE22" vendor_sources: {} # optional, see "Fetch invoices that are not in your inbox" +email_accounts: [] # optional, see "Multiple email accounts" ``` Point at a config elsewhere with `FETCH_CONFIG`. Change the output folder with `FETCH_OUTPUT_DIR`. @@ -238,7 +269,9 @@ src/ reconcile.py Parse a missing list, match it, build a report retrieval.py Plan fetching invoices not in the inbox audit.py Local audit log - gmail.py Gmail search, download, draft reply (via gog) + providers.py Mailbox abstraction + multi-account search + gmail.py Google-hosted mailboxes (via gog), draft replies + imap.py Any other mailbox via IMAP classify.py AI classification (claude) payment_xml.py SEPA payment XML config.py, models.py, helpers.py, process.py, dropbox.py, summary.py diff --git a/SKILL.md b/SKILL.md index 9539eb3..cd20412 100644 --- a/SKILL.md +++ b/SKILL.md @@ -12,6 +12,10 @@ This server stores **no passwords or tokens**. It runs locally and shells out to - **`gog`**: your Gmail session (search, fetch attachments, draft replies). - **`claude`**: classifies each PDF (runs the `haiku` model locally). +Non-Google mailboxes connect over IMAP instead; the password comes from an +environment variable named in `config.yml` (`imap.password_env`), never from +the config file itself. + If those aren't set up, every tool can tell you: call `health_check` first. **If you already have your own Gmail connector** (e.g. running inside Cowork), use @@ -63,26 +67,32 @@ only Dropbox), and your debtor bank accounts. Verify with | Tool | Purpose | |------|---------| -| `health_check()` | Check gog CLI, claude CLI, Gmail auth, and config.yml. Returns `{ok, checks[]}`. Call this first. | -| `search_inbox(query?, max_results?)` | Raw Gmail search, returns `{query, count, messages[{id, threadId}]}`. `query` defaults to the built-in invoice search. | -| `get_message(message_id)` | One message's headers plus `attachments[{attachment_id, filename, mime_type}]`. Use it to pick which PDF to pull. | -| `download_attachment(message_id, attachment_id, filename)` | Download one attachment to staging, returns `{path, filename, exists}`. | -| `fetch_invoices(max_emails?, query?)` | The full pipeline: search Gmail, download PDFs, classify, dedupe, file under `output/`, write `invoices.json` and `SUMMARY.md`. Returns `{count, invoices[], output_dir, summary_md, invoices_json}`. | +| `health_check()` | Check gog CLI, claude CLI, per-account mail auth (Gmail and IMAP), and config.yml. Returns `{ok, checks[], warnings[]}`. Call this first. | +| `search_inbox(query?, max_results?, account?)` | Search every configured mailbox (or one `account`), returns `{query, count, messages[{id, threadId, account}]}`. `query` defaults to the built-in invoice search; IMAP accounts run it as plain text. | +| `get_message(message_id, account?)` | One message's headers plus `attachments[{attachment_id, filename, mime_type}]`. Use it to pick which PDF to pull. | +| `download_attachment(message_id, attachment_id, filename, account?)` | Download one attachment to staging, returns `{path, filename, exists}`. | +| `fetch_invoices(max_emails?, query?)` | The full pipeline: search all mailboxes, download PDFs, classify, dedupe, file under `output/`, write `invoices.json` and `SUMMARY.md`. Returns `{count, invoices[], output_dir, summary_md, invoices_json}`. | | `classify_invoice(pdf_path, subject?, sender?, snippet?)` | Classify one local PDF (e.g. one you downloaded yourself) into the same record shape. Does **not** move the file. | | `list_invoices(status?, company?)` | Read back the last fetch from `invoices.json`, filtered by status (`Paid`/`To-Pay`) and/or company slug. No network. | | `copy_to_dropbox(status?, companies?)` | Copy filed PDFs into each company's configured folder (`dropbox_dirs` in config.yml, any folder and not only Dropbox). Returns `{copied, skipped, errors, details[]}`. | | `generate_payments(companies?, execution_date?)` | Build SEPA `pain.001.001.03` XML for To-Pay invoices, one file per debtor company. Derives missing BICs from Estonian IBANs. Returns `{count, files[], derived_bics[], skipped[]}`. | -| `archive_thread(thread_id)` | Remove the INBOX label from a Gmail thread. | +| `archive_thread(thread_id, account?)` | Remove a thread from its account's inbox. | | `parse_missing_list(text)` | Turn an accountant's free-form missing/expected-invoice list (pasted text, CSV, table, email) into a structured checklist, returns `{count, items[]}`. | | `reconcile(checklist, collected?)` | Match the checklist against collected invoices (or the last fetch if `collected` omitted). Returns `{summary, matched[], still_missing[], unmatched_collected[]}`. Deterministic. | | `build_report(reconciliation, title?)` | Render a reconcile result as Markdown, write `REPORT.md`, return `{report_md, path}`. Optional summary for the accountant. | | `plan_retrieval(items)` | For invoices NOT in the inbox, return retrieval tasks per item (vendor, identifiers, any `config.yml` per-vendor recipe, suggested sources, instructions). You fetch each with your own browser, then attach via `draft_reply`. | -| `draft_reply(body, attachments?, reply_to_message_id?, to?, subject?)` | Create a **draft** Gmail reply to the accountant with the invoice files attached. Never auto-sends; a human reviews and sends. | +| `draft_reply(body, attachments?, reply_to_message_id?, to?, subject?, account?)` | Create a **draft** Gmail reply to the accountant with the invoice files attached. Never auto-sends; a human reviews and sends. | | `read_audit(limit?)` | Read the local audit log (`output/audit.log`). Every fetch, download, and draft is recorded. | Statuses are case-insensitive (`paid`, `to-pay`, `all`). Company values are the slugs from `config.yml`. +**Multiple accounts.** With `email_accounts` configured in `config.yml`, message, +thread, and attachment ids are scoped to the account they came from. Pass each +message's `account` (from `search_inbox`) back to `get_message`, +`download_attachment`, and `archive_thread`. With a single account, omit it. +Drafts always go through a Google-hosted account. + ## Main workflow: clear a missing-invoice list The core job: the accountant emails a list of invoices they're missing, you fetch diff --git a/config.example.yml b/config.example.yml index 8d0729c..183d3a7 100644 --- a/config.example.yml +++ b/config.example.yml @@ -35,3 +35,21 @@ vendor_sources: # login_hint: "log in with the company Google account; invoices under Billing > History" # einvoice_operator: "Envoice" # notes: "monthly subscription, issued on the 1st" + +# Optional. Mailboxes to search. Without this, the Gmail account `gog` is +# signed in to is used. Google-hosted addresses (gmail.com or any Workspace +# domain) need `gog auth add
` once each. Anything else goes through +# IMAP; the password comes from the env var named in `password_env`, never +# from this file (use an app password for providers with two-factor auth). +email_accounts: + # - address: you@example.com + # - address: info@other-domain.com + # - address: user@elsewhere.com + # provider: imap + # imap: + # host: imap.elsewhere.com + # port: 993 # optional, default 993 + # username: user@elsewhere.com # optional, defaults to address + # password_env: ELSEWHERE_IMAP_PASSWORD + # folder: INBOX # optional + # archive_folder: Archive # optional, used when archiving diff --git a/pyproject.toml b/pyproject.toml index 3536745..55f7e47 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -7,6 +7,7 @@ dependencies = [ "click", "httpx", "pyyaml", + "imap-tools", "mcp>=1.27.2", "anyio>=4.12.1", ] diff --git a/src/invoices/__init__.py b/src/invoices/__init__.py index e0cd4e9..854df0e 100644 --- a/src/invoices/__init__.py +++ b/src/invoices/__init__.py @@ -13,15 +13,16 @@ from .config import BASE_DIR, COMPANIES, DEBTOR_ACCOUNTS, require_config from .dropbox import copy_to_dropbox -from .gmail import archive_thread, create_directories, search_messages +from .gmail import GmailProvider from .helpers import console -from .models import InvoiceRecord +from .models import EmailAccount, InvoiceRecord from .payment_xml import ( fill_missing_bics, generate_payment_xml, incomplete_beneficiaries, ) -from .process import process_messages +from .process import create_directories, process_messages +from .providers import build_providers, search_all_accounts from .reconcile import build_report, parse_missing_list from .reconcile import reconcile as reconcile_items from .summary import write_summary @@ -37,7 +38,7 @@ def invoices(): '--max-emails', default=None, type=int, help='Limit number of emails to process' ) def fetch(max_emails: int | None): - """Fetch invoices from Gmail, classify, and organize.""" + """Fetch invoices from your mailboxes, classify, and organize.""" # Save terminal state before asyncio takes over signal handling saved_attrs = None if sys.stdin.isatty(): @@ -54,7 +55,8 @@ def fetch(max_emails: int | None): async def _main(max_emails: int | None): require_config() create_directories() - message_dicts = await search_messages(max_emails) + providers = build_providers() + message_dicts = await search_all_accounts(providers, max_emails) if not message_dicts: console.print('\n[bold]No messages found. Done.[/bold]') @@ -252,12 +254,14 @@ def archive( console.print('[yellow]No invoices match the given filters.[/yellow]') return - # Deduplicate by thread_id — group records per thread - threads: dict[str, list[InvoiceRecord]] = {} + # Deduplicate by (account, thread_id) — group records per thread. + # Thread IDs are account-scoped, so archiving must go through the + # provider the record came from. + threads: dict[tuple[str, str], list[InvoiceRecord]] = {} for r in records: if not r.thread_id: continue - threads.setdefault(r.thread_id, []).append(r) + threads.setdefault((r.account, r.thread_id), []).append(r) if not threads: console.print('[yellow]No threads found (records missing thread_id).[/yellow]') @@ -267,6 +271,8 @@ def archive( if max_emails is not None: thread_items = thread_items[:max_emails] + providers = {p.address: p for p in build_providers()} + console.print(f'\nFound [bold]{len(thread_items)}[/bold] thread(s) to review.\n') archived = 0 @@ -277,7 +283,12 @@ def archive( saved_attrs = termios.tcgetattr(sys.stdin) try: - for thread_id, thread_records in thread_items: + for (account, thread_id), thread_records in thread_items: + # Records from accounts no longer in config fall back to a + # Gmail provider for that address (gog may still be authed) + provider = providers.get(account) or GmailProvider( + EmailAccount(address=account) + ) first = thread_records[0] pdf_names = ', '.join(r.renamed_pdf for r in thread_records) amount_str = ( @@ -285,7 +296,6 @@ def archive( if first.amount_inc_vat else 'N/A' ) - gmail_link = f'https://mail.google.com/mail/u/0/#inbox/{thread_id}' date_str = first.doc_date or 'N/A' @@ -293,13 +303,16 @@ def archive( f'[bold]Subject:[/bold] {first.subject}\n' f'[bold]From:[/bold] {first.sender}\n' f'[bold]Date:[/bold] {date_str}\n' + f'[bold]Account:[/bold] {provider.label}\n' f'[bold]Status:[/bold] {first.status}\n' f'[bold]Company:[/bold] {first.company}\n' f'[bold]Amount:[/bold] {amount_str}\n' f'[bold]Summary:[/bold] {first.summary}\n' - f'[bold]PDF(s):[/bold] {pdf_names}\n' - f'[bold]Thread:[/bold] {gmail_link}' + f'[bold]PDF(s):[/bold] {pdf_names}' ) + if isinstance(provider, GmailProvider): + gmail_link = f'https://mail.google.com/mail/u/0/#inbox/{thread_id}' + panel_text += f'\n[bold]Thread:[/bold] {gmail_link}' console.print(Panel(panel_text, title=f'Thread {thread_id[:12]}…')) if dry_run: @@ -310,7 +323,7 @@ def archive( should_archive = yes or Confirm.ask(' Archive this thread?', default=False) if should_archive: - asyncio.run(archive_thread(thread_id)) + asyncio.run(provider.archive_thread(thread_id)) console.print('[green] Archived.[/green]\n') archived += 1 else: diff --git a/src/invoices/config.py b/src/invoices/config.py index 0e6a6a4..d96e944 100644 --- a/src/invoices/config.py +++ b/src/invoices/config.py @@ -3,6 +3,8 @@ import yaml +from .models import EmailAccount + BASE_DIR = ( Path(os.environ['FETCH_OUTPUT_DIR']).expanduser() if 'FETCH_OUTPUT_DIR' in os.environ @@ -19,11 +21,35 @@ ) STATUSES = ['Paid', 'To-Pay'] + # Load config.yml tolerantly. If it is missing or invalid, the package still # imports (so the MCP server starts and health_check can report the problem) and # only operations that actually need it raise, via require_config(). # VENDOR_SOURCES holds optional per-vendor retrieval recipes (portal URL / login # hint / e-invoice operator), matched against checklist vendor names. +def _parse_email_accounts(cfg: dict) -> list[EmailAccount]: + accounts: list[EmailAccount] = [] + for entry in cfg.get('email_accounts') or []: + if isinstance(entry, str): + accounts.append(EmailAccount(address=entry)) + continue + imap = entry.get('imap') or {} + accounts.append( + EmailAccount( + address=entry.get('address', ''), + provider=entry.get('provider', 'gmail'), + imap_host=imap.get('host', ''), + imap_port=int(imap.get('port', 993)), + imap_username=imap.get('username', ''), + imap_password_env=imap.get('password_env', ''), + imap_folder=imap.get('folder', 'INBOX'), + imap_archive_folder=imap.get('archive_folder', 'Archive'), + ) + ) + # No accounts configured: fall back to gog's default Gmail account + return accounts or [EmailAccount(address='')] + + CONFIG_ERROR: str | None = None try: with _CONFIG_PATH.open() as f: @@ -33,7 +59,15 @@ DROPBOX_DIRS = {k: Path(v) for k, v in _cfg['dropbox_dirs'].items()} DEBTOR_ACCOUNTS = _cfg.get('debtor_accounts', {}) VENDOR_SOURCES = _cfg.get('vendor_sources', {}) -except (OSError, KeyError, TypeError, AttributeError, yaml.YAMLError) as exc: + EMAIL_ACCOUNTS = _parse_email_accounts(_cfg) +except ( + OSError, + KeyError, + TypeError, + AttributeError, + ValueError, + yaml.YAMLError, +) as exc: CONFIG_ERROR = ( f'config.yml could not be loaded from {_CONFIG_PATH} ({exc}). ' 'Create it from config.example.yml, or set FETCH_CONFIG to its path.' @@ -43,6 +77,7 @@ DROPBOX_DIRS = {} DEBTOR_ACCOUNTS = {} VENDOR_SOURCES = {} + EMAIL_ACCOUNTS = [EmailAccount(address='')] def require_config() -> None: @@ -55,10 +90,12 @@ def require_config() -> None: raise RuntimeError(CONFIG_ERROR) +INVOICE_KEYWORDS = ['invoice', 'arve', 'receipt', 'payment', 'facture', 'paiement'] + GMAIL_QUERY = ( 'in:inbox -category:(promotions OR social) ' 'has:attachment filename:pdf ' - '(invoice OR arve OR receipt OR payment OR facture OR paiement)' + f'({" OR ".join(INVOICE_KEYWORDS)})' ) diff --git a/src/invoices/gmail.py b/src/invoices/gmail.py index 12f1bed..e45fd41 100644 --- a/src/invoices/gmail.py +++ b/src/invoices/gmail.py @@ -1,10 +1,13 @@ +from __future__ import annotations + import os import re import tempfile from pathlib import Path -from .config import BASE_DIR, COMPANIES, GMAIL_QUERY, STAGING_DIR, STATUSES -from .helpers import async_run, async_run_json, console +from .config import GMAIL_QUERY +from .helpers import async_run, async_run_json +from .providers import MailProvider _ID_RE = re.compile(r'^[A-Za-z0-9][A-Za-z0-9_-]*$') @@ -21,161 +24,137 @@ def _safe_id(value: str, kind: str) -> str: return value -def create_directories(): - console.rule('[bold]Step 1: Creating directory structure') - for status in STATUSES: - for slug in COMPANIES: - d = BASE_DIR / status / slug - d.mkdir(parents=True, exist_ok=True) - STAGING_DIR.mkdir(parents=True, exist_ok=True) - console.print( - f' Created directories under [bold]{BASE_DIR}[/bold] and [bold]{STAGING_DIR}[/bold]' - ) - - -async def search_messages( - max_emails: int | None, query: str | None = None -) -> list[dict]: - console.rule('[bold]Step 2: Searching Gmail for invoices') - data = await async_run_json( - [ - 'gog', - 'gmail', - 'messages', - 'search', - query or GMAIL_QUERY, - '--json', - '--all', - ] - ) - - # The output may be a list of messages or have a "messages" key - if isinstance(data, dict): - messages = data.get('messages', data.get('results', [])) - else: - messages = data - - message_dicts = [] - for msg in messages: - mid = msg.get('id') or msg.get('messageId') - tid = msg.get('threadId', '') - if mid: - message_dicts.append({'id': mid, 'threadId': tid}) - - if max_emails is not None: - message_dicts = message_dicts[:max_emails] - - console.print(f' Found [bold]{len(message_dicts)}[/bold] messages to process') - return message_dicts - - -async def fetch_message(message_id: str) -> dict: - """Fetch full message metadata.""" - message_id = _safe_id(message_id, 'message id') - return await async_run_json(['gog', 'gmail', 'get', message_id, '--json']) - - -async def fetch_thread(thread_id: str) -> dict: - """Fetch full thread with all messages.""" - thread_id = _safe_id(thread_id, 'thread id') - data = await async_run_json( - ['gog', 'gmail', 'thread', 'get', thread_id, '--full', '--json'] - ) - # gog wraps the result in a {"thread": ...} envelope - if isinstance(data, dict) and 'thread' in data: - return data['thread'] - return data - - -async def download_attachment( - message_id: str, attachment_id: str, filename: str -) -> Path: - """Download a single attachment into a per-message staging subfolder. - - ``filename`` comes from the email (attacker-influenced), so it is reduced to - a bare basename and the download is namespaced under the (sanitized) message - id — preventing path traversal and silent same-name collisions between items. - """ - message_id = _safe_id(message_id, 'message id') - attachment_id = _safe_id(attachment_id, 'attachment id') - safe_name = Path(filename).name - if not safe_name or safe_name in {'.', '..'}: - raise ValueError(f'unsafe attachment filename: {filename!r}') - safe_mid = re.sub(r'[^A-Za-z0-9_-]', '', message_id) or 'message' - - dest_dir = STAGING_DIR / safe_mid - dest_dir.mkdir(parents=True, exist_ok=True) - - await async_run( - [ - 'gog', - 'gmail', - 'attachment', - message_id, - attachment_id, - '--out', - str(dest_dir), - '--name', - safe_name, - ] - ) - return dest_dir / safe_name - - -async def archive_thread(thread_id: str) -> None: - """Remove INBOX label from a thread (archive it).""" - thread_id = _safe_id(thread_id, 'thread id') - await async_run(['gog', 'gmail', 'labels', 'modify', thread_id, '--remove=INBOX']) - - -async def create_draft_reply( - body: str, - attachments: list[str], - reply_to_message_id: str | None = None, - to: str | None = None, - subject: str | None = None, -) -> dict: - """Create a Gmail DRAFT (never auto-sends) replying with files attached. - - Pass ``reply_to_message_id`` (the accountant's message) or ``to``. Returns a - status dict. Defaulting to a draft keeps a human in the loop before anything - goes to an external recipient. - """ - if not reply_to_message_id and not to: - raise ValueError('provide reply_to_message_id or to') - - paths = [Path(a).expanduser() for a in attachments] - missing = [str(p) for p in paths if not p.exists()] - if missing: - raise FileNotFoundError(f'attachment(s) not found: {", ".join(missing)}') - - cmd = ['gog', 'gmail', 'drafts', 'create'] - if reply_to_message_id: - cmd += ['--reply-to-message-id', reply_to_message_id] - if to: - cmd += ['--to', to] - if subject: - cmd += ['--subject', subject] - - # Body via a temp file so multi-line content survives intact. The file is - # created before the try, so the finally guarantees cleanup even if the - # write itself fails. - fd, name = tempfile.mkstemp(suffix='.txt', prefix='fetch-draft-') - os.close(fd) - body_file = Path(name) - try: - body_file.write_text(body) - cmd += ['--body-file', str(body_file)] - for p in paths: - cmd += ['--attach', str(p)] - proc = await async_run(cmd, check=False) - finally: - body_file.unlink(missing_ok=True) - - if proc.returncode != 0: - raise RuntimeError(f'gog drafts create failed: {proc.stderr_text[:300]}') - return { - 'status': 'drafted', - 'attachments': [str(p) for p in paths], - 'output': proc.stdout_text.strip()[:500], - } +class GmailProvider(MailProvider): + """Google-hosted mailbox (gmail.com or any Workspace domain) via the gog CLI.""" + + def _gog(self, *args: str) -> list[str]: + cmd = ['gog'] + if self.address: + cmd += ['-a', self.address] + cmd.extend(args) + return cmd + + async def search_messages( + self, max_emails: int | None, query: str | None = None + ) -> list[dict]: + data = await async_run_json( + self._gog( + 'gmail', 'messages', 'search', query or GMAIL_QUERY, '--json', '--all' + ) + ) + + # The output may be a list of messages or have a "messages" key + if isinstance(data, dict): + messages = data.get('messages', data.get('results', [])) + else: + messages = data + + message_dicts = [] + for msg in messages: + mid = msg.get('id') or msg.get('messageId') + tid = msg.get('threadId', '') + if mid: + message_dicts.append({'id': mid, 'threadId': tid}) + + if max_emails is not None: + message_dicts = message_dicts[:max_emails] + + return message_dicts + + async def fetch_message(self, message_id: str) -> dict: + """Fetch full message metadata.""" + message_id = _safe_id(message_id, 'message id') + return await async_run_json(self._gog('gmail', 'get', message_id, '--json')) + + async def fetch_thread(self, thread_id: str) -> dict: + """Fetch full thread with all messages.""" + thread_id = _safe_id(thread_id, 'thread id') + data = await async_run_json( + self._gog('gmail', 'thread', 'get', thread_id, '--full', '--json') + ) + # gog wraps the result in a {"thread": ...} envelope + if isinstance(data, dict) and 'thread' in data: + return data['thread'] + return data + + async def download_attachment( + self, message_id: str, attachment_id: str, filename: str + ) -> Path: + """Download a single attachment into the sanitized staging subfolder.""" + message_id = _safe_id(message_id, 'message id') + attachment_id = _safe_id(attachment_id, 'attachment id') + dest_dir, safe_name = self._staging_target(message_id, filename) + + await async_run( + self._gog( + 'gmail', + 'attachment', + message_id, + attachment_id, + '--out', + str(dest_dir), + '--name', + safe_name, + ) + ) + return dest_dir / safe_name + + async def archive_thread(self, thread_id: str) -> None: + """Remove INBOX label from a thread (archive it).""" + thread_id = _safe_id(thread_id, 'thread id') + await async_run( + self._gog('gmail', 'labels', 'modify', thread_id, '--remove=INBOX') + ) + + async def create_draft_reply( + self, + body: str, + attachments: list[str], + reply_to_message_id: str | None = None, + to: str | None = None, + subject: str | None = None, + ) -> dict: + """Create a Gmail DRAFT (never auto-sends) replying with files attached. + + Pass ``reply_to_message_id`` (the accountant's message) or ``to``. Returns a + status dict. Defaulting to a draft keeps a human in the loop before anything + goes to an external recipient. + """ + if not reply_to_message_id and not to: + raise ValueError('provide reply_to_message_id or to') + + paths = [Path(a).expanduser() for a in attachments] + missing = [str(p) for p in paths if not p.exists()] + if missing: + raise FileNotFoundError(f'attachment(s) not found: {", ".join(missing)}') + + cmd = self._gog('gmail', 'drafts', 'create') + if reply_to_message_id: + cmd += ['--reply-to-message-id', reply_to_message_id] + if to: + cmd += ['--to', to] + if subject: + cmd += ['--subject', subject] + + # Body via a temp file so multi-line content survives intact. The file is + # created before the try, so the finally guarantees cleanup even if the + # write itself fails. + fd, name = tempfile.mkstemp(suffix='.txt', prefix='fetch-draft-') + os.close(fd) + body_file = Path(name) + try: + body_file.write_text(body) + cmd += ['--body-file', str(body_file)] + for p in paths: + cmd += ['--attach', str(p)] + proc = await async_run(cmd, check=False) + finally: + body_file.unlink(missing_ok=True) + + if proc.returncode != 0: + raise RuntimeError(f'gog drafts create failed: {proc.stderr_text[:300]}') + return { + 'status': 'drafted', + 'attachments': [str(p) for p in paths], + 'output': proc.stdout_text.strip()[:500], + } diff --git a/src/invoices/imap.py b/src/invoices/imap.py new file mode 100644 index 0000000..d4315e6 --- /dev/null +++ b/src/invoices/imap.py @@ -0,0 +1,133 @@ +from __future__ import annotations + +import asyncio +import os +from typing import TYPE_CHECKING + +from imap_tools import AND, OR, MailBox + +from .config import INVOICE_KEYWORDS +from .providers import MailProvider + +if TYPE_CHECKING: + from pathlib import Path + + from imap_tools import MailMessage + + from .models import EmailAccount + +# Snippet length passed to the classifier; full HTML emails can be huge +_BODY_LIMIT = 4000 + + +class ImapProvider(MailProvider): + """Universal fallback for mailboxes not hosted on Google. + + IMAP has no thread concept, so every message is treated as its own + single-message thread (threadId == message UID). Search fetches full + messages once and caches them, so classification and attachment + downloads never touch the network again. + """ + + def __init__(self, account: EmailAccount): + super().__init__(account) + self._messages: dict[str, MailMessage] = {} + + def _login(self) -> MailBox: + env = self.account.imap_password_env + password = os.environ.get(env, '') if env else '' + if not password: + raise RuntimeError( + f'IMAP password for {self.address} not found; ' + f'set imap.password_env in config.yml and export {env or ""}' + ) + mailbox = MailBox(self.account.imap_host, self.account.imap_port) + return mailbox.login( + self.account.imap_username or self.address, + password, + initial_folder=self.account.imap_folder, + ) + + def _search_sync(self, max_emails: int | None, query: str | None) -> list[dict]: + # IMAP cannot evaluate Gmail search syntax; a custom query is used as a + # plain text search, the default is the invoice keyword search. + criteria = OR(text=[query]) if query else OR(text=INVOICE_KEYWORDS) + refs: list[dict] = [] + with self._login() as mailbox: + for msg in mailbox.fetch(criteria, mark_seen=False, reverse=True): + has_pdf = any( + (att.filename or '').lower().endswith('.pdf') + for att in msg.attachments + ) + if not has_pdf: + continue + self._messages[msg.uid] = msg + refs.append({'id': msg.uid, 'threadId': msg.uid}) + if max_emails is not None and len(refs) >= max_emails: + break + return refs + + def _fetch_sync(self, message_id: str) -> MailMessage: + with self._login() as mailbox: + for msg in mailbox.fetch(AND(uid=message_id), mark_seen=False): + return msg + raise RuntimeError(f'Message {message_id} not found on {self.label}') + + async def _get_message(self, message_id: str) -> MailMessage: + msg = self._messages.get(message_id) + if msg is None: + msg = await asyncio.to_thread(self._fetch_sync, message_id) + self._messages[message_id] = msg + return msg + + def _normalize(self, msg: MailMessage) -> dict: + return { + 'id': msg.uid, + 'threadId': msg.uid, + 'headers': { + 'subject': msg.subject, + 'from': msg.from_, + 'date': str(msg.date or ''), + }, + 'body': (msg.text or msg.html or '')[:_BODY_LIMIT], + 'attachments': [ + { + 'filename': att.filename, + 'attachmentId': str(i), + 'mimeType': att.content_type, + } + for i, att in enumerate(msg.attachments) + if att.filename + ], + } + + async def search_messages( + self, max_emails: int | None, query: str | None = None + ) -> list[dict]: + return await asyncio.to_thread(self._search_sync, max_emails, query) + + async def fetch_message(self, message_id: str) -> dict: + return self._normalize(await self._get_message(message_id)) + + async def fetch_thread(self, thread_id: str) -> dict: + return {'messages': [await self.fetch_message(thread_id)]} + + async def download_attachment( + self, message_id: str, attachment_id: str, filename: str + ) -> Path: + msg = await self._get_message(message_id) + attachment = msg.attachments[int(attachment_id)] + dest_dir, safe_name = self._staging_target(message_id, filename) + path = dest_dir / safe_name + path.write_bytes(attachment.payload) + return path + + def _archive_sync(self, message_id: str) -> None: + folder = self.account.imap_archive_folder + with self._login() as mailbox: + if not mailbox.folder.exists(folder): + mailbox.folder.create(folder) + mailbox.move(message_id, folder) + + async def archive_thread(self, thread_id: str) -> None: + await asyncio.to_thread(self._archive_sync, thread_id) diff --git a/src/invoices/models.py b/src/invoices/models.py index 0e96edd..4f7e906 100644 --- a/src/invoices/models.py +++ b/src/invoices/models.py @@ -2,6 +2,18 @@ from typing import TypedDict +@dataclass +class EmailAccount: + address: str + provider: str = 'gmail' # 'gmail' or 'imap' + imap_host: str = '' + imap_port: int = 993 + imap_username: str = '' # defaults to address + imap_password_env: str = '' # env var holding the IMAP password + imap_folder: str = 'INBOX' + imap_archive_folder: str = 'Archive' + + @dataclass class Attachment: message_id: str @@ -44,6 +56,7 @@ class InvoiceRecord: is_overdue: bool = False doc_date: str = '' thread_id: str = '' + account: str = '' # mailbox the invoice came from; IDs are scoped to it sha1: str = '' invoice_number: str | None = None beneficiary_name: str | None = None @@ -66,6 +79,7 @@ class ChecklistItem: class InvoiceRecordDict(TypedDict): thread_id: str + account: str subject: str sender: str renamed_pdf: str @@ -89,6 +103,7 @@ class InvoiceRecordDict(TypedDict): def record_to_dict(r: InvoiceRecord) -> InvoiceRecordDict: return InvoiceRecordDict( thread_id=r.thread_id, + account=r.account, subject=r.subject, sender=r.sender, renamed_pdf=r.renamed_pdf, diff --git a/src/invoices/process.py b/src/invoices/process.py index a654d84..b0a5615 100644 --- a/src/invoices/process.py +++ b/src/invoices/process.py @@ -13,8 +13,7 @@ ) from .classify import _format_thread_context, classify_pdf -from .config import BASE_DIR -from .gmail import download_attachment, fetch_message, fetch_thread +from .config import BASE_DIR, COMPANIES, STAGING_DIR, STATUSES from .helpers import console, sha1_file from .models import InvoiceRecord @@ -22,6 +21,20 @@ from pathlib import Path from .models import Classification + from .providers import MailProvider + + +def create_directories(): + console.rule('[bold]Step 1: Creating directory structure') + for status in STATUSES: + for slug in COMPANIES: + d = BASE_DIR / status / slug + d.mkdir(parents=True, exist_ok=True) + STAGING_DIR.mkdir(parents=True, exist_ok=True) + console.print( + f' Created directories under [bold]{BASE_DIR}[/bold] and [bold]{STAGING_DIR}[/bold]' + ) + # ── Dedup state ────────────────────────────────────────────────────────────── @@ -45,6 +58,7 @@ def is_duplicate(pdf_path, target_dir): async def process_thread( + provider: MailProvider, thread_id: str, matched_message_ids: list[str], index: int, @@ -58,12 +72,12 @@ async def process_thread( async with semaphore: console.print( - f'\n[bold]--- Thread {index}/{total}: {thread_id} ' + f'\n[bold]--- Thread {index}/{total}: {thread_id} [{provider.label}] ' f'({len(matched_message_ids)} matched message(s)) ---[/bold]' ) try: - thread = await fetch_thread(thread_id) + thread = await provider.fetch_thread(thread_id) except Exception as exc: console.print(f' [red]ERROR:[/red] Could not fetch thread: {exc}') progress.advance(task_id) @@ -84,7 +98,7 @@ async def process_thread( # Thread messages use raw Gmail format; fetch normalized version try: - msg = await fetch_message(mid) + msg = await provider.fetch_message(mid) except Exception as exc: console.print(f' [dim]Could not fetch message {mid}: {exc}[/dim]') continue @@ -111,7 +125,7 @@ async def process_thread( att_filename = att['filename'] console.print(f'\n Attachment: [cyan]{att_filename}[/cyan]') - pdf_path = await download_attachment(mid, att_id, att_filename) + pdf_path = await provider.download_attachment(mid, att_id, att_filename) if not pdf_path.exists(): console.print( f' [red]ERROR:[/red] Download failed for {att_filename}' @@ -217,6 +231,7 @@ async def process_thread( is_overdue=classification.is_overdue, doc_date=classification.doc_date, thread_id=thread_id, + account=provider.address, sha1=sha1, invoice_number=classification.invoice_number, beneficiary_name=classification.beneficiary_name, @@ -233,11 +248,13 @@ async def process_thread( async def process_messages(message_dicts: list[dict]) -> list[InvoiceRecord]: console.rule('[bold]Step 3: Processing threads') - # Group messages by threadId - threads: dict[str, list[str]] = {} + # Group messages by (account, threadId) — thread IDs are account-scoped, + # so every operation must go through the originating provider + threads: dict[tuple[str, str], tuple[MailProvider, list[str]]] = {} for md in message_dicts: tid = md.get('threadId', '') or md['id'] - threads.setdefault(tid, []).append(md['id']) + key = (md['account'], tid) + threads.setdefault(key, (md['provider'], []))[1].append(md['id']) semaphore = asyncio.Semaphore(5) records: list[InvoiceRecord] = [] @@ -253,9 +270,11 @@ async def process_messages(message_dicts: list[dict]) -> list[InvoiceRecord]: tasks = [ asyncio.ensure_future( - process_thread(tid, mids, i, len(threads), semaphore, progress, task_id) + process_thread( + provider, tid, mids, i, len(threads), semaphore, progress, task_id + ) ) - for i, (tid, mids) in enumerate(threads.items(), 1) + for i, ((_account, tid), (provider, mids)) in enumerate(threads.items(), 1) ] for coro in asyncio.as_completed(tasks): diff --git a/src/invoices/providers.py b/src/invoices/providers.py new file mode 100644 index 0000000..e0d40c4 --- /dev/null +++ b/src/invoices/providers.py @@ -0,0 +1,129 @@ +from __future__ import annotations + +import asyncio +import re +from abc import ABC, abstractmethod +from pathlib import Path +from typing import TYPE_CHECKING + +from .config import EMAIL_ACCOUNTS, STAGING_DIR +from .helpers import console + +if TYPE_CHECKING: + from .models import EmailAccount + + +class MailProvider(ABC): + """One connected mailbox. + + Message, thread, and attachment IDs are scoped to the mailbox they came + from — an ID from one account cannot be fetched through another. + """ + + def __init__(self, account: EmailAccount): + self.account = account + + @property + def address(self) -> str: + return self.account.address + + @property + def label(self) -> str: + return self.address or 'default Gmail account' + + def _staging_target(self, message_id: str, filename: str) -> tuple[Path, str]: + """Sanitized (directory, name) for a staged attachment download. + + ``filename`` comes from the email (attacker-influenced), so it is + reduced to a bare basename, and the path is namespaced by account and + (sanitized) message id — preventing path traversal, collisions between + same-named attachments in concurrent threads, and collisions between + accounts (IMAP UIDs are small integers that repeat across mailboxes). + """ + safe_name = Path(filename).name + if not safe_name or safe_name in {'.', '..'}: + raise ValueError(f'unsafe attachment filename: {filename!r}') + safe_account = ( + re.sub(r'[^A-Za-z0-9_.@-]', '', self.address).strip('.') or 'default' + ) + safe_mid = re.sub(r'[^A-Za-z0-9_-]', '', message_id) or 'message' + + dest_dir = STAGING_DIR / safe_account / safe_mid + dest_dir.mkdir(parents=True, exist_ok=True) + return dest_dir, safe_name + + @abstractmethod + async def search_messages( + self, max_emails: int | None, query: str | None = None + ) -> list[dict]: + """Return candidate invoice messages as [{'id': ..., 'threadId': ...}].""" + + @abstractmethod + async def fetch_message(self, message_id: str) -> dict: + """Return a normalized message: headers dict, body text, attachments list.""" + + @abstractmethod + async def fetch_thread(self, thread_id: str) -> dict: + """Return {'messages': [...]} for the full conversation.""" + + @abstractmethod + async def download_attachment( + self, message_id: str, attachment_id: str, filename: str + ) -> Path: + """Download one attachment into staging and return its path.""" + + @abstractmethod + async def archive_thread(self, thread_id: str) -> None: + """Remove the thread from the inbox.""" + + +def build_providers() -> list[MailProvider]: + # Imported here because both modules subclass MailProvider from this file + from .gmail import GmailProvider # noqa: PLC0415 + from .imap import ImapProvider # noqa: PLC0415 + + classes = {'gmail': GmailProvider, 'imap': ImapProvider} + providers: list[MailProvider] = [] + for account in EMAIL_ACCOUNTS: + cls = classes.get(account.provider) + if cls is None: + console.print( + f' [yellow]WARNING:[/yellow] Unknown provider ' + f"'{account.provider}' for {account.address}, skipping" + ) + continue + providers.append(cls(account)) + return providers + + +async def search_all_accounts( + providers: list[MailProvider], max_emails: int | None, query: str | None = None +) -> list[dict]: + """Search every mailbox concurrently and merge the results. + + Each returned dict carries the provider it came from, since all + downstream operations must go through the same account. One failing + account is reported and skipped instead of aborting the run. + """ + console.rule('[bold]Step 2: Searching mailboxes for invoices') + + results = await asyncio.gather( + *(p.search_messages(max_emails, query) for p in providers), + return_exceptions=True, + ) + + message_dicts: list[dict] = [] + for provider, result in zip(providers, results, strict=True): + if isinstance(result, BaseException): + console.print(f' [red]ERROR:[/red] {provider.label}: {result}') + continue + console.print(f' {provider.label}: [bold]{len(result)}[/bold] message(s)') + message_dicts.extend( + {**md, 'account': provider.address, 'provider': provider} for md in result + ) + + if max_emails is not None: + message_dicts = message_dicts[:max_emails] + + console.print(f' Found [bold]{len(message_dicts)}[/bold] messages to process') + return message_dicts diff --git a/src/mcp_server.py b/src/mcp_server.py index c6d94ec..c343501 100644 --- a/src/mcp_server.py +++ b/src/mcp_server.py @@ -36,25 +36,11 @@ require_config, ) from src.invoices.dropbox import copy_to_dropbox as _copy_to_dropbox -from src.invoices.gmail import ( - archive_thread as _archive_thread, -) -from src.invoices.gmail import ( - create_directories, - search_messages, -) -from src.invoices.gmail import ( - create_draft_reply as _create_draft_reply, -) -from src.invoices.gmail import ( - download_attachment as _download_attachment, -) -from src.invoices.gmail import ( - fetch_message as _fetch_message, -) +from src.invoices.gmail import GmailProvider from src.invoices.helpers import console, sha1_file -from src.invoices.models import InvoiceRecord, record_to_dict -from src.invoices.process import process_messages +from src.invoices.models import EmailAccount, InvoiceRecord, record_to_dict +from src.invoices.process import create_directories, process_messages +from src.invoices.providers import MailProvider, build_providers, search_all_accounts from src.invoices.reconcile import ( build_report as _build_report, ) @@ -106,6 +92,47 @@ def _normalize_status(status: str | None) -> str | None: return norm +def _resolve_provider(account: str | None) -> MailProvider: + """Pick the mailbox provider an account-scoped id belongs to. + + With one configured account (the default), ``account`` can be omitted. + An address not in config falls back to a Gmail provider for it (gog may + still be authenticated), mirroring the CLI's archive behavior. + """ + providers = build_providers() + if account is None: + if len(providers) == 1: + return providers[0] + raise ValueError( + 'Multiple accounts configured; pass account= ' + f'(one of: {", ".join(p.label for p in providers)})' + ) + for p in providers: + if p.address == account: + return p + return GmailProvider(EmailAccount(address=account)) + + +def _resolve_gmail_provider(account: str | None) -> GmailProvider: + """Like _resolve_provider, but for operations only Gmail supports (drafts).""" + gmail = [p for p in build_providers() if isinstance(p, GmailProvider)] + if account is not None: + for p in gmail: + if p.address == account: + return p + return GmailProvider(EmailAccount(address=account)) + if len(gmail) == 1: + return gmail[0] + if not gmail: + raise ValueError( + 'No Gmail account configured; pass account=' + ) + raise ValueError( + 'Multiple Gmail accounts configured; pass account= ' + f'(one of: {", ".join(p.address for p in gmail)})' + ) + + def _validate_companies(companies: list[str] | None) -> None: """Raise ValueError if any company slug is unknown (mirrors the CLI).""" if not companies: @@ -132,19 +159,25 @@ async def health_check() -> dict: @mcp.tool() async def search_inbox( - query: str | None = None, max_results: int | None = None + query: str | None = None, max_results: int | None = None, account: str | None = None ) -> dict: - """Search the user's Gmail and return matching message/thread ids. + """Search the user's mailboxes and return matching message/thread ids. - ``query`` is a Gmail search expression (e.g. "from:vendor has:attachment"). - When omitted, the built-in invoice query is used. Useful to preview what - ``fetch_invoices`` would process, or to locate a specific email. + ``query`` is a Gmail search expression (e.g. "from:vendor has:attachment"); + IMAP accounts run it as a plain text search. When omitted, the built-in + invoice query is used. All configured accounts are searched unless + ``account`` narrows it to one address. Ids are account-scoped: pass each + message's ``account`` back to ``get_message``/``download_attachment``. """ - messages = await search_messages(max_results, query) + providers = [_resolve_provider(account)] if account else build_providers() + messages = await search_all_accounts(providers, max_results, query) return { 'query': query or GMAIL_QUERY, 'count': len(messages), - 'messages': messages, + 'messages': [ + {'id': m['id'], 'threadId': m['threadId'], 'account': m['account']} + for m in messages + ], } @@ -152,9 +185,9 @@ async def search_inbox( async def fetch_invoices( max_emails: int | None = None, query: str | None = None ) -> dict: - """Fetch invoices from Gmail, classify each PDF with AI, and file them. + """Fetch invoices from all configured mailboxes, classify, and file them. - Runs the full pipeline: search Gmail -> download PDF attachments -> + Runs the full pipeline: search every account -> download PDF attachments -> classify (company, status, amounts, beneficiary bank details) -> dedupe -> organize under the output directory -> write invoices.json + SUMMARY.md. @@ -163,7 +196,7 @@ async def fetch_invoices( """ require_config() create_directories() - messages = await search_messages(max_emails, query) + messages = await search_all_accounts(build_providers(), max_emails, query) if not messages: log_event('fetch_invoices', {'count': 0, 'query': query or GMAIL_QUERY}) # No fetch happened, so no summary/json was written; keep the documented @@ -325,21 +358,29 @@ def generate_payments( @mcp.tool() -async def archive_thread(thread_id: str) -> dict: - """Archive a Gmail thread (remove its INBOX label) by thread id.""" - await _archive_thread(thread_id) - return {'archived': thread_id} +async def archive_thread(thread_id: str, account: str | None = None) -> dict: + """Archive a thread by id: remove it from the inbox of its account. + + Thread ids are account-scoped; pass the ``account`` the id came from + (required when several accounts are configured). + """ + provider = _resolve_provider(account) + await provider.archive_thread(thread_id) + return {'archived': thread_id, 'account': provider.address} @mcp.tool() -async def get_message(message_id: str) -> dict: - """Fetch one Gmail message's headers and attachment list. +async def get_message(message_id: str, account: str | None = None) -> dict: + """Fetch one message's headers and attachment list. Use after ``search_inbox`` to inspect a candidate email and pick which PDF to - pull with ``download_attachment``. Returns subject/sender/date/snippet and + pull with ``download_attachment``. Message ids are account-scoped; pass the + ``account`` the id came from (required when several accounts are configured). + Returns subject/sender/date/snippet and ``attachments[{attachment_id, filename, mime_type}]``. """ - msg = await _fetch_message(message_id) + provider = _resolve_provider(account) + msg = await provider.fetch_message(message_id) headers = msg.get('headers', {}) headers = headers if isinstance(headers, dict) else {} attachments = [ @@ -352,6 +393,7 @@ async def get_message(message_id: str) -> dict: ] return { 'message_id': message_id, + 'account': provider.address, 'thread_id': msg.get('threadId', ''), 'subject': headers.get('subject', ''), 'sender': headers.get('from', ''), @@ -363,17 +405,24 @@ async def get_message(message_id: str) -> dict: @mcp.tool() async def download_attachment( - message_id: str, attachment_id: str, filename: str + message_id: str, attachment_id: str, filename: str, account: str | None = None ) -> dict: - """Download one Gmail attachment to the staging dir; returns its local path. + """Download one attachment to the staging dir; returns its local path. Pair with ``get_message`` to retrieve the invoice PDF for a checklist item, - then attach the path via ``draft_reply``. + then attach the path via ``draft_reply``. Ids are account-scoped; pass the + ``account`` the message came from (required when several are configured). """ - path = await _download_attachment(message_id, attachment_id, filename) + provider = _resolve_provider(account) + path = await provider.download_attachment(message_id, attachment_id, filename) log_event( 'download_attachment', - {'message_id': message_id, 'filename': filename, 'path': str(path)}, + { + 'message_id': message_id, + 'account': provider.address, + 'filename': filename, + 'path': str(path), + }, ) return {'path': str(path), 'filename': filename, 'exists': path.exists()} @@ -450,6 +499,7 @@ async def draft_reply( reply_to_message_id: str | None = None, to: str | None = None, subject: str | None = None, + account: str | None = None, ) -> dict: """Create a DRAFT Gmail reply to the accountant with the invoice files attached. @@ -458,8 +508,11 @@ async def draft_reply( request email, e.g. from ``search_inbox``/``get_message``) or ``to`` (an email address). ``attachments`` is a list of local PDF paths gathered from the inbox and/or beyond-inbox retrieval. Put the "still couldn't find: …" note in ``body``. + Drafts need a Google-hosted mailbox; with several configured, pass ``account`` + to pick which one the draft is created in. """ - result = await _create_draft_reply( + provider = _resolve_gmail_provider(account) + result = await provider.create_draft_reply( body=body, attachments=attachments or [], reply_to_message_id=reply_to_message_id, @@ -471,6 +524,7 @@ async def draft_reply( { 'reply_to_message_id': reply_to_message_id, 'to': to, + 'account': provider.address, 'attachment_count': len(attachments or []), 'attachments': attachments or [], }, diff --git a/src/onboarding.py b/src/onboarding.py index 1932662..3e00f2d 100644 --- a/src/onboarding.py +++ b/src/onboarding.py @@ -1,11 +1,14 @@ +import json import os import shutil import subprocess from pathlib import Path import yaml +from imap_tools import MailBox from src import fetch +from src.invoices.config import EMAIL_ACCOUNTS from src.invoices.helpers import console _CONFIG_PATH = ( @@ -13,27 +16,34 @@ if 'FETCH_CONFIG' in os.environ else Path(__file__).resolve().parents[1] / 'config.yml' ) -_REQUIRED_KEYS = ['companies', 'dropbox_dirs', 'debtor_accounts'] +_REQUIRED_KEYS = ['default_slug', 'companies', 'dropbox_dirs', 'debtor_accounts'] def check_prerequisites() -> dict: """Check runtime prerequisites and return a structured report. - Returns ``{'ok': bool, 'checks': [{'name', 'ok', 'detail', 'hint'}, ...]}``. + Returns ``{'ok': bool, 'checks': [{'name', 'ok', 'detail', 'hint'}, ...], + 'warnings': [str, ...]}``. Warnings are advisory (e.g. a gmail-configured + domain whose mail is not Google-hosted) and do not flip ``ok``. Shared by the ``onboarding`` CLI command and the MCP ``health_check`` tool. """ checks: list[dict] = [] + warnings: list[str] = [] - # gog CLI — used for all Gmail access. + gmail_accounts = [a for a in EMAIL_ACCOUNTS if a.provider == 'gmail'] + imap_accounts = [a for a in EMAIL_ACCOUNTS if a.provider == 'imap'] + + # gog CLI — needed for Google-hosted mailboxes (and for draft replies). gog_path = shutil.which('gog') - checks.append( - { - 'name': 'gog CLI', - 'ok': bool(gog_path), - 'detail': gog_path or 'not found on PATH', - 'hint': '' if gog_path else 'Install: brew install openclaw/tap/gogcli', - } - ) + if gmail_accounts: + checks.append( + { + 'name': 'gog CLI', + 'ok': bool(gog_path), + 'detail': gog_path or 'not found on PATH', + 'hint': '' if gog_path else 'Install: brew install openclaw/tap/gogcli', + } + ) # claude CLI — used to classify invoice PDFs. claude_path = shutil.which('claude') @@ -48,80 +58,217 @@ def check_prerequisites() -> dict: } ) - # Gmail authentication — only checkable if gog is present. - if gog_path: - result = subprocess.run( # noqa: S603 - [gog_path, 'auth', 'list'], - stdin=subprocess.DEVNULL, - capture_output=True, - text=True, - check=False, - ) - authed = result.returncode == 0 and bool(result.stdout.strip()) - checks.append( + # Gmail authentication, per configured account. + if gmail_accounts: + if gog_path: + warnings.extend(_mx_warnings(gmail_accounts)) + warnings.extend(_multi_domain_notes(gmail_accounts)) + checks.extend(_gmail_auth_checks(gog_path, gmail_accounts)) + else: + checks.append( + { + 'name': 'Gmail authentication', + 'ok': False, + 'detail': 'skipped (gog CLI not found)', + 'hint': 'Install gog first', + } + ) + + # IMAP accounts: config completeness + a real login test. + checks.extend(_imap_checks(imap_accounts)) + + # config.yml — companies, destination folders, debtor accounts. + checks.append(_config_check()) + + return { + 'ok': all(c['ok'] for c in checks), + 'checks': checks, + 'warnings': warnings, + } + + +def _config_check() -> dict: + if not _CONFIG_PATH.exists(): + return { + 'name': 'config.yml', + 'ok': False, + 'detail': f'{_CONFIG_PATH} not found', + 'hint': 'Copy the template: cp config.example.yml config.yml', + } + try: + with _CONFIG_PATH.open() as f: + cfg = yaml.safe_load(f) or {} + except yaml.YAMLError as exc: + return { + 'name': 'config.yml', + 'ok': False, + 'detail': f'invalid YAML: {exc}', + 'hint': 'Fix the YAML syntax in config.yml', + } + missing = [k for k in _REQUIRED_KEYS if k not in cfg] + if missing: + return { + 'name': 'config.yml', + 'ok': False, + 'detail': f'missing keys: {", ".join(missing)}', + 'hint': 'See config.example.yml for the expected shape', + } + return {'name': 'config.yml', 'ok': True, 'detail': 'valid', 'hint': ''} + + +def _authed_addresses(gog_path: str) -> list[str] | None: + """Return lowercase authenticated addresses from gog, or None on failure.""" + result = subprocess.run( # noqa: S603 + [gog_path, 'auth', 'list', '--json'], + stdin=subprocess.DEVNULL, + capture_output=True, + text=True, + check=False, + ) + if result.returncode != 0: + return None + try: + accounts = json.loads(result.stdout).get('accounts', []) + except (json.JSONDecodeError, AttributeError): + return None + addresses = [] + for entry in accounts: + if isinstance(entry, str): + addresses.append(entry.lower()) + elif isinstance(entry, dict): + address = entry.get('email') or entry.get('account') or '' + if address: + addresses.append(address.lower()) + return addresses + + +def _gmail_auth_checks(gog_path: str, gmail_accounts: list) -> list[dict]: + authed = _authed_addresses(gog_path) + if authed is None: + return [ + { + 'name': 'Gmail authentication', + 'ok': False, + 'detail': 'could not read gog auth state', + 'hint': 'Run: gog auth list', + } + ] + + expected = [a.address for a in gmail_accounts if a.address] + if not expected: + # No explicit accounts configured: any authenticated account will do + return [ { 'name': 'Gmail authentication', - 'ok': authed, + 'ok': bool(authed), 'detail': 'authenticated' if authed else 'no authenticated accounts', 'hint': '' if authed else 'Run: gog auth credentials && gog auth add you@gmail.com', } - ) - else: - checks.append( - { - 'name': 'Gmail authentication', - 'ok': False, - 'detail': 'skipped (gog CLI not found)', - 'hint': 'Install gog first', - } - ) + ] - # config.yml — companies, destination folders, debtor accounts. - if not _CONFIG_PATH.exists(): - checks.append( - { - 'name': 'config.yml', - 'ok': False, - 'detail': f'{_CONFIG_PATH} not found', - 'hint': 'Copy the template: cp config.example.yml config.yml', - } + return [ + { + 'name': f'Gmail account {addr}', + 'ok': addr.lower() in authed, + 'detail': 'authenticated' + if addr.lower() in authed + else 'not authenticated', + 'hint': '' if addr.lower() in authed else f'Run: gog auth add {addr}', + } + for addr in expected + ] + + +def _mx_warnings(gmail_accounts: list) -> list[str]: + """Warn when a gmail-configured domain's mail is not actually Google-hosted.""" + dig_path = shutil.which('dig') + if not dig_path: + return [] + warnings = [] + domains = sorted( + {a.address.split('@', 1)[1] for a in gmail_accounts if '@' in a.address} + ) + for domain in domains: + result = subprocess.run( # noqa: S603 + [dig_path, '+short', 'mx', domain], + stdin=subprocess.DEVNULL, + capture_output=True, + text=True, + check=False, ) - else: - try: - with _CONFIG_PATH.open() as f: - cfg = yaml.safe_load(f) or {} - missing = [k for k in _REQUIRED_KEYS if k not in cfg] - if missing: - checks.append( - { - 'name': 'config.yml', - 'ok': False, - 'detail': f'missing keys: {", ".join(missing)}', - 'hint': 'See config.example.yml for the expected shape', - } - ) - else: - checks.append( - { - 'name': 'config.yml', - 'ok': True, - 'detail': 'valid', - 'hint': '', - } - ) - except yaml.YAMLError as exc: + mx = result.stdout.strip() + if result.returncode != 0 or not mx: + continue # lookup failed; not a config problem + if 'google' not in mx.lower(): + warnings.append( + f'{domain} mail is not Google-hosted; ' + f"use 'provider: imap' for this account in config.yml" + ) + return warnings + + +def _multi_domain_notes(gmail_accounts: list) -> list[str]: + domains = {a.address.split('@', 1)[1] for a in gmail_accounts if '@' in a.address} + if len(domains) > 1: + return [ + 'Gmail accounts span multiple Google Workspace domains. The OAuth ' + 'client must be "External" (not "Internal"), and while unverified ' + 'each address must be added as a test user. A Workspace admin of ' + 'each domain may also need to allow the client.' + ] + return [] + + +def _imap_checks(imap_accounts: list) -> list[dict]: + checks = [] + for account in imap_accounts: + label = account.address or account.imap_host or 'imap account' + problems = [] + if not account.address: + problems.append('missing address') + if not account.imap_host: + problems.append('missing imap.host') + if not account.imap_password_env: + problems.append('missing imap.password_env') + elif not os.environ.get(account.imap_password_env): + problems.append(f'env var {account.imap_password_env} not set') + + if problems: checks.append( { - 'name': 'config.yml', + 'name': f'IMAP {label}', 'ok': False, - 'detail': f'invalid YAML: {exc}', - 'hint': 'Fix the YAML syntax in config.yml', + 'detail': ', '.join(problems), + 'hint': 'Set address, imap.host, and imap.password_env in ' + 'config.yml, and export the password env var', } ) + continue - return {'ok': all(c['ok'] for c in checks), 'checks': checks} + try: + mailbox = MailBox(account.imap_host, account.imap_port, timeout=15) + mailbox.login( + account.imap_username or account.address, + os.environ[account.imap_password_env], + initial_folder=account.imap_folder, + ) + mailbox.logout() + checks.append( + {'name': f'IMAP {label}', 'ok': True, 'detail': 'login OK', 'hint': ''} + ) + except Exception as exc: + checks.append( + { + 'name': f'IMAP {label}', + 'ok': False, + 'detail': f'login failed: {exc}', + 'hint': 'Check host/username/password; for providers with ' + 'two-factor auth, use an app password', + } + ) + return checks @fetch.command() @@ -133,6 +280,8 @@ def onboarding(): console.print(f'{mark} {c["name"]}: {c["detail"]}') if not c['ok'] and c['hint']: console.print(f' [dim]{c["hint"]}[/dim]') + for w in report['warnings']: + console.print(f'[yellow]![/yellow] {w}') console.print() if report['ok']: diff --git a/tests/test_gmail.py b/tests/test_gmail.py index 8af807a..97f6bdc 100644 --- a/tests/test_gmail.py +++ b/tests/test_gmail.py @@ -4,6 +4,7 @@ import src.invoices.gmail as gm from src.invoices.config import STAGING_DIR +from src.invoices.models import EmailAccount class _FakeProc: @@ -12,6 +13,19 @@ class _FakeProc: stderr_text = '' +def _provider(address: str = '') -> gm.GmailProvider: + return gm.GmailProvider(EmailAccount(address=address)) + + +def test_gog_command_default_account(): + assert _provider()._gog('gmail', 'get', 'm1') == ['gog', 'gmail', 'get', 'm1'] + + +def test_gog_command_selects_account(): + cmd = _provider('me@work.com')._gog('gmail', 'get', 'm1') + assert cmd == ['gog', '-a', 'me@work.com', 'gmail', 'get', 'm1'] + + def test_create_draft_reply_builds_command(monkeypatch, tmp_path): captured = {} @@ -23,7 +37,7 @@ async def fake(cmd, **kwargs): pdf = tmp_path / 'inv.pdf' pdf.write_bytes(b'%PDF') res = asyncio.run( - gm.create_draft_reply( + _provider().create_draft_reply( body='hi\nthere', attachments=[str(pdf)], reply_to_message_id='m1' ) ) @@ -35,6 +49,24 @@ async def fake(cmd, **kwargs): assert res['status'] == 'drafted' +def test_create_draft_reply_targets_account(monkeypatch, tmp_path): + captured = {} + + async def fake(cmd, **kwargs): + captured['cmd'] = cmd + return _FakeProc() + + monkeypatch.setattr(gm, 'async_run', fake) + pdf = tmp_path / 'inv.pdf' + pdf.write_bytes(b'%PDF') + asyncio.run( + _provider('me@work.com').create_draft_reply( + body='x', attachments=[str(pdf)], to='acct@firm.ee' + ) + ) + assert captured['cmd'][:3] == ['gog', '-a', 'me@work.com'] + + def test_create_draft_reply_missing_attachment(monkeypatch): async def fake(cmd, **kwargs): return _FakeProc() @@ -42,13 +74,15 @@ async def fake(cmd, **kwargs): monkeypatch.setattr(gm, 'async_run', fake) with pytest.raises(FileNotFoundError): asyncio.run( - gm.create_draft_reply(body='x', attachments=['/no/such.pdf'], to='a@b.c') + _provider().create_draft_reply( + body='x', attachments=['/no/such.pdf'], to='a@b.c' + ) ) def test_create_draft_reply_requires_recipient(): with pytest.raises(ValueError): - asyncio.run(gm.create_draft_reply(body='x', attachments=[])) + asyncio.run(_provider().create_draft_reply(body='x', attachments=[])) def test_download_attachment_sanitizes_filename(monkeypatch): @@ -59,28 +93,40 @@ async def fake(cmd, **kwargs): return _FakeProc() monkeypatch.setattr(gm, 'async_run', fake) - p = asyncio.run(gm.download_attachment('msg123', 'a1', '../../evil.pdf')) + p = asyncio.run(_provider().download_attachment('msg123', 'a1', '../../evil.pdf')) assert p.name == 'evil.pdf' assert STAGING_DIR in p.parents assert '../../evil.pdf' not in captured['cmd'] +def test_download_attachment_namespaced_by_account(monkeypatch): + async def fake(cmd, **kwargs): + return _FakeProc() + + monkeypatch.setattr(gm, 'async_run', fake) + p1 = asyncio.run(_provider('a@x.com').download_attachment('m1', 'a1', 'inv.pdf')) + p2 = asyncio.run(_provider('b@y.com').download_attachment('m1', 'a1', 'inv.pdf')) + assert p1 != p2 + assert 'a@x.com' in p1.parts and 'b@y.com' in p2.parts + + def test_download_attachment_rejects_dotdot(monkeypatch): async def fake(cmd, **kwargs): return _FakeProc() monkeypatch.setattr(gm, 'async_run', fake) with pytest.raises(ValueError): - asyncio.run(gm.download_attachment('m1', 'a1', '..')) + asyncio.run(_provider().download_attachment('m1', 'a1', '..')) def test_gmail_ids_reject_flag_like_values(): # ids that could be read as CLI flags (argument injection) are refused + provider = _provider() with pytest.raises(ValueError): - asyncio.run(gm.download_attachment('-x', 'a1', 'inv.pdf')) + asyncio.run(provider.download_attachment('-x', 'a1', 'inv.pdf')) with pytest.raises(ValueError): - asyncio.run(gm.download_attachment('m1', '-y', 'inv.pdf')) + asyncio.run(provider.download_attachment('m1', '-y', 'inv.pdf')) with pytest.raises(ValueError): - asyncio.run(gm.fetch_message('-o')) + asyncio.run(provider.fetch_message('-o')) with pytest.raises(ValueError): - asyncio.run(gm.archive_thread('--remove=INBOX')) + asyncio.run(provider.archive_thread('--remove=INBOX')) diff --git a/tests/test_imap.py b/tests/test_imap.py new file mode 100644 index 0000000..5ffb004 --- /dev/null +++ b/tests/test_imap.py @@ -0,0 +1,75 @@ +import asyncio +from types import SimpleNamespace + +import pytest + +from src.invoices.config import STAGING_DIR +from src.invoices.imap import ImapProvider +from src.invoices.models import EmailAccount + + +def _provider() -> ImapProvider: + return ImapProvider( + EmailAccount( + address='u@elsewhere.com', + provider='imap', + imap_host='imap.elsewhere.com', + imap_password_env='ELSEWHERE_PW', + ) + ) + + +def _msg(uid='7', filename='inv.pdf', payload=b'%PDF'): + att = SimpleNamespace( + filename=filename, payload=payload, content_type='application/pdf' + ) + return SimpleNamespace( + uid=uid, + subject='Invoice 1', + from_='vendor@x.com', + date=None, + text='hello', + html='', + attachments=[att], + ) + + +def test_normalize_shapes_message(): + norm = _provider()._normalize(_msg()) + assert norm['id'] == '7' and norm['threadId'] == '7' + assert norm['headers']['subject'] == 'Invoice 1' + assert norm['headers']['from'] == 'vendor@x.com' + assert norm['body'] == 'hello' + assert norm['attachments'] == [ + {'filename': 'inv.pdf', 'attachmentId': '0', 'mimeType': 'application/pdf'} + ] + + +def test_fetch_thread_wraps_single_message(): + provider = _provider() + provider._messages['7'] = _msg() + thread = asyncio.run(provider.fetch_thread('7')) + assert [m['id'] for m in thread['messages']] == ['7'] + + +def test_download_attachment_sanitizes_and_namespaces(): + provider = _provider() + provider._messages['7'] = _msg(filename='../../evil.pdf') + path = asyncio.run(provider.download_attachment('7', '0', '../../evil.pdf')) + assert path.name == 'evil.pdf' + assert STAGING_DIR in path.parents + assert 'u@elsewhere.com' in path.parts + assert path.read_bytes() == b'%PDF' + + +def test_download_attachment_rejects_non_numeric_attachment_id(): + provider = _provider() + provider._messages['7'] = _msg() + with pytest.raises(ValueError): + asyncio.run(provider.download_attachment('7', '0; rm -rf /', 'inv.pdf')) + + +def test_login_requires_password_env(monkeypatch): + monkeypatch.delenv('ELSEWHERE_PW', raising=False) + with pytest.raises(RuntimeError, match='IMAP password'): + _provider()._login() diff --git a/tests/test_providers.py b/tests/test_providers.py new file mode 100644 index 0000000..e1b51f0 --- /dev/null +++ b/tests/test_providers.py @@ -0,0 +1,164 @@ +import asyncio + +import pytest + +import src.invoices.providers as pr +import src.mcp_server as mcp +from src.invoices.config import _parse_email_accounts +from src.invoices.gmail import GmailProvider +from src.invoices.imap import ImapProvider +from src.invoices.models import EmailAccount + + +class _FakeProvider(pr.MailProvider): + def __init__(self, address, result=None): + super().__init__(EmailAccount(address=address)) + self._result = result or [] + + async def search_messages(self, max_emails, query=None): + if isinstance(self._result, Exception): + raise self._result + return self._result + + async def fetch_message(self, message_id): + return {} + + async def fetch_thread(self, thread_id): + return {} + + async def download_attachment(self, message_id, attachment_id, filename): + raise NotImplementedError + + async def archive_thread(self, thread_id): + pass + + +# ── config parsing ─────────────────────────────────────────────────────────── + + +def test_parse_email_accounts_defaults_to_gog_default(): + assert _parse_email_accounts({}) == [EmailAccount(address='')] + + +def test_parse_email_accounts_shapes(): + cfg = { + 'email_accounts': [ + 'plain@example.com', + {'address': 'g@work.com'}, + { + 'address': 'u@elsewhere.com', + 'provider': 'imap', + 'imap': { + 'host': 'imap.elsewhere.com', + 'port': 143, + 'password_env': 'X_PW', + }, + }, + ] + } + plain, gmail, imap = _parse_email_accounts(cfg) + assert plain == EmailAccount(address='plain@example.com') + assert gmail.provider == 'gmail' and gmail.address == 'g@work.com' + assert imap.provider == 'imap' + assert imap.imap_host == 'imap.elsewhere.com' and imap.imap_port == 143 + assert imap.imap_username == '' # defaults to address at login time + assert imap.imap_folder == 'INBOX' and imap.imap_archive_folder == 'Archive' + + +# ── provider construction ──────────────────────────────────────────────────── + + +def test_build_providers_maps_types(monkeypatch): + monkeypatch.setattr( + pr, + 'EMAIL_ACCOUNTS', + [ + EmailAccount(address='g@x.com'), + EmailAccount(address='i@y.com', provider='imap'), + EmailAccount(address='p@z.com', provider='pigeon'), # unknown: skipped + ], + ) + providers = pr.build_providers() + assert len(providers) == 2 + assert isinstance(providers[0], GmailProvider) + assert isinstance(providers[1], ImapProvider) + + +# ── multi-account search ───────────────────────────────────────────────────── + + +def test_search_all_accounts_merges_and_tags(): + p1 = _FakeProvider('a@x.com', [{'id': '1', 'threadId': 't1'}]) + p2 = _FakeProvider('b@y.com', [{'id': '1', 'threadId': '1'}]) + msgs = asyncio.run(pr.search_all_accounts([p1, p2], None)) + assert len(msgs) == 2 + assert msgs[0]['account'] == 'a@x.com' and msgs[0]['provider'] is p1 + assert msgs[1]['account'] == 'b@y.com' and msgs[1]['provider'] is p2 + + +def test_search_all_accounts_isolates_failures(): + p1 = _FakeProvider('a@x.com', RuntimeError('boom')) + p2 = _FakeProvider('b@y.com', [{'id': '2', 'threadId': 't2'}]) + msgs = asyncio.run(pr.search_all_accounts([p1, p2], None)) + assert [m['id'] for m in msgs] == ['2'] + + +def test_search_all_accounts_caps_total(): + p1 = _FakeProvider( + 'a@x.com', [{'id': str(i), 'threadId': str(i)} for i in range(3)] + ) + p2 = _FakeProvider('b@y.com', [{'id': 'z', 'threadId': 'z'}]) + msgs = asyncio.run(pr.search_all_accounts([p1, p2], 2)) + assert len(msgs) == 2 + + +# ── staging isolation ──────────────────────────────────────────────────────── + + +def test_staging_target_is_namespaced_and_sanitized(): + d, name = _FakeProvider('a@x.com')._staging_target('42', '../../evil.pdf') + assert name == 'evil.pdf' + assert d.parts[-2:] == ('a@x.com', '42') + # Same IMAP UID on two accounts cannot clobber each other + d2, _ = _FakeProvider('b@y.com')._staging_target('42', 'evil.pdf') + assert d != d2 + + +def test_staging_target_rejects_bad_names(): + with pytest.raises(ValueError): + _FakeProvider('a@x.com')._staging_target('42', '..') + with pytest.raises(ValueError): + _FakeProvider('a@x.com')._staging_target('42', '') + + +# ── MCP account routing ────────────────────────────────────────────────────── + + +def test_resolve_provider_single_account_needs_no_address(): + # The test config has no email_accounts, so there is exactly one provider + provider = mcp._resolve_provider(None) + assert isinstance(provider, GmailProvider) and provider.address == '' + + +def test_resolve_provider_multiple_accounts(monkeypatch): + g1 = GmailProvider(EmailAccount(address='a@x.com')) + g2 = GmailProvider(EmailAccount(address='b@y.com')) + monkeypatch.setattr(mcp, 'build_providers', lambda: [g1, g2]) + with pytest.raises(ValueError, match='pass account='): + mcp._resolve_provider(None) + assert mcp._resolve_provider('a@x.com') is g1 + # Unknown address: falls back to a Gmail provider (gog may still be authed) + fallback = mcp._resolve_provider('ghost@x.com') + assert isinstance(fallback, GmailProvider) and fallback.address == 'ghost@x.com' + + +def test_resolve_gmail_provider_for_drafts(monkeypatch): + gmail = GmailProvider(EmailAccount(address='a@x.com')) + imap = ImapProvider(EmailAccount(address='i@y.com', provider='imap', imap_host='h')) + monkeypatch.setattr(mcp, 'build_providers', lambda: [gmail, imap]) + # The lone Gmail account is picked even though two accounts exist + assert mcp._resolve_gmail_provider(None) is gmail + + monkeypatch.setattr(mcp, 'build_providers', lambda: [imap]) + with pytest.raises(ValueError, match='No Gmail account'): + mcp._resolve_gmail_provider(None) diff --git a/uv.lock b/uv.lock index 944e02c..0925113 100644 --- a/uv.lock +++ b/uv.lock @@ -200,6 +200,7 @@ dependencies = [ { name = "anyio" }, { name = "click" }, { name = "httpx" }, + { name = "imap-tools" }, { name = "mcp" }, { name = "pyyaml" }, { name = "rich" }, @@ -215,6 +216,7 @@ requires-dist = [ { name = "anyio", specifier = ">=4.12.1" }, { name = "click" }, { name = "httpx" }, + { name = "imap-tools" }, { name = "mcp", specifier = ">=1.27.2" }, { name = "pyyaml" }, { name = "rich" }, @@ -278,6 +280,15 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/0e/61/66938bbb5fc52dbdf84594873d5b51fb1f7c7794e9c0f5bd885f30bc507b/idna-3.11-py3-none-any.whl", hash = "sha256:771a87f49d9defaf64091e6e6fe9c18d4833f140bd19464795bc32d966ca37ea", size = 71008, upload-time = "2025-10-12T14:55:18.883Z" }, ] +[[package]] +name = "imap-tools" +version = "1.13.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/f7/cb/76d8697739439be6dd0261db5a27c945fb6a43e054f2d2e90283be502058/imap_tools-1.13.0.tar.gz", hash = "sha256:0da0d72c921a724cba09b959bad9bfaf60bca537a697e69a076fdf607ef5775c", size = 47683, upload-time = "2026-05-12T07:14:54.488Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/19/a8/0f58c13d2660d5fc8f808ce8b46828d2941752ec21e4015bde99c08b37d7/imap_tools-1.13.0-py3-none-any.whl", hash = "sha256:656c37beba22ab2929b73c07d0ca397ae8805b670d390b1127723e3335244e6d", size = 35849, upload-time = "2026-05-12T07:14:52.669Z" }, +] + [[package]] name = "iniconfig" version = "2.3.0"