Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,15 @@
# Changelog

## Unreleased

Added:
- Multiple email accounts. List mailboxes under `email_accounts` in config.yml: Google-hosted addresses on any domain (via gog) and any other mailbox via IMAP. One run searches them all, and one failing account no longer stops the rest. IMAP passwords come from environment variables, never from config.
- Per-account setup checks. Onboarding and health_check verify gog auth for each Google account, warn when a gmail-configured domain is not actually Google-hosted, note the "External" OAuth client requirement for multi-domain setups, and login-test IMAP accounts.

Changed:
- Message, thread, and attachment ids are account-scoped. The mailbox MCP tools (search_inbox, get_message, download_attachment, archive_thread, draft_reply) take an optional `account` parameter; it is only needed when several accounts are configured.
- Staged attachment downloads are namespaced by account as well as message, so same-named PDFs from different mailboxes cannot overwrite each other.

## 0.1.1

Fixes and docs. No breaking changes.
Expand Down
15 changes: 12 additions & 3 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,13 @@ pyproject.toml # dependencies + ruff config
./fetch invoices generate-payments
```

## Gmail
## Email access

Mailboxes are configured under `email_accounts` in `config.yml`. Each is served
by a `MailProvider` (`providers.py`): `GmailProvider` (`gmail.py`, gog CLI) for
Google-hosted addresses, `ImapProvider` (`imap.py`, imap-tools) for everything
else. Message/thread/attachment IDs are account-scoped — always go through the
provider the message came from (records carry an `account` field).

When working **on this codebase**, route Gmail through the `gog gmail` CLI, not a
Gmail MCP connector, so calls go through the audited `helpers.async_run` wrapper
Expand All @@ -32,12 +38,14 @@ end users: a host agent (e.g. Cowork) may use its own Gmail connector for
search / read / draft / archive. The one job that still needs `gog` is landing an
attachment's bytes on disk (`gog gmail attachment`), since connectors return an
attachment id but not its content, and the rest of the pipeline reads real files.
Pass `-a <address>` to select an account.

```bash
gog gmail search "is:unread" --json
gog -a you@example.com gmail search "is:unread" --json
gog gmail get <messageId> --json
gog gmail thread get <threadId> --full --json
gog gmail attachment <messageId> <attachmentId> --out <dir> --name <filename>
gog auth list --json
gog gmail -h
```

Expand All @@ -58,8 +66,9 @@ uvx ruff format src/

## Key conventions

- Config values (companies, destination folders, debtor accounts) live in `config.yml`, not hard-coded
- Config values (companies, destination folders, debtor accounts, email accounts) live in `config.yml`, not hard-coded
- `DEFAULT_SLUG` in `config.yml` is the fallback company slug
- IMAP passwords come from env vars (`imap.password_env`), never from config.yml
- Classification uses `claude` CLI with haiku model (`classify.py`)
- Models are plain dataclasses in `models.py`
- All async subprocess calls go through `helpers.py` (`async_run`, `async_run_json`)
43 changes: 38 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,14 +11,14 @@ Small companies (and their agents) that get a periodic "please send the invoices
## What it does

1. Reads the accountant's list, in any format (pasted text, a table, an email).
2. Finds each invoice. First in your Gmail inbox. For anything not there, it hands your agent a clear task to fetch it from the vendor's portal or e-invoice platform.
2. Finds each invoice. First in your email (one mailbox or several, Gmail or IMAP). For anything not there, it hands your agent a clear task to fetch it from the vendor's portal or e-invoice platform.
3. Drafts a reply to the accountant with the found PDFs attached, plus a note on anything still missing. You review the draft and send it.

It can also classify and file invoices, copy them to a folder of your choice (Dropbox, a shared drive, or any local folder), and build a SEPA payment file. See the tool list below.

## How it stays safe

- No stored credentials. It uses your local `gog` (Gmail) and `claude` CLIs.
- No stored credentials. It uses your local `gog` (Gmail) and `claude` CLIs. IMAP passwords come from environment variables, never from config.
- It drafts, it does not send. Outgoing replies are Gmail drafts you review first.
- Local audit log. Every fetch, download, and draft is recorded in `output/audit.log`.

Expand Down Expand Up @@ -93,12 +93,12 @@ uv sync
|------|--------------|
| `health_check` | Check gog, claude, Gmail auth, and config.yml |
| `parse_missing_list` | Turn a free-form missing list into a checklist |
| `search_inbox`, `get_message`, `download_attachment` | Find and pull invoice PDFs from Gmail |
| `search_inbox`, `get_message`, `download_attachment` | Find and pull invoice PDFs from your mailboxes |
| `plan_retrieval` | Build tasks to fetch invoices that are not in the inbox |
| `draft_reply` | Draft a reply to the accountant with files attached (never sends) |
| `read_audit` | Read the local audit log |
| `reconcile`, `build_report` | Optional: match the list to what you found and write a summary |
| `fetch_invoices`, `classify_invoice`, `list_invoices` | Fetch and classify invoices from Gmail |
| `fetch_invoices`, `classify_invoice`, `list_invoices` | Fetch and classify invoices from all your mailboxes |
| `copy_to_dropbox`, `generate_payments`, `archive_thread` | File to a folder (Dropbox or any path), build SEPA payments, archive threads |

Full reference and the step-by-step workflow: [SKILL.md](SKILL.md).
Expand Down Expand Up @@ -136,6 +136,36 @@ token without your own Cloud project is Google's OAuth Playground.

Either way, `health_check` tells you whether Gmail is connected.

## Multiple email accounts

By default Fetch searches the Gmail account `gog` is signed in to. To search several mailboxes in one run, list them under `email_accounts` in `config.yml`:

```yaml
email_accounts:
# Google-hosted mailboxes (gmail.com or any Workspace domain)
- address: you@example.com
- address: info@other-domain.com

# Any other mailbox via IMAP
- address: user@elsewhere.com
provider: imap
imap:
host: imap.elsewhere.com
port: 993 # optional, default 993
username: user@elsewhere.com # optional, defaults to address
password_env: ELSEWHERE_IMAP_PASSWORD
folder: INBOX # optional
archive_folder: Archive # optional, used when archiving
```

Google accounts need `gog auth add <address>` once each. The domain does not matter, only that the mailbox is hosted on Google. When accounts span more than one Workspace domain, create the OAuth client as "External" and add each address as a test user while the app is unverified.

IMAP passwords are never stored in `config.yml`. Set `password_env` to the name of an environment variable and export it before running. For providers with two-factor auth, use an app password.

Searches cover all accounts at once, and one failing account does not stop the others. Message and thread ids are scoped to the account they came from, so the MCP tools take an optional `account` parameter (only needed when several accounts are configured). Draft replies always go through a Google account.

`./fetch onboarding` (or the `health_check` tool) checks all of this: gog auth for each Google account, whether each Gmail-configured domain is actually Google-hosted, and IMAP connectivity.

## Use it inside Cowork (or any agent that already has Gmail)

If your agent already has its own Gmail connector (for example Claude Cowork's
Expand Down Expand Up @@ -198,6 +228,7 @@ debtor_accounts:
bic: "LHVBEE22"

vendor_sources: {} # optional, see "Fetch invoices that are not in your inbox"
email_accounts: [] # optional, see "Multiple email accounts"
```

Point at a config elsewhere with `FETCH_CONFIG`. Change the output folder with `FETCH_OUTPUT_DIR`.
Expand Down Expand Up @@ -238,7 +269,9 @@ src/
reconcile.py Parse a missing list, match it, build a report
retrieval.py Plan fetching invoices not in the inbox
audit.py Local audit log
gmail.py Gmail search, download, draft reply (via gog)
providers.py Mailbox abstraction + multi-account search
gmail.py Google-hosted mailboxes (via gog), draft replies
imap.py Any other mailbox via IMAP
classify.py AI classification (claude)
payment_xml.py SEPA payment XML
config.py, models.py, helpers.py, process.py, dropbox.py, summary.py
Expand Down
24 changes: 17 additions & 7 deletions SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,10 @@ This server stores **no passwords or tokens**. It runs locally and shells out to
- **`gog`**: your Gmail session (search, fetch attachments, draft replies).
- **`claude`**: classifies each PDF (runs the `haiku` model locally).

Non-Google mailboxes connect over IMAP instead; the password comes from an
environment variable named in `config.yml` (`imap.password_env`), never from
the config file itself.

If those aren't set up, every tool can tell you: call `health_check` first.

**If you already have your own Gmail connector** (e.g. running inside Cowork), use
Expand Down Expand Up @@ -63,26 +67,32 @@ only Dropbox), and your debtor bank accounts. Verify with

| Tool | Purpose |
|------|---------|
| `health_check()` | Check gog CLI, claude CLI, Gmail auth, and config.yml. Returns `{ok, checks[]}`. Call this first. |
| `search_inbox(query?, max_results?)` | Raw Gmail search, returns `{query, count, messages[{id, threadId}]}`. `query` defaults to the built-in invoice search. |
| `get_message(message_id)` | One message's headers plus `attachments[{attachment_id, filename, mime_type}]`. Use it to pick which PDF to pull. |
| `download_attachment(message_id, attachment_id, filename)` | Download one attachment to staging, returns `{path, filename, exists}`. |
| `fetch_invoices(max_emails?, query?)` | The full pipeline: search Gmail, download PDFs, classify, dedupe, file under `output/`, write `invoices.json` and `SUMMARY.md`. Returns `{count, invoices[], output_dir, summary_md, invoices_json}`. |
| `health_check()` | Check gog CLI, claude CLI, per-account mail auth (Gmail and IMAP), and config.yml. Returns `{ok, checks[], warnings[]}`. Call this first. |
| `search_inbox(query?, max_results?, account?)` | Search every configured mailbox (or one `account`), returns `{query, count, messages[{id, threadId, account}]}`. `query` defaults to the built-in invoice search; IMAP accounts run it as plain text. |
| `get_message(message_id, account?)` | One message's headers plus `attachments[{attachment_id, filename, mime_type}]`. Use it to pick which PDF to pull. |
| `download_attachment(message_id, attachment_id, filename, account?)` | Download one attachment to staging, returns `{path, filename, exists}`. |
| `fetch_invoices(max_emails?, query?)` | The full pipeline: search all mailboxes, download PDFs, classify, dedupe, file under `output/`, write `invoices.json` and `SUMMARY.md`. Returns `{count, invoices[], output_dir, summary_md, invoices_json}`. |
| `classify_invoice(pdf_path, subject?, sender?, snippet?)` | Classify one local PDF (e.g. one you downloaded yourself) into the same record shape. Does **not** move the file. |
| `list_invoices(status?, company?)` | Read back the last fetch from `invoices.json`, filtered by status (`Paid`/`To-Pay`) and/or company slug. No network. |
| `copy_to_dropbox(status?, companies?)` | Copy filed PDFs into each company's configured folder (`dropbox_dirs` in config.yml, any folder and not only Dropbox). Returns `{copied, skipped, errors, details[]}`. |
| `generate_payments(companies?, execution_date?)` | Build SEPA `pain.001.001.03` XML for To-Pay invoices, one file per debtor company. Derives missing BICs from Estonian IBANs. Returns `{count, files[], derived_bics[], skipped[]}`. |
| `archive_thread(thread_id)` | Remove the INBOX label from a Gmail thread. |
| `archive_thread(thread_id, account?)` | Remove a thread from its account's inbox. |
| `parse_missing_list(text)` | Turn an accountant's free-form missing/expected-invoice list (pasted text, CSV, table, email) into a structured checklist, returns `{count, items[]}`. |
| `reconcile(checklist, collected?)` | Match the checklist against collected invoices (or the last fetch if `collected` omitted). Returns `{summary, matched[], still_missing[], unmatched_collected[]}`. Deterministic. |
| `build_report(reconciliation, title?)` | Render a reconcile result as Markdown, write `REPORT.md`, return `{report_md, path}`. Optional summary for the accountant. |
| `plan_retrieval(items)` | For invoices NOT in the inbox, return retrieval tasks per item (vendor, identifiers, any `config.yml` per-vendor recipe, suggested sources, instructions). You fetch each with your own browser, then attach via `draft_reply`. |
| `draft_reply(body, attachments?, reply_to_message_id?, to?, subject?)` | Create a **draft** Gmail reply to the accountant with the invoice files attached. Never auto-sends; a human reviews and sends. |
| `draft_reply(body, attachments?, reply_to_message_id?, to?, subject?, account?)` | Create a **draft** Gmail reply to the accountant with the invoice files attached. Never auto-sends; a human reviews and sends. |
| `read_audit(limit?)` | Read the local audit log (`output/audit.log`). Every fetch, download, and draft is recorded. |

Statuses are case-insensitive (`paid`, `to-pay`, `all`). Company values are the
slugs from `config.yml`.

**Multiple accounts.** With `email_accounts` configured in `config.yml`, message,
thread, and attachment ids are scoped to the account they came from. Pass each
message's `account` (from `search_inbox`) back to `get_message`,
`download_attachment`, and `archive_thread`. With a single account, omit it.
Drafts always go through a Google-hosted account.

## Main workflow: clear a missing-invoice list

The core job: the accountant emails a list of invoices they're missing, you fetch
Expand Down
18 changes: 18 additions & 0 deletions config.example.yml
Original file line number Diff line number Diff line change
Expand Up @@ -35,3 +35,21 @@ vendor_sources:
# login_hint: "log in with the company Google account; invoices under Billing > History"
# einvoice_operator: "Envoice"
# notes: "monthly subscription, issued on the 1st"

# Optional. Mailboxes to search. Without this, the Gmail account `gog` is
# signed in to is used. Google-hosted addresses (gmail.com or any Workspace
# domain) need `gog auth add <address>` once each. Anything else goes through
# IMAP; the password comes from the env var named in `password_env`, never
# from this file (use an app password for providers with two-factor auth).
email_accounts:
# - address: you@example.com
# - address: info@other-domain.com
# - address: user@elsewhere.com
# provider: imap
# imap:
# host: imap.elsewhere.com
# port: 993 # optional, default 993
# username: user@elsewhere.com # optional, defaults to address
# password_env: ELSEWHERE_IMAP_PASSWORD
# folder: INBOX # optional
# archive_folder: Archive # optional, used when archiving
1 change: 1 addition & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ dependencies = [
"click",
"httpx",
"pyyaml",
"imap-tools",
"mcp>=1.27.2",
"anyio>=4.12.1",
]
Expand Down
39 changes: 26 additions & 13 deletions src/invoices/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,15 +13,16 @@

from .config import BASE_DIR, COMPANIES, DEBTOR_ACCOUNTS, require_config
from .dropbox import copy_to_dropbox
from .gmail import archive_thread, create_directories, search_messages
from .gmail import GmailProvider
from .helpers import console
from .models import InvoiceRecord
from .models import EmailAccount, InvoiceRecord
from .payment_xml import (
fill_missing_bics,
generate_payment_xml,
incomplete_beneficiaries,
)
from .process import process_messages
from .process import create_directories, process_messages
from .providers import build_providers, search_all_accounts
from .reconcile import build_report, parse_missing_list
from .reconcile import reconcile as reconcile_items
from .summary import write_summary
Expand All @@ -37,7 +38,7 @@ def invoices():
'--max-emails', default=None, type=int, help='Limit number of emails to process'
)
def fetch(max_emails: int | None):
"""Fetch invoices from Gmail, classify, and organize."""
"""Fetch invoices from your mailboxes, classify, and organize."""
# Save terminal state before asyncio takes over signal handling
saved_attrs = None
if sys.stdin.isatty():
Expand All @@ -54,7 +55,8 @@ def fetch(max_emails: int | None):
async def _main(max_emails: int | None):
require_config()
create_directories()
message_dicts = await search_messages(max_emails)
providers = build_providers()
message_dicts = await search_all_accounts(providers, max_emails)

if not message_dicts:
console.print('\n[bold]No messages found. Done.[/bold]')
Expand Down Expand Up @@ -252,12 +254,14 @@ def archive(
console.print('[yellow]No invoices match the given filters.[/yellow]')
return

# Deduplicate by thread_id — group records per thread
threads: dict[str, list[InvoiceRecord]] = {}
# Deduplicate by (account, thread_id) — group records per thread.
# Thread IDs are account-scoped, so archiving must go through the
# provider the record came from.
threads: dict[tuple[str, str], list[InvoiceRecord]] = {}
for r in records:
if not r.thread_id:
continue
threads.setdefault(r.thread_id, []).append(r)
threads.setdefault((r.account, r.thread_id), []).append(r)

if not threads:
console.print('[yellow]No threads found (records missing thread_id).[/yellow]')
Expand All @@ -267,6 +271,8 @@ def archive(
if max_emails is not None:
thread_items = thread_items[:max_emails]

providers = {p.address: p for p in build_providers()}

console.print(f'\nFound [bold]{len(thread_items)}[/bold] thread(s) to review.\n')

archived = 0
Expand All @@ -277,29 +283,36 @@ def archive(
saved_attrs = termios.tcgetattr(sys.stdin)

try:
for thread_id, thread_records in thread_items:
for (account, thread_id), thread_records in thread_items:
# Records from accounts no longer in config fall back to a
# Gmail provider for that address (gog may still be authed)
provider = providers.get(account) or GmailProvider(
EmailAccount(address=account)
)
first = thread_records[0]
pdf_names = ', '.join(r.renamed_pdf for r in thread_records)
amount_str = (
f'{first.currency_code} {first.amount_inc_vat}'
if first.amount_inc_vat
else 'N/A'
)
gmail_link = f'https://mail.google.com/mail/u/0/#inbox/{thread_id}'

date_str = first.doc_date or 'N/A'

panel_text = (
f'[bold]Subject:[/bold] {first.subject}\n'
f'[bold]From:[/bold] {first.sender}\n'
f'[bold]Date:[/bold] {date_str}\n'
f'[bold]Account:[/bold] {provider.label}\n'
f'[bold]Status:[/bold] {first.status}\n'
f'[bold]Company:[/bold] {first.company}\n'
f'[bold]Amount:[/bold] {amount_str}\n'
f'[bold]Summary:[/bold] {first.summary}\n'
f'[bold]PDF(s):[/bold] {pdf_names}\n'
f'[bold]Thread:[/bold] {gmail_link}'
f'[bold]PDF(s):[/bold] {pdf_names}'
)
if isinstance(provider, GmailProvider):
gmail_link = f'https://mail.google.com/mail/u/0/#inbox/{thread_id}'
panel_text += f'\n[bold]Thread:[/bold] {gmail_link}'
console.print(Panel(panel_text, title=f'Thread {thread_id[:12]}…'))

if dry_run:
Expand All @@ -310,7 +323,7 @@ def archive(
should_archive = yes or Confirm.ask(' Archive this thread?', default=False)

if should_archive:
asyncio.run(archive_thread(thread_id))
asyncio.run(provider.archive_thread(thread_id))
console.print('[green] Archived.[/green]\n')
archived += 1
else:
Expand Down
Loading
Loading