Batch incremental IMAP UID fetch ranges#313
Open
MaxRink wants to merge 72 commits into
Open
Conversation
…009-979d-6da0331bff63 - Failed to export an email. Aborting process... rustmailer#275
- Treat MIME parts with Content-ID but no Content-Disposition as inline - Add account_ids filter to CLI export search to avoid pulling all accounts - Skip failed emails during export instead of aborting the entire batch
When the request returns an empty body, choose to skip it and print the account ID and UID information, leaving it for the user to investigate themselves. Otherwise, the IMAP download process will be blocked by this.
This commit adds support for IMAP servers that don't provide UIDVALIDITY, such as Tencent Enterprise Mail (腾讯企业邮箱). Changes: - Added `generate_synthetic_uidvalidity()` function that creates a stable hash-based UIDVALIDITY from the mailbox name - Modified `reconcile_mailboxes()` to use synthetic UIDVALIDITY when the server doesn't provide one - Servers without UIDVALIDITY can now sync all mailboxes including system folders (Sent Messages, Drafts, Deleted Messages) - Incremental sync is supported via the synthetic UIDVALIDITY - Added warning logs to indicate when synthetic UIDVALIDITY is in use - Updated mailbox metadata to store the resolved UIDVALIDITY Fixes issues with: - Tencent Enterprise Mail (腾讯企业邮箱) - Other non-compliant IMAP servers - Mailboxes that don't properly support UIDVALIDITY"
…exing Use (account_id, mailbox_id, content_hash) as dedup key with time-based eviction to bound memory at ~50MB. Check happens before mail parsing, so duplicate emails skip all expensive work entirely. - Populate cache from Tantivy FAST columns on startup (7-day window) - Evict oldest 1/4 of entries when exceeding 300K capacity - Graceful degradation: populate failure → empty cache, still works
Previously, do_migrate_segment created a fresh NewIndexWriter (and
therefore a new Fjall Database) on every call, meaning the Fjall
database at bichon-storage/ was opened and closed once per segment.
This caused the migration to fail mid-way through (observed at segment
9/16) with:
Storage(InvalidTag(("ChecksumType", 171)))
Root cause: after segment N writes email blobs via Fjall's ingestion
API (start_ingestion / write / finish), those SSTables and KV-separated
blob files are flushed to disk and the Database is dropped. When segment
N+1 calls Database::builder(storage_dir).open(), Fjall must discover and
catalog all on-disk files produced by the previous segments. During that
discovery it reads SSTable or blob-file block headers and encounters a
ChecksumType discriminant byte (171 / 0xAB) that lsm-tree 3.1.4 does
not recognise, causing the fatal error.
The first N segments succeed because the cumulative set of ingested
SSTables stays small enough that Fjall does not need to read the
offending headers during reopen. Once enough data has accumulated the
reopen triggers a manifest or compaction read that exposes the mismatch.
Fix: open NewIndexWriter once, before the segment loop, and pass a
&mut reference into each do_migrate_segment call. finish_writers() is
called a single time after all segments complete. The Fjall Database
stays open for the entire migration and is never closed and reopened,
eliminating the incompatible-reopen path entirely.
fix: open NewIndexWriter once across all migration segments
…dity fix: Add fallback UIDVALIDITY support for non-compliant IMAP servers
update_envelope_tags lost f_body, f_from_text, f_to_text, f_cc_text, f_bcc_text, f_attachment_name_text and f_attachment_name_exact because they are not STORED and field_values() skipped them during delete+add. Rebuild these from stored counterparts and the blob-store EML.
…th regex validation
- Make MBOX/PST upload size limits configurable via SETTINGS
(bichon_web_mbox_upload_limit_mb defaults to 1 GB,
bichon_web_pst_upload_limit_mb defaults to 2 GB)
…pansion feat: expand proxy provider support
MaxRink
marked this pull request as ready for review
July 7, 2026 12:59
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
start:*behavior only as a fallback when UIDNEXT is unavailable*Why
Large backlogs can currently use one long-lived
UID FETCH start:*stream. If the server stalls or closes the stream mid-backlog, scheduled sync can remain behind without making reliable progress.Test
BICHON_ROOT_DIR=$(mktemp -d) cargo test -p bichon-core uid_range_batches_do_not_use_unbounded_fetch -- --nocaptureDraft because this was reproduced against a live stalled Bichon instance, and I want upstream feedback on whether they prefer preserving progress semantics for skipped/empty batches differently.