feat(mail): Phase 4 archive jobs by zoorpha · Pull Request #159 · kubedoio/rustshare

zoorpha · 2026-07-10T23:18:03Z

Implements RustShare Mail Phase 4: folder/date-range IMAP archive jobs with incremental sync, retention, retry/backoff, and failure handling.

Scope:

Extends mail_import_jobs with source_mode = 'imap_archive'.
Adds archive_job_id scoping to mail_messages for retention.
MetadataStore CRUD, service methods, background worker dispatch, and REST endpoints.
Audit events for create/start/complete/fail.
Domain + integration tests; mock IMAP unit tests.

Refs #147

Design doc: docs/plans/2026-07-09-mail-phase4-archive-jobs-plan.md

Note: this touches migrations, external IMAP imports, retention/soft-delete, and tenant/owner scoping; per project safety boundaries it should have human review before merge.

Refs #147 Signed-off-by: Senol Colak <senol@kubedo.com>

- Add create_archive_job, list_archive_jobs, get_archive_job, cancel_archive_job, delete_archive_job, and process_archive_job to MailService. - Add update_mail_import_job_status to MetadataStore for atomic status transitions. - Add FromStr impl for MailImportJobStatus to support status parsing. - Update SQLx offline query cache. Refs #147 Signed-off-by: Senol Colak <senol@kubedo.com>

Refs #147 Signed-off-by: Senol Colak <senol@kubedo.com>

- Advance the UID watermark across IMAP gaps while stopping at the first failure, so deleted-message gaps do not stall progress. - Propagate retention cleanup failures instead of swallowing them. - Log event-store append failures for MailImported, MailArchiveJobCompleted, MailArchiveJobFailed, and MailArchiveJobStarted events. - Cap max_retries validation at 100 in the archive-job handler. - Add integration test for non-contiguous UID watermark advancement. Refs #147 Signed-off-by: Senol Colak <senol@kubedo.com>

The test created two independent mock sessions, so the fetch counter on the first session only counted one fetch. Capture the second session's counter and assert the total fetch count across both sessions. Refs #147 Signed-off-by: Senol Colak <senol@kubedo.com>

Signed-off-by: Senol Colak <senol@kubedo.com>

senolcolak · 2026-07-11T12:02:53Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9a18cf5d2e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Signed-off-by: Senol Colak <senol@kubedo.com>

senolcolak · 2026-07-11T12:21:01Z

@codex review

chatgpt-codex-connector · 2026-07-11T12:21:08Z

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, you can upgrade your account or add credits to your account and enable them for code reviews in your settings.

senolcolak · 2026-07-11T13:38:42Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7545d7558a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Signed-off-by: Senol Colak <senol@kubedo.com>

zoorpha · 2026-07-11T14:15:32Z

@codex review

chatgpt-codex-connector · 2026-07-11T14:15:37Z

To use Codex here, create a Codex account and connect to github.

senolcolak · 2026-07-11T14:17:27Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 74f1a7a4f8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…jobs - Skip archive retention when the job is no longer running, so cancelling a scan does not delete aged messages. - Reassign archived messages from a deleted archive job to a new archive job so the new job's retention policy applies; active overlapping jobs keep their ownership to avoid a ping-pong race. - Owner-scope the new update_mail_message_archive_job_id query. - Cancel a running archive job before soft-deleting it so a worker does not get stuck in a running state against a deleted row. - Hard-delete mail_message_parts and mail_attachments rows when applying archive retention, so internal metadata is also purged. - Add integration tests for retention-on-cancel, ownership reassignment, and deleting a running archive job; extend retention test to assert parts and attachments are removed. Refs #147 Signed-off-by: Senol Colak <senol@kubedo.com>

senolcolak · 2026-07-11T16:10:35Z

@codex review

Follow-up to the deep-review pass: - soft_delete_mail_archive_job now atomically sets status = 'cancelled' so deleting a running job cannot leave it stuck in 'running' against a deleted row. delete_archive_job drops the separate cancel call. - apply_archive_retention verifies the owning archive job is still 'running' inside its transaction, closing the gap between the service-layer guard and the retention UPDATE. - maybe_reassign_archive_message_ownership now also takes over messages from cancelled or failed archive jobs, not only deleted ones, so those messages remain under an active retention policy. - Add integration test for ownership reassignment from a cancelled job. Refs #147 Signed-off-by: Senol Colak <senol@kubedo.com>

zoorpha · 2026-07-11T16:11:52Z

All Codex review comments have been addressed and resolved.

Summary of fixes pushed:

UID watermark only advances across a contiguous successful prefix (already fixed; verified and covered by test).
Incomplete deduplication rows are reclaimed instead of skipped (already fixed; verified and covered by test).
Archive jobs reschedule after each successful scan (already fixed; verified).
Retention soft-deletes visible artifact folders/files and now also hard-deletes internal mail_message_parts/mail_attachments rows.
Completion event is gated on successful requeue (already fixed; verified).
Retention is skipped when the job is cancelled/stopped (added status guard in apply_archive_retention).
Messages owned by a deleted/cancelled/failed archive job are reassigned to the current archive job so retention keeps covering them; active jobs keep ownership to avoid ping-pong.
Deleting a running archive job atomically cancels it in the same UPDATE, preventing a stuck running state.

Verification run:

cargo fmt --check
SQLX_OFFLINE=true cargo clippy --all-features -- -D warnings
SQLX_OFFLINE=true cargo check --workspace
SQLX_OFFLINE=true cargo test --workspace --lib
cargo sqlx prepare --workspace --check

All pass. Per project safety boundaries this still needs human review before merge (external IMAP imports, retention/soft-delete, storage queries).

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: dc0f354d29

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

IMAP SINCE/BEFORE search criteria are date-only, so the previous DateTime<Utc> API silently dropped the time component and produced unexpected inclusions/exclusions. Change the API, domain type, migration, and IMAP client to use NaiveDate so callers get exactly the behavior the IMAP query implements. Also fixes two new Codex findings: - update_mail_message_archive_job_id is now a compare-and-swap that matches the previous archive_job_id, preventing concurrent replacement jobs from stealing ownership from each other. - apply_archive_retention verifies the owning job is still running inside the transaction, and ownership is reassigned from cancelled/failed jobs as well as deleted ones. Refs #147 Signed-off-by: Senol Colak <senol@kubedo.com>

zoorpha · 2026-07-11T16:38:19Z

Follow-up fixes pushed (6a3c092):

archive_since/archive_before are now NaiveDate end-to-end (migration, domain type, handlers, IMAP client, tests, docs). This matches the date-only semantics of IMAP SINCE/BEFORE and prevents the time-component surprises Codex flagged.
update_mail_message_archive_job_id is now a compare-and-swap on the previous archive_job_id, so concurrent replacement jobs cannot steal ownership from each other.
apply_archive_retention verifies the owning job is still running inside the transaction, and ownership is reassigned from cancelled/failed jobs as well as deleted ones.

All verification commands still pass.

…ation test The retention pass now verifies the owning archive job is still running inside the transaction. Update the integration test to mark the job running before the second pass so the aged message is soft-deleted as expected. Also remove the now-unused NaiveDate import. Signed-off-by: Senol Colak <senol@kubedo.com>

senolcolak · 2026-07-11T20:24:16Z

@codex review

chatgpt-codex-connector · 2026-07-11T20:33:37Z

Codex Review: Didn't find any major issues. Another round soon, please!

Reviewed commit: 5fa4e16a63

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

senolcolak added 28 commits July 10, 2026 20:18

feat(mail): add archive job columns to mail_import_jobs

f20ace4

Refs #147 Signed-off-by: Senol Colak <senol@kubedo.com>

feat(mail): extend MailImportJob for archive jobs

0d74ef0

Refs #147 Signed-off-by: Senol Colak <senol@kubedo.com>

feat(mail): add archive job event types

6c67267

Refs #147 Signed-off-by: Senol Colak <senol@kubedo.com>

feat(mail): add MetadataStore archive job queries

ac3eb13

Refs #147 Signed-off-by: Senol Colak <senol@kubedo.com>

fix(mail): address archive job storage review feedback

29747a8

Refs #147 Signed-off-by: Senol Colak <senol@kubedo.com>

fix(mail): restore import job failure semantics and add archive retry

35328ba

Refs #147 Signed-off-by: Senol Colak <senol@kubedo.com>

feat(mail): add IMAP date-range UID search

fe1c238

Refs #147 Signed-off-by: Senol Colak <senol@kubedo.com>

fix(mail): add timeout to IMAP date-range UID search

9505b4e

Refs #147 Signed-off-by: Senol Colak <senol@kubedo.com>

fix(mail): add tests and validation for IMAP date-range search

b92fedf

Refs #147 Signed-off-by: Senol Colak <senol@kubedo.com>

fix(mail): address archive job service review feedback

c8f7a2d

Refs #147 Signed-off-by: Senol Colak <senol@kubedo.com>

fix(mail): harden archive job service error handling and atomicity

e21aedb

Refs #147 Signed-off-by: Senol Colak <senol@kubedo.com>

fix(mail): archive job failed events and retry exhaustion

dee3fb4

Refs #147 Signed-off-by: Senol Colak <senol@kubedo.com>

docs(mail): clarify max_retries as total attempts

69e7137

Refs #147 Signed-off-by: Senol Colak <senol@kubedo.com>

feat(mail): dispatch archive jobs in import worker

c2739d1

Refs #147 Signed-off-by: Senol Colak <senol@kubedo.com>

feat(mail): add archive job handlers

4440329

Refs #147 Signed-off-by: Senol Colak <senol@kubedo.com>

feat(mail): register archive job routes

0078906

Refs #147 Signed-off-by: Senol Colak <senol@kubedo.com>

feat(mail): register archive job OpenAPI schemas

7820af9

Refs #147 Signed-off-by: Senol Colak <senol@kubedo.com>

docs(mail): regenerate OpenAPI contract for archive jobs

65cbf97

Refs #147 Signed-off-by: Senol Colak <senol@kubedo.com>

fix(mail): return 409 Conflict for invalid archive job cancellation

cc49c49

Refs #147 Signed-off-by: Senol Colak <senol@kubedo.com>

test(mail): add archive job domain tests

16eea76

Refs #147 Signed-off-by: Senol Colak <senol@kubedo.com>

test(mail): add archive job lifecycle integration test

40412e4

Refs #147 Signed-off-by: Senol Colak <senol@kubedo.com>

docs(mail): add phase 4 plan docs and changelog entry

31ef13b

Refs #147 Signed-off-by: Senol Colak <senol@kubedo.com>

fix(mail): use POWER for exponential backoff instead of bitwise XOR

5e1feb9

Refs #147 Signed-off-by: Senol Colak <senol@kubedo.com>

feat(mail): scope archive retention by archive_job_id

ae78729

Refs #147 Signed-off-by: Senol Colak <senol@kubedo.com>

fix(mail): validate retention_days and deduplicate archive imports

fe98d2c

Refs #147 Signed-off-by: Senol Colak <senol@kubedo.com>

fix(mail): do not delay fresh archive jobs on first claim

e6c5bb0

Refs #147 Signed-off-by: Senol Colak <senol@kubedo.com>

chore(mail): add root workspace SQLx offline metadata

c338a33

Refs #147 Signed-off-by: Senol Colak <senol@kubedo.com>

zoorpha requested a review from senolcolak as a code owner July 10, 2026 23:18

senolcolak approved these changes Jul 11, 2026

View reviewed changes

senolcolak added 2 commits July 11, 2026 12:13

zoorpha force-pushed the feat/mail-phase4-archive-jobs branch from 692fd0a to 97881c6 Compare July 11, 2026 10:51

fix(mail): correct archive job retry and delete handling

43c3315

Signed-off-by: Senol Colak <senol@kubedo.com>

zoorpha force-pushed the feat/mail-phase4-archive-jobs branch from 97881c6 to 43c3315 Compare July 11, 2026 10:54

fix(mail): avoid stealing active archive partial rows

9a18cf5

Signed-off-by: Senol Colak <senol@kubedo.com>

chatgpt-codex-connector Bot reviewed Jul 11, 2026

View reviewed changes

Comment thread backend/server/src/services/mail_service.rs Outdated

fix(mail): reschedule completed archive scans

7545d75

Signed-off-by: Senol Colak <senol@kubedo.com>

chatgpt-codex-connector Bot reviewed Jul 11, 2026

View reviewed changes

Comment thread backend/crates/storage/src/metadata.rs Outdated

Comment thread backend/server/src/services/mail_service.rs Outdated

fix(mail): address archive retention review

74f1a7a

Signed-off-by: Senol Colak <senol@kubedo.com>

chatgpt-codex-connector Bot reviewed Jul 11, 2026

View reviewed changes

Comment thread backend/server/src/services/mail_service.rs Outdated

Comment thread backend/server/src/services/mail_service.rs

chatgpt-codex-connector Bot reviewed Jul 11, 2026

View reviewed changes

Comment thread backend/server/src/services/imap_client.rs

Comment thread backend/crates/storage/src/metadata.rs

zoorpha merged commit 6be1bcf into main Jul 11, 2026
20 checks passed

zoorpha deleted the feat/mail-phase4-archive-jobs branch July 11, 2026 20:36

Uh oh!

Conversation

zoorpha commented Jul 10, 2026

Uh oh!

senolcolak commented Jul 11, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

senolcolak commented Jul 11, 2026

Uh oh!

chatgpt-codex-connector Bot commented Jul 11, 2026

Uh oh!

senolcolak commented Jul 11, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

zoorpha commented Jul 11, 2026

Uh oh!

chatgpt-codex-connector Bot commented Jul 11, 2026

Uh oh!

senolcolak commented Jul 11, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

senolcolak commented Jul 11, 2026

Uh oh!

zoorpha commented Jul 11, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

zoorpha commented Jul 11, 2026

Uh oh!

senolcolak commented Jul 11, 2026

Uh oh!

chatgpt-codex-connector Bot commented Jul 11, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants