Skip to content

Loader speedup#11

Merged
gkennos merged 5 commits into
mainfrom
loader_speedup
May 19, 2026
Merged

Loader speedup#11
gkennos merged 5 commits into
mainfrom
loader_speedup

Conversation

@gkennos
Copy link
Copy Markdown
Member

@gkennos gkennos commented May 19, 2026

CLOSES: #9 , CLOSES #10

@gkennos gkennos requested a review from Copilot May 19, 2026 12:20
@gkennos gkennos self-assigned this May 19, 2026
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR improves loader observability and performance by adding phase-level timing logs around index/merge operations and introducing an insert_if_empty fast path (and automatic routing for replace/upsert when the target table is empty), addressing issues #9 and #10.

Changes:

  • Added detailed, timed logging for index dropping, FK toggle, merge phases, and index rebuild steps.
  • Implemented insert_if_empty merge strategy plus an empty-target probe that routes replace/upsert to the fast path.
  • Added Postgres + e2e tests covering the new merge strategy and the new logging behavior; bumped version to 0.4.1 and updated changelog.

Reviewed changes

Copilot reviewed 6 out of 7 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
uv.lock Updates lockfile artifacts and bumps project version reference to 0.4.1.
pyproject.toml Bumps package version to 0.4.1.
CHANGELOG.md Documents 0.4.1 changes (logging improvements and empty-table merge optimization).
src/orm_loader/tables/typing.py Extends CSVTableProtocol with _target_has_rows() for typing support.
src/orm_loader/tables/loadable_table.py Adds phase timing/logging, implements empty-target routing and insert_if_empty merge strategy.
tests/loaders/test_pg_loader.py Adds Postgres tests for insert_if_empty behavior (success + error on non-empty).
tests/loaders/test_loader_e2e.py Adds e2e tests for insert_if_empty, empty-target routing logs, and merge-phase logging.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/orm_loader/tables/loadable_table.py Outdated
Comment thread src/orm_loader/tables/loadable_table.py Outdated
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 7 changed files in this pull request and generated 3 comments.

Comment thread src/orm_loader/tables/loadable_table.py
Comment thread src/orm_loader/tables/loadable_table.py
Comment thread src/orm_loader/tables/loadable_table.py
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 7 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (1)

tests/loaders/test_loader_e2e.py:615

  • This assertion expects a "Merge operation SQL completed in ..." log message, but the current implementation only logs per-phase timings (delete/insert/upsert) and does not emit an overall SQL-completion message with that wording. Either adjust the test to match the existing logs, or add a total-merge-SQL timing log around merge_from_staging() to satisfy this expectation.
    assert any("Merge operation SQL completed in " in message for message in messages)
    assert any("Committing merged rows." in message for message in messages)

Comment thread src/orm_loader/tables/loadable_table.py
Comment thread tests/loaders/test_loader_e2e.py Outdated
@gkennos gkennos closed this May 19, 2026
@gkennos gkennos reopened this May 19, 2026
@gkennos gkennos merged commit 4ceb432 into main May 19, 2026
6 checks passed
@gkennos gkennos deleted the loader_speedup branch May 19, 2026 12:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

if a table is empty on load we should skip the merge steps improve logging for loader

2 participants