Loader speedup#11
Merged
Merged
Conversation
There was a problem hiding this comment.
Pull request overview
This PR improves loader observability and performance by adding phase-level timing logs around index/merge operations and introducing an insert_if_empty fast path (and automatic routing for replace/upsert when the target table is empty), addressing issues #9 and #10.
Changes:
- Added detailed, timed logging for index dropping, FK toggle, merge phases, and index rebuild steps.
- Implemented
insert_if_emptymerge strategy plus an empty-target probe that routesreplace/upsertto the fast path. - Added Postgres + e2e tests covering the new merge strategy and the new logging behavior; bumped version to 0.4.1 and updated changelog.
Reviewed changes
Copilot reviewed 6 out of 7 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| uv.lock | Updates lockfile artifacts and bumps project version reference to 0.4.1. |
| pyproject.toml | Bumps package version to 0.4.1. |
| CHANGELOG.md | Documents 0.4.1 changes (logging improvements and empty-table merge optimization). |
| src/orm_loader/tables/typing.py | Extends CSVTableProtocol with _target_has_rows() for typing support. |
| src/orm_loader/tables/loadable_table.py | Adds phase timing/logging, implements empty-target routing and insert_if_empty merge strategy. |
| tests/loaders/test_pg_loader.py | Adds Postgres tests for insert_if_empty behavior (success + error on non-empty). |
| tests/loaders/test_loader_e2e.py | Adds e2e tests for insert_if_empty, empty-target routing logs, and merge-phase logging. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 6 out of 7 changed files in this pull request and generated 2 comments.
Comments suppressed due to low confidence (1)
tests/loaders/test_loader_e2e.py:615
- This assertion expects a "Merge operation SQL completed in ..." log message, but the current implementation only logs per-phase timings (delete/insert/upsert) and does not emit an overall SQL-completion message with that wording. Either adjust the test to match the existing logs, or add a total-merge-SQL timing log around merge_from_staging() to satisfy this expectation.
assert any("Merge operation SQL completed in " in message for message in messages)
assert any("Committing merged rows." in message for message in messages)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CLOSES: #9 , CLOSES #10