Skip to content

[FIX] Make BIDSLayout.save idempotent when re-saving to the same path#1260

Open
Leonard013 wants to merge 1 commit into
bids-standard:mainfrom
Leonard013:fix/1079-save-database-idempotent
Open

[FIX] Make BIDSLayout.save idempotent when re-saving to the same path#1260
Leonard013 wants to merge 1 commit into
bids-standard:mainfrom
Leonard013:fix/1079-save-database-idempotent

Conversation

@Leonard013

Copy link
Copy Markdown

Closes #1079.

Why

Calling layout.save(path) a second time against the same path raises:

sqlite3.OperationalError: table associations already exists

This bites users who re-run a script: the first run creates
<path>/layout_index.sqlite, and a second save() to the same path fails.
@effigies reproduced it in the issue thread and noted it wasn't obvious to fix.

Root cause

ConnectionManager.save_database (src/bids/layout/db.py) snapshots the index by
replaying sqlite3.Connection.iterdump() into sqlite3.connect(database_file).
iterdump() emits plain CREATE TABLE ... (no IF NOT EXISTS, no DROP), so
replaying into a database_file that already holds the schema (from a prior
save) collides on the first table — associations.

The reason a naive fix (deleting/truncating the destination first) does not
work is subtle, and is likely why this looked unfixable: with the default
replace_connection=True, the first save() re-points the layout's connection at
the saved file. A subsequent save() to the same path therefore has old_db
(source) and database_file (destination) aliasing the same file
— clearing the
destination in place would destroy the source before iterdump() reads it
(confirmed: it produces an empty / no such table: layout_info database).

What

Dump into a fresh temporary file in the same directory, then atomically move it
into place:

temp_file = database_file.with_name(f'.{database_file.name}.tmp')
temp_file.unlink(missing_ok=True)
new_db = sqlite3.connect(str(temp_file))
with new_db:
    for line in old_db.iterdump():
        if line not in ('BEGIN;', 'COMMIT;'):
            new_db.execute(line)
    new_db.commit()
new_db.close()
temp_file.replace(database_file)
  • The destination always starts empty, so CREATE TABLE never collides.
  • The destination is only touched at the final replace(), so re-saving stays
    correct even when source and destination are the same file.
  • Streaming via iterdump() is preserved; replace() is a same-directory atomic
    rename.

Tests

Added test_save_database_twice_same_path to src/bids/layout/tests/test_db.py:
saves a layout to the same path twice (default replace_connection=True,
exercising the same-file path) and asserts the re-saved database is complete and
reloadable.

  • Before: fails with OperationalError: table associations already exists.
  • After: passes.
  • pytest src/bids/layout/tests/ → 193 passed, 71 skipped (network/submodule
    gated), 1 xfailed, 0 failed.

Caveats (flagging honestly)

  • Windows, same-process same-file re-save: on the second same-file save,
    old_db still holds the destination open when replace() runs, which can raise
    PermissionError on Windows. This is not a regression (the current code
    already raises on the second save everywhere), and pybids CI is Linux/macOS only;
    first saves and cross-path saves are unaffected. Happy to guard it (close
    old_db before replace()) if you'd like it fully Windows-safe.
  • Concurrent saves to the same path remain unsupported (unchanged from before).
  • Happy to fold in polish if preferred: a process-unique temp name
    (tempfile.mkstemp(dir=...)), a try/finally to clean the temp on error, and/or
    a stronger completeness assertion in the test.

No CHANGELOG.rst edit — this project compiles the changelog from PR titles via
release-drafter.


Prepared with AI assistance (Claude Code): the reproduction, root-cause analysis,
fix, and test were AI-drafted; the reasoning and verification are documented above.

Calling ``layout.save(path)`` twice against the same path raised
``sqlite3.OperationalError: table associations already exists``.

``ConnectionManager.save_database`` snapshots the index by replaying
``sqlite3.iterdump()`` into ``sqlite3.connect(database_file)``. ``iterdump``
emits plain ``CREATE TABLE`` statements, so replaying them into a file that
already holds the schema (a previous save, e.g. re-running a script) fails.

The destination cannot simply be cleared in place: with the default
``replace_connection=True`` the first save re-points the layout at the saved
file, so a subsequent save has ``old_db`` (source) and ``database_file``
(destination) aliasing the same path. Dump into a fresh temporary file and
atomically ``replace`` it into position instead, which starts from an empty
database and stays correct even when source and destination are the same file.

Adds a regression test that saves a layout to the same path twice and checks
the re-saved database is still complete.

Closes bids-standard#1079.

This change was developed with the assistance of Claude, an AI coding
assistant; all changes were authored, reviewed, and verified by a human
contributor.

Co-authored-by: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

sqlite3.OperationalError: table associations already exists

1 participant