[bug-fix] Fix reinstall-overwrites-kept-config: preserve config on plain reinstall after --keep-config (#3449)

* Fix reinstall-overwrites-kept-config: preserve config on plain reinstall after --keep-config

Apply the remediation from the bug assessment on issue #3427.

Before the unconditional shutil.rmtree(dest_dir), scan dest_dir for any
*-config.yml and *-config.local.yml files and hold their contents in memory.
After shutil.copytree succeeds, write them back so user-customized values
always win over the packaged defaults.

This mirrors the existing backup/restore logic for the --force reinstall path
but handles the case where remove --keep-config left config files behind in
an unregistered extension directory.

Refs #3427

Assisted-by: GitHub Copilot (model: claude-sonnet-4.6, autonomous)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix: restore method decl, move config restore before registration, preserve file mode

- Restore missing `test_install_force_without_existing` method declaration in
  tests/test_extensions.py so pytest collects it as a separate test.
- Move stranded-config restoration to immediately after `copytree`, before
  command/skill/hook registration, so a failed registration step can't leave
  preserved configs permanently lost.
- Store `(bytes, mode)` tuples instead of bare bytes when rescuing stranded
  configs, and reapply the original file mode after writing so permission bits
  (e.g. 0600 for credential files) are faithfully restored.

Assisted-by: GitHub Copilot (model: claude-sonnet-4.5, autonomous)

* fix: mask setuid/setgid bits when restoring stranded config file mode

Only preserve user/group read-write bits (mode & 0o660) to avoid
restoring setuid, setgid, or world-writable permissions from a
user-modified config file.

Assisted-by: GitHub Copilot (model: claude-sonnet-4.5, autonomous)

* fix: add copytree rollback path and strengthen regression test with packaged default config

- Wrap shutil.copytree in a try/except BaseException so stranded configs
  rescued before rmtree are written back even if copytree fails mid-way
  (addresses review comment: configs were permanently lost on copy failure)
- Add a packaged default config to extension_dir in the regression test so
  a naive 'restore only when absent' implementation would fail; assert the
  user's customized values beat the packaged defaults after reinstall

Assisted-by: GitHub Copilot (model: claude-sonnet-4.6, autonomous)

* fix: restore configs with secure atomic writes

Assisted-by: GitHub Copilot (model: gpt-5, autonomous)

* fix: write secure temp file then chmod to preserved_mode; add copytree-failure test

- _restore_stranded_config_file: write content while temp file is at its
  secure OS-default mode (typically 0600 on POSIX), then apply the
  original preserved_mode after the file is fully written and before the
  atomic os.replace. Removes the & 0o660 mask that was silently stripping
  world-read and executable bits (e.g. 0644 → 0640).

- Add test_copytree_failure_restores_stranded_config: patches
  shutil.copytree to create a partial destination then raise OSError,
  then asserts that the preserved config bytes and file mode are restored
  by the rollback path and that the extension remains unregistered.

Assisted-by: GitHub Copilot (model: claude-sonnet-4.6, autonomous)

* Potential fix for pull request finding 'Unused local variable'

Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>

* Potential fix for pull request finding 'Module is imported with 'import' and 'import from''

Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>

* fix: durable staging for stranded configs and import style fix

- Stage stranded config files to a durable rescue_staging_dir
  (extensions_dir/.rescue-staging-<id>) before rmtree so original bytes
  survive partial rmtree, copytree failure, or partial restore on retry.
  On retry the staging dir is detected and its content reused instead of
  whatever mix of packaged defaults and partial restores remains on disk.
  The staging dir is cleaned up only after every restore succeeds.
- Fix CodeQL: change `import specify_cli.extensions as _ext_module` to
  `from specify_cli import extensions as _ext_module` in test file.

Assisted-by: GitHub Copilot (model: claude-sonnet-4.6, autonomous)

* fix: harden rescue staging dir - symlink checks, secure writes, cleanup errors

- Thread 14: Change except BaseException to except Exception in the staging
  fallback block so KeyboardInterrupt/SystemExit propagate correctly
- Thread 15: Add explanatory comment to the bare pass in the chmod except
  block to satisfy static analysis
- Thread 16: Reject a symlinked staging directory and only reload
  non-symlinked files whose names match the two recognised config suffixes
- Thread 17: Create each staging file via os.open with mode 0600 and
  O_CREAT|O_EXCL before writing so preserved bytes are never transiently
  exposed to other local users
- Thread 18: Remove ignore_errors=True from the final staging-dir cleanup
  so a failed rmtree propagates rather than silently leaving a stale
  backup that could be misread on the next retry

Assisted-by: GitHub Copilot (model: claude-sonnet-4.6, autonomous)

* fix: mask file-type bits from chmod, harden staging dir symlink check

- Add `import stat` to imports
- Use `stat.S_IMODE(mode)` before chmod in staging write (thread 20, line 1464)
- Use `stat.S_IMODE(preserved_mode)` and make chmod best-effort in
  `_restore_stranded_config_file` (thread 18, line 1492)
- Add `not rescue_staging_dir.is_symlink()` guard to cleanup (thread 19, line 1522)

Assisted-by: GitHub Copilot (model: claude-sonnet-4.6, autonomous)

* fix: use completion marker for rescue staging, abort on staging failure, full os.write

Assisted-by: GitHub Copilot (model: GPT-5.3-Codex, autonomous)

* fix(extensions): make rescue staging durable

Assisted-by: GitHub Copilot (model: MAI-Code-1-Flash, autonomous)

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* test(extensions): fix flaky copytree regression test

Assisted-by: GitHub Copilot (model: MAI-Code-1-Flash, autonomous)

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* test(extensions): fix module import alias for review feedback

Assisted-by: GitHub Copilot (model: MAI-Code-1-Flash, autonomous)
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* fix(workflows): keep cleanup warnings single-line and remove dead helper

Assisted-by: GitHub Copilot (model: MAI-Code-1-Flash, autonomous)
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* Potential fix for pull request finding 'Module is imported with 'import' and 'import from''

Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>

* Preserve rescued extension config across retry

Assisted-by: GitHub Copilot (model: MAI-Code-1-Flash, autonomous)
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* Clarify ignored directory fsync cleanup errors

Assisted-by: GitHub Copilot (model: MAI-Code-1-Flash, autonomous)
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* Fix reinstall durability and workflow cleanup warnings

Assisted-by: GitHub Copilot (model: MAI-Code-1-Flash, autonomous)

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* Open rescue staging file in binary mode to fix Windows CRLF corruption

On Windows os.open() defaults to text mode, so os.write() of preserved
config bytes containing \r\n was translated to \r\r\n, corrupting the
staged backup and failing the retry-restore regression test. Add
O_BINARY (0 on POSIX) to the staging file open flags.

Assisted-by: GitHub Copilot (model: Claude Opus 4.8, autonomous)

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* Load .extensionignore before deleting dest_dir on reinstall

The .extensionignore loader can raise ValidationError (invalid UTF-8) or
OSError. Previously it ran after dest_dir was removed, so such a failure
left the kept config only in the hidden staging directory rather than its
documented location. Load/validate it before the rmtree so every
post-deletion failure path restores the config. Adds a regression test.

Assisted-by: GitHub Copilot (model: Claude Opus 4.8, autonomous)

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* Validate .extensionignore before publishing rescue staging

Loading .extensionignore after the rescue staging directory was published
meant a validation failure left a complete staging copy behind. A later
retry (after the user fixed the ignore file and edited the kept config)
would reload the stale staged bytes and silently overwrite the newer
config. Move the loader ahead of reading/creating rescue staging so a
failure aborts while the kept config is still authoritative on disk, and
extend the regression test to prove no staging is published and a retry
adopts the newer bytes.

Assisted-by: GitHub Copilot (model: Claude Opus 4.8, autonomous)

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* Harden preserved-config rescue against divergence and long names

Address three review findings on the reinstall config-rescue path:

- A complete .rescue-complete marker proves only that staging finished,
  not that dest_dir was modified. A crash after staging sync but before
  the rmtree leaves the live kept config intact; if the user edits it
  before retrying, preferring the staged bytes silently overwrote the
  newer config. The two copies are indistinguishable in provenance from
  disk, so detect divergence between a complete staging copy and the live
  config and abort (preserving both) instead of unconditionally choosing
  staging.
- The staging directory embedded the full extension ID in one path
  component. Extension IDs are length-unbounded, so a valid long ID could
  install at dest_dir yet fail every reinstall-after-keep-config with
  ENAMETOOLONG. Derive the staging component from a fixed-length hash via
  a new _rescue_staging_dir() helper.
- The stranded-config restore used the full config filename as a
  NamedTemporaryFile prefix; a name already near the component limit plus
  the random suffix raised ENAMETOOLONG. Use a short fixed prefix.

Updates the retry regression test to the new divergence semantics and
adds conflict-abort, long-ID, and fixed-prefix coverage.

Assisted-by: GitHub Copilot (model: Claude Opus 4.8, autonomous)

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* Harden preserved-config rescue divergence check and fix test path

Assisted-by: GitHub Copilot (model: GPT-5.6, autonomous)

* fix: reject/flag symlinked preserved configs on reinstall

Assisted-by: GitHub Copilot (model: GPT-5.6-Sol, autonomous)

* fix: include symlinks in live-dir config enumeration and address review feedback

- _recognized_config_names() now accepts follow_symlinks=False for live dir
  so symlinked *-config.yml entries are detected and treated as conflicts
  rather than being silently deleted by rmtree.
- Add explanatory comment to bare 'except OSError: pass' in
  _restore_stranded_config_file's finally block.
- Resolve CodeQL dual-import style: use 'from specify_cli import extensions
  as _ext_module' instead of 'import specify_cli.extensions as _ext_module'.

Assisted-by: GitHub Copilot (model: claude-sonnet-4, autonomous)

* test: add staging-failure fault-injection test for rescue staging block

Add test_staging_failure_aborts_before_dest_dir_removal covering three
failure modes (mkdir, os.open/O_CREAT, fsync with EIO) in the rescue
staging block. Each parametrized case verifies:
- the install aborts before dest_dir is removed
- the preserved config bytes remain authoritative
- any partial staging is cleaned up and not left as complete
- the extension stays unregistered

Addresses review feedback on PRRT_kwDOPiFCnc6R351t.

Assisted-by: GitHub Copilot (model: claude-sonnet-4.5, autonomous)

* test: add test_retry_restores_config_from_staging_when_live_absent

Exercises the retry-from-staging branch (if staging_is_complete at
line 1505 of extensions/__init__.py) in a scenario where the live
config is absent — simulating a power loss that interrupted the
rollback before it could write the config back.

When the live copy is gone, the live-dir fallback (elif dest_dir.exists())
finds no stranded configs and the packaged default would be kept. Only the
staging-complete branch can restore the original bytes and mode. This proves
staging (not the fallback) is used on retry.

Addresses review feedback on PRRT_kwDOPiFCnc6SAL3L.

Assisted-by: GitHub Copilot (model: claude-sonnet-4.5, autonomous)

* fix: keep staging files writable; record modes in .rescue-modes.json; fix live-only conflict message

Thread 64: Remove os.fchmod/chmod from staged files to avoid Windows
read-only attribute that prevents shutil.rmtree from cleaning up.
Original permission bits are now written to a .rescue-modes.json sidecar
in the staging dir and reloaded during retry, with a fall-back to the
staged file's own mode for backwards-compat with pre-sidecar staging dirs.

Thread 65: Split the ValidationError message for staging-vs-live conflicts
into two accurate cases: files that diverged between both locations
("Both copies have been preserved") and live-only files that have no
backup counterpart, which previously incorrectly claimed "Both copies
have been preserved" and offered a restore instruction that was impossible.

Assisted-by: GitHub Copilot (model: claude-sonnet-4.5, autonomous)

* Potential fix for pull request finding 'Empty except'

Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>

* fix: add .keep-config provenance marker to guard rescue path against partially-failed installs

When `remove --keep-config` strands config files, write a `.keep-config`
marker into the extension directory.  `install_from_directory` now only
enters the rescue path when that marker is present, preventing a partially-
failed install (which also leaves dest_dir with no registry entry but no
marker) from having its packaged default configs treated as user-preserved
data on a retry from an updated package.

Refs: #3449 (comment)

Assisted-by: GitHub Copilot (model: claude-sonnet-4.5, autonomous)

* refactor: extract _has_keep_config_marker helper and document empty-content choice

Assisted-by: GitHub Copilot (model: claude-sonnet-4.5, autonomous)

* fix: defer rescue-backup cleanup until registry commit; validate modes sidecar shape

Assisted-by: GitHub Copilot (model: GPT-5.6, autonomous)

* fix legacy keep-config rescue and retry baseline handling

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: Manfred Riem <15701806+mnriem@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>

Author: 5 people