fix(rag): stop hard-deleting shared chunks; wire soft-delete cleanup

[abdulrafey1]

Closes: [RAG] ChunkStoreService.delete_by_source deletes chunk rows instead of only their links

What

Stop delete_by_source from hard-deleting DocumentChunk rows shared across documents, and wire every Google Drive delete path to soft-delete its linked Document so the reference-counted RAG cleanup job actually runs.

Changes

• fix(rag): remove dangerous delete_by_source chunk deletion
• feat(googledrive): add soft_delete_drive_file helper
• fix(googledrive): soft-delete linked Document on file delete
• fix(googledrive): soft-delete linked Documents on folder delete
• fix(googledrive): soft-delete linked Document on folder sync removal
• test(googledrive): exercise folder-delete over multiple files

Background

ChunkStoreService.delete_by_source issued DELETE FROM rag_document_chunks WHERE source_name = .... Because ingestion deduplicates chunks by content hash, one DocumentChunk row can be linked (via DocumentChunkLink) to several documents — so hard-deleting on one document's removal would destroy chunks other documents still share. The method had no production callers and bypassed the sanctioned reference-counted cleanup in sparkth/rag/cleanup.py.

It was removed rather than rewritten: an inline unlink-only rewrite (as originally proposed on the issue) would instead leak orphaned chunks forever, because cleanup.py only discovers chunks reachable through soft-deleted documents' links.

The sanctioned path is soft-delete driven — soft-delete a Document, and the cleanup_deleted_documents job removes that document's links and deletes only chunks with no remaining link to a live document (shared chunks preserved). That chain was dormant because the Google Drive delete routes soft-deleted the DriveFile but never its linked Document. This PR adds one helper, soft_delete_drive_file, and routes all three DriveFile soft-delete sites (delete_file, delete_folder, folder-sync removal) through it. Each DriveFile owns its own Document, so soft-deleting the file's own Document is safe — chunk sharing is ref-counted at the chunk-link level by the cleanup job.

How to Test

1. Run the full backend suite: make test.backend.pytest → all pass (1369).
2. Focused suites: uv run pytest tests/rag/test_cleanup.py sparkth/plugins/googledrive/tests/test_routes.py sparkth/plugins/googledrive/tests/test_utils.py -v → green (shared-chunk survival + all three wired delete paths + helper unit tests).
3. Confirm the dangerous method is gone: grep -rn "delete_by_source" sparkth/ tests/ → no matches.
4. Types + lint: make mypy (strict, clean) and make lint.backend (clean).
5. Behavioural end-to-end: ingest a Google Drive file so a Document + chunks exist, DELETE the file, confirm Document.is_deleted becomes True, then run make rag-cleanup — the document's links are removed and only chunks with no remaining live-document link are deleted; a chunk shared with another live document survives.

Notes

• No migration, no breaking change, no new env var, no dependency bump.
• This branch is stacked on rafey/migrate-permission-routes-to-central-exception-management and targets it (not main); GitHub will auto-close [RAG] ChunkStoreService.delete_by_source deletes chunk rows instead of only their links #456 once the change reaches main.
• Out of scope (candidate follow-ups): ChunkStoreService.get_sources is also dead code; and the cleanup job (cleanup_deleted_documents) is only exposed via make rag-cleanup — it is not yet wired to a deployed schedule.

This PR description was written with the assistance of an LLM (Claude).

[zaira-bibi]

LGTM!