Improve WebDAV compatibility and performance#381
Conversation
…upport - Add `src/utils/http_validators.rs` with shared HTTP validation helpers: - `if_match_header_matches`: strong ETag comparison per RFC 9110 - `if_none_match_header_matches`: weak ETag comparison per RFC 9110 - `format_http_date` / `parse_http_date`: HTTP-date serialization and parsing - `http_date_epoch_seconds`: SystemTime to epoch seconds for date comparison - Remove duplicated ETag helpers from `webdav/protocol.rs` and `file_service/common.rs`, replacing them with calls to `http_validators` - Add `evaluate_http_download_preconditions` to `webdav/protocol.rs` supporting `If-Match`, `If-None-Match`, `If-Modified-Since`, and `If-Unmodified-Since` with correct header precedence rules - Update `webdav/transfer/mod.rs` to use `evaluate_http_download_preconditions` and include `Last-Modified` in GET/HEAD responses - Add `get_range` stub to `TrailingErrorStreamDriver` in handler tests and add test `handle_get_range_uses_driver_range_without_opening_full_stream` - Add integration tests for `Last-Modified` header presence, `If-Modified-Since`, and `If-Unmodified-Since` preconditions Closed #362
Optimize PROPFIND performance for large directories by batching property and lock queries: - Add `find_by_entities` to property repository for batch property lookup across multiple entities - Implement `get_props_many` in DavFileSystem trait for batch dead property loading - Implement `discover_many` in DavLockSystem trait for batch lock discovery - Add PropfindPreload structure to preload all properties and locks before building responses - Skip dead property loading when PROPFIND requests only standard live properties - Skip lock discovery when PROPFIND does not request lockdiscovery - Add integration test verifying large directory PROPFIND performance with live-only properties - Add unit tests validating batch loading behavior and selective property/lock loading closed #363
Split resolve_path_cached_in_scope into separate read and write variants to prevent stale cache hits after write operations. Write operations (PUT/MKCOL/COPY/MOVE/DELETE/LOCK) now use writer_db() for authoritative resolution, while read operations (GET/PROPFIND metadata) use reader_db() for better load distribution. - Add resolve_path_cached_for_read_in_scope() that queries reader_db - Keep resolve_path_cached_in_scope() pinned to writer_db for write preconditions - Extract resolve_path_cached_in_scope_with_db() as internal implementation - Add resolve_entity_for_read() helper for property queries on read paths - Update validate_cached_folder_path() to accept explicit db connection - Update validate_cached_parent() to accept explicit db connection - Update load_cached_resolved_node() to accept explicit db connection - Switch open_file(), read_dir(), metadata(), get_props(), get_props_bulk(), has_props(), get_quota() to use reader_db - Add comprehensive test_webdav_immediate_read_after_write_operations() covering PUT→GET, COPY→GET, MOVE→GET, LOCK→PROPFIND, DELETE→GET sequences Closed #364
…approach Refactor storage change cache invalidation to use targeted key deletion instead of broad prefix-based invalidation where possible. This improves cache efficiency by only invalidating affected entries. Changes: - Add audience-aware WebDAV path cache prefix generation (personal/team/any) - Introduce `CacheInvalidationTargets` struct to support both prefix and key-based invalidation - Replace blanket folder path cache prefix invalidation with targeted key deletion for specific folder changes - Add batch invalidation support for folder path chains - Expose `folder_path_cache_key` for targeted cache invalidation - Update cache invalidation logic to use event context instead of just event kind - Add helper functions for personal/team-scoped cache prefix generation - Remove unnecessary full cache invalidation calls from folder operations - Update all tests to use event-based cache invalidation with proper scope context
Add configurable coalescing window for WebDAV download audit records to prevent log flooding from repeated reads by the same client. **Features:** - Add `webdav_download_audit_coalesce_window_secs` configuration (default 30s) - Coalesce repeated downloads by account, file, request type (full/ranged), and client fingerprint - Track WebDAV account ID through authentication and filesystem layers - Separate coalescing for full vs ranged reads - Support disabling coalescing by setting window to 0 **Implementation:** - New `download_audit` module with cache-based deduplication logic - Client fingerprint derived from IP address and User-Agent SHA256 hash - Integrate audit identity (account_id, scope, root_folder_id) into download recording - Add i18n labels for English and Chinese admin settings **Testing:** - Verify default 30-second coalescing reduces multiple range requests to single audit row - Confirm full and ranged reads tracked independently - Validate disabling coalescing (window=0) records every read - Update existing auth tests to handle new account_id field Closed #365
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (7)
🚧 Files skipped from review as they are similar to previous changes (4)
📝 WalkthroughWalkthrough本次改动新增 WebDAV 下载审计合并窗口、按文件夹 ID 定向缓存失效、共享 HTTP 校验器与条件下载、PROPFIND 批量预加载,并将部分 WebDAV 只读路径切到 reader DB,同时补充相关测试与配置文案。 ChangesWebDAV 审计、缓存、条件请求与读取优化
Estimated code review effort: 5 (Critical) | ~120 minutes Sequence Diagram(s)sequenceDiagram
participant Client
participant AsterDavFs
participant DownloadAudit
participant Cache
participant AuditService
Client->>AsterDavFs: GET/Range 请求
AsterDavFs->>DownloadAudit: record_download(identity, kind)
DownloadAudit->>Cache: set_bytes_if_absent(key)
Cache-->>DownloadAudit: 是否首次占位
alt 窗口内首次
DownloadAudit->>AuditService: log_with_details
else 窗口内重复
DownloadAudit-->>AsterDavFs: 跳过记录
end
sequenceDiagram
participant Client
participant PropfindHandler
participant PropfindPreload
participant AsterDavFs
participant DbLockSystem
Client->>PropfindHandler: PROPFIND (Depth:1)
PropfindHandler->>PropfindHandler: collect_propfind_resources
PropfindHandler->>PropfindPreload: load(resources)
PropfindPreload->>AsterDavFs: get_props_many(paths)
PropfindPreload->>DbLockSystem: discover_many(paths)
PropfindHandler->>PropfindPreload: dead_props_for / locks_for
PropfindHandler-->>Client: multistatus 响应
Possibly related PRs
Suggested labels: Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (1)
src/webdav/fs/mod.rs (1)
700-742: 🚀 Performance & Scalability | 🔵 Trivial | 🏗️ Heavy liftPROPFIND 的批量只省了一半
get_props_many里还是逐个resolve_entity_for_read(...),而缓存命中的路径也会先find_by_id,再通过validate_cached_folder_path做一轮 DB 校验。read_dir目前只往下游传path/meta,没把 folder/file id 带出来,所以大目录 Depth:1 还是会留下 N 次路径解析。要真吃到这轮优化,得把实体 id 一路传到这里,或者把属性预取并进read_dir。🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/webdav/fs/mod.rs` around lines 700 - 742, get_props_many still resolves each DavPath one by one via resolve_entity_for_read, so PROPFIND Depth:1 remains N path lookups despite the batch property fetch. Update the read path so read_dir (or its callers) carries the entity identifiers needed here, and then use those ids in get_props_many instead of per-path resolution; if that refactor is too broad, fold the property prefetch into read_dir and keep the existing property_repo::find_by_entities flow.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/services/file_service/common.rs`:
- Around line 32-35: The ETag matching logic in if_none_match_matches_value has
changed semantics from the old case-insensitive comparison to
http_validators::if_none_match_header_matches, which now compares If-None-Match
against etag_value differently. Update this helper to preserve the previous
trim_matches('"').eq_ignore_ascii_case(etag_value) behavior, or add explicit
compatibility handling plus tests in if_none_match_matches_value so existing
blob.hash values continue to match as before.
In `@src/webdav/db_lock_system.rs`:
- Around line 521-565: The discover_many flow currently gathers all ancestors
into one lock_repo::find_ancestors query, which can exceed sqlx-sqlite parameter
limits on large Depth:1 directories. Update discover_many to batch or chunk
all_ancestors before calling lock_repo::find_ancestors, then merge the per-chunk
results back into locks_by_path while preserving the existing filtering and
ordering logic. Also add an integration test in tests/test_webdav.rs that
exercises a directory with more than 500 children so the chunking path is
covered.
---
Nitpick comments:
In `@src/webdav/fs/mod.rs`:
- Around line 700-742: get_props_many still resolves each DavPath one by one via
resolve_entity_for_read, so PROPFIND Depth:1 remains N path lookups despite the
batch property fetch. Update the read path so read_dir (or its callers) carries
the entity identifiers needed here, and then use those ids in get_props_many
instead of per-path resolution; if that refactor is too broad, fold the property
prefetch into read_dir and keep the existing property_repo::find_by_entities
flow.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: ddd4e0af-1d8f-4a04-8ee2-26950f1bc7ff
📒 Files selected for processing (28)
frontend-panel/src/i18n/locales/en/admin/settings-common.jsonfrontend-panel/src/i18n/locales/zh/admin/settings-common.jsonsrc/config/definitions.rssrc/db/repository/property_repo.rssrc/services/file_service/common.rssrc/services/folder_service/cache.rssrc/services/folder_service/copy.rssrc/services/folder_service/hierarchy.rssrc/services/folder_service/mod.rssrc/services/folder_service/mutation.rssrc/services/storage_change_service.rssrc/services/task_service/archive/extract/mod.rssrc/services/trash_service/common.rssrc/services/webdav_service.rssrc/utils/http_validators.rssrc/utils/mod.rssrc/webdav/auth.rssrc/webdav/dav.rssrc/webdav/db_lock_system.rssrc/webdav/download_audit.rssrc/webdav/fs/mod.rssrc/webdav/handler_tests/mod.rssrc/webdav/mod.rssrc/webdav/path_resolver.rssrc/webdav/props/mod.rssrc/webdav/protocol.rssrc/webdav/transfer/mod.rstests/test_webdav.rs
💤 Files with no reviewable changes (1)
- src/services/folder_service/copy.rs
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Summary
Tests
Closes #362
Closes #363
Closes #364
Closes #365
Summary by CodeRabbit
Last-Modified与条件请求逻辑完善;PROPFIND 支持批量属性与锁发现,提升深层目录响应准确性。