G3MINI: web dashboard, validators, watcher state, compliance scanner & Docker#4
Draft
JohanDevl wants to merge 143 commits into
Draft
G3MINI: web dashboard, validators, watcher state, compliance scanner & Docker#4JohanDevl wants to merge 143 commits into
JohanDevl wants to merge 143 commits into
Conversation
…and 3D/DTS-HD HRA support
Add blacklist mechanism for team tags, mirroring the existing TAGS_TEAM whitelist system. Releases with a team tag matching EXCLUDED_TAGS are skipped before upload. Applied to Video, Game, and Docu content types.
feat: tracker rule validators, dynamic piece sizing, and team tag exclusion
Replace destination_path with a JSON state file (watcher_state.json) in the config directory. Tracks uploaded and skipped entries with reasons (validation_error, duplicate, excluded_tag, etc.) so the watcher skips already-processed items instantly instead of re-running the full pipeline. Also fix indentation bugs in GameManager and DocuManager where the continue statement was at the wrong level, causing all games and docs to be skipped in watcher mode.
…ategory Support multiple source folders in watcher mode via WATCHER_PATHS config, each with an optional qBittorrent category for cross-seed organization. Backward compatible with legacy single WATCHER_PATH format.
…arate tracker categories
Apply 9 fixes from updated .claude/ rules: H.265/H.264 with dots for WEB/HDTV, DTS-HDMA single token, preserve 10bit/12bit, add OPUS audio detection, DD→AC3 alias, HDTV/HDLight/4KLight codec conventions, 4.0/6.1 channel support, and NoGRP/NoTAG validation for teamless releases.
- preserve "Max" in titles by removing HBO MAX from streaming cleanup - support compound team names with dash and ampersand (Tsundere-Raws, TsundereRaws&T3KASHi) - prevent codecs from being used as false team tags (x264-x264) - strip parenthesized non-year content early to prevent title pollution - add BRRip as distinct source type, normalize HE-AAC/EAC3/AC3@bitrate - remove AD from extras when in title position, keep when after year - clean 6CH/HEVC/EN/KO/READNFO/DUAL residual tags from title - fix 10bits plural, FR-EN compound, duplicate language tags - preserve 6CH as valid audio channel format per tracker rules
Upscaled content is strictly forbidden by tracker rules (encodage.md, upload.md). Add release-name-based detection that blocks uploads containing the UpScaled tag, complementing the existing height-based check. Also extract UpScaled as a proper extras tag in the normalizer so it no longer pollutes the title.
Add Check 5 in NamingValidator to block releases where the team is a placeholder (NoTag, NoGRP) or a year mistakenly used as team name. Severity ERROR ensures the upload is skipped entirely.
Store the full validation report (severity, rule, message, doc) in watcher_state.json and watcher_dryrun.json for both skipped entries (validation errors) and uploaded entries (warnings/infos). Bump version to 0.9.7.
Include the guessit source (BluRay, WEB-DL, WEBRip, HDTV, BDRip, etc.) in watcher_state.json and watcher_dryrun.json for both uploaded and skipped entries. Bump version to 0.9.8.
…aves Items whose torrent file already exists in the archive are now marked as "uploaded" instead of "skipped", since the torrent presence means the content was previously uploaded. Also makes state file writes atomic (tmp + os.replace) to prevent corruption on interrupt, and logs a warning instead of silently ignoring corrupt state files.
Single source of truth for the version: the VERSION file used by pyproject.toml. The hardcoded string in settings.py was stale (0.8.x) and never referenced externally. Bump to 0.9.9.
Release v0.15.0
Move fastapi, uvicorn, jinja2, python-multipart to [project.optional-dependencies] so CLI-only users don't need to install them. Install with pip install Unit3Dup[web] for web dashboard support.
Infer subtitle/audio language from track title when the language field is missing (e.g. Title: French → fr) and add a web UI to edit track languages with prez regeneration before upload approval.
- Click on current/proposed name (list row or detail modal) copies to clipboard; hover reveals a copy icon and success flashes the cell green. - Pre-fuse H.264/H.265 before the dot→space step so the codec is no longer split into "H" + "264" tokens that leak into the title; codec stays at the canonical tail position per G3MINI naming.
- normalizer: SUBFRENCH kept distinct from VOSTFR, add SUBFORCED; VFI now routed through FRENCH disambiguation (VFF/VFQ via MediaInfo); add COMPLETE, 4K.REMASTER, DoVi, HDTVRip, BDMV, DVD9, DVD5 tokens - naming: NoTAG/NoGRP accepted as placeholder per upload.md §1 - naming: add HARDSUB→SUBFRENCH check and MULTi (VO+VF+FR subs) invariant - encoding: add source-reencoded detection (untouched tag + x264/x265 library) and basic crop heuristic (16:9 stored + cinema DAR) - upload: add pack consistency (multi-team, mix FRENCH/MULTi) - compliance: extract subtitle/library/dimensions from MediaInfo text, extend _FakeMediaFile, run EncodingValidator alongside NamingValidator
Previously, VOF/VOQ/VOB (Version Originale) tags were stripped or replaced by VFF when combined with MULTi or FRENCH: - MULTi.VOF was reduced to bare MULTi - FRENCH.VOF was overwritten to VFF - MULTi.FRENCH.VOF became MULTi.VFF VO tags carry distinct information (original audio, not a dub) and must never be lost or downgraded to a dub indicator.
When an audio track has Title=VOF but Language=French (FR), the previous logic returned VFF because the language code was checked first. Audio with an explicit VO title is original French audio, not a dub — VOF must win. Title-based VOF/VOQ/VOB now take priority over the generic Language: French (FR)/(CA) signal.
Release 0.16.0
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A long-running fork with a comprehensive set of features and fixes targeted at the G3MINI tracker workflow. Highlights:
naming,encoding,upload) — rejects no-group / upscaled releases, classifies x264/x265/AV1 as encode.docker-compose.yml, watcher mode, TZ env, arbitrary-UID friendly, non-interactiveEOFErrorhandling.external_ids, robust kwargs filtering.VERSIONfile as source of truth.Diff aggregate: 78 files, +12 591 / −611 lines.
Detailed changes by domain
1. Web Dashboard (Starlette + HTMX)
Initial scaffold then ~60 follow-up commits:
c7721cafeat(web): add web dashboard for upload approval workflow26354bffeat(web): modern dark dashboard with sidebar, interactive tables, responsive layout2735fa3feat(web): add TMDB ID editing and rescan from item detail page8bdeac0feat(web): add save button to persist edits without approving or rejectingedd42a4feat(web): add reason labels, status filter dropdown, fix date sorting9613f1ffeat(web): add client-side pagination (25/50/100/250/All) on all tablesf357461feat(web): add skip reason column with human-readable labels in pending table5885468feat(web): show current page number in pagination2fbec21feat(web): replace confirm alert with approve modal showing upload summary2cf3746feat(web): add full rescan button on pending itemsc0ac05dfix(api): make approve endpoint accept empty body to prevent 422 errorscd7cba1fix(web): format all dates as dd/mm/yyyy HH:MM instead of raw ISO timestamps220609afeat(web): fetch TMDB title in French (fr-FR) with English fallback on rescan2330b9dfix(web): complete mobile responsive overhaul (cards on mobile, no horizontal scroll)ebe9a06feat(web): add multi-select column filters and mobile sort dropdownb1b84a1fix(web): preserve pending list pagination across HTMX refreshes229cd03feat(web): add similar torrents link on detail page60b8ee8feat(web): add category adjustment on pending item detail pageac44404fix(web): preserve table sort state across HTMX pending refresh8a685ecfeat(web): auto-update release name when TMDB ID is rescanned5400499fix(web): redirect to pending page after approve/reject instead of history4b06c27fix(web): use full rescan instead of retry for skipped/rejected items3abdd6dfeat(web): add season/episode display and editing for TV series6fc1fcdfeat(web): add bulk reject UI for pending items list70f1465feat(web): add async upload queue with background worker and bulk approve modal1a3259afeat(web): add queue status page with sidebar badge and live refresh0687d1ffeat(web): normalize separators in search bar for flexible name matching5bbdfe2feat(web): add upload-cloud favicon matching sidebar icon5a15750feat(web): add cancel queued and reset uploaded item actions790756efeat(web): add visual analyzing indicator for watcher processing114c081feat(web): display human-readable category labels instead of raw valuesd5dc25afeat(web): add season/episode editing to bulk approve modal for TV series18d4c90feat(web): add track language editor and title-based language fallback3a48237feat(web): add async bulk rescan queue with web visibility8f787b9update(web): replace native confirm with proper modal for bulk rescan898a925/bd72633fix(web): batch all transitions before enqueueing in bulk rescan / approve60a8ca2feat(web): link duplicate match on tracker and add force-rescan actionf820ed0fix(web): persist tracker_name on duplicate skip and add similar-url fallback2. Compliance Module
967a6dafeat(web): add compliance menu to audit past upload naming against G3MINI rules650d413fix(compliance): align normalizer & validators with G3MINI docs rules23ee647feat(web): show description & mediainfo in compliance detail modal4855637feat(web): regenerate prez-format description in compliance modal90a8c2cfix(compliance): read UnIT3D media_info/bd_info snake_case API keys4a0bb2bfix(compliance): flatten unwrapped torrent response and modal-ify recheck7a265ecupdate(web): auto-reload compliance page after single-row recheck0733e7cfeat(compliance): add regenerate button and per-track language editoraf5867dfeat(compliance): add ack/ignore buttons in modal with in-place status updateffcf7d4feat(compliance): add delete & rescan action with bulk selection6d1abedfix(web): copy-on-click compliance names + keep H.264/H.265 codec at end3. Validator Suite
7782e62feat(validation): add tracker rule validators, dynamic piece sizing, and 3D/DTS-HD HRA supportf1de5d3feat(validator): reject uploads without a real release groupbfa7f2afeat(validator): reject upscaled releases with ERROR severity9e41948fix(validator): treat x264/x265/AV1 tags as encode, not untouched41ef372fix(tmdb): filter unknown kwargs before dataclass init4. Watcher State & Performance
3d0a322feat(watcher): add persistent state tracking to avoid reprocessing9031645fix(watcher): skip state persistence in dry-run mode (-noup/-noseed)d176202feat(watcher): write dry-run results to separate watcher_dryrun.json333640afix(watcher): treat already_in_archive as uploaded and atomic state saves125b331feat(watcher): add multi-folder support with per-folder qBittorrent category0d21689perf(watcher): reduce countdown loop wake-ups from 100/s to 1/s3dda7c6fix(watcher): break infinite loop on already_in_archive in web mode0702130fix(watcher): handle stale torrent archive causing false 'already uploaded' for series8b946a4fix(watcher): skip non-media files (.nfo, .txt, etc.) in watcher scan4f1c4edfix(watcher): preserve validation reports on dry-run re-runs (41c49d4)d6a8df7feat(watcher): add release source type to watcher JSON4a03f7afeat(watcher): include validation report details in watcher JSON5. Release Normalizer
568c771feat(normalizer): use single-lang tags (VFF/VFQ/VOSTFR) for mono-track releases5b8b8dbfix(naming): preserve VOF/VOQ/VOB tags in release namesef10a65fix(naming): prioritize MediaInfo Title VO markers over Language codeade3a7cfeat(naming): inject TMDB year into series release names7a22d6ffix(normalizer): resolve 22 anomalies in release name normalization60afe99fix(normalizer): add NoTag fallback and clean separator/bitrate artifactsadff990fix(normalizer): transliterate accented characters instead of stripping them462bf4dfeat(category): detect animated content via TMDB genre and assign separate tracker categoriesca8a23dfix(normalizer): normalize SUBFRENCH to VOSTFR in release titlesdeaacc4fix(normalizer): align release naming with updated tracker rules6. TMDB / External Services
c54952ffix(tmdb): use French language for API requests and fix rescan title extraction73584cdfeat(tmdb): fetch IMDb ID during discover via details/external_ids endpoints6cc7ed0fix(tmdb): add missing fields to ExternalIds dataclass for TMDB API compatibility41ef372fix(tmdb): filter unknown kwargs before dataclass initada37d7feat: add TMDB-based documentary detection mirroring anime pattern930cd2bfix(prez): pre-select detected track language and add fr-CA/fr-BE00f7247fix(prez): normalize language tags with region and english namesf2f933dfix(mediainfo): strip full path from Complete name in displayed mediainfo7. Prez / BBCode Description
bc944aefeat(prez): replace screenshot description with BBCode prez style 38078da3fix(prez): use width-based resolution classification for quality label2cc0ba9fix(web): show generated NFO (cleaned mediainfo) instead of source NFO filefcd3a4ffix(web): remove NFO source display from dashboard, avoid confusion with mediainfo8. Docker / Deployment
f3052bcfeat(docker): add docker-compose.yml with watcher mode and first-run setupb15c42dfeat(docker): add TZ environment variable for timezone support455bbd2fix(docker): make home directory traversable for arbitrary UIDsa1eb35cfix(docker): make venv world-executable for arbitrary UIDs33812f1fix(docker): set HOME env for arbitrary UIDs and remove unused destination pathd4eb555update(docker): remove done volume and document watcher state files95391a2fix(docker): handle EOFError on input() calls for non-interactive containerscafe4dafix(cache): use absolute paths for diskcache and prez file output9. CI / Release Automation
668c6f2feat(ci): add GitHub Actions for Docker build and publish to GHCR and Docker Hub10. Packaging / Settings
b81cacffeat(packaging): make web dependencies optional via pip extras7f3d30cfix(settings): read version from VERSION file instead of hardcoded value724964bfix(settings): use importlib.metadata for version in installed packages682a153fix(settings): handle empty string for WEB_PORT in existing configs33e29bdfix(settings): handle empty string for WEB_HOST in existing configs5b6db02fix(settings): handle empty string for QBIT_SKIP_HASH_CHECK in existing configsc1a5e7cfix(settings): use model defaults for new torrent config attributes0c9907ffeat(torrent): add skip hash check option for qBittorrentb9ec363fix(qbittorrent): let category handle save path instead of forcing it11. Other Fixes
71abe94fix(media): accept season 0 and episode 0 for TV series uploads70b720ffix(video): guard against None db.result before calling get_title()a803176fix(trackers): correct video TYPE_ID mappings and infer source from resolutiond473845fix(trackers): pass title instead of full path to filter_type to prevent directory names matching TYPE_ID keys5aafc81fix(db): use localtime for discovered_at default to respect TZ env variableed97a51fix(db): use Python datetime.now() for discovered_at instead of SQLite defaultce55ac8fix(web): restrict field editing (resolution, source, TMDB) to pending status only5a94838feat(web): add resolution correction dropdown with exact tracker values94f4f85fix(web): use exact tracker type_id values with optgroups in source dropdownf2bc095feat(web): add source type (type_id) correction dropdown in item detailee7fa88feat(upload): add EXCLUDED_TAGS to skip uploads by team tageddaeccfeat(dry-run): count processed media as uploaded and handle all EOFError cases10f5446feat(dry-run): store normalized release name in watcher stateNote on the 2 commits behind upstream
The 2 upstream commits not yet merged into the fork are both about the initial Docker-stack support:
d495e69Clarify Docker watcher startup behavior (Clarify Docker watcher startup behavior #2)e7df18aAdd Docker stack support for containerized deployments (Add Docker stack support for containerized deployments #1)The fork has reworked Docker more deeply (
docker-compose.yml, multi-folder watcher, arbitrary UIDs, TZ, EOF handling). These commits will be reconciled when conflicts are resolved.Test plan