From 69b0cc0937938299ef72ad5571e4d6fb56f6ed7d Mon Sep 17 00:00:00 2001 From: Hyeon-Mook Jelly Choi Date: Tue, 23 Jun 2026 18:26:15 +0900 Subject: [PATCH 1/6] =?UTF-8?q?feat(enhancer):=20Voice=20Enhancer=20?= =?UTF-8?q?=ED=8C=8C=EC=9D=B4=ED=94=84=EB=9D=BC=EC=9D=B8=20=EC=B6=94?= =?UTF-8?q?=EA=B0=80=20(PRD=20v0.2)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 기존 DeepFilterNet 노이즈 캔슬링 위에 교체 가능한 2-stage 파이프라인과 DSP 보이스 인핸서를 추가한다 (krisp_like_mac_voice_enhancer_prd_v02.docx). 엔진 (CrispEngine): - AudioProcessing: ProcessingMode(Off/Noise/Voice/Clean+Enhance), AudioProcessingConfig, AudioProcessor 프로토콜 (PRD §4.5 교체 가능 stage) - Biquad: RBJ cookbook IIR + EnvelopeFollower (할당 없는 per-sample) - VoiceEnhancer: HPF→톤EQ(3종)→컴프레서→디에서→−1dBFS 리미터. zero-latency, 생성형 아님(화자 보존). wetMix=0이면 bit-identical 패스스루 - PipelineProcessor: denoise→enhance stage 그래프. 모드는 라우팅이 아니라 파라미터(atten·dry/wet 램프)만 바꿔 클릭 없이 전환 (PRD §4.1/§8.4) - Loudness: BS.1770 K-weighting + 게이팅 LUFS 측정/정규화 + peak 천장 - FileEnhancer.enhance(options:): 파일 HQ 파이프라인 (Fast/HQ, 톤, 미리듣기 N초, LUFS −16/−18, before/after 리포트) 앱: - AppState에 mode/enhanceStrength/tonePreset 추가 + 영속화. 파라미터 변경은 엔진 재시작 없이 update()로 라이브 적용 - LiveAudioEngine을 PipelineProcessor로 전환 - MenuBar/Settings: 처리 모드 + 인핸스 강도 + 톤 picker - File 탭: 모드/품질/톤/음량정규화/미리듣기 + 결과 리포트 - Diagnostics: 모드/강도/톤/추정 지연 표기 검증: - vetool(신규 CLI): 패스스루 bit-identical, aligned==chunked 결정성, 리미터 안전, RTF enhancer 0.012 / clean+enhance 0.11 - filetool enhance: 4모드 파일 처리, 48k mono wav/m4a, 길이 정확 보존 - 신규 서드파티/모델 weight 0개 (라이선스 BOM 변화 없음) docs: voice-enhancer.md(신규), architecture/STATUS/test-report/README/LICENSES 갱신 Co-Authored-By: Claude Opus 4.8 (1M context) --- LICENSES.md | 7 + README.md | 40 ++-- app/Package.swift | 7 + .../Audio/AudioEngineController.swift | 41 ++-- .../CrispApp/Diagnostics/Diagnostics.swift | 14 +- app/Sources/CrispApp/Model/AppState.swift | 98 +++++++-- app/Sources/CrispApp/Views/FileTabView.swift | 190 ++++++++++-------- .../CrispApp/Views/MenuBarContentView.swift | 57 +++++- app/Sources/CrispApp/Views/SettingsView.swift | 35 +++- app/Sources/CrispEngine/AudioProcessing.swift | 109 ++++++++++ app/Sources/CrispEngine/Biquad.swift | 103 ++++++++++ app/Sources/CrispEngine/FileEnhancer.swift | 159 +++++++++++++++ app/Sources/CrispEngine/Loudness.swift | 98 +++++++++ app/Sources/CrispEngine/NoiseSuppressor.swift | 3 + .../CrispEngine/PipelineProcessor.swift | 72 +++++++ app/Sources/CrispEngine/VoiceEnhancer.swift | 161 +++++++++++++++ app/Sources/filetool/main.swift | 35 +++- app/Sources/vetool/main.swift | 119 +++++++++++ docs/STATUS.md | 13 ++ docs/architecture.md | 25 +++ docs/test-report.md | 28 +++ docs/voice-enhancer.md | 110 ++++++++++ 22 files changed, 1377 insertions(+), 147 deletions(-) create mode 100644 app/Sources/CrispEngine/AudioProcessing.swift create mode 100644 app/Sources/CrispEngine/Biquad.swift create mode 100644 app/Sources/CrispEngine/Loudness.swift create mode 100644 app/Sources/CrispEngine/PipelineProcessor.swift create mode 100644 app/Sources/CrispEngine/VoiceEnhancer.swift create mode 100644 app/Sources/vetool/main.swift create mode 100644 docs/voice-enhancer.md diff --git a/LICENSES.md b/LICENSES.md index 0f33771..8398cbb 100644 --- a/LICENSES.md +++ b/LICENSES.md @@ -8,6 +8,13 @@ Crisp가 사용하는 서드파티 구성요소와 상용 배포 가능 여부. | **tract** (tract-onnx/core/pulse) | 순수 Rust ONNX 추론 엔진 | MIT / Apache-2.0 (dual) | ✅ 가능 | | Crisp HAL 드라이버 코드 | 가상 마이크 | 자체 작성(원본). Apple Audio Server Plug-in 아키텍처 참고 | ✅ 자체 저작 | | Crisp 앱/엔진 코드 | UI, 실시간/파일 처리 | 자체 작성 | ✅ 자체 저작 | +| Crisp Voice Enhancer DSP | EQ/컴프레서/디에서/리미터/LUFS (PRD v0.2) | 자체 작성(RBJ EQ cookbook·ITU-R BS.1770 공개 공식 기반, 코드 원본) | ✅ 자체 저작 | + +> **Voice Enhancer(PRD v0.2)는 신규 서드파티·모델 weight를 추가하지 않는다.** 인핸서를 생성형 +> 모델 대신 순수 DSP(자체 코드)로 구현했기 때문이다. PRD가 후보로 든 LocalVQE(Apache-2.0), +> Resemble Enhance(MIT), ClearerVoice(Apache-2.0)는 추후 동일 `AudioProcessor` seam에 드롭인할 +> 수 있으나, 그 가중치·코드 라이선스는 **탑재 시점에 별도 BOM 확인 필요**(PRD §7.2, §9 M1). +> NVIDIA RE-USE 등 비상용 모델은 출시 빌드 제외 원칙 유지. > **ffmpeg 의존 제거됨.** 파일 처리(Phase 4)는 AVFoundation(시스템 프레임워크) + libDF로 구현해 > 외부 프로세스/ffmpeg 없이 동작한다. ffmpeg는 테스트 픽스처 생성에만 쓰이며 제품에 포함되지 않는다. diff --git a/README.md b/README.md index d0660fd..607ba92 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,12 @@ -# Crisp — macOS 실시간 노이즈 캔슬링 +# Crisp — macOS 실시간 노이즈 캔슬링 + 보이스 인핸서 -물리 마이크 입력을 **온디바이스 AI(DeepFilterNet3)** 로 실시간 노이즈 억제한 뒤, Zoom·Meet·Slack·Discord·Teams -등에서 선택할 수 있는 **가상 마이크("Noise Cancelled Microphone")** 로 출력. 로컬 파일 음성 개선도 지원. +물리 마이크 입력을 **온디바이스 AI(DeepFilterNet3)** 로 실시간 노이즈 억제하고, **DSP 보이스 +인핸서**(EQ·컴프레서·디에서·리미터)로 명료도·음량을 다듬은 뒤, Zoom·Meet·Slack·Discord·Teams +등에서 선택할 수 있는 **가상 마이크("Noise Cancelled Microphone")** 로 출력. 로컬 파일 음성 개선(파일 HQ)도 지원. + +**처리 모드**: Off · Noise Cancellation · Voice Enhancer · Clean + Enhance (PRD v0.2 — +[`docs/voice-enhancer.md`](docs/voice-enhancer.md)). 두 stage(denoise → enhance)는 교체 가능하며 +보이스 인핸서는 생성형이 아니라 화자/발화를 보존한다. ## 빌드 & 설치 — 복사해서 실행 (또는 AI 에이전트에 붙여넣기) @@ -54,10 +59,13 @@ open build/Crisp-0.1.0.pkg ## 사용법 -**실시간**: 메뉴바 Crisp → 노이즈 캔슬링 ON + 입력 마이크 선택 → 회의 앱 마이크에서 "Noise Cancelled Microphone" 선택. 강도(약/보통/강)·바이패스·저지연 모드 조절 가능. +**실시간**: 메뉴바 Crisp → 실시간 처리 ON + 입력 마이크 선택 → 회의 앱 마이크에서 "Noise Cancelled Microphone" 선택. +**처리 모드**(Off/Noise/Voice/Clean+Enhance), 노이즈 강도, 인핸스 강도, 톤(자연/따뜻/밝게), +바이패스(A/B), 저지연 모드를 조절. 모드/강도/톤 전환은 클릭 없이 라이브 적용. -**파일**: 설정 창 → 파일 탭 → 오디오/비디오(wav/mp3/m4a/mp4) 드래그앤드롭 → 출력 포맷(wav/m4a) 선택 → -**강도 슬라이더(0–100 dB)** 로 단일 처리, 또는 **배치 모드**로 여러 강도(예 12/24/48/100 dB)를 한 번에 출력해 비교·저장. +**파일**: 설정 창 → 파일 탭 → 오디오/비디오(wav/mp3/m4a/mp4/mov) 드래그앤드롭 → +처리 모드 + 품질(Fast/HQ) + 톤 + 음량 정규화(-16/-18 LUFS) 선택 → **미리듣기(20초)** 로 결과 방향 확인 후 +**음성 개선 시작**. 원본은 덮어쓰지 않고 wav/m4a로 저장하며 before/after LUFS·peak 리포트를 표시. ## 검증 / 테스트 @@ -66,8 +74,10 @@ open build/Crisp-0.1.0.pkg ./scripts/e2e-test.sh # end-to-end: model→가상마이크→녹음 (드라이버 설치 필요) ./scripts/stability-test.sh 1800 # 30분 안정성 (crash/dropout) BIN="$(cd app && swift build --show-bin-path)" -"$BIN/dftool" engine/models/DeepFilterNet3_onnx.tar.gz 100 build/in.f32 out.f32 chunked # 스트리밍 정확성 -"$BIN/filetool" input.mp3 out.m4a m4a # 파일 처리 +"$BIN/dftool" engine/models/DeepFilterNet3_onnx.tar.gz 100 build/in.f32 out.f32 chunked # 노이즈 stage 스트리밍 정확성 +"$BIN/vetool" engine/models/DeepFilterNet3_onnx.tar.gz # 인핸서/파이프라인(결정성·리미터·RTF) +"$BIN/filetool" input.mp3 out.m4a m4a # 파일 denoise +"$BIN/filetool" enhance input.mp3 out.wav clean hq natural podcast # 파일 Voice Enhancer 파이프라인 ``` ## 프로젝트 구조 @@ -78,8 +88,10 @@ app/Package.swift SwiftPM: CDeepFilter / CrispEngine / CrispApp / dftoo Sources/CDeepFilter/ libdf C 헤더(df.h) + modulemap Sources/CrispEngine/ 순수 DSP/모델(AVFoundation): DeepFilterSuppressor, FileEnhancer, VirtualMicOutput(AUHAL), CoreAudioDevices ← 앱·CLI 공유 + Voice Enhancer: AudioProcessing(모드/config/protocol), PipelineProcessor, + VoiceEnhancer, Biquad, Loudness Sources/CrispApp/ SwiftUI 메뉴바 앱 (AppState, AudioEngineController, Views) - Sources/{dftool,filetool,mictool}/ 검증 CLI + Sources/{dftool,vetool,filetool,mictool}/ 검증 CLI (vetool = 인핸서/파이프라인) engine/CDeepFilter/lib/ libdf.dylib (fetch-deps 생성, gitignore) engine/models/ DeepFilterNet3 모델 (fetch-deps 생성, gitignore) scripts/ fetch-deps / install / verify / package / e2e / stability @@ -98,6 +110,7 @@ poc/model/ DeepFilterNet 클론(gitignore) + run_poc.sh | 2 실시간 엔진 | capture→model→가상마이크 | ✅ end-to-end loopback | | 3 앱 | 메뉴바/온보딩/설정 | ✅ | | 4 파일 처리 | ffmpeg-free (AVFoundation) | ✅ mp3→m4a, mp4→wav | +| VE Voice Enhancer (v0.2) | 2-stage 파이프라인 + DSP 인핸서, 4모드, 파일 HQ | ✅ vetool 검증, RTF 0.11 | | 5 패키징 | 서명/notarization/pkg | 🔶 pkg ✅, notarize는 Apple 계정 | --- @@ -108,8 +121,10 @@ poc/model/ DeepFilterNet 클론(gitignore) + run_poc.sh **Repo map (수정 위치):** - 가상 마이크 동작 → `driver/CrispAudioDriver/CrispAudioDriver.c` (`Crisp_DoIOOperation`의 loopback ring) -- 모델 추론(실시간·파일 공통) → `app/Sources/CrispEngine/DeepFilterSuppressor.swift` (libdf C-API) -- 파일 처리 → `app/Sources/CrispEngine/FileEnhancer.swift` (AVAssetReader→libdf→AVAudioFile) +- 모델 추론(노이즈 stage) → `app/Sources/CrispEngine/DeepFilterSuppressor.swift` (libdf C-API) +- 보이스 인핸서(DSP) → `app/Sources/CrispEngine/VoiceEnhancer.swift` · 필터 `Biquad.swift` · LUFS `Loudness.swift` +- 파이프라인 그래프/모드 → `app/Sources/CrispEngine/{PipelineProcessor,AudioProcessing}.swift` +- 파일 처리 → `app/Sources/CrispEngine/FileEnhancer.swift` (AVAssetReader→pipeline→AVAudioFile) - 가상 마이크 출력 → `app/Sources/CrispEngine/VirtualMicOutput.swift` (AUHAL) - 실시간 캡처/라우팅 → `app/Sources/CrispApp/Audio/AudioEngineController.swift` - UI/상태 → `app/Sources/CrispApp/{Views,Model}` @@ -118,6 +133,7 @@ poc/model/ DeepFilterNet 클론(gitignore) + run_poc.sh - 장치 UID `"CrispAudioDevice:0"` 는 드라이버(`kDevice_UID`)와 Swift(`CoreAudioDevices.crispVirtualUID`)에서 동일. - 모델은 48 kHz mono, hop 480(=10ms). 다른 레이트는 `AVAudioConverter`로 48k 변환 후 처리. - `DeepFilterSuppressor`는 stateful 스트리밍 — hop을 연속·순서대로 공급(carry 버퍼 보장). 검증: `dftool ... chunked` 출력이 aligned와 bit-identical. +- 두 stage(denoise→enhance)는 모든 모드에서 항상 신호 경로에 존재 — 모드는 파라미터(atten·dry/wet)만 바꿈. `VoiceEnhancer`는 `wetMix==0`에서 bit-identical 패스스루여야 함(클릭 방지). 검증: `vetool` (aligned==chunked, 패스스루, 리미터). - 실시간/파일 경로에 ffmpeg·외부 프로세스 의존 금지 (파일도 AVFoundation). - UI 메터 갱신은 ~15Hz throttle + `MeterState` 격리 (오디오 탭 ~100Hz → 메인스레드 폭주 방지). 되돌리지 말 것. - HAL 드라이버는 Apple Silicon에서 서명 필수(ad-hoc 가능), 팩토리 심볼 `CrispAudioDriver_Create` export 유지(`CRISP_EXPORT`). @@ -127,4 +143,4 @@ poc/model/ DeepFilterNet 클론(gitignore) + run_poc.sh ## 라이선스 -DeepFilterNet(libDF + 모델 weight), tract: **MIT / Apache-2.0** (상용 가능). Crisp 드라이버/앱/엔진: 자체 작성. 상세: [`LICENSES.md`](LICENSES.md) +DeepFilterNet(libDF + 모델 weight), tract: **MIT / Apache-2.0** (상용 가능). Crisp 드라이버/앱/엔진(보이스 인핸서 DSP 포함): 자체 작성 — Voice Enhancer 추가로 **서드파티/모델 weight 변화 없음**. 상세: [`LICENSES.md`](LICENSES.md) diff --git a/app/Package.swift b/app/Package.swift index 6c0e60b..f683e87 100644 --- a/app/Package.swift +++ b/app/Package.swift @@ -55,6 +55,13 @@ let package = Package( dependencies: ["CrispEngine", "CDeepFilter"], path: "Sources/mictool", linkerSettings: [dfLinker] + ), + // Verifies the Voice Enhancer DSP stage + pipeline (streaming determinism, RTF). + .executableTarget( + name: "vetool", + dependencies: ["CrispEngine", "CDeepFilter"], + path: "Sources/vetool", + linkerSettings: [dfLinker] ) ] ) diff --git a/app/Sources/CrispApp/Audio/AudioEngineController.swift b/app/Sources/CrispApp/Audio/AudioEngineController.swift index a53cf0c..e93f590 100644 --- a/app/Sources/CrispApp/Audio/AudioEngineController.swift +++ b/app/Sources/CrispApp/Audio/AudioEngineController.swift @@ -7,20 +7,21 @@ import CrispEngine private let audioLog = Logger(subsystem: "ai.rtzr.crisp", category: "audio") protocol AudioEngineControlling: AnyObject { - func start(inputUID: String?, strength: NoiseStrength, bypassed: Bool, lowLatency: Bool) + func start(config: AudioProcessingConfig, inputUID: String?, lowLatency: Bool) func stop() - func updateParameters(strength: NoiseStrength, bypassed: Bool) + func update(config: AudioProcessingConfig) } /// Live capture engine (PRD 6.2): -/// physical mic capture → AVAudioConverter(→48k mono) → DeepFilter inference -/// → ring buffer → AUHAL output → Crisp virtual device → loopback → meeting app +/// physical mic capture → AVAudioConverter(→48k mono) → two-stage pipeline +/// (DeepFilter denoise → Voice Enhancer DSP) → ring buffer → AUHAL output +/// → Crisp virtual device → loopback → meeting app /// /// Any mic sample rate is supported: the converter normalizes to the model's 48 kHz mono. final class LiveAudioEngine: AudioEngineControlling { private weak var state: AppState? private let engine = AVAudioEngine() - private var suppressor: NoiseSuppressor = PassthroughSuppressor() + private var pipeline: PipelineProcessor? private var output: VirtualMicOutput? private var converter: AVAudioConverter? private let canonical = AVAudioFormat(commonFormat: .pcmFormatFloat32, sampleRate: 48_000, channels: 1, interleaved: false)! @@ -31,18 +32,18 @@ final class LiveAudioEngine: AudioEngineControlling { self.state = state } - func start(inputUID: String?, strength: NoiseStrength, bypassed: Bool, lowLatency: Bool) { + func start(config: AudioProcessingConfig, inputUID: String?, lowLatency: Bool) { requestMicAccess { [weak self] granted in guard let self else { return } guard granted else { self.setStatus(.error("마이크 권한이 거부되었습니다.")) return } - self.startLocked(inputUID: inputUID, strength: strength, bypassed: bypassed, lowLatency: lowLatency) + self.startLocked(config: config, inputUID: inputUID, lowLatency: lowLatency) } } - private func startLocked(inputUID: String?, strength: NoiseStrength, bypassed: Bool, lowLatency: Bool) { + private func startLocked(config: AudioProcessingConfig, inputUID: String?, lowLatency: Bool) { stop() if let uid = inputUID, let deviceID = Self.deviceID(forUID: uid) { @@ -56,18 +57,20 @@ final class LiveAudioEngine: AudioEngineControlling { return } - // DeepFilter model: low-latency or full quality (PRD SET-01). Load here (not RT-safe). + // Noise Cancellation stage: DeepFilter model, low-latency or full (PRD SET-01). + // Load here (not RT-safe); fall back to passthrough if it fails. let model: DeepFilterSuppressor.Model = lowLatency ? .lowLatency : .full + let suppressor: NoiseSuppressor if let modelPath = DeepFilterSuppressor.modelPath(model), - let df = DeepFilterSuppressor(modelPath: modelPath, - attenuationLimitDb: strength.attenuationLimitDb, - bypassed: bypassed) { + let df = DeepFilterSuppressor(modelPath: modelPath) { suppressor = df } else { suppressor = PassthroughSuppressor() } - suppressor.attenuationLimitDb = strength.attenuationLimitDb - suppressor.bypassed = bypassed + // Two-stage pipeline (denoise → Voice Enhancer DSP). Mode/strength/tone come from config. + let pipe = PipelineProcessor(suppressor: suppressor, sampleRate: 48_000, config: config) + pipe.prepare(config: config) + pipeline = pipe // Resampler: input (any rate, any channels) → 48 kHz mono for the model. converter = AVAudioConverter(from: format, to: canonical) @@ -87,7 +90,8 @@ final class LiveAudioEngine: AudioEngineControlling { input.installTap(onBus: 0, bufferSize: 480, format: format) { [weak self] buffer, _ in guard let self, let canonicalSamples = self.toCanonicalMono(buffer), !canonicalSamples.isEmpty else { return } let inLevel = Self.rms(canonicalSamples) - let processed = self.suppressor.process(canonicalSamples) // DeepFilter (0.97ms/hop) + // Two-stage pipeline: DeepFilter denoise (~0.97 ms/hop) → Voice Enhancer DSP. + let processed = self.pipeline?.process(canonicalSamples) ?? canonicalSamples if !processed.isEmpty { processed.withUnsafeBufferPointer { self.output?.pushMono($0.baseAddress!, count: $0.count) } } @@ -118,14 +122,15 @@ final class LiveAudioEngine: AudioEngineControlling { output?.stop() output = nil converter = nil + pipeline = nil running = false DispatchQueue.main.async { self.state?.meters.set(input: 0, output: 0) } setStatus(.stopped) } - func updateParameters(strength: NoiseStrength, bypassed: Bool) { - suppressor.attenuationLimitDb = strength.attenuationLimitDb - suppressor.bypassed = bypassed + /// Apply mode/strength/tone/bypass live — ramps inside the pipeline, no teardown (PRD 8.4). + func update(config: AudioProcessingConfig) { + pipeline?.update(config: config) } // MARK: - Helpers diff --git a/app/Sources/CrispApp/Diagnostics/Diagnostics.swift b/app/Sources/CrispApp/Diagnostics/Diagnostics.swift index 7fdb5ab..f959e51 100644 --- a/app/Sources/CrispApp/Diagnostics/Diagnostics.swift +++ b/app/Sources/CrispApp/Diagnostics/Diagnostics.swift @@ -8,7 +8,14 @@ enum Diagnostics { static var buildNumber: String { Bundle.main.infoDictionary?["CFBundleVersion"] as? String ?? "1" } - static let modelVersion = "DeepFilterNet3 (tract)" + static let modelVersion = "DeepFilterNet3 (tract) + DSP Voice Enhancer" + + /// Rough added latency of the realtime pipeline, in ms (PRD 6.1 / SET-02). The Voice + /// Enhancer DSP is zero-latency (IIR, no lookahead); the figure is the DeepFilter stage's + /// algorithmic latency (~10 ms hop, ~20 ms initial buffer for the low-latency model). + static func estimatedLatencyMs(lowLatency: Bool) -> Double { + lowLatency ? 20 : 30 + } /// Export status/error info only — never voice data (PRD privacy / SET-03). @MainActor @@ -18,7 +25,10 @@ enum Diagnostics { lines.append("app: \(appVersion) (build \(buildNumber))") lines.append("model: \(modelVersion)") lines.append("os: \(ProcessInfo.processInfo.operatingSystemVersionString)") - lines.append("enabled: \(state.isEnabled) bypassed: \(state.isBypassed) strength: \(state.strength.rawValue)") + lines.append("enabled: \(state.isEnabled) bypassed: \(state.isBypassed)") + lines.append("mode: \(state.mode.rawValue)") + lines.append("noiseStrength: \(state.strength.rawValue) enhanceStrength: \(state.enhanceStrength.rawValue) tone: \(state.tonePreset.rawValue)") + lines.append("lowLatencyModel: \(state.lowLatencyMode) est.addedLatencyMs: \(estimatedLatencyMs(lowLatency: state.lowLatencyMode))") lines.append("selectedInputUID: \(state.selectedInputUID ?? "default")") lines.append("virtualMicInstalled: \(state.virtualMicInstalled)") switch state.status { diff --git a/app/Sources/CrispApp/Model/AppState.swift b/app/Sources/CrispApp/Model/AppState.swift index 2c00ef5..27cf577 100644 --- a/app/Sources/CrispApp/Model/AppState.swift +++ b/app/Sources/CrispApp/Model/AppState.swift @@ -1,6 +1,7 @@ import Foundation import Combine import ServiceManagement +import CrispEngine /// Noise-suppression strength. PRD RT-04: at least 3 steps or a continuous slider. enum NoiseStrength: String, CaseIterable, Identifiable { @@ -25,6 +26,47 @@ enum NoiseStrength: String, CaseIterable, Identifiable { } } +// Korean display labels for the engine's processing enums (PRD 2.2 / 3.2). Kept in the app +// layer so CrispEngine stays UI/locale-free. +extension ProcessingMode { + var label: String { + switch self { + case .off: return "끄기" + case .noiseCancellation: return "노이즈 캔슬링" + case .voiceEnhancer: return "보이스 인핸서" + case .cleanAndEnhance: return "클린 + 인핸스" + } + } + var shortLabel: String { + switch self { + case .off: return "Off" + case .noiseCancellation: return "노이즈" + case .voiceEnhancer: return "인핸스" + case .cleanAndEnhance: return "클린+인핸스" + } + } +} + +extension EnhanceStrength { + var label: String { + switch self { + case .low: return "약하게" + case .medium: return "보통" + case .high: return "강하게" + } + } +} + +extension TonePreset { + var label: String { + switch self { + case .natural: return "자연스럽게" + case .warm: return "따뜻하게" + case .bright: return "밝게" + } + } +} + /// Holds only the high-frequency level-meter values, observed solely by the meter views. /// Keeping these out of AppState prevents 100 Hz audio updates from invalidating the /// entire popover (pickers, toggles, etc.) — the main UI-responsiveness fix. @@ -50,11 +92,14 @@ enum EngineStatus: Equatable { /// Single source of truth for UI + engine, with UserDefaults persistence (PRD APP-03). @MainActor final class AppState: ObservableObject { - @Published var isEnabled: Bool { didSet { persist(); syncEngine() } } - @Published var isBypassed: Bool { didSet { persist(); syncEngine() } } - @Published var strength: NoiseStrength { didSet { persist(); syncEngine() } } - @Published var selectedInputUID: String? { didSet { persist(); syncEngine() } } - @Published var lowLatencyMode: Bool { didSet { persist(); syncEngine() } } // PRD SET-01 + @Published var isEnabled: Bool { didSet { persist(); syncEnable() } } + @Published var isBypassed: Bool { didSet { persist(); applyParams() } } // A/B monitor (FR-RT-005) + @Published var mode: ProcessingMode { didSet { persist(); applyParams() } } // FR-RT-002 + @Published var strength: NoiseStrength { didSet { persist(); applyParams() } } // noise (RT-04) + @Published var enhanceStrength: EnhanceStrength { didSet { persist(); applyParams() } } // FR-RT-003 + @Published var tonePreset: TonePreset { didSet { persist(); applyParams() } } // tone (3.2) + @Published var selectedInputUID: String? { didSet { persist(); restartEngine() } } + @Published var lowLatencyMode: Bool { didSet { persist(); restartEngine() } } // PRD SET-01 (reloads model) @Published var launchAtLogin: Bool { didSet { persist(); applyLaunchAtLogin() } } // Live telemetry. Level meters live in their own object (`meters`) so high-frequency @@ -70,7 +115,10 @@ final class AppState: ObservableObject { private enum Keys { static let isEnabled = "isEnabled" static let isBypassed = "isBypassed" + static let mode = "mode" static let strength = "strength" + static let enhanceStrength = "enhanceStrength" + static let tonePreset = "tonePreset" static let selectedInputUID = "selectedInputUID" static let lowLatencyMode = "lowLatencyMode" static let launchAtLogin = "launchAtLogin" @@ -80,7 +128,10 @@ final class AppState: ObservableObject { let d = UserDefaults.standard self.isEnabled = d.object(forKey: Keys.isEnabled) as? Bool ?? false self.isBypassed = d.object(forKey: Keys.isBypassed) as? Bool ?? false + self.mode = ProcessingMode(rawValue: d.string(forKey: Keys.mode) ?? "") ?? .cleanAndEnhance self.strength = NoiseStrength(rawValue: d.string(forKey: Keys.strength) ?? "") ?? .medium + self.enhanceStrength = EnhanceStrength(rawValue: d.string(forKey: Keys.enhanceStrength) ?? "") ?? .medium + self.tonePreset = TonePreset(rawValue: d.string(forKey: Keys.tonePreset) ?? "") ?? .natural self.selectedInputUID = d.string(forKey: Keys.selectedInputUID) self.lowLatencyMode = d.bool(forKey: Keys.lowLatencyMode) self.launchAtLogin = d.bool(forKey: Keys.launchAtLogin) @@ -97,16 +148,30 @@ final class AppState: ObservableObject { } .store(in: &cancellables) - if isEnabled { syncEngine() } + if isEnabled { restartEngine() } } func toggleEnabled() { isEnabled.toggle() } + /// The pipeline configuration for the current UI state. A bypass forces passthrough + /// (mode `.off`) for instant A/B without tearing the engine down (PRD FR-RT-005). + private func currentConfig() -> AudioProcessingConfig { + AudioProcessingConfig(mode: isBypassed ? .off : mode, + sampleRate: 48_000, + noiseAttenuationDb: strength.attenuationLimitDb, + enhanceStrength: enhanceStrength, + tonePreset: tonePreset, + modelId: lowLatencyMode ? "DeepFilterNet3_ll_onnx" : "DeepFilterNet3_onnx") + } + private func persist() { let d = UserDefaults.standard d.set(isEnabled, forKey: Keys.isEnabled) d.set(isBypassed, forKey: Keys.isBypassed) + d.set(mode.rawValue, forKey: Keys.mode) d.set(strength.rawValue, forKey: Keys.strength) + d.set(enhanceStrength.rawValue, forKey: Keys.enhanceStrength) + d.set(tonePreset.rawValue, forKey: Keys.tonePreset) d.set(selectedInputUID, forKey: Keys.selectedInputUID) d.set(lowLatencyMode, forKey: Keys.lowLatencyMode) d.set(launchAtLogin, forKey: Keys.launchAtLogin) @@ -122,11 +187,20 @@ final class AppState: ObservableObject { } } - private func syncEngine() { - if isEnabled { - engine.start(inputUID: selectedInputUID, strength: strength, bypassed: isBypassed, lowLatency: lowLatencyMode) - } else { - engine.stop() - } + /// Master on/off: start or stop the capture engine. + private func syncEnable() { + if isEnabled { restartEngine() } else { engine.stop() } + } + + /// (Re)start the engine — needed when the input device or model (low-latency) changes. + private func restartEngine() { + guard isEnabled else { return } + engine.start(config: currentConfig(), inputUID: selectedInputUID, lowLatency: lowLatencyMode) + } + + /// Apply mode/strength/tone/bypass live — no teardown, so it is click-free (PRD 8.4). + private func applyParams() { + guard isEnabled else { return } + engine.update(config: currentConfig()) } } diff --git a/app/Sources/CrispApp/Views/FileTabView.swift b/app/Sources/CrispApp/Views/FileTabView.swift index 429d9a0..a8d34bc 100644 --- a/app/Sources/CrispApp/Views/FileTabView.swift +++ b/app/Sources/CrispApp/Views/FileTabView.swift @@ -6,67 +6,77 @@ import CrispEngine @MainActor final class FileTabModel: ObservableObject { @Published var inputURL: URL? + @Published var mode: ProcessingMode = .cleanAndEnhance + @Published var quality: FileQuality = .hq + @Published var enhanceStrength: EnhanceStrength = .medium + @Published var tonePreset: TonePreset = .natural + @Published var noiseStrength: NoiseStrength = .high + @Published var loudness: LoudnessTarget = .podcast @Published var format: OutputFormat = .wav - @Published var attenDb: Double = 100 // single-mode strength (atten-lim dB) - @Published var batchMode = false - @Published var batchLevels: Set = [12, 24, 48, 100] + @Published var progress: Double = 0 @Published var statusText = "" @Published var isProcessing = false @Published var results: [URL] = [] + @Published var report: FileEnhanceReport? @Published var errorText: String? - static let presetLevels = [6, 12, 24, 48, 100] // dB; 100 = full suppression - private let enhancer = FileEnhancer() private var player: AVAudioPlayer? + private var lastPreview: URL? func setInput(_ url: URL) { - inputURL = url; results = []; errorText = nil; progress = 0; statusText = "" + inputURL = url; results = []; report = nil; errorText = nil; progress = 0; statusText = "" } - func toggleLevel(_ l: Int) { if batchLevels.contains(l) { batchLevels.remove(l) } else { batchLevels.insert(l) } } func cancel() { enhancer.cancel() } func play(_ url: URL) { player = try? AVAudioPlayer(contentsOf: url); player?.play() } - func startSingle() { + private func options(previewSeconds: Double? = nil) -> FileEnhanceOptions { + FileEnhanceOptions(mode: mode, quality: quality, enhanceStrength: enhanceStrength, + tonePreset: tonePreset, noiseAttenuationDb: noiseStrength.attenuationLimitDb, + loudness: loudness, format: format, previewSeconds: previewSeconds) + } + + /// Full-file processing → user-chosen output (PRD FR-FILE-005: never overwrites the input). + func start() { guard let input = inputURL, !isProcessing else { return } let panel = NSSavePanel() panel.nameFieldStringValue = input.deletingPathExtension().lastPathComponent + "_crisp." + format.ext panel.allowedContentTypes = [format == .wav ? .wav : .mpeg4Audio] guard panel.runModal() == .OK, let output = panel.url else { return } - let fmt = format, atten = Float(attenDb), enhancer = self.enhancer - begin() + let opts = options(), enhancer = self.enhancer + begin("처리 중…") Task.detached { do { - try enhancer.enhance(input: input, output: output, format: fmt, attenuationDb: atten) { p in + let r = try enhancer.enhance(input: input, output: output, options: opts) { p in Task { @MainActor in self.progress = p } } - await MainActor.run { self.done([output]) } + await MainActor.run { self.done([output], report: r) } } catch { await MainActor.run { self.fail(error) } } } } - func startBatch() { - guard let input = inputURL, !isProcessing, !batchLevels.isEmpty else { return } - let panel = NSOpenPanel() - panel.canChooseDirectories = true; panel.canChooseFiles = false; panel.prompt = "이 폴더에 저장" - guard panel.runModal() == .OK, let dir = panel.url else { return } - let fmt = format, base = input.deletingPathExtension().lastPathComponent + "_crisp" - let levels = batchLevels.sorted().map(Float.init), enhancer = self.enhancer - begin() + /// 15–30 s before/after preview to a temp file (PRD FR-FILE-003), then play it. + func startPreview() { + guard let input = inputURL, !isProcessing else { return } + if let old = lastPreview { try? FileManager.default.removeItem(at: old) } + let tmp = FileManager.default.temporaryDirectory + .appendingPathComponent("crisp_preview_\(UInt32(truncatingIfNeeded: input.hashValue)).\(format.ext)") + lastPreview = tmp + let opts = options(previewSeconds: 20), enhancer = self.enhancer + begin("미리듣기 생성 중…") Task.detached { do { - let out = try enhancer.enhanceBatch(input: input, outputDirectory: dir, baseName: base, - format: fmt, attenuationLevels: levels) { p, lvl in - Task { @MainActor in self.progress = p; self.statusText = "강도 \(Int(lvl))dB 처리 중…" } + _ = try enhancer.enhance(input: input, output: tmp, options: opts) { p in + Task { @MainActor in self.progress = p } } - await MainActor.run { self.done(out) } + await MainActor.run { self.isProcessing = false; self.statusText = "미리듣기 재생"; self.play(tmp) } } catch { await MainActor.run { self.fail(error) } } } } - private func begin() { isProcessing = true; errorText = nil; statusText = "처리 중…"; progress = 0; results = [] } - private func done(_ urls: [URL]) { results = urls; isProcessing = false; statusText = "완료 (\(urls.count)개)" } + private func begin(_ text: String) { isProcessing = true; errorText = nil; statusText = text; progress = 0; results = []; report = nil } + private func done(_ urls: [URL], report: FileEnhanceReport?) { results = urls; self.report = report; isProcessing = false; statusText = "완료" } private func fail(_ error: Error) { isProcessing = false; errorText = (error as? LocalizedError)?.errorDescription ?? error.localizedDescription } } @@ -75,77 +85,85 @@ struct FileTabView: View { @State private var isTargeted = false var body: some View { - VStack(spacing: 14) { - dropZone - if let input = model.inputURL { - LabeledContent("입력 파일", value: input.lastPathComponent) - } - - Picker("출력 포맷", selection: $model.format) { - ForEach(OutputFormat.allCases) { Text($0.rawValue.uppercased()).tag($0) } - } - .pickerStyle(.segmented).disabled(model.isProcessing) - - Toggle("여러 강도 일괄 출력 (배치)", isOn: $model.batchMode).disabled(model.isProcessing) - - if model.batchMode { - batchLevelPicker - } else { - strengthSlider - } + ScrollView { + VStack(spacing: 14) { + dropZone + if let input = model.inputURL { + LabeledContent("입력 파일", value: input.lastPathComponent) + } - if model.isProcessing { - ProgressView(value: model.progress) { Text(model.statusText) } - Button("취소") { model.cancel() } - } else { - Button(model.batchMode ? "일괄 처리 (\(model.batchLevels.count)개 강도)" : "음성 개선 시작") { - if model.batchMode { model.startBatch() } else { model.startSingle() } + settings + + if model.isProcessing { + ProgressView(value: model.progress) { Text(model.statusText) } + Button("취소") { model.cancel() } + } else { + HStack { + Button("미리듣기 (20초)") { model.startPreview() } + .disabled(model.inputURL == nil) + Spacer() + Button("음성 개선 시작") { model.start() } + .disabled(model.inputURL == nil) + .keyboardShortcut(.defaultAction) + } } - .disabled(model.inputURL == nil || (model.batchMode && model.batchLevels.isEmpty)) - .keyboardShortcut(.defaultAction) - } - resultsView - if let err = model.errorText { - Label(err, systemImage: "xmark.octagon.fill").foregroundStyle(.red).font(.caption) + resultsView + if let err = model.errorText { + Label(err, systemImage: "xmark.octagon.fill").foregroundStyle(.red).font(.caption) + } } - Spacer() + .padding(4) } - .padding(4) } - private var strengthSlider: some View { - VStack(alignment: .leading, spacing: 2) { - HStack { - Text("강도 (감쇠 상한)").font(.caption).foregroundStyle(.secondary) - Spacer() - Text("\(Int(model.attenDb)) dB").font(.caption.monospacedDigit()) - } - Slider(value: $model.attenDb, in: 0...100, step: 1).disabled(model.isProcessing) - Text("0 = 원본 유지 · 100 = 최대 노이즈 제거").font(.caption2).foregroundStyle(.tertiary) - } - } + private var settings: some View { + GroupBox { + VStack(alignment: .leading, spacing: 10) { + Picker("처리 모드", selection: $model.mode) { + ForEach(ProcessingMode.allCases) { Text($0.label).tag($0) } + } + Picker("품질", selection: $model.quality) { + Text("빠르게 (Fast)").tag(FileQuality.fast) + Text("고품질 (HQ)").tag(FileQuality.hq) + } + .pickerStyle(.segmented) - private var batchLevelPicker: some View { - VStack(alignment: .leading, spacing: 6) { - Text("생성할 강도 (감쇠 상한 dB) — 선택한 만큼 파일 생성").font(.caption).foregroundStyle(.secondary) - HStack { - ForEach(FileTabModel.presetLevels, id: \.self) { lvl in - Toggle("\(lvl)", isOn: Binding( - get: { model.batchLevels.contains(lvl) }, - set: { _ in model.toggleLevel(lvl) })) - .toggleStyle(.button) - .disabled(model.isProcessing) + if model.mode == .noiseCancellation || model.mode == .cleanAndEnhance { + Picker("노이즈 강도", selection: $model.noiseStrength) { + ForEach(NoiseStrength.allCases) { Text($0.label).tag($0) } + } } + if model.mode.usesEnhancer { + Picker("인핸스 강도", selection: $model.enhanceStrength) { + ForEach(EnhanceStrength.allCases) { Text($0.label).tag($0) } + } + Picker("톤", selection: $model.tonePreset) { + ForEach(TonePreset.allCases) { Text($0.label).tag($0) } + } + } + if model.quality == .hq { + Picker("음량 정규화", selection: $model.loudness) { + Text("팟캐스트 (-16 LUFS)").tag(LoudnessTarget.podcast) + Text("회의 (-18 LUFS)").tag(LoudnessTarget.meeting) + Text("끄기").tag(LoudnessTarget.none) + } + } + Picker("출력 포맷", selection: $model.format) { + ForEach(OutputFormat.allCases) { Text($0.rawValue.uppercased()).tag($0) } + } + .pickerStyle(.segmented) } + .disabled(model.isProcessing) + .padding(6) } } private var resultsView: some View { Group { if !model.results.isEmpty { - GroupBox("결과 (\(model.results.count)개) — 전후/레벨 비교") { - VStack(alignment: .leading, spacing: 4) { + GroupBox("결과 — 전후 비교") { + VStack(alignment: .leading, spacing: 6) { if let input = model.inputURL { HStack { Button("원본 재생") { model.play(input) }; Spacer() } } @@ -157,6 +175,12 @@ struct FileTabView: View { Button("표시") { NSWorkspace.shared.activateFileViewerSelecting([url]) } } } + if let r = model.report { + Divider() + Text(String(format: "출력 음량 %.1f LUFS · 피크 %.1f → %.1f dBFS · 적용 게인 %+.1f dB", + r.outputLUFS, r.inputPeakDb, r.outputPeakDb, r.loudnessGainDb)) + .font(.caption2.monospacedDigit()).foregroundStyle(.secondary) + } } } } @@ -167,12 +191,12 @@ struct FileTabView: View { RoundedRectangle(cornerRadius: 12) .strokeBorder(style: StrokeStyle(lineWidth: 2, dash: [8])) .foregroundStyle(isTargeted ? Color.accentColor : Color.secondary) - .frame(height: 110) + .frame(height: 100) .overlay { VStack(spacing: 6) { Image(systemName: "arrow.down.doc").font(.system(size: 26)).foregroundStyle(.tint) Text("오디오/비디오 파일을 끌어다 놓거나 클릭해서 선택").font(.callout).foregroundStyle(.secondary) - Text("wav · mp3 · m4a · mp4").font(.caption2).foregroundStyle(.tertiary) + Text("wav · mp3 · m4a · mp4 · mov").font(.caption2).foregroundStyle(.tertiary) } } .contentShape(Rectangle()) diff --git a/app/Sources/CrispApp/Views/MenuBarContentView.swift b/app/Sources/CrispApp/Views/MenuBarContentView.swift index 208be09..a6a2b12 100644 --- a/app/Sources/CrispApp/Views/MenuBarContentView.swift +++ b/app/Sources/CrispApp/Views/MenuBarContentView.swift @@ -1,25 +1,38 @@ import SwiftUI +import CrispEngine struct MenuBarContentView: View { @EnvironmentObject var state: AppState @Environment(\.openWindow) private var openWindow + private var isActive: Bool { state.isEnabled && !state.isBypassed && state.mode != .off } + var body: some View { VStack(alignment: .leading, spacing: 12) { HStack { Text("Crisp") .font(.headline) Spacer() - Text(state.isEnabled && !state.isBypassed ? "노이즈 캔슬링 켜짐" : "꺼짐") + Text(isActive ? "\(state.mode.label) 켜짐" : "꺼짐") .font(.caption) - .foregroundStyle(state.isEnabled && !state.isBypassed ? .green : .secondary) + .foregroundStyle(isActive ? .green : .secondary) } - Toggle("노이즈 캔슬링", isOn: $state.isEnabled) + Toggle("실시간 처리", isOn: $state.isEnabled) .toggleStyle(.switch) Divider() + // Processing mode (PRD FR-RT-002). + VStack(alignment: .leading, spacing: 4) { + Text("처리 모드").font(.caption).foregroundStyle(.secondary) + Picker("처리 모드", selection: $state.mode) { + ForEach(ProcessingMode.allCases) { m in Text(m.shortLabel).tag(m) } + } + .labelsHidden() + .pickerStyle(.segmented) + } + // Input mic picker (PRD RT-01). VStack(alignment: .leading, spacing: 4) { Text("입력 마이크").font(.caption).foregroundStyle(.secondary) @@ -33,21 +46,43 @@ struct MenuBarContentView: View { .pickerStyle(.menu) } - // Strength (PRD RT-04). - VStack(alignment: .leading, spacing: 4) { - Text("강도").font(.caption).foregroundStyle(.secondary) - Picker("강도", selection: $state.strength) { - ForEach(NoiseStrength.allCases) { s in Text(s.label).tag(s) } + // Noise strength — only when the Noise Cancellation stage is active (PRD RT-04). + if state.mode == .noiseCancellation || state.mode == .cleanAndEnhance { + VStack(alignment: .leading, spacing: 4) { + Text("노이즈 강도").font(.caption).foregroundStyle(.secondary) + Picker("노이즈 강도", selection: $state.strength) { + ForEach(NoiseStrength.allCases) { s in Text(s.label).tag(s) } + } + .labelsHidden() + .pickerStyle(.segmented) + } + } + + // Enhance strength + tone — only when the Voice Enhancer stage is active (PRD FR-RT-003 / 3.2). + if state.mode.usesEnhancer { + VStack(alignment: .leading, spacing: 4) { + Text("인핸스 강도").font(.caption).foregroundStyle(.secondary) + Picker("인핸스 강도", selection: $state.enhanceStrength) { + ForEach(EnhanceStrength.allCases) { s in Text(s.label).tag(s) } + } + .labelsHidden() + .pickerStyle(.segmented) + } + VStack(alignment: .leading, spacing: 4) { + Text("톤").font(.caption).foregroundStyle(.secondary) + Picker("톤", selection: $state.tonePreset) { + ForEach(TonePreset.allCases) { t in Text(t.label).tag(t) } + } + .labelsHidden() + .pickerStyle(.segmented) } - .labelsHidden() - .pickerStyle(.segmented) } // Level meters (PRD APP/RT status display). Isolated in their own observer so // high-frequency level updates don't re-render the rest of this popover. MetersView(meters: state.meters) - Toggle("바이패스 (원본 전달)", isOn: $state.isBypassed) + Toggle("바이패스 (원본 전달 · A/B 비교)", isOn: $state.isBypassed) .toggleStyle(.checkbox) .font(.caption) diff --git a/app/Sources/CrispApp/Views/SettingsView.swift b/app/Sources/CrispApp/Views/SettingsView.swift index 50bbec6..426d506 100644 --- a/app/Sources/CrispApp/Views/SettingsView.swift +++ b/app/Sources/CrispApp/Views/SettingsView.swift @@ -1,4 +1,5 @@ import SwiftUI +import CrispEngine struct SettingsView: View { @EnvironmentObject var state: AppState @@ -22,12 +23,25 @@ struct AudioSettingsView: View { var body: some View { Form { Section("실시간 처리") { - Toggle("노이즈 캔슬링 켜기", isOn: $state.isEnabled) - Toggle("바이패스 (원본 신호 전달)", isOn: $state.isBypassed) - Picker("강도", selection: $state.strength) { - ForEach(NoiseStrength.allCases) { Text($0.label).tag($0) } + Toggle("실시간 처리 켜기", isOn: $state.isEnabled) + Toggle("바이패스 (원본 신호 전달 · A/B)", isOn: $state.isBypassed) + Picker("처리 모드", selection: $state.mode) { + ForEach(ProcessingMode.allCases) { Text($0.label).tag($0) } } - Picker("처리 모드", selection: $state.lowLatencyMode) { + if state.mode == .noiseCancellation || state.mode == .cleanAndEnhance { + Picker("노이즈 강도", selection: $state.strength) { + ForEach(NoiseStrength.allCases) { Text($0.label).tag($0) } + } + } + if state.mode.usesEnhancer { + Picker("인핸스 강도", selection: $state.enhanceStrength) { + ForEach(EnhanceStrength.allCases) { Text($0.label).tag($0) } + } + Picker("톤", selection: $state.tonePreset) { + ForEach(TonePreset.allCases) { Text($0.label).tag($0) } + } + } + Picker("모델 지연", selection: $state.lowLatencyMode) { Text("기본 (고품질)").tag(false) Text("저지연").tag(true) } @@ -77,6 +91,17 @@ struct DiagnosticsView: View { LabeledContent("엔진", value: statusText) LabeledContent("가상 마이크", value: state.virtualMicInstalled ? "설치됨" : "없음") } + Section("처리 설정") { + LabeledContent("처리 모드", value: state.mode.label) + if state.mode == .noiseCancellation || state.mode == .cleanAndEnhance { + LabeledContent("노이즈 강도", value: state.strength.label) + } + if state.mode.usesEnhancer { + LabeledContent("인핸스 강도", value: state.enhanceStrength.label) + LabeledContent("톤", value: state.tonePreset.label) + } + LabeledContent("추정 추가 지연", value: "≈ \(Int(Diagnostics.estimatedLatencyMs(lowLatency: state.lowLatencyMode))) ms") + } Section("진단 로그") { Text("진단 로그에는 음성 원본이 포함되지 않으며, 상태/오류 정보만 기록됩니다. (PRD SET-03)") .font(.caption).foregroundStyle(.secondary) diff --git a/app/Sources/CrispEngine/AudioProcessing.swift b/app/Sources/CrispEngine/AudioProcessing.swift new file mode 100644 index 0000000..d5933c5 --- /dev/null +++ b/app/Sources/CrispEngine/AudioProcessing.swift @@ -0,0 +1,109 @@ +import Foundation + +/// Processing mode the user selects (PRD 2.2 / 4.5 / FR-RT-002). +/// +/// The pipeline is a two-stage graph — a *Noise Cancellation* stage (DeepFilter) and a +/// *Voice Enhancer* stage (DSP) — and the mode just selects which stages are active. +/// Both stages stay structurally in the signal path at all times (the enhancer blends +/// dry→wet, the suppressor runs at a varying attenuation) so switching modes never +/// re-routes the stream and therefore never clicks (PRD 8.4 "모드 전환 시 pop/click 없음"). +public enum ProcessingMode: String, CaseIterable, Identifiable, Sendable { + case off // passthrough — A/B baseline, low power + case noiseCancellation // denoise only (existing MVP) + case voiceEnhancer // light denoise + DSP enhancement + case cleanAndEnhance // full denoise → DSP enhancement (recommended default) + + public var id: String { rawValue } + + /// Whether the Voice Enhancer (DSP) stage contributes to the output. + public var usesEnhancer: Bool { + self == .voiceEnhancer || self == .cleanAndEnhance + } + + /// How hard the Noise Cancellation stage works, relative to the user's strength. + /// `voiceEnhancer` keeps denoise gentle (PRD 2.2: "잡음 제거는 약하게 적용"). + public enum DenoisePolicy { case none, light, full } + public var denoisePolicy: DenoisePolicy { + switch self { + case .off: return .none + case .noiseCancellation: return .full + case .voiceEnhancer: return .light + case .cleanAndEnhance: return .full + } + } +} + +/// Enhancement intensity (PRD 3.2 / FR-RT-003). Defaults stay conservative — High can +/// introduce artifacts and is never the default (PRD 6.2 / 8.3). +public enum EnhanceStrength: String, CaseIterable, Identifiable, Sendable { + case low, medium, high + public var id: String { rawValue } + + /// 0…1 scalar that scales every enhancer band/amount. + public var intensity: Float { + switch self { + case .low: return 0.45 + case .medium: return 0.75 + case .high: return 1.0 + } + } +} + +/// Tone shaping (PRD 3.2). Three understandable options, not a parametric EQ. +public enum TonePreset: String, CaseIterable, Identifiable, Sendable { + case natural // gentle presence lift, true-to-source + case warm // body boost, softened highs — close, podcast-like + case bright // air/presence boost — clarity on dull mics + public var id: String { rawValue } +} + +/// Light denoise attenuation limit (dB) used by `voiceEnhancer` mode. +public let lightDenoiseAttenuationDb: Float = 10 + +/// Full pipeline configuration (PRD 4.5 `AudioProcessingConfig`). Carries everything the +/// stage graph needs; the app maps its own UI enums onto this. +public struct AudioProcessingConfig: Sendable { + public var mode: ProcessingMode + public var sampleRate: Int + public var noiseAttenuationDb: Float // user-chosen Noise Cancellation strength + public var enhanceStrength: EnhanceStrength + public var tonePreset: TonePreset + public var outputGainDb: Float + public var modelId: String + + public init(mode: ProcessingMode = .cleanAndEnhance, + sampleRate: Int = 48_000, + noiseAttenuationDb: Float = 100, + enhanceStrength: EnhanceStrength = .medium, + tonePreset: TonePreset = .natural, + outputGainDb: Float = 0, + modelId: String = "") { + self.mode = mode + self.sampleRate = sampleRate + self.noiseAttenuationDb = noiseAttenuationDb + self.enhanceStrength = enhanceStrength + self.tonePreset = tonePreset + self.outputGainDb = outputGainDb + self.modelId = modelId + } + + /// Attenuation the Noise Cancellation stage should run at for this mode. + public var effectiveDenoiseDb: Float { + switch mode.denoisePolicy { + case .none: return 0 + case .light: return min(noiseAttenuationDb, lightDenoiseAttenuationDb) + case .full: return noiseAttenuationDb + } + } +} + +/// Streaming audio-processing seam (PRD 4.5 `AudioProcessor`). Mirrors `NoiseSuppressor`'s +/// any-buffer-size contract: feed mono `sampleRate` Float samples, receive processed +/// samples for the work that is ready (a stage may carry a remainder internally). +public protocol AudioProcessor: AnyObject { + func prepare(config: AudioProcessingConfig) + func process(_ input: [Float]) -> [Float] + func reset() + /// Best-effort added latency of this processor, in milliseconds (PRD 6.1 / SET-02). + var addedLatencyMs: Double { get } +} diff --git a/app/Sources/CrispEngine/Biquad.swift b/app/Sources/CrispEngine/Biquad.swift new file mode 100644 index 0000000..e1d9813 --- /dev/null +++ b/app/Sources/CrispEngine/Biquad.swift @@ -0,0 +1,103 @@ +import Foundation + +/// A single biquad IIR filter (RBJ Audio-EQ cookbook), transposed Direct Form II. +/// +/// Allocation-free and stateful: hold one per channel and call `process` per sample in the +/// real-time path. Coefficients are normalized by `a0` at design time so the hot loop is +/// five multiplies and four adds. +public struct Biquad { + // Normalized coefficients (a0 == 1). + private var b0: Float = 1, b1: Float = 0, b2: Float = 0 + private var a1: Float = 0, a2: Float = 0 + // State (transposed DF-II). + private var z1: Float = 0, z2: Float = 0 + + public init() {} + + @inline(__always) + public mutating func process(_ x: Float) -> Float { + let y = b0 * x + z1 + z1 = b1 * x - a1 * y + z2 + z2 = b2 * x - a2 * y + return y + } + + public mutating func reset() { z1 = 0; z2 = 0 } + + private mutating func set(_ b0: Float, _ b1: Float, _ b2: Float, _ a0: Float, _ a1: Float, _ a2: Float) { + self.b0 = b0 / a0; self.b1 = b1 / a0; self.b2 = b2 / a0 + self.a1 = a1 / a0; self.a2 = a2 / a0 + } + + // MARK: - Designers (RBJ cookbook) + + public mutating func setHighpass(freq: Float, q: Float, sampleRate: Float) { + let w0 = 2 * Float.pi * freq / sampleRate + let cw = cos(w0), sw = sin(w0) + let alpha = sw / (2 * q) + set((1 + cw) / 2, -(1 + cw), (1 + cw) / 2, + 1 + alpha, -2 * cw, 1 - alpha) + } + + public mutating func setPeaking(freq: Float, q: Float, gainDb: Float, sampleRate: Float) { + let A = pow(10, gainDb / 40) + let w0 = 2 * Float.pi * freq / sampleRate + let cw = cos(w0), sw = sin(w0) + let alpha = sw / (2 * q) + set(1 + alpha * A, -2 * cw, 1 - alpha * A, + 1 + alpha / A, -2 * cw, 1 - alpha / A) + } + + public mutating func setLowShelf(freq: Float, gainDb: Float, sampleRate: Float, slope: Float = 1) { + let A = pow(10, gainDb / 40) + let w0 = 2 * Float.pi * freq / sampleRate + let cw = cos(w0), sw = sin(w0) + let alpha = sw / 2 * sqrt((A + 1 / A) * (1 / slope - 1) + 2) + let twoSqrtAalpha = 2 * sqrt(A) * alpha + set(A * ((A + 1) - (A - 1) * cw + twoSqrtAalpha), + 2 * A * ((A - 1) - (A + 1) * cw), + A * ((A + 1) - (A - 1) * cw - twoSqrtAalpha), + (A + 1) + (A - 1) * cw + twoSqrtAalpha, + -2 * ((A - 1) + (A + 1) * cw), + (A + 1) + (A - 1) * cw - twoSqrtAalpha) + } + + public mutating func setHighShelf(freq: Float, gainDb: Float, sampleRate: Float, slope: Float = 1) { + let A = pow(10, gainDb / 40) + let w0 = 2 * Float.pi * freq / sampleRate + let cw = cos(w0), sw = sin(w0) + let alpha = sw / 2 * sqrt((A + 1 / A) * (1 / slope - 1) + 2) + let twoSqrtAalpha = 2 * sqrt(A) * alpha + set(A * ((A + 1) + (A - 1) * cw + twoSqrtAalpha), + -2 * A * ((A - 1) + (A + 1) * cw), + A * ((A + 1) + (A - 1) * cw - twoSqrtAalpha), + (A + 1) - (A - 1) * cw + twoSqrtAalpha, + 2 * ((A - 1) - (A + 1) * cw), + (A + 1) - (A - 1) * cw - twoSqrtAalpha) + } +} + +/// One-pole envelope follower with separate attack/release time constants. +/// Tracks the peak magnitude of a signal for compressor/de-esser side-chains. +public struct EnvelopeFollower { + private var env: Float = 0 + private var attackCoef: Float = 0 + private var releaseCoef: Float = 0 + + public init() {} + + public mutating func configure(attackMs: Float, releaseMs: Float, sampleRate: Float) { + attackCoef = exp(-1 / (attackMs * 0.001 * sampleRate)) + releaseCoef = exp(-1 / (releaseMs * 0.001 * sampleRate)) + } + + @inline(__always) + public mutating func process(_ x: Float) -> Float { + let mag = abs(x) + let coef = mag > env ? attackCoef : releaseCoef + env = coef * (env - mag) + mag + return env + } + + public mutating func reset() { env = 0 } +} diff --git a/app/Sources/CrispEngine/FileEnhancer.swift b/app/Sources/CrispEngine/FileEnhancer.swift index 04d3a44..5d54a33 100644 --- a/app/Sources/CrispEngine/FileEnhancer.swift +++ b/app/Sources/CrispEngine/FileEnhancer.swift @@ -8,6 +8,56 @@ public enum OutputFormat: String, CaseIterable, Identifiable, Sendable { public var ext: String { rawValue } } +/// File processing quality (PRD FR-FILE-002). Fast = denoise/DSP only; HQ adds loudness +/// normalization + peak limiting (and is where a heavier offline model would slot in). +public enum FileQuality: String, CaseIterable, Identifiable, Sendable { + case fast, hq + public var id: String { rawValue } +} + +/// Everything the file HQ pipeline needs (PRD 4.3 / 5.2). +public struct FileEnhanceOptions: Sendable { + public var mode: ProcessingMode + public var quality: FileQuality + public var enhanceStrength: EnhanceStrength + public var tonePreset: TonePreset + public var noiseAttenuationDb: Float + public var loudness: LoudnessTarget + public var format: OutputFormat + public var lowLatencyModel: Bool + /// Process only the first N seconds (PRD FR-FILE-003 preview). nil = whole file. + public var previewSeconds: Double? + + public init(mode: ProcessingMode = .cleanAndEnhance, + quality: FileQuality = .hq, + enhanceStrength: EnhanceStrength = .medium, + tonePreset: TonePreset = .natural, + noiseAttenuationDb: Float = 100, + loudness: LoudnessTarget = .podcast, + format: OutputFormat = .wav, + lowLatencyModel: Bool = false, + previewSeconds: Double? = nil) { + self.mode = mode + self.quality = quality + self.enhanceStrength = enhanceStrength + self.tonePreset = tonePreset + self.noiseAttenuationDb = noiseAttenuationDb + self.loudness = loudness + self.format = format + self.lowLatencyModel = lowLatencyModel + self.previewSeconds = previewSeconds + } +} + +/// Result of a file enhance run — surfaced to the UI for the before/after report (PRD FR-FILE-006). +public struct FileEnhanceReport: Sendable { + public let output: URL + public let inputPeakDb: Double + public let outputPeakDb: Double + public let outputLUFS: Double + public let loudnessGainDb: Double +} + public enum FileEnhanceError: LocalizedError { case noAudioTrack, modelLoadFailed, readFailed(String), writeFailed(String), cancelled public var errorDescription: String? { @@ -68,6 +118,115 @@ public final class FileEnhancer { return outputs } + // MARK: - Voice Enhancer pipeline (PRD 4.3 / 5.2) + + /// Full file pipeline: decode → (denoise) → (Voice Enhancer DSP) → (HQ loudness/peak) → write. + /// Honors `options.previewSeconds` to render a short before/after sample (FR-FILE-003). + @discardableResult + public func enhance(input: URL, + output: URL, + options: FileEnhanceOptions, + progress: ((Double) -> Void)? = nil) throws -> FileEnhanceReport { + cancelled = false + var samples = try decodeTo48kMono(input) + if let secs = options.previewSeconds { + let limit = Int(secs * 48_000) + if samples.count > limit { samples = Array(samples[0.. Float { + var p: Float = 0 + for v in s { p = max(p, abs(v)) } + return p + } + + private func write(_ samples: [Float], to output: URL, format: OutputFormat) throws { + let outSettings = Self.outputSettings(format) + let procFormat = AVAudioFormat(commonFormat: .pcmFormatFloat32, sampleRate: 48_000, channels: 1, interleaved: false)! + let outFile: AVAudioFile + do { outFile = try AVAudioFile(forWriting: output, settings: outSettings, commonFormat: .pcmFormatFloat32, interleaved: false) } + catch { throw FileEnhanceError.writeFailed(error.localizedDescription) } + // Write in chunks so a huge buffer isn't required. + let chunk = 48_000 + var i = 0 + while i < samples.count { + let n = min(chunk, samples.count - i) + guard let buf = AVAudioPCMBuffer(pcmFormat: procFormat, frameCapacity: AVAudioFrameCount(n)) else { break } + buf.frameLength = AVAudioFrameCount(n) + samples.withUnsafeBufferPointer { memcpy(buf.floatChannelData![0], $0.baseAddress! + i, n * MemoryLayout.size) } + do { try outFile.write(from: buf) } catch { throw FileEnhanceError.writeFailed(error.localizedDescription) } + i += n + } + } + + private static func outputSettings(_ format: OutputFormat) -> [String: Any] { + switch format { + case .wav: + return [AVFormatIDKey: kAudioFormatLinearPCM, AVSampleRateKey: 48_000, + AVNumberOfChannelsKey: 1, AVLinearPCMBitDepthKey: 16, + AVLinearPCMIsFloatKey: false, AVLinearPCMIsBigEndianKey: false] + case .m4a: + return [AVFormatIDKey: kAudioFormatMPEG4AAC, AVSampleRateKey: 48_000, + AVNumberOfChannelsKey: 1, AVEncoderBitRateKey: 192_000] + } + } + // MARK: - Decode (once) private func decodeTo48kMono(_ input: URL) throws -> [Float] { diff --git a/app/Sources/CrispEngine/Loudness.swift b/app/Sources/CrispEngine/Loudness.swift new file mode 100644 index 0000000..c6b48ae --- /dev/null +++ b/app/Sources/CrispEngine/Loudness.swift @@ -0,0 +1,98 @@ +import Foundation + +/// Loudness targets for the file HQ pipeline (PRD 4.3). +public enum LoudnessTarget: String, CaseIterable, Identifiable, Sendable { + case podcast // -16 LUFS + case meeting // -18 LUFS + case none // leave level untouched + public var id: String { rawValue } + public var lufs: Double? { + switch self { + case .podcast: return -16 + case .meeting: return -18 + case .none: return nil + } + } +} + +/// Integrated-loudness measurement (ITU-R BS.1770 K-weighting + gating) and normalization. +/// +/// The K-weighting and gating follow BS.1770; the final "true-peak" stage is approximated by +/// a sample-peak ceiling (no 4× oversampling), which is conservative for speech and keeps +/// the pipeline dependency-free. Documented as such in the test report. +public enum Loudness { + /// Gated integrated loudness in LUFS (mono). Returns -.infinity for silence. + public static func integratedLUFS(_ samples: [Float], sampleRate: Double) -> Double { + guard !samples.isEmpty else { return -.infinity } + let sr = Float(sampleRate) + + // K-weighting: stage 1 high-shelf (~+4 dB @ 1.5 kHz), stage 2 high-pass (~38 Hz). + var shelf = Biquad(); shelf.setHighShelf(freq: 1500, gainDb: 4, sampleRate: sr) + var hp = Biquad(); hp.setHighpass(freq: 38, q: 0.5, sampleRate: sr) + var weighted = [Float](repeating: 0, count: samples.count) + for i in 0.. 0, samples.count >= blockLen else { + let ms = meanSquare(weighted, 0, weighted.count) + return ms > 0 ? -0.691 + 10 * log10(Double(ms)) : -.infinity + } + var blockLoudness: [Double] = [] + var start = 0 + while start + blockLen <= weighted.count { + let ms = Double(meanSquare(weighted, start, blockLen)) + blockLoudness.append(ms > 0 ? -0.691 + 10 * log10(ms) : -.infinity) + start += hop + } + + // Absolute gate at -70 LUFS, then relative gate at (gated mean − 10 LU). + let absKept = blockLoudness.enumerated().filter { $0.element > -70 }.map { $0.offset } + guard !absKept.isEmpty else { return -.infinity } + let relThreshold = gatedMeanLUFS(blockLoudness, absKept) - 10 + let relKept = absKept.filter { blockLoudness[$0] > relThreshold } + let kept = relKept.isEmpty ? absKept : relKept + return gatedMeanLUFS(blockLoudness, kept) + } + + /// Normalize to `targetLUFS`, then bring sample peaks down to `ceilingDb` if needed. + /// Returns the applied gain (dB) and the measured input loudness. + @discardableResult + public static func normalize(_ samples: inout [Float], + toLUFS targetLUFS: Double, + sampleRate: Double, + ceilingDb: Double = -1.0) -> (gainDb: Double, measuredLUFS: Double) { + let measured = integratedLUFS(samples, sampleRate: sampleRate) + guard measured.isFinite else { return (0, measured) } + var gainDb = targetLUFS - measured + + // Don't let the loudness gain push peaks over the ceiling. + let ceilingLin = Float(pow(10, ceilingDb / 20)) + var peak: Float = 0 + for s in samples { peak = max(peak, abs(s)) } + if peak > 0 { + let maxGainLin = ceilingLin / peak + let maxGainDb = 20 * log10(Double(maxGainLin)) + gainDb = min(gainDb, maxGainDb) + } + let gainLin = Float(pow(10, gainDb / 20)) + for i in 0.. Float { + var sum: Float = 0 + let end = start + count + for i in start.. Double { + var energy = 0.0 + for i in indices { energy += pow(10, (blockLoudness[i] + 0.691) / 10) } + let mean = energy / Double(indices.count) + return mean > 0 ? -0.691 + 10 * log10(mean) : -.infinity + } +} diff --git a/app/Sources/CrispEngine/NoiseSuppressor.swift b/app/Sources/CrispEngine/NoiseSuppressor.swift index bdc8664..df74b98 100644 --- a/app/Sources/CrispEngine/NoiseSuppressor.swift +++ b/app/Sources/CrispEngine/NoiseSuppressor.swift @@ -10,6 +10,8 @@ public protocol NoiseSuppressor: AnyObject { var bypassed: Bool { get set } /// Returns processed samples for complete hops; may be empty while carrying. func process(_ input: [Float]) -> [Float] + /// Drain any carried sub-hop remainder (offline/file use). Streaming callers skip this. + func flush() -> [Float] func reset() } @@ -19,5 +21,6 @@ public final class PassthroughSuppressor: NoiseSuppressor { public var bypassed: Bool = false public init() {} public func process(_ input: [Float]) -> [Float] { input } + public func flush() -> [Float] { [] } public func reset() {} } diff --git a/app/Sources/CrispEngine/PipelineProcessor.swift b/app/Sources/CrispEngine/PipelineProcessor.swift new file mode 100644 index 0000000..a747d04 --- /dev/null +++ b/app/Sources/CrispEngine/PipelineProcessor.swift @@ -0,0 +1,72 @@ +import Foundation + +/// The two-stage processing graph (PRD 4.1): Noise Cancellation stage → Voice Enhancer stage. +/// +/// Both stages stay structurally in the path for every mode. The mode only changes +/// *parameters* — the suppressor's attenuation (0 dB = passthrough) and whether the +/// enhancer's dry→wet blend ramps up. Because nothing re-routes, switching modes mid-stream +/// can never introduce a sample-count discontinuity or a click (PRD 8.4). +/// +/// Streaming contract matches `NoiseSuppressor`: the suppressor emits only complete model +/// hops (carrying a sub-hop remainder), and the enhancer is 1:1, so `process` returns the +/// enhanced version of exactly the hops the suppressor produced this call. +public final class PipelineProcessor: AudioProcessor { + private let suppressor: NoiseSuppressor + private let enhancer: VoiceEnhancer + private var config: AudioProcessingConfig + + /// Inject the Noise Cancellation stage (a real `DeepFilterSuppressor`, or a passthrough + /// when the model fails to load). The Voice Enhancer stage is owned here. + public init(suppressor: NoiseSuppressor, + sampleRate: Float = 48_000, + config: AudioProcessingConfig = AudioProcessingConfig()) { + self.suppressor = suppressor + self.enhancer = VoiceEnhancer(sampleRate: sampleRate) + self.config = config + } + + public var addedLatencyMs: Double { enhancer.addedLatencyMs } + + public func prepare(config: AudioProcessingConfig) { + self.config = config + enhancer.prepare(config: config) + applyMode() + } + + /// Apply a new config live (mode / strength / tone / atten). Safe to call from the UI. + public func update(config: AudioProcessingConfig) { + let toneOrStrengthChanged = config.enhanceStrength != self.config.enhanceStrength + || config.tonePreset != self.config.tonePreset + self.config = config + if toneOrStrengthChanged { enhancer.prepare(config: config) } // preserves filter state + applyMode() + } + + private func applyMode() { + // Noise Cancellation stage: run at the mode's effective attenuation (0 → passthrough). + let denoiseDb = config.effectiveDenoiseDb + suppressor.attenuationLimitDb = denoiseDb + suppressor.bypassed = denoiseDb <= 0 + // Voice Enhancer stage: ramp in/out for the mode. + enhancer.setActive(config.mode.usesEnhancer) + } + + public func process(_ input: [Float]) -> [Float] { + let denoised = suppressor.process(input) // complete hops (carries remainder) + return enhancer.process(denoised) // 1:1 + } + + public func reset() { + suppressor.reset() + enhancer.reset() + } + + /// Drain the suppressor's carried tail through the enhancer (offline/file use only). + public func flush() -> [Float] { + enhancer.process(suppressor.flush()) + } + + /// For offline/file use: jump the enhancer blend straight to its target so the very + /// first samples are already fully processed (no listener to protect from a transient). + public func snapEnhancerToTarget() { enhancer.snapToTarget() } +} diff --git a/app/Sources/CrispEngine/VoiceEnhancer.swift b/app/Sources/CrispEngine/VoiceEnhancer.swift new file mode 100644 index 0000000..12de7b8 --- /dev/null +++ b/app/Sources/CrispEngine/VoiceEnhancer.swift @@ -0,0 +1,161 @@ +import Foundation + +/// The Voice Enhancer stage (PRD 4.2 "Realtime: lightweight enhancer + DSP", and the +/// Post-DSP block of PRD 4.1). A zero-latency, identity-preserving DSP chain — *not* a +/// generative model, so the speaker is never altered (PRD 2.3 / 6.2 / explicit exclusions). +/// +/// Chain (per sample): high-pass (rumble) → tone EQ → compressor (level uniformity) → +/// de-esser (sibilance) → output limiter (clip safety). The whole chain runs continuously +/// even when `targetWetMix == 0`, so its filter/compressor state stays warm and the +/// dry→wet blend can ramp without a startup transient — this is what makes mode switching +/// click-free (PRD 8.4). At `wetMix == 0` the output is bit-identical to the input. +/// +/// This is a swappable stage (PRD 4.5): a neural enhancer (LocalVQE / Resemble) can replace +/// it behind the same `AudioProcessor`/streaming contract without touching the pipeline. +public final class VoiceEnhancer: AudioProcessor { + private let sampleRate: Float + private var prepared = false + + // EQ + filters (one set; mono path). + private var hpf = Biquad() + private var eqA = Biquad() + private var eqB = Biquad() + private var eqC = Biquad() + private var deEssShelf = Biquad() // fixed high-shelf cut, blended in dynamically + private var deEssDetector = Biquad() // side-chain high-pass + + private var compEnv = EnvelopeFollower() + private var deEssEnv = EnvelopeFollower() + private var limiterEnv = EnvelopeFollower() + + // Derived parameters. + private var compThresholdDb: Float = -24 + private var compRatio: Float = 2 + private var makeupGainLin: Float = 1 + private var deEssThresholdDb: Float = -28 + private var deEssRangeDb: Float = 12 + private var limiterCeilingLin: Float = 0.891 // -1 dBFS + + // Dry→wet blend (ramped to avoid clicks on mode/preset change). + private var wetMix: Float = 0 + private var targetWetMix: Float = 0 + private var mixStep: Float = 0 + + public init(sampleRate: Float = 48_000) { + self.sampleRate = sampleRate + } + + public var addedLatencyMs: Double { 0 } // IIR chain, no lookahead + + /// Whether the enhancer contributes to the output. Setting this ramps the blend. + public var bypassed: Bool = false { + didSet { targetWetMix = (bypassed || baseTargetMix == 0) ? 0 : baseTargetMix } + } + private var baseTargetMix: Float = 0 + + public func prepare(config: AudioProcessingConfig) { + let intensity = config.enhanceStrength.intensity + + // Rumble / DC: gentle 2nd-order high-pass (PRD 4.2 "60~80 Hz HPF"). + hpf.setHighpass(freq: config.tonePreset == .bright ? 90 : 80, q: 0.707, sampleRate: sampleRate) + + // Tone EQ — three understandable shapes (PRD 3.2). Unused slots are 0 dB (identity). + // Designers only rewrite coefficients (not delay state), so re-preparing live to + // change tone/strength does not zero the filters and therefore does not click. + switch config.tonePreset { + case .natural: + eqA.setPeaking(freq: 3000, q: 0.9, gainDb: 2.0 * intensity, sampleRate: sampleRate) + eqB.setPeaking(freq: 1000, q: 1.0, gainDb: 0, sampleRate: sampleRate) + eqC.setPeaking(freq: 1000, q: 1.0, gainDb: 0, sampleRate: sampleRate) + case .warm: + eqA.setLowShelf(freq: 180, gainDb: 3.0 * intensity, sampleRate: sampleRate) + eqB.setHighShelf(freq: 9000, gainDb: -2.0 * intensity, sampleRate: sampleRate) + eqC.setPeaking(freq: 2500, q: 1.2, gainDb: 1.0 * intensity, sampleRate: sampleRate) + case .bright: + eqA.setPeaking(freq: 4000, q: 1.0, gainDb: 2.5 * intensity, sampleRate: sampleRate) + eqB.setHighShelf(freq: 7500, gainDb: 3.0 * intensity, sampleRate: sampleRate) + eqC.setPeaking(freq: 1000, q: 1.0, gainDb: 0, sampleRate: sampleRate) + } + + // Compressor — level uniformity (PRD 4.2 "mild compressor"). Conservative defaults. + compThresholdDb = -24 + compRatio = 1.5 + 1.5 * intensity // 1.5:1 … 3:1 + makeupGainLin = pow(10, (2.0 * intensity) / 20) // up to +2 dB + compEnv.configure(attackMs: 8, releaseMs: 120, sampleRate: sampleRate) + + // De-esser — dynamic high-shelf cut blended in on sibilance (PRD 4.2 "de-esser"). + let deEssMaxCutDb = -(3 + 5 * intensity) // up to ~ -8 dB + deEssShelf.setHighShelf(freq: 5500, gainDb: deEssMaxCutDb, sampleRate: sampleRate) + deEssDetector.setHighpass(freq: 5000, q: 0.707, sampleRate: sampleRate) + deEssThresholdDb = -30 + deEssRangeDb = 14 + deEssEnv.configure(attackMs: 1, releaseMs: 60, sampleRate: sampleRate) + + // Output limiter — clip safety at -1 dBFS (PRD 4.2 "output limiter"). + limiterCeilingLin = pow(10, -1.0 / 20) + limiterEnv.configure(attackMs: 0.5, releaseMs: 50, sampleRate: sampleRate) + + // 40 ms equal-rate dry→wet ramp (PRD 4.2 "30~80 ms crossfade"). + mixStep = 1 / (0.040 * sampleRate) + prepared = true + } + + /// Set whether enhancement is active (ramped). Pipeline calls this on mode changes. + public func setActive(_ active: Bool) { + baseTargetMix = active ? 1 : 0 + targetWetMix = (bypassed || !active) ? 0 : 1 + } + + /// Jump the blend straight to its target — for offline/file processing where there is + /// no listener to protect from a transient. + public func snapToTarget() { wetMix = targetWetMix } + + public func reset() { + hpf.reset(); eqA.reset(); eqB.reset(); eqC.reset() + deEssShelf.reset(); deEssDetector.reset() + compEnv.reset(); deEssEnv.reset(); limiterEnv.reset() + } + + public func process(_ input: [Float]) -> [Float] { + guard prepared else { return input } + var out = [Float](repeating: 0, count: input.count) + let eps: Float = 1e-7 + for i in 0.. targetWetMix { wetMix = max(targetWetMix, wetMix - mixStep) } + + // Always run the chain to keep state warm (even at wetMix 0). + var x = hpf.process(dry) + x = eqA.process(x); x = eqB.process(x); x = eqC.process(x) + + // Compressor (downward). + let env = compEnv.process(x) + let envDb = 20 * log10(env + eps) + if envDb > compThresholdDb { + let grDb = (compThresholdDb - envDb) * (1 - 1 / compRatio) // ≤ 0 + x *= pow(10, grDb / 20) + } + x *= makeupGainLin + + // De-esser: blend a fixed high-shelf cut in proportion to sibilance energy. + let det = deEssEnv.process(deEssDetector.process(x)) + let detDb = 20 * log10(det + eps) + let shelved = deEssShelf.process(x) + if detDb > deEssThresholdDb { + let a = min(1, (detDb - deEssThresholdDb) / deEssRangeDb) + x = x * (1 - a) + shelved * a + } + + // Output limiter (-1 dBFS), then hard safety clamp. + let pk = limiterEnv.process(x) + if pk > limiterCeilingLin { x *= limiterCeilingLin / pk } + x = max(-1, min(1, x)) + + out[i] = dry * (1 - wetMix) + x * wetMix + } + return out + } +} diff --git a/app/Sources/filetool/main.swift b/app/Sources/filetool/main.swift index c4b3d21..49a33cb 100644 --- a/app/Sources/filetool/main.swift +++ b/app/Sources/filetool/main.swift @@ -2,17 +2,44 @@ import CrispEngine import Foundation // Verifies FileEnhancer (the ffmpeg-free file path the app uses). -// single: filetool [ll] -// batch: filetool batch e.g. 6,24,100 +// single: filetool [ll] +// batch: filetool batch e.g. 6,24,100 +// enhance: filetool enhance [natural|warm|bright] [podcast|meeting|none] +// → full Voice Enhancer pipeline (denoise → DSP enhance → HQ loudness/peak) func die(_ msg: String, _ code: Int32 = 1) -> Never { FileHandle.standardError.write((msg + "\n").data(using: .utf8)!) exit(code) } +func parseMode(_ s: String) -> ProcessingMode? { + switch s { + case "off": return .off + case "noise": return .noiseCancellation + case "voice": return .voiceEnhancer + case "clean": return .cleanAndEnhance + default: return ProcessingMode(rawValue: s) + } +} + let a = CommandLine.arguments do { - if a.count >= 6 && a[1] == "batch", let fmt = OutputFormat(rawValue: a[4]) { + if a.count >= 6 && a[1] == "enhance", let mode = parseMode(a[4]), let q = FileQuality(rawValue: a[5]) { + let tone = a.count >= 7 ? (TonePreset(rawValue: a[6]) ?? .natural) : .natural + let loud = a.count >= 8 ? (LoudnessTarget(rawValue: a[7]) ?? .podcast) : .podcast + let ext = URL(fileURLWithPath: a[3]).pathExtension.lowercased() + let fmt = OutputFormat(rawValue: ext) ?? .wav + let opts = FileEnhanceOptions(mode: mode, quality: q, enhanceStrength: .medium, + tonePreset: tone, noiseAttenuationDb: 100, + loudness: loud, format: fmt) + let r = try FileEnhancer().enhance(input: URL(fileURLWithPath: a[2]), + output: URL(fileURLWithPath: a[3]), options: opts) { p in + FileHandle.standardError.write(String(format: "\r%.0f%% ", p * 100).data(using: .utf8)!) + } + FileHandle.standardError.write(String(format: "\nOK out=%.1f LUFS peak %.1f→%.1f dBFS gain %+.1f dB\n", + r.outputLUFS, r.inputPeakDb, r.outputPeakDb, r.loudnessGainDb).data(using: .utf8)!) + print(r.output.path) + } else if a.count >= 6 && a[1] == "batch", let fmt = OutputFormat(rawValue: a[4]) { let levels = a[5].split(separator: ",").compactMap { Float($0) } let urls = try FileEnhancer().enhanceBatch( input: URL(fileURLWithPath: a[2]), @@ -31,7 +58,7 @@ do { } FileHandle.standardError.write("\nOK\n".data(using: .utf8)!) } else { - die("usage:\n filetool [ll]\n filetool batch ", 2) + die("usage:\n filetool [ll]\n filetool batch \n filetool enhance [natural|warm|bright] [podcast|meeting|none]", 2) } } catch { die("\n\(error.localizedDescription)") diff --git a/app/Sources/vetool/main.swift b/app/Sources/vetool/main.swift new file mode 100644 index 0000000..2fbdb57 --- /dev/null +++ b/app/Sources/vetool/main.swift @@ -0,0 +1,119 @@ +import CrispEngine +import Foundation + +// Verifies the Voice Enhancer DSP stage and the two-stage pipeline. +// vetool [model.tar.gz] +// With no args it runs the self-contained DSP checks (no model needed). Given a model path +// it additionally measures the full pipeline RTF (DeepFilter denoise → enhancer). +// +// Checks (mirroring dftool's streaming-determinism guarantee for the suppressor): +// 1. wetMix==0 is bit-identical passthrough (so Off / Noise-only modes don't color audio) +// 2. enhancer output is independent of buffer chunking (state correctness) +// 3. output never exceeds full scale (limiter / clamp safety) +// 4. enhancement measurably changes the signal when active + +func log(_ s: String) { FileHandle.standardError.write((s + "\n").data(using: .utf8)!) } +func fail(_ s: String) -> Never { log("FAIL: \(s)"); exit(1) } + +let sr = 48_000.0 + +/// Deterministic 48 kHz mono test signal: voice-like tones + sibilance band + noise + a +/// loud transient (to exercise the limiter). Fixed-seed LCG → reproducible. +func synth(seconds: Double) -> [Float] { + let n = Int(seconds * sr) + var out = [Float](repeating: 0, count: n) + var seed: UInt64 = 0x1234_5678 + func rnd() -> Float { + seed = seed &* 6364136223846793005 &+ 1442695040888963407 + return Float(Int32(truncatingIfNeeded: seed >> 33)) / Float(Int32.max) + } + for i in 0.. VoiceEnhancer { + let e = VoiceEnhancer(sampleRate: Float(sr)) + e.prepare(config: AudioProcessingConfig(enhanceStrength: strength, tonePreset: tone)) + e.setActive(active) + e.snapToTarget() + return e +} + +func processChunked(_ e: VoiceEnhancer, _ x: [Float], sizes: [Int]) -> [Float] { + var out: [Float] = []; out.reserveCapacity(x.count) + var i = 0, k = 0 + while i < x.count { + let n = min(sizes[k % sizes.count], x.count - i) + out.append(contentsOf: e.process(Array(x[i..<(i + n)]))) + i += n; k += 1 + } + return out +} + +let signal = synth(seconds: 5) +log("test signal: \(signal.count) samples (\(Double(signal.count) / sr)s @ 48k)") + +// 1. wetMix == 0 → bit-identical passthrough. +let pass = makeEnhancer(active: false) +let passOut = pass.process(signal) +if passOut != signal { fail("inactive enhancer is not bit-identical passthrough") } +log("PASS 1/4 — inactive enhancer is exact passthrough") + +// 2. Chunk independence — aligned vs varied chunk sizes must match bit-for-bit. +let aligned = makeEnhancer(active: true) +let alignedOut = aligned.process(signal) +let chunked = makeEnhancer(active: true) +let chunkedOut = processChunked(chunked, signal, sizes: [137, 480, 53, 911, 256, 1000]) +if alignedOut.count != chunkedOut.count { fail("length mismatch \(alignedOut.count) vs \(chunkedOut.count)") } +var maxDiff: Float = 0 +for i in 0.. 1.0001 { fail("output peak \(peak) exceeds full scale") } +log(String(format: "PASS 3/4 — output peak %.4f within full scale (limiter ok)", peak)) + +// 4. Enhancement actually changes the signal. +var changeEnergy: Double = 0, refEnergy: Double = 0 +for i in 0.. Void) { + let t0 = DispatchTime.now().uptimeNanoseconds + body() + let proc = Double(DispatchTime.now().uptimeNanoseconds - t0) / 1e9 + let audio = Double(signal.count) / sr + log(String(format: "RTF %@: proc=%.4fs audio=%.2fs RTF=%.4f", label, proc, audio, proc / audio)) +} + +let benchEnh = makeEnhancer(active: true) +rtf("enhancer-only") { _ = benchEnh.process(signal) } + +if CommandLine.arguments.count >= 2 { + let modelPath = CommandLine.arguments[1] + guard let df = DeepFilterSuppressor(modelPath: modelPath) else { fail("model load failed: \(modelPath)") } + let pipe = PipelineProcessor(suppressor: df) + pipe.prepare(config: AudioProcessingConfig(mode: .cleanAndEnhance, enhanceStrength: .medium, tonePreset: .natural)) + pipe.snapEnhancerToTarget() + rtf("clean+enhance pipeline") { + var i = 0 + while i < signal.count { let n = min(2400, signal.count - i); _ = pipe.process(Array(signal[i..<(i + n)])); i += n } + _ = pipe.flush() + } +} + +log("ALL CHECKS PASSED") diff --git a/docs/STATUS.md b/docs/STATUS.md index 2602348..5590f1e 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -17,6 +17,7 @@ PRD(`krisp_like_mac_prd_workplan.docx`)의 M0–M6 마일스톤을 **검증 가 | 4 | M4 / FILE-* | 파일 처리 | ✅ | **ffmpeg-free**(AVFoundation+libdf), mp3→m4a, mp4→wav | | 5 | M5 | 서명/notarization/패키징 | 🔶 | pkg 생성(앱+드라이버+2모델), **notarize는 Apple 계정 필요** | | 6 | M6 | 문서/라이선스/리포트 | ✅ | architecture/install/LICENSES/known-issues | +| **VE** | **v0.2** | **Voice Enhancer 파이프라인** | ✅ | 2-stage 파이프라인 + DSP 인핸서, 4모드, 파일 HQ(LUFS/peak), `vetool` 검증 | ## 검증된 성공 기준 (PRD 1.4 / 8장 대비) @@ -35,6 +36,18 @@ PRD(`krisp_like_mac_prd_workplan.docx`)의 M0–M6 마일스톤을 **검증 가 - ✅ **30분 연속 안정성: crash 0 · dropout 0** (PASS) - ✅ 객관 음질 DNSMOS: OVRL +0.55, BAK +1.45 +## Voice Enhancer 파이프라인 (PRD v0.2) — 이번 추가 + +PRD `krisp_like_mac_voice_enhancer_prd_v02.docx` 구현. 상세: [`docs/voice-enhancer.md`](voice-enhancer.md). + +- ✅ **4개 처리 모드** — Off / Noise Cancellation / Voice Enhancer / Clean + Enhance (`ProcessingMode`, UI/영속화) +- ✅ **2-stage 파이프라인** — `PipelineProcessor`(DeepFilter denoise → `VoiceEnhancer` DSP), 교체 가능 stage(PRD §4.5) +- ✅ **DSP 인핸서** — HPF·톤 EQ(3종)·컴프레서·디에서·−1dBFS 리미터, 추가 지연 ≈ 0ms +- ✅ **클릭 없는 전환** — 항상 in-path + dry→wet 40ms 램프, `wetMix=0` bit-identical 패스스루(검증) +- ✅ **스트리밍 결정성** — `vetool`: aligned == chunked bit-identical, 리미터 안전, RTF enhancer 0.012 / clean+enhance 0.11 +- ✅ **파일 HQ** — Fast/HQ, 미리듣기(20초), LUFS 정규화(-16/-18) + peak 천장(-1dBFS), before/after 리포트 +- ✅ **모델/가중치 추가 0** — 생성형 아님(화자 보존), 라이선스 BOM 변화 없음. LocalVQE/Resemble는 동일 seam에 드롭인 가능(PRD §9 M1 별도) + ## 남은 항목 — 사람/계정/GUI 앱 필요 (자동화 불가) 절차: [`docs/acceptance-checklist.md`](acceptance-checklist.md) diff --git a/docs/architecture.md b/docs/architecture.md index 194d2d9..e3187cc 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -57,6 +57,31 @@ PRD 6장(권장 기술 아키텍처)을 구현 기준으로 구체화한 문서. **input stream**으로 loopback 되어 회의 앱이 "Noise Cancelled Microphone"으로 수신한다. (`driver/CrispAudioDriver/CrispAudioDriver.c` 의 `Crisp_DoIOOperation`) +## Voice Enhancer 2-stage 파이프라인 (PRD v0.2) + +기존 단일 denoiser를 **교체 가능한 2-stage 그래프**로 확장했다(PRD §4.1/4.5). 상세: +[`docs/voice-enhancer.md`](voice-enhancer.md). + +``` +입력(48k mono) + → [Stage 1] Noise Cancellation : DeepFilterSuppressor (atten = mode별, 0 = passthrough) + → [Stage 2] Voice Enhancer : VoiceEnhancer (HPF→톤EQ→comp→de-ess→limiter, dry→wet 램프) + → 출력 +``` + +| 구성 | 파일 | 역할 | +|---|---|---| +| `ProcessingMode` / `AudioProcessingConfig` / `AudioProcessor` | `CrispEngine/AudioProcessing.swift` | 모드·설정·교체 가능 stage 인터페이스(PRD §4.5) | +| `PipelineProcessor` | `CrispEngine/PipelineProcessor.swift` | stage 그래프, 모드→파라미터 매핑, flush | +| `VoiceEnhancer` | `CrispEngine/VoiceEnhancer.swift` | DSP 인핸서(zero-latency, 생성형 아님) | +| `Biquad` / `EnvelopeFollower` | `CrispEngine/Biquad.swift` | RBJ 필터 + 엔벨로프(할당 없는 per-sample) | +| `Loudness` | `CrispEngine/Loudness.swift` | BS.1770 LUFS 측정 + 정규화 + peak 천장(파일 HQ) | + +**모드는 라우팅을 바꾸지 않는다.** 두 stage 모두 항상 경로에 있고, 모드는 suppressor의 +attenuation과 enhancer의 dry→wet 블렌드만 조정한다 → 모드 전환 시 샘플 불연속/클릭 없음(§8.4). +`wetMix==0`이면 enhancer 출력은 입력과 bit-identical(`vetool` 검증). 라이브 강도/톤 변경은 +필터 계수만 재계산(상태 보존)하므로 역시 클릭이 없다. + ## libDF C-API (Phase 2 연동 지점) ```c diff --git a/docs/test-report.md b/docs/test-report.md index 9ca5db6..3d09d87 100644 --- a/docs/test-report.md +++ b/docs/test-report.md @@ -70,6 +70,34 @@ no-reference DNSMOS (1–5, 높을수록 좋음), noisy_snr0 샘플, 16kHz: | **30분 연속 안정성** (model+AUHAL+driver) | 1800s, 10/10 spot-check non-silent, **crash 0 · dropout 0** | ✅ | | 설치 패키지 (앱→/Applications, 드라이버→/Library) | relocation 버그 수정 후 검증 | ✅ | +## Voice Enhancer 파이프라인 (PRD v0.2) — `vetool` / `filetool enhance` + +DSP 인핸서 + 2-stage 파이프라인. 구현/설계: [`voice-enhancer.md`](voice-enhancer.md). + +| 검증 항목 | 기대 | 측정 | 판정 | +|---|---|---|---| +| 비활성 인핸서 = 패스스루 | bit-identical | `wetMix=0` 출력 == 입력 | ✅ | +| 스트리밍 결정성 | aligned == chunked | 임의 버퍼(137/480/53/911/256/1000) bit-identical | ✅ | +| 리미터/클램프 안전 | peak ≤ full scale | 입력 1.5 트랜지언트 → 출력 peak **0.75** | ✅ | +| 인핸스 효과 | 신호 변화 | Δenergy 측정됨 | ✅ | +| RTF (enhancer 단독) | ≪ 1 | **0.012** (≈84×) | ✅ | +| RTF (clean+enhance) | < 1 | **0.11** (≈8.9×) | ✅ | +| 추가 지연 | §6.1 ≤ 50ms | DSP **0 ms**(IIR, no lookahead) + denoise hop | ✅ | + +파일 HQ 파이프라인 (`filetool enhance`, noisy_snr0.wav 10.6s): + +| 모드/품질 | 출력 LUFS | peak in→out | gain | 비고 | +|---|---|---|---|---| +| clean+enhance / HQ podcast | −19.3 | −3.5 → −1.0 dBFS | +6.9 dB | peak 천장에 의해 −16 미달(클립 방지) | +| voice·warm / HQ meeting | −19.0 | −3.5 → −1.0 dBFS | +7.3 dB | | +| noise / Fast | −23.7 | −3.5 → −3.9 dBFS | +0.0 dB | Fast=정규화 미적용 | +| off / HQ podcast | −21.0 | −3.5 → −1.0 dBFS | +2.5 dB | 변환+정규화만 | + +출력 48k mono Int16 WAV / AAC m4a, 길이 정확 보존(10.595646s in=out). `wav/mp3/m4a/mp4/mov` 입력. + +> **한계**: true-peak(-1 dBTP)는 오버샘플링 없이 sample-peak 천장으로 근사. LUFS는 BS.1770 +> 근사(K-weighting + 절대/상대 게이팅). LocalVQE/Resemble 실모델은 동일 seam 드롭인(PRD §9 M1 별도). + ## 미검증 (사람/계정 필요 — 자동화 불가) - A/B blind 청취 ≥80% 개선 (PRD 8.3/8.4) — **사람 청취 필요**. 객관 proxy(DNSMOS)는 별도 측정 가능. diff --git a/docs/voice-enhancer.md b/docs/voice-enhancer.md new file mode 100644 index 0000000..4e94fb9 --- /dev/null +++ b/docs/voice-enhancer.md @@ -0,0 +1,110 @@ +# Crisp — Voice Enhancer 파이프라인 (PRD v0.2 구현) + +`krisp_like_mac_voice_enhancer_prd_v02.docx` 의 Voice Enhancer 추가 요구사항을 기존 +DeepFilterNet 노이즈 캔슬링 MVP 위에 구현한 내용. 본 문서는 PRD 각 절 ↔ 구현 매핑과 +모델 선택 판단, 검증 결과를 기록한다. + +## 핵심 설계 결정 — 모델 vs DSP + +PRD는 실시간 Voice Enhancer 후보로 **LocalVQE**, 파일 HQ 후보로 **Resemble Enhance / +ClearerVoice** 를 "1순위 PoC"(§1, §2.1, §9 M1)로 제시한다. 이들은 **다운로드·벤치마크· +라이선스 확인이 선행되어야 하는 별도 R&D 마일스톤**이며, 본 변경에서 가중치를 번들하지 +않는다. + +대신 PRD가 명시적으로 허용하는 경로를 택했다: + +- §4.2 "Realtime: LocalVQE **or lightweight enhancer + DSP**" +- §4.5 "Noise Cancellation stage와 Voice Enhancer stage를 **독립 모듈로 교체 가능한 구조**" + +→ Voice Enhancer stage를 **순수 Swift DSP 인핸서**(HPF·톤 EQ·컴프레서·디에서·리미터)로 +구현하고, 동일한 `AudioProcessor` 스트리밍 인터페이스 뒤에 둔다. 추후 LocalVQE/Resemble +래퍼를 **같은 seam에 드롭인**할 수 있다(코드 변경 없이 stage 교체). + +**이 선택의 이점** + +- 신규 서드파티/모델 weight **0개** → 라이선스 BOM 변화 없음(PRD §0.3 상용성, §7.2). +- **생성형 아님** → 화자 정체성·발화 내용 보존(PRD §2.3, §6.2, 명시적 제외 범위). +- 추가 지연 **≈ 0 ms**(IIR, lookahead 없음) → PRD §6.1 실시간 예산 여유 유지. + +## PRD ↔ 구현 매핑 + +| PRD | 요구 | 구현 | +|---|---|---| +| §2.2 / FR-RT-002 | Off / Noise Cancellation / Voice Enhancer / Clean + Enhance | `ProcessingMode` (`AudioProcessing.swift`), UI 4-mode picker | +| §4.1 | 2-stage 파이프라인 (Clean → Enhance + Post-DSP) | `PipelineProcessor` (suppressor → enhancer) | +| §4.2 | 실시간: HPF, mild compressor, de-esser, output limiter, preset EQ | `VoiceEnhancer` (Biquad HPF/EQ + comp + de-ess + limiter) | +| §4.2 | 모드 전환 30~80ms crossfade, pop/click 없음 | 항상 in-path + dry→wet 40ms 램프, atten 0 = bit-identical | +| §4.3 | 파일 HQ: decode → (de)noise → enhance → loudness → true-peak | `FileEnhancer.enhance(options:)` + `Loudness` | +| §4.5 | `AudioProcessingConfig` / `AudioProcessor` / 교체 가능 stage | `AudioProcessing.swift` (config + protocol) | +| §3.2 / FR-RT-003 | Enhance Strength Low/Med/High | `EnhanceStrength` (intensity 0.45/0.75/1.0) | +| §3.2 | Tone Preset 3종 | `TonePreset` natural/warm/bright | +| FR-FILE-002 | Fast / HQ | `FileQuality`; HQ에서만 loudness 정규화 적용 | +| FR-FILE-003 | 처리 전 15~30초 미리듣기 | `FileEnhanceOptions.previewSeconds` + UI "미리듣기(20초)" | +| FR-FILE-005 | 원본 비덮어쓰기, WAV/오디오교체 저장 | NSSavePanel 별도 출력, wav/m4a | +| FR-FILE-006 | before/after loudness/peak 리포트 | `FileEnhanceReport` (LUFS·peak·gain) UI 표기 | +| §4.3 step6 | -16 LUFS(podcast) / -18 LUFS(meeting) | `LoudnessTarget` (BS.1770 K-weighting + 게이팅) | +| §4.3 step7 | -1.0 dBTP true-peak | sample-peak 천장 -1 dBFS 근사(아래 한계 참고) | +| SET-02 | model/runtime/sr/latency 진단 | Diagnostics 확장 (mode/강도/톤/추정 지연) | +| §6.2 | High는 기본값 아님, 보수적 기본 | 기본 Clean+Enhance / Medium / Natural | + +## 실시간 파이프라인 (PRD §4.2) + +``` +mic → AVAudioConverter(48k mono) + → DeepFilterSuppressor (Noise Cancellation stage; atten = mode별) + → VoiceEnhancer (HPF → 톤 EQ → compressor → de-esser → limiter; dry→wet 램프) + → ring → AUHAL → 가상 마이크 +``` + +**클릭 없는 모드 전환의 핵심**: 두 stage는 모든 모드에서 **구조적으로 항상 신호 경로에 +존재**한다. 모드는 파라미터만 바꾼다 — suppressor의 attenuation(0dB=패스스루)과 enhancer의 +dry→wet 블렌드(40ms 램프). 라우팅이 재구성되지 않으므로 샘플 수 불연속이나 클릭이 원천적으로 +발생하지 않는다(PRD §8.4). `wetMix==0`이면 enhancer 출력은 입력과 **bit-identical**(검증됨). + +## Voice Enhancer DSP 체인 + +1. **High-pass** ~80Hz (rumble/DC 제거; PRD §4.2 "60~80 Hz HPF") +2. **Tone EQ** — natural(presence +), warm(low-shelf +, high-shelf −), bright(presence + air +) +3. **Compressor** — downward, ratio 1.5~3:1(강도별), makeup ≤+2dB → 음량 균일도 +4. **De-esser** — 5kHz 사이드체인 검출 → 고정 high-shelf cut(≤−8dB)을 사이빌런스 비례 블렌드 +5. **Output limiter** — −1 dBFS 천장 + 하드 클램프(클립 방지) + +전 단계 `Biquad`(RBJ cookbook, transposed DF-II) + `EnvelopeFollower` 기반, **할당 없는 per-sample** +처리. 강도/톤 변경은 계수만 재계산(딜레이 상태 보존) → 라이브 변경도 클릭 없음. + +## 검증 결과 + +`vetool`(DSP/파이프라인 검증 CLI, `dftool`의 enhancer 버전): + +``` +$ vetool # DSP 단독 (모델 불필요) +PASS 1/4 — inactive enhancer is exact passthrough # wetMix=0 bit-identical +PASS 2/4 — enhancer output is independent of buffer chunking # 스트리밍 결정성 +PASS 3/4 — output peak 0.7537 within full scale (limiter ok) # 리미터/클램프 +PASS 4/4 — active enhancer changes signal +RTF enhancer-only: 0.0119 # ≈ 84× 실시간 + +$ vetool engine/models/DeepFilterNet3_onnx.tar.gz +RTF clean+enhance pipeline: 0.1126 # ≈ 8.9× 실시간 (denoise+enhance) +``` + +`filetool enhance` (파일 HQ 파이프라인, `test/fixtures/noisy_snr0.wav` 10.6s): + +| 모드 | 품질 | 출력 LUFS | peak(in→out) | gain | 비고 | +|---|---|---|---|---|---| +| clean+enhance | HQ podcast | −19.3 | −3.5 → −1.0 dBFS | +6.9 dB | peak 천장에 의해 −16 미달(클립 방지) | +| voice/warm | HQ meeting | −19.0 | −3.5 → −1.0 dBFS | +7.3 dB | | +| noise | Fast | −23.7 | −3.5 → −3.9 dBFS | +0.0 dB | Fast=정규화 미적용 | +| off | HQ podcast | −21.0 | −3.5 → −1.0 dBFS | +2.5 dB | 변환+정규화만 | + +출력은 48k mono Int16 WAV / AAC m4a, **길이 정확 보존**(10.595646s in=out → hop carry+flush +샘플 정확). 모든 모드 정상. + +## 한계 / 후속 + +- **True-peak(-1 dBTP)** 는 4× 오버샘플링 없이 **sample-peak 천장**으로 근사(음성에서 보수적). + 방송 수준 정밀 true-peak가 필요하면 oversampling 리미터로 교체. +- **LUFS** 게이팅은 절대(-70)·상대(-10 LU) 게이트 구현, K-weighting은 BS.1770 근사 계수. +- **LocalVQE / Resemble / ClearerVoice 실모델**: 동일 `AudioProcessor` seam에 래퍼로 드롭인 + 가능. 가중치 다운로드·벤치마크·라이선스 확인은 PRD §9 M1 별도 마일스톤. +- A/B blind 청취(PRD §8.3) 및 회의 앱 실사용은 사람 검수 필요(`docs/acceptance-checklist.md`). From 9159ea2ff3b2a2ccc6874a19e79f6bf692c869f2 Mon Sep 17 00:00:00 2001 From: Hyeon-Mook Jelly Choi Date: Tue, 23 Jun 2026 20:51:21 +0900 Subject: [PATCH 2/6] =?UTF-8?q?test(enhancer):=20=ED=8F=89=EA=B0=80=20?= =?UTF-8?q?=ED=85=8C=EC=8A=A4=ED=8A=B8=EC=85=8B=20+=20XCTest=20=EC=8A=A4?= =?UTF-8?q?=EC=9C=84=ED=8A=B8=20+=20=EB=B0=B0=EC=B9=98=20=ED=92=88?= =?UTF-8?q?=EC=A7=88=20=ED=8F=89=EA=B0=80?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit PRD §8.1 테스트셋, §8.2 자동 평가, §9.4 batch 스크립트+CSV 납품물을 구현한다. 다운로드 없이 레포의 실제 음성·노이즈 자산에서 재현 가능. 엔진: - AudioIO: 48k mono WAV 읽기/쓰기 (AVFoundation, ffmpeg-free) - Metrics: RMS/peak dB, SI-SDR, 지연 보정 SI-SDR(cross-correlation 정렬), non-finite 검출 - TestCorpus: 실제 음성+노이즈에서 코퍼스 생성 — noisy(SNR 0/5/10/20), reverb(Schroeder), clipping, bandlimit(8/16k), 저음량, hum - Biquad: setLowpass 추가 (코퍼스 대역 제한용) CLI: - maketestset: 코퍼스 생성 → test/corpus/*.wav + manifest.csv - evaltool: 파일 메트릭(LUFS/peak/RMS/SI-SDR) CSV 출력 테스트 (swift test, 26개 0 실패): - BiquadTests/VoiceEnhancerTests/LoudnessTests/MetricsTests/PipelineTests - IntegrationTests: 코퍼스 생성 유효성 + 파일 end-to-end(모델). 자산/모델 없으면 XCTSkip 스크립트: - scripts/quality-eval.sh: 코퍼스 × 모드 처리 → quality-report.csv. non-finite·클리핑 시 exit 1 (회귀 게이트). 측정: noisy_snr0 denoise SI-SDR 0→+4.89dB, 모든 출력 −1dBFS 천장 준수 test/corpus 산출물(wav/manifest/report)은 gitignore (재현 가능), README만 추적. docs: voice-enhancer/test-report/README에 테스트셋·평가 결과 반영. Co-Authored-By: Claude Opus 4.8 (1M context) --- .gitignore | 5 + README.md | 20 ++- app/Package.swift | 21 +++ app/Sources/CrispEngine/AudioIO.swift | 72 ++++++++ app/Sources/CrispEngine/Biquad.swift | 8 + app/Sources/CrispEngine/Metrics.swift | 82 +++++++++ app/Sources/CrispEngine/TestCorpus.swift | 168 ++++++++++++++++++ app/Sources/evaltool/main.swift | 36 ++++ app/Sources/maketestset/main.swift | 30 ++++ app/Tests/CrispEngineTests/BiquadTests.swift | 55 ++++++ .../CrispEngineTests/IntegrationTests.swift | 71 ++++++++ .../CrispEngineTests/LoudnessTests.swift | 39 ++++ app/Tests/CrispEngineTests/MetricsTests.swift | 40 +++++ .../CrispEngineTests/PipelineTests.swift | 45 +++++ app/Tests/CrispEngineTests/TestSupport.swift | 63 +++++++ .../CrispEngineTests/VoiceEnhancerTests.swift | 70 ++++++++ docs/test-report.md | 33 ++++ docs/voice-enhancer.md | 30 ++++ scripts/quality-eval.sh | 67 +++++++ test/corpus/README.md | 40 +++++ 20 files changed, 987 insertions(+), 8 deletions(-) create mode 100644 app/Sources/CrispEngine/AudioIO.swift create mode 100644 app/Sources/CrispEngine/Metrics.swift create mode 100644 app/Sources/CrispEngine/TestCorpus.swift create mode 100644 app/Sources/evaltool/main.swift create mode 100644 app/Sources/maketestset/main.swift create mode 100644 app/Tests/CrispEngineTests/BiquadTests.swift create mode 100644 app/Tests/CrispEngineTests/IntegrationTests.swift create mode 100644 app/Tests/CrispEngineTests/LoudnessTests.swift create mode 100644 app/Tests/CrispEngineTests/MetricsTests.swift create mode 100644 app/Tests/CrispEngineTests/PipelineTests.swift create mode 100644 app/Tests/CrispEngineTests/TestSupport.swift create mode 100644 app/Tests/CrispEngineTests/VoiceEnhancerTests.swift create mode 100755 scripts/quality-eval.sh create mode 100644 test/corpus/README.md diff --git a/.gitignore b/.gitignore index ee1f0ad..c51a6e3 100644 --- a/.gitignore +++ b/.gitignore @@ -28,5 +28,10 @@ models/*.tar.gz *.f32 !test/fixtures/*.wav +# Generated evaluation corpus + report (reproduce: maketestset / scripts/quality-eval.sh) +/test/corpus/manifest.csv +/test/corpus/out/ +/test/corpus/quality-report.csv + # Internal PRD spec — not published *.docx diff --git a/README.md b/README.md index 607ba92..2fc3b30 100644 --- a/README.md +++ b/README.md @@ -70,14 +70,16 @@ open build/Crisp-0.1.0.pkg ## 검증 / 테스트 ```sh +swift test --package-path app # 단위·통합 테스트 (26개: DSP·파이프라인·LUFS·메트릭·파일 end-to-end) +./scripts/quality-eval.sh # 테스트셋 생성+배치 처리 → quality-report.csv (회귀 게이트) ./poc/model/run_poc.sh # 모델 RTF·노이즈 감소 ./scripts/e2e-test.sh # end-to-end: model→가상마이크→녹음 (드라이버 설치 필요) ./scripts/stability-test.sh 1800 # 30분 안정성 (crash/dropout) BIN="$(cd app && swift build --show-bin-path)" -"$BIN/dftool" engine/models/DeepFilterNet3_onnx.tar.gz 100 build/in.f32 out.f32 chunked # 노이즈 stage 스트리밍 정확성 -"$BIN/vetool" engine/models/DeepFilterNet3_onnx.tar.gz # 인핸서/파이프라인(결정성·리미터·RTF) -"$BIN/filetool" input.mp3 out.m4a m4a # 파일 denoise -"$BIN/filetool" enhance input.mp3 out.wav clean hq natural podcast # 파일 Voice Enhancer 파이프라인 +"$BIN/dftool" engine/models/DeepFilterNet3_onnx.tar.gz 100 build/in.f32 out.f32 chunked # 노이즈 stage 스트리밍 정확성 +"$BIN/vetool" engine/models/DeepFilterNet3_onnx.tar.gz # 인핸서/파이프라인(결정성·리미터·RTF) +"$BIN/maketestset" test/corpus # PRD 8.1 테스트셋 생성(실제 자산 기반) +"$BIN/filetool" enhance input.mp3 out.wav clean hq natural podcast # 파일 Voice Enhancer 파이프라인 ``` ## 프로젝트 구조 @@ -89,13 +91,15 @@ app/Package.swift SwiftPM: CDeepFilter / CrispEngine / CrispApp / dftoo Sources/CrispEngine/ 순수 DSP/모델(AVFoundation): DeepFilterSuppressor, FileEnhancer, VirtualMicOutput(AUHAL), CoreAudioDevices ← 앱·CLI 공유 Voice Enhancer: AudioProcessing(모드/config/protocol), PipelineProcessor, - VoiceEnhancer, Biquad, Loudness + VoiceEnhancer, Biquad, Loudness · 평가: Metrics, AudioIO, TestCorpus Sources/CrispApp/ SwiftUI 메뉴바 앱 (AppState, AudioEngineController, Views) - Sources/{dftool,vetool,filetool,mictool}/ 검증 CLI (vetool = 인핸서/파이프라인) + Sources/{dftool,vetool,filetool,mictool,maketestset,evaltool}/ 검증·평가 CLI + Tests/CrispEngineTests/ XCTest (swift test): DSP·파이프라인·LUFS·메트릭·파일 end-to-end engine/CDeepFilter/lib/ libdf.dylib (fetch-deps 생성, gitignore) engine/models/ DeepFilterNet3 모델 (fetch-deps 생성, gitignore) -scripts/ fetch-deps / install / verify / package / e2e / stability -docs/ architecture · STATUS · test-report · install · acceptance-checklist · known-issues +test/corpus/ 평가 테스트셋 (maketestset 생성, gitignore) + README +scripts/ fetch-deps / install / verify / package / e2e / stability / quality-eval +docs/ architecture · voice-enhancer · STATUS · test-report · install · acceptance-checklist poc/model/ DeepFilterNet 클론(gitignore) + run_poc.sh ``` diff --git a/app/Package.swift b/app/Package.swift index f683e87..5a9afcf 100644 --- a/app/Package.swift +++ b/app/Package.swift @@ -62,6 +62,27 @@ let package = Package( dependencies: ["CrispEngine", "CDeepFilter"], path: "Sources/vetool", linkerSettings: [dfLinker] + ), + // Generates the evaluation corpus (PRD 8.1) from in-repo speech + noise. + .executableTarget( + name: "maketestset", + dependencies: ["CrispEngine"], + path: "Sources/maketestset", + linkerSettings: [dfLinker] + ), + // Objective metrics for a processed file (PRD 8.2): LUFS, peak, RMS, SI-SDR. + .executableTarget( + name: "evaltool", + dependencies: ["CrispEngine"], + path: "Sources/evaltool", + linkerSettings: [dfLinker] + ), + // Automated unit + integration tests (`swift test`). + .testTarget( + name: "CrispEngineTests", + dependencies: ["CrispEngine", "CDeepFilter"], + path: "Tests/CrispEngineTests", + linkerSettings: [dfLinker] ) ] ) diff --git a/app/Sources/CrispEngine/AudioIO.swift b/app/Sources/CrispEngine/AudioIO.swift new file mode 100644 index 0000000..3743293 --- /dev/null +++ b/app/Sources/CrispEngine/AudioIO.swift @@ -0,0 +1,72 @@ +import Foundation +import AVFoundation + +/// Minimal, dependency-free audio file I/O at the engine's canonical format (48 kHz mono +/// Float). Shared by the test-set generator, the eval CLI, and tests. Uses AVFoundation only +/// (no ffmpeg), matching `FileEnhancer`. +public enum AudioIO { + public static let sampleRate = 48_000.0 + + public enum IOError: LocalizedError { + case noAudioTrack, readFailed(String), writeFailed(String) + public var errorDescription: String? { + switch self { + case .noAudioTrack: return "오디오 트랙 없음" + case .readFailed(let m): return "읽기 실패: \(m)" + case .writeFailed(let m): return "쓰기 실패: \(m)" + } + } + } + + /// Decode + resample + downmix any audio/video file to 48 kHz mono Float. + public static func read48kMono(_ url: URL) throws -> [Float] { + let asset = AVURLAsset(url: url) + guard let track = asset.tracks(withMediaType: .audio).first else { throw IOError.noAudioTrack } + let reader: AVAssetReader + do { reader = try AVAssetReader(asset: asset) } catch { throw IOError.readFailed(error.localizedDescription) } + let settings: [String: Any] = [ + AVFormatIDKey: kAudioFormatLinearPCM, AVLinearPCMBitDepthKey: 32, + AVLinearPCMIsFloatKey: true, AVLinearPCMIsNonInterleaved: false, AVLinearPCMIsBigEndianKey: false, + AVSampleRateKey: sampleRate, AVNumberOfChannelsKey: 1 + ] + let out = AVAssetReaderTrackOutput(track: track, outputSettings: settings) + out.alwaysCopiesSampleData = false + reader.add(out) + guard reader.startReading() else { throw IOError.readFailed("reader start") } + + var samples: [Float] = [] + while let sb = out.copyNextSampleBuffer() { + guard let bb = CMSampleBufferGetDataBuffer(sb) else { continue } + var length = 0; var dataPtr: UnsafeMutablePointer? = nil + CMBlockBufferGetDataPointer(bb, atOffset: 0, lengthAtOffsetOut: nil, totalLengthOut: &length, dataPointerOut: &dataPtr) + guard let dp = dataPtr, length > 0 else { continue } + let count = length / MemoryLayout.size + dp.withMemoryRebound(to: Float.self, capacity: count) { samples.append(contentsOf: UnsafeBufferPointer(start: $0, count: count)) } + } + if reader.status == .failed { throw IOError.readFailed(reader.error?.localizedDescription ?? "unknown") } + return samples + } + + /// Write 48 kHz mono Float samples to a 16-bit PCM WAV file. + public static func writeWav(_ samples: [Float], to url: URL) throws { + let settings: [String: Any] = [ + AVFormatIDKey: kAudioFormatLinearPCM, AVSampleRateKey: sampleRate, + AVNumberOfChannelsKey: 1, AVLinearPCMBitDepthKey: 16, + AVLinearPCMIsFloatKey: false, AVLinearPCMIsBigEndianKey: false + ] + let fmt = AVAudioFormat(commonFormat: .pcmFormatFloat32, sampleRate: sampleRate, channels: 1, interleaved: false)! + let file: AVAudioFile + do { file = try AVAudioFile(forWriting: url, settings: settings, commonFormat: .pcmFormatFloat32, interleaved: false) } + catch { throw IOError.writeFailed(error.localizedDescription) } + let chunk = 48_000 + var i = 0 + while i < samples.count { + let n = min(chunk, samples.count - i) + guard let buf = AVAudioPCMBuffer(pcmFormat: fmt, frameCapacity: AVAudioFrameCount(n)) else { break } + buf.frameLength = AVAudioFrameCount(n) + samples.withUnsafeBufferPointer { _ = memcpy(buf.floatChannelData![0], $0.baseAddress! + i, n * MemoryLayout.size) } + do { try file.write(from: buf) } catch { throw IOError.writeFailed(error.localizedDescription) } + i += n + } + } +} diff --git a/app/Sources/CrispEngine/Biquad.swift b/app/Sources/CrispEngine/Biquad.swift index e1d9813..bb2911f 100644 --- a/app/Sources/CrispEngine/Biquad.swift +++ b/app/Sources/CrispEngine/Biquad.swift @@ -39,6 +39,14 @@ public struct Biquad { 1 + alpha, -2 * cw, 1 - alpha) } + public mutating func setLowpass(freq: Float, q: Float, sampleRate: Float) { + let w0 = 2 * Float.pi * freq / sampleRate + let cw = cos(w0), sw = sin(w0) + let alpha = sw / (2 * q) + set((1 - cw) / 2, 1 - cw, (1 - cw) / 2, + 1 + alpha, -2 * cw, 1 - alpha) + } + public mutating func setPeaking(freq: Float, q: Float, gainDb: Float, sampleRate: Float) { let A = pow(10, gainDb / 40) let w0 = 2 * Float.pi * freq / sampleRate diff --git a/app/Sources/CrispEngine/Metrics.swift b/app/Sources/CrispEngine/Metrics.swift new file mode 100644 index 0000000..1d88808 --- /dev/null +++ b/app/Sources/CrispEngine/Metrics.swift @@ -0,0 +1,82 @@ +import Foundation + +/// Reference-free and reference-based objective audio metrics (PRD 8.2). Dependency-free +/// (no PESQ/STOI libs); these are the metrics computable in-tree. DNSMOS/PESQ remain +/// optional external steps documented in the test report. +public enum Metrics { + public static func rmsDb(_ x: [Float]) -> Double { + guard !x.isEmpty else { return -.infinity } + var sum = 0.0 + for v in x { sum += Double(v) * Double(v) } + let rms = (sum / Double(x.count)).squareRoot() + return rms > 0 ? 20 * log10(rms) : -.infinity + } + + public static func peakDb(_ x: [Float]) -> Double { + var p: Float = 0 + for v in x { p = max(p, abs(v)) } + return p > 0 ? 20 * log10(Double(p)) : -.infinity + } + + /// Scale-invariant SDR (Le Roux et al.) between a clean reference and an estimate, in dB. + /// Higher is better. Scale-invariant but not EQ-invariant, so a coloring enhancer scores + /// lower than pure denoise — informative, not a defect. + public static func siSDR(reference: [Float], estimate: [Float]) -> Double { + let n = min(reference.count, estimate.count) + guard n > 0 else { return -.infinity } + var dotRE = 0.0, dotRR = 0.0 + for i in 0.. 0 else { return -.infinity } + let scale = dotRE / dotRR + var targetEnergy = 0.0, noiseEnergy = 0.0 + for i in 0.. 0, targetEnergy > 0 else { return .infinity } + return 10 * log10(targetEnergy / noiseEnergy) + } + + /// Best integer lag (samples) of `estimate` relative to `reference`, found by maximizing + /// cross-correlation over a central window. Needed because neural denoisers add a + /// processing delay, and SI-SDR is delay-sensitive. + public static func estimateLag(reference: [Float], estimate: [Float], maxLag: Int, window: Int) -> Int { + let n = min(reference.count, estimate.count) + guard n > 0 else { return 0 } + let w = min(window, n) + let start = (n - w) / 2 + var bestLag = 0, bestCorr = -Double.greatestFiniteMagnitude + for lag in -maxLag...maxLag { + var corr = 0.0 + var i = start + let end = start + w + while i < end { + let j = i + lag + if j >= 0 && j < estimate.count { corr += Double(reference[i]) * Double(estimate[j]) } + i += 1 + } + if corr > bestCorr { bestCorr = corr; bestLag = lag } + } + return bestLag + } + + /// Delay-compensated SI-SDR: aligns `estimate` to `reference` first (so a denoiser's + /// processing latency doesn't collapse the score), then computes SI-SDR on the overlap. + public static func alignedSiSDR(reference: [Float], estimate: [Float], + maxLagSamples: Int = 1500, window: Int = 24_000) -> Double { + let lag = estimateLag(reference: reference, estimate: estimate, maxLag: maxLagSamples, window: window) + let shifted: [Float] + if lag > 0 { shifted = Array(estimate[lag...]) } + else if lag < 0 { shifted = Array(repeating: 0, count: -lag) + estimate } + else { shifted = estimate } + return siSDR(reference: reference, estimate: shifted) + } + + /// True if any sample is NaN or infinite — a hard failure for an audio processor. + public static func hasNonFinite(_ x: [Float]) -> Bool { + for v in x where !v.isFinite { return true } + return false + } +} diff --git a/app/Sources/CrispEngine/TestCorpus.swift b/app/Sources/CrispEngine/TestCorpus.swift new file mode 100644 index 0000000..2644ab3 --- /dev/null +++ b/app/Sources/CrispEngine/TestCorpus.swift @@ -0,0 +1,168 @@ +import Foundation + +/// Builds a small, reproducible evaluation corpus from real in-repo speech + noise assets, +/// covering the degradation categories of PRD 8.1 (noise at several SNRs, reverb, clipping, +/// low bandwidth, level extremes, mains hum). No downloads — deterministic from the sources. +/// +/// The signal ops live here (not just in the CLI) so tests can build a corpus directly. +public enum TestCorpus { + public struct Entry: Sendable { + public let name: String // file name (without dir) + public let category: String + public let snrDb: String // "" if N/A + public let reference: String // clean reference file name for SI-SDR, or "" + } + + /// Generate the corpus into `outDir` from a clean speech file + a noise file. + /// Returns manifest entries. Writes each entry as a 48 kHz mono WAV plus `manifest.csv`. + @discardableResult + public static func generate(speechURL: URL, noiseURL: URL, outDir: URL) throws -> [Entry] { + try FileManager.default.createDirectory(at: outDir, withIntermediateDirectories: true) + let sr = Float(AudioIO.sampleRate) + var speech = try AudioIO.read48kMono(speechURL) + var noise = try AudioIO.read48kMono(noiseURL) + guard !speech.isEmpty, !noise.isEmpty else { throw AudioIO.IOError.readFailed("empty source") } + // Normalize the clean reference to a consistent -3 dBFS peak headroom. + normalizePeak(&speech, toDb: -3) + noise = tile(noise, toCount: speech.count) + + var entries: [Entry] = [] + func emit(_ samples: [Float], _ name: String, _ category: String, snr: String = "", ref: String = "clean.wav") throws { + try AudioIO.writeWav(samples, to: outDir.appendingPathComponent(name)) + entries.append(Entry(name: name, category: category, snrDb: snr, reference: ref)) + } + + try emit(speech, "clean.wav", "clean", ref: "clean.wav") + + for snr in [0, 5, 10, 20] { + try emit(mix(speech, noise, snrDb: Float(snr)), "noisy_snr\(snr).wav", "noisy", snr: "\(snr)") + } + + let rev = reverb(speech, sampleRate: sr, rt60: 0.6) + try emit(rev, "reverb.wav", "reverb") + try emit(mix(rev, noise, snrDb: 5), "reverb_noisy_snr5.wav", "reverb+noise", snr: "5") + + try emit(hardClip(speech, threshold: 0.4), "clipped.wav", "clipping") + + try emit(lowpass(speech, cutoff: 4000, sampleRate: sr), "lowband_8k.wav", "bandlimit") + try emit(lowpass(speech, cutoff: 8000, sampleRate: sr), "lowband_16k.wav", "bandlimit") + + try emit(gain(speech, db: -22), "quiet.wav", "level") + try emit(mix(addHum(speech, sampleRate: sr, freq: 60, level: 0.05), noise, snrDb: 15), + "hum_snr15.wav", "hum", snr: "15") + + // Manifest CSV. + var csv = "file,category,snr_db,reference\n" + for e in entries { csv += "\(e.name),\(e.category),\(e.snrDb),\(e.reference)\n" } + try csv.write(to: outDir.appendingPathComponent("manifest.csv"), atomically: true, encoding: .utf8) + return entries + } + + // MARK: - Signal ops + + static func tile(_ x: [Float], toCount n: Int) -> [Float] { + guard !x.isEmpty else { return [Float](repeating: 0, count: n) } + var out = [Float](repeating: 0, count: n) + for i in 0.. [Float] { + let sRms = rms(signal), nRms = rms(noise) + guard sRms > 0, nRms > 0 else { return signal } + let targetNoiseRms = sRms / pow(10, snrDb / 20) + let scale = targetNoiseRms / nRms + var out = [Float](repeating: 0, count: signal.count) + for i in 0.. [Float] { + x.map { max(-t, min(t, $0)) } + } + + public static func gain(_ x: [Float], db: Float) -> [Float] { + let g = pow(10, db / 20) + return x.map { $0 * g } + } + + /// 4th-order Butterworth-ish lowpass (two cascaded biquads) — band-limits for BWE tests. + public static func lowpass(_ x: [Float], cutoff: Float, sampleRate sr: Float) -> [Float] { + var b1 = Biquad(), b2 = Biquad() + // Reuse highpass designer's structure via a lowpass: build from RBJ lowpass coefficients. + b1.setLowpass(freq: cutoff, q: 0.541, sampleRate: sr) + b2.setLowpass(freq: cutoff, q: 1.307, sampleRate: sr) + var out = [Float](repeating: 0, count: x.count) + for i in 0.. [Float] { + var out = x + for i in 0.. [Float] { + let combMs: [Float] = [29.7, 37.1, 41.1, 43.7] + let allpassMs: [Float] = [5.0, 1.7] + var combs = combMs.map { ms -> (buf: [Float], idx: Int, g: Float) in + let d = max(1, Int(ms * 0.001 * sr)) + let g = pow(10, -3 * (ms * 0.001) / rt60) // feedback for target RT60 + return ([Float](repeating: 0, count: d), 0, g) + } + var allpasses = allpassMs.map { ms -> (buf: [Float], idx: Int, g: Float) in + let d = max(1, Int(ms * 0.001 * sr)) + return ([Float](repeating: 0, count: d), 0, 0.7) + } + var out = [Float](repeating: 0, count: x.count) + for i in 0.. Float { + guard !x.isEmpty else { return 0 } + var s: Float = 0 + for v in x { s += v * v } + return (s / Float(x.count)).squareRoot() + } + + static func normalizePeak(_ x: inout [Float], toDb: Float, onlyIfLouder: Bool = false) { + var peak: Float = 0 + for v in x { peak = max(peak, abs(v)) } + guard peak > 0 else { return } + let target = pow(10, toDb / 20) + if onlyIfLouder && peak <= target { return } + let g = target / peak + for i in 0.. [reference.wav] +// Prints CSV-friendly: lufs,peak_db,rms_db,si_sdr_db,nonfinite + +func die(_ msg: String, _ code: Int32 = 1) -> Never { + FileHandle.standardError.write((msg + "\n").data(using: .utf8)!) + exit(code) +} + +let a = CommandLine.arguments +guard a.count >= 2 else { die("usage: evaltool [reference.wav]", 2) } + +do { + let x = try AudioIO.read48kMono(URL(fileURLWithPath: a[1])) + let lufs = Loudness.integratedLUFS(x, sampleRate: AudioIO.sampleRate) + let peak = Metrics.peakDb(x) + let rms = Metrics.rmsDb(x) + var siSdr = Double.nan + if a.count >= 3 { + let ref = try AudioIO.read48kMono(URL(fileURLWithPath: a[2])) + // Delay-compensated: the denoiser adds latency, so align before scoring. + siSdr = Metrics.alignedSiSDR(reference: ref, estimate: x) + } + let nonFinite = Metrics.hasNonFinite(x) + // Human line on stderr, machine line on stdout. + FileHandle.standardError.write(String(format: "LUFS %.1f peak %.1f dBFS rms %.1f dBFS SI-SDR %@ finite=%@\n", + lufs, peak, rms, siSdr.isNaN ? "n/a" : String(format: "%.1f dB", siSdr), nonFinite ? "NO" : "yes").data(using: .utf8)!) + print(String(format: "%.2f,%.2f,%.2f,%@,%@", + lufs, peak, rms, siSdr.isNaN ? "" : String(format: "%.2f", siSdr), nonFinite ? "1" : "0")) +} catch { + die("eval failed: \(error.localizedDescription)") +} diff --git a/app/Sources/maketestset/main.swift b/app/Sources/maketestset/main.swift new file mode 100644 index 0000000..d943c96 --- /dev/null +++ b/app/Sources/maketestset/main.swift @@ -0,0 +1,30 @@ +import CrispEngine +import Foundation + +// Builds the evaluation corpus (PRD 8.1) from in-repo speech + noise assets. +// maketestset [outDir] [speechWav] [noiseWav] +// Defaults: out = test/corpus, speech = poc/model/in/speech.wav, noise = poc/model/in/noise.wav + +func die(_ msg: String, _ code: Int32 = 1) -> Never { + FileHandle.standardError.write((msg + "\n").data(using: .utf8)!) + exit(code) +} + +let a = CommandLine.arguments +let outDir = URL(fileURLWithPath: a.count >= 2 ? a[1] : "test/corpus") +let speech = URL(fileURLWithPath: a.count >= 3 ? a[2] : "poc/model/in/speech.wav") +let noise = URL(fileURLWithPath: a.count >= 4 ? a[3] : "poc/model/in/noise.wav") + +guard FileManager.default.fileExists(atPath: speech.path) else { die("speech not found: \(speech.path)\n(run scripts/fetch-deps.sh first, or pass a path)") } +guard FileManager.default.fileExists(atPath: noise.path) else { die("noise not found: \(noise.path)") } + +do { + let entries = try TestCorpus.generate(speechURL: speech, noiseURL: noise, outDir: outDir) + for e in entries { + FileHandle.standardError.write(" \(e.name) [\(e.category)\(e.snrDb.isEmpty ? "" : " SNR \(e.snrDb)dB")]\n".data(using: .utf8)!) + } + print("\(entries.count) files → \(outDir.path)") + print("manifest: \(outDir.appendingPathComponent("manifest.csv").path)") +} catch { + die("corpus generation failed: \(error.localizedDescription)") +} diff --git a/app/Tests/CrispEngineTests/BiquadTests.swift b/app/Tests/CrispEngineTests/BiquadTests.swift new file mode 100644 index 0000000..13dff1b --- /dev/null +++ b/app/Tests/CrispEngineTests/BiquadTests.swift @@ -0,0 +1,55 @@ +import XCTest +@testable import CrispEngine + +final class BiquadTests: XCTestCase { + let sr: Float = 48_000 + + private func rms(_ x: [Float]) -> Float { + var s: Float = 0; for v in x { s += v * v }; return (s / Float(x.count)).squareRoot() + } + private func tone(_ hz: Float, _ n: Int = 48_000) -> [Float] { + (0.. [Float] { + let n = Int(seconds * sr) + return (0.. [Float] { + ref.map { v in seed = seed &* 6364136223846793005 &+ 1; return v + amp * (Float(Int32(truncatingIfNeeded: seed >> 33)) / Float(Int32.max)) } + } + let little = Metrics.siSDR(reference: ref, estimate: noise(0.02)) + let lots = Metrics.siSDR(reference: ref, estimate: noise(0.2)) + XCTAssertGreaterThan(little, lots, "less added noise → higher SI-SDR") + } + + func testAlignedSiSDRRecoversFromDelay() { + let x = TestSupport.synth(seconds: 1) + let delayed = [Float](repeating: 0, count: 300) + x // x delayed by 300 samples + XCTAssertLessThan(Metrics.siSDR(reference: x, estimate: delayed), 10, "raw SI-SDR collapses under delay") + XCTAssertGreaterThan(Metrics.alignedSiSDR(reference: x, estimate: delayed, maxLagSamples: 1000, window: 20_000), + 30, "alignment should recover a pure delay") + } + + func testHasNonFiniteDetectsNaN() { + XCTAssertTrue(Metrics.hasNonFinite([0, 1, .nan, 2])) + XCTAssertTrue(Metrics.hasNonFinite([0, .infinity])) + XCTAssertFalse(Metrics.hasNonFinite([0, 0.5, -0.5])) + } +} diff --git a/app/Tests/CrispEngineTests/PipelineTests.swift b/app/Tests/CrispEngineTests/PipelineTests.swift new file mode 100644 index 0000000..6999631 --- /dev/null +++ b/app/Tests/CrispEngineTests/PipelineTests.swift @@ -0,0 +1,45 @@ +import XCTest +@testable import CrispEngine + +final class PipelineTests: XCTestCase { + /// Pipeline driven by a PassthroughSuppressor isolates the routing/enhancer behavior + /// (no model needed). The suppressor is then 1:1, so lengths are exact. + private func pipeline(mode: ProcessingMode) -> PipelineProcessor { + let p = PipelineProcessor(suppressor: PassthroughSuppressor()) + p.prepare(config: AudioProcessingConfig(mode: mode)) + p.snapEnhancerToTarget() + return p + } + + func testOffModeIsPassthrough() { + let x = TestSupport.synth(seconds: 1) + let out = pipeline(mode: .off).process(x) + XCTAssertEqual(out, x, "Off mode (passthrough suppressor + inactive enhancer) must be bit-identical") + } + + func testNoiseCancellationModeDoesNotColor() { + // PassthroughSuppressor + inactive enhancer → output equals input (enhancer wetMix 0). + let x = TestSupport.synth(seconds: 1) + let out = pipeline(mode: .noiseCancellation).process(x) + XCTAssertEqual(out, x, "Noise mode must not apply enhancer coloring") + } + + func testCleanAndEnhanceChangesSignalAndPreservesLength() { + let x = TestSupport.synth(seconds: 1) + let out = pipeline(mode: .cleanAndEnhance).process(x) + XCTAssertEqual(out.count, x.count, "1:1 length with passthrough suppressor") + var diff = 0.0; for i in 0.. String? { + let candidates = [ + "engine/models/DeepFilterNet3_onnx.tar.gz", + "poc/model/DeepFilterNet/models/DeepFilterNet3_onnx.tar.gz", + ] + for c in candidates { + let p = repoRoot.appendingPathComponent(c).path + if FileManager.default.fileExists(atPath: p) { return p } + } + return DeepFilterSuppressor.modelPath(.full) + } + + static func tempDir(_ name: String) -> URL { + let dir = FileManager.default.temporaryDirectory.appendingPathComponent("crisp_test_\(name)") + try? FileManager.default.createDirectory(at: dir, withIntermediateDirectories: true) + return dir + } + + /// A deterministic synthetic 48 kHz mono signal (tones + sibilance + noise + transient). + static func synth(seconds: Double) -> [Float] { + let sr = 48_000.0 + let n = Int(seconds * sr) + var out = [Float](repeating: 0, count: n) + var seed: UInt64 = 0xABCD_1234 + func rnd() -> Float { + seed = seed &* 6364136223846793005 &+ 1442695040888963407 + return Float(Int32(truncatingIfNeeded: seed >> 33)) / Float(Int32.max) + } + for i in 0.. VoiceEnhancer { + let e = VoiceEnhancer(sampleRate: 48_000) + e.prepare(config: AudioProcessingConfig(enhanceStrength: strength, tonePreset: tone)) + e.setActive(active) + e.snapToTarget() + return e + } + + func testInactiveIsBitIdenticalPassthrough() { + let e = make(active: false) + let x = TestSupport.synth(seconds: 2) + XCTAssertEqual(e.process(x), x, "inactive enhancer must be an exact passthrough (click-free Off/Noise modes)") + } + + func testChunkIndependence() { + let x = TestSupport.synth(seconds: 2) + let aligned = make(active: true).process(x) + + let chunked = make(active: true) + var out: [Float] = [] + let sizes = [137, 480, 53, 911, 256, 1000] + var i = 0, k = 0 + while i < x.count { + let n = min(sizes[k % sizes.count], x.count - i) + out.append(contentsOf: chunked.process(Array(x[i..<(i + n)]))) + i += n; k += 1 + } + XCTAssertEqual(aligned.count, out.count) + var maxDiff: Float = 0 + for j in 0.. **한계**: true-peak(-1 dBTP)는 오버샘플링 없이 sample-peak 천장으로 근사. LUFS는 BS.1770 > 근사(K-weighting + 절대/상대 게이팅). LocalVQE/Resemble 실모델은 동일 seam 드롭인(PRD §9 M1 별도). +## 테스트셋 & 자동화 테스트 (PRD §8.1 / §8.2 / §9.4) + +**테스트셋** — 레포의 실제 음성·노이즈·잔향 자산에서 다운로드 없이 구성(`maketestset`), +12개 카테고리: clean / noisy(SNR 0·5·10·20) / reverb / reverb+noise / clipping / +bandlimit(8·16k) / 저음량 / hum. 재현: `test/corpus/README.md`. + +**XCTest** (`swift test --package-path app`): + +| Suite | 검증 | 결과 | +|---|---|---| +| BiquadTests (4) | 0dB=identity, LPF/HPF 거동, 엔벨로프 수렴 | ✅ | +| VoiceEnhancerTests (5) | 패스스루 bit-identical·chunk 독립·리미터·톤·램프 | ✅ | +| LoudnessTests (4) | 무음/선형성/정규화/peak 천장 | ✅ | +| MetricsTests (5) | RMS·peak·SI-SDR·지연보정 SI-SDR | ✅ | +| PipelineTests (4) | 모드 라우팅·길이보존·라이브 전환 finite | ✅ | +| IntegrationTests (4) | 코퍼스 생성·파일 end-to-end(모델) | ✅ | +| **합계** | | **26 tests, 0 failures** | + +**배치 평가** (`scripts/quality-eval.sh` → `test/corpus/quality-report.csv`) — 코퍼스 × +{noise, voice, clean} HQ 처리. 모든 출력 finite·peak −1 dBFS 천장 준수 → **PASS(회귀 게이트)**. +지연 보정 SI-SDR(noise mode): + +| 입력 | in SI-SDR | out SI-SDR | 비고 | +|---|---|---|---| +| noisy_snr0 | 0.00 | **+4.89** | 저-SNR denoise 이득 | +| noisy_snr5 | 5.00 | 5.51 | 모델 상한(≈6dB) 수렴 | +| noisy_snr10 | 10.0 | 6.00 | | +| noisy_snr20 | 20.0 | 6.33 | 이미 깨끗 → fidelity 상한 | +| clean | ∞ | 6.25 | DeepFilter clean 재구성 상한 | + +> SI-SDR 상한(≈6dB)은 신경망 denoiser의 pristine-clean 대비 재구성 한계(샘플 지표 특성)이며 결함 아님. +> 지각 품질은 위 DNSMOS(OVRL +0.55) 참조. 단일 화자 합성셋이며 PRD §8.3 사람 blind 청취 대체 불가. + ## 미검증 (사람/계정 필요 — 자동화 불가) - A/B blind 청취 ≥80% 개선 (PRD 8.3/8.4) — **사람 청취 필요**. 객관 proxy(DNSMOS)는 별도 측정 가능. diff --git a/docs/voice-enhancer.md b/docs/voice-enhancer.md index 4e94fb9..cd3a170 100644 --- a/docs/voice-enhancer.md +++ b/docs/voice-enhancer.md @@ -100,6 +100,36 @@ RTF clean+enhance pipeline: 0.1126 # ≈ 8.9× 실시간 (denoise+enh 출력은 48k mono Int16 WAV / AAC m4a, **길이 정확 보존**(10.595646s in=out → hop carry+flush 샘플 정확). 모든 모드 정상. +## 테스트셋 & 자동화 테스트 + +PRD §8.1 테스트셋과 §8.2 자동 평가, §9.4 "batch 처리 스크립트 + 결과 CSV" 납품물을 구현했다. + +**테스트셋** — 레포의 실제 음성·노이즈 자산으로 다운로드 없이 구성(`TestCorpus`): +clean / noisy(SNR 0·5·10·20) / reverb / reverb+noise / clipping / bandlimit(8·16k) / 저음량 / hum. +생성: `maketestset test/corpus` (`test/corpus/README.md` 참고). + +**XCTest 스위트** (`swift test`) — **26개 테스트, 0 실패**: +- `BiquadTests` — 0dB peaking=identity, LPF가 HF 감쇠, HPF가 DC 제거, 엔벨로프 수렴 +- `VoiceEnhancerTests` — 비활성 패스스루 bit-identical, chunk 독립성, 리미터 full-scale 이내, 톤 프리셋 finite, 램프 클릭 없음 +- `LoudnessTests` — 무음=-inf, 음량 선형성(+20LU), 정규화 목표 도달, peak 천장 준수 +- `MetricsTests` — RMS/peak, SI-SDR(동일=∞, 노이즈↑=점수↓), **지연 보정 SI-SDR**(순수 지연 복원) +- `PipelineTests` — Off/Noise 무채색, Clean+Enhance 변화+길이보존, 라이브 모드 전환 finite +- `IntegrationTests`(자산/모델 있을 때) — 코퍼스 생성 유효성, 파일 end-to-end(길이 보존·천장·finite) + +**배치 평가** (`scripts/quality-eval.sh`) — 코퍼스 × 모드 처리 → `quality-report.csv` +(LUFS/peak/SI-SDR). 비정상 출력(non-finite·클리핑) 시 exit 1 → **회귀 게이트**. +SI-SDR은 모델 지연을 cross-correlation으로 정렬 후 측정. 측정 결과: + +| 입력 | noise mode SI-SDR(in→out) | 해석 | +|---|---|---| +| noisy_snr0 | 0.00 → **+4.89** | 저-SNR에서 denoise 이득 큼 | +| noisy_snr5 | 5.00 → 5.51 | 모델 상한(≈6dB)에 수렴 | +| noisy_snr20 | 20.0 → 6.33 | 이미 깨끗 → 처리가 재구성 한계로 fidelity 소폭↓ | +| clean | (∞) → 6.25 | DeepFilter의 clean 재구성 상한 | + +> SI-SDR 상한(≈6dB)은 신경망 denoiser가 pristine clean 대비 갖는 재구성 한계(샘플 단위 지표 특성)이며 +> 결함이 아니다. 지각 품질은 DNSMOS(`poc/model/run_poc.sh`)로 별도 측정(OVRL +0.55). + ## 한계 / 후속 - **True-peak(-1 dBTP)** 는 4× 오버샘플링 없이 **sample-peak 천장**으로 근사(음성에서 보수적). diff --git a/scripts/quality-eval.sh b/scripts/quality-eval.sh new file mode 100755 index 0000000..1ea5e97 --- /dev/null +++ b/scripts/quality-eval.sh @@ -0,0 +1,67 @@ +#!/usr/bin/env bash +# +# Batch quality evaluation (PRD 8.2 / 9.4 deliverable). +# Generates the test corpus (if missing), runs the Voice Enhancer pipeline over every +# corpus file in several modes, computes objective metrics (LUFS / peak / SI-SDR vs the +# clean reference), and writes a CSV report. Exits non-zero if any output is invalid +# (non-finite, or clipped above 0 dBFS) — so it doubles as a regression gate. +# +# Usage: ./scripts/quality-eval.sh [modes...] (default: noise voice clean) +set -euo pipefail + +ROOT="$(cd "$(dirname "$0")/.." && pwd)" +cd "$ROOT" +CORPUS="$ROOT/test/corpus" +OUTDIR="$CORPUS/out" +REPORT="$CORPUS/quality-report.csv" +MODES=("${@:-noise voice clean}") +read -r -a MODES <<< "${MODES[*]}" +QUALITY="hq" + +echo "==> Building tools" +swift build --package-path "$ROOT/app" >/dev/null +BIN="$(swift build --package-path "$ROOT/app" --show-bin-path)" + +if [[ ! -f "$CORPUS/manifest.csv" ]]; then + echo "==> Generating corpus" + "$BIN/maketestset" "$CORPUS" >/dev/null +fi +mkdir -p "$OUTDIR" + +echo "==> Evaluating (modes: ${MODES[*]}, quality: $QUALITY)" +echo "file,category,snr_db,mode,in_lufs,out_lufs,in_peak_db,out_peak_db,in_sisdr_db,out_sisdr_db,sisdr_gain_db,finite" > "$REPORT" + +REF="$CORPUS/clean.wav" +fail=0 +# Skip the header line of the manifest. +tail -n +2 "$CORPUS/manifest.csv" | while IFS=, read -r file category snr reference; do + [[ -z "$file" ]] && continue + IN="$CORPUS/$file" + [[ -f "$IN" ]] || continue + # Baseline metrics of the unprocessed input vs the clean reference. + in_csv="$("$BIN/evaltool" "$IN" "$REF" 2>/dev/null)" + in_lufs="$(echo "$in_csv" | cut -d, -f1)"; in_peak="$(echo "$in_csv" | cut -d, -f2)"; in_sisdr="$(echo "$in_csv" | cut -d, -f4)" + for mode in "${MODES[@]}"; do + base="${file%.wav}" + OUT="$OUTDIR/${base}__${mode}.wav" + "$BIN/filetool" enhance "$IN" "$OUT" "$mode" "$QUALITY" natural podcast >/dev/null 2>&1 + out_csv="$("$BIN/evaltool" "$OUT" "$REF" 2>/dev/null)" + out_lufs="$(echo "$out_csv" | cut -d, -f1)"; out_peak="$(echo "$out_csv" | cut -d, -f2)" + out_sisdr="$(echo "$out_csv" | cut -d, -f4)"; finite="$(echo "$out_csv" | cut -d, -f5)" + gain="" + if [[ -n "$in_sisdr" && -n "$out_sisdr" ]]; then gain="$(awk "BEGIN{printf \"%.2f\", $out_sisdr-($in_sisdr)}")"; fi + echo "$file,$category,$snr,$mode,$in_lufs,$out_lufs,$in_peak,$out_peak,$in_sisdr,$out_sisdr,$gain,$finite" >> "$REPORT" + # Gate: finite and no clipping above 0 dBFS. + if [[ "$finite" != "0" ]]; then echo " FAIL non-finite: $OUT"; fail=1; fi + if [[ -n "$out_peak" ]] && awk "BEGIN{exit !($out_peak > 0.1)}"; then echo " FAIL clipping ($out_peak dBFS): $OUT"; fail=1; fi + done + echo "$fail" > /tmp/crisp_eval_fail # propagate out of the subshell (pipe) +done +fail="$(cat /tmp/crisp_eval_fail 2>/dev/null || echo 0)"; rm -f /tmp/crisp_eval_fail + +echo +echo "==> Report: $REPORT" +column -s, -t "$REPORT" | sed 's/^/ /' +echo +if [[ "$fail" != "0" ]]; then echo "RESULT: FAIL (invalid output detected)"; exit 1; fi +echo "RESULT: PASS — all outputs finite and within 0 dBFS" diff --git a/test/corpus/README.md b/test/corpus/README.md new file mode 100644 index 0000000..1320e11 --- /dev/null +++ b/test/corpus/README.md @@ -0,0 +1,40 @@ +# 평가 테스트셋 (PRD 8.1) + +이 디렉터리의 WAV/manifest/리포트는 **생성물**이라 git에 포함하지 않는다(재현 가능). +레포의 실제 음성·노이즈 자산에서 다운로드 없이 구성한다. + +## 재생성 + +```sh +# 소스 자산 준비(최초 1회): poc/model/in/speech.wav, noise.wav +./scripts/fetch-deps.sh + +# 테스트셋 생성 → test/corpus/*.wav + manifest.csv +BIN="$(swift build --package-path app --show-bin-path)" +"$BIN/maketestset" test/corpus +``` + +## 카테고리 (PRD 8.1 대응) + +| 파일 | 카테고리 | 구성 | +|---|---|---| +| `clean.wav` | clean | 원본 음성(−3 dBFS 정규화). SI-SDR 기준 reference | +| `noisy_snr{0,5,10,20}.wav` | noisy | 음성 + 실제 노이즈, 목표 SNR | +| `reverb.wav` | reverb | Schroeder 잔향(RT60≈0.6s) | +| `reverb_noisy_snr5.wav` | reverb+noise | 잔향 + 노이즈 | +| `clipped.wav` | clipping | 하드 클리핑(±0.4) | +| `lowband_8k.wav` / `lowband_16k.wav` | bandlimit | 4k/8k LPF(8/16 kHz 대역) — BWE 평가용 | +| `quiet.wav` | level | −22 dB 저음량 | +| `hum_snr15.wav` | hum | 60 Hz 험 + 노이즈 | + +## 자동 평가 (PRD 8.2 / 9.4) + +```sh +./scripts/quality-eval.sh # 기본 modes: noise voice clean +./scripts/quality-eval.sh noise clean # 특정 모드만 +# → test/corpus/quality-report.csv (LUFS/peak/SI-SDR), 비정상 출력 시 exit 1 +``` + +> SI-SDR는 모델 지연 보정(정렬) 후 측정한다. 저-SNR 입력에서 denoise 이득이 보이고, +> 고-SNR(이미 깨끗한) 입력에서는 모델/인핸서의 재구성 한계로 SI-SDR이 상한(≈6 dB)에 수렴한다. +> 단일 화자 합성 테스트셋이며, PRD 8.3의 사람 blind 청취를 대체하지 않는다. From bb73b472faf1ab91fa9efb2ed5df1f84e18e749e Mon Sep 17 00:00:00 2001 From: Hyeon-Mook Jelly Choi Date: Wed, 24 Jun 2026 21:05:19 +0900 Subject: [PATCH 3/6] =?UTF-8?q?fix(realtime):=20=EB=A0=88=EB=B2=A8?= =?UTF-8?q?=EB=AF=B8=ED=84=B0=20=ED=91=9C=EC=8B=9C=20+=20=EC=9E=85?= =?UTF-8?q?=EB=A0=A5=20=ED=8F=AC=EB=A7=B7=20=EB=B6=88=EC=9D=BC=EC=B9=98=20?= =?UTF-8?q?=ED=81=AC=EB=9E=98=EC=8B=9C=20=EC=88=98=EC=A0=95?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 레벨미터: - 출력 미터가 suppressor의 hop carry 프레임(processed.isEmpty)마다 입력 레벨을 그대로 미러링하고, 매 ~10ms 창의 RMS를 그대로 써서 깜빡이던 문제 수정. - peak-hold-with-decay 엔벨로프로 변경(빠른 상승·완만한 하강). carry 프레임에는 출력 엔벨로프만 감쇠하고 입력을 따라가지 않음. 크래시(SIGABRT): - 44.1kHz 내장 마이크에서 installTap(format:)에 명시 포맷을 넘기면 AVFAudio가 "format mismatch" NSException을 던져(Swift에서 catch 불가) 앱이 즉시 종료. isEnabled가 영속화돼 실행 즉시 엔진이 시작되며 재현됨. - installTap(format: nil)로 노드 자체 포맷 사용 + 리샘플러를 실제 버퍼 포맷에서 지연 생성(포맷 변경 시 재생성)하도록 수정. 임의 하드웨어 레이트에 견고. 문서/평가: - docs/enhance-behavior.md(신규): 인핸서가 실제로 들리게 하는 변화와 한계를 측정값(LUFS/peak/crest/저역·고역)과 함께 요약. BWE·디클리핑·dereverb 불가 명시. - Metrics.spectralTilt + evaltool에 crest/저역/고역 출력 추가(톤 특성 측정). evaltool CSV 컬럼 변경에 맞춰 quality-eval.sh 인덱스 갱신. Co-Authored-By: Claude Opus 4.8 (1M context) --- README.md | 2 +- .../Audio/AudioEngineController.swift | 44 ++++++++-- app/Sources/CrispEngine/Metrics.swift | 16 ++++ app/Sources/evaltool/main.swift | 11 ++- docs/enhance-behavior.md | 81 +++++++++++++++++++ docs/voice-enhancer.md | 3 +- scripts/quality-eval.sh | 4 +- 7 files changed, 146 insertions(+), 15 deletions(-) create mode 100644 docs/enhance-behavior.md diff --git a/README.md b/README.md index 2fc3b30..8e38cd6 100644 --- a/README.md +++ b/README.md @@ -99,7 +99,7 @@ engine/CDeepFilter/lib/ libdf.dylib (fetch-deps 생성, gitignore) engine/models/ DeepFilterNet3 모델 (fetch-deps 생성, gitignore) test/corpus/ 평가 테스트셋 (maketestset 생성, gitignore) + README scripts/ fetch-deps / install / verify / package / e2e / stability / quality-eval -docs/ architecture · voice-enhancer · STATUS · test-report · install · acceptance-checklist +docs/ architecture · voice-enhancer · enhance-behavior · STATUS · test-report · install poc/model/ DeepFilterNet 클론(gitignore) + run_poc.sh ``` diff --git a/app/Sources/CrispApp/Audio/AudioEngineController.swift b/app/Sources/CrispApp/Audio/AudioEngineController.swift index e93f590..adc8dd7 100644 --- a/app/Sources/CrispApp/Audio/AudioEngineController.swift +++ b/app/Sources/CrispApp/Audio/AudioEngineController.swift @@ -24,9 +24,13 @@ final class LiveAudioEngine: AudioEngineControlling { private var pipeline: PipelineProcessor? private var output: VirtualMicOutput? private var converter: AVAudioConverter? + private var converterInFormat: AVAudioFormat? // format the current converter was built for private let canonical = AVAudioFormat(commonFormat: .pcmFormatFloat32, sampleRate: 48_000, channels: 1, interleaved: false)! private var running = false private var lastMeterNs: UInt64 = 0 // throttle gate for UI meter updates + private var inMeter: Float = 0 // smoothed input level (audio thread only) + private var outMeter: Float = 0 // smoothed output level (audio thread only) + private static let meterDecay: Float = 0.82 // per ~10 ms callback → ~0.3 s fall time init(state: AppState) { self.state = state @@ -72,8 +76,12 @@ final class LiveAudioEngine: AudioEngineControlling { pipe.prepare(config: config) pipeline = pipe - // Resampler: input (any rate, any channels) → 48 kHz mono for the model. - converter = AVAudioConverter(from: format, to: canonical) + // Resampler is built lazily in the tap from the *actual* delivered buffer format + // (see toCanonicalMono) — installing the tap with an explicit format throws an + // uncatchable ObjC "format mismatch" exception when the node negotiates a different + // rate (e.g. a 44.1 kHz built-in mic), so we let the tap use the node's own format. + converter = nil + converterInFormat = nil // PRD SET-02: log input / model / output sample rates (no PII). audioLog.info("engine start — input SR=\(format.sampleRate, privacy: .public)Hz ch=\(format.channelCount, privacy: .public) → model/virtual-mic SR=48000Hz mono") @@ -87,21 +95,34 @@ final class LiveAudioEngine: AudioEngineControlling { setStatus(.error("가상 마이크가 설치되지 않았습니다. pkg 설치 후 다시 시도하세요.")) } - input.installTap(onBus: 0, bufferSize: 480, format: format) { [weak self] buffer, _ in + inMeter = 0; outMeter = 0 + // format: nil → use the node's own format (avoids the format-mismatch crash). + input.installTap(onBus: 0, bufferSize: 480, format: nil) { [weak self] buffer, _ in guard let self, let canonicalSamples = self.toCanonicalMono(buffer), !canonicalSamples.isEmpty else { return } - let inLevel = Self.rms(canonicalSamples) // Two-stage pipeline: DeepFilter denoise (~0.97 ms/hop) → Voice Enhancer DSP. + // `process` returns only the complete model hops (carrying a sub-hop remainder), + // so `processed` may be shorter than the input or briefly empty. let processed = self.pipeline?.process(canonicalSamples) ?? canonicalSamples if !processed.isEmpty { processed.withUnsafeBufferPointer { self.output?.pushMono($0.baseAddress!, count: $0.count) } } - let outLevel = processed.isEmpty ? inLevel : Self.rms(processed) + // Peak-hold-with-decay envelopes: fast rise, smooth fall — stable, not jittery. + // The output envelope only decays on carry frames; it never mirrors the input + // (the old `processed.isEmpty ? inLevel` fallback made the output bar flicker and + // falsely track the input level). + self.inMeter = max(Self.level(canonicalSamples), self.inMeter * Self.meterDecay) + if processed.isEmpty { + self.outMeter *= Self.meterDecay + } else { + self.outMeter = max(Self.level(processed), self.outMeter * Self.meterDecay) + } // Throttle UI meter updates to ~15 Hz (the tap fires ~100 Hz). Flooding the main // thread with @Published updates was the UI-responsiveness bottleneck. let now = DispatchTime.now().uptimeNanoseconds if now &- self.lastMeterNs >= 66_000_000 { self.lastMeterNs = now - DispatchQueue.main.async { self.state?.meters.set(input: inLevel, output: outLevel) } + let i = self.inMeter, o = self.outMeter + DispatchQueue.main.async { self.state?.meters.set(input: i, output: o) } } } @@ -122,8 +143,10 @@ final class LiveAudioEngine: AudioEngineControlling { output?.stop() output = nil converter = nil + converterInFormat = nil pipeline = nil running = false + inMeter = 0; outMeter = 0 DispatchQueue.main.async { self.state?.meters.set(input: 0, output: 0) } setStatus(.stopped) } @@ -151,7 +174,13 @@ final class LiveAudioEngine: AudioEngineControlling { } /// Convert any-rate/any-channel tap buffer → 48 kHz mono Float samples. + /// The converter is built (and rebuilt) from the buffer's actual format, since the tap + /// uses the node's own format (which may differ from the rate we read at setup time). private func toCanonicalMono(_ buffer: AVAudioPCMBuffer) -> [Float]? { + if converter == nil || !(converterInFormat?.isEqual(buffer.format) ?? false) { + converter = AVAudioConverter(from: buffer.format, to: canonical) + converterInFormat = buffer.format + } guard let converter else { return nil } let ratio = canonical.sampleRate / buffer.format.sampleRate let cap = AVAudioFrameCount(Double(buffer.frameLength) * ratio) + 16 @@ -167,7 +196,8 @@ final class LiveAudioEngine: AudioEngineControlling { return Array(UnsafeBufferPointer(start: ptr, count: n)) } - private static func rms(_ samples: [Float]) -> Float { + /// Per-frame meter level: RMS mapped to 0…1 over a ~-60 dB floor. + private static func level(_ samples: [Float]) -> Float { guard !samples.isEmpty else { return 0 } var sum: Float = 0 for s in samples { sum += s * s } diff --git a/app/Sources/CrispEngine/Metrics.swift b/app/Sources/CrispEngine/Metrics.swift index 1d88808..43ebf16 100644 --- a/app/Sources/CrispEngine/Metrics.swift +++ b/app/Sources/CrispEngine/Metrics.swift @@ -74,6 +74,22 @@ public enum Metrics { return siSDR(reference: reference, estimate: shifted) } + /// Low-band vs high-band RMS (dB), for characterizing tone/EQ. Low = below ~500 Hz, + /// high = above ~4 kHz (each a 4th-order filter). `tilt = highDb - lowDb` rises for a + /// brighter signal, falls for a warmer one. + public static func spectralTilt(_ x: [Float], sampleRate sr: Double) -> (lowDb: Double, highDb: Double) { + let s = Float(sr) + var lp1 = Biquad(), lp2 = Biquad(), hp1 = Biquad(), hp2 = Biquad() + lp1.setLowpass(freq: 500, q: 0.541, sampleRate: s); lp2.setLowpass(freq: 500, q: 1.307, sampleRate: s) + hp1.setHighpass(freq: 4000, q: 0.541, sampleRate: s); hp2.setHighpass(freq: 4000, q: 1.307, sampleRate: s) + var low = [Float](repeating: 0, count: x.count), high = [Float](repeating: 0, count: x.count) + for i in 0.. Bool { for v in x where !v.isFinite { return true } diff --git a/app/Sources/evaltool/main.swift b/app/Sources/evaltool/main.swift index 536224f..2f5be5e 100644 --- a/app/Sources/evaltool/main.swift +++ b/app/Sources/evaltool/main.swift @@ -26,11 +26,14 @@ do { siSdr = Metrics.alignedSiSDR(reference: ref, estimate: x) } let nonFinite = Metrics.hasNonFinite(x) + let tilt = Metrics.spectralTilt(x, sampleRate: AudioIO.sampleRate) + let crest = peak - rms // Human line on stderr, machine line on stdout. - FileHandle.standardError.write(String(format: "LUFS %.1f peak %.1f dBFS rms %.1f dBFS SI-SDR %@ finite=%@\n", - lufs, peak, rms, siSdr.isNaN ? "n/a" : String(format: "%.1f dB", siSdr), nonFinite ? "NO" : "yes").data(using: .utf8)!) - print(String(format: "%.2f,%.2f,%.2f,%@,%@", - lufs, peak, rms, siSdr.isNaN ? "" : String(format: "%.2f", siSdr), nonFinite ? "1" : "0")) + FileHandle.standardError.write(String(format: "LUFS %.1f peak %.1f rms %.1f crest %.1f low %.1f high %.1f SI-SDR %@ finite=%@\n", + lufs, peak, rms, crest, tilt.lowDb, tilt.highDb, siSdr.isNaN ? "n/a" : String(format: "%.1f", siSdr), nonFinite ? "NO" : "yes").data(using: .utf8)!) + // CSV: lufs,peak,rms,crest,low_db,high_db,si_sdr,nonfinite + print(String(format: "%.2f,%.2f,%.2f,%.2f,%.2f,%.2f,%@,%@", + lufs, peak, rms, crest, tilt.lowDb, tilt.highDb, siSdr.isNaN ? "" : String(format: "%.2f", siSdr), nonFinite ? "1" : "0")) } catch { die("eval failed: \(error.localizedDescription)") } diff --git a/docs/enhance-behavior.md b/docs/enhance-behavior.md new file mode 100644 index 0000000..dd9a244 --- /dev/null +++ b/docs/enhance-behavior.md @@ -0,0 +1,81 @@ +# Voice Enhancer — 현재 동작 느낌 & 한계 요약 + +현재 구현된 Voice Enhancer(순수 DSP)가 **실제로 소리에 어떤 변화를 주는지**, 그리고 +**무엇을 못 하는지**를 측정값과 함께 정리한 문서. 설계/구현 상세는 +[`voice-enhancer.md`](voice-enhancer.md), 측정 방법은 [`test-report.md`](test-report.md). + +> 한 줄 요약: 지금의 인핸서는 **"믹싱 엔지니어의 기본 채널 스트립"** 에 가깝다 — +> 저역 정리(HPF) → 톤 보정(EQ) → 음량 고르기(컴프레서) → 치찰음 제어(디에서) → +> 음량 맞추고 클립 방지(LUFS 정규화 + 리미터). **없는 정보를 만들어내지는 않는다.** + +## 1. 단계별로 무엇이 들리는가 + +| 단계 | 처리 | 귀로 느끼는 변화 | +|---|---|---| +| High-pass 80Hz | 초저역/DC 제거 | 책상 울림·에어컨 럼블·"웅—" 저역이 사라져 **깔끔/타이트**해짐 | +| Tone EQ | 프리셋별 EQ | 톤이 **자연/따뜻/밝게** 중 하나로 살짝 기움(아래 §3) | +| Compressor | 1.5:1~3:1, makeup ≤+2dB | 큰 소리는 누르고 작은 소리는 올려 **음량이 고르게**, 또렷해짐 | +| De-esser | 5kHz 사이드체인 → 동적 high-shelf 컷 | "스/시/ㅊ" **치찰음이 덜 쏘게** | +| Limiter −1dBFS | 피크 천장 | 갑작스런 피크에도 **클리핑/찢어짐 없이** 안전 | +| (HQ만) LUFS 정규화 | −16/−18 LUFS | 파일 음량이 **방송/팟캐스트 수준으로 일정**하게 | + +## 2. 모드별 느낌 + +- **Off** — 원본 그대로(A/B 비교용). +- **Noise Cancellation** — 기존 DeepFilter 잡음 제거만. 톤/음량 **무채색**(인핸서 미적용). +- **Voice Enhancer** — 약한 잡음 제거 + 위 DSP. "마이크 톤만 다듬는" 느낌. 조용한 방에 적합. +- **Clean + Enhance** (기본 추천) — 풀 잡음 제거 후 DSP까지. 가장 **또렷하고 다듬어진** 결과. + +## 3. 강도/톤 — 실측 (clean 음성, voice/fast, 정규화 OFF로 인핸서 고유 효과만) + +| 케이스 | LUFS | peak | rms | crest(피크-rms) | 저역(<500Hz) | 고역(>4kHz) | +|---|---|---|---|---|---|---| +| 원본(baseline) | −23.0 | −3.0 | −23.3 | **20.3** | −25.4 | −33.7 | +| voice/natural | −26.0 | −7.7 | −27.3 | 19.6 | −29.5 | −38.2 | +| voice/**warm** | −26.1 | −8.1 | −27.1 | **19.0** | −29.1 | **−38.6** | +| voice/**bright** | −26.0 | −7.5 | −27.3 | 19.7 | −29.5 | **−37.6** | +| clean/**hq**/podcast | −19.4 | **−1.0** | −20.7 | 19.7 | −22.9 | −31.8 | + +읽는 법: +- **톤이 의도대로 작동** — warm은 고역이 가장 낮고(−38.6, 가장 어두움), bright는 가장 높다(−37.6, 가장 밝음). 차이는 **0.5~1dB로 미묘**(PRD 6.2 "기본값 보수적" 준수). +- **컴프레션은 약하게** — crest factor 20.3 → 19.0~19.7dB(약 0.6~1.3dB↓). 과하지 않음. +- **Fast 모드는 오히려 음량이 낮아진다**(−23 → −26 LUFS): 정규화가 없고 HPF+약denoise가 에너지를 덜어내기 때문. **HQ 모드는 −19 LUFS·피크 −1dBFS로 크고 일정**하게 맞춰진다. → 음량 일관성이 필요하면 **HQ + LUFS 정규화 권장**. + +## 4. 한계 — 못 하는 것 (degraded 입력, HQ clean 실측) + +현재 인핸서는 **DSP 후처리**라, "이미 있는 소리를 다듬을" 뿐 **손상된/없는 정보를 복원하지 못한다.** + +| 한계 | 실측 근거 | 설명 | +|---|---|---| +| **대역폭 복원(BWE) 없음** | lowband_8k 고역 −45.7 → −39.9 (정규화로 +5dB 올랐을 뿐, 풀밴드 −33.7엔 한참 못 미침) | 8/16kHz로 잘린 음성의 **잃어버린 고역을 만들어내지 못함**. 먹먹함이 크게 개선되지 않음 | +| **디클리핑 없음** | clipped crest 15.4 → 18.6 (음량만 올라감) | 클리핑으로 깨진 파형을 **복원하지 못함**. 왜곡(하모닉)은 남음 | +| **잔향 제거 거의 없음** | reverb 처리 후에도 잔향 꼬리 유지 | DSP에 dereverb 없음(DeepFilter가 소폭만). 울림이 크게 줄지 않음 | +| **생성형 복원 불가** | (설계상) | 화자/발화 보존이 원칙 → 없는 디테일을 **생성하지 않음**(보이스 클로닝·합성 제외) | +| **고정 프리셋** | — | 화자·언어·마이크별 **자동 적응 없음**. 3강도×3톤 수동 선택 | +| **true-peak 근사** | — | −1dBTP를 **sample-peak**로 근사(오버샘플링 없음). 인터샘플 피크 이론상 초과 가능 | +| **High 강도 주의** | — | 강하게는 아티팩트/부자연 가능 → **기본값 아님**(보통 권장) | +| **SI-SDR 상한 ≈6dB** | test-report 참조 | 신경망 denoiser의 pristine 대비 재구성 한계(샘플지표 특성, 결함 아님) | + +요점: **잡음↓·톤 보정·음량 고르기·치찰음 제어·클립 방지**는 잘 한다. +**대역 복원·디클리핑·강한 dereverb·음질 "업스케일"** 은 못 한다(원리상). + +## 5. 한계를 넘으려면 (후속) + +BWE/디클리핑/강한 dereverb는 **신경망 모델**의 영역이다. 파이프라인의 Voice Enhancer는 +교체 가능한 `AudioProcessor` stage이므로, PRD 후보(**Resemble Enhance / ClearerVoice / +LavaSR**)를 **같은 seam에 드롭인**하면 위 한계를 보완할 수 있다(가중치 다운로드·벤치마크· +라이선스 확인은 PRD §9 M1 별도). 현재 DSP 인핸서는 그 전까지의 **무의존·저지연·화자보존** +기본값으로 동작한다. + +## 재현 + +```sh +BIN="$(swift build --package-path app --show-bin-path)" +"$BIN/maketestset" test/corpus +# 인핸서 고유 효과(정규화 OFF): +"$BIN/filetool" enhance test/corpus/clean.wav /tmp/warm.wav voice fast warm none +"$BIN/evaltool" /tmp/warm.wav # LUFS/peak/rms/crest/low/high 출력 +# 한계(대역복원 안 됨): +"$BIN/filetool" enhance test/corpus/lowband_8k.wav /tmp/lb.wav clean hq natural podcast +"$BIN/evaltool" /tmp/lb.wav test/corpus/clean.wav +``` diff --git a/docs/voice-enhancer.md b/docs/voice-enhancer.md index cd3a170..6823fb8 100644 --- a/docs/voice-enhancer.md +++ b/docs/voice-enhancer.md @@ -2,7 +2,8 @@ `krisp_like_mac_voice_enhancer_prd_v02.docx` 의 Voice Enhancer 추가 요구사항을 기존 DeepFilterNet 노이즈 캔슬링 MVP 위에 구현한 내용. 본 문서는 PRD 각 절 ↔ 구현 매핑과 -모델 선택 판단, 검증 결과를 기록한다. +모델 선택 판단, 검증 결과를 기록한다. 실제 들리는 효과·한계 요약은 +[`enhance-behavior.md`](enhance-behavior.md). ## 핵심 설계 결정 — 모델 vs DSP diff --git a/scripts/quality-eval.sh b/scripts/quality-eval.sh index 1ea5e97..b486234 100755 --- a/scripts/quality-eval.sh +++ b/scripts/quality-eval.sh @@ -40,14 +40,14 @@ tail -n +2 "$CORPUS/manifest.csv" | while IFS=, read -r file category snr refere [[ -f "$IN" ]] || continue # Baseline metrics of the unprocessed input vs the clean reference. in_csv="$("$BIN/evaltool" "$IN" "$REF" 2>/dev/null)" - in_lufs="$(echo "$in_csv" | cut -d, -f1)"; in_peak="$(echo "$in_csv" | cut -d, -f2)"; in_sisdr="$(echo "$in_csv" | cut -d, -f4)" + in_lufs="$(echo "$in_csv" | cut -d, -f1)"; in_peak="$(echo "$in_csv" | cut -d, -f2)"; in_sisdr="$(echo "$in_csv" | cut -d, -f7)" for mode in "${MODES[@]}"; do base="${file%.wav}" OUT="$OUTDIR/${base}__${mode}.wav" "$BIN/filetool" enhance "$IN" "$OUT" "$mode" "$QUALITY" natural podcast >/dev/null 2>&1 out_csv="$("$BIN/evaltool" "$OUT" "$REF" 2>/dev/null)" out_lufs="$(echo "$out_csv" | cut -d, -f1)"; out_peak="$(echo "$out_csv" | cut -d, -f2)" - out_sisdr="$(echo "$out_csv" | cut -d, -f4)"; finite="$(echo "$out_csv" | cut -d, -f5)" + out_sisdr="$(echo "$out_csv" | cut -d, -f7)"; finite="$(echo "$out_csv" | cut -d, -f8)" gain="" if [[ -n "$in_sisdr" && -n "$out_sisdr" ]]; then gain="$(awk "BEGIN{printf \"%.2f\", $out_sisdr-($in_sisdr)}")"; fi echo "$file,$category,$snr,$mode,$in_lufs,$out_lufs,$in_peak,$out_peak,$in_sisdr,$out_sisdr,$gain,$finite" >> "$REPORT" From d79007a452ad9ea944898bfc90fbc86715cca0cb Mon Sep 17 00:00:00 2001 From: Hyeon-Mook Jelly Choi Date: Thu, 25 Jun 2026 02:13:57 +0900 Subject: [PATCH 4/6] =?UTF-8?q?docs(research):=20=ED=95=9C=EA=B3=84=20?= =?UTF-8?q?=EA=B7=B9=EB=B3=B5=20SOTA=20=EB=A6=AC=EC=84=9C=EC=B9=98=20?= =?UTF-8?q?=E2=80=94=20=EC=B0=A8=EC=84=B8=EB=8C=80=20=EC=9D=B8=ED=95=B8?= =?UTF-8?q?=EC=84=9C=20=EB=AA=A8=EB=8D=B8=20=EB=A1=9C=EB=93=9C=EB=A7=B5?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit deep-research(다중소스 웹검색 + 적대적 검증, 110 에이전트/27소스/131주장) 결과를 우리 코드베이스 기준으로 합성. enhance-limitations-research.md 신규. 한계별(BWE/디클립/dereverb/생성형 통합/적응형/true-peak/SI-SDR 상한) SOTA·모델· 라이선스·온디바이스 실현성·통합 권고. 핵심 결론: - 실시간 즉시 가능: true-peak 오버샘플 리미터(자체), AP-BWE(MIT, CPU 고속) PoC - 파일 HQ: Resemble Enhance / ClearerVoice-Studio 도입(가중치 라이선스 검증 선행) - 생성형 통합 복원(FINALLY/AnyEnhance/Miipher-2)은 대부분 파일 전용 + 라이선스 제약 - 실시간 생성형(Stream.FM, 32ms)은 연구 최전선 → watch - RE-USE(NSCLv1)·NISQA 가중치·AnyEnhance 비상용 → 출시 제외/내부용 검증 12건(3-0), 일부 항목은 세션 한도로 verify 미완(⚠️ 표기, 출시 전 재확인). Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/enhance-behavior.md | 9 +- docs/enhance-limitations-research.md | 132 +++++++++++++++++++++++++++ 2 files changed, 138 insertions(+), 3 deletions(-) create mode 100644 docs/enhance-limitations-research.md diff --git a/docs/enhance-behavior.md b/docs/enhance-behavior.md index dd9a244..e994b21 100644 --- a/docs/enhance-behavior.md +++ b/docs/enhance-behavior.md @@ -62,9 +62,12 @@ ## 5. 한계를 넘으려면 (후속) BWE/디클리핑/강한 dereverb는 **신경망 모델**의 영역이다. 파이프라인의 Voice Enhancer는 -교체 가능한 `AudioProcessor` stage이므로, PRD 후보(**Resemble Enhance / ClearerVoice / -LavaSR**)를 **같은 seam에 드롭인**하면 위 한계를 보완할 수 있다(가중치 다운로드·벤치마크· -라이선스 확인은 PRD §9 M1 별도). 현재 DSP 인핸서는 그 전까지의 **무의존·저지연·화자보존** +교체 가능한 `AudioProcessor` stage이므로, 후보 모델을 **같은 seam에 드롭인**하면 위 한계를 +보완할 수 있다. 어떤 모델을 실시간 vs 파일에 쓸지, 라이선스·온디바이스 실현성·통합 경로를 +조사한 결과는 → **[`enhance-limitations-research.md`](enhance-limitations-research.md)**. +요약: 지금 자체로 해결 가능한 건 **true-peak 리미터**뿐이고, 파일 HQ는 **Resemble Enhance/ +ClearerVoice**(라이선스 검증 후), 실시간 BWE는 **AP-BWE**(MIT, CPU 고속) PoC, 실시간 생성형 +복원은 **Stream.FM** 성숙까지 보류. 현재 DSP 인핸서는 그 전까지의 **무의존·저지연·화자보존** 기본값으로 동작한다. ## 재현 diff --git a/docs/enhance-limitations-research.md b/docs/enhance-limitations-research.md new file mode 100644 index 0000000..b4dddcd --- /dev/null +++ b/docs/enhance-limitations-research.md @@ -0,0 +1,132 @@ +# 한계 극복 리서치 — Voice Enhancer 차세대 모델 (2024–2026) + +[`enhance-behavior.md`](enhance-behavior.md)에 정리한 현재 DSP 인핸서의 한계(대역폭 복원· +디클리핑·dereverb·생성형 복원·적응형·true-peak·SI-SDR 상한)를 **넘기 위한** SOTA 조사. +다중 소스 웹 리서치 + 주장별 적대적 검증(3표 중 2표) 결과를 우리 코드베이스 기준으로 합성했다. + +> **신뢰도 표기** — ✅ 검증됨(3-0 적대적 통과) · ⚠️ 1차 소스 수집됨·검증 미완(세션 한도로 verify +> 중단) · 💡 도메인 지식 보강. **라이선스/수치는 출시 전 원문 재확인 필수**(PRD §7.2). 본 리포트는 +> 합성 단계가 세션 한도로 중단돼 **수동 합성**했다 — verify 보강 재실행 가능. + +--- + +## 한 줄 결론 + +| 경로 | 지금 도입 가능 | 차기(검증 후) | 레퍼런스/보류 | +|---|---|---|---| +| **실시간(causal)** | DeepFilter 유지 + **true-peak 리미터 자체 구현** | **AP-BWE**(경량 BWE, MIT, CPU 빠름)·WPE dereverb | Stream.FM(32ms 생성형, 연구최전선) | +| **파일 HQ** | — | **Resemble Enhance**·**ClearerVoice-Studio**(라이선스 검증 후) | FINALLY·Miipher-2(가중치/라이선스)·AnyEnhance·RE-USE(비상용) | + +핵심: **생성형 통합 복원은 거의 다 파일(offline) 전용**이고, **실시간 생성형은 이제 막 +연구 단계(Stream.FM, 2025.12)**다. 실시간에서 당장 넘을 수 있는 한계는 **BWE(AP-BWE)**와 +**true-peak**뿐이며, 디클리핑·강한 dereverb·studio급 복원은 **파일 모드의 생성형 모델**로 가야 한다. + +--- + +## 한계별 SOTA & 권고 + +### 1. 대역폭 복원 / 음성 초해상 (BWE/SR) +- **SOTA 방향**: 판별형(discriminative) 매핑은 regression-to-mean → 고역 over-smoothing. + 필드가 **생성형(diffusion/flow/bridge)** 으로 이동 중. ⚠️(출처 arxiv 2605.16681) +- **AP-BWE** ⚠️ (arxiv [2401.06387](https://arxiv.org/abs/2401.06387)) — 진폭·위상 **병렬 예측 + GAN**, 8/16/24k→48k. 리포트상 **CPU에서 18.1× 실시간(48k), 16k는 ~29×**, all-conv 구조라 + ONNX 변환 친화적, **MIT(코드+가중치)**. → **실시간 BWE 1순위 후보**. (속도/라이선스는 검증 미완 → 재확인) +- ClearerVoice-Studio에 16k→48k **super-resolution** 포함 ⚠️ (파일 경로). +- 후보(💡): mdctGAN, NU-Wave2, AudioSR(diffusion, 파일·느림). Resemble Enhance enhancer가 BWE 내장 ✅. +- **권고**: 실시간 = AP-BWE를 causal/스트리밍으로 만들 수 있는지 PoC(현재 코드의 enhancer stage 자리). + 파일 = Resemble/ClearerVoice가 이미 BWE 포함 → 별도 모델 불필요. + +### 2. 디클리핑 / 새츄레이션 복구 +- 단독 경량 상용 모델은 두드러진 것이 없음. **통합 생성형 모델의 부분기능**으로 해결되는 추세: + **AnyEnhance**(declip 포함) ✅, **FINALLY**(digital distortion 학습) ✅, Resemble Enhance(distortion 복원) ✅. +- 고전 DSP(A-SPADE 등)는 경미한 클리핑만. 💡 +- **권고**: 디클리핑은 **파일 HQ 생성형 모델에 위임**(전용 모델 도입 안 함). 실시간은 보류. + +### 3. 잔향 제거 (Dereverberation) +- **실시간**: **WPE / NARA-WPE**(통계적 적응형, MIT) — 다채널 강점, 단채널도 가능하나 분석버퍼 + 지연·이득 제한. 💡 DeepFilter가 약한 dereverb 이미 수행. +- **파일/통합**: FINALLY·AnyEnhance·Resemble가 dereverb 포함 ✅. +- **권고**: 실시간 강한 dereverb는 난제 → 보류(또는 NARA-WPE 실험). 파일은 통합 모델로 해결. + +### 4. 생성형 "스튜디오급" 통합 복원 (denoise+dereverb+declip+BWE 동시) +이번 리서치의 핵심 영역. **거의 전부 파일 전용**이며 접근법이 갈린다: + +| 모델 | 접근 | 통합 범위 | 스트리밍 | 속도 | 라이선스 | 판정 | +|---|---|---|---|---|---|---| +| **Resemble Enhance** ✅ | latent **flow matching**(생성형) | denoise+왜곡복원+BWE | ❌ 없음(PyTorch CLI) | RTF 미공개 | 코드 MIT 주장 **반박됨**⚠️ → 검증 | **파일 1순위 후보**(라이선스 확인) | +| **FINALLY** ✅ | **GAN**(HiFi++ +WavLM, 1-pass) | noise+reverb+BWE+mic | ❌(WavLM 미래컨텍스트) | RTF 0.03@V100 | 연구·가중치/라이선스 미확인 | **품질 최상**(MOS 4.54>Miipher), 파일·검증 | +| **AnyEnhance** ✅ | masked generative | denoise+dereverb+declip+SR+TSE | ❌ | RTF 0.254@GPU, 363.6M | **비상용 추정**(검증 충돌)⚠️ | **레퍼런스만**(상용 불가 가능성↑) | +| **Miipher-2** ✅ | **feature regression**(USM+WaveFit) | 범용 복원 | ❌ | RTF **0.0078**@가속기 | Google, 공개 상용가중치 없음 | **레퍼런스**(배포 불가) | +| **NVIDIA RE-USE** | 범용 SE+BWE | 범용 | — | — | **NSCLv1 비상용** | **출시 제외**(PRD 확정) | +| **Stream.FM** ✅ | frame-causal **flow matching** | SE 중심(통합은 반박 1-2) | ✅ **32ms/48ms**(24ms SE변형) | — | 연구최전선(2025.12) | **실시간 생성형의 미래** — watch | + +- **품질 vs 보존**: FINALLY는 "새 음성 생성이 아니라 정제"를 목표로 main-mode 회귀 → **화자/내용 + 보존**, WER 0.18로 입력(0.23)·Miipher(0.22)보다 우수 ✅. 우리 "화자 보존" 제약에 부합. +- **권고**: **파일 HQ = Resemble Enhance 도입 PoC**(BWE+왜곡복원 한번에, 라이선스 확정 선행). + 품질 상한이 필요하면 FINALLY를 벤치 레퍼런스로. AnyEnhance/Miipher-2/RE-USE는 **상용 배포 불가/ + 불확실** → 비교용. 실시간 생성형은 Stream.FM 성숙까지 보류. + +### 5. 적응형 / 콘텐츠 인지 (자동 EQ·게인, 화자/언어/마이크 적응) +- **무참조 품질 예측기**를 루프에 사용: **DNSMOS**(MS), **NISQA**, **UTMOS**. + ⚠️ **NISQA 가중치는 별도 제한 라이선스**(코드≠가중치) → **내부 평가용만, 출시 비포함**. +- 자동 게인은 이미 보유(LUFS 정규화). 자동 EQ는 스펙트럼 분석 기반 DSP로 자체 구현 가능 💡. +- **권고**: 품질 예측기는 **오프라인 튜닝/회귀 게이트**(이미 있는 `quality-eval.sh`에 DNSMOS 연동) + 로만. 실시간 적응 EQ는 경량 스펙트럼 타겟 매칭으로 점진 도입. + +### 6. True-peak 리미팅 (−1 dBTP) +- **ITU-R BS.1770-4/5**: 4× 오버샘플링 후 피크 검출 = inter-sample peak 포함. ✅(ITU 원문 확보) +- 현재 우리는 **sample-peak** 근사 → **자체 구현으로 즉시 해소 가능**(오버샘플 리미터). +- **권고**: **지금 바로 자체 구현**(외부 의존 0). 한계 7개 중 **유일하게 모델 없이 해결**되는 항목. + +### 7. 마스크 기반 denoiser의 fidelity 상한 (SI-SDR caps) +- 판별형은 regression-to-mean으로 상한 → **생성형(GAN/flow/diffusion)** 이 돌파. FINALLY가 GAN + 1-pass로 회귀형 Miipher 대비 MOS/WER 우위 입증 ✅. +- **평가지표**: SI-SDR 단독 금지 → **UTMOS/DNSMOS/PESQ/NISQA + WER** 병행(이미 enhance-behavior에 + 명시). 💡 +- **권고**: 파일 경로를 생성형으로 옮기면 자연히 해소. 평가를 perceptual+WER로 확장. + +--- + +## 런타임 / 변환 feasibility (Apple Silicon) + +| 경로 | 적합 모델 | 비고 | +|---|---|---| +| **tract(순수 Rust)** | DeepFilter류 conv/GRU | 생성형/대형 op 미지원 → 신규 생성형엔 부적합 💡 | +| **ONNX Runtime + CoreML EP** | all-conv(AP-BWE) ✅ 친화 | STFT/dynamic shape op가 변환 난관(HT-Demucs 사례) ⚠️ | +| **Core ML** | 변환되면 ANE 가속 | stateful streaming 변환 검증 필요 | +| **MLX / mlx-audio** | Apple Silicon 네이티브 오디오 | [github.com/Blaizzy/mlx-audio](https://github.com/Blaizzy/mlx-audio) — 온디바이스 후보 ⚠️ | +| **PyTorch(번들)** | Resemble/FINALLY PoC | 앱 번들 크기·cold start 큼 → 파일 beta까지만(PRD 4.4) | + +- **변환 난이도**: AP-BWE(all-conv) < ClearerVoice(FRCRN) < Resemble Enhance(CFM+vocoder, 다단·반복) < diffusion. +- 실시간 후보는 **ONNX/CoreML 변환성 + causal 가능성**을 PoC 1순위 기준으로. + +## 라이선스 함정 (출시 전 필수 확인) + +- **코드 ≠ 가중치**: Resemble Enhance(MIT 주장 **반박**), NISQA(가중치 제한), RE-USE(NSCLv1 비상용), + AnyEnhance(비상용 가능성). HuggingFace 모델카드 license 별도 확인. +- **비상용 확정/유력 → 출시 제외**: NVIDIA RE-USE(NSCLv1) ✅확정, AnyEnhance ⚠️, NISQA 가중치 ⚠️. +- **상용 가능 유력**: AP-BWE(MIT 주장) ⚠️검증, ClearerVoice(Apache-2.0 주장) ⚠️검증, DeepFilter(현행 사용중) ✅. + +## 우선순위 로드맵 + +1. **지금(자체, 의존 0)** — **true-peak 오버샘플 리미터** 구현(한계 #6 해소), 평가에 DNSMOS 연동. +2. **다음(검증 후 PoC)** — 파일 HQ에 **Resemble Enhance** 또는 **ClearerVoice-Studio** 드롭인 + (denoise+왜곡복원+BWE 일괄). **라이선스(가중치) 확정 → 벤치(품질/속도/WER) → `AudioProcessor` 어댑터**. +3. **다음(실시간)** — **AP-BWE** causal PoC(ONNX/CoreML), DeepFilter 뒤 BWE stage로. WER/지연 검증. +4. **watch(보류)** — Stream.FM(실시간 생성형), FINALLY(품질 상한 레퍼런스), Miipher-2/AnyEnhance/RE-USE(라이선스). + +## 출처 (1차) + +- Miipher-2: arxiv [2505.04457](https://arxiv.org/abs/2505.04457) · [google.github.io/df-conformer/miipher2](https://google.github.io/df-conformer/miipher2/) +- Resemble Enhance: [github.com/resemble-ai/resemble-enhance](https://github.com/resemble-ai/resemble-enhance) +- Stream.FM: arxiv [2512.19442](https://arxiv.org/abs/2512.19442) +- FINALLY: NeurIPS 2024 [proceedings](https://proceedings.neurips.cc/paper_files/paper/2024/file/01b3dea1871f7cea1e0e6be1f2f085bc-Paper-Conference.pdf) +- AnyEnhance: arxiv [2501.15417](https://arxiv.org/abs/2501.15417) +- AP-BWE: arxiv [2401.06387](https://arxiv.org/abs/2401.06387) +- ClearerVoice-Studio: [github.com/modelscope/ClearerVoice-Studio](https://github.com/modelscope/ClearerVoice-Studio) +- NVIDIA RE-USE: [huggingface.co/nvidia/RE-USE](https://huggingface.co/nvidia/RE-USE) (NSCLv1) +- ITU-R BS.1770-5 (true-peak): [itu.int](https://www.itu.int/dms_pubrec/itu-r/rec/bs/R-REC-BS.1770-5-202311-I!!PDF-E.pdf) +- 런타임: [mlx-audio](https://github.com/Blaizzy/mlx-audio) · [ONNX Runtime CoreML EP](https://onnxruntime.ai/docs/execution-providers/CoreML-ExecutionProvider.html) · DeepFilterNet ONNX export + +> 검증 미완(⚠️) 항목은 세션 한도(KST 01:10 리셋) 이후 verify 재실행으로 보강 가능. 특히 +> AP-BWE 속도/라이선스, ClearerVoice 라이선스, AnyEnhance 라이선스는 도입 결정 전 재확인 권장. From 414577530fb27f9797f288795afa554ed6b9d78c Mon Sep 17 00:00:00 2001 From: Hyeon-Mook Jelly Choi Date: Thu, 25 Jun 2026 13:22:46 +0900 Subject: [PATCH 5/6] =?UTF-8?q?feat(enhance):=20true-peak=20=EB=A6=AC?= =?UTF-8?q?=EB=AF=B8=ED=84=B0=20+=20=EC=99=B8=EB=B6=80=20HQ=20=EB=AA=A8?= =?UTF-8?q?=EB=8D=B8=20=ED=86=B5=ED=95=A9=20seam=20+=20=ED=95=9C=EA=B3=84?= =?UTF-8?q?=20=EB=A6=AC=EC=84=9C=EC=B9=98=20=EA=B2=80=EC=A6=9D?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit (a) true-peak (한계 #6 해소): - TruePeak.swift — ITU-R BS.1770 4× 오버샘플 inter-sample peak(윈도우드 sinc fractional-delay, DC 정규화, 엣지 truncation 가드). Loudness.normalize가 sample-peak 대신 true-peak로 −1 dBTP 천장 적용. FileEnhanceReport에 truePeak 추가. evaltool에 true-peak 컬럼. 파일 HQ 출력 −1.0 dBTP 검증. (b) 리서치 ⚠️ 항목 1차 소스 직접 검증: - AP-BWE: 코드+가중치 모두 MIT, CPU 18.1× 실시간, 8/12/16/24k→48k (확정) - ClearerVoice-Studio: 코드 Apache-2.0, 16k→48k SR (확정) - AnyEnhance: 재현코드 MIT지만 공개 가중치 무라이선스 → 배포불가(레퍼런스) enhance-limitations-research.md 갱신. (c) 파일 HQ 외부 모델 통합 PoC: - ExternalEnhancer.swift — {in}/{out} subprocess worker(PyTorch 모델용, PRD §4.4). FileEnhancer.enhanceExternal: decode→외부모델→HQ post-DSP(LUFS+true-peak)→write, temp 삭제. filetool external 서브커맨드. cp 패스스루로 plumbing 테스트. - scripts/hq-model-poc.sh — 코퍼스 × {dsp,외부모델} 벤치 → CSV. 미설치 시 graceful skip. --print-setup으로 Resemble/ClearerVoice 설치·래퍼 안내. docs/hq-model-integration.md. 테스트 32개 0 실패(+TruePeak 4, +External 2). 앱 번들 빌드 정상. Co-Authored-By: Claude Opus 4.8 (1M context) --- .gitignore | 4 +- README.md | 2 +- .../CrispEngine/ExternalEnhancer.swift | 54 ++++++++++++++ app/Sources/CrispEngine/FileEnhancer.swift | 51 +++++++++++++ app/Sources/CrispEngine/Loudness.swift | 15 ++-- app/Sources/CrispEngine/TruePeak.swift | 53 ++++++++++++++ app/Sources/evaltool/main.swift | 11 +-- app/Sources/filetool/main.swift | 18 ++++- .../ExternalEnhancerTests.swift | 35 +++++++++ .../CrispEngineTests/TruePeakTests.swift | 39 ++++++++++ docs/enhance-behavior.md | 2 +- docs/enhance-limitations-research.md | 35 ++++++--- docs/hq-model-integration.md | 62 ++++++++++++++++ scripts/hq-model-poc.sh | 73 +++++++++++++++++++ 14 files changed, 424 insertions(+), 30 deletions(-) create mode 100644 app/Sources/CrispEngine/ExternalEnhancer.swift create mode 100644 app/Sources/CrispEngine/TruePeak.swift create mode 100644 app/Tests/CrispEngineTests/ExternalEnhancerTests.swift create mode 100644 app/Tests/CrispEngineTests/TruePeakTests.swift create mode 100644 docs/hq-model-integration.md create mode 100755 scripts/hq-model-poc.sh diff --git a/.gitignore b/.gitignore index c51a6e3..c54016a 100644 --- a/.gitignore +++ b/.gitignore @@ -28,10 +28,12 @@ models/*.tar.gz *.f32 !test/fixtures/*.wav -# Generated evaluation corpus + report (reproduce: maketestset / scripts/quality-eval.sh) +# Generated evaluation corpus + reports (reproduce: maketestset / quality-eval.sh / hq-model-poc.sh) /test/corpus/manifest.csv /test/corpus/out/ /test/corpus/quality-report.csv +/test/corpus/hq-poc/ +/test/corpus/hq-poc-report.csv # Internal PRD spec — not published *.docx diff --git a/README.md b/README.md index 8e38cd6..4c3e60f 100644 --- a/README.md +++ b/README.md @@ -99,7 +99,7 @@ engine/CDeepFilter/lib/ libdf.dylib (fetch-deps 생성, gitignore) engine/models/ DeepFilterNet3 모델 (fetch-deps 생성, gitignore) test/corpus/ 평가 테스트셋 (maketestset 생성, gitignore) + README scripts/ fetch-deps / install / verify / package / e2e / stability / quality-eval -docs/ architecture · voice-enhancer · enhance-behavior · STATUS · test-report · install +docs/ architecture · voice-enhancer · enhance-behavior · enhance-limitations-research · hq-model-integration · STATUS · test-report poc/model/ DeepFilterNet 클론(gitignore) + run_poc.sh ``` diff --git a/app/Sources/CrispEngine/ExternalEnhancer.swift b/app/Sources/CrispEngine/ExternalEnhancer.swift new file mode 100644 index 0000000..7267517 --- /dev/null +++ b/app/Sources/CrispEngine/ExternalEnhancer.swift @@ -0,0 +1,54 @@ +import Foundation + +/// A file-mode "Voice Enhancer" stage backed by an EXTERNAL process (e.g. a PyTorch model +/// CLI like Resemble Enhance or ClearerVoice). This is the integration seam for heavy HQ +/// models that can't be bundled into the Swift/tract runtime (PRD 4.4 "PyTorch helper … +/// 파일 HQ beta까지만 허용"): the model runs out-of-process and exchanges WAV files, then our +/// pipeline applies the post-DSP (loudness + true-peak) and writes the final output. +/// +/// The model itself is NOT shipped here — `command` points at whatever the operator has +/// installed (gated on a commercial license, PRD §7.2). With no model installed the file +/// pipeline keeps using the built-in DSP enhancer. +public struct ExternalEnhancer: Sendable { + public enum ExternalError: LocalizedError { + case empty, launchFailed(String), nonZeroExit(Int32, String), noOutput + public var errorDescription: String? { + switch self { + case .empty: return "외부 인핸서 명령이 비어 있습니다." + case .launchFailed(let m): return "외부 인핸서 실행 실패: \(m)" + case .nonZeroExit(let c, let m): return "외부 인핸서 오류 (코드 \(c)): \(m)" + case .noOutput: return "외부 인핸서가 출력 파일을 생성하지 않았습니다." + } + } + } + + /// Argv with `{in}` / `{out}` placeholders, e.g. ["resemble-enhance-file", "{in}", "{out}"]. + /// Resolved via `/usr/bin/env` so the binary is found on PATH. + public let command: [String] + public let name: String + + public init(name: String, command: [String]) { + self.name = name + self.command = command + } + + /// Run the external model: `input` (48 kHz mono WAV) → `output` WAV. Throws on failure. + public func run(input: URL, output: URL) throws { + guard !command.isEmpty else { throw ExternalError.empty } + let args = command.map { + $0.replacingOccurrences(of: "{in}", with: input.path) + .replacingOccurrences(of: "{out}", with: output.path) + } + let proc = Process() + proc.executableURL = URL(fileURLWithPath: "/usr/bin/env") + proc.arguments = args + let errPipe = Pipe() + proc.standardError = errPipe + proc.standardOutput = Pipe() + do { try proc.run() } catch { throw ExternalError.launchFailed(error.localizedDescription) } + proc.waitUntilExit() + let errText = String(data: errPipe.fileHandleForReading.readDataToEndOfFile(), encoding: .utf8) ?? "" + if proc.terminationStatus != 0 { throw ExternalError.nonZeroExit(proc.terminationStatus, errText) } + guard FileManager.default.fileExists(atPath: output.path) else { throw ExternalError.noOutput } + } +} diff --git a/app/Sources/CrispEngine/FileEnhancer.swift b/app/Sources/CrispEngine/FileEnhancer.swift index 5d54a33..12c283c 100644 --- a/app/Sources/CrispEngine/FileEnhancer.swift +++ b/app/Sources/CrispEngine/FileEnhancer.swift @@ -54,6 +54,7 @@ public struct FileEnhanceReport: Sendable { public let output: URL public let inputPeakDb: Double public let outputPeakDb: Double + public let outputTruePeakDb: Double // ITU-R BS.1770 inter-sample peak (−1 dBTP target) public let outputLUFS: Double public let loudnessGainDb: Double } @@ -186,6 +187,56 @@ public final class FileEnhancer { return FileEnhanceReport(output: output, inputPeakDb: inputPeakDb, outputPeakDb: 20 * log10(Double(peak(processed)) + 1e-9), + outputTruePeakDb: TruePeak.truePeakDb(processed), + outputLUFS: outLUFS, + loudnessGainDb: gainDb) + } + + /// File HQ via an EXTERNAL model worker (PRD 4.4): decode → run `model` out-of-process + /// → apply our HQ post-DSP (loudness + true-peak ceiling) → write. Temp WAVs are deleted + /// on completion/failure (PRD privacy §7.1). The built-in stages are bypassed; the external + /// model is the enhancer. Use when a licensed HQ model (Resemble/ClearerVoice/…) is installed. + @discardableResult + public func enhanceExternal(input: URL, + output: URL, + model: ExternalEnhancer, + options: FileEnhanceOptions, + progress: ((Double) -> Void)? = nil) throws -> FileEnhanceReport { + cancelled = false + let work = FileManager.default.temporaryDirectory.appendingPathComponent("crisp_ext_\(UUID().uuidString)") + try FileManager.default.createDirectory(at: work, withIntermediateDirectories: true) + defer { try? FileManager.default.removeItem(at: work) } // privacy: clean temp files + let inWav = work.appendingPathComponent("in.wav") + let outWav = work.appendingPathComponent("out.wav") + + var samples = try decodeTo48kMono(input) + if let secs = options.previewSeconds { + let limit = Int(secs * 48_000) + if samples.count > limit { samples = Array(samples[0.. 0 { - let maxGainLin = ceilingLin / peak - let maxGainDb = 20 * log10(Double(maxGainLin)) + let tp = TruePeak.truePeak(samples) + if tp > 0 { + let maxGainDb = 20 * log10(Double(ceilingLin / tp)) gainDb = min(gainDb, maxGainDb) } let gainLin = Float(pow(10, gainDb / 20)) diff --git a/app/Sources/CrispEngine/TruePeak.swift b/app/Sources/CrispEngine/TruePeak.swift new file mode 100644 index 0000000..8349ed0 --- /dev/null +++ b/app/Sources/CrispEngine/TruePeak.swift @@ -0,0 +1,53 @@ +import Foundation + +/// True-peak estimation per ITU-R BS.1770-4 (Annex 2): oversample ≥4× and take the peak of +/// the interpolated signal, capturing inter-sample peaks that a raw sample-peak meter misses. +/// +/// Implemented as 4× interpolation — the original samples plus three fractional points +/// (¼, ½, ¾) per sample via windowed-sinc fractional-delay FIRs (DC-normalized, so a constant +/// signal never produces a phantom peak). Used to drive the −1 dBTP ceiling in the file +/// pipeline so the output respects true-peak, not just sample-peak. +public enum TruePeak { + static let taps = 16 + static let half = taps / 2 + /// Fractional-delay kernels for the ¼, ½, ¾ inter-sample positions. + static let kernels: [[Float]] = [0.25, 0.5, 0.75].map { kernel(frac: Float($0)) } + + private static func kernel(frac: Float) -> [Float] { + var k = [Float](repeating: 0, count: taps) + var sum: Float = 0 + for j in 0.. Float { + guard !x.isEmpty else { return 0 } + var pk: Float = 0 + for i in 0..= 0 && base + taps <= x.count else { continue } + for k in kernels { + var acc: Float = 0 + for j in 0.. Double { + let pk = truePeak(x) + return pk > 0 ? 20 * log10(Double(pk)) : -.infinity + } +} diff --git a/app/Sources/evaltool/main.swift b/app/Sources/evaltool/main.swift index 2f5be5e..0e511dd 100644 --- a/app/Sources/evaltool/main.swift +++ b/app/Sources/evaltool/main.swift @@ -28,12 +28,13 @@ do { let nonFinite = Metrics.hasNonFinite(x) let tilt = Metrics.spectralTilt(x, sampleRate: AudioIO.sampleRate) let crest = peak - rms + let truePeak = TruePeak.truePeakDb(x) // Human line on stderr, machine line on stdout. - FileHandle.standardError.write(String(format: "LUFS %.1f peak %.1f rms %.1f crest %.1f low %.1f high %.1f SI-SDR %@ finite=%@\n", - lufs, peak, rms, crest, tilt.lowDb, tilt.highDb, siSdr.isNaN ? "n/a" : String(format: "%.1f", siSdr), nonFinite ? "NO" : "yes").data(using: .utf8)!) - // CSV: lufs,peak,rms,crest,low_db,high_db,si_sdr,nonfinite - print(String(format: "%.2f,%.2f,%.2f,%.2f,%.2f,%.2f,%@,%@", - lufs, peak, rms, crest, tilt.lowDb, tilt.highDb, siSdr.isNaN ? "" : String(format: "%.2f", siSdr), nonFinite ? "1" : "0")) + FileHandle.standardError.write(String(format: "LUFS %.1f peak %.1f truePeak %.1f dBTP rms %.1f crest %.1f low %.1f high %.1f SI-SDR %@ finite=%@\n", + lufs, peak, truePeak, rms, crest, tilt.lowDb, tilt.highDb, siSdr.isNaN ? "n/a" : String(format: "%.1f", siSdr), nonFinite ? "NO" : "yes").data(using: .utf8)!) + // CSV: lufs,peak,rms,crest,low_db,high_db,si_sdr,nonfinite,true_peak_db (true_peak appended last to keep prior indices stable) + print(String(format: "%.2f,%.2f,%.2f,%.2f,%.2f,%.2f,%@,%@,%.2f", + lufs, peak, rms, crest, tilt.lowDb, tilt.highDb, siSdr.isNaN ? "" : String(format: "%.2f", siSdr), nonFinite ? "1" : "0", truePeak)) } catch { die("eval failed: \(error.localizedDescription)") } diff --git a/app/Sources/filetool/main.swift b/app/Sources/filetool/main.swift index 49a33cb..2a392b7 100644 --- a/app/Sources/filetool/main.swift +++ b/app/Sources/filetool/main.swift @@ -24,7 +24,21 @@ func parseMode(_ s: String) -> ProcessingMode? { let a = CommandLine.arguments do { - if a.count >= 6 && a[1] == "enhance", let mode = parseMode(a[4]), let q = FileQuality(rawValue: a[5]) { + if a.count >= 7 && a[1] == "external", let q = FileQuality(rawValue: a[4]) { + // filetool external [args...] (cmd uses {in}/{out}) + let loud = LoudnessTarget(rawValue: a[5]) ?? .podcast + let ext = URL(fileURLWithPath: a[3]).pathExtension.lowercased() + let fmt = OutputFormat(rawValue: ext) ?? .wav + let model = ExternalEnhancer(name: "external", command: Array(a[6...])) + let opts = FileEnhanceOptions(quality: q, loudness: loud, format: fmt) + let r = try FileEnhancer().enhanceExternal(input: URL(fileURLWithPath: a[2]), + output: URL(fileURLWithPath: a[3]), model: model, options: opts) { p in + FileHandle.standardError.write(String(format: "\r%.0f%% ", p * 100).data(using: .utf8)!) + } + FileHandle.standardError.write(String(format: "\nOK out=%.1f LUFS truePeak %.1f dBTP gain %+.1f dB\n", + r.outputLUFS, r.outputTruePeakDb, r.loudnessGainDb).data(using: .utf8)!) + print(r.output.path) + } else if a.count >= 6 && a[1] == "enhance", let mode = parseMode(a[4]), let q = FileQuality(rawValue: a[5]) { let tone = a.count >= 7 ? (TonePreset(rawValue: a[6]) ?? .natural) : .natural let loud = a.count >= 8 ? (LoudnessTarget(rawValue: a[7]) ?? .podcast) : .podcast let ext = URL(fileURLWithPath: a[3]).pathExtension.lowercased() @@ -58,7 +72,7 @@ do { } FileHandle.standardError.write("\nOK\n".data(using: .utf8)!) } else { - die("usage:\n filetool [ll]\n filetool batch \n filetool enhance [natural|warm|bright] [podcast|meeting|none]", 2) + die("usage:\n filetool [ll]\n filetool batch \n filetool enhance [natural|warm|bright] [podcast|meeting|none]\n filetool external [args...] (cmd uses {in}/{out})", 2) } } catch { die("\n\(error.localizedDescription)") diff --git a/app/Tests/CrispEngineTests/ExternalEnhancerTests.swift b/app/Tests/CrispEngineTests/ExternalEnhancerTests.swift new file mode 100644 index 0000000..04cae69 --- /dev/null +++ b/app/Tests/CrispEngineTests/ExternalEnhancerTests.swift @@ -0,0 +1,35 @@ +import XCTest +@testable import CrispEngine + +/// Verifies the external-model integration seam end-to-end using `cp` as a stand-in "model" +/// (passthrough). Proves the plumbing — decode → external process → HQ post-DSP (loudness + +/// true-peak) → write — works; a real model (Resemble/ClearerVoice) is just a different command. +final class ExternalEnhancerTests: XCTestCase { + + func testExternalSeamWithPassthroughCommand() throws { + try XCTSkipUnless(TestSupport.hasSourceAssets, "source assets not present") + let outDir = TestSupport.tempDir("ext") + let out = outDir.appendingPathComponent("ext_out.wav") + // Stand-in "model": copy input → output (env resolves `cp` on PATH). + let model = ExternalEnhancer(name: "passthrough", command: ["cp", "{in}", "{out}"]) + let opts = FileEnhanceOptions(mode: .cleanAndEnhance, quality: .hq, loudness: .podcast, format: .wav) + let report = try FileEnhancer().enhanceExternal(input: TestSupport.speechURL, output: out, model: model, options: opts) + + XCTAssertTrue(FileManager.default.fileExists(atPath: out.path)) + let samples = try AudioIO.read48kMono(out) + XCTAssertGreaterThan(samples.count, 0) + XCTAssertFalse(Metrics.hasNonFinite(samples)) + // HQ post-DSP must have applied the −1 dBTP true-peak ceiling even though the "model" + // was a passthrough. + XCTAssertLessThanOrEqual(report.outputTruePeakDb, -1.0 + 0.25, "HQ post must enforce −1 dBTP") + XCTAssertTrue(report.outputLUFS.isFinite) + } + + func testExternalEnhancerSurfacesFailure() { + let model = ExternalEnhancer(name: "bogus", command: ["this-binary-does-not-exist-xyz", "{in}", "{out}"]) + let tmp = TestSupport.tempDir("extfail") + XCTAssertThrowsError(try model.run(input: tmp.appendingPathComponent("a"), + output: tmp.appendingPathComponent("b")), + "a missing model binary must throw, not crash") + } +} diff --git a/app/Tests/CrispEngineTests/TruePeakTests.swift b/app/Tests/CrispEngineTests/TruePeakTests.swift new file mode 100644 index 0000000..7b7521c --- /dev/null +++ b/app/Tests/CrispEngineTests/TruePeakTests.swift @@ -0,0 +1,39 @@ +import XCTest +@testable import CrispEngine + +final class TruePeakTests: XCTestCase { + let sr = 48_000.0 + + func testConstantHasNoPhantomPeak() { + // A DC/constant signal has no inter-sample overshoot → true peak ≈ sample peak. + let x = [Float](repeating: 0.5, count: 2000) + XCTAssertEqual(Double(TruePeak.truePeak(x)), 0.5, accuracy: 0.01) + } + + func testTruePeakAtLeastSamplePeak() { + let x = TestSupport.synth(seconds: 1) + let sample = x.map { abs($0) }.max() ?? 0 + XCTAssertGreaterThanOrEqual(TruePeak.truePeak(x) + 1e-4, sample, "true peak must be ≥ sample peak") + } + + func testDetectsInterSamplePeak() { + // A high-frequency tone whose true peak exceeds the sampled maxima between samples. + // Near Nyquist/3, sampling lands off the crests, so true peak > sample peak. + let f = 16_000.0 + let n = 4800 + let x = (0..Miipher), 파일·검증 | -| **AnyEnhance** ✅ | masked generative | denoise+dereverb+declip+SR+TSE | ❌ | RTF 0.254@GPU, 363.6M | **비상용 추정**(검증 충돌)⚠️ | **레퍼런스만**(상용 불가 가능성↑) | +| **AnyEnhance** ✅ | masked generative | denoise+dereverb+declip+SR+TSE | ❌ | RTF 0.254@GPU, 363.6M | 재현코드 MIT, **가중치 라이선스 없음**⚠️검증 | **레퍼런스만**(가중치 미라이선스 → 배포 불가) | | **Miipher-2** ✅ | **feature regression**(USM+WaveFit) | 범용 복원 | ❌ | RTF **0.0078**@가속기 | Google, 공개 상용가중치 없음 | **레퍼런스**(배포 불가) | | **NVIDIA RE-USE** | 범용 SE+BWE | 범용 | — | — | **NSCLv1 비상용** | **출시 제외**(PRD 확정) | | **Stream.FM** ✅ | frame-causal **flow matching** | SE 중심(통합은 반박 1-2) | ✅ **32ms/48ms**(24ms SE변형) | — | 연구최전선(2025.12) | **실시간 생성형의 미래** — watch | @@ -102,17 +105,25 @@ ## 라이선스 함정 (출시 전 필수 확인) -- **코드 ≠ 가중치**: Resemble Enhance(MIT 주장 **반박**), NISQA(가중치 제한), RE-USE(NSCLv1 비상용), - AnyEnhance(비상용 가능성). HuggingFace 모델카드 license 별도 확인. -- **비상용 확정/유력 → 출시 제외**: NVIDIA RE-USE(NSCLv1) ✅확정, AnyEnhance ⚠️, NISQA 가중치 ⚠️. -- **상용 가능 유력**: AP-BWE(MIT 주장) ⚠️검증, ClearerVoice(Apache-2.0 주장) ⚠️검증, DeepFilter(현행 사용중) ✅. +- **코드 ≠ 가중치**: Resemble Enhance(MIT 주장 **반박** → 재확인), NISQA(가중치 제한), RE-USE(NSCLv1 비상용), + AnyEnhance(코드 MIT지만 **가중치 무라이선스**). HuggingFace/ModelScope 모델카드 license 별도 확인. +- **비상용/배포불가 → 출시 제외**: NVIDIA RE-USE(NSCLv1) ✅확정, AnyEnhance 가중치 무라이선스 ✅확정, NISQA 가중치 ⚠️. +- **상용 가능 확인됨**: **AP-BWE 코드+가중치 MIT** ✅검증, **ClearerVoice 코드 Apache-2.0** ✅검증(가중치는 ModelScope에서 재확인), + DeepFilter(현행 사용중) ✅. + +> **검증 보강(2026-06-25)**: 세션 한도로 미검증이던 ⚠️ 항목을 1차 소스(GitHub/arxiv) 직접 확인. +> AP-BWE = MIT(코드+가중치)·CPU 18.1×RT·8/12/16/24k→48k 확정. ClearerVoice = Apache-2.0·SR 16→48k 확정. +> AnyEnhance = 재현 코드 MIT지만 공개 가중치에 라이선스 명시 없음 → 가중치 배포 불가(레퍼런스). ## 우선순위 로드맵 -1. **지금(자체, 의존 0)** — **true-peak 오버샘플 리미터** 구현(한계 #6 해소), 평가에 DNSMOS 연동. -2. **다음(검증 후 PoC)** — 파일 HQ에 **Resemble Enhance** 또는 **ClearerVoice-Studio** 드롭인 - (denoise+왜곡복원+BWE 일괄). **라이선스(가중치) 확정 → 벤치(품질/속도/WER) → `AudioProcessor` 어댑터**. -3. **다음(실시간)** — **AP-BWE** causal PoC(ONNX/CoreML), DeepFilter 뒤 BWE stage로. WER/지연 검증. +1. ~~**지금(자체)** — true-peak 오버샘플 리미터~~ **✅ 완료**(`TruePeak.swift`, BS.1770 4×, 파일 HQ가 + −1 dBTP 준수·테스트 통과). 평가에 DNSMOS 연동은 추가 TODO. +2. **다음(검증 후 PoC)** — 파일 HQ에 **ClearerVoice-Studio**(Apache-2.0 ✅) 우선 / **Resemble Enhance** + (가중치 라이선스 확인) 드롭인. **통합 seam·벤치 스캐폴드 구축 완료** → + [`hq-model-integration.md`](hq-model-integration.md) (`ExternalEnhancer` + `scripts/hq-model-poc.sh`). + 남은 일: 모델 설치 → DSP 대비 벤치 → UI 노출. +3. **다음(실시간)** — **AP-BWE**(MIT ✅, CPU 18.1×RT) causal/ONNX PoC, DeepFilter 뒤 BWE stage로. 4. **watch(보류)** — Stream.FM(실시간 생성형), FINALLY(품질 상한 레퍼런스), Miipher-2/AnyEnhance/RE-USE(라이선스). ## 출처 (1차) diff --git a/docs/hq-model-integration.md b/docs/hq-model-integration.md new file mode 100644 index 0000000..583967e --- /dev/null +++ b/docs/hq-model-integration.md @@ -0,0 +1,62 @@ +# 파일 HQ 외부 모델 통합 (PoC) + +[`enhance-limitations-research.md`](enhance-limitations-research.md)의 권고를 실행하기 위한 +**파일 HQ 생성형 모델 통합 seam**과 벤치 스캐폴드. 한계(대역폭 복원·디클리핑·dereverb)는 +파일 경로의 신경망 모델로 넘어가야 하는데, 그 모델은 PyTorch라 Swift/tract 런타임에 번들할 수 +없다 → **out-of-process worker**로 붙인다(PRD §4.4 "PyTorch helper … 파일 HQ beta까지만 허용"). + +## Seam 구조 + +``` +입력파일 → decode(48k mono) → in.wav + → [외부 모델 프로세스] (Resemble Enhance / ClearerVoice / …) → out.wav + → HQ post-DSP: LUFS 정규화 + true-peak(−1 dBTP) 천장 + → 최종 파일 + before/after 리포트 + (temp WAV는 완료/실패 시 삭제 — PRD §7.1 프라이버시) +``` + +- `ExternalEnhancer`(`CrispEngine/ExternalEnhancer.swift`) — `{in}`/`{out}` placeholder를 가진 + argv를 `/usr/bin/env`로 실행. 비정상 종료/미생성 시 throw(앱 크래시 없음). +- `FileEnhancer.enhanceExternal(input:output:model:options:)` — decode→외부실행→post-DSP→write. +- 모델 미설치 시 파일 경로는 **기존 내장 DSP 인핸서**를 그대로 사용(graceful). + +**핵심**: 외부 모델은 레포에 포함하지 않는다. 운영자가 **상용 라이선스를 확인하고** 설치한 +명령을 가리킬 뿐이다. + +CLI: `filetool external [args…]` +(검증: `cp {in} {out}` 패스스루로 plumbing + post-DSP(−1 dBTP) 동작 테스트 통과) + +## 벤치마크 (PRD M1 "동일 테스트셋 비교") + +```sh +./scripts/hq-model-poc.sh --print-setup # Resemble/ClearerVoice 설치·래퍼 방법 출력 +# 외부 모델을 file-in/out 명령으로 export 후: +export RESEMBLE_CMD="$PWD/resemble-enhance-file {in} {out}" +./scripts/hq-model-poc.sh # 내장 DSP + 외부 모델 비교 → test/corpus/hq-poc-report.csv +``` + +스캐폴드는 코퍼스 × 엔진(dsp + 설치된 외부) 처리 → `evaltool`로 LUFS/peak/true-peak/SI-SDR 채점. +외부 모델 미설정이면 **내장 DSP만** 돌리고 건너뛴다(실패 아님). + +## 후보별 통합 메모 (라이선스 검증 결과) + +| 모델 | 통합 방식 | 라이선스 | 비고 | +|---|---|---|---| +| **Resemble Enhance** | dir-CLI → file-in/out 래퍼 | 코드 MIT(검증서 **반박** 이력) · **가중치 출시 전 확인** | denoise+왜곡복원+BWE 일괄, 파일 전용 | +| **ClearerVoice-Studio** | Python API → 래퍼 | **코드 Apache-2.0 ✅** · 가중치(ModelScope) 확인 | FRCRN denoise / 16k→48k SR | +| **AP-BWE** | (파일은 가능하나) **실시간은 네이티브 ONNX/CoreML stage 권장** | **코드+가중치 MIT ✅** · CPU 18.1×RT | BWE 전용, 실시간 후보 → 별도 작업 | +| AnyEnhance | — | 재현코드 MIT, **가중치 무라이선스** → 배포불가 | 레퍼런스만 | +| NVIDIA RE-USE | — | **NSCLv1 비상용** | 출시 제외 | + +## 실시간은 이 seam이 아니다 + +본 seam은 **파일 전용**(subprocess + PyTorch, 지연 무제한). 실시간 BWE(AP-BWE)는 subprocess가 +아니라 **`VoiceEnhancer` 자리에 들어가는 네이티브 causal 스테이지**(ONNX Runtime/CoreML, 무할당) +로 가야 한다 — 별도 PoC. true-peak 리미터는 이미 실시간/파일 공통 DSP로 구현됨. + +## 다음 단계 + +1. 상용 라이선스 확정된 모델 1종(우선 ClearerVoice = Apache-2.0) 설치 → `hq-model-poc.sh`로 + 내장 DSP 대비 품질(LUFS/true-peak/SI-SDR/WER) 벤치. +2. 이기면 `FileTabView`에 "HQ 모델" 옵션 노출(설치 시에만), 미설치 시 DSP fallback. +3. 실시간 BWE는 AP-BWE causal/ONNX PoC를 별도 트랙으로. diff --git a/scripts/hq-model-poc.sh b/scripts/hq-model-poc.sh new file mode 100755 index 0000000..6c847e2 --- /dev/null +++ b/scripts/hq-model-poc.sh @@ -0,0 +1,73 @@ +#!/usr/bin/env bash +# +# File-HQ model PoC / benchmark (PRD M1 "동일 테스트셋으로 비교"). +# Runs the built-in DSP enhancer AND any installed external HQ model (Resemble Enhance / +# ClearerVoice / …) over the test corpus, scores every output with evaltool (LUFS / peak / +# true-peak / SI-SDR vs clean), and writes a comparison CSV. External models are OPTIONAL: +# point env vars at a file-in/file-out command; if unset, that engine is skipped (not failed). +# +# ./scripts/hq-model-poc.sh # baseline DSP + whatever external cmds are set +# ./scripts/hq-model-poc.sh --print-setup # how to install Resemble/ClearerVoice as a wrapper +# +# External command env vars (use {in}/{out} placeholders; must accept/produce a 48k mono WAV): +# RESEMBLE_CMD="resemble-enhance-file {in} {out}" +# CLEARERVOICE_CMD="clearvoice-sr {in} {out}" +set -euo pipefail +ROOT="$(cd "$(dirname "$0")/.." && pwd)"; cd "$ROOT" +CORPUS="$ROOT/test/corpus"; OUT="$CORPUS/hq-poc"; REPORT="$CORPUS/hq-poc-report.csv" + +if [[ "${1:-}" == "--print-setup" ]]; then +cat <<'SETUP' +# ── Resemble Enhance (flow-matching; denoise+왜곡복원+BWE; 파일 전용) ────────────── +python3 -m venv .venv-resemble && source .venv-resemble/bin/activate +pip install resemble-enhance --upgrade # 가중치 자동 다운로드(최초). 라이선스 출시 전 확인! +# resemble-enhance는 디렉터리 단위 CLI → file-in/out 래퍼: +cat > resemble-enhance-file <<'EOF' +#!/usr/bin/env bash +set -e; d=$(mktemp -d); cp "$1" "$d/a.wav" +resemble-enhance "$d" "$d/out" >/dev/null 2>&1 +mv "$d/out/a.wav" "$2"; rm -rf "$d" +EOF +chmod +x resemble-enhance-file +export RESEMBLE_CMD="$PWD/resemble-enhance-file {in} {out}" + +# ── ClearerVoice-Studio (Apache-2.0; FRCRN denoise / MossFormer2 / 16k→48k SR) ────── +git clone https://github.com/modelscope/ClearerVoice-Studio # 가중치는 ModelScope, 라이선스 확인 +# clearvoice Python API를 file-in/out 래퍼로 감싸 CLEARERVOICE_CMD 로 export (repo 예제 참고) +SETUP +exit 0 +fi + +echo "==> Building tools"; swift build --package-path "$ROOT/app" >/dev/null +BIN="$(swift build --package-path "$ROOT/app" --show-bin-path)" +[[ -f "$CORPUS/manifest.csv" ]] || { echo "==> Generating corpus"; "$BIN/maketestset" "$CORPUS" >/dev/null; } +mkdir -p "$OUT"; REF="$CORPUS/clean.wav" + +# engine list: built-in DSP always; external engines only if their env var is set. +declare -a ENGINES=("dsp") +[[ -n "${RESEMBLE_CMD:-}" ]] && ENGINES+=("resemble") +[[ -n "${CLEARERVOICE_CMD:-}" ]] && ENGINES+=("clearervoice") +echo "==> Engines: ${ENGINES[*]}" +[[ ${#ENGINES[@]} -eq 1 ]] && echo " (no external model configured — run with --print-setup to add one)" + +echo "file,category,snr_db,engine,lufs,peak_db,true_peak_db,si_sdr_db,finite" > "$REPORT" +score(){ "$BIN/evaltool" "$1" "$REF" 2>/dev/null; } # CSV: lufs,peak,rms,crest,low,high,sisdr,nonfinite,truepeak + +tail -n +2 "$CORPUS/manifest.csv" | while IFS=, read -r file category snr reference; do + [[ -z "$file" || ! -f "$CORPUS/$file" ]] && continue + IN="$CORPUS/$file"; base="${file%.wav}" + for eng in "${ENGINES[@]}"; do + OUTF="$OUT/${base}__${eng}.wav" + case "$eng" in + dsp) "$BIN/filetool" enhance "$IN" "$OUTF" clean hq natural podcast >/dev/null 2>&1 || continue ;; + resemble) "$BIN/filetool" external "$IN" "$OUTF" hq podcast ${RESEMBLE_CMD} >/dev/null 2>&1 || { echo " resemble failed on $file"; continue; } ;; + clearervoice) "$BIN/filetool" external "$IN" "$OUTF" hq podcast ${CLEARERVOICE_CMD} >/dev/null 2>&1 || { echo " clearervoice failed on $file"; continue; } ;; + esac + c="$(score "$OUTF")" + lufs=$(echo "$c"|cut -d, -f1); peak=$(echo "$c"|cut -d, -f2); tp=$(echo "$c"|cut -d, -f9) + sisdr=$(echo "$c"|cut -d, -f7); fin=$(echo "$c"|cut -d, -f8) + echo "$file,$category,$snr,$eng,$lufs,$peak,$tp,$sisdr,$fin" >> "$REPORT" + done +done + +echo; echo "==> Report: $REPORT"; column -s, -t "$REPORT" | sed 's/^/ /' From ea82dbeeede0d498d8764f883e68c90e2f25ee47 Mon Sep 17 00:00:00 2001 From: Hyeon-Mook Jelly Choi Date: Fri, 26 Jun 2026 22:19:21 +0900 Subject: [PATCH 6/6] feat(file): expose optional ClearerVoice HQ model --- .gitignore | 3 + LICENSES.md | 6 +- README.md | 8 +- app/Sources/CrispApp/Views/FileTabView.swift | 143 ++++++++++++++++--- docs/STATUS.md | 8 +- docs/enhance-limitations-research.md | 15 +- docs/hq-model-integration.md | 42 +++++- docs/test-report.md | 19 ++- docs/voice-enhancer.md | 14 +- scripts/clearvoice-wrapper.py | 36 +++++ scripts/hq-model-poc.sh | 9 +- 11 files changed, 256 insertions(+), 47 deletions(-) create mode 100755 scripts/clearvoice-wrapper.py diff --git a/.gitignore b/.gitignore index c54016a..2b62cf6 100644 --- a/.gitignore +++ b/.gitignore @@ -20,6 +20,9 @@ engine/models/ models/*.onnx models/*.tar.gz +# External HQ model checkpoints downloaded during local PoC runs +/checkpoints/ + # macOS .DS_Store diff --git a/LICENSES.md b/LICENSES.md index 8398cbb..d9cd627 100644 --- a/LICENSES.md +++ b/LICENSES.md @@ -9,10 +9,12 @@ Crisp가 사용하는 서드파티 구성요소와 상용 배포 가능 여부. | Crisp HAL 드라이버 코드 | 가상 마이크 | 자체 작성(원본). Apple Audio Server Plug-in 아키텍처 참고 | ✅ 자체 저작 | | Crisp 앱/엔진 코드 | UI, 실시간/파일 처리 | 자체 작성 | ✅ 자체 저작 | | Crisp Voice Enhancer DSP | EQ/컴프레서/디에서/리미터/LUFS (PRD v0.2) | 자체 작성(RBJ EQ cookbook·ITU-R BS.1770 공개 공식 기반, 코드 원본) | ✅ 자체 저작 | +| ClearerVoice-Studio `clearvoice` + MossFormer2_SE_48K | 선택적 파일 HQ 외부 모델(Python/Torch 프로세스) | Apache-2.0(코드·가중치 README 확인) | ✅ 가능, **제품 번들 아님** | > **Voice Enhancer(PRD v0.2)는 신규 서드파티·모델 weight를 추가하지 않는다.** 인핸서를 생성형 -> 모델 대신 순수 DSP(자체 코드)로 구현했기 때문이다. PRD가 후보로 든 LocalVQE(Apache-2.0), -> Resemble Enhance(MIT), ClearerVoice(Apache-2.0)는 추후 동일 `AudioProcessor` seam에 드롭인할 +> 모델 대신 순수 DSP(자체 코드)로 구현했기 때문이다. ClearerVoice는 설치된 외부 명령이 있을 때만 +> out-of-process 파일 HQ 옵션으로 연결하며 앱/패키지에 Python·Torch·가중치를 번들하지 않는다. PRD가 후보로 든 +> LocalVQE(Apache-2.0), Resemble Enhance(MIT)는 추후 동일 `AudioProcessor` seam에 드롭인할 > 수 있으나, 그 가중치·코드 라이선스는 **탑재 시점에 별도 BOM 확인 필요**(PRD §7.2, §9 M1). > NVIDIA RE-USE 등 비상용 모델은 출시 빌드 제외 원칙 유지. diff --git a/README.md b/README.md index 4c3e60f..6958cbb 100644 --- a/README.md +++ b/README.md @@ -66,12 +66,14 @@ open build/Crisp-0.1.0.pkg **파일**: 설정 창 → 파일 탭 → 오디오/비디오(wav/mp3/m4a/mp4/mov) 드래그앤드롭 → 처리 모드 + 품질(Fast/HQ) + 톤 + 음량 정규화(-16/-18 LUFS) 선택 → **미리듣기(20초)** 로 결과 방향 확인 후 **음성 개선 시작**. 원본은 덮어쓰지 않고 wav/m4a로 저장하며 before/after LUFS·peak 리포트를 표시. +ClearerVoice가 외부 명령/venv로 설치되어 있으면 HQ에서 "HQ 모델" 선택지가 나타나고, 없으면 내장 DSP만 사용한다. ## 검증 / 테스트 ```sh -swift test --package-path app # 단위·통합 테스트 (26개: DSP·파이프라인·LUFS·메트릭·파일 end-to-end) +swift test --package-path app # 단위·통합 테스트 (32개: DSP·파이프라인·LUFS·true-peak·외부 모델 seam) ./scripts/quality-eval.sh # 테스트셋 생성+배치 처리 → quality-report.csv (회귀 게이트) +./scripts/hq-model-poc.sh # 파일 HQ 외부 모델 PoC (CLEARERVOICE_CMD/RESEMBLE_CMD 설정 시 비교) ./poc/model/run_poc.sh # 모델 RTF·노이즈 감소 ./scripts/e2e-test.sh # end-to-end: model→가상마이크→녹음 (드라이버 설치 필요) ./scripts/stability-test.sh 1800 # 30분 안정성 (crash/dropout) @@ -98,7 +100,7 @@ app/Package.swift SwiftPM: CDeepFilter / CrispEngine / CrispApp / dftoo engine/CDeepFilter/lib/ libdf.dylib (fetch-deps 생성, gitignore) engine/models/ DeepFilterNet3 모델 (fetch-deps 생성, gitignore) test/corpus/ 평가 테스트셋 (maketestset 생성, gitignore) + README -scripts/ fetch-deps / install / verify / package / e2e / stability / quality-eval +scripts/ fetch-deps / install / verify / package / e2e / stability / quality-eval / hq-model-poc docs/ architecture · voice-enhancer · enhance-behavior · enhance-limitations-research · hq-model-integration · STATUS · test-report poc/model/ DeepFilterNet 클론(gitignore) + run_poc.sh ``` @@ -114,7 +116,7 @@ poc/model/ DeepFilterNet 클론(gitignore) + run_poc.sh | 2 실시간 엔진 | capture→model→가상마이크 | ✅ end-to-end loopback | | 3 앱 | 메뉴바/온보딩/설정 | ✅ | | 4 파일 처리 | ffmpeg-free (AVFoundation) | ✅ mp3→m4a, mp4→wav | -| VE Voice Enhancer (v0.2) | 2-stage 파이프라인 + DSP 인핸서, 4모드, 파일 HQ | ✅ vetool 검증, RTF 0.11 | +| VE Voice Enhancer (v0.2) | 2-stage 파이프라인 + DSP 인핸서, 4모드, 파일 HQ | ✅ vetool 검증, 외부 HQ PoC | | 5 패키징 | 서명/notarization/pkg | 🔶 pkg ✅, notarize는 Apple 계정 | --- diff --git a/app/Sources/CrispApp/Views/FileTabView.swift b/app/Sources/CrispApp/Views/FileTabView.swift index a8d34bc..a5d8078 100644 --- a/app/Sources/CrispApp/Views/FileTabView.swift +++ b/app/Sources/CrispApp/Views/FileTabView.swift @@ -3,11 +3,76 @@ import UniformTypeIdentifiers import AVFoundation import CrispEngine +struct FileHQEngineOption: Hashable, Identifiable { + let id: String + let label: String + let command: [String]? + + static let builtIn = FileHQEngineOption(id: "dsp", label: "내장 DSP", command: nil) + + var isExternal: Bool { command != nil } +} + +enum FileHQEngineDiscovery { + static func available() -> [FileHQEngineOption] { + var options = [FileHQEngineOption.builtIn] + if let clearVoice = clearVoice() { options.append(clearVoice) } + return options + } + + private static func clearVoice() -> FileHQEngineOption? { + let env = ProcessInfo.processInfo.environment + if let raw = env["CRISP_CLEARERVOICE_CMD"] ?? env["CLEARERVOICE_CMD"], + !raw.trimmingCharacters(in: .whitespacesAndNewlines).isEmpty { + return FileHQEngineOption(id: "clearervoice", label: "ClearerVoice", command: ["/bin/zsh", "-lc", raw]) + } + + for root in candidateRoots() { + let python = root.appendingPathComponent(".clearvoice.venv/bin/python") + let wrapper = root.appendingPathComponent("scripts/clearvoice-wrapper.py") + let checkpoint = root.appendingPathComponent("checkpoints/MossFormer2_SE_48K/last_best_checkpoint.pt") + let fm = FileManager.default + if fm.isExecutableFile(atPath: python.path), + fm.isReadableFile(atPath: wrapper.path), + fm.fileExists(atPath: checkpoint.path) { + return FileHQEngineOption(id: "clearervoice", label: "ClearerVoice", + command: [python.path, wrapper.path, "{in}", "{out}"]) + } + } + return nil + } + + private static func candidateRoots() -> [URL] { + var seeds: [URL] = [] + let env = ProcessInfo.processInfo.environment + if let root = env["CRISP_REPO_ROOT"] { seeds.append(URL(fileURLWithPath: root, isDirectory: true)) } + seeds.append(URL(fileURLWithPath: FileManager.default.currentDirectoryPath, isDirectory: true)) + if let executable = Bundle.main.executableURL { + seeds.append(executable.deletingLastPathComponent()) + } + + var roots: [URL] = [] + var seen = Set() + for seed in seeds { + var url = seed.standardizedFileURL + while true { + let path = url.path + if seen.insert(path).inserted { roots.append(url) } + let parent = url.deletingLastPathComponent() + if parent.path == path { break } + url = parent + } + } + return roots + } +} + @MainActor final class FileTabModel: ObservableObject { @Published var inputURL: URL? @Published var mode: ProcessingMode = .cleanAndEnhance @Published var quality: FileQuality = .hq + @Published var hqEngine: FileHQEngineOption @Published var enhanceStrength: EnhanceStrength = .medium @Published var tonePreset: TonePreset = .natural @Published var noiseStrength: NoiseStrength = .high @@ -24,6 +89,12 @@ final class FileTabModel: ObservableObject { private let enhancer = FileEnhancer() private var player: AVAudioPlayer? private var lastPreview: URL? + let hqEngineOptions: [FileHQEngineOption] + + init(hqEngineOptions: [FileHQEngineOption] = FileHQEngineDiscovery.available()) { + self.hqEngineOptions = hqEngineOptions + self.hqEngine = hqEngineOptions.first ?? .builtIn + } func setInput(_ url: URL) { inputURL = url; results = []; report = nil; errorText = nil; progress = 0; statusText = "" @@ -37,19 +108,40 @@ final class FileTabModel: ObservableObject { loudness: loudness, format: format, previewSeconds: previewSeconds) } + var usesExternalHQ: Bool { + quality == .hq && hqEngine.isExternal + } + + private var selectedExternalEnhancer: ExternalEnhancer? { + guard quality == .hq, let command = hqEngine.command else { return nil } + return ExternalEnhancer(name: hqEngine.id, command: command) + } + + private var outputSuffix: String { + selectedExternalEnhancer == nil ? "crisp" : "crisp_\(hqEngine.id)" + } + /// Full-file processing → user-chosen output (PRD FR-FILE-005: never overwrites the input). func start() { guard let input = inputURL, !isProcessing else { return } let panel = NSSavePanel() - panel.nameFieldStringValue = input.deletingPathExtension().lastPathComponent + "_crisp." + format.ext + panel.nameFieldStringValue = input.deletingPathExtension().lastPathComponent + "_\(outputSuffix)." + format.ext panel.allowedContentTypes = [format == .wav ? .wav : .mpeg4Audio] guard panel.runModal() == .OK, let output = panel.url else { return } let opts = options(), enhancer = self.enhancer + let external = selectedExternalEnhancer begin("처리 중…") Task.detached { do { - let r = try enhancer.enhance(input: input, output: output, options: opts) { p in - Task { @MainActor in self.progress = p } + let r: FileEnhanceReport + if let external { + r = try enhancer.enhanceExternal(input: input, output: output, model: external, options: opts) { p in + Task { @MainActor in self.progress = p } + } + } else { + r = try enhancer.enhance(input: input, output: output, options: opts) { p in + Task { @MainActor in self.progress = p } + } } await MainActor.run { self.done([output], report: r) } } catch { await MainActor.run { self.fail(error) } } @@ -61,14 +153,21 @@ final class FileTabModel: ObservableObject { guard let input = inputURL, !isProcessing else { return } if let old = lastPreview { try? FileManager.default.removeItem(at: old) } let tmp = FileManager.default.temporaryDirectory - .appendingPathComponent("crisp_preview_\(UInt32(truncatingIfNeeded: input.hashValue)).\(format.ext)") + .appendingPathComponent("crisp_preview_\(hqEngine.id)_\(UInt32(truncatingIfNeeded: input.hashValue)).\(format.ext)") lastPreview = tmp let opts = options(previewSeconds: 20), enhancer = self.enhancer + let external = selectedExternalEnhancer begin("미리듣기 생성 중…") Task.detached { do { - _ = try enhancer.enhance(input: input, output: tmp, options: opts) { p in - Task { @MainActor in self.progress = p } + if let external { + _ = try enhancer.enhanceExternal(input: input, output: tmp, model: external, options: opts) { p in + Task { @MainActor in self.progress = p } + } + } else { + _ = try enhancer.enhance(input: input, output: tmp, options: opts) { p in + Task { @MainActor in self.progress = p } + } } await MainActor.run { self.isProcessing = false; self.statusText = "미리듣기 재생"; self.play(tmp) } } catch { await MainActor.run { self.fail(error) } } @@ -120,26 +219,36 @@ struct FileTabView: View { private var settings: some View { GroupBox { VStack(alignment: .leading, spacing: 10) { - Picker("처리 모드", selection: $model.mode) { - ForEach(ProcessingMode.allCases) { Text($0.label).tag($0) } - } Picker("품질", selection: $model.quality) { Text("빠르게 (Fast)").tag(FileQuality.fast) Text("고품질 (HQ)").tag(FileQuality.hq) } .pickerStyle(.segmented) - if model.mode == .noiseCancellation || model.mode == .cleanAndEnhance { - Picker("노이즈 강도", selection: $model.noiseStrength) { - ForEach(NoiseStrength.allCases) { Text($0.label).tag($0) } + if model.quality == .hq && model.hqEngineOptions.count > 1 { + Picker("HQ 모델", selection: $model.hqEngine) { + ForEach(model.hqEngineOptions) { option in + Text(option.label).tag(option) + } } } - if model.mode.usesEnhancer { - Picker("인핸스 강도", selection: $model.enhanceStrength) { - ForEach(EnhanceStrength.allCases) { Text($0.label).tag($0) } + + if !model.usesExternalHQ { + Picker("처리 모드", selection: $model.mode) { + ForEach(ProcessingMode.allCases) { Text($0.label).tag($0) } + } + if model.mode == .noiseCancellation || model.mode == .cleanAndEnhance { + Picker("노이즈 강도", selection: $model.noiseStrength) { + ForEach(NoiseStrength.allCases) { Text($0.label).tag($0) } + } } - Picker("톤", selection: $model.tonePreset) { - ForEach(TonePreset.allCases) { Text($0.label).tag($0) } + if model.mode.usesEnhancer { + Picker("인핸스 강도", selection: $model.enhanceStrength) { + ForEach(EnhanceStrength.allCases) { Text($0.label).tag($0) } + } + Picker("톤", selection: $model.tonePreset) { + ForEach(TonePreset.allCases) { Text($0.label).tag($0) } + } } } if model.quality == .hq { diff --git a/docs/STATUS.md b/docs/STATUS.md index 5590f1e..bc2121e 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -46,7 +46,11 @@ PRD `krisp_like_mac_voice_enhancer_prd_v02.docx` 구현. 상세: [`docs/voice-en - ✅ **클릭 없는 전환** — 항상 in-path + dry→wet 40ms 램프, `wetMix=0` bit-identical 패스스루(검증) - ✅ **스트리밍 결정성** — `vetool`: aligned == chunked bit-identical, 리미터 안전, RTF enhancer 0.012 / clean+enhance 0.11 - ✅ **파일 HQ** — Fast/HQ, 미리듣기(20초), LUFS 정규화(-16/-18) + peak 천장(-1dBFS), before/after 리포트 -- ✅ **모델/가중치 추가 0** — 생성형 아님(화자 보존), 라이선스 BOM 변화 없음. LocalVQE/Resemble는 동일 seam에 드롭인 가능(PRD §9 M1 별도) +- ✅ **외부 파일 HQ PoC** — ClearerVoice `MossFormer2_SE_48K` file-in/out 래퍼 + `filetool external` 검증, + 12개 코퍼스 평균 SI-SDR 2.55→6.04 dB +- ✅ **외부 파일 HQ UI** — ClearerVoice 명령/venv/체크포인트가 설치된 경우에만 "HQ 모델" 선택지 노출, + 미설치 시 내장 DSP fallback +- ✅ **번들 모델/가중치 추가 0** — 기본 DSP 인핸서는 생성형이 아니며 화자/발화를 보존. ClearerVoice는 설치된 외부 명령으로만 연결(PRD §9 M1 별도) ## 남은 항목 — 사람/계정/GUI 앱 필요 (자동화 불가) @@ -59,5 +63,5 @@ PRD `krisp_like_mac_voice_enhancer_prd_v02.docx` 구현. 상세: [`docs/voice-en ## 다음 작업 - end-to-end 실시간 검증 (마이크→모델→가상마이크→회의앱) — **드라이버 설치 후 가능** -- limiter/AGC (PRD 6.2) — 선택적 음질 보호 +- 파일 HQ 모델 제품 노출 판단 — 사람 A/B 청취 + WER/DNSMOS/PESQ 등 추가 지표 - Developer ID 서명 + notarization (Apple 계정) diff --git a/docs/enhance-limitations-research.md b/docs/enhance-limitations-research.md index f4b1303..7684ddb 100644 --- a/docs/enhance-limitations-research.md +++ b/docs/enhance-limitations-research.md @@ -108,7 +108,7 @@ - **코드 ≠ 가중치**: Resemble Enhance(MIT 주장 **반박** → 재확인), NISQA(가중치 제한), RE-USE(NSCLv1 비상용), AnyEnhance(코드 MIT지만 **가중치 무라이선스**). HuggingFace/ModelScope 모델카드 license 별도 확인. - **비상용/배포불가 → 출시 제외**: NVIDIA RE-USE(NSCLv1) ✅확정, AnyEnhance 가중치 무라이선스 ✅확정, NISQA 가중치 ⚠️. -- **상용 가능 확인됨**: **AP-BWE 코드+가중치 MIT** ✅검증, **ClearerVoice 코드 Apache-2.0** ✅검증(가중치는 ModelScope에서 재확인), +- **상용 가능 확인됨**: **AP-BWE 코드+가중치 MIT** ✅검증, **ClearerVoice 코드+MossFormer2_SE_48K 가중치 Apache-2.0** ✅검증, DeepFilter(현행 사용중) ✅. > **검증 보강(2026-06-25)**: 세션 한도로 미검증이던 ⚠️ 항목을 1차 소스(GitHub/arxiv) 직접 확인. @@ -119,12 +119,15 @@ 1. ~~**지금(자체)** — true-peak 오버샘플 리미터~~ **✅ 완료**(`TruePeak.swift`, BS.1770 4×, 파일 HQ가 −1 dBTP 준수·테스트 통과). 평가에 DNSMOS 연동은 추가 TODO. -2. **다음(검증 후 PoC)** — 파일 HQ에 **ClearerVoice-Studio**(Apache-2.0 ✅) 우선 / **Resemble Enhance** - (가중치 라이선스 확인) 드롭인. **통합 seam·벤치 스캐폴드 구축 완료** → - [`hq-model-integration.md`](hq-model-integration.md) (`ExternalEnhancer` + `scripts/hq-model-poc.sh`). - 남은 일: 모델 설치 → DSP 대비 벤치 → UI 노출. +2. ~~**다음(검증 후 PoC)** — 파일 HQ에 **ClearerVoice-Studio**(Apache-2.0 ✅) 우선~~ **✅ 완료**: + `ExternalEnhancer` + `scripts/clearvoice-wrapper.py` + `scripts/hq-model-poc.sh`로 DSP 대비 벤치 완료 + (12개 코퍼스 평균 SI-SDR 2.55→6.04 dB, +3.50 dB). 상세: + [`hq-model-integration.md`](hq-model-integration.md). + `FileTabView` UI 노출도 완료(설치 시 ClearerVoice 선택, 미설치 시 DSP fallback). + 남은 일: 사람/인식 품질 검증. 3. **다음(실시간)** — **AP-BWE**(MIT ✅, CPU 18.1×RT) causal/ONNX PoC, DeepFilter 뒤 BWE stage로. -4. **watch(보류)** — Stream.FM(실시간 생성형), FINALLY(품질 상한 레퍼런스), Miipher-2/AnyEnhance/RE-USE(라이선스). +4. **watch(보류)** — Resemble Enhance(가중치 라이선스 확인), Stream.FM(실시간 생성형), + FINALLY(품질 상한 레퍼런스), Miipher-2/AnyEnhance/RE-USE(라이선스). ## 출처 (1차) diff --git a/docs/hq-model-integration.md b/docs/hq-model-integration.md index 583967e..ea6e1e5 100644 --- a/docs/hq-model-integration.md +++ b/docs/hq-model-integration.md @@ -32,18 +32,48 @@ CLI: `filetool external [args ./scripts/hq-model-poc.sh --print-setup # Resemble/ClearerVoice 설치·래퍼 방법 출력 # 외부 모델을 file-in/out 명령으로 export 후: export RESEMBLE_CMD="$PWD/resemble-enhance-file {in} {out}" +export CLEARERVOICE_CMD="$PWD/.clearvoice.venv/bin/python $PWD/scripts/clearvoice-wrapper.py {in} {out}" ./scripts/hq-model-poc.sh # 내장 DSP + 외부 모델 비교 → test/corpus/hq-poc-report.csv ``` 스캐폴드는 코퍼스 × 엔진(dsp + 설치된 외부) 처리 → `evaltool`로 LUFS/peak/true-peak/SI-SDR 채점. 외부 모델 미설정이면 **내장 DSP만** 돌리고 건너뛴다(실패 아님). +### ClearerVoice PoC 결과 (2026-06-25) + +환경: Apple Silicon CPU, `clearvoice==0.1.2`, `MossFormer2_SE_48K`, +`scripts/clearvoice-wrapper.py` → `filetool external` → HQ post-DSP. + +| 엔진 | 샘플 수 | 평균 SI-SDR | 평균 LUFS | true-peak | finite | +|---|---:|---:|---:|---:|---:| +| 내장 DSP HQ | 12 | **2.55 dB** | −19.41 | −1.00 dBTP | 12/12 | +| ClearerVoice HQ | 12 | **6.04 dB** | −20.34 | −1.00 dBTP | 12/12 | + +ClearerVoice는 전체 샘플에서 DSP보다 SI-SDR이 높았다(평균 **+3.50 dB**, 최소 +1.92 dB +`reverb_noisy_snr5`, 최대 +7.25 dB `quiet`). 특히 bandlimit(평균 6.11 vs 1.96 dB), +clipping(6.14 vs 2.86 dB), noisy(5.62 vs 2.59 dB)에서 파일 HQ 후보로 의미 있는 개선을 보였다. + +주의: SI-SDR은 객관 proxy일 뿐이다. 실제 제품 노출 전에는 사람 A/B 청취, WER, DNSMOS/PESQ/NISQA +같은 지각/인식 지표를 추가해야 한다. ClearerVoice는 Python+Torch+체크포인트를 요구하므로 현재 앱 +번들에 포함하지 않고, 설치된 외부 명령이 있을 때만 file-HQ beta로 연결하는 구조가 맞다. + +### 앱 노출 방식 + +`FileTabView`는 HQ 품질에서만 "HQ 모델" Picker를 보인다. 발견 순서: + +1. `CRISP_CLEARERVOICE_CMD` 또는 `CLEARERVOICE_CMD` 환경변수 +2. 현재 작업 디렉터리/실행 파일 상위 경로에서 `.clearvoice.venv/bin/python`, + `scripts/clearvoice-wrapper.py`, `checkpoints/MossFormer2_SE_48K/last_best_checkpoint.pt` 동시 발견 + +ClearerVoice가 없으면 Picker가 나타나지 않고 기존 내장 DSP 경로를 그대로 쓴다. ClearerVoice 선택 시 +내장 DSP의 모드/강도/톤 컨트롤은 숨기고, 외부 모델 출력 뒤 Crisp의 LUFS/true-peak post-DSP만 적용한다. + ## 후보별 통합 메모 (라이선스 검증 결과) | 모델 | 통합 방식 | 라이선스 | 비고 | |---|---|---|---| | **Resemble Enhance** | dir-CLI → file-in/out 래퍼 | 코드 MIT(검증서 **반박** 이력) · **가중치 출시 전 확인** | denoise+왜곡복원+BWE 일괄, 파일 전용 | -| **ClearerVoice-Studio** | Python API → 래퍼 | **코드 Apache-2.0 ✅** · 가중치(ModelScope) 확인 | FRCRN denoise / 16k→48k SR | +| **ClearerVoice-Studio** | Python API → 래퍼 | **코드 Apache-2.0 ✅** · MossFormer2_SE_48K 가중치 README Apache-2.0 ✅ | FRCRN denoise / 16k→48k SR | | **AP-BWE** | (파일은 가능하나) **실시간은 네이티브 ONNX/CoreML stage 권장** | **코드+가중치 MIT ✅** · CPU 18.1×RT | BWE 전용, 실시간 후보 → 별도 작업 | | AnyEnhance | — | 재현코드 MIT, **가중치 무라이선스** → 배포불가 | 레퍼런스만 | | NVIDIA RE-USE | — | **NSCLv1 비상용** | 출시 제외 | @@ -56,7 +86,9 @@ export RESEMBLE_CMD="$PWD/resemble-enhance-file {in} {out}" ## 다음 단계 -1. 상용 라이선스 확정된 모델 1종(우선 ClearerVoice = Apache-2.0) 설치 → `hq-model-poc.sh`로 - 내장 DSP 대비 품질(LUFS/true-peak/SI-SDR/WER) 벤치. -2. 이기면 `FileTabView`에 "HQ 모델" 옵션 노출(설치 시에만), 미설치 시 DSP fallback. -3. 실시간 BWE는 AP-BWE causal/ONNX PoC를 별도 트랙으로. +1. ~~상용 라이선스 확정된 모델 1종(우선 ClearerVoice = Apache-2.0) 설치 → `hq-model-poc.sh` 벤치~~ + **완료**: ClearerVoice가 SI-SDR 기준 DSP 대비 평균 +3.50 dB. +2. ~~`FileTabView`에 "HQ 모델" 옵션 노출 방식 설계~~ **완료**: 설치된 외부 명령/venv/체크포인트를 + 발견할 때만 ClearerVoice 선택지를 보이고, 미설치 시 DSP fallback. 앱 번들에 Python/Torch를 포함하지 않는다. +3. 사람 A/B 청취 + WER/DNSMOS/PESQ 등 추가 지표로 제품 노출 여부 결정. +4. 실시간 BWE는 AP-BWE causal/ONNX PoC를 별도 트랙으로. diff --git a/docs/test-report.md b/docs/test-report.md index 14145ea..17db57c 100644 --- a/docs/test-report.md +++ b/docs/test-report.md @@ -95,8 +95,19 @@ DSP 인핸서 + 2-stage 파이프라인. 구현/설계: [`voice-enhancer.md`](vo 출력 48k mono Int16 WAV / AAC m4a, 길이 정확 보존(10.595646s in=out). `wav/mp3/m4a/mp4/mov` 입력. -> **한계**: true-peak(-1 dBTP)는 오버샘플링 없이 sample-peak 천장으로 근사. LUFS는 BS.1770 -> 근사(K-weighting + 절대/상대 게이팅). LocalVQE/Resemble 실모델은 동일 seam 드롭인(PRD §9 M1 별도). +파일 HQ 외부 모델 PoC (`scripts/hq-model-poc.sh`, 12개 코퍼스, `MossFormer2_SE_48K`): + +| 엔진 | 평균 SI-SDR | 평균 LUFS | true-peak | finite | +|---|---:|---:|---:|---:| +| 내장 DSP HQ | 2.55 dB | −19.41 | −1.00 dBTP | 12/12 | +| ClearerVoice HQ | **6.04 dB** | −20.34 | −1.00 dBTP | 12/12 | + +ClearerVoice는 전체 샘플에서 DSP보다 SI-SDR이 높았다(평균 **+3.50 dB**). 통합 경로: +`filetool external` → `scripts/clearvoice-wrapper.py` → HQ LUFS/true-peak post-DSP. + +> **한계**: LUFS는 BS.1770 근사(K-weighting + 절대/상대 게이팅). ClearerVoice 실모델은 파일 HQ +> beta 후보이며 Python/Torch 외부 프로세스로만 연결한다. 제품 노출 전 사람 A/B 청취와 WER/DNSMOS/PESQ +> 같은 지각/인식 지표가 필요하다. ## 테스트셋 & 자동화 테스트 (PRD §8.1 / §8.2 / §9.4) @@ -111,10 +122,12 @@ bandlimit(8·16k) / 저음량 / hum. 재현: `test/corpus/README.md`. | BiquadTests (4) | 0dB=identity, LPF/HPF 거동, 엔벨로프 수렴 | ✅ | | VoiceEnhancerTests (5) | 패스스루 bit-identical·chunk 독립·리미터·톤·램프 | ✅ | | LoudnessTests (4) | 무음/선형성/정규화/peak 천장 | ✅ | +| TruePeakTests (4) | inter-sample peak 감지·true-peak 천장 | ✅ | | MetricsTests (5) | RMS·peak·SI-SDR·지연보정 SI-SDR | ✅ | | PipelineTests (4) | 모드 라우팅·길이보존·라이브 전환 finite | ✅ | | IntegrationTests (4) | 코퍼스 생성·파일 end-to-end(모델) | ✅ | -| **합계** | | **26 tests, 0 failures** | +| ExternalEnhancerTests (2) | 외부 모델 seam·실패 전파 | ✅ | +| **합계** | | **32 tests, 0 failures** | **배치 평가** (`scripts/quality-eval.sh` → `test/corpus/quality-report.csv`) — 코퍼스 × {noise, voice, clean} HQ 처리. 모든 출력 finite·peak −1 dBFS 천장 준수 → **PASS(회귀 게이트)**. diff --git a/docs/voice-enhancer.md b/docs/voice-enhancer.md index 6823fb8..1a53712 100644 --- a/docs/voice-enhancer.md +++ b/docs/voice-enhancer.md @@ -44,7 +44,7 @@ ClearerVoice** 를 "1순위 PoC"(§1, §2.1, §9 M1)로 제시한다. 이들은 | FR-FILE-005 | 원본 비덮어쓰기, WAV/오디오교체 저장 | NSSavePanel 별도 출력, wav/m4a | | FR-FILE-006 | before/after loudness/peak 리포트 | `FileEnhanceReport` (LUFS·peak·gain) UI 표기 | | §4.3 step6 | -16 LUFS(podcast) / -18 LUFS(meeting) | `LoudnessTarget` (BS.1770 K-weighting + 게이팅) | -| §4.3 step7 | -1.0 dBTP true-peak | sample-peak 천장 -1 dBFS 근사(아래 한계 참고) | +| §4.3 step7 | -1.0 dBTP true-peak | `TruePeak.swift` 4× 오버샘플 true-peak 천장 | | SET-02 | model/runtime/sr/latency 진단 | Diagnostics 확장 (mode/강도/톤/추정 지연) | | §6.2 | High는 기본값 아님, 보수적 기본 | 기본 Clean+Enhance / Medium / Natural | @@ -109,13 +109,15 @@ PRD §8.1 테스트셋과 §8.2 자동 평가, §9.4 "batch 처리 스크립트 clean / noisy(SNR 0·5·10·20) / reverb / reverb+noise / clipping / bandlimit(8·16k) / 저음량 / hum. 생성: `maketestset test/corpus` (`test/corpus/README.md` 참고). -**XCTest 스위트** (`swift test`) — **26개 테스트, 0 실패**: +**XCTest 스위트** (`swift test`) — **32개 테스트, 0 실패**: - `BiquadTests` — 0dB peaking=identity, LPF가 HF 감쇠, HPF가 DC 제거, 엔벨로프 수렴 - `VoiceEnhancerTests` — 비활성 패스스루 bit-identical, chunk 독립성, 리미터 full-scale 이내, 톤 프리셋 finite, 램프 클릭 없음 - `LoudnessTests` — 무음=-inf, 음량 선형성(+20LU), 정규화 목표 도달, peak 천장 준수 +- `TruePeakTests` — inter-sample peak 감지, true-peak 천장 준수 - `MetricsTests` — RMS/peak, SI-SDR(동일=∞, 노이즈↑=점수↓), **지연 보정 SI-SDR**(순수 지연 복원) - `PipelineTests` — Off/Noise 무채색, Clean+Enhance 변화+길이보존, 라이브 모드 전환 finite - `IntegrationTests`(자산/모델 있을 때) — 코퍼스 생성 유효성, 파일 end-to-end(길이 보존·천장·finite) +- `ExternalEnhancerTests` — 외부 모델 seam, 비정상 명령 실패 전파 **배치 평가** (`scripts/quality-eval.sh`) — 코퍼스 × 모드 처리 → `quality-report.csv` (LUFS/peak/SI-SDR). 비정상 출력(non-finite·클리핑) 시 exit 1 → **회귀 게이트**. @@ -133,9 +135,9 @@ SI-SDR은 모델 지연을 cross-correlation으로 정렬 후 측정. 측정 결 ## 한계 / 후속 -- **True-peak(-1 dBTP)** 는 4× 오버샘플링 없이 **sample-peak 천장**으로 근사(음성에서 보수적). - 방송 수준 정밀 true-peak가 필요하면 oversampling 리미터로 교체. +- **True-peak(-1 dBTP)** 는 `TruePeak.swift`의 4× 오버샘플링 경로로 검증한다. - **LUFS** 게이팅은 절대(-70)·상대(-10 LU) 게이트 구현, K-weighting은 BS.1770 근사 계수. -- **LocalVQE / Resemble / ClearerVoice 실모델**: 동일 `AudioProcessor` seam에 래퍼로 드롭인 - 가능. 가중치 다운로드·벤치마크·라이선스 확인은 PRD §9 M1 별도 마일스톤. +- **ClearerVoice 실모델**: 파일 HQ 외부 모델로 연결 완료. 설치된 명령/venv/체크포인트가 있을 때만 + `FileTabView`의 "HQ 모델" 선택지로 노출한다. +- **LocalVQE / Resemble 실모델**: 동일 seam에 래퍼로 드롭인 가능하나, 가중치·라이선스 확인은 별도. - A/B blind 청취(PRD §8.3) 및 회의 앱 실사용은 사람 검수 필요(`docs/acceptance-checklist.md`). diff --git a/scripts/clearvoice-wrapper.py b/scripts/clearvoice-wrapper.py new file mode 100755 index 0000000..b69ad10 --- /dev/null +++ b/scripts/clearvoice-wrapper.py @@ -0,0 +1,36 @@ +#!/usr/bin/env python3 +"""File-in/file-out adapter for ClearerVoice-Studio (ClearVoice), matching Crisp's +ExternalEnhancer {in}/{out} contract so it can be driven by `filetool external` and +`scripts/hq-model-poc.sh`. + + clearvoice-wrapper.py [model_name] + +Default model: MossFormer2_SE_48K (48 kHz speech enhancement). Pass MossFormer2_SR_48K for +super-resolution / bandwidth extension. Weights download on first run to checkpoints/. +ClearerVoice code and MossFormer2_SE_48K weights are Apache-2.0; confirm any other model +weight license before shipping. +""" +import sys +from pathlib import Path + +from clearvoice import ClearVoice + +def main(): + if len(sys.argv) < 3: + sys.exit("usage: clearvoice-wrapper.py [model_name]") + inp, outp = sys.argv[1], sys.argv[2] + model = sys.argv[3] if len(sys.argv) > 3 else "MossFormer2_SE_48K" + task = "speech_super_resolution" if "_SR_" in model else "speech_enhancement" + out = Path(outp) + if out.exists() and out.is_dir(): + sys.exit(f"output path is a directory, expected file: {outp}") + out.parent.mkdir(parents=True, exist_ok=True) + + cv = ClearVoice(task=task, model_names=[model]) + result = cv(input_path=inp, online_write=False) + cv.write(result, output_path=outp) + if not out.is_file(): + sys.exit(f"clearvoice did not create output file: {outp}") + +if __name__ == "__main__": + main() diff --git a/scripts/hq-model-poc.sh b/scripts/hq-model-poc.sh index 6c847e2..e7ed617 100755 --- a/scripts/hq-model-poc.sh +++ b/scripts/hq-model-poc.sh @@ -11,7 +11,7 @@ # # External command env vars (use {in}/{out} placeholders; must accept/produce a 48k mono WAV): # RESEMBLE_CMD="resemble-enhance-file {in} {out}" -# CLEARERVOICE_CMD="clearvoice-sr {in} {out}" +# CLEARERVOICE_CMD=".clearvoice.venv/bin/python scripts/clearvoice-wrapper.py {in} {out}" set -euo pipefail ROOT="$(cd "$(dirname "$0")/.." && pwd)"; cd "$ROOT" CORPUS="$ROOT/test/corpus"; OUT="$CORPUS/hq-poc"; REPORT="$CORPUS/hq-poc-report.csv" @@ -32,8 +32,11 @@ chmod +x resemble-enhance-file export RESEMBLE_CMD="$PWD/resemble-enhance-file {in} {out}" # ── ClearerVoice-Studio (Apache-2.0; FRCRN denoise / MossFormer2 / 16k→48k SR) ────── -git clone https://github.com/modelscope/ClearerVoice-Studio # 가중치는 ModelScope, 라이선스 확인 -# clearvoice Python API를 file-in/out 래퍼로 감싸 CLEARERVOICE_CMD 로 export (repo 예제 참고) +python3 -m venv .clearvoice.venv && . .clearvoice.venv/bin/activate +pip install --upgrade pip setuptools wheel +pip install clearvoice==0.1.2 +# MossFormer2_SE_48K 가중치는 최초 실행 시 checkpoints/MossFormer2_SE_48K 로 다운로드된다. +export CLEARERVOICE_CMD="$PWD/.clearvoice.venv/bin/python $PWD/scripts/clearvoice-wrapper.py {in} {out}" SETUP exit 0 fi