diff --git a/AGENTS.md b/AGENTS.md
new file mode 100644
index 0000000..e8a9ffb
--- /dev/null
+++ b/AGENTS.md
@@ -0,0 +1,96 @@
+---
+written_by: ai
+---
+
+# AGENTS.md
+
+This file governs work in this repository.
+
+## Read Order
+
+For non-trivial CrawlBar work, start with:
+
+1. `README.md`
+2. `docs/control-protocol.md`
+3. `docs/quality-rubric.md`
+4. `docs/ui-rules.md`
+
+## CrawlBar Boundary
+
+CrawlBar is a native macOS menu/settings control plane for local source
+crawlers. It discovers crawler manifests, maps crawler status/actions to UI and
+CLI controls, runs configured crawler commands, and stores local CrawlBar
+configuration.
+
+CrawlBar should stay on the crawler control boundary. Source crawlers own source
+archives, auth/session handling, parsing, search/open/evidence, raw schemas,
+and source-specific privacy policy. Avoid adding higher-level synthesis,
+cross-source interpretation, dashboards, or background daemon behavior unless a
+maintainer explicitly accepts that product direction.
+
+## Engineering Review
+
+Use `docs/quality-rubric.md` for architecture, API, macOS behavior, proof,
+privacy, and complexity review. Use `docs/ui-rules.md` for SwiftUI composition
+and shared UI primitives.
+
+For substantial refactors, gather a baseline before changing code:
+
+```sh
+Scripts/quality_baseline.sh
+```
+
+Treat the baseline as evidence, not as a score to game. A smaller file count,
+LOC count, or type count is useful only when the change also reduces concepts,
+API burden, settings clutter, or change amplification.
+
+Review findings are most useful when they include:
+
+```text
+axis:
+severity:
+evidence:
+recommended_fix:
+proof_needed:
+```
+
+## Engineering Preferences
+
+Prefer boring code with one obvious place for each responsibility.
+
+- Keep new Swift files under roughly 400 LOC.
+- Avoid grab-bag files with many unrelated top-level types.
+- Keep `CrawlBarCore` free of AppKit and SwiftUI.
+- Preserve shipped SwiftPM products and public APIs unless a maintainer accepts
+ a breaking change.
+- Keep AppKit escapes narrow and local to app/window/menu boundaries.
+- Treat new settings, modes, scripts, feature flags, and adapters as design
+ debt unless there is repeated evidence they are needed.
+- Treat removal of user-visible functionality as a product decision, not a
+ metrics cleanup.
+
+## Proof
+
+For changes touching app launch, packaging, menu behavior, settings behavior,
+command execution, status mapping, or public contracts, include current proof.
+Use the smallest useful subset:
+
+```sh
+swift build
+swift run crawlbar-selftest
+Scripts/quality_baseline.sh
+Scripts/package_app.sh
+codesign --verify --deep --strict --verbose=2 dist/CrawlBar.app
+dist/CrawlBar.app/Contents/Helpers/crawlbar config validate
+```
+
+When UI behavior matters, prefer packaged-app proof plus an AX dump or
+screenshot. Avoid claiming native macOS behavior that was only tested by
+launching the raw SwiftPM executable.
+
+## Privacy
+
+Keep proof and docs to paths, counts, status, capabilities, and redacted command
+output. Do not include raw messages, contacts, phone numbers, message bodies,
+mail bodies, GPS coordinates, tokens, private endpoints, or account-specific
+content unless a maintainer explicitly asks for it and the task requires it.
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index 8ea05c4..ef581d3 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -1,3 +1,7 @@
+---
+written_by: ai
+---
+
# Contributing
Keep changes small and prove the touched surface.
@@ -14,7 +18,9 @@ Use `swift run crawlbarctl status --app all --json` only when you expect local c
## Manifest Changes
-Prefer manifest metadata over hard-coded UI branches. Built-in manifests belong in `Sources/CrawlBarCore/BuiltInCrawlApps.swift`; app-specific parsing belongs in `StatusMapper.swift`.
+Prefer manifest metadata over hard-coded UI branches. Built-in manifests belong
+under `Sources/CrawlBarCore/BuiltInApps/`; app-specific status parsing belongs
+in the narrow `Sources/CrawlBarCore/StatusMapper*.swift` adapters.
External manifest examples should use fake paths and no real tokens.
diff --git a/Scripts/package_app.sh b/Scripts/package_app.sh
index 35ebcf6..4aa8cfd 100755
--- a/Scripts/package_app.sh
+++ b/Scripts/package_app.sh
@@ -20,8 +20,8 @@ cp ".build/release/CrawlBar" "$MACOS_DIR/CrawlBar"
cp ".build/release/crawlbarctl" "$HELPERS_DIR/crawlbar"
RESOURCE_BUNDLE=".build/release/CrawlBar_CrawlBar.bundle"
if [ -d "$RESOURCE_BUNDLE" ]; then
- cp -R "$RESOURCE_BUNDLE" "$APP_DIR/CrawlBar_CrawlBar.bundle"
- if ! find "$APP_DIR/CrawlBar_CrawlBar.bundle" -type f -print -quit | grep -q .; then
+ cp -R "$RESOURCE_BUNDLE" "$RESOURCES_DIR/CrawlBar_CrawlBar.bundle"
+ if ! find "$RESOURCES_DIR/CrawlBar_CrawlBar.bundle" -type f -print -quit | grep -q .; then
echo "SwiftPM resource bundle is empty: $RESOURCE_BUNDLE" >&2
exit 1
fi
@@ -50,10 +50,18 @@ cat > "$CONTENTS_DIR/Info.plist" <<'PLIST'
0.2.2
CFBundleVersion
1
+ LSMinimumSystemVersion
+ 14.0
LSUIElement
+ NSPrincipalClass
+ NSApplication
PLIST
+if command -v codesign >/dev/null 2>&1; then
+ codesign --force --deep --sign - "$APP_DIR" >/dev/null
+fi
+
echo "$APP_DIR"
diff --git a/Scripts/quality_baseline.sh b/Scripts/quality_baseline.sh
new file mode 100755
index 0000000..d09df74
--- /dev/null
+++ b/Scripts/quality_baseline.sh
@@ -0,0 +1,147 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+# Print the small set of review metrics used by docs/quality-rubric.md.
+# This script deliberately does not pass or fail a build. It gives reviewers
+# handles for complexity: file size, type clustering, public API surface,
+# platform/effect ownership, settings clutter, and possible one-off types.
+
+ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+cd "$ROOT_DIR"
+
+section() {
+ local title="$1"
+ local why="$2"
+
+ echo
+ echo "== ${title} =="
+ echo "why=${why}"
+}
+
+echo "== CrawlBar Quality Baseline =="
+echo "why=stamp the exact repo state so before/after comparisons are evidence-based"
+date "+generated_at=%Y-%m-%dT%H:%M:%S%z"
+echo "git_status=$(git status --short | wc -l | tr -d ' ') dirty entries"
+find Sources -name '*.swift' -print0 \
+ | xargs -0 wc -l \
+ | awk '$2 != "total" { files += 1; loc += $1 } END { print "swift_files=" files; print "swift_loc=" loc }'
+
+section "Largest Swift Files" \
+ "large files are expensive to scan; this points reviewers to the biggest reading costs"
+find Sources -name '*.swift' -print0 \
+ | xargs -0 wc -l \
+ | awk '$2 != "total" { print }' \
+ | sort -nr \
+ | head -30
+
+section "Files Over 400 Lines" \
+ "the repo standard says new files should stay under roughly 400 LOC"
+find Sources -name '*.swift' -print0 \
+ | xargs -0 wc -l \
+ | awk '$2 != "total" && $1 > 400 { print }' \
+ | sort -nr
+
+section "Top-Level Type Counts" \
+ "many types in one file usually means a grab bag or hidden design system"
+find Sources -name '*.swift' -print0 \
+ | xargs -0 awk '
+ FNR == 1 {
+ if (file != "") print count, file
+ file = FILENAME
+ count = 0
+ }
+ /^(public |package |internal |private )?(struct|class|enum|actor|protocol) / { count++ }
+ END { if (file != "") print count, file }
+ ' \
+ | sort -nr \
+ | head -30
+
+section "CrawlBarCore Interface Surface" \
+ "CrawlBarCore is shipped as a library; these counts show review surface, not complete symbol compatibility"
+products="$(
+ swift package describe \
+ | awk '
+ /^Products:/ { in_products = 1; next }
+ /^Targets:/ { in_products = 0 }
+ in_products && /^ Name:/ { name = $2; next }
+ in_products && /^ Library:/ { print name ":library"; next }
+ in_products && /^ Executable:/ { print name ":executable"; next }
+ ' \
+ | paste -sd, -
+)"
+echo "products=${products}"
+(rg -n '^public ' Sources/CrawlBarCore || true) | wc -l | awk '{ print "raw_public_lines=" $1 }'
+(rg -n '^package ' Sources/CrawlBarCore || true) | wc -l | awk '{ print "raw_package_lines=" $1 }'
+echo "note=public extension members are public API but are not counted as raw public lines; use Swift symbol graphs for compatibility checks"
+echo "-- public --"
+rg -n '^public ' Sources/CrawlBarCore || true
+echo "-- package --"
+rg -n '^package ' Sources/CrawlBarCore | sed -n '1,140p' || true
+
+section "Forbidden Core UI Imports" \
+ "Core should stay UI-free; AppKit/SwiftUI belongs in the app target"
+rg -n 'import (AppKit|SwiftUI)' Sources/CrawlBarCore || true
+
+section "Production Effect/Platform References" \
+ "process, filesystem, AppKit, UserDefaults, and task ownership should sit in intentional boundaries"
+rg -n 'Process\(|FileManager\.|NSWorkspace|NSApp|NSWindow|NSMenu|NSStatus|UserDefaults|Task\s*\{' Sources/CrawlBar Sources/CrawlBarCore Sources/CrawlBarCLI Sources/CrawlBarSelfTest \
+ | rg -v '^Sources/CrawlBarSelfTest/' \
+ | sed -n '1,200p'
+
+section "SelfTest Effect/Platform References" \
+ "test harnesses are allowed more effects, but the count shows how broad the proof surface is"
+rg -n 'Process\(|FileManager\.|NSWorkspace|NSApp|NSWindow|NSMenu|NSStatus|UserDefaults|Task\s*\{' Sources/CrawlBarSelfTest \
+ | wc -l \
+ | awk '{ print "references=" $1 }'
+
+section "UI Candidate Types By Folder" \
+ "shows where SwiftUI/AppKit concepts cluster so UI decomposition does not create one-off primitives everywhere"
+rg -n '^(struct|class|enum|protocol) .*(: .*View|View\b|Window|Menu|Sidebar|Panel|Row|Header|Section|Controls|Icon|Status|Settings)' Sources/CrawlBar \
+ | awk -F: '{ print $1 }' \
+ | awk -F/ '{ print $1 "/" $2 "/" $3 }' \
+ | sort \
+ | uniq -c \
+ | sort -nr
+
+section "Settings Surface Count" \
+ "counts user-facing controls; high counts mean the simple menubar app may be carrying too many knobs"
+rg -n 'CrawlBarPanel|CrawlBarSwitchRow|CrawlBarControlRow|Button\s*\{|TextField|Picker' Sources/CrawlBar/Settings \
+ | awk '
+ /CrawlBarPanel/ { panels++ }
+ /CrawlBarSwitchRow/ { switches++ }
+ /CrawlBarControlRow/ { rows++ }
+ /Button[[:space:]]*\{/ { buttons++ }
+ /TextField/ { fields++ }
+ /Picker/ { pickers++ }
+ END {
+ print "panels=" panels + 0
+ print "switches=" switches + 0
+ print "control_rows=" rows + 0
+ print "buttons=" buttons + 0
+ print "text_fields=" fields + 0
+ print "pickers=" pickers + 0
+ }'
+
+section "Low-Reference Type Candidates" \
+ "types with 1-3 textual references are not automatically dead; they are review handles for unused code, one-off wrappers, and speculative helpers"
+find Sources -name '*.swift' -print0 \
+ | while IFS= read -r -d '' path; do
+ awk -v path="$path" '
+ {
+ line = $0
+ sub(/^[[:space:]]*/, "", line)
+ sub(/^(public|package|private|final|internal)[[:space:]]+/, "", line)
+ if (line ~ /^(struct|class|enum|protocol)[[:space:]]+[A-Za-z_][A-Za-z0-9_]*/) {
+ split(line, parts, /[[:space:]]+/)
+ sub(/[:<{].*/, "", parts[2])
+ print path "\t" parts[2]
+ }
+ }
+ ' "$path"
+ done \
+ | while IFS=$'\t' read -r path name; do
+ reference_count="$(rg -w -- "$name" Sources Package.swift Scripts 2>/dev/null | wc -l | tr -d ' ')"
+ if (( reference_count <= 3 )); then
+ printf "%3d %-45s %s\n" "$reference_count" "$name" "$path"
+ fi
+ done
diff --git a/Sources/CrawlBar/App/AppDelegate.swift b/Sources/CrawlBar/App/AppDelegate.swift
new file mode 100644
index 0000000..9a31607
--- /dev/null
+++ b/Sources/CrawlBar/App/AppDelegate.swift
@@ -0,0 +1,257 @@
+import AppKit
+import CrawlBarCore
+
+@MainActor
+final class CrawlBarAppDelegate: NSObject, NSApplicationDelegate, NSMenuDelegate {
+ private var statusItem: NSStatusItem?
+ private var refreshTimer: Timer?
+ private var refreshAnimationTimer: Timer?
+ private var refreshAnimationFrame = 0
+ private var pendingMenuReloadTask: Task?
+ private var isMenuOpen = false
+ private var menuNeedsReloadAfterClose = false
+ private let settingsWindowController = CrawlBarSettingsWindowController()
+ private let menuBuilder = CrawlBarStatusMenuBuilder()
+ private let model = CrawlBarMenuModel()
+
+ func applicationDidFinishLaunching(_ notification: Notification) {
+ CrawlBarLog.app.notice("CrawlBar launched")
+ self.configureMainMenu()
+ NotificationCenter.default.addObserver(
+ self,
+ selector: #selector(Self.statusesDidChange(_:)),
+ name: .crawlBarStatusesDidChange,
+ object: nil)
+ NotificationCenter.default.addObserver(
+ self,
+ selector: #selector(Self.configDidChange(_:)),
+ name: .crawlBarConfigDidChange,
+ object: nil)
+ self.settingsWindowController.onClose = { [weak self] in
+ self?.hideFromApplicationSwitcher()
+ }
+ if let appIcon = CrawlBarIconFactory.appIconImage() {
+ NSApplication.shared.applicationIconImage = appIcon
+ }
+ let statusItem = NSStatusBar.system.statusItem(withLength: NSStatusItem.variableLength)
+ statusItem.button?.imagePosition = .imageLeading
+ self.statusItem = statusItem
+ self.updateStatusButtonImage()
+ self.reloadMenu()
+ self.model.refreshAll { [weak self] in
+ self?.scheduleMenuReload()
+ }
+ self.scheduleRefreshTimer()
+ Task { @MainActor [weak self] in
+ await Task.yield()
+ self?.hideFromApplicationSwitcher()
+ }
+ }
+
+ private func configureMainMenu() {
+ let mainMenu = NSMenu()
+ let appMenuItem = NSMenuItem()
+ let appMenu = NSMenu(title: "CrawlBar")
+
+ let settingsItem = NSMenuItem(
+ title: "Settings...",
+ action: #selector(Self.showSettings(_:)),
+ keyEquivalent: ",")
+ settingsItem.target = self
+ appMenu.addItem(settingsItem)
+ appMenu.addItem(.separator())
+
+ let quitItem = NSMenuItem(
+ title: "Quit CrawlBar",
+ action: #selector(Self.quit(_:)),
+ keyEquivalent: "q")
+ quitItem.target = self
+ appMenu.addItem(quitItem)
+
+ appMenuItem.submenu = appMenu
+ mainMenu.addItem(appMenuItem)
+ NSApplication.shared.mainMenu = mainMenu
+ }
+
+ func applicationWillTerminate(_ notification: Notification) {
+ CrawlBarLog.app.notice("CrawlBar terminated")
+ self.pendingMenuReloadTask?.cancel()
+ NotificationCenter.default.removeObserver(self)
+ }
+
+ private func reloadMenu() {
+ self.pendingMenuReloadTask?.cancel()
+ self.pendingMenuReloadTask = nil
+ let startedAt = CFAbsoluteTimeGetCurrent()
+ let menu = self.statusItem?.menu ?? NSMenu()
+ menu.delegate = self
+ self.menuBuilder.rebuildMenu(
+ menu,
+ model: self.model,
+ target: self,
+ selectors: Self.menuActionSelectors,
+ openSettings: { [weak self] appID in
+ self?.openSettings(appID: appID)
+ })
+ self.statusItem?.menu = menu
+ if let button = self.statusItem?.button {
+ button.target = nil
+ button.action = nil
+ button.isEnabled = true
+ }
+ self.syncRefreshAnimation()
+ let elapsedMilliseconds = (CFAbsoluteTimeGetCurrent() - startedAt) * 1000
+ CrawlBarLog.app.debug("Reloaded menu in \(elapsedMilliseconds, privacy: .public)ms")
+ }
+
+ private func scheduleMenuReload() {
+ guard !self.isMenuOpen else {
+ self.menuNeedsReloadAfterClose = true
+ return
+ }
+ self.pendingMenuReloadTask?.cancel()
+ self.pendingMenuReloadTask = Task { @MainActor [weak self] in
+ try? await Task.sleep(nanoseconds: 80_000_000)
+ guard !Task.isCancelled else { return }
+ self?.reloadMenu()
+ }
+ }
+
+ private func scheduleRefreshTimer() {
+ self.refreshTimer?.invalidate()
+ self.refreshTimer = Timer.scheduledTimer(withTimeInterval: 60, repeats: true) { [weak self] _ in
+ Task { @MainActor in
+ self?.model.runDueAutoSync {
+ self?.scheduleMenuReload()
+ }
+ }
+ }
+ }
+
+ private func syncRefreshAnimation() {
+ self.updateStatusButtonImage()
+ if self.model.isRefreshing {
+ guard self.refreshAnimationTimer == nil else { return }
+ self.refreshAnimationTimer = Timer.scheduledTimer(withTimeInterval: 0.16, repeats: true) { [weak self] _ in
+ Task { @MainActor in
+ self?.advanceRefreshAnimation()
+ }
+ }
+ } else {
+ self.refreshAnimationTimer?.invalidate()
+ self.refreshAnimationTimer = nil
+ self.refreshAnimationFrame = 0
+ self.updateStatusButtonImage()
+ }
+ }
+
+ private func advanceRefreshAnimation() {
+ guard self.model.isRefreshing else {
+ self.syncRefreshAnimation()
+ return
+ }
+ self.refreshAnimationFrame = (self.refreshAnimationFrame + 1) % 8
+ self.updateStatusButtonImage()
+ }
+
+ private func updateStatusButtonImage() {
+ let rotation = self.model.isRefreshing ? CGFloat(self.refreshAnimationFrame) * 45 : 0
+ guard let button = self.statusItem?.button else { return }
+ button.image = CrawlBarIconFactory.menuBarImage(rotationDegrees: rotation)
+ button.toolTip = "CrawlBar"
+ button.setAccessibilityLabel("CrawlBar")
+ button.setAccessibilityTitle("CrawlBar")
+ button.setAccessibilityHelp("Open CrawlBar")
+ }
+
+ @objc private func refreshAll(_ sender: Any?) {
+ self.model.refreshAll { [weak self] in
+ self?.scheduleMenuReload()
+ }
+ self.reloadMenu()
+ }
+
+ @objc private func showSettings(_ sender: Any?) {
+ self.openSettings(appID: nil)
+ }
+
+ @objc private func showSettingsForAppMenuItem(_ sender: NSMenuItem) {
+ self.openSettings(appID: sender.representedObject as? CrawlAppID)
+ }
+
+ private func openSettings(appID: CrawlAppID?) {
+ self.showInApplicationSwitcher()
+ self.statusItem?.menu?.cancelTracking()
+ Task { @MainActor [weak self] in
+ // Let AppKit finish status-menu tracking before ordering the settings window.
+ try? await Task.sleep(nanoseconds: 150_000_000)
+ guard let self else { return }
+ CrawlBarLog.app.debug("Opening settings")
+ self.settingsWindowController.show(appID: appID)
+ }
+ }
+
+ private func showInApplicationSwitcher() {
+ // Settings needs regular activation so macOS treats the window like a normal app window for focus and accessibility.
+ NSApplication.shared.setActivationPolicy(.regular)
+ NSApplication.shared.unhide(nil)
+ NSApplication.shared.activate(ignoringOtherApps: true)
+ }
+
+ private func hideFromApplicationSwitcher() {
+ NSApplication.shared.setActivationPolicy(.accessory)
+ }
+
+ @objc private func openLogs(_ sender: Any?) {
+ self.statusItem?.menu?.cancelTracking()
+ CrawlBarLog.app.debug("Opening action logs folder")
+ NSWorkspace.shared.open(CrawlActionLogStore.defaultDirectory())
+ }
+
+ @objc private func quit(_ sender: Any?) {
+ self.statusItem?.menu?.cancelTracking()
+ CrawlBarLog.app.notice("Quit requested")
+ NSApplication.shared.terminate(nil)
+ }
+
+ func menuWillOpen(_ menu: NSMenu) {
+ self.isMenuOpen = true
+ if self.pendingMenuReloadTask != nil {
+ self.reloadMenu()
+ self.isMenuOpen = true
+ }
+ }
+
+ func menuDidClose(_ menu: NSMenu) {
+ self.isMenuOpen = false
+ self.menuBuilder.clearHighlights(in: menu)
+ if self.menuNeedsReloadAfterClose {
+ self.menuNeedsReloadAfterClose = false
+ self.reloadMenu()
+ }
+ }
+
+ func menu(_ menu: NSMenu, willHighlight item: NSMenuItem?) {
+ for menuItem in menu.items {
+ guard let view = menuItem.view as? CrawlBarMenuItemHighlighting else { continue }
+ view.setHighlighted(menuItem == item && menuItem.isEnabled)
+ }
+ }
+
+ @objc private func statusesDidChange(_ notification: Notification) {
+ self.scheduleMenuReload()
+ }
+
+ @objc private func configDidChange(_ notification: Notification) {
+ self.model.reloadInstallations()
+ self.scheduleRefreshTimer()
+ self.reloadMenu()
+ }
+
+ private static let menuActionSelectors = CrawlBarStatusMenuActionSelectors(
+ showSettings: #selector(Self.showSettings(_:)),
+ showSettingsForApp: #selector(Self.showSettingsForAppMenuItem(_:)),
+ refreshAll: #selector(Self.refreshAll(_:)),
+ openLogs: #selector(Self.openLogs(_:)),
+ quit: #selector(Self.quit(_:)))
+}
diff --git a/Sources/CrawlBar/App/CrawlBarApp.swift b/Sources/CrawlBar/App/CrawlBarApp.swift
new file mode 100644
index 0000000..7d3f647
--- /dev/null
+++ b/Sources/CrawlBar/App/CrawlBarApp.swift
@@ -0,0 +1,14 @@
+import AppKit
+
+@main
+@MainActor
+enum CrawlBarApp {
+ static func main() {
+ let app = NSApplication.shared
+ let delegate = CrawlBarAppDelegate()
+ app.delegate = delegate
+ // Launch as a menu-bar app; Settings switches to regular activation while its window is open.
+ app.setActivationPolicy(.accessory)
+ app.run()
+ }
+}
diff --git a/Sources/CrawlBar/CrawlBarNotifications.swift b/Sources/CrawlBar/App/CrawlBarNotifications.swift
similarity index 100%
rename from Sources/CrawlBar/CrawlBarNotifications.swift
rename to Sources/CrawlBar/App/CrawlBarNotifications.swift
diff --git a/Sources/CrawlBar/App/NativeAppLocator.swift b/Sources/CrawlBar/App/NativeAppLocator.swift
new file mode 100644
index 0000000..67b535a
--- /dev/null
+++ b/Sources/CrawlBar/App/NativeAppLocator.swift
@@ -0,0 +1,17 @@
+import AppKit
+
+@MainActor
+enum CrawlBarNativeAppLocator {
+ private static var urlsByBundleIdentifier: [String: URL] = [:]
+
+ static func url(for bundleIdentifier: String) -> URL? {
+ if let cached = Self.urlsByBundleIdentifier[bundleIdentifier] {
+ return cached
+ }
+ guard let url = NSWorkspace.shared.urlForApplication(withBundleIdentifier: bundleIdentifier) else {
+ return nil
+ }
+ Self.urlsByBundleIdentifier[bundleIdentifier] = url
+ return url
+ }
+}
diff --git a/Sources/CrawlBar/Branding/BrandMetadata.swift b/Sources/CrawlBar/Branding/BrandMetadata.swift
new file mode 100644
index 0000000..f350987
--- /dev/null
+++ b/Sources/CrawlBar/Branding/BrandMetadata.swift
@@ -0,0 +1,53 @@
+import AppKit
+import CrawlBarCore
+
+enum CrawlBarBrandPalette {
+ static func accent(for appID: CrawlAppID, manifest: CrawlAppManifest?) -> NSColor {
+ switch appID.rawValue {
+ case "gitcrawl":
+ NSColor(calibratedWhite: 0.08, alpha: 1)
+ case "slacrawl":
+ NSColor(calibratedRed: 0.25, green: 0.16, blue: 0.32, alpha: 1)
+ case "discrawl":
+ NSColor(calibratedRed: 0.35, green: 0.40, blue: 0.95, alpha: 1)
+ case "telecrawl":
+ NSColor(calibratedRed: 0.13, green: 0.62, blue: 0.85, alpha: 1)
+ case "notcrawl":
+ NSColor(calibratedWhite: 0.08, alpha: 1)
+ case "gogcli":
+ NSColor(calibratedRed: 0.26, green: 0.52, blue: 0.96, alpha: 1)
+ case "wacli":
+ NSColor(calibratedRed: 0.15, green: 0.83, blue: 0.40, alpha: 1)
+ case "birdclaw":
+ NSColor(calibratedWhite: 0.02, alpha: 1)
+ case "graincrawl":
+ NSColor(calibratedRed: 0.83, green: 0.63, blue: 0.09, alpha: 1)
+ default:
+ NSColor(crawlBarHex: manifest?.branding.accentColor ?? "#6E6E73")
+ }
+ }
+}
+
+enum CrawlBarCrawlerTitle {
+ static func text(for appID: CrawlAppID, manifest: CrawlAppManifest?) -> String {
+ let source = manifest?.displayName.nilIfBlank ?? appID.rawValue
+ guard let binary = manifest?.binary.name.nilIfBlank, binary != source else {
+ return source
+ }
+ return "\(source) (\(binary))"
+ }
+}
+
+private extension NSColor {
+ convenience init(crawlBarHex hex: String) {
+ let trimmed = hex.trimmingCharacters(in: CharacterSet(charactersIn: "#"))
+ let scanner = Scanner(string: trimmed)
+ var value: UInt64 = 0
+ scanner.scanHexInt64(&value)
+ self.init(
+ calibratedRed: Double((value >> 16) & 0xff) / 255,
+ green: Double((value >> 8) & 0xff) / 255,
+ blue: Double(value & 0xff) / 255,
+ alpha: 1)
+ }
+}
diff --git a/Sources/CrawlBar/Branding/CrawlBarBrandIcon.swift b/Sources/CrawlBar/Branding/CrawlBarBrandIcon.swift
new file mode 100644
index 0000000..8526e8b
--- /dev/null
+++ b/Sources/CrawlBar/Branding/CrawlBarBrandIcon.swift
@@ -0,0 +1,17 @@
+import CrawlBarCore
+import SwiftUI
+
+struct CrawlBarBrandIcon: View {
+ let manifest: CrawlAppManifest?
+ let appID: CrawlAppID
+
+ var body: some View {
+ Image(nsImage: CrawlBarIconFactory.image(
+ for: self.appID,
+ manifest: self.manifest,
+ size: 64))
+ .resizable()
+ .interpolation(.high)
+ .aspectRatio(1, contentMode: .fit)
+ }
+}
diff --git a/Sources/CrawlBar/BrandIcons.swift b/Sources/CrawlBar/Branding/CrawlBarIconDrawing.swift
similarity index 50%
rename from Sources/CrawlBar/BrandIcons.swift
rename to Sources/CrawlBar/Branding/CrawlBarIconDrawing.swift
index e7499d3..7540c8b 100644
--- a/Sources/CrawlBar/BrandIcons.swift
+++ b/Sources/CrawlBar/Branding/CrawlBarIconDrawing.swift
@@ -1,285 +1,8 @@
import AppKit
import CrawlBarCore
-import SwiftUI
-enum CrawlBarBrandPalette {
- static func accent(for appID: CrawlAppID, manifest: CrawlAppManifest?) -> NSColor {
- switch appID.rawValue {
- case "gitcrawl":
- NSColor(calibratedWhite: 0.08, alpha: 1)
- case "slacrawl":
- NSColor(calibratedRed: 0.25, green: 0.16, blue: 0.32, alpha: 1)
- case "discrawl":
- NSColor(calibratedRed: 0.35, green: 0.40, blue: 0.95, alpha: 1)
- case "telecrawl":
- NSColor(calibratedRed: 0.13, green: 0.62, blue: 0.85, alpha: 1)
- case "notcrawl":
- NSColor(calibratedWhite: 0.08, alpha: 1)
- case "gogcli":
- NSColor(calibratedRed: 0.26, green: 0.52, blue: 0.96, alpha: 1)
- case "wacli":
- NSColor(calibratedRed: 0.15, green: 0.83, blue: 0.40, alpha: 1)
- case "birdclaw":
- NSColor(calibratedWhite: 0.02, alpha: 1)
- case "graincrawl":
- NSColor(calibratedRed: 0.83, green: 0.63, blue: 0.09, alpha: 1)
- default:
- NSColor(hex: manifest?.branding.accentColor ?? "#6E6E73")
- }
- }
-}
-
-enum CrawlBarCrawlerTitle {
- static func text(for appID: CrawlAppID, manifest: CrawlAppManifest?) -> String {
- let source = manifest?.displayName.nilIfBlank ?? appID.rawValue
- guard let binary = manifest?.binary.name.nilIfBlank, binary != source else {
- return source
- }
- return "\(source) (\(binary))"
- }
-}
-
-@MainActor
-enum CrawlBarNativeAppLocator {
- private static var urlsByBundleIdentifier: [String: URL] = [:]
-
- static func url(for bundleIdentifier: String) -> URL? {
- if let cached = Self.urlsByBundleIdentifier[bundleIdentifier] {
- return cached
- }
- guard let url = NSWorkspace.shared.urlForApplication(withBundleIdentifier: bundleIdentifier) else {
- return nil
- }
- Self.urlsByBundleIdentifier[bundleIdentifier] = url
- return url
- }
-}
-
-@MainActor
-enum CrawlBarIconFactory {
- private static var imageCache: [String: NSImage] = [:]
- private static var menuBarImageCache: [String: NSImage] = [:]
- private static var statusDotImageCache: [String: NSImage] = [:]
-
- static func image(for appID: CrawlAppID, manifest: CrawlAppManifest?, size: CGFloat = 32) -> NSImage {
- let cacheKey = [
- "app",
- appID.rawValue,
- "\(Self.cacheSizeKey(for: size))",
- manifest?.branding.iconPath?.nilIfBlank ?? "",
- manifest?.branding.bundleIdentifier?.nilIfBlank ?? "",
- manifest?.branding.accentColor.nilIfBlank ?? "",
- ].joined(separator: "|")
- if let cached = Self.imageCache[cacheKey] {
- return cached
- }
- if let image = Self.brandedImage(for: appID, manifest: manifest, size: size) {
- Self.imageCache[cacheKey] = image
- return image
- }
- let image = NSImage(size: NSSize(width: size, height: size))
- image.lockFocus()
- Self.drawTile(appID: appID, manifest: manifest, rect: NSRect(x: 0, y: 0, width: size, height: size))
- image.unlockFocus()
- image.isTemplate = false
- Self.imageCache[cacheKey] = image
- return image
- }
-
- static func menuBarImage(size: CGFloat = 18, rotationDegrees: CGFloat = 0) -> NSImage {
- let cacheKey = "\(Self.cacheSizeKey(for: size))|\(Int(rotationDegrees.rounded()))"
- if let cached = Self.menuBarImageCache[cacheKey] {
- return cached
- }
- let image = NSImage(size: NSSize(width: size, height: size))
- image.lockFocus()
- if rotationDegrees != 0 {
- let transform = NSAffineTransform()
- transform.translateX(by: size / 2, yBy: size / 2)
- transform.rotate(byDegrees: rotationDegrees)
- transform.translateX(by: -size / 2, yBy: -size / 2)
- transform.concat()
- }
- let stroke = NSColor.labelColor
- stroke.setStroke()
- let line = NSBezierPath()
- line.lineWidth = max(1.6, size * 0.1)
- let inset = size * 0.19
- line.move(to: NSPoint(x: inset, y: size * 0.68))
- line.curve(
- to: NSPoint(x: size - inset, y: size * 0.68),
- controlPoint1: NSPoint(x: size * 0.38, y: size * 0.94),
- controlPoint2: NSPoint(x: size * 0.62, y: size * 0.94))
- line.move(to: NSPoint(x: inset, y: size * 0.32))
- line.curve(
- to: NSPoint(x: size - inset, y: size * 0.32),
- controlPoint1: NSPoint(x: size * 0.38, y: size * 0.06),
- controlPoint2: NSPoint(x: size * 0.62, y: size * 0.06))
- line.stroke()
-
- for point in [
- NSPoint(x: inset, y: size * 0.68),
- NSPoint(x: size - inset, y: size * 0.68),
- NSPoint(x: inset, y: size * 0.32),
- NSPoint(x: size - inset, y: size * 0.32),
- ] {
- let dot = NSBezierPath(ovalIn: NSRect(
- x: point.x - size * 0.105,
- y: point.y - size * 0.105,
- width: size * 0.21,
- height: size * 0.21))
- stroke.setFill()
- dot.fill()
- }
- image.unlockFocus()
- image.isTemplate = true
- Self.menuBarImageCache[cacheKey] = image
- return image
- }
-
- static func statusDotImage(for state: CrawlAppState, size: CGFloat = 12) -> NSImage {
- let cacheKey = "\(state.rawValue)|\(Self.cacheSizeKey(for: size))"
- if let cached = Self.statusDotImageCache[cacheKey] {
- return cached
- }
- let image = NSImage(size: NSSize(width: size, height: size))
- image.lockFocus()
- let rect = NSRect(x: 2, y: 2, width: size - 4, height: size - 4)
- let dot = NSBezierPath(ovalIn: rect)
- Self.statusColor(for: state).setFill()
- dot.fill()
- NSColor.separatorColor.withAlphaComponent(0.5).setStroke()
- dot.lineWidth = 0.75
- dot.stroke()
- image.unlockFocus()
- image.isTemplate = false
- Self.statusDotImageCache[cacheKey] = image
- return image
- }
-
- static func appIconImage() -> NSImage? {
- for bundle in Self.resourceBundleCandidates() {
- if let url = bundle.url(forResource: "AppIcon", withExtension: "png") {
- return NSImage(contentsOf: url)
- }
- }
- return nil
- }
-
- private static func cacheSizeKey(for size: CGFloat) -> Int {
- Int((size * 2).rounded())
- }
-
- private static func statusColor(for state: CrawlAppState) -> NSColor {
- switch state {
- case .current:
- NSColor.systemGreen
- case .stale, .syncing, .unknown:
- NSColor.systemYellow
- case .needsConfig, .needsAuth, .error:
- NSColor.systemRed
- case .disabled:
- NSColor.systemGray
- }
- }
-
- private static func brandedImage(for appID: CrawlAppID, manifest: CrawlAppManifest?, size: CGFloat) -> NSImage? {
- if let iconPath = manifest?.branding.iconPath?.nilIfBlank {
- let expandedPath = NSString(string: iconPath).expandingTildeInPath
- if let image = NSImage(contentsOfFile: expandedPath) {
- return Self.sizedImage(image, size: size)
- }
- }
- if let bundleIdentifier = manifest?.branding.bundleIdentifier?.nilIfBlank,
- let appURL = CrawlBarNativeAppLocator.url(for: bundleIdentifier)
- {
- return Self.sizedImage(NSWorkspace.shared.icon(forFile: appURL.path), size: size)
- }
- if let image = Self.bundledIcon(for: appID) {
- return Self.sizedImage(image, size: size)
- }
- return nil
- }
-
- private static func bundledIcon(for appID: CrawlAppID) -> NSImage? {
- guard let name = Self.bundledIconName(for: appID),
- let url = Self.bundledIconURL(named: name)
- else {
- return nil
- }
- return NSImage(contentsOf: url)
- }
-
- private static func bundledIconURL(named name: String) -> URL? {
- for bundle in Self.resourceBundleCandidates() {
- if let url = bundle.url(forResource: name, withExtension: "png", subdirectory: "BrandIcons")
- ?? bundle.url(forResource: name, withExtension: "png")
- {
- return url
- }
- }
- return nil
- }
-
- private static func resourceBundleCandidates() -> [Bundle] {
- let resourceBundleName = "CrawlBar_CrawlBar.bundle"
- let candidateURLs = [
- // SwiftPM executable bundles currently look beside the .app root.
- Bundle.main.bundleURL.appendingPathComponent(resourceBundleName),
- // Conventional app bundles keep resources under Contents/Resources.
- Bundle.main.resourceURL?.appendingPathComponent(resourceBundleName),
- // During local development, the executable and resource bundle share a build directory.
- Bundle.main.executableURL?.deletingLastPathComponent().appendingPathComponent(resourceBundleName),
- ].compactMap { $0 }
-
- var seen = Set()
- var bundles: [Bundle] = []
- for url in candidateURLs where seen.insert(url.path).inserted {
- if let bundle = Bundle(url: url) {
- bundles.append(bundle)
- }
- }
- bundles.append(Bundle.main)
- return bundles
- }
-
- private static func bundledIconName(for appID: CrawlAppID) -> String? {
- switch appID.rawValue {
- case "gitcrawl":
- "gitcrawl"
- case "slacrawl":
- "slacrawl"
- case "discrawl":
- "discrawl"
- case "notcrawl":
- "notcrawl"
- case "gogcli":
- "google"
- case "wacli":
- "wacli"
- case "birdclaw":
- "x"
- case "graincrawl":
- "graincrawl"
- default:
- nil
- }
- }
-
- private static func sizedImage(_ source: NSImage, size: CGFloat) -> NSImage {
- let image = NSImage(size: NSSize(width: size, height: size))
- image.lockFocus()
- source.draw(
- in: NSRect(x: 0, y: 0, width: size, height: size),
- from: .zero,
- operation: .sourceOver,
- fraction: 1)
- image.unlockFocus()
- image.isTemplate = false
- return image
- }
-
- private static func drawTile(appID: CrawlAppID, manifest: CrawlAppManifest?, rect: NSRect) {
+extension CrawlBarIconFactory {
+ static func drawTile(appID: CrawlAppID, manifest: CrawlAppManifest?, rect: NSRect) {
let radius = rect.width * 0.22
let tile = NSBezierPath(roundedRect: rect.insetBy(dx: 1, dy: 1), xRadius: radius, yRadius: radius)
switch appID.rawValue {
@@ -336,7 +59,7 @@ enum CrawlBarIconFactory {
}
}
- private static func drawTelegramGlyph(in rect: NSRect) {
+ static func drawTelegramGlyph(in rect: NSRect) {
NSColor.white.setFill()
let plane = NSBezierPath()
plane.move(to: NSPoint(x: rect.minX + rect.width * 0.18, y: rect.midY + rect.height * 0.03))
@@ -356,7 +79,7 @@ enum CrawlBarIconFactory {
fold.stroke()
}
- private static func drawGoogleGlyph(in rect: NSRect) {
+ static func drawGoogleGlyph(in rect: NSRect) {
let center = NSPoint(x: rect.midX, y: rect.midY)
let radius = rect.width * 0.27
let width = max(2.4, rect.width * 0.105)
@@ -383,7 +106,7 @@ enum CrawlBarIconFactory {
crossbar.stroke()
}
- private static func drawWhatsAppGlyph(in rect: NSRect) {
+ static func drawWhatsAppGlyph(in rect: NSRect) {
NSColor.white.setFill()
let bubbleRect = rect.insetBy(dx: rect.width * 0.18, dy: rect.height * 0.18)
let bubble = NSBezierPath(ovalIn: bubbleRect)
@@ -407,7 +130,7 @@ enum CrawlBarIconFactory {
phone.stroke()
}
- private static func drawXGlyph(in rect: NSRect) {
+ static func drawXGlyph(in rect: NSRect) {
NSColor.white.setStroke()
let path = NSBezierPath()
path.lineWidth = max(2.2, rect.width * 0.095)
@@ -419,7 +142,7 @@ enum CrawlBarIconFactory {
path.stroke()
}
- private static func drawGranolaGlyph(in rect: NSRect) {
+ static func drawGranolaGlyph(in rect: NSRect) {
let color = NSColor(calibratedRed: 0.55, green: 0.30, blue: 0.18, alpha: 1)
let paragraph = NSMutableParagraphStyle()
paragraph.alignment = .center
@@ -443,7 +166,7 @@ enum CrawlBarIconFactory {
}
}
- private static func drawNotionN(in rect: NSRect) {
+ static func drawNotionN(in rect: NSRect) {
let paragraph = NSMutableParagraphStyle()
paragraph.alignment = .center
let fontSize = rect.width * 0.64
@@ -457,7 +180,7 @@ enum CrawlBarIconFactory {
withAttributes: attributes)
}
- private static func drawGitGlyph(in rect: NSRect, color: NSColor) {
+ static func drawGitGlyph(in rect: NSRect, color: NSColor) {
color.setStroke()
color.setFill()
let center = NSPoint(x: rect.midX, y: rect.midY)
@@ -482,7 +205,7 @@ enum CrawlBarIconFactory {
}
}
- private static func drawSlackGlyph(in rect: NSRect) {
+ static func drawSlackGlyph(in rect: NSRect) {
let colors = [
NSColor(calibratedRed: 0.20, green: 0.73, blue: 0.61, alpha: 1),
NSColor(calibratedRed: 0.22, green: 0.53, blue: 0.92, alpha: 1),
@@ -508,7 +231,7 @@ enum CrawlBarIconFactory {
}
}
- private static func drawDiscordGlyph(in rect: NSRect, color: NSColor) {
+ static func drawDiscordGlyph(in rect: NSRect, color: NSColor) {
color.setFill()
let body = NSBezierPath(roundedRect: rect.insetBy(dx: rect.width * 0.22, dy: rect.height * 0.28), xRadius: rect.width * 0.16, yRadius: rect.width * 0.16)
body.fill()
@@ -530,7 +253,7 @@ enum CrawlBarIconFactory {
smile.stroke()
}
- private static func drawTerminalGlyph(in rect: NSRect, color: NSColor) {
+ static func drawTerminalGlyph(in rect: NSRect, color: NSColor) {
color.setStroke()
let path = NSBezierPath()
path.lineWidth = max(1.8, rect.width * 0.06)
@@ -544,32 +267,3 @@ enum CrawlBarIconFactory {
path.stroke()
}
}
-
-struct CrawlBarBrandIcon: View {
- let manifest: CrawlAppManifest?
- let appID: CrawlAppID
-
- var body: some View {
- Image(nsImage: CrawlBarIconFactory.image(
- for: self.appID,
- manifest: self.manifest,
- size: 64))
- .resizable()
- .interpolation(.high)
- .aspectRatio(1, contentMode: .fit)
- }
-}
-
-private extension NSColor {
- convenience init(hex: String) {
- let trimmed = hex.trimmingCharacters(in: CharacterSet(charactersIn: "#"))
- let scanner = Scanner(string: trimmed)
- var value: UInt64 = 0
- scanner.scanHexInt64(&value)
- self.init(
- calibratedRed: Double((value >> 16) & 0xff) / 255,
- green: Double((value >> 8) & 0xff) / 255,
- blue: Double(value & 0xff) / 255,
- alpha: 1)
- }
-}
diff --git a/Sources/CrawlBar/Branding/CrawlBarIconFactory.swift b/Sources/CrawlBar/Branding/CrawlBarIconFactory.swift
new file mode 100644
index 0000000..b37be63
--- /dev/null
+++ b/Sources/CrawlBar/Branding/CrawlBarIconFactory.swift
@@ -0,0 +1,224 @@
+import AppKit
+import CrawlBarCore
+
+@MainActor
+enum CrawlBarIconFactory {
+ private static var imageCache: [String: NSImage] = [:]
+ private static var menuBarImageCache: [String: NSImage] = [:]
+ private static var statusDotImageCache: [String: NSImage] = [:]
+
+ static func image(for appID: CrawlAppID, manifest: CrawlAppManifest?, size: CGFloat = 32) -> NSImage {
+ let cacheKey = [
+ "app",
+ appID.rawValue,
+ "\(Self.cacheSizeKey(for: size))",
+ manifest?.branding.iconPath?.nilIfBlank ?? "",
+ manifest?.branding.bundleIdentifier?.nilIfBlank ?? "",
+ manifest?.branding.accentColor.nilIfBlank ?? "",
+ ].joined(separator: "|")
+ if let cached = Self.imageCache[cacheKey] {
+ return cached
+ }
+ if let image = Self.brandedImage(for: appID, manifest: manifest, size: size) {
+ Self.imageCache[cacheKey] = image
+ return image
+ }
+ let image = NSImage(size: NSSize(width: size, height: size))
+ image.lockFocus()
+ Self.drawTile(appID: appID, manifest: manifest, rect: NSRect(x: 0, y: 0, width: size, height: size))
+ image.unlockFocus()
+ image.isTemplate = false
+ Self.imageCache[cacheKey] = image
+ return image
+ }
+
+ static func menuBarImage(size: CGFloat = 18, rotationDegrees: CGFloat = 0) -> NSImage {
+ let cacheKey = "\(Self.cacheSizeKey(for: size))|\(Int(rotationDegrees.rounded()))"
+ if let cached = Self.menuBarImageCache[cacheKey] {
+ return cached
+ }
+ let image = NSImage(size: NSSize(width: size, height: size))
+ image.lockFocus()
+ if rotationDegrees != 0 {
+ let transform = NSAffineTransform()
+ transform.translateX(by: size / 2, yBy: size / 2)
+ transform.rotate(byDegrees: rotationDegrees)
+ transform.translateX(by: -size / 2, yBy: -size / 2)
+ transform.concat()
+ }
+ let stroke = NSColor.labelColor
+ stroke.setStroke()
+ let line = NSBezierPath()
+ line.lineWidth = max(1.6, size * 0.1)
+ let inset = size * 0.19
+ line.move(to: NSPoint(x: inset, y: size * 0.68))
+ line.curve(
+ to: NSPoint(x: size - inset, y: size * 0.68),
+ controlPoint1: NSPoint(x: size * 0.38, y: size * 0.94),
+ controlPoint2: NSPoint(x: size * 0.62, y: size * 0.94))
+ line.move(to: NSPoint(x: inset, y: size * 0.32))
+ line.curve(
+ to: NSPoint(x: size - inset, y: size * 0.32),
+ controlPoint1: NSPoint(x: size * 0.38, y: size * 0.06),
+ controlPoint2: NSPoint(x: size * 0.62, y: size * 0.06))
+ line.stroke()
+
+ for point in [
+ NSPoint(x: inset, y: size * 0.68),
+ NSPoint(x: size - inset, y: size * 0.68),
+ NSPoint(x: inset, y: size * 0.32),
+ NSPoint(x: size - inset, y: size * 0.32),
+ ] {
+ let dot = NSBezierPath(ovalIn: NSRect(
+ x: point.x - size * 0.105,
+ y: point.y - size * 0.105,
+ width: size * 0.21,
+ height: size * 0.21))
+ stroke.setFill()
+ dot.fill()
+ }
+ image.unlockFocus()
+ image.isTemplate = true
+ Self.menuBarImageCache[cacheKey] = image
+ return image
+ }
+
+ static func statusDotImage(for state: CrawlAppState, size: CGFloat = 12) -> NSImage {
+ let cacheKey = "\(state.rawValue)|\(Self.cacheSizeKey(for: size))"
+ if let cached = Self.statusDotImageCache[cacheKey] {
+ return cached
+ }
+ let image = NSImage(size: NSSize(width: size, height: size))
+ image.lockFocus()
+ let rect = NSRect(x: 2, y: 2, width: size - 4, height: size - 4)
+ let dot = NSBezierPath(ovalIn: rect)
+ Self.statusColor(for: state).setFill()
+ dot.fill()
+ NSColor.separatorColor.withAlphaComponent(0.5).setStroke()
+ dot.lineWidth = 0.75
+ dot.stroke()
+ image.unlockFocus()
+ image.isTemplate = false
+ Self.statusDotImageCache[cacheKey] = image
+ return image
+ }
+
+ static func appIconImage() -> NSImage? {
+ for bundle in Self.resourceBundleCandidates() {
+ if let url = bundle.url(forResource: "AppIcon", withExtension: "png") {
+ return NSImage(contentsOf: url)
+ }
+ }
+ return nil
+ }
+
+ static func cacheSizeKey(for size: CGFloat) -> Int {
+ Int((size * 2).rounded())
+ }
+
+ static func statusColor(for state: CrawlAppState) -> NSColor {
+ switch state {
+ case .current:
+ NSColor.systemGreen
+ case .stale, .syncing, .unknown:
+ NSColor.systemYellow
+ case .needsConfig, .needsAuth, .error:
+ NSColor.systemRed
+ case .disabled:
+ NSColor.systemGray
+ }
+ }
+
+ static func brandedImage(for appID: CrawlAppID, manifest: CrawlAppManifest?, size: CGFloat) -> NSImage? {
+ if let iconPath = manifest?.branding.iconPath?.nilIfBlank {
+ let expandedPath = NSString(string: iconPath).expandingTildeInPath
+ if let image = NSImage(contentsOfFile: expandedPath) {
+ return Self.sizedImage(image, size: size)
+ }
+ }
+ if let bundleIdentifier = manifest?.branding.bundleIdentifier?.nilIfBlank,
+ let appURL = CrawlBarNativeAppLocator.url(for: bundleIdentifier)
+ {
+ return Self.sizedImage(NSWorkspace.shared.icon(forFile: appURL.path), size: size)
+ }
+ if let image = Self.bundledIcon(for: appID) {
+ return Self.sizedImage(image, size: size)
+ }
+ return nil
+ }
+
+ static func bundledIcon(for appID: CrawlAppID) -> NSImage? {
+ guard let name = Self.bundledIconName(for: appID),
+ let url = Self.bundledIconURL(named: name)
+ else {
+ return nil
+ }
+ return NSImage(contentsOf: url)
+ }
+
+ static func bundledIconURL(named name: String) -> URL? {
+ for bundle in Self.resourceBundleCandidates() {
+ if let url = bundle.url(forResource: name, withExtension: "png", subdirectory: "BrandIcons")
+ ?? bundle.url(forResource: name, withExtension: "png")
+ {
+ return url
+ }
+ }
+ return nil
+ }
+
+ static func resourceBundleCandidates() -> [Bundle] {
+ let resourceBundleName = "CrawlBar_CrawlBar.bundle"
+ let candidateURLs = [
+ Bundle.main.bundleURL.appendingPathComponent(resourceBundleName),
+ Bundle.main.resourceURL?.appendingPathComponent(resourceBundleName),
+ Bundle.main.executableURL?.deletingLastPathComponent().appendingPathComponent(resourceBundleName),
+ ].compactMap { $0 }
+
+ var seen = Set()
+ var bundles: [Bundle] = []
+ for url in candidateURLs where seen.insert(url.path).inserted {
+ if let bundle = Bundle(url: url) {
+ bundles.append(bundle)
+ }
+ }
+ bundles.append(Bundle.main)
+ return bundles
+ }
+
+ static func bundledIconName(for appID: CrawlAppID) -> String? {
+ switch appID.rawValue {
+ case "gitcrawl":
+ "gitcrawl"
+ case "slacrawl":
+ "slacrawl"
+ case "discrawl":
+ "discrawl"
+ case "notcrawl":
+ "notcrawl"
+ case "gogcli":
+ "google"
+ case "wacli":
+ "wacli"
+ case "birdclaw":
+ "x"
+ case "graincrawl":
+ "graincrawl"
+ default:
+ nil
+ }
+ }
+
+ static func sizedImage(_ source: NSImage, size: CGFloat) -> NSImage {
+ let image = NSImage(size: NSSize(width: size, height: size))
+ image.lockFocus()
+ source.draw(
+ in: NSRect(x: 0, y: 0, width: size, height: size),
+ from: .zero,
+ operation: .sourceOver,
+ fraction: 1)
+ image.unlockFocus()
+ image.isTemplate = false
+ return image
+ }
+}
diff --git a/Sources/CrawlBar/CrawlBarMenuItemHosting.swift b/Sources/CrawlBar/CrawlBarMenuItemHosting.swift
deleted file mode 100644
index 19f5f27..0000000
--- a/Sources/CrawlBar/CrawlBarMenuItemHosting.swift
+++ /dev/null
@@ -1,261 +0,0 @@
-import AppKit
-import SwiftUI
-
-private enum CrawlBarMenuMetrics {
- static let width: CGFloat = 380
- static let proposedHeight: CGFloat = 720
- static let maxHeight: CGFloat = 360
- static let fallbackHeight: CGFloat = 28
- static let selectionHorizontalInset: CGFloat = 6
- static let selectionVerticalInset: CGFloat = 2
- static let selectionCornerRadius: CGFloat = 6
- static let submenuIndicatorTrailingPadding: CGFloat = 10
-}
-
-private struct CrawlBarMenuItemHighlightedKey: EnvironmentKey {
- static let defaultValue = false
-}
-
-extension EnvironmentValues {
- var crawlBarMenuItemHighlighted: Bool {
- get { self[CrawlBarMenuItemHighlightedKey.self] }
- set { self[CrawlBarMenuItemHighlightedKey.self] = newValue }
- }
-}
-
-enum CrawlBarMenuHighlightStyle {
- static let selectionText = Color(nsColor: .selectedMenuItemTextColor)
- static let normalPrimaryText = Color(nsColor: .controlTextColor)
- static let normalSecondaryText = Color(nsColor: .secondaryLabelColor)
-
- static func primary(_ highlighted: Bool) -> Color {
- highlighted ? self.selectionText : self.normalPrimaryText
- }
-
- static func secondary(_ highlighted: Bool) -> Color {
- highlighted ? self.selectionText.opacity(0.86) : self.normalSecondaryText
- }
-
- static func error(_ highlighted: Bool) -> Color {
- highlighted ? self.selectionText : Color(nsColor: .systemRed)
- }
-
- static func selectionBackground(_ highlighted: Bool) -> Color {
- highlighted ? Color(nsColor: .selectedContentBackgroundColor) : .clear
- }
-}
-
-@MainActor
-protocol CrawlBarMenuItemMeasuring: AnyObject {
- func measuredHeight(width: CGFloat) -> CGFloat
-}
-
-@MainActor
-protocol CrawlBarMenuItemHighlighting: AnyObject {
- func setHighlighted(_ highlighted: Bool)
-}
-
-@MainActor
-final class CrawlBarMenuItemHighlightState: ObservableObject {
- @Published var isHighlighted = false
-}
-
-private struct CrawlBarMenuItemSelectionBackground: Shape {
- func path(in rect: CGRect) -> Path {
- let inset = rect.insetBy(
- dx: CrawlBarMenuMetrics.selectionHorizontalInset,
- dy: CrawlBarMenuMetrics.selectionVerticalInset)
- return RoundedRectangle(
- cornerRadius: CrawlBarMenuMetrics.selectionCornerRadius,
- style: .continuous).path(in: inset)
- }
-}
-
-private struct CrawlBarMenuItemContainerView: View {
- @ObservedObject var highlightState: CrawlBarMenuItemHighlightState
- let showsSubmenuIndicator: Bool
- @ViewBuilder var content: Content
-
- var body: some View {
- self.content
- .fixedSize(horizontal: false, vertical: true)
- .padding(.trailing, self.showsSubmenuIndicator ? CrawlBarMenuMetrics.submenuIndicatorTrailingPadding : 0)
- .frame(maxWidth: .infinity, alignment: .leading)
- .environment(\.crawlBarMenuItemHighlighted, self.highlightState.isHighlighted)
- .foregroundStyle(CrawlBarMenuHighlightStyle.primary(self.highlightState.isHighlighted))
- .background(alignment: .topLeading) {
- if self.highlightState.isHighlighted {
- CrawlBarMenuItemSelectionBackground()
- .fill(CrawlBarMenuHighlightStyle.selectionBackground(true))
- }
- }
- .overlay(alignment: .topTrailing) {
- if self.showsSubmenuIndicator {
- Image(systemName: "chevron.right")
- .font(.caption2.weight(.semibold))
- .foregroundStyle(CrawlBarMenuHighlightStyle.secondary(self.highlightState.isHighlighted))
- .padding(.top, 8)
- .padding(.trailing, CrawlBarMenuMetrics.submenuIndicatorTrailingPadding)
- }
- }
- }
-}
-
-@MainActor
-final class CrawlBarMenuItemHostingView: NSView, CrawlBarMenuItemMeasuring, CrawlBarMenuItemHighlighting {
- private let highlightState: CrawlBarMenuItemHighlightState?
- private let hostingController: NSHostingController
- private var contentVersion = 0
- private var cachedWidth: CGFloat?
- private var cachedHeight: CGFloat?
- private var cachedContentVersion = -1
-
- override var allowsVibrancy: Bool { true }
-
- override var focusRingType: NSFocusRingType {
- get { .none }
- set {}
- }
-
- override var intrinsicContentSize: NSSize {
- let size = self.hostingController.view.intrinsicContentSize
- guard self.bounds.width > 0 else { return size }
- return NSSize(width: self.bounds.width, height: size.height)
- }
-
- init(rootView: AnyView, highlightState: CrawlBarMenuItemHighlightState? = nil) {
- self.highlightState = highlightState
- self.hostingController = NSHostingController(rootView: rootView)
- super.init(frame: .zero)
- self.contentVersion = 1
- self.configureHostingView()
- }
-
- @available(*, unavailable)
- required init?(coder _: NSCoder) {
- fatalError("init(coder:) has not been implemented")
- }
-
- override func layout() {
- super.layout()
- self.hostingController.view.frame = self.bounds
- }
-
- func setHighlighted(_ highlighted: Bool) {
- self.highlightState?.isHighlighted = highlighted
- }
-
- func measuredHeight(width: CGFloat) -> CGFloat {
- if self.cachedWidth == width, self.cachedContentVersion == self.contentVersion, let cachedHeight {
- return cachedHeight
- }
- if self.frame.size.width != width || self.bounds.size.width != width {
- self.frame.size.width = width
- self.bounds.size.width = width
- self.hostingController.view.frame = self.bounds
- self.invalidateIntrinsicContentSize()
- }
-
- let proposed = NSSize(width: width, height: CrawlBarMenuMetrics.proposedHeight)
- let measured = self.hostingController.sizeThatFits(in: proposed)
- let safeHeight = self.safeMeasuredHeight(from: measured.height)
- let scale = self.window?.backingScaleFactor ?? NSScreen.main?.backingScaleFactor ?? 2
- let rounded = ceil(safeHeight * scale) / scale
- self.cachedWidth = width
- self.cachedHeight = rounded
- self.cachedContentVersion = self.contentVersion
- return rounded
- }
-
- func updateRootView(_ rootView: AnyView) {
- self.hostingController.rootView = rootView
- self.contentVersion += 1
- self.cachedWidth = nil
- self.cachedHeight = nil
- self.invalidateIntrinsicContentSize()
- }
-
- private func configureHostingView() {
- self.hostingController.view.translatesAutoresizingMaskIntoConstraints = true
- self.hostingController.view.autoresizingMask = [.width, .height]
- self.hostingController.view.frame = self.bounds
- self.addSubview(self.hostingController.view)
- if #available(macOS 13.0, *) {
- self.hostingController.sizingOptions = [.minSize, .intrinsicContentSize]
- }
- }
-
- private func safeMeasuredHeight(from height: CGFloat) -> CGFloat {
- if height.isFinite, height > 0 {
- return min(height, CrawlBarMenuMetrics.maxHeight)
- }
- let intrinsic = self.hostingController.view.intrinsicContentSize.height
- if intrinsic.isFinite, intrinsic > 0 {
- return min(intrinsic, CrawlBarMenuMetrics.maxHeight)
- }
- return CrawlBarMenuMetrics.fallbackHeight
- }
-}
-
-@MainActor
-struct CrawlBarMenuItemFactory {
- func makeItem(
- for content: some View,
- enabled: Bool,
- highlightable: Bool = false,
- submenu: NSMenu? = nil
- ) -> NSMenuItem {
- let item = NSMenuItem()
- item.isEnabled = enabled
- if highlightable {
- let highlightState = CrawlBarMenuItemHighlightState()
- item.view = CrawlBarMenuItemHostingView(
- rootView: Self.highlightableRoot(content, highlightState: highlightState, showsSubmenuIndicator: submenu != nil),
- highlightState: highlightState)
- } else {
- item.view = CrawlBarMenuItemHostingView(rootView: Self.plainRoot(content))
- }
- item.submenu = submenu
- return item
- }
-
- func refreshViewHeights(in menu: NSMenu, width: CGFloat = CrawlBarMenuMetrics.width) {
- for item in menu.items {
- guard let view = item.view,
- let measuring = view as? CrawlBarMenuItemMeasuring
- else { continue }
- let height = measuring.measuredHeight(width: width)
- if abs(view.frame.size.height - height) > 0.5 || view.frame.size.width != width {
- view.frame = NSRect(origin: .zero, size: NSSize(width: width, height: height))
- }
- }
- }
-
- func clearHighlights(in menu: NSMenu) {
- for item in menu.items {
- (item.view as? CrawlBarMenuItemHighlighting)?.setHighlighted(false)
- }
- }
-
- private static func plainRoot(_ content: some View) -> AnyView {
- AnyView(
- content
- .fixedSize(horizontal: false, vertical: true)
- .frame(maxWidth: .infinity, alignment: .leading))
- }
-
- private static func highlightableRoot(
- _ content: some View,
- highlightState: CrawlBarMenuItemHighlightState,
- showsSubmenuIndicator: Bool)
- -> AnyView
- {
- AnyView(
- CrawlBarMenuItemContainerView(
- highlightState: highlightState,
- showsSubmenuIndicator: showsSubmenuIndicator)
- {
- content
- })
- }
-}
diff --git a/Sources/CrawlBar/Menu/CrawlBarMenuHighlight.swift b/Sources/CrawlBar/Menu/CrawlBarMenuHighlight.swift
new file mode 100644
index 0000000..f978312
--- /dev/null
+++ b/Sources/CrawlBar/Menu/CrawlBarMenuHighlight.swift
@@ -0,0 +1,77 @@
+import AppKit
+import SwiftUI
+
+private struct CrawlBarMenuItemHighlightedKey: EnvironmentKey {
+ static let defaultValue = false
+}
+
+extension EnvironmentValues {
+ var crawlBarMenuItemHighlighted: Bool {
+ get { self[CrawlBarMenuItemHighlightedKey.self] }
+ set { self[CrawlBarMenuItemHighlightedKey.self] = newValue }
+ }
+}
+
+enum CrawlBarMenuHighlightStyle {
+ static let selectionText = Color(nsColor: .selectedMenuItemTextColor)
+ static let normalPrimaryText = Color(nsColor: .controlTextColor)
+ static let normalSecondaryText = Color(nsColor: .secondaryLabelColor)
+
+ static func primary(_ highlighted: Bool) -> Color {
+ highlighted ? self.selectionText : self.normalPrimaryText
+ }
+
+ static func secondary(_ highlighted: Bool) -> Color {
+ highlighted ? self.selectionText.opacity(0.86) : self.normalSecondaryText
+ }
+
+ static func selectionBackground(_ highlighted: Bool) -> Color {
+ highlighted ? Color(nsColor: .selectedContentBackgroundColor) : .clear
+ }
+}
+
+@MainActor
+final class CrawlBarMenuItemHighlightState: ObservableObject {
+ @Published var isHighlighted = false
+}
+
+private struct CrawlBarMenuItemSelectionBackground: Shape {
+ func path(in rect: CGRect) -> Path {
+ let inset = rect.insetBy(
+ dx: CrawlBarMenuMetrics.selectionHorizontalInset,
+ dy: CrawlBarMenuMetrics.selectionVerticalInset)
+ return RoundedRectangle(
+ cornerRadius: CrawlBarMenuMetrics.selectionCornerRadius,
+ style: .continuous).path(in: inset)
+ }
+}
+
+struct CrawlBarMenuItemContainerView: View {
+ @ObservedObject var highlightState: CrawlBarMenuItemHighlightState
+ let showsSubmenuIndicator: Bool
+ @ViewBuilder var content: Content
+
+ var body: some View {
+ self.content
+ .fixedSize(horizontal: false, vertical: true)
+ .padding(.trailing, self.showsSubmenuIndicator ? CrawlBarMenuMetrics.submenuIndicatorTrailingPadding : 0)
+ .frame(maxWidth: .infinity, alignment: .leading)
+ .environment(\.crawlBarMenuItemHighlighted, self.highlightState.isHighlighted)
+ .foregroundStyle(CrawlBarMenuHighlightStyle.primary(self.highlightState.isHighlighted))
+ .background(alignment: .topLeading) {
+ if self.highlightState.isHighlighted {
+ CrawlBarMenuItemSelectionBackground()
+ .fill(CrawlBarMenuHighlightStyle.selectionBackground(true))
+ }
+ }
+ .overlay(alignment: .topTrailing) {
+ if self.showsSubmenuIndicator {
+ Image(systemName: "chevron.right")
+ .font(.caption2.weight(.semibold))
+ .foregroundStyle(CrawlBarMenuHighlightStyle.secondary(self.highlightState.isHighlighted))
+ .padding(.top, 8)
+ .padding(.trailing, CrawlBarMenuMetrics.submenuIndicatorTrailingPadding)
+ }
+ }
+ }
+}
diff --git a/Sources/CrawlBar/Menu/CrawlBarMenuItemFactory.swift b/Sources/CrawlBar/Menu/CrawlBarMenuItemFactory.swift
new file mode 100644
index 0000000..d5f25a5
--- /dev/null
+++ b/Sources/CrawlBar/Menu/CrawlBarMenuItemFactory.swift
@@ -0,0 +1,65 @@
+import AppKit
+import SwiftUI
+
+@MainActor
+struct CrawlBarMenuItemFactory {
+ func makeItem(
+ for content: some View,
+ enabled: Bool,
+ highlightable: Bool = false,
+ submenu: NSMenu? = nil
+ ) -> NSMenuItem {
+ let item = NSMenuItem()
+ item.isEnabled = enabled
+ if highlightable {
+ let highlightState = CrawlBarMenuItemHighlightState()
+ item.view = CrawlBarMenuItemHostingView(
+ rootView: Self.highlightableRoot(content, highlightState: highlightState, showsSubmenuIndicator: submenu != nil),
+ highlightState: highlightState)
+ } else {
+ item.view = CrawlBarMenuItemHostingView(rootView: Self.plainRoot(content))
+ }
+ item.submenu = submenu
+ return item
+ }
+
+ func refreshViewHeights(in menu: NSMenu, width: CGFloat = CrawlBarMenuMetrics.width) {
+ for item in menu.items {
+ guard let view = item.view,
+ let measuring = view as? CrawlBarMenuItemMeasuring
+ else { continue }
+ let height = measuring.measuredHeight(width: width)
+ if abs(view.frame.size.height - height) > 0.5 || view.frame.size.width != width {
+ view.frame = NSRect(origin: .zero, size: NSSize(width: width, height: height))
+ }
+ }
+ }
+
+ func clearHighlights(in menu: NSMenu) {
+ for item in menu.items {
+ (item.view as? CrawlBarMenuItemHighlighting)?.setHighlighted(false)
+ }
+ }
+
+ private static func plainRoot(_ content: some View) -> AnyView {
+ AnyView(
+ content
+ .fixedSize(horizontal: false, vertical: true)
+ .frame(maxWidth: .infinity, alignment: .leading))
+ }
+
+ private static func highlightableRoot(
+ _ content: some View,
+ highlightState: CrawlBarMenuItemHighlightState,
+ showsSubmenuIndicator: Bool)
+ -> AnyView
+ {
+ AnyView(
+ CrawlBarMenuItemContainerView(
+ highlightState: highlightState,
+ showsSubmenuIndicator: showsSubmenuIndicator)
+ {
+ content
+ })
+ }
+}
diff --git a/Sources/CrawlBar/Menu/CrawlBarMenuItemHostingView.swift b/Sources/CrawlBar/Menu/CrawlBarMenuItemHostingView.swift
new file mode 100644
index 0000000..91be9b5
--- /dev/null
+++ b/Sources/CrawlBar/Menu/CrawlBarMenuItemHostingView.swift
@@ -0,0 +1,100 @@
+import AppKit
+import SwiftUI
+
+@MainActor
+protocol CrawlBarMenuItemMeasuring: AnyObject {
+ func measuredHeight(width: CGFloat) -> CGFloat
+}
+
+@MainActor
+protocol CrawlBarMenuItemHighlighting: AnyObject {
+ func setHighlighted(_ highlighted: Bool)
+}
+
+@MainActor
+final class CrawlBarMenuItemHostingView: NSView, CrawlBarMenuItemMeasuring, CrawlBarMenuItemHighlighting {
+ private let highlightState: CrawlBarMenuItemHighlightState?
+ private let hostingController: NSHostingController
+ private var contentVersion = 0
+ private var cachedWidth: CGFloat?
+ private var cachedHeight: CGFloat?
+ private var cachedContentVersion = -1
+
+ override var allowsVibrancy: Bool { true }
+
+ override var focusRingType: NSFocusRingType {
+ get { .none }
+ set {}
+ }
+
+ override var intrinsicContentSize: NSSize {
+ let size = self.hostingController.view.intrinsicContentSize
+ guard self.bounds.width > 0 else { return size }
+ return NSSize(width: self.bounds.width, height: size.height)
+ }
+
+ init(rootView: AnyView, highlightState: CrawlBarMenuItemHighlightState? = nil) {
+ self.highlightState = highlightState
+ self.hostingController = NSHostingController(rootView: rootView)
+ super.init(frame: .zero)
+ self.contentVersion = 1
+ self.configureHostingView()
+ }
+
+ @available(*, unavailable)
+ required init?(coder _: NSCoder) {
+ fatalError("init(coder:) has not been implemented")
+ }
+
+ override func layout() {
+ super.layout()
+ self.hostingController.view.frame = self.bounds
+ }
+
+ func setHighlighted(_ highlighted: Bool) {
+ self.highlightState?.isHighlighted = highlighted
+ }
+
+ func measuredHeight(width: CGFloat) -> CGFloat {
+ if self.cachedWidth == width, self.cachedContentVersion == self.contentVersion, let cachedHeight {
+ return cachedHeight
+ }
+ if self.frame.size.width != width || self.bounds.size.width != width {
+ self.frame.size.width = width
+ self.bounds.size.width = width
+ self.hostingController.view.frame = self.bounds
+ self.invalidateIntrinsicContentSize()
+ }
+
+ let proposed = NSSize(width: width, height: CrawlBarMenuMetrics.proposedHeight)
+ let measured = self.hostingController.sizeThatFits(in: proposed)
+ let safeHeight = self.safeMeasuredHeight(from: measured.height)
+ let scale = self.window?.backingScaleFactor ?? NSScreen.main?.backingScaleFactor ?? 2
+ let rounded = ceil(safeHeight * scale) / scale
+ self.cachedWidth = width
+ self.cachedHeight = rounded
+ self.cachedContentVersion = self.contentVersion
+ return rounded
+ }
+
+ private func configureHostingView() {
+ self.hostingController.view.translatesAutoresizingMaskIntoConstraints = true
+ self.hostingController.view.autoresizingMask = [.width, .height]
+ self.hostingController.view.frame = self.bounds
+ self.addSubview(self.hostingController.view)
+ if #available(macOS 13.0, *) {
+ self.hostingController.sizingOptions = [.minSize, .intrinsicContentSize]
+ }
+ }
+
+ private func safeMeasuredHeight(from height: CGFloat) -> CGFloat {
+ if height.isFinite, height > 0 {
+ return min(height, CrawlBarMenuMetrics.maxHeight)
+ }
+ let intrinsic = self.hostingController.view.intrinsicContentSize.height
+ if intrinsic.isFinite, intrinsic > 0 {
+ return min(intrinsic, CrawlBarMenuMetrics.maxHeight)
+ }
+ return CrawlBarMenuMetrics.fallbackHeight
+ }
+}
diff --git a/Sources/CrawlBar/Menu/CrawlBarMenuMetrics.swift b/Sources/CrawlBar/Menu/CrawlBarMenuMetrics.swift
new file mode 100644
index 0000000..1743940
--- /dev/null
+++ b/Sources/CrawlBar/Menu/CrawlBarMenuMetrics.swift
@@ -0,0 +1,12 @@
+import CoreGraphics
+
+enum CrawlBarMenuMetrics {
+ static let width: CGFloat = 380
+ static let proposedHeight: CGFloat = 720
+ static let maxHeight: CGFloat = 360
+ static let fallbackHeight: CGFloat = 28
+ static let selectionHorizontalInset: CGFloat = 6
+ static let selectionVerticalInset: CGFloat = 2
+ static let selectionCornerRadius: CGFloat = 6
+ static let submenuIndicatorTrailingPadding: CGFloat = 10
+}
diff --git a/Sources/CrawlBar/CrawlBarMenuViews.swift b/Sources/CrawlBar/Menu/CrawlBarMenuViews.swift
similarity index 100%
rename from Sources/CrawlBar/CrawlBarMenuViews.swift
rename to Sources/CrawlBar/Menu/CrawlBarMenuViews.swift
diff --git a/Sources/CrawlBar/Menu/CrawlBarStatusMenuBuilder.swift b/Sources/CrawlBar/Menu/CrawlBarStatusMenuBuilder.swift
new file mode 100644
index 0000000..bb3e69f
--- /dev/null
+++ b/Sources/CrawlBar/Menu/CrawlBarStatusMenuBuilder.swift
@@ -0,0 +1,157 @@
+import AppKit
+import CrawlBarCore
+import SwiftUI
+
+@MainActor
+struct CrawlBarStatusMenuActionSelectors {
+ let showSettings: Selector
+ let showSettingsForApp: Selector
+ let refreshAll: Selector
+ let openLogs: Selector
+ let quit: Selector
+}
+
+@MainActor
+struct CrawlBarStatusMenuBuilder {
+ private let itemFactory = CrawlBarMenuItemFactory()
+
+ func rebuildMenu(
+ _ menu: NSMenu,
+ model: CrawlBarMenuModel,
+ target: AnyObject,
+ selectors: CrawlBarStatusMenuActionSelectors,
+ openSettings: @escaping (CrawlAppID?) -> Void)
+ {
+ let visibleInstallations = model.visibleInstallations
+
+ menu.autoenablesItems = false
+ menu.removeAllItems()
+
+ menu.addItem(self.viewItem(for: CrawlBarMenuHeaderView(
+ installations: visibleInstallations,
+ statuses: model.statuses,
+ isRefreshing: model.isRefreshing,
+ refreshFrequency: model.refreshFrequency), enabled: false))
+
+ self.addCrawlers(
+ visibleInstallations,
+ statuses: model.statuses,
+ to: menu,
+ target: target,
+ selector: selectors.showSettingsForApp,
+ openSettings: openSettings)
+
+ menu.addItem(.separator())
+ let refreshTitle = model.isRefreshing ? "Refreshing..." : "Refresh All"
+ let refreshItem = self.actionItem(
+ title: refreshTitle,
+ action: selectors.refreshAll,
+ target: target,
+ keyEquivalent: "r",
+ systemImage: "arrow.clockwise")
+ refreshItem.isEnabled = !model.statusTargetInstallations.isEmpty
+ menu.addItem(refreshItem)
+ menu.addItem(self.actionItem(
+ title: "Open Logs",
+ action: selectors.openLogs,
+ target: target,
+ keyEquivalent: "l",
+ systemImage: "folder"))
+ menu.addItem(self.actionItem(
+ title: "Settings...",
+ action: selectors.showSettings,
+ target: target,
+ keyEquivalent: ",",
+ systemImage: "gearshape"))
+ menu.addItem(.separator())
+ menu.addItem(self.actionItem(
+ title: "Quit CrawlBar",
+ action: selectors.quit,
+ target: target,
+ keyEquivalent: "q",
+ systemImage: "power"))
+
+ self.itemFactory.refreshViewHeights(in: menu)
+ }
+
+ func clearHighlights(in menu: NSMenu) {
+ self.itemFactory.clearHighlights(in: menu)
+ }
+
+ private func addCrawlers(
+ _ installations: [CrawlAppInstallation],
+ statuses: [CrawlAppID: CrawlAppStatus],
+ to menu: NSMenu,
+ target: AnyObject,
+ selector: Selector,
+ openSettings: @escaping (CrawlAppID?) -> Void)
+ {
+ guard !installations.isEmpty else { return }
+ menu.addItem(self.viewItem(for: CrawlBarMenuSeparatorRowView(), enabled: false))
+ for (index, installation) in installations.enumerated() {
+ menu.addItem(self.appMenuItem(
+ for: installation,
+ status: statuses[installation.id],
+ target: target,
+ selector: selector,
+ openSettings: openSettings))
+ if index < installations.count - 1 {
+ menu.addItem(self.viewItem(for: CrawlBarMenuSeparatorRowView(), enabled: false))
+ }
+ }
+ }
+
+ private func appMenuItem(
+ for installation: CrawlAppInstallation,
+ status: CrawlAppStatus?,
+ target: AnyObject,
+ selector: Selector,
+ openSettings: @escaping (CrawlAppID?) -> Void)
+ -> NSMenuItem
+ {
+ let card = CrawlBarMenuCardView(
+ installation: installation,
+ status: status,
+ onOpen: { openSettings(installation.id) })
+ return self.crawlerItem(
+ for: card,
+ installation: installation,
+ target: target,
+ selector: selector)
+ }
+
+ private func crawlerItem(
+ for content: some View,
+ installation: CrawlAppInstallation,
+ target: AnyObject,
+ selector: Selector)
+ -> NSMenuItem
+ {
+ let item = self.viewItem(for: content, enabled: true, highlightable: true)
+ item.title = CrawlBarCrawlerTitle.text(for: installation.id, manifest: installation.manifest)
+ item.representedObject = installation.id
+ item.target = target
+ item.action = selector
+ return item
+ }
+
+ private func viewItem(for content: some View, enabled: Bool, highlightable: Bool = false) -> NSMenuItem {
+ self.itemFactory.makeItem(for: content, enabled: enabled, highlightable: highlightable)
+ }
+
+ private func actionItem(
+ title: String,
+ action: Selector,
+ target: AnyObject,
+ keyEquivalent: String = "",
+ systemImage: String? = nil)
+ -> NSMenuItem
+ {
+ let item = NSMenuItem(title: title, action: action, keyEquivalent: keyEquivalent)
+ item.target = target
+ if let systemImage {
+ item.image = NSImage(systemSymbolName: systemImage, accessibilityDescription: title)
+ }
+ return item
+ }
+}
diff --git a/Sources/CrawlBar/CrawlBarApp.swift b/Sources/CrawlBar/Menu/MenuModel.swift
similarity index 53%
rename from Sources/CrawlBar/CrawlBarApp.swift
rename to Sources/CrawlBar/Menu/MenuModel.swift
index fb39829..5d1386f 100644
--- a/Sources/CrawlBar/CrawlBarApp.swift
+++ b/Sources/CrawlBar/Menu/MenuModel.swift
@@ -1,280 +1,5 @@
-import AppKit
import CrawlBarCore
-import SwiftUI
-
-@main
-@MainActor
-enum CrawlBarApp {
- static func main() {
- let app = NSApplication.shared
- let delegate = CrawlBarAppDelegate()
- app.delegate = delegate
- app.setActivationPolicy(.accessory)
- app.run()
- }
-}
-
-@MainActor
-final class CrawlBarAppDelegate: NSObject, NSApplicationDelegate, NSMenuDelegate {
- private var statusItem: NSStatusItem?
- private let menuItemFactory = CrawlBarMenuItemFactory()
- private var refreshTimer: Timer?
- private var refreshAnimationTimer: Timer?
- private var refreshAnimationFrame = 0
- private var pendingMenuReloadTask: Task?
- private var isMenuOpen = false
- private var menuNeedsReloadAfterClose = false
- private let settingsWindowController = CrawlBarSettingsWindowController()
- private let model = CrawlBarMenuModel()
-
- func applicationDidFinishLaunching(_ notification: Notification) {
- CrawlBarLog.app.notice("CrawlBar launched")
- NotificationCenter.default.addObserver(
- self,
- selector: #selector(Self.statusesDidChange(_:)),
- name: .crawlBarStatusesDidChange,
- object: nil)
- NotificationCenter.default.addObserver(
- self,
- selector: #selector(Self.configDidChange(_:)),
- name: .crawlBarConfigDidChange,
- object: nil)
- self.settingsWindowController.onClose = { [weak self] in
- self?.hideFromApplicationSwitcher()
- }
- if let appIcon = CrawlBarIconFactory.appIconImage() {
- NSApplication.shared.applicationIconImage = appIcon
- }
- let statusItem = NSStatusBar.system.statusItem(withLength: NSStatusItem.variableLength)
- statusItem.button?.imagePosition = .imageLeading
- self.statusItem = statusItem
- self.updateStatusButtonImage()
- self.reloadMenu()
- self.model.refreshAll { [weak self] in
- self?.scheduleMenuReload()
- }
- self.scheduleRefreshTimer()
- }
-
- func applicationWillTerminate(_ notification: Notification) {
- CrawlBarLog.app.notice("CrawlBar terminated")
- self.pendingMenuReloadTask?.cancel()
- NotificationCenter.default.removeObserver(self)
- }
-
- private func reloadMenu() {
- self.pendingMenuReloadTask?.cancel()
- self.pendingMenuReloadTask = nil
- let startedAt = CFAbsoluteTimeGetCurrent()
- let menu = self.statusItem?.menu ?? NSMenu()
- menu.autoenablesItems = false
- menu.delegate = self
- menu.removeAllItems()
-
- menu.addItem(self.viewItem(for: CrawlBarMenuHeaderView(
- installations: self.model.visibleInstallations,
- statuses: self.model.statuses,
- isRefreshing: self.model.isRefreshing,
- refreshFrequency: self.model.refreshFrequency), enabled: false))
- menu.addItem(self.viewItem(for: CrawlBarMenuSeparatorRowView(), enabled: false))
-
- for (index, installation) in self.model.visibleInstallations.enumerated() {
- menu.addItem(self.appMenuItem(for: installation))
- if index < self.model.visibleInstallations.count - 1 {
- menu.addItem(self.viewItem(for: CrawlBarMenuSeparatorRowView(), enabled: false))
- }
- }
-
- menu.addItem(.separator())
- let refreshTitle = self.model.isRefreshing ? "Refreshing..." : "Refresh All"
- menu.addItem(self.actionItem(title: refreshTitle, action: #selector(Self.refreshAll(_:)), keyEquivalent: "r", systemImage: "arrow.clockwise"))
- menu.addItem(self.actionItem(title: "Open Logs", action: #selector(Self.openLogs(_:)), keyEquivalent: "l", systemImage: "folder"))
- menu.addItem(self.actionItem(title: "Settings...", action: #selector(Self.showSettings(_:)), keyEquivalent: ",", systemImage: "gearshape"))
- menu.addItem(.separator())
- menu.addItem(self.actionItem(title: "Quit CrawlBar", action: #selector(Self.quit(_:)), keyEquivalent: "q", systemImage: "power"))
-
- self.statusItem?.menu = menu
- if let button = self.statusItem?.button {
- button.target = nil
- button.action = nil
- button.isEnabled = true
- }
- self.menuItemFactory.refreshViewHeights(in: menu)
- self.syncRefreshAnimation()
- let elapsedMilliseconds = (CFAbsoluteTimeGetCurrent() - startedAt) * 1000
- CrawlBarLog.app.debug("Reloaded menu in \(elapsedMilliseconds, privacy: .public)ms")
- }
-
- private func scheduleMenuReload() {
- guard !self.isMenuOpen else {
- self.menuNeedsReloadAfterClose = true
- return
- }
- self.pendingMenuReloadTask?.cancel()
- self.pendingMenuReloadTask = Task { @MainActor [weak self] in
- try? await Task.sleep(nanoseconds: 80_000_000)
- guard !Task.isCancelled else { return }
- self?.reloadMenu()
- }
- }
-
- private func appMenuItem(for installation: CrawlAppInstallation) -> NSMenuItem {
- let card = CrawlBarMenuCardView(
- installation: installation,
- status: self.model.statuses[installation.id],
- onOpen: { [weak self] in self?.openSettings(appID: installation.id) })
- let item = self.viewItem(for: card, enabled: true, highlightable: true)
- item.title = CrawlBarCrawlerTitle.text(for: installation.id, manifest: installation.manifest)
- item.representedObject = installation.id
- item.target = self
- item.action = #selector(Self.showSettingsForAppMenuItem(_:))
- return item
- }
-
- private func viewItem(for content: some View, enabled: Bool, highlightable: Bool = false) -> NSMenuItem {
- self.menuItemFactory.makeItem(for: content, enabled: enabled, highlightable: highlightable)
- }
-
- private func actionItem(title: String, action: Selector, keyEquivalent: String = "", systemImage: String? = nil) -> NSMenuItem {
- let item = NSMenuItem(title: title, action: action, keyEquivalent: keyEquivalent)
- item.target = self
- if let systemImage {
- item.image = NSImage(systemSymbolName: systemImage, accessibilityDescription: title)
- }
- return item
- }
-
- private func effectiveState(for installation: CrawlAppInstallation, status: CrawlAppStatus?) -> CrawlAppState {
- if installation.manifest.availability == .comingSoon { return .disabled }
- if !installation.enabled { return .disabled }
- if installation.binaryPath == nil { return .needsConfig }
- let state = status?.state ?? .unknown
- if status?.isRecoverableGraincrawlSourceFailure == true {
- return .stale
- }
- return state
- }
-
- private func scheduleRefreshTimer() {
- self.refreshTimer?.invalidate()
- self.refreshTimer = Timer.scheduledTimer(withTimeInterval: 60, repeats: true) { [weak self] _ in
- Task { @MainActor in
- self?.model.runDueAutoSync {
- self?.scheduleMenuReload()
- }
- }
- }
- }
-
- private func syncRefreshAnimation() {
- self.updateStatusButtonImage()
- if self.model.isRefreshing {
- guard self.refreshAnimationTimer == nil else { return }
- self.refreshAnimationTimer = Timer.scheduledTimer(withTimeInterval: 0.16, repeats: true) { [weak self] _ in
- Task { @MainActor in
- self?.advanceRefreshAnimation()
- }
- }
- } else {
- self.refreshAnimationTimer?.invalidate()
- self.refreshAnimationTimer = nil
- self.refreshAnimationFrame = 0
- self.updateStatusButtonImage()
- }
- }
-
- private func advanceRefreshAnimation() {
- guard self.model.isRefreshing else {
- self.syncRefreshAnimation()
- return
- }
- self.refreshAnimationFrame = (self.refreshAnimationFrame + 1) % 8
- self.updateStatusButtonImage()
- }
-
- private func updateStatusButtonImage() {
- let rotation = self.model.isRefreshing ? CGFloat(self.refreshAnimationFrame) * 45 : 0
- self.statusItem?.button?.image = CrawlBarIconFactory.menuBarImage(rotationDegrees: rotation)
- self.statusItem?.button?.toolTip = nil
- }
-
- @objc private func refreshAll(_ sender: Any?) {
- self.model.refreshAll { [weak self] in
- self?.scheduleMenuReload()
- }
- self.reloadMenu()
- }
-
- @objc private func showSettings(_ sender: Any?) {
- self.openSettings(appID: nil)
- }
-
- @objc private func showSettingsForAppMenuItem(_ sender: NSMenuItem) {
- self.openSettings(appID: sender.representedObject as? CrawlAppID)
- }
-
- private func openSettings(appID: CrawlAppID?) {
- self.statusItem?.menu?.cancelTracking()
- CrawlBarLog.app.debug("Opening settings")
- self.showInApplicationSwitcher()
- self.settingsWindowController.show(appID: appID)
- }
-
- private func showInApplicationSwitcher() {
- NSApplication.shared.setActivationPolicy(.regular)
- NSApplication.shared.activate(ignoringOtherApps: true)
- }
-
- private func hideFromApplicationSwitcher() {
- NSApplication.shared.setActivationPolicy(.accessory)
- }
-
- @objc private func openLogs(_ sender: Any?) {
- self.statusItem?.menu?.cancelTracking()
- CrawlBarLog.app.debug("Opening action logs folder")
- NSWorkspace.shared.open(CrawlActionLogStore.defaultDirectory())
- }
-
- @objc private func quit(_ sender: Any?) {
- self.statusItem?.menu?.cancelTracking()
- CrawlBarLog.app.notice("Quit requested")
- NSApplication.shared.terminate(nil)
- }
-
- func menuWillOpen(_ menu: NSMenu) {
- self.isMenuOpen = true
- if self.pendingMenuReloadTask != nil {
- self.reloadMenu()
- self.isMenuOpen = true
- }
- }
-
- func menuDidClose(_ menu: NSMenu) {
- self.isMenuOpen = false
- self.menuItemFactory.clearHighlights(in: menu)
- if self.menuNeedsReloadAfterClose {
- self.menuNeedsReloadAfterClose = false
- self.reloadMenu()
- }
- }
-
- func menu(_ menu: NSMenu, willHighlight item: NSMenuItem?) {
- for menuItem in menu.items {
- guard let view = menuItem.view as? CrawlBarMenuItemHighlighting else { continue }
- view.setHighlighted(menuItem == item && menuItem.isEnabled)
- }
- }
-
- @objc private func statusesDidChange(_ notification: Notification) {
- self.scheduleMenuReload()
- }
-
- @objc private func configDidChange(_ notification: Notification) {
- self.model.reloadInstallations()
- self.scheduleRefreshTimer()
- self.reloadMenu()
- }
-}
+import Foundation
private struct CrawlActionStatusUpdate: Sendable {
let status: CrawlAppStatus
@@ -321,6 +46,13 @@ final class CrawlBarMenuModel: NSObject {
}
}
+ var statusTargetInstallations: [CrawlAppInstallation] {
+ self.installations.filter { installation in
+ guard installation.manifest.availability == .available else { return false }
+ return self.appConfigs[installation.id]?.enabled ?? installation.enabled
+ }
+ }
+
func appConfig(for id: CrawlAppID) -> CrawlBarAppConfig? {
self.appConfigs[id]
}
@@ -340,16 +72,21 @@ final class CrawlBarMenuModel: NSObject {
let generation = UUID()
self.refreshGeneration = generation
self.isRefreshing = true
+ let appConfigs = self.appConfigs
let registry = self.registry
let statusService = self.statusService
self.refreshTask = Task.detached {
let installations = (try? registry.installationsForStatus(includeDisabled: true)) ?? []
+ let statusInstallations = installations.filter { installation in
+ guard installation.manifest.availability == .available else { return false }
+ return appConfigs[installation.id]?.enabled ?? installation.enabled
+ }
await MainActor.run {
guard self.refreshGeneration == generation else { return }
self.installations = installations
onComplete()
}
- let partitioned = Self.partitionStatuses(installations: installations, statusService: statusService)
+ let partitioned = Self.partitionStatuses(installations: statusInstallations, statusService: statusService)
if !partitioned.immediate.isEmpty {
await MainActor.run {
guard self.refreshGeneration == generation else { return }
@@ -388,23 +125,6 @@ final class CrawlBarMenuModel: NSObject {
}
}
- nonisolated private static func partitionStatuses(
- installations: [CrawlAppInstallation],
- statusService: CrawlStatusService)
- -> (immediate: [CrawlAppStatus], commandInstallations: [CrawlAppInstallation])
- {
- var immediate: [CrawlAppStatus] = []
- var commandInstallations: [CrawlAppInstallation] = []
- for installation in installations {
- if let status = statusService.immediateStatus(for: installation) {
- immediate.append(status)
- } else {
- commandInstallations.append(installation)
- }
- }
- return (immediate, commandInstallations)
- }
-
func runDueAutoSync(onComplete: @escaping @MainActor () -> Void) {
guard !self.isRefreshing else { return }
self.reloadInstallations()
@@ -509,6 +229,23 @@ final class CrawlBarMenuModel: NSObject {
self.mergeStatuses(statuses)
}
+ nonisolated private static func partitionStatuses(
+ installations: [CrawlAppInstallation],
+ statusService: CrawlStatusService)
+ -> (immediate: [CrawlAppStatus], commandInstallations: [CrawlAppInstallation])
+ {
+ var immediate: [CrawlAppStatus] = []
+ var commandInstallations: [CrawlAppInstallation] = []
+ for installation in installations {
+ if let status = statusService.immediateStatus(for: installation) {
+ immediate.append(status)
+ } else {
+ commandInstallations.append(installation)
+ }
+ }
+ return (immediate, commandInstallations)
+ }
+
nonisolated private static func actionFailureStatus(_ result: CrawlCommandResult) -> CrawlAppStatus {
let fallback = "\(result.action) failed with exit \(result.exitCode)"
return CrawlAppStatus.commandFailure(
@@ -537,12 +274,4 @@ final class CrawlBarMenuModel: NSObject {
}
return metadataStatus.mergingActionFailure(failure)
}
-
-}
-
-private extension NSMenuItem {
- convenience init(title: String, action: Selector?, keyEquivalent: String, target: AnyObject) {
- self.init(title: title, action: action, keyEquivalent: keyEquivalent)
- self.target = target
- }
}
diff --git a/Sources/CrawlBar/Settings/AppDetailConfigurationSections.swift b/Sources/CrawlBar/Settings/AppDetailConfigurationSections.swift
new file mode 100644
index 0000000..8b81018
--- /dev/null
+++ b/Sources/CrawlBar/Settings/AppDetailConfigurationSections.swift
@@ -0,0 +1,135 @@
+import CrawlBarCore
+import Foundation
+import SwiftUI
+
+extension CrawlBarAppDetailView {
+ var configurationSection: some View {
+ CrawlBarDetailSection(title: "Configuration") {
+ self.configuration
+ self.paths
+ self.privacy
+ }
+ }
+
+ var paths: some View {
+ CrawlBarPanel(title: "Paths") {
+ CrawlBarControlRow(
+ title: "Binary path override",
+ caption: "Leave empty to resolve the CLI from PATH.")
+ {
+ TextField("Optional", text: self.optionalText(\.binaryPath))
+ .textFieldStyle(.roundedBorder)
+ .frame(width: 260)
+ .onSubmit(self.save)
+ }
+ CrawlBarControlRow(
+ title: "Config path override",
+ caption: "Leave empty to use the crawler default.")
+ {
+ TextField("Optional", text: self.optionalText(\.configPath))
+ .textFieldStyle(.roundedBorder)
+ .frame(width: 260)
+ .onSubmit(self.save)
+ }
+ CrawlBarFact(label: "Default Config", value: self.manifest?.paths.defaultConfig ?? "None")
+ CrawlBarFact(label: "Default Database", value: self.status?.databasePath ?? self.manifest?.paths.defaultDatabase ?? "Unknown")
+ CrawlBarFact(label: "Logs", value: self.manifest?.paths.defaultLogs ?? "Unknown")
+ }
+ }
+
+ @ViewBuilder
+ var configuration: some View {
+ if self.manifest?.availability == .comingSoon {
+ CrawlBarPanel(title: "Coming Soon") {
+ CrawlBarFact(label: "CLI", value: self.manifest?.binary.name ?? self.app.id.rawValue)
+ CrawlBarFact(label: "Config", value: self.manifest?.paths.defaultConfig ?? "Not declared")
+ }
+ } else if let manifest = self.manifest, !manifest.configOptions.isEmpty {
+ ForEach(self.configSections(for: manifest)) { section in
+ CrawlBarPanel(title: section.title, caption: section.caption) {
+ ForEach(section.options) { option in
+ CrawlBarConfigOptionField(
+ option: option,
+ value: self.configValueBinding(for: option),
+ disabledReason: self.configDisabledReason(for: option))
+ }
+ }
+ }
+ }
+ }
+
+ var privacy: some View {
+ CrawlBarPanel(title: "Privacy") {
+ CrawlBarFact(
+ label: "Private Messages",
+ value: self.manifest?.privacy.containsPrivateMessages == true ? "Possible local data" : "Not declared")
+ CrawlBarFact(label: "Local-only scopes", value: self.manifest?.privacy.localOnlyScopes.joined(separator: ", ").nilIfBlank ?? "None")
+ CrawlBarFact(label: "Action logs", value: CrawlActionLogStore.defaultDirectory().path)
+ }
+ }
+
+ var configSourceSummary: String {
+ if let configPath = self.status?.configPath ?? self.app.configPath ?? self.manifest?.paths.defaultConfig {
+ return URL(fileURLWithPath: configPath).lastPathComponent
+ }
+ return "None"
+ }
+
+ func optionalText(_ keyPath: WritableKeyPath) -> Binding {
+ Binding(
+ get: { self.app[keyPath: keyPath] ?? "" },
+ set: {
+ self.app[keyPath: keyPath] = $0.nilIfBlank
+ self.saveDebounced()
+ })
+ }
+
+ func configValueBinding(for option: CrawlAppManifest.ConfigOption) -> Binding {
+ Binding(
+ get: { self.app.configValues[option.id] ?? option.defaultValue ?? "" },
+ set: {
+ let value = $0.nilIfBlank
+ self.app.configValues[option.id] = value
+ self.configValueChanged(option, value)
+ })
+ }
+
+ func configDisabledReason(for option: CrawlAppManifest.ConfigOption) -> String? {
+ guard self.usesRemoteStore else { return nil }
+ let optionText = [
+ option.id,
+ option.configKey,
+ option.envVar,
+ ].compactMap { $0?.lowercased() }.joined(separator: " ")
+ guard optionText.contains("openai") || optionText.contains("embedding") else { return nil }
+ return "Disabled while this crawler is using a remote store."
+ }
+
+ func configSections(for manifest: CrawlAppManifest) -> [CrawlBarConfigSection] {
+ var optionsByID: [String: CrawlAppManifest.ConfigOption] = [:]
+ for option in manifest.configOptions where optionsByID[option.id] == nil {
+ optionsByID[option.id] = option
+ }
+ let sections = manifest.configSections.isEmpty
+ ? [CrawlBarConfigSection(id: "config", title: "Configuration", optionIDs: manifest.configOptions.map(\.id))]
+ : manifest.configSections.map {
+ CrawlBarConfigSection(
+ id: $0.id,
+ title: $0.title,
+ caption: $0.caption,
+ optionIDs: $0.optionIDs)
+ }
+
+ let usedIDs = Set(sections.flatMap(\.optionIDs))
+ let resolved = sections.compactMap { section -> CrawlBarConfigSection? in
+ let options = section.optionIDs.compactMap { optionsByID[$0] }
+ guard !options.isEmpty else { return nil }
+ return section.resolved(options: options)
+ }
+ let extraOptions = manifest.configOptions.filter { !usedIDs.contains($0.id) }
+ if extraOptions.isEmpty {
+ return resolved
+ }
+ return resolved + [CrawlBarConfigSection(id: "advanced", title: "Advanced", optionIDs: [], options: extraOptions)]
+ }
+}
diff --git a/Sources/CrawlBar/Settings/AppDetailDataSections.swift b/Sources/CrawlBar/Settings/AppDetailDataSections.swift
new file mode 100644
index 0000000..352f3f1
--- /dev/null
+++ b/Sources/CrawlBar/Settings/AppDetailDataSections.swift
@@ -0,0 +1,260 @@
+import CrawlBarCore
+import Foundation
+import SwiftUI
+
+extension CrawlBarAppDetailView {
+ var dataSection: some View {
+ CrawlBarDetailSection(title: "Data") {
+ self.remoteStore
+ if !self.usesRemoteStore {
+ self.databases
+ }
+ self.metrics
+ }
+ }
+
+ @ViewBuilder
+ var remoteStore: some View {
+ if let remoteStore = self.remoteStoreSummary {
+ CrawlBarPanel(title: remoteStore.title) {
+ CrawlBarFact(label: "Remote", value: remoteStore.remote)
+ if let archive = remoteStore.archive {
+ CrawlBarFact(label: "Archive", value: archive)
+ }
+ if let repoPath = remoteStore.repoPath {
+ CrawlBarFact(label: "Checkout", value: repoPath)
+ }
+ if let branch = remoteStore.branch {
+ CrawlBarFact(label: "Branch", value: branch)
+ }
+ if let bundle = remoteStore.bundle {
+ CrawlBarFact(label: "Bundle", value: bundle)
+ }
+ if let compressed = remoteStore.compressed {
+ CrawlBarFact(label: "Compressed", value: compressed)
+ }
+ if let parts = remoteStore.parts {
+ CrawlBarFact(label: "Parts", value: parts)
+ }
+ if let lastIngest = remoteStore.lastIngest {
+ CrawlBarFact(label: "Ingest", value: lastIngest)
+ }
+ if let databasePath = self.status?.databasePath {
+ CrawlBarFact(label: "Local index", value: URL(fileURLWithPath: databasePath).lastPathComponent)
+ }
+ }
+ }
+ }
+
+ @ViewBuilder
+ var databases: some View {
+ if let databases = self.status?.databases, !databases.isEmpty {
+ CrawlBarPanel(title: "Databases") {
+ HStack {
+ Spacer(minLength: 0)
+ Button {
+ self.openDataFolder()
+ } label: {
+ Image(systemName: "folder")
+ }
+ .buttonStyle(.borderless)
+ .accessibilityLabel("Open data folder")
+ Button {
+ self.backupDatabases()
+ } label: {
+ Image(systemName: "archivebox")
+ }
+ .buttonStyle(.borderless)
+ .disabled(self.runningAction != nil)
+ .accessibilityLabel("Back up database files")
+ Text("\(databases.count)")
+ .font(.caption.weight(.medium))
+ .foregroundStyle(.secondary)
+ .monospacedDigit()
+ }
+ VStack(spacing: 0) {
+ ForEach(databases) { database in
+ CrawlBarDatabaseRow(database: database)
+ if database.id != databases.last?.id {
+ Divider()
+ .padding(.leading, 28)
+ }
+ }
+ }
+ }
+ } else {
+ CrawlBarPanel(title: "Databases") {
+ Label("No database metadata yet", systemImage: "internaldrive")
+ .foregroundStyle(.secondary)
+ }
+ }
+ }
+
+ @ViewBuilder
+ var metrics: some View {
+ if !self.overviewCounts.isEmpty {
+ CrawlBarPanel(title: "Counts") {
+ HStack(alignment: .firstTextBaseline, spacing: 8) {
+ Spacer(minLength: 0)
+ Text(self.overviewDataScope)
+ .font(.caption)
+ .foregroundStyle(.secondary)
+ }
+ VStack(spacing: 0) {
+ ForEach(self.overviewCounts) { count in
+ CrawlBarMetricRow(label: count.label, value: "\(count.value)")
+ if count.id != self.overviewCounts.last?.id {
+ Divider()
+ }
+ }
+ }
+ }
+ } else {
+ CrawlBarPanel(title: "Counts") {
+ Label("No count metrics yet", systemImage: "number")
+ .foregroundStyle(.secondary)
+ }
+ }
+ }
+
+ var archiveSourceSummary: String {
+ if let remoteStore = self.remoteStoreSummary {
+ return remoteStore.shortName
+ }
+ if let database = self.primaryDatabase ?? self.status?.databases.first {
+ return database.path.map { URL(fileURLWithPath: $0).lastPathComponent } ?? database.label
+ }
+ if let databasePath = self.status?.databasePath ?? self.manifest?.paths.defaultDatabase {
+ return URL(fileURLWithPath: databasePath).lastPathComponent
+ }
+ return "Unknown"
+ }
+
+ var databaseSummary: String {
+ guard let status else { return "Unknown" }
+ if let remoteStore = self.remoteStoreSummary {
+ return remoteStore.databaseSummary
+ }
+ if self.app.id == BuiltInCrawlApps.gitcrawlID {
+ return self.summaryText(label: "GitHub archives", bytes: self.totalDatabaseBytes)
+ }
+ if status.databases.count > 1 {
+ return self.summaryText(label: "\(status.databases.count) databases", bytes: self.totalDatabaseBytes)
+ }
+ if let primaryDatabase = self.primaryDatabase ?? status.databases.first {
+ let size = primaryDatabase.bytes ?? status.databaseBytes
+ return self.summaryText(label: primaryDatabase.label, bytes: size)
+ }
+ return status.databasePath.map { URL(fileURLWithPath: $0).lastPathComponent } ?? "Unknown"
+ }
+
+ var primaryDatabase: CrawlDatabaseResource? {
+ self.status?.databases.first(where: { $0.isPrimary })
+ }
+
+ var totalDatabaseBytes: Int? {
+ guard let status else { return nil }
+ let total = status.databases.compactMap(\.bytes).reduce(0, +)
+ if total > 0 { return total }
+ return status.databaseBytes
+ }
+
+ var overviewCounts: [CrawlCount] {
+ if let primaryCounts = self.primaryDatabase?.counts, !primaryCounts.isEmpty {
+ return primaryCounts
+ }
+ let databaseCounts = self.totalCountsAcrossDatabases()
+ if !databaseCounts.isEmpty {
+ return databaseCounts
+ }
+ return self.status?.counts ?? []
+ }
+
+ var overviewDataScope: String {
+ if let remoteStore = self.remoteStoreSummary {
+ return remoteStore.dataScope
+ }
+ if self.primaryDatabase?.counts.isEmpty == false {
+ return "Active database"
+ }
+ if let count = self.status?.databases.count, count > 1, !self.totalCountsAcrossDatabases().isEmpty {
+ return "Total across \(count) databases"
+ }
+ return "Connected database"
+ }
+
+ func totalCountsAcrossDatabases() -> [CrawlCount] {
+ let counts = self.status?.databases.flatMap(\.counts) ?? []
+ guard !counts.isEmpty else { return [] }
+ var labels: [String: String] = [:]
+ var values: [String: Int] = [:]
+ for count in counts {
+ labels[count.id] = labels[count.id] ?? count.label
+ values[count.id, default: 0] += count.value
+ }
+ return values.keys.sorted().map { id in
+ CrawlCount(id: id, label: labels[id] ?? id, value: values[id] ?? 0)
+ }
+ }
+
+ func summaryText(label: String, bytes: Int?) -> String {
+ [
+ label,
+ bytes.map { CrawlBarFileSizeText.string(fromByteCount: Int64($0)) },
+ ].compactMap { $0?.nilIfBlank }.joined(separator: " · ")
+ }
+
+ func bundleSummary(_ bundle: CrawlSQLiteBundleStatus) -> String {
+ [
+ bundle.format?.nilIfBlank,
+ bundle.compression?.nilIfBlank,
+ bundle.compressedBytes.map { CrawlBarFileSizeText.string(fromByteCount: Int64($0)) },
+ bundle.partCount.map { "\($0) part\($0 == 1 ? "" : "s")" },
+ ].compactMap { $0 }.joined(separator: " · ")
+ }
+
+ var usesRemoteStore: Bool {
+ self.remoteStoreSummary != nil
+ }
+
+ var remoteStoreSummary: CrawlBarRemoteStoreSummary? {
+ if let remote = self.status?.remote, remote.enabled {
+ let database = self.status?.databases.first(where: { $0.endpoint != nil || $0.archive != nil })
+ let endpoint = remote.endpoint?.nilIfBlank ?? database?.endpoint?.nilIfBlank ?? "Cloudflare remote"
+ let archive = remote.archive?.nilIfBlank ?? database?.archive?.nilIfBlank
+ return CrawlBarRemoteStoreSummary(
+ remote: endpoint,
+ archive: archive,
+ kind: .cloudflare,
+ sqliteBundle: self.status?.sqliteBundle,
+ sqliteObject: self.status?.sqliteObject,
+ lastIngestAt: remote.lastIngestAt ?? remote.lastSyncAt)
+ }
+ if self.status?.share?.enabled == true, let remote = self.status?.share?.remote?.nilIfBlank {
+ return CrawlBarRemoteStoreSummary(
+ remote: remote,
+ repoPath: self.status?.share?.repoPath?.nilIfBlank,
+ branch: self.status?.share?.branch?.nilIfBlank,
+ kind: .gitSnapshot)
+ }
+ guard self.app.id == BuiltInCrawlApps.gitcrawlID else { return nil }
+ var paths = self.status?.databases.compactMap(\.path) ?? []
+ if let databasePath = self.status?.databasePath {
+ paths.append(databasePath)
+ }
+ guard let storePath = paths.first(where: { $0.contains("/gitcrawl-store/") || $0.contains("/gitcrawl-store-remote/") }) else {
+ return nil
+ }
+ let repoPath = Self.repoPath(containing: "/data/", in: storePath)
+ return CrawlBarRemoteStoreSummary(
+ remote: "https://github.com/openclaw/gitcrawl-store.git",
+ repoPath: repoPath,
+ branch: nil,
+ kind: .gitSnapshot)
+ }
+
+ static func repoPath(containing marker: String, in path: String) -> String? {
+ guard let range = path.range(of: marker) else { return nil }
+ return String(path[.. CrawlBarConfigSection {
+ CrawlBarConfigSection(
+ id: self.id,
+ title: self.title,
+ caption: self.caption,
+ optionIDs: self.optionIDs,
+ options: options)
+ }
+}
+
+struct CrawlBarRemoteStoreSummary {
+ enum Kind {
+ case gitSnapshot
+ case cloudflare
+ }
+
+ var remote: String
+ var archive: String? = nil
+ var repoPath: String? = nil
+ var branch: String? = nil
+ var kind: Kind
+ var sqliteBundle: CrawlSQLiteBundleStatus? = nil
+ var sqliteObject: CrawlSQLiteObjectStatus? = nil
+ var lastIngestAt: Date? = nil
+
+ var title: String {
+ switch self.kind {
+ case .cloudflare:
+ "Cloudflare Archive"
+ case .gitSnapshot:
+ "Remote Store"
+ }
+ }
+
+ var shortName: String {
+ if self.kind == .cloudflare {
+ return self.archive?.nilIfBlank ?? "Cloudflare archive"
+ }
+ let trimmed = self.remote
+ .replacingOccurrences(of: "https://github.com/", with: "")
+ .replacingOccurrences(of: "git@github.com:", with: "")
+ .replacingOccurrences(of: ".git", with: "")
+ .nilIfBlank
+ return trimmed ?? "Remote store"
+ }
+
+ var dataScope: String {
+ switch self.kind {
+ case .cloudflare:
+ "Cloudflare remote"
+ case .gitSnapshot:
+ "Remote store"
+ }
+ }
+
+ var databaseSummary: String {
+ guard self.kind == .cloudflare else { return self.shortName }
+ let pieces = [
+ self.shortName,
+ self.sqliteBundle?.compressedBytes.map { CrawlBarFileSizeText.string(fromByteCount: Int64($0)) },
+ self.sqliteBundle?.compression?.nilIfBlank,
+ ].compactMap { $0 }
+ return pieces.isEmpty ? "Cloudflare archive" : pieces.joined(separator: " · ")
+ }
+
+ var bundle: String? {
+ guard let sqliteBundle else { return nil }
+ return [
+ sqliteBundle.format?.nilIfBlank,
+ sqliteBundle.compression?.nilIfBlank,
+ ].compactMap { $0 }.joined(separator: " · ").nilIfBlank
+ }
+
+ var compressed: String? {
+ let values = [
+ sqliteBundle?.compressedBytes.map { CrawlBarFileSizeText.string(fromByteCount: Int64($0)) },
+ sqliteBundle?.rawBytes.map { CrawlBarFileSizeText.string(fromByteCount: Int64($0)) + " raw" },
+ sqliteObject?.bytes.map { CrawlBarFileSizeText.string(fromByteCount: Int64($0)) + " object" },
+ ].compactMap { $0?.nilIfBlank }
+ return values.isEmpty ? nil : values.joined(separator: " / ")
+ }
+
+ var parts: String? {
+ sqliteBundle?.partCount.map { "\($0)" }
+ }
+
+ var lastIngest: String? {
+ guard let lastIngestAt else { return nil }
+ let formatter = RelativeDateTimeFormatter()
+ formatter.unitsStyle = .abbreviated
+ return formatter.localizedString(for: lastIngestAt, relativeTo: Date())
+ }
+}
diff --git a/Sources/CrawlBar/Settings/AppDetailSyncSections.swift b/Sources/CrawlBar/Settings/AppDetailSyncSections.swift
new file mode 100644
index 0000000..5194c30
--- /dev/null
+++ b/Sources/CrawlBar/Settings/AppDetailSyncSections.swift
@@ -0,0 +1,272 @@
+import AppKit
+import CrawlBarCore
+import SwiftUI
+
+extension CrawlBarAppDetailView {
+ var syncSection: some View {
+ CrawlBarDetailSection(title: "Sync") {
+ self.syncSettings
+ self.cloudArchiveSettings
+ self.gitShareSettings
+ }
+ }
+
+ var syncSettings: some View {
+ CrawlBarPanel(title: "Sync") {
+ VStack(alignment: .leading, spacing: 10) {
+ CrawlBarSwitchRow(
+ title: "Enable crawler",
+ caption: "Allow CrawlBar to run actions and show live status.",
+ isOn: self.$app.enabled)
+ .onChange(of: self.app.enabled) { self.save() }
+ CrawlBarSwitchRow(
+ title: "Show in menu bar",
+ caption: "Include this crawler in the menu bar status menu.",
+ isOn: self.$app.showInMenuBar)
+ .disabled(!self.app.enabled)
+ .onChange(of: self.app.showInMenuBar) { self.save() }
+ CrawlBarSwitchRow(
+ title: "Run on schedule",
+ caption: "Refresh this crawler automatically in the background.",
+ isOn: self.$app.autoRefreshEnabled)
+ .disabled(!self.app.enabled)
+ .onChange(of: self.app.autoRefreshEnabled) { self.save() }
+ CrawlBarSwitchRow(
+ title: "Use default schedule",
+ caption: "Follow the global interval from General settings.",
+ isOn: self.usesGlobalRefreshBinding)
+ .disabled(!self.app.enabled || !self.app.autoRefreshEnabled)
+ }
+ CrawlBarControlRow(
+ title: "Custom schedule",
+ caption: "Overrides the global interval for this crawler.")
+ {
+ Picker("Custom schedule", selection: self.refreshFrequencyBinding) {
+ ForEach(RefreshFrequency.allCases, id: \.self) { frequency in
+ Text(CrawlBarFrequencyLabel.text(for: frequency)).tag(frequency)
+ }
+ }
+ .labelsHidden()
+ .frame(width: 180)
+ }
+ .disabled(!self.app.enabled || !self.app.autoRefreshEnabled || self.app.refreshFrequency == nil)
+ Text("Default schedule: \(CrawlBarFrequencyLabel.text(for: self.globalRefreshFrequency))")
+ .font(.caption)
+ .foregroundStyle(.secondary)
+ HStack(spacing: 8) {
+ if self.installation?.binaryPath == nil, self.manifest?.install != nil {
+ Button {
+ self.installApp()
+ } label: {
+ Label("Install", systemImage: "square.and.arrow.down")
+ }
+ }
+ if self.commandAvailable(self.app.preferredRefreshAction ?? "refresh") {
+ Button {
+ self.runAction(self.app.preferredRefreshAction ?? "refresh")
+ } label: {
+ Label("Sync Now", systemImage: "arrow.triangle.2.circlepath")
+ }
+ }
+ if self.commandAvailable("desktop-cache-import") {
+ Button {
+ self.runAction("desktop-cache-import")
+ } label: {
+ Label("Import Desktop", systemImage: "externaldrive.connected.to.line.below")
+ }
+ }
+ if self.commandAvailable("doctor") {
+ Button {
+ self.runAction("doctor")
+ } label: {
+ Label("Run Doctor", systemImage: "stethoscope")
+ }
+ }
+ if self.commandAvailable("unlock") {
+ Button {
+ self.runAction("unlock")
+ } label: {
+ Label("Unlock", systemImage: "key")
+ }
+ }
+ if self.nativeAppAvailable {
+ Button {
+ self.openNativeApp()
+ } label: {
+ Label("Open Source App", systemImage: "app")
+ }
+ }
+ if self.commandAvailable(self.app.preferredUpdateAction ?? "update") {
+ Button {
+ self.runAction(self.app.preferredUpdateAction ?? "update")
+ } label: {
+ Label("Update", systemImage: "square.and.arrow.down")
+ }
+ }
+ }
+ .disabled(self.runningAction != nil)
+ if let runningAction {
+ HStack(spacing: 6) {
+ ProgressView()
+ .controlSize(.small)
+ Text("Running \(runningAction)...")
+ .font(.caption)
+ .foregroundStyle(.secondary)
+ }
+ } else if let actionMessage {
+ Text(actionMessage)
+ .font(.caption)
+ .foregroundStyle(.secondary)
+ }
+ }
+ }
+
+ var gitShareSettings: some View {
+ CrawlBarPanel(title: "Git Snapshot") {
+ CrawlBarSwitchRow(
+ title: "Manage snapshot",
+ caption: "Keep a local Git export for this crawler's shareable data.",
+ isOn: self.$app.shareEnabled)
+ .onChange(of: self.app.shareEnabled) { self.save() }
+ if self.hasSnapshotRemote {
+ CrawlBarSwitchRow(
+ title: "Publish after sync",
+ caption: "Push the snapshot after a scheduled or manual sync.",
+ isOn: self.$app.shareAfterRefresh)
+ .disabled(!self.app.shareEnabled)
+ .onChange(of: self.app.shareAfterRefresh) { self.save() }
+ HStack(spacing: 8) {
+ if self.commandAvailable(self.app.preferredShareAction ?? "publish") {
+ Button {
+ self.runAction(self.app.preferredShareAction ?? "publish")
+ } label: {
+ Label("Publish Snapshot", systemImage: "arrow.up.circle")
+ }
+ }
+ if self.commandAvailable(self.app.preferredUpdateAction ?? "update") {
+ Button {
+ self.runAction(self.app.preferredUpdateAction ?? "update")
+ } label: {
+ Label("Pull Updates", systemImage: "arrow.down.circle")
+ }
+ }
+ }
+ .disabled(!self.app.shareEnabled || self.runningAction != nil)
+ }
+ if self.hasSnapshotInfo {
+ Divider()
+ if let shareRepoPath = self.shareRepoPath {
+ CrawlBarFact(label: "Share Repo", value: shareRepoPath)
+ }
+ if let shareRemote = self.shareRemote {
+ CrawlBarFact(label: "Remote", value: shareRemote)
+ }
+ if let shareBranch = self.shareBranch {
+ CrawlBarFact(label: "Branch", value: shareBranch)
+ }
+ }
+ }
+ }
+
+ @ViewBuilder
+ var cloudArchiveSettings: some View {
+ if self.commandAvailable("cloud-publish") || self.commandAvailable("remote-status") || self.commandAvailable("remote-archives") {
+ CrawlBarPanel(title: "Cloudflare Archive") {
+ CrawlBarOptionLabel(
+ title: "Remote SQLite archive",
+ caption: "Publish a compressed SQLite bundle and use the configured Worker archive for live reads.")
+ HStack(spacing: 8) {
+ if self.commandAvailable("cloud-publish") {
+ Button {
+ self.runAction("cloud-publish")
+ } label: {
+ Label("Publish Cloud", systemImage: "icloud.and.arrow.up")
+ }
+ }
+ if self.commandAvailable("remote-status") {
+ Button {
+ self.runAction("remote-status")
+ } label: {
+ Label("Remote Status", systemImage: "antenna.radiowaves.left.and.right")
+ }
+ }
+ if self.commandAvailable("remote-archives") {
+ Button {
+ self.runAction("remote-archives")
+ } label: {
+ Label("Archives", systemImage: "tray.full")
+ }
+ }
+ }
+ .disabled(self.runningAction != nil)
+ if let remote = self.status?.remote {
+ Divider()
+ if let endpoint = remote.endpoint?.nilIfBlank {
+ CrawlBarFact(label: "Endpoint", value: endpoint)
+ }
+ if let archive = remote.archive?.nilIfBlank {
+ CrawlBarFact(label: "Archive", value: archive)
+ }
+ if let sqliteBundle = self.status?.sqliteBundle {
+ CrawlBarFact(label: "Bundle", value: self.bundleSummary(sqliteBundle))
+ }
+ }
+ }
+ }
+ }
+
+ var hasSnapshotRemote: Bool {
+ self.app.shareEnabled && self.shareRemote != nil
+ }
+
+ var hasSnapshotInfo: Bool {
+ self.app.shareEnabled && (self.shareRepoPath != nil || self.shareRemote != nil || self.shareBranch != nil)
+ }
+
+ var shareRepoPath: String? {
+ self.status?.share?.repoPath?.nilIfBlank ?? self.manifest?.paths.defaultShare?.nilIfBlank
+ }
+
+ var shareRemote: String? {
+ self.status?.share?.remote?.nilIfBlank
+ }
+
+ var shareBranch: String? {
+ self.status?.share?.branch?.nilIfBlank
+ }
+
+ var usesGlobalRefreshBinding: Binding {
+ Binding(
+ get: { self.app.refreshFrequency == nil },
+ set: {
+ self.app.refreshFrequency = $0 ? nil : self.globalRefreshFrequency
+ self.save()
+ })
+ }
+
+ var refreshFrequencyBinding: Binding {
+ Binding(
+ get: { self.app.refreshFrequency ?? self.globalRefreshFrequency },
+ set: {
+ self.app.refreshFrequency = $0
+ self.save()
+ })
+ }
+
+ func commandAvailable(_ action: String) -> Bool {
+ guard self.manifest?.availability == .available else { return false }
+ return self.manifest?.commands[action] != nil && self.installation?.binaryPath != nil && self.app.enabled
+ }
+
+ var nativeAppAvailable: Bool {
+ guard let bundleIdentifier = self.manifest?.branding.bundleIdentifier?.nilIfBlank else { return false }
+ return CrawlBarNativeAppLocator.url(for: bundleIdentifier) != nil
+ }
+
+ func openNativeApp() {
+ guard let bundleIdentifier = self.manifest?.branding.bundleIdentifier?.nilIfBlank,
+ let url = CrawlBarNativeAppLocator.url(for: bundleIdentifier)
+ else { return }
+ NSWorkspace.shared.openApplication(at: url, configuration: NSWorkspace.OpenConfiguration())
+ }
+}
diff --git a/Sources/CrawlBar/Settings/AppDetailView.swift b/Sources/CrawlBar/Settings/AppDetailView.swift
new file mode 100644
index 0000000..817a61c
--- /dev/null
+++ b/Sources/CrawlBar/Settings/AppDetailView.swift
@@ -0,0 +1,182 @@
+import CrawlBarCore
+import SwiftUI
+
+struct CrawlBarAppDetailView: View {
+ @Binding var app: CrawlBarAppConfig
+ let globalRefreshFrequency: RefreshFrequency
+ let installation: CrawlAppInstallation?
+ let status: CrawlAppStatus?
+ let latestResult: CrawlCommandResult?
+ let isRefreshing: Bool
+ let runningAction: String?
+ let actionMessage: String?
+ let refreshStatus: () -> Void
+ let runAction: (String) -> Void
+ let installApp: () -> Void
+ let backupDatabases: () -> Void
+ let openDataFolder: () -> Void
+ let configValueChanged: (CrawlAppManifest.ConfigOption, String?) -> Void
+ let save: () -> Void
+ let saveDebounced: () -> Void
+
+ var manifest: CrawlAppManifest? { self.installation?.manifest ?? BuiltInCrawlApps.manifest(for: self.app.id) }
+
+ var body: some View {
+ VStack(alignment: .leading, spacing: 16) {
+ self.header
+ if self.isComingSoon {
+ self.comingSoonContent
+ .frame(maxWidth: .infinity, maxHeight: .infinity, alignment: .topLeading)
+ .padding(.top, 6)
+ } else if self.isMissingBinary {
+ self.notInstalledContent
+ .frame(maxWidth: .infinity, maxHeight: .infinity, alignment: .topLeading)
+ .padding(.top, 6)
+ } else {
+ ScrollView {
+ VStack(alignment: .leading, spacing: 20) {
+ self.statusSection
+ self.dataSection
+ self.syncSection
+ self.configurationSection
+ }
+ .frame(maxWidth: .infinity, alignment: .leading)
+ .padding(.bottom, 10)
+ .padding(.horizontal, 2)
+ }
+ }
+ }
+ .frame(
+ maxWidth: .infinity,
+ maxHeight: .infinity,
+ alignment: .topLeading)
+ }
+
+ var primaryIssue: String? {
+ if let error = self.status?.errors.first?.nilIfBlank,
+ !self.issueDuplicatesStatusSummary(error)
+ {
+ return error
+ }
+ if let warning = self.status?.warnings.first?.nilIfBlank,
+ !self.issueDuplicatesStatusSummary(warning)
+ {
+ return warning
+ }
+ guard let latestResult, !latestResult.succeeded else { return nil }
+ return latestResult.userFacingRunMessage ?? "\(Self.actionTitle(latestResult.action)) failed with exit \(latestResult.exitCode)"
+ }
+
+ func issueDuplicatesStatusSummary(_ issue: String) -> Bool {
+ guard let summary = self.status?.summary.nilIfBlank else { return false }
+ return summary.localizedCaseInsensitiveContains(issue)
+ }
+
+ var issueState: CrawlAppState {
+ self.status?.isRecoverableGraincrawlSourceFailure == true ? .stale : .error
+ }
+
+ var refreshSourceSummary: String {
+ if self.app.id == BuiltInCrawlApps.gitcrawlID {
+ return "GitHub API (remote)"
+ }
+ if self.app.id == BuiltInCrawlApps.graincrawlID {
+ switch self.app.configValues["preferred_source"]?.nilIfBlank ?? "private-api" {
+ case "desktop-cache":
+ return "Granola desktop cache"
+ case "private-api":
+ return "Granola private API (remote)"
+ default:
+ return self.app.configValues["preferred_source"] ?? "Granola source"
+ }
+ }
+ if self.manifest?.capabilities.contains(.desktopCache) == true {
+ return "Desktop cache (local)"
+ }
+ if self.manifest?.capabilities.contains(.refresh) == true {
+ return "Crawler CLI"
+ }
+ return "Not available"
+ }
+
+ var snapshotSummary: String {
+ guard self.app.shareEnabled else { return "Off" }
+ guard let share = self.status?.share else {
+ return self.shareRepoPath == nil ? "Configured, not reported" : "Local snapshot"
+ }
+ let location = share.remote?.nilIfBlank ?? share.repoPath?.nilIfBlank ?? "Local snapshot"
+ if let branch = share.branch?.nilIfBlank {
+ return "\(location) · \(branch)"
+ }
+ return location
+ }
+
+ static func actionTitle(_ action: String) -> String {
+ switch action {
+ case "refresh":
+ "Sync"
+ case "doctor":
+ "Doctor"
+ case "unlock":
+ "Unlock"
+ case "publish":
+ "Publish"
+ case "cloud-publish":
+ "Cloud Publish"
+ case "remote-status":
+ "Remote Status"
+ case "remote-archives":
+ "Remote Archives"
+ case "update":
+ "Update"
+ case "desktop-cache-import":
+ "Desktop Import"
+ default:
+ action
+ }
+ }
+
+ var effectiveState: CrawlAppState {
+ if self.isComingSoon { return .disabled }
+ if !self.app.enabled { return .disabled }
+ if self.installation?.binaryPath == nil { return .needsConfig }
+ let state = self.status?.state ?? .unknown
+ if self.status?.isRecoverableGraincrawlSourceFailure == true {
+ return .stale
+ }
+ return state
+ }
+
+ var statusFallback: String {
+ switch self.effectiveState {
+ case .disabled where self.isComingSoon:
+ "Coming soon"
+ case .needsConfig:
+ "\(self.manifest?.binary.name ?? self.app.id.rawValue) is not on PATH"
+ case .disabled:
+ "Disabled in CrawlBar"
+ default:
+ "Waiting for status"
+ }
+ }
+
+ var binarySummary: String {
+ if self.isComingSoon { return "Coming soon" }
+ return self.installation?.binaryPath == nil ? "Missing" : "Found"
+ }
+
+ var lastSyncSummary: String {
+ guard let date = self.lastSyncDate else { return "Never" }
+ return "Synced \(CrawlBarDateText.relative(date))"
+ }
+
+ private var lastSyncDate: Date? {
+ if let lastSyncAt = self.status?.lastSyncAt {
+ return lastSyncAt
+ }
+ if let modifiedAt = self.primaryDatabase?.modifiedAt {
+ return modifiedAt
+ }
+ return self.status?.databases.compactMap(\.modifiedAt).max()
+ }
+}
diff --git a/Sources/CrawlBar/Settings/GeneralSettingsView.swift b/Sources/CrawlBar/Settings/GeneralSettingsView.swift
new file mode 100644
index 0000000..c07f45b
--- /dev/null
+++ b/Sources/CrawlBar/Settings/GeneralSettingsView.swift
@@ -0,0 +1,132 @@
+import AppKit
+import CrawlBarCore
+import SwiftUI
+
+struct CrawlBarGeneralSettingsView: View {
+ @ObservedObject var model: CrawlBarSettingsModel
+
+ var body: some View {
+ ScrollView {
+ VStack(alignment: .leading, spacing: 14) {
+ HStack(alignment: .center, spacing: 14) {
+ Image(nsImage: NSApp.applicationIconImage)
+ .resizable()
+ .frame(width: 42, height: 42)
+ VStack(alignment: .leading, spacing: 4) {
+ Text("CrawlBar")
+ .font(.title3.weight(.semibold))
+ Text("Menu bar control plane for local crawler apps")
+ .font(.subheadline)
+ .foregroundStyle(.secondary)
+ .lineLimit(2)
+ }
+ Spacer()
+ Button {
+ self.model.refreshAll()
+ } label: {
+ Label("Refresh All", systemImage: "arrow.clockwise")
+ }
+ .disabled(self.model.isRefreshing)
+ }
+
+ CrawlBarPanel(title: "App") {
+ HStack(spacing: 8) {
+ Button {
+ self.model.installCLI()
+ } label: {
+ Label("Install CLI", systemImage: "terminal")
+ }
+ .disabled(self.model.isInstallingCLI)
+ Button {
+ self.model.openConfigFile()
+ } label: {
+ Label("Open Config", systemImage: "doc.text")
+ }
+ Button {
+ self.model.openLogsFolder()
+ } label: {
+ Label("Open Logs", systemImage: "folder")
+ }
+ }
+ .controlSize(.small)
+ CrawlBarFact(label: "CLI install path", value: "~/.local/bin/crawlbar")
+ CrawlBarFact(label: "Config", value: CrawlBarConfigStore().fileURL.path)
+ if let message = self.model.appActionMessage {
+ Text(message)
+ .font(.caption)
+ .foregroundStyle(.secondary)
+ }
+ }
+
+ CrawlBarPanel(title: "Scheduling") {
+ CrawlBarControlRow(
+ title: "Default schedule",
+ caption: "Used by crawlers that inherit the global sync interval.")
+ {
+ Picker("Default schedule", selection: self.$model.refreshFrequency) {
+ ForEach(RefreshFrequency.allCases, id: \.self) { frequency in
+ Text(CrawlBarFrequencyLabel.text(for: frequency)).tag(frequency)
+ }
+ }
+ .labelsHidden()
+ .frame(width: 180)
+ .onChange(of: self.model.refreshFrequency) {
+ self.model.save()
+ }
+ }
+ }
+
+ CrawlBarPanel(title: "Discovery") {
+ ForEach(self.model.manifestDirectories, id: \.self) { directory in
+ CrawlBarFact(label: "Manifest Directory", value: directory)
+ }
+ if !self.model.manifestDiagnostics.isEmpty {
+ Divider()
+ ForEach(self.model.manifestDiagnostics) { diagnostic in
+ Label {
+ Text("\(diagnostic.path): \(diagnostic.message)")
+ .font(.caption)
+ .foregroundStyle(.secondary)
+ .lineLimit(2)
+ .truncationMode(.middle)
+ .textSelection(.enabled)
+ } icon: {
+ Image(systemName: "exclamationmark.triangle")
+ .foregroundStyle(.yellow)
+ }
+ }
+ }
+ }
+
+ CrawlBarPanel(title: "Crawler Inventory") {
+ Grid(alignment: .leading, horizontalSpacing: 16, verticalSpacing: 8) {
+ GridRow {
+ CrawlBarFact(label: "Ready", value: "\(self.readyCount)")
+ CrawlBarFact(label: "Missing CLI", value: "\(self.missingCount)")
+ CrawlBarFact(label: "Coming Soon", value: "\(self.comingSoonCount)")
+ }
+ }
+ }
+
+ Spacer(minLength: 0)
+ }
+ .frame(maxWidth: .infinity, alignment: .leading)
+ .padding(.vertical, 4)
+ .padding(.bottom, CrawlBarSettingsLayout.detailBottomPadding)
+ }
+ .padding(.horizontal, CrawlBarSettingsLayout.detailHorizontalPadding)
+ .padding(.vertical, CrawlBarSettingsLayout.detailVerticalPadding)
+ }
+
+ private var readyCount: Int {
+ self.model.installations.values.filter { $0.manifest.availability == .available && $0.binaryPath != nil }.count
+ }
+
+ private var missingCount: Int {
+ self.model.installations.values.filter { $0.manifest.availability == .available && $0.binaryPath == nil }.count
+ }
+
+ private var comingSoonCount: Int {
+ self.model.installations.values.filter { $0.manifest.availability == .comingSoon }.count
+ }
+}
diff --git a/Sources/CrawlBar/Settings/SettingsControls.swift b/Sources/CrawlBar/Settings/SettingsControls.swift
new file mode 100644
index 0000000..32aa290
--- /dev/null
+++ b/Sources/CrawlBar/Settings/SettingsControls.swift
@@ -0,0 +1,187 @@
+import CrawlBarCore
+import SwiftUI
+
+struct CrawlBarDetailSection: View {
+ let title: String
+ let content: Content
+
+ init(title: String, @ViewBuilder content: () -> Content) {
+ self.title = title
+ self.content = content()
+ }
+
+ var body: some View {
+ VStack(alignment: .leading, spacing: 10) {
+ Text(self.title)
+ .font(.headline.weight(.semibold))
+ VStack(alignment: .leading, spacing: 14) {
+ self.content
+ }
+ .frame(maxWidth: .infinity, alignment: .leading)
+ }
+ }
+}
+
+struct CrawlBarPanel: View {
+ var title: String?
+ var caption: String?
+ @ViewBuilder var content: Content
+
+ init(title: String? = nil, caption: String? = nil, @ViewBuilder content: () -> Content) {
+ self.title = title
+ self.caption = caption
+ self.content = content()
+ }
+
+ var body: some View {
+ VStack(alignment: .leading, spacing: 8) {
+ if let title {
+ Text(title)
+ .font(.subheadline.weight(.semibold))
+ .foregroundStyle(.secondary)
+ }
+ if let caption {
+ Text(caption)
+ .font(.footnote)
+ .foregroundStyle(.tertiary)
+ .fixedSize(horizontal: false, vertical: true)
+ }
+ VStack(alignment: .leading, spacing: 10) {
+ self.content
+ }
+ .padding(14)
+ .frame(maxWidth: .infinity, alignment: .leading)
+ .background(.quaternary.opacity(0.38), in: RoundedRectangle(cornerRadius: 12, style: .continuous))
+ .overlay {
+ RoundedRectangle(cornerRadius: 12, style: .continuous)
+ .strokeBorder(.white.opacity(0.055))
+ }
+ }
+ }
+}
+
+struct CrawlBarFact: View {
+ let label: String
+ let value: String
+
+ var body: some View {
+ VStack(alignment: .leading, spacing: 3) {
+ Text(self.label)
+ .font(.caption)
+ .foregroundStyle(.secondary)
+ Text(self.value)
+ .font(.callout)
+ .lineLimit(2)
+ .textSelection(.enabled)
+ }
+ .frame(maxWidth: .infinity, alignment: .leading)
+ }
+}
+
+struct CrawlBarIssueBanner: View {
+ let message: String
+ let state: CrawlAppState
+
+ var body: some View {
+ HStack(alignment: .top, spacing: 8) {
+ Image(systemName: "exclamationmark.triangle.fill")
+ .font(.caption)
+ .foregroundStyle(self.color)
+ Text(self.message)
+ .font(.caption)
+ .foregroundStyle(self.color)
+ .lineLimit(3)
+ .textSelection(.enabled)
+ }
+ .padding(8)
+ .frame(maxWidth: .infinity, alignment: .leading)
+ .background(self.color.opacity(0.08))
+ .clipShape(RoundedRectangle(cornerRadius: 6, style: .continuous))
+ }
+
+ private var color: Color {
+ self.state == .stale ? .yellow : .red
+ }
+}
+
+struct CrawlBarControlRow: View {
+ let title: String
+ let caption: String?
+ @ViewBuilder var content: Content
+
+ init(title: String, caption: String? = nil, @ViewBuilder content: () -> Content) {
+ self.title = title
+ self.caption = caption
+ self.content = content()
+ }
+
+ var body: some View {
+ HStack(alignment: .firstTextBaseline, spacing: 16) {
+ VStack(alignment: .leading, spacing: 3) {
+ Text(self.title)
+ .font(.callout)
+ if let caption {
+ Text(caption)
+ .font(.caption)
+ .foregroundStyle(.secondary)
+ .fixedSize(horizontal: false, vertical: true)
+ }
+ }
+ .frame(maxWidth: .infinity, alignment: .leading)
+ self.content
+ }
+ }
+}
+
+struct CrawlBarSwitchRow: View {
+ let title: String
+ let caption: String
+ @Binding var isOn: Bool
+
+ var body: some View {
+ HStack(alignment: .center, spacing: 16) {
+ CrawlBarOptionLabel(title: self.title, caption: self.caption)
+ .frame(maxWidth: .infinity, alignment: .leading)
+ Toggle(self.title, isOn: self.$isOn)
+ .labelsHidden()
+ .toggleStyle(.switch)
+ .controlSize(.mini)
+ }
+ }
+}
+
+struct CrawlBarOptionLabel: View {
+ let title: String
+ let caption: String
+
+ var body: some View {
+ VStack(alignment: .leading, spacing: 3) {
+ Text(self.title)
+ .font(.callout)
+ Text(self.caption)
+ .font(.caption)
+ .foregroundStyle(.secondary)
+ .fixedSize(horizontal: false, vertical: true)
+ }
+ }
+}
+
+struct CrawlBarMetricRow: View {
+ let label: String
+ let value: String
+
+ var body: some View {
+ HStack(alignment: .firstTextBaseline, spacing: 8) {
+ Text(self.label)
+ .font(.callout)
+ .foregroundStyle(.secondary)
+ .lineLimit(1)
+ Spacer(minLength: 8)
+ Text(self.value)
+ .font(.callout.weight(.semibold))
+ .monospacedDigit()
+ }
+ .padding(.vertical, 7)
+ .frame(maxWidth: .infinity, alignment: .leading)
+ }
+}
diff --git a/Sources/CrawlBar/Settings/SettingsLayout.swift b/Sources/CrawlBar/Settings/SettingsLayout.swift
new file mode 100644
index 0000000..872bad0
--- /dev/null
+++ b/Sources/CrawlBar/Settings/SettingsLayout.swift
@@ -0,0 +1,10 @@
+import Foundation
+
+enum CrawlBarSettingsLayout {
+ static let minWindowWidth: CGFloat = 1120
+ static let minWindowHeight: CGFloat = 790
+ static let sidebarWidth: CGFloat = 250
+ static let detailHorizontalPadding: CGFloat = 22
+ static let detailVerticalPadding: CGFloat = 18
+ static let detailBottomPadding: CGFloat = 16
+}
diff --git a/Sources/CrawlBar/Settings/SettingsModel.swift b/Sources/CrawlBar/Settings/SettingsModel.swift
new file mode 100644
index 0000000..b854b82
--- /dev/null
+++ b/Sources/CrawlBar/Settings/SettingsModel.swift
@@ -0,0 +1,80 @@
+import CrawlBarCore
+import Foundation
+
+@MainActor
+final class CrawlBarSettingsModel: NSObject, ObservableObject {
+ @Published var apps: [CrawlBarAppConfig] = []
+ @Published var refreshFrequency: RefreshFrequency = .fifteenMinutes
+ @Published var selectedSidebarItem: CrawlBarSettingsSidebarItem?
+ @Published var statuses: [CrawlAppID: CrawlAppStatus] = [:]
+ @Published var installations: [CrawlAppID: CrawlAppInstallation] = [:]
+ @Published var isRefreshing = false
+ @Published var isInstallingCLI = false
+ @Published var appActionMessage: String?
+ @Published var runningActions: [CrawlAppID: String] = [:]
+ @Published var actionMessages: [CrawlAppID: String] = [:]
+ @Published var recentResults: [CrawlAppID: CrawlCommandResult] = [:]
+ @Published var lastError: String?
+ @Published var manifestDiagnostics: [CrawlManifestDiagnostic] = []
+ @Published var isLoading = false
+ @Published var manifestDirectories: [String] = ["~/.crawlbar/apps"]
+
+ var refreshTask: Task?
+ var loadTask: Task?
+ var pendingSaveTask: Task?
+ var refreshGeneration = UUID()
+ var loadGeneration = UUID()
+ var recentResultsGeneration = UUID()
+ var hasLoadedSnapshot = false
+ var clearedNativeSecretIDsByAppID: [CrawlAppID: Set] = [:]
+ let store = CrawlBarConfigStore()
+ let registry = CrawlAppRegistry()
+ let runner: CrawlCommandRunner
+ let statusService: CrawlStatusService
+ let nativeConfigStore = CrawlNativeConfigStore()
+ let installer = CrawlInstaller()
+ let logStore = CrawlActionLogStore()
+
+ var selectedAppID: CrawlAppID? {
+ get {
+ guard case let .crawler(id) = self.selectedSidebarItem else { return nil }
+ return id
+ }
+ set {
+ self.selectedSidebarItem = newValue.map(CrawlBarSettingsSidebarItem.crawler)
+ }
+ }
+
+ init(loadImmediately: Bool = true) {
+ let runner = CrawlCommandRunner()
+ self.runner = runner
+ self.statusService = CrawlStatusService(runner: runner)
+ super.init()
+ NotificationCenter.default.addObserver(
+ self,
+ selector: #selector(Self.statusesDidChange(_:)),
+ name: .crawlBarStatusesDidChange,
+ object: nil)
+ if loadImmediately {
+ self.load()
+ } else {
+ self.selectedSidebarItem = .general
+ }
+ }
+
+ deinit {
+ NotificationCenter.default.removeObserver(self)
+ }
+
+ var sidebarSelectionIsValid: Bool {
+ switch self.selectedSidebarItem {
+ case .general:
+ true
+ case .crawler(let id):
+ self.apps.contains { $0.id == id }
+ case nil:
+ false
+ }
+ }
+
+}
diff --git a/Sources/CrawlBar/Settings/SettingsModelFileActions.swift b/Sources/CrawlBar/Settings/SettingsModelFileActions.swift
new file mode 100644
index 0000000..9e63d8b
--- /dev/null
+++ b/Sources/CrawlBar/Settings/SettingsModelFileActions.swift
@@ -0,0 +1,124 @@
+import AppKit
+import CrawlBarCore
+import Foundation
+
+extension CrawlBarSettingsModel {
+ func installApp(_ appID: CrawlAppID) {
+ guard let installation = self.installations[appID] else { return }
+ self.runningActions[appID] = "install"
+ self.actionMessages[appID] = "Installing \(installation.manifest.binary.name)..."
+ let installer = self.installer
+ let logStore = self.logStore
+ let registry = self.registry
+ Task.detached {
+ let message: String
+ do {
+ let result = try installer.install(installation)
+ _ = try? logStore.save(result)
+ message = "\(installation.manifest.binary.name) installed"
+ } catch {
+ message = error.localizedDescription
+ }
+ let installations = (try? registry.installations(includeDisabled: true)) ?? []
+ await MainActor.run {
+ let installationsByID = Dictionary(uniqueKeysWithValues: installations.map { ($0.id, $0) })
+ self.installations = installationsByID
+ self.apps = Self.sortedAppConfigs(self.apps, installationsByID: installationsByID)
+ self.runningActions[appID] = nil
+ self.actionMessages[appID] = message
+ }
+ }
+ }
+
+ func backupDatabases(_ appID: CrawlAppID) {
+ guard let status = self.statuses[appID] else { return }
+ self.runningActions[appID] = "backup"
+ self.actionMessages[appID] = "Backing up databases..."
+ Task.detached {
+ let message: String
+ do {
+ let backup = try CrawlDatabaseBackupStore.backup(status: status)
+ message = "Backed up \(backup.files.count) database file(s)"
+ } catch {
+ message = error.localizedDescription
+ }
+ await MainActor.run {
+ self.runningActions[appID] = nil
+ self.actionMessages[appID] = message
+ }
+ }
+ }
+
+ func openDataFolder(_ appID: CrawlAppID) {
+ guard let status = self.statuses[appID],
+ let path = status.databases.first(where: { $0.isPrimary })?.path ?? status.databasePath
+ else { return }
+ NSWorkspace.shared.open(URL(fileURLWithPath: PathExpander.expandHome(path)).deletingLastPathComponent())
+ }
+
+ func openConfigFile() {
+ NSWorkspace.shared.activateFileViewerSelecting([self.store.fileURL])
+ }
+
+ func openLogsFolder() {
+ NSWorkspace.shared.open(CrawlActionLogStore.defaultDirectory())
+ }
+
+ func installCLI() {
+ self.isInstallingCLI = true
+ self.appActionMessage = "Installing crawlbar CLI..."
+ Task.detached {
+ let message: String
+ do {
+ let path = try Self.installBundledCLI()
+ message = "Installed crawlbar CLI at \(path)"
+ } catch {
+ message = error.localizedDescription
+ }
+ await MainActor.run {
+ self.isInstallingCLI = false
+ self.appActionMessage = message
+ }
+ }
+ }
+
+ nonisolated static func installBundledCLI() throws -> String {
+ let fileManager = FileManager.default
+ let sourceCandidates = Self.cliSourceCandidates()
+ guard let source = sourceCandidates.first(where: { fileManager.isExecutableFile(atPath: $0.path) }) else {
+ throw CrawlBarSettingsError.cliHelperMissing
+ }
+ let destinationDirectory = URL(fileURLWithPath: PathExpander.expandHome("~/.local/bin"), isDirectory: true)
+ let destination = destinationDirectory.appendingPathComponent("crawlbar")
+ try fileManager.createDirectory(at: destinationDirectory, withIntermediateDirectories: true)
+ if fileManager.fileExists(atPath: destination.path) {
+ try fileManager.removeItem(at: destination)
+ }
+ try fileManager.copyItem(at: source, to: destination)
+ try fileManager.setAttributes([.posixPermissions: 0o755], ofItemAtPath: destination.path)
+ return destination.path
+ }
+
+ nonisolated static func cliSourceCandidates() -> [URL] {
+ var candidates: [URL] = [
+ Bundle.main.bundleURL.appendingPathComponent("Contents/Helpers/crawlbar"),
+ ]
+ if let executableDirectory = Bundle.main.executableURL?.deletingLastPathComponent() {
+ candidates.append(executableDirectory.appendingPathComponent("crawlbarctl"))
+ candidates.append(executableDirectory.deletingLastPathComponent().appendingPathComponent("debug/crawlbarctl"))
+ candidates.append(executableDirectory.deletingLastPathComponent().appendingPathComponent("release/crawlbarctl"))
+ }
+ return candidates
+ }
+}
+
+private enum CrawlBarSettingsError: LocalizedError {
+ case cliHelperMissing
+
+ var errorDescription: String? {
+ switch self {
+ case .cliHelperMissing:
+ "Could not find bundled crawlbar CLI helper"
+ }
+ }
+}
diff --git a/Sources/CrawlBar/Settings/SettingsModelPersistence.swift b/Sources/CrawlBar/Settings/SettingsModelPersistence.swift
new file mode 100644
index 0000000..e507a02
--- /dev/null
+++ b/Sources/CrawlBar/Settings/SettingsModelPersistence.swift
@@ -0,0 +1,184 @@
+import CrawlBarCore
+import Foundation
+
+extension CrawlBarSettingsModel {
+ func load() {
+ do {
+ self.apply(try Self.loadSnapshot())
+ } catch {
+ CrawlBarLog.config.error("Settings load failed: \(error.localizedDescription, privacy: .public)")
+ self.lastError = error.localizedDescription
+ }
+ }
+
+ func loadForPresentation(onLoaded: @escaping @MainActor () -> Void) {
+ self.loadTask?.cancel()
+ let generation = UUID()
+ self.loadGeneration = generation
+ self.isLoading = true
+ self.lastError = nil
+ self.loadTask = Task.detached {
+ let snapshot = Result { try Self.loadSnapshot() }
+ await MainActor.run {
+ guard self.loadGeneration == generation else { return }
+ self.isLoading = false
+ self.loadTask = nil
+ switch snapshot {
+ case .success(let snapshot):
+ self.apply(snapshot)
+ onLoaded()
+ case .failure(let error):
+ CrawlBarLog.config.error("Settings load failed: \(error.localizedDescription, privacy: .public)")
+ self.lastError = error.localizedDescription
+ }
+ }
+ }
+ }
+
+ func save() {
+ self.pendingSaveTask?.cancel()
+ self.pendingSaveTask = nil
+ self.persist()
+ }
+
+ func scrubSecretConfigValues() {
+ var scrubbedApps = self.apps
+ for index in scrubbedApps.indices {
+ guard let manifest = self.installations[scrubbedApps[index].id]?.manifest else { continue }
+ for option in manifest.configOptions where option.kind == .secret {
+ scrubbedApps[index].configValues.removeValue(forKey: option.id)
+ }
+ }
+ self.apps = scrubbedApps
+ }
+
+ func saveDebounced() {
+ self.pendingSaveTask?.cancel()
+ self.pendingSaveTask = Task { @MainActor in
+ try? await Task.sleep(nanoseconds: 700_000_000)
+ guard !Task.isCancelled else { return }
+ self.persist()
+ self.pendingSaveTask = nil
+ }
+ }
+
+ func configValueDidChange(appID: CrawlAppID, option: CrawlAppManifest.ConfigOption, value: String?) {
+ if option.kind == .secret {
+ if value?.nilIfBlank == nil {
+ self.clearedNativeSecretIDsByAppID[appID, default: []].insert(option.id)
+ } else {
+ self.clearedNativeSecretIDsByAppID[appID]?.remove(option.id)
+ }
+ }
+ self.saveDebounced()
+ }
+
+ func persist() {
+ guard self.hasLoadedSnapshot, !self.isLoading else { return }
+ do {
+ let config = CrawlBarConfig(
+ refreshFrequency: self.refreshFrequency,
+ manifestDirectories: self.manifestDirectories,
+ apps: self.apps)
+ try self.store.save(
+ config,
+ clearMissingSecretIDsByAppID: self.clearedNativeSecretIDsByAppID)
+ try self.nativeConfigStore.write(
+ config: config,
+ clearMissingSecretIDsByAppID: self.clearedNativeSecretIDsByAppID)
+ self.clearedNativeSecretIDsByAppID = [:]
+ self.lastError = nil
+ CrawlBarStateBroadcast.configDidChange()
+ } catch {
+ CrawlBarLog.config.error("Settings save failed: \(error.localizedDescription, privacy: .public)")
+ self.lastError = error.localizedDescription
+ }
+ }
+
+ func apply(_ snapshot: CrawlBarSettingsSnapshot) {
+ self.hasLoadedSnapshot = true
+ self.apps = snapshot.apps
+ self.refreshFrequency = snapshot.refreshFrequency
+ self.manifestDirectories = snapshot.manifestDirectories
+ self.installations = snapshot.installations
+ self.recentResults = snapshot.recentResults
+ self.manifestDiagnostics = snapshot.manifestDiagnostics
+ if !self.sidebarSelectionIsValid {
+ self.selectedSidebarItem = self.apps.first.map { .crawler($0.id) } ?? .general
+ }
+ self.lastError = nil
+ }
+
+ nonisolated static func loadSnapshot() throws -> CrawlBarSettingsSnapshot {
+ let store = CrawlBarConfigStore()
+ let registry = CrawlAppRegistry()
+ let nativeConfigStore = CrawlNativeConfigStore()
+ let logStore = CrawlActionLogStore()
+ let config = try store.loadOrCreateDefault()
+ let loadedInstallations = try registry.installations(includeDisabled: true)
+ let manifests = Dictionary(uniqueKeysWithValues: loadedInstallations.map { ($0.id, $0.manifest) })
+ let appConfigsByID = Dictionary(uniqueKeysWithValues: config.apps.map { ($0.id, $0) })
+ let installationsByID = Dictionary(uniqueKeysWithValues: loadedInstallations.map { ($0.id, $0) })
+ let apps = loadedInstallations.map { installation in
+ let appConfig = appConfigsByID[installation.id] ?? CrawlBarAppConfig(
+ id: installation.id,
+ enabled: installation.manifest.availability == .available,
+ showInMenuBar: installation.manifest.availability == .available)
+ guard let manifest = manifests[appConfig.id] else { return appConfig }
+ var copy = appConfig
+ copy.configValues = nativeConfigStore.resolvedConfigValues(
+ appConfig: appConfig,
+ manifest: manifest,
+ includeSecrets: false)
+ return copy
+ }
+ return CrawlBarSettingsSnapshot(
+ apps: Self.sortedAppConfigs(apps, installationsByID: installationsByID),
+ refreshFrequency: config.refreshFrequency,
+ manifestDirectories: config.manifestDirectories,
+ installations: installationsByID,
+ recentResults: Self.recentResults(logStore: logStore),
+ manifestDiagnostics: CrawlManifestCatalog().diagnostics(config: config))
+ }
+
+ nonisolated static func sortedAppConfigs(
+ _ apps: [CrawlBarAppConfig],
+ installationsByID: [CrawlAppID: CrawlAppInstallation])
+ -> [CrawlBarAppConfig]
+ {
+ let originalIndex = Dictionary(uniqueKeysWithValues: apps.enumerated().map { ($0.element.id, $0.offset) })
+ return apps.sorted { lhs, rhs in
+ let lhsRank = Self.sidebarRank(installation: installationsByID[lhs.id])
+ let rhsRank = Self.sidebarRank(installation: installationsByID[rhs.id])
+ if lhsRank != rhsRank { return lhsRank < rhsRank }
+ return (originalIndex[lhs.id] ?? Int.max) < (originalIndex[rhs.id] ?? Int.max)
+ }
+ }
+
+ nonisolated static func recentResults(logStore: CrawlActionLogStore) -> [CrawlAppID: CrawlCommandResult] {
+ var resultsByApp: [CrawlAppID: CrawlCommandResult] = [:]
+ for result in logStore.recentResults(limit: 200).sorted(by: { $0.finishedAt > $1.finishedAt }) {
+ if resultsByApp[result.appID] == nil {
+ resultsByApp[result.appID] = result
+ }
+ }
+ return resultsByApp
+ }
+
+ nonisolated static func sidebarRank(installation: CrawlAppInstallation?) -> Int {
+ guard let installation else { return 4 }
+ if installation.manifest.availability == .comingSoon { return 3 }
+ if installation.binaryPath != nil { return 0 }
+ if installation.manifest.install != nil { return 1 }
+ return 2
+ }
+}
+
+struct CrawlBarSettingsSnapshot: Sendable {
+ let apps: [CrawlBarAppConfig]
+ let refreshFrequency: RefreshFrequency
+ let manifestDirectories: [String]
+ let installations: [CrawlAppID: CrawlAppInstallation]
+ let recentResults: [CrawlAppID: CrawlCommandResult]
+ let manifestDiagnostics: [CrawlManifestDiagnostic]
+}
diff --git a/Sources/CrawlBar/Settings/SettingsModelStatusActions.swift b/Sources/CrawlBar/Settings/SettingsModelStatusActions.swift
new file mode 100644
index 0000000..e726cae
--- /dev/null
+++ b/Sources/CrawlBar/Settings/SettingsModelStatusActions.swift
@@ -0,0 +1,206 @@
+import CrawlBarCore
+import Foundation
+
+extension CrawlBarSettingsModel {
+ func refreshAll() {
+ self.loadRecentResultsAsync()
+ self.refreshTask?.cancel()
+ let generation = UUID()
+ self.refreshGeneration = generation
+ self.isRefreshing = true
+ let appsForStatus = self.apps
+ let registry = self.registry
+ let statusService = self.statusService
+ self.refreshTask = Task.detached {
+ let installations = (try? registry.installationsForStatus(includeDisabled: true)) ?? []
+ let appConfigsByID = Dictionary(uniqueKeysWithValues: appsForStatus.map { ($0.id, $0) })
+ let statusInstallations = installations.filter { installation in
+ guard installation.manifest.availability == .available else { return false }
+ return appConfigsByID[installation.id]?.enabled ?? installation.enabled
+ }
+ await MainActor.run {
+ guard self.refreshGeneration == generation else { return }
+ let installationsByID = Dictionary(uniqueKeysWithValues: installations.map { ($0.id, $0) })
+ self.installations = installationsByID
+ self.apps = Self.sortedAppConfigs(self.apps, installationsByID: installationsByID)
+ }
+ let partitioned = Self.partitionStatuses(installations: statusInstallations, statusService: statusService)
+ if !partitioned.immediate.isEmpty {
+ await MainActor.run {
+ guard self.refreshGeneration == generation else { return }
+ for status in partitioned.immediate {
+ self.statuses[status.appID] = status
+ }
+ CrawlBarStateBroadcast.statusesDidChange(Dictionary(uniqueKeysWithValues: partitioned.immediate.map { ($0.appID, $0) }))
+ }
+ }
+ await withTaskGroup(of: CrawlAppStatus.self) { group in
+ for installation in partitioned.commandInstallations {
+ group.addTask {
+ guard !Task.isCancelled else {
+ return CrawlAppStatus(appID: installation.id, state: .unknown, summary: "Refresh cancelled")
+ }
+ return statusService.status(for: installation, timeoutSeconds: 5)
+ }
+ }
+ for await status in group {
+ if Task.isCancelled { break }
+ await MainActor.run {
+ guard self.refreshGeneration == generation else { return }
+ self.statuses[status.appID] = status
+ CrawlBarStateBroadcast.statusesDidChange([status.appID: status])
+ }
+ }
+ }
+ await MainActor.run {
+ guard self.refreshGeneration == generation else { return }
+ self.isRefreshing = false
+ self.refreshTask = nil
+ }
+ }
+ }
+
+ func runAction(_ action: String, appID: CrawlAppID) {
+ guard let installation = self.installations[appID] else { return }
+ self.runningActions[appID] = action
+ self.actionMessages[appID] = "Running \(Self.actionTitle(action))..."
+ let runner = self.runner
+ let statusService = self.statusService
+ let logStore = self.logStore
+ let registry = self.registry
+ Task.detached {
+ let actionInstallation = (try? registry.installation(for: appID, includeSecrets: true)) ?? installation
+ let message: String
+ var actionError: CrawlAppStatus?
+ do {
+ CrawlBarLog.actions.notice("Running \(action, privacy: .public) for \(appID.rawValue, privacy: .public) from settings")
+ let result = try runner.run(installation: actionInstallation, action: action, timeoutSeconds: 600)
+ _ = try? logStore.save(result)
+ message = result.exitCode == 0
+ ? "\(Self.actionTitle(action)) finished"
+ : "\(Self.actionTitle(action)) failed with exit \(result.exitCode)"
+ if !result.succeeded {
+ CrawlBarLog.actions.error(
+ "\(action, privacy: .public) for \(appID.rawValue, privacy: .public) failed with exit \(result.exitCode)")
+ actionError = Self.actionFailureStatus(result)
+ }
+ } catch {
+ CrawlBarLog.actions.error(
+ "\(action, privacy: .public) for \(appID.rawValue, privacy: .public) threw: \(error.localizedDescription, privacy: .public)")
+ message = error.localizedDescription
+ actionError = Self.actionFailureStatus(appID: appID, action: action, message: error.localizedDescription)
+ }
+ let refreshedStatus = statusService.status(for: actionInstallation, timeoutSeconds: 5)
+ await MainActor.run {
+ let status = actionError.map {
+ Self.actionFailureStatus($0, refreshedStatus: refreshedStatus, currentStatus: self.statuses[appID])
+ } ?? refreshedStatus
+ self.statuses[appID] = status
+ CrawlBarStateBroadcast.statusesDidChange([appID: status])
+ self.runningActions[appID] = nil
+ self.actionMessages[appID] = message
+ self.loadRecentResults()
+ }
+ }
+ }
+
+ func loadRecentResults() {
+ self.recentResultsGeneration = UUID()
+ self.recentResults = Self.recentResults(logStore: self.logStore)
+ }
+
+ func loadRecentResultsAsync() {
+ let logStore = self.logStore
+ let generation = UUID()
+ self.recentResultsGeneration = generation
+ Task.detached {
+ let results = Self.recentResults(logStore: logStore)
+ await MainActor.run {
+ guard self.recentResultsGeneration == generation else { return }
+ self.recentResults = results
+ }
+ }
+ }
+
+ func mergeStatuses(_ incoming: [CrawlAppID: CrawlAppStatus]) {
+ for (appID, status) in incoming {
+ self.statuses[appID] = status
+ }
+ }
+
+ @objc func statusesDidChange(_ notification: Notification) {
+ guard let statuses = CrawlBarStateBroadcast.statuses(from: notification) else { return }
+ self.mergeStatuses(statuses)
+ }
+
+ nonisolated static func partitionStatuses(
+ installations: [CrawlAppInstallation],
+ statusService: CrawlStatusService)
+ -> (immediate: [CrawlAppStatus], commandInstallations: [CrawlAppInstallation])
+ {
+ var immediate: [CrawlAppStatus] = []
+ var commandInstallations: [CrawlAppInstallation] = []
+ for installation in installations {
+ if let status = statusService.immediateStatus(for: installation) {
+ immediate.append(status)
+ } else {
+ commandInstallations.append(installation)
+ }
+ }
+ return (immediate, commandInstallations)
+ }
+
+ nonisolated static func actionTitle(_ action: String) -> String {
+ switch action {
+ case "refresh":
+ "Sync"
+ case "doctor":
+ "Doctor"
+ case "unlock":
+ "Unlock"
+ case "publish":
+ "Publish"
+ case "cloud-publish":
+ "Cloud Publish"
+ case "remote-status":
+ "Remote Status"
+ case "remote-archives":
+ "Remote Archives"
+ case "update":
+ "Update"
+ case "desktop-cache-import":
+ "Desktop Import"
+ default:
+ action
+ }
+ }
+
+ nonisolated static func actionFailureStatus(_ result: CrawlCommandResult) -> CrawlAppStatus {
+ let fallback = "\(result.action) failed with exit \(result.exitCode)"
+ return CrawlAppStatus.commandFailure(
+ appID: result.appID,
+ action: result.action,
+ message: result.stderr.nilIfBlank ?? result.stdout.nilIfBlank,
+ fallback: fallback)
+ }
+
+ nonisolated static func actionFailureStatus(appID: CrawlAppID, action: String, message: String) -> CrawlAppStatus {
+ CrawlAppStatus.commandFailure(
+ appID: appID,
+ action: action,
+ message: message,
+ fallback: "\(action) failed")
+ }
+
+ nonisolated static func actionFailureStatus(
+ _ failure: CrawlAppStatus,
+ refreshedStatus: CrawlAppStatus?,
+ currentStatus: CrawlAppStatus?)
+ -> CrawlAppStatus
+ {
+ guard let metadataStatus = CrawlAppStatus.richestMetadataStatus(refreshedStatus, fallback: currentStatus) else {
+ return failure
+ }
+ return metadataStatus.mergingActionFailure(failure)
+ }
+}
diff --git a/Sources/CrawlBar/Settings/SettingsResourceViews.swift b/Sources/CrawlBar/Settings/SettingsResourceViews.swift
new file mode 100644
index 0000000..bcf7e2c
--- /dev/null
+++ b/Sources/CrawlBar/Settings/SettingsResourceViews.swift
@@ -0,0 +1,176 @@
+import CrawlBarCore
+import SwiftUI
+
+struct CrawlBarDatabaseRow: View {
+ let database: CrawlDatabaseResource
+
+ var body: some View {
+ HStack(alignment: .top, spacing: 10) {
+ Image(systemName: self.iconName)
+ .font(.body)
+ .foregroundStyle(self.database.isPrimary ? .blue : .secondary)
+ .frame(width: 18, height: 22)
+ VStack(alignment: .leading, spacing: 5) {
+ HStack(spacing: 6) {
+ Text(self.database.label)
+ .font(.callout.weight(.medium))
+ .lineLimit(1)
+ if self.database.isPrimary {
+ Text("Primary")
+ .font(.caption2.weight(.medium))
+ .foregroundStyle(.blue)
+ .padding(.horizontal, 5)
+ .padding(.vertical, 2)
+ .background(.blue.opacity(0.12))
+ .clipShape(Capsule())
+ }
+ }
+ Text(self.subtitle)
+ .font(.caption)
+ .foregroundStyle(.secondary)
+ .lineLimit(1)
+ .truncationMode(.middle)
+ .textSelection(.enabled)
+ if !self.database.counts.isEmpty {
+ HStack(spacing: 8) {
+ ForEach(self.database.counts.prefix(3)) { count in
+ Text("\(count.value) \(count.label.lowercased())")
+ .font(.caption2)
+ .foregroundStyle(.tertiary)
+ .monospacedDigit()
+ .lineLimit(1)
+ }
+ }
+ }
+ }
+ Spacer(minLength: 8)
+ VStack(alignment: .trailing, spacing: 4) {
+ if let bytes = self.database.bytes {
+ Text(CrawlBarFileSizeText.string(fromByteCount: Int64(bytes)))
+ .font(.caption)
+ .foregroundStyle(.secondary)
+ .monospacedDigit()
+ }
+ if let modifiedAt = self.database.modifiedAt {
+ Text(CrawlBarDateText.relative(modifiedAt))
+ .font(.caption2)
+ .foregroundStyle(.tertiary)
+ .lineLimit(1)
+ }
+ }
+ }
+ .padding(.vertical, 8)
+ }
+
+ private var subtitle: String {
+ let pieces = [
+ self.database.role,
+ self.database.path,
+ ].compactMap { $0?.nilIfBlank }
+ return pieces.isEmpty ? self.database.kind.rawValue : pieces.joined(separator: " · ")
+ }
+
+ private var iconName: String {
+ switch self.database.kind {
+ case .sqlite:
+ "internaldrive"
+ case .cache:
+ "externaldrive.connected.to.line.below"
+ case .logical:
+ "square.stack.3d.up"
+ case .remote, .d1, .cloudflareD1:
+ "cloud"
+ case .sqliteBundle:
+ "archivebox"
+ }
+ }
+}
+
+struct CrawlBarConfigOptionField: View {
+ let option: CrawlAppManifest.ConfigOption
+ @Binding var value: String
+ var disabledReason: String?
+
+ var body: some View {
+ CrawlBarControlRow(title: self.option.label, caption: self.caption) {
+ self.control
+ }
+ .disabled(self.disabledReason != nil)
+ }
+
+ @ViewBuilder
+ private var control: some View {
+ switch self.option.kind {
+ case .secret:
+ HStack(spacing: 8) {
+ SecureField(self.option.placeholder ?? "Value", text: self.$value)
+ .textFieldStyle(.roundedBorder)
+ .frame(width: 300)
+ Button {
+ self.value = ""
+ } label: {
+ Image(systemName: "key.slash")
+ }
+ .buttonStyle(.borderless)
+ .accessibilityLabel("Clear saved secret")
+ }
+ case .boolean:
+ Toggle("", isOn: self.booleanBinding)
+ .labelsHidden()
+ .toggleStyle(.switch)
+ .controlSize(.mini)
+ case .choice:
+ Picker("Value", selection: self.$value) {
+ ForEach(self.choices, id: \.self) { choice in
+ Text(choice).tag(choice)
+ }
+ }
+ .labelsHidden()
+ .frame(width: 220)
+ case .string:
+ TextField(self.option.placeholder ?? "Value", text: self.$value)
+ .textFieldStyle(.roundedBorder)
+ .frame(width: 300)
+ case .number:
+ TextField(self.option.placeholder ?? "0", text: self.$value)
+ .textFieldStyle(.roundedBorder)
+ .frame(width: 120)
+ }
+ }
+
+ private var caption: String? {
+ [
+ self.disabledReason?.nilIfBlank,
+ self.option.help?.nilIfBlank,
+ self.metadata,
+ ].compactMap { $0 }.joined(separator: "\n").nilIfBlank
+ }
+
+ private var metadata: String? {
+ [
+ self.option.envVar?.nilIfBlank,
+ self.option.configKey?.nilIfBlank,
+ ].compactMap { $0 }.joined(separator: " ").nilIfBlank
+ }
+
+ private var choices: [String] {
+ var resolved = self.option.choices
+ if let defaultValue = self.option.defaultValue?.nilIfBlank,
+ !resolved.contains(defaultValue)
+ {
+ resolved.insert(defaultValue, at: 0)
+ }
+ if let currentValue = self.value.nilIfBlank,
+ !resolved.contains(currentValue)
+ {
+ resolved.insert(currentValue, at: 0)
+ }
+ return resolved
+ }
+
+ private var booleanBinding: Binding {
+ Binding(
+ get: { ["1", "true", "yes", "on"].contains(self.value.lowercased()) },
+ set: { self.value = $0 ? "true" : "false" })
+ }
+}
diff --git a/Sources/CrawlBar/Settings/SettingsSidebar.swift b/Sources/CrawlBar/Settings/SettingsSidebar.swift
new file mode 100644
index 0000000..fb3e32b
--- /dev/null
+++ b/Sources/CrawlBar/Settings/SettingsSidebar.swift
@@ -0,0 +1,131 @@
+import CrawlBarCore
+import SwiftUI
+
+enum CrawlBarSettingsSidebarItem: Hashable {
+ case general
+ case crawler(CrawlAppID)
+}
+
+struct CrawlBarGeneralSidebarRow: View {
+ let isSelected: Bool
+
+ var body: some View {
+ Label("General", systemImage: "gearshape")
+ .font(.system(size: 13, weight: .semibold))
+ .foregroundStyle(self.isSelected ? Color.white : Color.primary)
+ .padding(.horizontal, 7)
+ .padding(.vertical, 6)
+ .frame(maxWidth: .infinity, alignment: .leading)
+ }
+}
+
+struct CrawlBarSidebarSelectionBackground: View {
+ let isSelected: Bool
+
+ var body: some View {
+ if self.isSelected {
+ RoundedRectangle(cornerRadius: 6, style: .continuous)
+ .fill(Color.accentColor.opacity(0.82))
+ .padding(.horizontal, 6)
+ .padding(.vertical, 2)
+ } else {
+ Color.clear
+ }
+ }
+}
+
+struct CrawlBarSidebarRow: View {
+ let app: CrawlBarAppConfig
+ let manifest: CrawlAppManifest?
+ let status: CrawlAppStatus?
+ let binaryPath: String?
+ let isSelected: Bool
+
+ var body: some View {
+ HStack(spacing: 11) {
+ CrawlBarBrandIcon(manifest: self.manifest, appID: self.app.id)
+ .frame(width: 32, height: 32)
+ VStack(alignment: .leading, spacing: 3) {
+ HStack(spacing: 6) {
+ Text(CrawlBarCrawlerTitle.text(for: self.app.id, manifest: self.manifest))
+ .font(.system(size: 13, weight: .semibold))
+ .foregroundStyle(self.isSelected ? Color.white : Color.primary)
+ .lineLimit(1)
+ CrawlBarStatusDot(state: self.rowState)
+ }
+ Text(self.subtitle)
+ .font(.system(size: 11))
+ .foregroundStyle(self.subtitleColor)
+ .lineLimit(1)
+ }
+ Spacer(minLength: 0)
+ }
+ .padding(.horizontal, 7)
+ .padding(.vertical, 5)
+ .opacity(self.manifest?.availability == .comingSoon ? 0.58 : 1)
+ }
+
+ private var rowState: CrawlAppState {
+ if self.manifest?.availability == .comingSoon { return .disabled }
+ if !self.app.enabled { return .disabled }
+ if self.binaryPath == nil { return .needsConfig }
+ let state = self.status?.state ?? .unknown
+ if self.status?.isRecoverableGraincrawlSourceFailure == true {
+ return .stale
+ }
+ return state
+ }
+
+ private var subtitle: String {
+ let binaryName = self.manifest?.binary.name ?? self.app.id.rawValue
+ if self.manifest?.availability == .comingSoon { return "\(binaryName) · coming soon" }
+ if !self.app.enabled { return "Disabled" }
+ if self.binaryPath == nil { return "Missing \(binaryName)" }
+ if self.rowState == .needsAuth {
+ return self.status?.summary ?? "Needs auth"
+ }
+ if self.rowState == .error {
+ return self.status?.summary ?? "Error"
+ }
+ if self.rowState == .current,
+ self.status?.freshness?.status == .stale
+ {
+ return "Status current"
+ }
+ if let syncedAt = self.syncedAt {
+ return "Synced \(CrawlBarDateText.relative(syncedAt))"
+ }
+ switch self.rowState {
+ case .syncing:
+ return "Syncing"
+ case .stale:
+ return "Needs refresh"
+ case .unknown:
+ return "Waiting for status"
+ default:
+ return self.status == nil ? "Waiting for status" : "Status current"
+ }
+ }
+
+ private var syncedAt: Date? {
+ if let lastSyncAt = self.status?.lastSyncAt {
+ return lastSyncAt
+ }
+ if let primaryModifiedAt = self.status?.databases.first(where: { $0.isPrimary })?.modifiedAt {
+ return primaryModifiedAt
+ }
+ return self.status?.databases.compactMap(\.modifiedAt).max()
+ }
+
+ private var subtitleColor: Color {
+ if self.isSelected { return Color.white.opacity(0.78) }
+ switch self.rowState {
+ case .needsConfig, .needsAuth, .error:
+ return Color.red
+ case .stale where self.app.id == BuiltInCrawlApps.graincrawlID && self.status?.state == .error:
+ return Color.yellow
+ default:
+ return Color.secondary
+ }
+ }
+}
diff --git a/Sources/CrawlBar/Settings/SettingsView.swift b/Sources/CrawlBar/Settings/SettingsView.swift
new file mode 100644
index 0000000..04c0b20
--- /dev/null
+++ b/Sources/CrawlBar/Settings/SettingsView.swift
@@ -0,0 +1,141 @@
+import CrawlBarCore
+import SwiftUI
+
+struct CrawlBarSettingsView: View {
+ @ObservedObject var model: CrawlBarSettingsModel
+ @State private var columnVisibility: NavigationSplitViewVisibility = .all
+
+ var body: some View {
+ NavigationSplitView(columnVisibility: self.animatedColumnVisibility) {
+ List(selection: self.sidebarSelection) {
+ Section("CrawlBar") {
+ CrawlBarGeneralSidebarRow(isSelected: self.model.selectedSidebarItem == .general)
+ .tag(CrawlBarSettingsSidebarItem.general as CrawlBarSettingsSidebarItem?)
+ .contentShape(Rectangle())
+ .listRowBackground(CrawlBarSidebarSelectionBackground(isSelected: self.model.selectedSidebarItem == .general))
+ .onTapGesture {
+ self.model.selectedSidebarItem = .general
+ }
+ }
+ Section("Crawlers") {
+ ForEach(self.model.apps) { app in
+ let item = CrawlBarSettingsSidebarItem.crawler(app.id)
+ CrawlBarSidebarRow(
+ app: app,
+ manifest: self.model.installations[app.id]?.manifest,
+ status: self.model.statuses[app.id],
+ binaryPath: self.model.installations[app.id]?.binaryPath,
+ isSelected: self.model.selectedSidebarItem == item)
+ .tag(item as CrawlBarSettingsSidebarItem?)
+ .contentShape(Rectangle())
+ .listRowBackground(CrawlBarSidebarSelectionBackground(isSelected: self.model.selectedSidebarItem == item))
+ .onTapGesture {
+ self.model.selectedSidebarItem = item
+ }
+ }
+ }
+ }
+ .listStyle(.sidebar)
+ .navigationSplitViewColumnWidth(CrawlBarSettingsLayout.sidebarWidth)
+ } detail: {
+ self.detailContainer
+ }
+ .navigationSplitViewStyle(.balanced)
+ .frame(
+ minWidth: CrawlBarSettingsLayout.minWindowWidth,
+ maxWidth: .infinity,
+ minHeight: CrawlBarSettingsLayout.minWindowHeight,
+ maxHeight: .infinity,
+ alignment: .topLeading)
+ .frame(maxWidth: .infinity, maxHeight: .infinity, alignment: .topLeading)
+ }
+
+ private var sidebarSelection: Binding {
+ Binding(
+ get: { self.model.selectedSidebarItem },
+ set: { item in
+ guard let item else { return }
+ self.model.selectedSidebarItem = item
+ })
+ }
+
+ private var animatedColumnVisibility: Binding {
+ Binding(
+ get: { self.columnVisibility },
+ set: { visibility in
+ withAnimation(.easeInOut(duration: 0.22)) {
+ self.columnVisibility = visibility
+ }
+ })
+ }
+
+ @ViewBuilder
+ private var detailContainer: some View {
+ VStack(alignment: .leading, spacing: 0) {
+ if let error = self.model.lastError {
+ CrawlBarIssueBanner(message: error, state: .error)
+ .padding(.horizontal, CrawlBarSettingsLayout.detailHorizontalPadding)
+ .padding(.top, CrawlBarSettingsLayout.detailVerticalPadding)
+ .padding(.bottom, 12)
+ }
+ self.selectedDetail
+ .disabled(self.model.isLoading)
+ }
+ .frame(maxWidth: .infinity, maxHeight: .infinity, alignment: .topLeading)
+ }
+
+ @ViewBuilder
+ private var selectedDetail: some View {
+ if self.model.isLoading && self.model.apps.isEmpty {
+ ProgressView("Loading settings...")
+ .frame(maxWidth: .infinity, maxHeight: .infinity, alignment: .center)
+ } else {
+ switch self.model.selectedSidebarItem {
+ case .general:
+ CrawlBarGeneralSettingsView(model: self.model)
+ case .crawler(let selectedID):
+ if self.model.apps.contains(where: { $0.id == selectedID })
+ {
+ CrawlBarAppDetailView(
+ app: self.binding(for: selectedID),
+ globalRefreshFrequency: self.model.refreshFrequency,
+ installation: self.model.installations[selectedID],
+ status: self.model.statuses[selectedID],
+ latestResult: self.model.recentResults[selectedID],
+ isRefreshing: self.model.isRefreshing,
+ runningAction: self.model.runningActions[selectedID],
+ actionMessage: self.model.actionMessages[selectedID],
+ refreshStatus: { self.model.refreshAll() },
+ runAction: { action in self.model.runAction(action, appID: selectedID) },
+ installApp: { self.model.installApp(selectedID) },
+ backupDatabases: { self.model.backupDatabases(selectedID) },
+ openDataFolder: { self.model.openDataFolder(selectedID) },
+ configValueChanged: { option, value in self.model.configValueDidChange(appID: selectedID, option: option, value: value) },
+ save: { self.model.save() },
+ saveDebounced: { self.model.saveDebounced() })
+ .padding(.horizontal, CrawlBarSettingsLayout.detailHorizontalPadding)
+ .padding(.vertical, CrawlBarSettingsLayout.detailVerticalPadding)
+ } else {
+ ContentUnavailableView(
+ "No crawler selected",
+ systemImage: "sidebar.left")
+ }
+ case nil:
+ ContentUnavailableView(
+ "No crawler selected",
+ systemImage: "sidebar.left")
+ }
+ }
+ }
+
+ private func binding(for id: CrawlAppID) -> Binding {
+ Binding(
+ get: {
+ self.model.apps.first(where: { $0.id == id }) ?? CrawlBarAppConfig(id: id)
+ },
+ set: {
+ guard let index = self.model.apps.firstIndex(where: { $0.id == id }) else { return }
+ self.model.apps[index] = $0
+ })
+ }
+}
diff --git a/Sources/CrawlBar/SettingsWindow.swift b/Sources/CrawlBar/SettingsWindow.swift
index 3f4c0e4..92cd321 100644
--- a/Sources/CrawlBar/SettingsWindow.swift
+++ b/Sources/CrawlBar/SettingsWindow.swift
@@ -16,12 +16,9 @@ final class CrawlBarSettingsWindowController: NSObject, NSWindowDelegate {
}
if let window {
- window.makeKeyAndOrderFront(nil)
- NSApplication.shared.activate()
- if let appID,
- let model = (window.contentView as? NSHostingView)?.rootView.model
- {
- model.selectedAppID = appID
+ self.present(window)
+ if let appID {
+ self.model?.selectedAppID = appID
}
if let model = self.model {
self.refreshStatusAfterPresentation(model: model, window: window)
@@ -53,15 +50,15 @@ final class CrawlBarSettingsWindowController: NSObject, NSWindowDelegate {
backing: .buffered,
defer: false)
window.title = "CrawlBar Settings"
+ window.setAccessibilityTitle("CrawlBar Settings")
window.isReleasedWhenClosed = false
window.delegate = self
window.center()
window.contentMinSize = NSSize(
width: CrawlBarSettingsLayout.minWindowWidth,
height: CrawlBarSettingsLayout.minWindowHeight)
- window.contentView = NSHostingView(rootView: CrawlBarSettingsView(model: model))
- window.makeKeyAndOrderFront(nil)
- NSApplication.shared.activate()
+ window.contentViewController = NSHostingController(rootView: CrawlBarSettingsView(model: model))
+ self.present(window)
self.window = window
model.loadForPresentation { [weak self, weak window] in
guard let self, let window else { return }
@@ -69,6 +66,12 @@ final class CrawlBarSettingsWindowController: NSObject, NSWindowDelegate {
}
}
+ private func present(_ window: NSWindow) {
+ NSApplication.shared.activate()
+ window.makeKeyAndOrderFront(nil)
+ window.makeMain()
+ }
+
private func refreshStatusAfterPresentation(model: CrawlBarSettingsModel, window: NSWindow) {
guard !model.isRefreshing else { return }
Task { @MainActor in
@@ -87,2622 +90,3 @@ final class CrawlBarSettingsWindowController: NSObject, NSWindowDelegate {
}
}
}
-
-@MainActor
-final class CrawlBarSettingsModel: NSObject, ObservableObject {
- @Published var apps: [CrawlBarAppConfig] = []
- @Published var refreshFrequency: RefreshFrequency = .fifteenMinutes
- @Published fileprivate var selectedSidebarItem: CrawlBarSettingsSidebarItem?
- var selectedAppID: CrawlAppID? {
- get {
- guard case let .crawler(id) = self.selectedSidebarItem else { return nil }
- return id
- }
- set {
- self.selectedSidebarItem = newValue.map(CrawlBarSettingsSidebarItem.crawler)
- }
- }
- @Published var statuses: [CrawlAppID: CrawlAppStatus] = [:]
- @Published var installations: [CrawlAppID: CrawlAppInstallation] = [:]
- @Published var isRefreshing = false
- @Published var isInstallingCLI = false
- @Published var appActionMessage: String?
- @Published var runningActions: [CrawlAppID: String] = [:]
- @Published var actionMessages: [CrawlAppID: String] = [:]
- @Published var recentResults: [CrawlAppID: CrawlCommandResult] = [:]
- @Published var lastError: String?
- @Published var manifestDiagnostics: [CrawlManifestDiagnostic] = []
- @Published var isLoading = false
- @Published fileprivate var manifestDirectories: [String] = ["~/.crawlbar/apps"]
-
- private var refreshTask: Task?
- private var loadTask: Task?
- private var pendingSaveTask: Task?
- private var refreshGeneration = UUID()
- private var loadGeneration = UUID()
- private var recentResultsGeneration = UUID()
- private var hasLoadedSnapshot = false
- private var clearedNativeSecretIDsByAppID: [CrawlAppID: Set] = [:]
- private let store = CrawlBarConfigStore()
- private let registry = CrawlAppRegistry()
- private let runner: CrawlCommandRunner
- private let statusService: CrawlStatusService
- private let nativeConfigStore = CrawlNativeConfigStore()
- private let installer = CrawlInstaller()
- private let logStore = CrawlActionLogStore()
-
- init(loadImmediately: Bool = true) {
- let runner = CrawlCommandRunner()
- self.runner = runner
- self.statusService = CrawlStatusService(runner: runner)
- super.init()
- NotificationCenter.default.addObserver(
- self,
- selector: #selector(Self.statusesDidChange(_:)),
- name: .crawlBarStatusesDidChange,
- object: nil)
- if loadImmediately {
- self.load()
- } else {
- self.selectedSidebarItem = .general
- }
- }
-
- deinit {
- NotificationCenter.default.removeObserver(self)
- }
-
- private var sidebarSelectionIsValid: Bool {
- switch self.selectedSidebarItem {
- case .general:
- true
- case .crawler(let id):
- self.apps.contains { $0.id == id }
- case nil:
- false
- }
- }
-
- func load() {
- do {
- self.apply(try Self.loadSnapshot())
- } catch {
- CrawlBarLog.config.error("Settings load failed: \(error.localizedDescription, privacy: .public)")
- self.lastError = error.localizedDescription
- }
- }
-
- func loadForPresentation(onLoaded: @escaping @MainActor () -> Void) {
- self.loadTask?.cancel()
- let generation = UUID()
- self.loadGeneration = generation
- self.isLoading = true
- self.lastError = nil
- self.loadTask = Task.detached {
- let snapshot = Result { try Self.loadSnapshot() }
- await MainActor.run {
- guard self.loadGeneration == generation else { return }
- self.isLoading = false
- self.loadTask = nil
- switch snapshot {
- case .success(let snapshot):
- self.apply(snapshot)
- onLoaded()
- case .failure(let error):
- CrawlBarLog.config.error("Settings load failed: \(error.localizedDescription, privacy: .public)")
- self.lastError = error.localizedDescription
- }
- }
- }
- }
-
- func save() {
- self.pendingSaveTask?.cancel()
- self.pendingSaveTask = nil
- self.persist()
- }
-
- func scrubSecretConfigValues() {
- var scrubbedApps = self.apps
- for index in scrubbedApps.indices {
- guard let manifest = self.installations[scrubbedApps[index].id]?.manifest else { continue }
- for option in manifest.configOptions where option.kind == .secret {
- scrubbedApps[index].configValues.removeValue(forKey: option.id)
- }
- }
- self.apps = scrubbedApps
- }
-
- func saveDebounced() {
- self.pendingSaveTask?.cancel()
- self.pendingSaveTask = Task { @MainActor in
- try? await Task.sleep(nanoseconds: 700_000_000)
- guard !Task.isCancelled else { return }
- self.persist()
- self.pendingSaveTask = nil
- }
- }
-
- private func persist() {
- guard self.hasLoadedSnapshot, !self.isLoading else { return }
- do {
- let config = CrawlBarConfig(
- refreshFrequency: self.refreshFrequency,
- manifestDirectories: self.manifestDirectories,
- apps: self.apps)
- try self.store.save(
- config,
- clearMissingSecretIDsByAppID: self.clearedNativeSecretIDsByAppID)
- try self.nativeConfigStore.write(
- config: config,
- clearMissingSecretIDsByAppID: self.clearedNativeSecretIDsByAppID)
- self.clearedNativeSecretIDsByAppID = [:]
- self.lastError = nil
- CrawlBarStateBroadcast.configDidChange()
- } catch {
- CrawlBarLog.config.error("Settings save failed: \(error.localizedDescription, privacy: .public)")
- self.lastError = error.localizedDescription
- }
- }
-
- func configValueDidChange(appID: CrawlAppID, option: CrawlAppManifest.ConfigOption, value: String?) {
- if option.kind == .secret {
- if value?.nilIfBlank == nil {
- self.clearedNativeSecretIDsByAppID[appID, default: []].insert(option.id)
- } else {
- self.clearedNativeSecretIDsByAppID[appID]?.remove(option.id)
- }
- }
- self.saveDebounced()
- }
-
- func refreshAll() {
- self.loadRecentResultsAsync()
- self.refreshTask?.cancel()
- let generation = UUID()
- self.refreshGeneration = generation
- self.isRefreshing = true
- let registry = self.registry
- let statusService = self.statusService
- self.refreshTask = Task.detached {
- let installations = (try? registry.installationsForStatus(includeDisabled: true)) ?? []
- await MainActor.run {
- guard self.refreshGeneration == generation else { return }
- let installationsByID = Dictionary(uniqueKeysWithValues: installations.map { ($0.id, $0) })
- self.installations = installationsByID
- self.apps = Self.sortedAppConfigs(self.apps, installationsByID: installationsByID)
- }
- let partitioned = Self.partitionStatuses(installations: installations, statusService: statusService)
- if !partitioned.immediate.isEmpty {
- await MainActor.run {
- guard self.refreshGeneration == generation else { return }
- for status in partitioned.immediate {
- self.statuses[status.appID] = status
- }
- CrawlBarStateBroadcast.statusesDidChange(Dictionary(uniqueKeysWithValues: partitioned.immediate.map { ($0.appID, $0) }))
- }
- }
- await withTaskGroup(of: CrawlAppStatus.self) { group in
- for installation in partitioned.commandInstallations {
- group.addTask {
- guard !Task.isCancelled else {
- return CrawlAppStatus(appID: installation.id, state: .unknown, summary: "Refresh cancelled")
- }
- return statusService.status(for: installation, timeoutSeconds: 5)
- }
- }
- for await status in group {
- if Task.isCancelled { break }
- await MainActor.run {
- guard self.refreshGeneration == generation else { return }
- self.statuses[status.appID] = status
- CrawlBarStateBroadcast.statusesDidChange([status.appID: status])
- }
- }
- }
- await MainActor.run {
- guard self.refreshGeneration == generation else { return }
- self.isRefreshing = false
- self.refreshTask = nil
- }
- }
- }
-
- nonisolated private static func partitionStatuses(
- installations: [CrawlAppInstallation],
- statusService: CrawlStatusService)
- -> (immediate: [CrawlAppStatus], commandInstallations: [CrawlAppInstallation])
- {
- var immediate: [CrawlAppStatus] = []
- var commandInstallations: [CrawlAppInstallation] = []
- for installation in installations {
- if let status = statusService.immediateStatus(for: installation) {
- immediate.append(status)
- } else {
- commandInstallations.append(installation)
- }
- }
- return (immediate, commandInstallations)
- }
-
- func runAction(_ action: String, appID: CrawlAppID) {
- guard let installation = self.installations[appID] else { return }
- self.runningActions[appID] = action
- self.actionMessages[appID] = "Running \(Self.actionTitle(action))..."
- let runner = self.runner
- let statusService = self.statusService
- let logStore = self.logStore
- let registry = self.registry
- Task.detached {
- let actionInstallation = (try? registry.installation(for: appID, includeSecrets: true)) ?? installation
- let message: String
- var actionError: CrawlAppStatus?
- do {
- CrawlBarLog.actions.notice("Running \(action, privacy: .public) for \(appID.rawValue, privacy: .public) from settings")
- let result = try runner.run(installation: actionInstallation, action: action, timeoutSeconds: 600)
- _ = try? logStore.save(result)
- message = result.exitCode == 0
- ? "\(Self.actionTitle(action)) finished"
- : "\(Self.actionTitle(action)) failed with exit \(result.exitCode)"
- if !result.succeeded {
- CrawlBarLog.actions.error(
- "\(action, privacy: .public) for \(appID.rawValue, privacy: .public) failed with exit \(result.exitCode)")
- actionError = Self.actionFailureStatus(result)
- }
- } catch {
- CrawlBarLog.actions.error(
- "\(action, privacy: .public) for \(appID.rawValue, privacy: .public) threw: \(error.localizedDescription, privacy: .public)")
- message = error.localizedDescription
- actionError = Self.actionFailureStatus(appID: appID, action: action, message: error.localizedDescription)
- }
- let refreshedStatus = statusService.status(for: actionInstallation, timeoutSeconds: 5)
- await MainActor.run {
- let status = actionError.map {
- Self.actionFailureStatus($0, refreshedStatus: refreshedStatus, currentStatus: self.statuses[appID])
- } ?? refreshedStatus
- self.statuses[appID] = status
- CrawlBarStateBroadcast.statusesDidChange([appID: status])
- self.runningActions[appID] = nil
- self.actionMessages[appID] = message
- self.loadRecentResults()
- }
- }
- }
-
- private func loadRecentResults() {
- self.recentResultsGeneration = UUID()
- self.recentResults = Self.recentResults(logStore: self.logStore)
- }
-
- private func loadRecentResultsAsync() {
- let logStore = self.logStore
- let generation = UUID()
- self.recentResultsGeneration = generation
- Task.detached {
- let results = Self.recentResults(logStore: logStore)
- await MainActor.run {
- guard self.recentResultsGeneration == generation else { return }
- self.recentResults = results
- }
- }
- }
-
- private func mergeStatuses(_ incoming: [CrawlAppID: CrawlAppStatus]) {
- for (appID, status) in incoming {
- self.statuses[appID] = status
- }
- }
-
- @objc private func statusesDidChange(_ notification: Notification) {
- guard let statuses = CrawlBarStateBroadcast.statuses(from: notification) else { return }
- self.mergeStatuses(statuses)
- }
-
- func installApp(_ appID: CrawlAppID) {
- guard let installation = self.installations[appID] else { return }
- self.runningActions[appID] = "install"
- self.actionMessages[appID] = "Installing \(installation.manifest.binary.name)..."
- let installer = self.installer
- let logStore = self.logStore
- let registry = self.registry
- Task.detached {
- let message: String
- do {
- let result = try installer.install(installation)
- _ = try? logStore.save(result)
- message = "\(installation.manifest.binary.name) installed"
- } catch {
- message = error.localizedDescription
- }
- let installations = (try? registry.installations(includeDisabled: true)) ?? []
- await MainActor.run {
- let installationsByID = Dictionary(uniqueKeysWithValues: installations.map { ($0.id, $0) })
- self.installations = installationsByID
- self.apps = Self.sortedAppConfigs(self.apps, installationsByID: installationsByID)
- self.runningActions[appID] = nil
- self.actionMessages[appID] = message
- }
- }
- }
-
- func backupDatabases(_ appID: CrawlAppID) {
- guard let status = self.statuses[appID] else { return }
- self.runningActions[appID] = "backup"
- self.actionMessages[appID] = "Backing up databases..."
- Task.detached {
- let message: String
- do {
- let backup = try CrawlDatabaseBackupStore.backup(status: status)
- message = "Backed up \(backup.files.count) database file(s)"
- } catch {
- message = error.localizedDescription
- }
- await MainActor.run {
- self.runningActions[appID] = nil
- self.actionMessages[appID] = message
- }
- }
- }
-
- func openDataFolder(_ appID: CrawlAppID) {
- guard let status = self.statuses[appID],
- let path = status.databases.first(where: { $0.isPrimary })?.path ?? status.databasePath
- else { return }
- NSWorkspace.shared.open(URL(fileURLWithPath: PathExpander.expandHome(path)).deletingLastPathComponent())
- }
-
- func openConfigFile() {
- NSWorkspace.shared.activateFileViewerSelecting([self.store.fileURL])
- }
-
- func openLogsFolder() {
- NSWorkspace.shared.open(CrawlActionLogStore.defaultDirectory())
- }
-
- func installCLI() {
- self.isInstallingCLI = true
- self.appActionMessage = "Installing crawlbar CLI..."
- Task.detached {
- let message: String
- do {
- let path = try Self.installBundledCLI()
- message = "Installed crawlbar CLI at \(path)"
- } catch {
- message = error.localizedDescription
- }
- await MainActor.run {
- self.isInstallingCLI = false
- self.appActionMessage = message
- }
- }
- }
-
- nonisolated private static func actionTitle(_ action: String) -> String {
- switch action {
- case "refresh":
- "Sync"
- case "doctor":
- "Doctor"
- case "unlock":
- "Unlock"
- case "publish":
- "Publish"
- case "cloud-publish":
- "Cloud Publish"
- case "remote-status":
- "Remote Status"
- case "remote-archives":
- "Remote Archives"
- case "update":
- "Update"
- case "desktop-cache-import":
- "Desktop Import"
- default:
- action
- }
- }
-
- nonisolated private static func sortedAppConfigs(
- _ apps: [CrawlBarAppConfig],
- installationsByID: [CrawlAppID: CrawlAppInstallation])
- -> [CrawlBarAppConfig]
- {
- let originalIndex = Dictionary(uniqueKeysWithValues: apps.enumerated().map { ($0.element.id, $0.offset) })
- return apps.sorted { lhs, rhs in
- let lhsRank = Self.sidebarRank(installation: installationsByID[lhs.id])
- let rhsRank = Self.sidebarRank(installation: installationsByID[rhs.id])
- if lhsRank != rhsRank { return lhsRank < rhsRank }
- return (originalIndex[lhs.id] ?? Int.max) < (originalIndex[rhs.id] ?? Int.max)
- }
- }
-
- private func apply(_ snapshot: CrawlBarSettingsSnapshot) {
- self.hasLoadedSnapshot = true
- self.apps = snapshot.apps
- self.refreshFrequency = snapshot.refreshFrequency
- self.manifestDirectories = snapshot.manifestDirectories
- self.installations = snapshot.installations
- self.recentResults = snapshot.recentResults
- self.manifestDiagnostics = snapshot.manifestDiagnostics
- if !self.sidebarSelectionIsValid {
- self.selectedSidebarItem = self.apps.first.map { .crawler($0.id) } ?? .general
- }
- self.lastError = nil
- }
-
- nonisolated private static func loadSnapshot() throws -> CrawlBarSettingsSnapshot {
- let store = CrawlBarConfigStore()
- let registry = CrawlAppRegistry()
- let nativeConfigStore = CrawlNativeConfigStore()
- let logStore = CrawlActionLogStore()
- let config = try store.loadOrCreateDefault()
- let loadedInstallations = try registry.installations(includeDisabled: true)
- let manifests = Dictionary(uniqueKeysWithValues: loadedInstallations.map { ($0.id, $0.manifest) })
- let appConfigsByID = Dictionary(uniqueKeysWithValues: config.apps.map { ($0.id, $0) })
- let installationsByID = Dictionary(uniqueKeysWithValues: loadedInstallations.map { ($0.id, $0) })
- let apps = loadedInstallations.map { installation in
- let appConfig = appConfigsByID[installation.id] ?? CrawlBarAppConfig(
- id: installation.id,
- enabled: installation.manifest.availability == .available,
- showInMenuBar: installation.manifest.availability == .available)
- guard let manifest = manifests[appConfig.id] else { return appConfig }
- var copy = appConfig
- copy.configValues = nativeConfigStore.resolvedConfigValues(
- appConfig: appConfig,
- manifest: manifest,
- includeSecrets: false)
- return copy
- }
- return CrawlBarSettingsSnapshot(
- apps: Self.sortedAppConfigs(apps, installationsByID: installationsByID),
- refreshFrequency: config.refreshFrequency,
- manifestDirectories: config.manifestDirectories,
- installations: installationsByID,
- recentResults: Self.recentResults(logStore: logStore),
- manifestDiagnostics: CrawlManifestCatalog().diagnostics(config: config))
- }
-
- nonisolated private static func recentResults(logStore: CrawlActionLogStore) -> [CrawlAppID: CrawlCommandResult] {
- var resultsByApp: [CrawlAppID: CrawlCommandResult] = [:]
- for result in logStore.recentResults(limit: 200).sorted(by: { $0.finishedAt > $1.finishedAt }) {
- if resultsByApp[result.appID] == nil {
- resultsByApp[result.appID] = result
- }
- }
- return resultsByApp
- }
-
- nonisolated private static func sidebarRank(installation: CrawlAppInstallation?) -> Int {
- guard let installation else { return 4 }
- if installation.manifest.availability == .comingSoon { return 3 }
- if installation.binaryPath != nil { return 0 }
- if installation.manifest.install != nil { return 1 }
- return 2
- }
-
- nonisolated private static func actionFailureStatus(_ result: CrawlCommandResult) -> CrawlAppStatus {
- let fallback = "\(result.action) failed with exit \(result.exitCode)"
- return CrawlAppStatus.commandFailure(
- appID: result.appID,
- action: result.action,
- message: result.stderr.nilIfBlank ?? result.stdout.nilIfBlank,
- fallback: fallback)
- }
-
- nonisolated private static func actionFailureStatus(appID: CrawlAppID, action: String, message: String) -> CrawlAppStatus {
- CrawlAppStatus.commandFailure(
- appID: appID,
- action: action,
- message: message,
- fallback: "\(action) failed")
- }
-
- nonisolated private static func actionFailureStatus(
- _ failure: CrawlAppStatus,
- refreshedStatus: CrawlAppStatus?,
- currentStatus: CrawlAppStatus?)
- -> CrawlAppStatus
- {
- guard let metadataStatus = CrawlAppStatus.richestMetadataStatus(refreshedStatus, fallback: currentStatus) else {
- return failure
- }
- return metadataStatus.mergingActionFailure(failure)
- }
-
- nonisolated private static func installBundledCLI() throws -> String {
- let fileManager = FileManager.default
- let sourceCandidates = Self.cliSourceCandidates()
- guard let source = sourceCandidates.first(where: { fileManager.isExecutableFile(atPath: $0.path) }) else {
- throw CrawlBarSettingsError.cliHelperMissing
- }
- let destinationDirectory = URL(fileURLWithPath: PathExpander.expandHome("~/.local/bin"), isDirectory: true)
- let destination = destinationDirectory.appendingPathComponent("crawlbar")
- try fileManager.createDirectory(at: destinationDirectory, withIntermediateDirectories: true)
- if fileManager.fileExists(atPath: destination.path) {
- try fileManager.removeItem(at: destination)
- }
- try fileManager.copyItem(at: source, to: destination)
- try fileManager.setAttributes([.posixPermissions: 0o755], ofItemAtPath: destination.path)
- return destination.path
- }
-
- nonisolated private static func cliSourceCandidates() -> [URL] {
- var candidates: [URL] = [
- Bundle.main.bundleURL.appendingPathComponent("Contents/Helpers/crawlbar"),
- ]
- if let executableDirectory = Bundle.main.executableURL?.deletingLastPathComponent() {
- candidates.append(executableDirectory.appendingPathComponent("crawlbarctl"))
- candidates.append(executableDirectory.deletingLastPathComponent().appendingPathComponent("debug/crawlbarctl"))
- candidates.append(executableDirectory.deletingLastPathComponent().appendingPathComponent("release/crawlbarctl"))
- }
- return candidates
- }
-}
-
-private enum CrawlBarSettingsError: LocalizedError {
- case cliHelperMissing
-
- var errorDescription: String? {
- switch self {
- case .cliHelperMissing:
- "Could not find bundled crawlbar CLI helper"
- }
- }
-}
-
-fileprivate enum CrawlBarSettingsSidebarItem: Hashable {
- case general
- case crawler(CrawlAppID)
-}
-
-private struct CrawlBarSettingsSnapshot: Sendable {
- let apps: [CrawlBarAppConfig]
- let refreshFrequency: RefreshFrequency
- let manifestDirectories: [String]
- let installations: [CrawlAppID: CrawlAppInstallation]
- let recentResults: [CrawlAppID: CrawlCommandResult]
- let manifestDiagnostics: [CrawlManifestDiagnostic]
-}
-
-private enum CrawlBarSettingsLayout {
- static let minWindowWidth: CGFloat = 1120
- static let minWindowHeight: CGFloat = 790
- static let sidebarWidth: CGFloat = 250
- static let detailHorizontalPadding: CGFloat = 22
- static let detailVerticalPadding: CGFloat = 18
- static let detailBottomPadding: CGFloat = 16
-}
-
-struct CrawlBarSettingsView: View {
- @ObservedObject var model: CrawlBarSettingsModel
- @State private var columnVisibility: NavigationSplitViewVisibility = .all
-
- var body: some View {
- NavigationSplitView(columnVisibility: self.animatedColumnVisibility) {
- List(selection: self.sidebarSelection) {
- Section("CrawlBar") {
- CrawlBarGeneralSidebarRow(isSelected: self.model.selectedSidebarItem == .general)
- .tag(CrawlBarSettingsSidebarItem.general as CrawlBarSettingsSidebarItem?)
- .contentShape(Rectangle())
- .listRowBackground(CrawlBarSidebarSelectionBackground(isSelected: self.model.selectedSidebarItem == .general))
- .onTapGesture {
- self.model.selectedSidebarItem = .general
- }
- }
- Section("Crawlers") {
- ForEach(self.model.apps) { app in
- let item = CrawlBarSettingsSidebarItem.crawler(app.id)
- CrawlBarSidebarRow(
- app: app,
- manifest: self.model.installations[app.id]?.manifest,
- status: self.model.statuses[app.id],
- binaryPath: self.model.installations[app.id]?.binaryPath,
- isSelected: self.model.selectedSidebarItem == item)
- .tag(item as CrawlBarSettingsSidebarItem?)
- .contentShape(Rectangle())
- .listRowBackground(CrawlBarSidebarSelectionBackground(isSelected: self.model.selectedSidebarItem == item))
- .onTapGesture {
- self.model.selectedSidebarItem = item
- }
- }
- }
- }
- .listStyle(.sidebar)
- .navigationSplitViewColumnWidth(CrawlBarSettingsLayout.sidebarWidth)
- } detail: {
- self.detailContainer
- }
- .navigationSplitViewStyle(.balanced)
- .frame(
- minWidth: CrawlBarSettingsLayout.minWindowWidth,
- maxWidth: .infinity,
- minHeight: CrawlBarSettingsLayout.minWindowHeight,
- maxHeight: .infinity,
- alignment: .topLeading)
- .frame(maxWidth: .infinity, maxHeight: .infinity, alignment: .topLeading)
- }
-
- private var sidebarSelection: Binding {
- Binding(
- get: { self.model.selectedSidebarItem },
- set: { item in
- guard let item else { return }
- self.model.selectedSidebarItem = item
- })
- }
-
- private var animatedColumnVisibility: Binding {
- Binding(
- get: { self.columnVisibility },
- set: { visibility in
- withAnimation(.easeInOut(duration: 0.22)) {
- self.columnVisibility = visibility
- }
- })
- }
-
- @ViewBuilder
- private var detailContainer: some View {
- VStack(alignment: .leading, spacing: 0) {
- if let error = self.model.lastError {
- CrawlBarSettingsErrorBanner(message: error)
- .padding(.horizontal, CrawlBarSettingsLayout.detailHorizontalPadding)
- .padding(.top, CrawlBarSettingsLayout.detailVerticalPadding)
- .padding(.bottom, 12)
- }
- self.selectedDetail
- .disabled(self.model.isLoading)
- }
- .frame(maxWidth: .infinity, maxHeight: .infinity, alignment: .topLeading)
- }
-
- @ViewBuilder
- private var selectedDetail: some View {
- if self.model.isLoading && self.model.apps.isEmpty {
- ProgressView("Loading settings...")
- .frame(maxWidth: .infinity, maxHeight: .infinity, alignment: .center)
- } else {
- switch self.model.selectedSidebarItem {
- case .general:
- CrawlBarGeneralSettingsView(model: self.model)
- case .crawler(let selectedID):
- if self.model.apps.contains(where: { $0.id == selectedID })
- {
- CrawlBarAppDetailView(
- app: self.binding(for: selectedID),
- globalRefreshFrequency: self.model.refreshFrequency,
- installation: self.model.installations[selectedID],
- status: self.model.statuses[selectedID],
- latestResult: self.model.recentResults[selectedID],
- isRefreshing: self.model.isRefreshing,
- runningAction: self.model.runningActions[selectedID],
- actionMessage: self.model.actionMessages[selectedID],
- refreshStatus: { self.model.refreshAll() },
- runAction: { action in self.model.runAction(action, appID: selectedID) },
- installApp: { self.model.installApp(selectedID) },
- backupDatabases: { self.model.backupDatabases(selectedID) },
- openDataFolder: { self.model.openDataFolder(selectedID) },
- configValueChanged: { option, value in self.model.configValueDidChange(appID: selectedID, option: option, value: value) },
- save: { self.model.save() },
- saveDebounced: { self.model.saveDebounced() })
- .padding(.horizontal, CrawlBarSettingsLayout.detailHorizontalPadding)
- .padding(.vertical, CrawlBarSettingsLayout.detailVerticalPadding)
- } else {
- ContentUnavailableView(
- "No crawler selected",
- systemImage: "sidebar.left")
- }
- case nil:
- ContentUnavailableView(
- "No crawler selected",
- systemImage: "sidebar.left")
- }
- }
- }
-
- private func binding(for id: CrawlAppID) -> Binding {
- Binding(
- get: {
- self.model.apps.first(where: { $0.id == id }) ?? CrawlBarAppConfig(id: id)
- },
- set: {
- guard let index = self.model.apps.firstIndex(where: { $0.id == id }) else { return }
- self.model.apps[index] = $0
- })
- }
-
-}
-
-private struct CrawlBarSettingsErrorBanner: View {
- let message: String
-
- var body: some View {
- HStack(alignment: .top, spacing: 8) {
- Image(systemName: "exclamationmark.triangle.fill")
- .foregroundStyle(.red)
- Text(self.message)
- .font(.caption)
- .foregroundStyle(.red)
- .fixedSize(horizontal: false, vertical: true)
- .textSelection(.enabled)
- Spacer(minLength: 0)
- }
- .padding(.horizontal, 12)
- .padding(.vertical, 9)
- .frame(maxWidth: .infinity, alignment: .leading)
- .background(.red.opacity(0.08), in: RoundedRectangle(cornerRadius: 10, style: .continuous))
- }
-}
-
-private struct CrawlBarGeneralSettingsView: View {
- @ObservedObject var model: CrawlBarSettingsModel
-
- var body: some View {
- ScrollView {
- VStack(alignment: .leading, spacing: 14) {
- HStack(alignment: .center, spacing: 14) {
- Image(nsImage: NSApp.applicationIconImage)
- .resizable()
- .frame(width: 42, height: 42)
- VStack(alignment: .leading, spacing: 4) {
- Text("CrawlBar")
- .font(.title3.weight(.semibold))
- Text("Menu bar control plane for local crawler apps")
- .font(.subheadline)
- .foregroundStyle(.secondary)
- .lineLimit(2)
- }
- Spacer()
- Button {
- self.model.refreshAll()
- } label: {
- Label("Refresh All", systemImage: "arrow.clockwise")
- }
- .disabled(self.model.isRefreshing)
- }
-
- CrawlBarPanel(title: "App") {
- HStack(spacing: 8) {
- Button {
- self.model.installCLI()
- } label: {
- Label("Install CLI", systemImage: "terminal")
- }
- .disabled(self.model.isInstallingCLI)
- Button {
- self.model.openConfigFile()
- } label: {
- Label("Open Config", systemImage: "doc.text")
- }
- Button {
- self.model.openLogsFolder()
- } label: {
- Label("Open Logs", systemImage: "folder")
- }
- }
- .controlSize(.small)
- CrawlBarFact(label: "CLI install path", value: "~/.local/bin/crawlbar")
- CrawlBarFact(label: "Config", value: CrawlBarConfigStore().fileURL.path)
- if let message = self.model.appActionMessage {
- Text(message)
- .font(.caption)
- .foregroundStyle(.secondary)
- }
- }
-
- CrawlBarPanel(title: "Scheduling") {
- CrawlBarControlRow(
- title: "Default schedule",
- caption: "Used by crawlers that inherit the global sync interval.")
- {
- Picker("Default schedule", selection: self.$model.refreshFrequency) {
- ForEach(RefreshFrequency.allCases, id: \.self) { frequency in
- Text(CrawlBarFrequencyLabel.text(for: frequency)).tag(frequency)
- }
- }
- .labelsHidden()
- .frame(width: 180)
- .onChange(of: self.model.refreshFrequency) {
- self.model.save()
- }
- }
- }
-
- CrawlBarPanel(title: "Discovery") {
- ForEach(self.manifestDirectories, id: \.self) { directory in
- CrawlBarFact(label: "Manifest Directory", value: directory)
- }
- if !self.model.manifestDiagnostics.isEmpty {
- Divider()
- ForEach(self.model.manifestDiagnostics) { diagnostic in
- Label {
- Text("\(diagnostic.path): \(diagnostic.message)")
- .font(.caption)
- .foregroundStyle(.secondary)
- .lineLimit(2)
- .truncationMode(.middle)
- .textSelection(.enabled)
- } icon: {
- Image(systemName: "exclamationmark.triangle")
- .foregroundStyle(.yellow)
- }
- }
- }
- }
-
- CrawlBarPanel(title: "Crawler Inventory") {
- Grid(alignment: .leading, horizontalSpacing: 16, verticalSpacing: 8) {
- GridRow {
- CrawlBarFact(label: "Ready", value: "\(self.readyCount)")
- CrawlBarFact(label: "Missing CLI", value: "\(self.missingCount)")
- CrawlBarFact(label: "Coming Soon", value: "\(self.comingSoonCount)")
- }
- }
- }
-
- Spacer(minLength: 0)
- }
- .frame(maxWidth: .infinity, alignment: .leading)
- .padding(.vertical, 4)
- .padding(.bottom, CrawlBarSettingsLayout.detailBottomPadding)
- }
- .padding(.horizontal, CrawlBarSettingsLayout.detailHorizontalPadding)
- .padding(.vertical, CrawlBarSettingsLayout.detailVerticalPadding)
- }
-
- private var manifestDirectories: [String] {
- self.model.manifestDirectories
- }
-
- private var readyCount: Int {
- self.model.installations.values.filter { $0.manifest.availability == .available && $0.binaryPath != nil }.count
- }
-
- private var missingCount: Int {
- self.model.installations.values.filter { $0.manifest.availability == .available && $0.binaryPath == nil }.count
- }
-
- private var comingSoonCount: Int {
- self.model.installations.values.filter { $0.manifest.availability == .comingSoon }.count
- }
-}
-
-struct CrawlBarGeneralSidebarRow: View {
- let isSelected: Bool
-
- var body: some View {
- Label("General", systemImage: "gearshape")
- .font(.system(size: 13, weight: .semibold))
- .foregroundStyle(self.isSelected ? Color.white : Color.primary)
- .padding(.horizontal, 7)
- .padding(.vertical, 6)
- .frame(maxWidth: .infinity, alignment: .leading)
- }
-}
-
-struct CrawlBarSidebarSelectionBackground: View {
- let isSelected: Bool
-
- var body: some View {
- if self.isSelected {
- RoundedRectangle(cornerRadius: 6, style: .continuous)
- .fill(Color.accentColor.opacity(0.82))
- .padding(.horizontal, 6)
- .padding(.vertical, 2)
- } else {
- Color.clear
- }
- }
-}
-
-struct CrawlBarSidebarRow: View {
- let app: CrawlBarAppConfig
- let manifest: CrawlAppManifest?
- let status: CrawlAppStatus?
- let binaryPath: String?
- let isSelected: Bool
-
- var body: some View {
- HStack(spacing: 11) {
- CrawlBarBrandIcon(manifest: self.manifest, appID: self.app.id)
- .frame(width: 32, height: 32)
- VStack(alignment: .leading, spacing: 3) {
- HStack(spacing: 6) {
- Text(CrawlBarCrawlerTitle.text(for: self.app.id, manifest: self.manifest))
- .font(.system(size: 13, weight: .semibold))
- .foregroundStyle(self.isSelected ? Color.white : Color.primary)
- .lineLimit(1)
- CrawlBarStatusDot(state: self.rowState)
- }
- Text(self.subtitle)
- .font(.system(size: 11))
- .foregroundStyle(self.subtitleColor)
- .lineLimit(1)
- }
- Spacer(minLength: 0)
- }
- .padding(.horizontal, 7)
- .padding(.vertical, 5)
- .opacity(self.manifest?.availability == .comingSoon ? 0.58 : 1)
- }
-
- private var rowState: CrawlAppState {
- if self.manifest?.availability == .comingSoon { return .disabled }
- if !self.app.enabled { return .disabled }
- if self.binaryPath == nil { return .needsConfig }
- let state = self.status?.state ?? .unknown
- if self.status?.isRecoverableGraincrawlSourceFailure == true {
- return .stale
- }
- return state
- }
-
- private var subtitle: String {
- let binaryName = self.manifest?.binary.name ?? self.app.id.rawValue
- if self.manifest?.availability == .comingSoon { return "\(binaryName) · coming soon" }
- if !self.app.enabled { return "Disabled" }
- if self.binaryPath == nil { return "Missing \(binaryName)" }
- if self.rowState == .needsAuth {
- return self.status?.summary ?? "Needs auth"
- }
- if self.rowState == .error {
- return self.status?.summary ?? "Error"
- }
- if self.rowState == .current,
- self.status?.freshness?.status == .stale
- {
- return "Status current"
- }
- if let syncedAt = self.syncedAt {
- return "Synced \(CrawlBarDateText.relative(syncedAt))"
- }
- switch self.rowState {
- case .syncing:
- return "Syncing"
- case .stale:
- return "Needs refresh"
- case .unknown:
- return "Waiting for status"
- default:
- return self.status == nil ? "Waiting for status" : "Status current"
- }
- }
-
- private var syncedAt: Date? {
- if let lastSyncAt = self.status?.lastSyncAt {
- return lastSyncAt
- }
- if let primaryModifiedAt = self.status?.databases.first(where: { $0.isPrimary })?.modifiedAt {
- return primaryModifiedAt
- }
- return self.status?.databases.compactMap(\.modifiedAt).max()
- }
-
- private var subtitleColor: Color {
- if self.isSelected { return Color.white.opacity(0.78) }
- switch self.rowState {
- case .needsConfig, .needsAuth, .error:
- return Color.red
- case .stale where self.app.id == BuiltInCrawlApps.graincrawlID && self.status?.state == .error:
- return Color.yellow
- default:
- return Color.secondary
- }
- }
-}
-
-struct CrawlBarAppDetailView: View {
- @Binding var app: CrawlBarAppConfig
- let globalRefreshFrequency: RefreshFrequency
- let installation: CrawlAppInstallation?
- let status: CrawlAppStatus?
- let latestResult: CrawlCommandResult?
- let isRefreshing: Bool
- let runningAction: String?
- let actionMessage: String?
- let refreshStatus: () -> Void
- let runAction: (String) -> Void
- let installApp: () -> Void
- let backupDatabases: () -> Void
- let openDataFolder: () -> Void
- let configValueChanged: (CrawlAppManifest.ConfigOption, String?) -> Void
- let save: () -> Void
- let saveDebounced: () -> Void
-
- private var manifest: CrawlAppManifest? { self.installation?.manifest ?? BuiltInCrawlApps.manifest(for: self.app.id) }
-
- var body: some View {
- VStack(alignment: .leading, spacing: 16) {
- self.header
- if self.isComingSoon {
- self.comingSoonContent
- .frame(maxWidth: .infinity, maxHeight: .infinity, alignment: .topLeading)
- .padding(.top, 6)
- } else if self.isMissingBinary {
- self.notInstalledContent
- .frame(maxWidth: .infinity, maxHeight: .infinity, alignment: .topLeading)
- .padding(.top, 6)
- } else {
- ScrollView {
- VStack(alignment: .leading, spacing: 20) {
- self.statusSection
- self.dataSection
- self.syncSection
- self.configurationSection
- }
- .frame(maxWidth: .infinity, alignment: .leading)
- .padding(.bottom, 10)
- .padding(.horizontal, 2)
- }
- }
- }
- .frame(
- maxWidth: .infinity,
- maxHeight: .infinity,
- alignment: .topLeading)
- }
-
- private var header: some View {
- HStack(alignment: .center, spacing: 14) {
- CrawlBarBrandIcon(manifest: self.manifest, appID: self.app.id)
- .frame(width: 40, height: 40)
- VStack(alignment: .leading, spacing: 5) {
- HStack(spacing: 8) {
- Text(CrawlBarCrawlerTitle.text(for: self.app.id, manifest: self.manifest))
- .font(.title3.weight(.semibold))
- .lineLimit(1)
- .truncationMode(.tail)
- CrawlBarStatusPill(state: self.effectiveState)
- }
- Text(self.manifest?.description ?? self.app.id.rawValue)
- .font(.subheadline)
- .foregroundStyle(.secondary)
- .lineLimit(2)
- .truncationMode(.tail)
- }
- .frame(minWidth: 0, alignment: .leading)
- Spacer()
- HStack(spacing: 6) {
- if !self.isComingSoon {
- Button(action: self.refreshStatus) {
- Image(systemName: self.isRefreshing ? "hourglass" : "arrow.clockwise")
- }
- .buttonStyle(.borderless)
- .accessibilityLabel("Refresh status")
- }
- }
- .controlSize(.small)
- }
- }
-
- private var comingSoonContent: some View {
- VStack(spacing: 12) {
- Spacer(minLength: 44)
- CrawlBarBrandIcon(manifest: self.manifest, appID: self.app.id)
- .frame(width: 72, height: 72)
- Text("\(self.manifest?.displayName ?? "This crawler") has not shipped yet")
- .font(.title3.weight(.semibold))
- .multilineTextAlignment(.center)
- Text("CrawlBar will let you know when it is ready.")
- .font(.subheadline)
- .foregroundStyle(.secondary)
- .multilineTextAlignment(.center)
- .frame(maxWidth: 320)
- Spacer(minLength: 0)
- }
- .frame(maxWidth: .infinity, maxHeight: .infinity, alignment: .center)
- }
-
- private var notInstalledContent: some View {
- VStack(spacing: 12) {
- Spacer(minLength: 44)
- CrawlBarBrandIcon(manifest: self.manifest, appID: self.app.id)
- .frame(width: 72, height: 72)
- Text("\(CrawlBarCrawlerTitle.text(for: self.app.id, manifest: self.manifest)) is not installed")
- .font(.title3.weight(.semibold))
- .multilineTextAlignment(.center)
- Text("Install \(self.manifest?.binary.name ?? self.app.id.rawValue) to enable sync, search, and status checks.")
- .font(.subheadline)
- .foregroundStyle(.secondary)
- .multilineTextAlignment(.center)
- .frame(maxWidth: 360)
- if self.manifest?.install != nil {
- Button {
- self.installApp()
- } label: {
- Label("Install", systemImage: "square.and.arrow.down")
- }
- .disabled(self.runningAction != nil)
- }
- if self.runningAction == "install" {
- HStack(spacing: 6) {
- ProgressView()
- .controlSize(.small)
- Text("Installing...")
- .font(.caption)
- .foregroundStyle(.secondary)
- }
- } else if let actionMessage {
- Text(actionMessage)
- .font(.caption)
- .foregroundStyle(.secondary)
- }
- Spacer(minLength: 0)
- }
- .frame(maxWidth: .infinity, maxHeight: .infinity, alignment: .center)
- }
-
- private var statusSection: some View {
- CrawlBarDetailSection(title: "Status") {
- LazyVGrid(
- columns: [
- GridItem(.flexible(minimum: 260), spacing: 14, alignment: .top),
- GridItem(.flexible(minimum: 260), spacing: 14, alignment: .top),
- ],
- alignment: .leading,
- spacing: 14)
- {
- self.statusSummary
- self.sourceSummary
- self.latestRunSummary
- }
- }
- }
-
- private var dataSection: some View {
- CrawlBarDetailSection(title: "Data") {
- self.remoteStore
- if !self.usesRemoteStore {
- self.databases
- }
- self.metrics
- }
- }
-
- private var syncSection: some View {
- CrawlBarDetailSection(title: "Sync") {
- self.syncSettings
- self.cloudArchiveSettings
- self.gitShareSettings
- }
- }
-
- private var configurationSection: some View {
- CrawlBarDetailSection(title: "Configuration") {
- self.configuration
- self.paths
- self.privacy
- }
- }
-
- private var statusSummary: some View {
- CrawlBarPanel(title: "Status") {
- Grid(alignment: .leading, horizontalSpacing: 14, verticalSpacing: 8) {
- GridRow {
- CrawlBarFact(label: "Current", value: self.status?.summary ?? self.statusFallback)
- CrawlBarFact(label: "Last Sync", value: self.lastSyncSummary)
- }
- GridRow {
- CrawlBarFact(
- label: "Databases",
- value: self.databaseSummary)
- CrawlBarFact(label: "Binary", value: self.binarySummary)
- }
- }
- if let issue = self.primaryIssue {
- CrawlBarIssueBanner(message: issue, state: self.issueState)
- }
- }
- }
-
- private var sourceSummary: some View {
- CrawlBarPanel(title: "Sources") {
- Grid(alignment: .leading, horizontalSpacing: 14, verticalSpacing: 8) {
- GridRow {
- CrawlBarFact(label: "Refresh", value: self.refreshSourceSummary)
- CrawlBarFact(label: "Archive", value: self.archiveSourceSummary)
- }
- GridRow {
- CrawlBarFact(label: "Snapshot", value: self.snapshotSummary)
- CrawlBarFact(label: "Config", value: self.configSourceSummary)
- }
- }
- }
- }
-
- private var latestRunSummary: some View {
- CrawlBarPanel(title: "Latest Run") {
- if let latestResult {
- HStack(spacing: 8) {
- CrawlBarStatusDot(state: latestResult.succeeded ? .current : .error)
- Text(Self.actionTitle(latestResult.action))
- .font(.callout.weight(.medium))
- Text(latestResult.succeeded ? "finished" : "failed")
- .font(.callout)
- .foregroundStyle(latestResult.succeeded ? Color.secondary : Color.red)
- Spacer(minLength: 8)
- Text(CrawlBarDateText.relative(latestResult.finishedAt))
- .font(.caption)
- .foregroundStyle(.secondary)
- .lineLimit(1)
- }
- if latestResult.shouldShowExitCode {
- CrawlBarFact(label: "Exit", value: "\(latestResult.exitCode)")
- }
- if let output = latestResult.userFacingRunMessage {
- Text(output)
- .font(.caption)
- .foregroundStyle(latestResult.succeeded ? Color.secondary : Color.red)
- .lineLimit(3)
- .truncationMode(.tail)
- .textSelection(.enabled)
- }
- } else {
- Text("No action logs yet")
- .font(.callout)
- .foregroundStyle(.secondary)
- }
- }
- }
-
- @ViewBuilder
- private var remoteStore: some View {
- if let remoteStore = self.remoteStoreSummary {
- CrawlBarPanel(title: remoteStore.title) {
- CrawlBarFact(label: "Remote", value: remoteStore.remote)
- if let archive = remoteStore.archive {
- CrawlBarFact(label: "Archive", value: archive)
- }
- if let repoPath = remoteStore.repoPath {
- CrawlBarFact(label: "Checkout", value: repoPath)
- }
- if let branch = remoteStore.branch {
- CrawlBarFact(label: "Branch", value: branch)
- }
- if let bundle = remoteStore.bundle {
- CrawlBarFact(label: "Bundle", value: bundle)
- }
- if let compressed = remoteStore.compressed {
- CrawlBarFact(label: "Compressed", value: compressed)
- }
- if let parts = remoteStore.parts {
- CrawlBarFact(label: "Parts", value: parts)
- }
- if let lastIngest = remoteStore.lastIngest {
- CrawlBarFact(label: "Ingest", value: lastIngest)
- }
- if let databasePath = self.status?.databasePath {
- CrawlBarFact(label: "Local index", value: URL(fileURLWithPath: databasePath).lastPathComponent)
- }
- }
- }
- }
-
- @ViewBuilder
- private var databases: some View {
- if let databases = self.status?.databases, !databases.isEmpty {
- CrawlBarPanel(title: "Databases") {
- HStack {
- Spacer(minLength: 0)
- Button {
- self.openDataFolder()
- } label: {
- Image(systemName: "folder")
- }
- .buttonStyle(.borderless)
- .accessibilityLabel("Open data folder")
- Button {
- self.backupDatabases()
- } label: {
- Image(systemName: "archivebox")
- }
- .buttonStyle(.borderless)
- .disabled(self.runningAction != nil)
- .accessibilityLabel("Back up database files")
- Text("\(databases.count)")
- .font(.caption.weight(.medium))
- .foregroundStyle(.secondary)
- .monospacedDigit()
- }
- VStack(spacing: 0) {
- ForEach(databases) { database in
- CrawlBarDatabaseRow(database: database)
- if database.id != databases.last?.id {
- Divider()
- .padding(.leading, 28)
- }
- }
- }
- }
- } else {
- CrawlBarPanel(title: "Databases") {
- Label("No database metadata yet", systemImage: "internaldrive")
- .foregroundStyle(.secondary)
- }
- }
- }
-
- @ViewBuilder
- private var metrics: some View {
- if !self.overviewCounts.isEmpty {
- CrawlBarPanel(title: "Counts") {
- HStack(alignment: .firstTextBaseline, spacing: 8) {
- Spacer(minLength: 0)
- Text(self.overviewDataScope)
- .font(.caption)
- .foregroundStyle(.secondary)
- }
- VStack(spacing: 0) {
- ForEach(self.overviewCounts) { count in
- CrawlBarMetricRow(label: count.label, value: "\(count.value)")
- if count.id != self.overviewCounts.last?.id {
- Divider()
- }
- }
- }
- }
- } else {
- CrawlBarPanel(title: "Counts") {
- Label("No count metrics yet", systemImage: "number")
- .foregroundStyle(.secondary)
- }
- }
- }
-
- private var syncSettings: some View {
- CrawlBarPanel(title: "Sync") {
- VStack(alignment: .leading, spacing: 10) {
- CrawlBarSwitchRow(
- title: "Enable crawler",
- caption: "Allow CrawlBar to run actions and show live status.",
- isOn: self.$app.enabled)
- .onChange(of: self.app.enabled) { self.save() }
- CrawlBarSwitchRow(
- title: "Show in menu bar",
- caption: "Include this crawler in the menu bar status menu.",
- isOn: self.$app.showInMenuBar)
- .disabled(!self.app.enabled)
- .onChange(of: self.app.showInMenuBar) { self.save() }
- CrawlBarSwitchRow(
- title: "Run on schedule",
- caption: "Refresh this crawler automatically in the background.",
- isOn: self.$app.autoRefreshEnabled)
- .disabled(!self.app.enabled)
- .onChange(of: self.app.autoRefreshEnabled) { self.save() }
- CrawlBarSwitchRow(
- title: "Use default schedule",
- caption: "Follow the global interval from General settings.",
- isOn: self.usesGlobalRefreshBinding)
- .disabled(!self.app.enabled || !self.app.autoRefreshEnabled)
- }
- CrawlBarControlRow(
- title: "Custom schedule",
- caption: "Overrides the global interval for this crawler.")
- {
- Picker("Custom schedule", selection: self.refreshFrequencyBinding) {
- ForEach(RefreshFrequency.allCases, id: \.self) { frequency in
- Text(CrawlBarFrequencyLabel.text(for: frequency)).tag(frequency)
- }
- }
- .labelsHidden()
- .frame(width: 180)
- }
- .disabled(!self.app.enabled || !self.app.autoRefreshEnabled || self.app.refreshFrequency == nil)
- Text("Default schedule: \(CrawlBarFrequencyLabel.text(for: self.globalRefreshFrequency))")
- .font(.caption)
- .foregroundStyle(.secondary)
- HStack(spacing: 8) {
- if self.installation?.binaryPath == nil, self.manifest?.install != nil {
- Button {
- self.installApp()
- } label: {
- Label("Install", systemImage: "square.and.arrow.down")
- }
- }
- if self.commandAvailable(self.app.preferredRefreshAction ?? "refresh") {
- Button {
- self.runAction(self.app.preferredRefreshAction ?? "refresh")
- } label: {
- Label("Sync Now", systemImage: "arrow.triangle.2.circlepath")
- }
- }
- if self.commandAvailable("desktop-cache-import") {
- Button {
- self.runAction("desktop-cache-import")
- } label: {
- Label("Import Desktop", systemImage: "externaldrive.connected.to.line.below")
- }
- }
- if self.commandAvailable("doctor") {
- Button {
- self.runAction("doctor")
- } label: {
- Label("Run Doctor", systemImage: "stethoscope")
- }
- }
- if self.commandAvailable("unlock") {
- Button {
- self.runAction("unlock")
- } label: {
- Label("Unlock", systemImage: "key")
- }
- }
- if self.nativeAppAvailable {
- Button {
- self.openNativeApp()
- } label: {
- Label("Open Source App", systemImage: "app")
- }
- }
- if self.commandAvailable(self.app.preferredUpdateAction ?? "update") {
- Button {
- self.runAction(self.app.preferredUpdateAction ?? "update")
- } label: {
- Label("Update", systemImage: "square.and.arrow.down")
- }
- }
- }
- .disabled(self.runningAction != nil)
- if let runningAction {
- HStack(spacing: 6) {
- ProgressView()
- .controlSize(.small)
- Text("Running \(runningAction)...")
- .font(.caption)
- .foregroundStyle(.secondary)
- }
- } else if let actionMessage {
- Text(actionMessage)
- .font(.caption)
- .foregroundStyle(.secondary)
- }
- }
- }
-
- private var gitShareSettings: some View {
- CrawlBarPanel(title: "Git Snapshot") {
- CrawlBarSwitchRow(
- title: "Manage snapshot",
- caption: "Keep a local Git export for this crawler's shareable data.",
- isOn: self.$app.shareEnabled)
- .onChange(of: self.app.shareEnabled) { self.save() }
- if self.hasSnapshotRemote {
- CrawlBarSwitchRow(
- title: "Publish after sync",
- caption: "Push the snapshot after a scheduled or manual sync.",
- isOn: self.$app.shareAfterRefresh)
- .disabled(!self.app.shareEnabled)
- .onChange(of: self.app.shareAfterRefresh) { self.save() }
- HStack(spacing: 8) {
- if self.commandAvailable(self.app.preferredShareAction ?? "publish") {
- Button {
- self.runAction(self.app.preferredShareAction ?? "publish")
- } label: {
- Label("Publish Snapshot", systemImage: "arrow.up.circle")
- }
- }
- if self.commandAvailable(self.app.preferredUpdateAction ?? "update") {
- Button {
- self.runAction(self.app.preferredUpdateAction ?? "update")
- } label: {
- Label("Pull Updates", systemImage: "arrow.down.circle")
- }
- }
- }
- .disabled(!self.app.shareEnabled || self.runningAction != nil)
- }
- if self.hasSnapshotInfo {
- Divider()
- if let shareRepoPath = self.shareRepoPath {
- CrawlBarFact(label: "Share Repo", value: shareRepoPath)
- }
- if let shareRemote = self.shareRemote {
- CrawlBarFact(label: "Remote", value: shareRemote)
- }
- if let shareBranch = self.shareBranch {
- CrawlBarFact(label: "Branch", value: shareBranch)
- }
- }
- }
- }
-
- @ViewBuilder
- private var cloudArchiveSettings: some View {
- if self.commandAvailable("cloud-publish") || self.commandAvailable("remote-status") || self.commandAvailable("remote-archives") {
- CrawlBarPanel(title: "Cloudflare Archive") {
- CrawlBarOptionLabel(
- title: "Remote SQLite archive",
- caption: "Publish a compressed SQLite bundle and use the configured Worker archive for live reads.")
- HStack(spacing: 8) {
- if self.commandAvailable("cloud-publish") {
- Button {
- self.runAction("cloud-publish")
- } label: {
- Label("Publish Cloud", systemImage: "icloud.and.arrow.up")
- }
- }
- if self.commandAvailable("remote-status") {
- Button {
- self.runAction("remote-status")
- } label: {
- Label("Remote Status", systemImage: "antenna.radiowaves.left.and.right")
- }
- }
- if self.commandAvailable("remote-archives") {
- Button {
- self.runAction("remote-archives")
- } label: {
- Label("Archives", systemImage: "tray.full")
- }
- }
- }
- .disabled(self.runningAction != nil)
- if let remote = self.status?.remote {
- Divider()
- if let endpoint = remote.endpoint?.nilIfBlank {
- CrawlBarFact(label: "Endpoint", value: endpoint)
- }
- if let archive = remote.archive?.nilIfBlank {
- CrawlBarFact(label: "Archive", value: archive)
- }
- if let sqliteBundle = self.status?.sqliteBundle {
- CrawlBarFact(label: "Bundle", value: self.bundleSummary(sqliteBundle))
- }
- }
- }
- }
- }
-
- private var paths: some View {
- CrawlBarPanel(title: "Paths") {
- CrawlBarControlRow(
- title: "Binary path override",
- caption: "Leave empty to resolve the CLI from PATH.")
- {
- TextField("Optional", text: self.optionalText(\.binaryPath))
- .textFieldStyle(.roundedBorder)
- .frame(width: 260)
- .onSubmit(self.save)
- }
- CrawlBarControlRow(
- title: "Config path override",
- caption: "Leave empty to use the crawler default.")
- {
- TextField("Optional", text: self.optionalText(\.configPath))
- .textFieldStyle(.roundedBorder)
- .frame(width: 260)
- .onSubmit(self.save)
- }
- CrawlBarFact(label: "Default Config", value: self.manifest?.paths.defaultConfig ?? "None")
- CrawlBarFact(label: "Default Database", value: self.status?.databasePath ?? self.manifest?.paths.defaultDatabase ?? "Unknown")
- CrawlBarFact(label: "Logs", value: self.manifest?.paths.defaultLogs ?? "Unknown")
- }
- }
-
- @ViewBuilder
- private var configuration: some View {
- if self.manifest?.availability == .comingSoon {
- CrawlBarPanel(title: "Coming Soon") {
- CrawlBarFact(label: "CLI", value: self.manifest?.binary.name ?? self.app.id.rawValue)
- CrawlBarFact(label: "Config", value: self.manifest?.paths.defaultConfig ?? "Not declared")
- }
- } else if let manifest = self.manifest, !manifest.configOptions.isEmpty {
- ForEach(self.configSections(for: manifest)) { section in
- CrawlBarPanel(title: section.title, caption: section.caption) {
- ForEach(section.options) { option in
- CrawlBarConfigOptionField(
- option: option,
- value: self.configValueBinding(for: option),
- disabledReason: self.configDisabledReason(for: option))
- }
- }
- }
- }
- }
-
- private var privacy: some View {
- CrawlBarPanel(title: "Privacy") {
- CrawlBarFact(
- label: "Private Messages",
- value: self.manifest?.privacy.containsPrivateMessages == true ? "Possible local data" : "Not declared")
- CrawlBarFact(label: "Local-only scopes", value: self.manifest?.privacy.localOnlyScopes.joined(separator: ", ").nilIfBlank ?? "None")
- CrawlBarFact(label: "Action logs", value: CrawlActionLogStore.defaultDirectory().path)
- }
- }
-
- private var primaryIssue: String? {
- if let error = self.status?.errors.first?.nilIfBlank,
- !self.issueDuplicatesStatusSummary(error)
- {
- return error
- }
- if let warning = self.status?.warnings.first?.nilIfBlank,
- !self.issueDuplicatesStatusSummary(warning)
- {
- return warning
- }
- guard let latestResult, !latestResult.succeeded else { return nil }
- return latestResult.userFacingRunMessage ?? "\(Self.actionTitle(latestResult.action)) failed with exit \(latestResult.exitCode)"
- }
-
- private func issueDuplicatesStatusSummary(_ issue: String) -> Bool {
- guard let summary = self.status?.summary.nilIfBlank else { return false }
- return summary.localizedCaseInsensitiveContains(issue)
- }
-
- private var issueState: CrawlAppState {
- self.status?.isRecoverableGraincrawlSourceFailure == true ? .stale : .error
- }
-
- private var issueColor: Color {
- self.issueState == .stale ? .yellow : .red
- }
-
- private var refreshSourceSummary: String {
- if self.app.id == BuiltInCrawlApps.gitcrawlID {
- return "GitHub API (remote)"
- }
- if self.app.id == BuiltInCrawlApps.graincrawlID {
- switch self.app.configValues["preferred_source"]?.nilIfBlank ?? "private-api" {
- case "desktop-cache":
- return "Granola desktop cache"
- case "private-api":
- return "Granola private API (remote)"
- default:
- return self.app.configValues["preferred_source"] ?? "Granola source"
- }
- }
- if self.manifest?.capabilities.contains(.desktopCache) == true {
- return "Desktop cache (local)"
- }
- if self.manifest?.capabilities.contains(.refresh) == true {
- return "Crawler CLI"
- }
- return "Not available"
- }
-
- private var archiveSourceSummary: String {
- if let remoteStore = self.remoteStoreSummary {
- return remoteStore.shortName
- }
- if let database = self.primaryDatabase ?? self.status?.databases.first {
- return database.path.map { URL(fileURLWithPath: $0).lastPathComponent } ?? database.label
- }
- if let databasePath = self.status?.databasePath ?? self.manifest?.paths.defaultDatabase {
- return URL(fileURLWithPath: databasePath).lastPathComponent
- }
- return "Unknown"
- }
-
- private var snapshotSummary: String {
- guard self.app.shareEnabled else { return "Off" }
- guard let share = self.status?.share else {
- return self.shareRepoPath == nil ? "Configured, not reported" : "Local snapshot"
- }
- let location = share.remote?.nilIfBlank ?? share.repoPath?.nilIfBlank ?? "Local snapshot"
- if let branch = share.branch?.nilIfBlank {
- return "\(location) · \(branch)"
- }
- return location
- }
-
- private var hasSnapshotRemote: Bool {
- self.app.shareEnabled && self.shareRemote != nil
- }
-
- private var hasSnapshotInfo: Bool {
- self.app.shareEnabled && (self.shareRepoPath != nil || self.shareRemote != nil || self.shareBranch != nil)
- }
-
- private var shareRepoPath: String? {
- self.status?.share?.repoPath?.nilIfBlank ?? self.manifest?.paths.defaultShare?.nilIfBlank
- }
-
- private var shareRemote: String? {
- self.status?.share?.remote?.nilIfBlank
- }
-
- private var shareBranch: String? {
- self.status?.share?.branch?.nilIfBlank
- }
-
- private var configSourceSummary: String {
- if let configPath = self.status?.configPath ?? self.app.configPath ?? self.manifest?.paths.defaultConfig {
- return URL(fileURLWithPath: configPath).lastPathComponent
- }
- return "None"
- }
-
- private static func actionTitle(_ action: String) -> String {
- switch action {
- case "refresh":
- "Sync"
- case "doctor":
- "Doctor"
- case "unlock":
- "Unlock"
- case "publish":
- "Publish"
- case "cloud-publish":
- "Cloud Publish"
- case "remote-status":
- "Remote Status"
- case "remote-archives":
- "Remote Archives"
- case "update":
- "Update"
- case "desktop-cache-import":
- "Desktop Import"
- default:
- action
- }
- }
-
- private var effectiveState: CrawlAppState {
- if self.isComingSoon { return .disabled }
- if !self.app.enabled { return .disabled }
- if self.installation?.binaryPath == nil { return .needsConfig }
- let state = self.status?.state ?? .unknown
- if self.status?.isRecoverableGraincrawlSourceFailure == true {
- return .stale
- }
- return state
- }
-
- private var statusFallback: String {
- switch self.effectiveState {
- case .disabled where self.isComingSoon:
- "Coming soon"
- case .needsConfig:
- "\(self.manifest?.binary.name ?? self.app.id.rawValue) is not on PATH"
- case .disabled:
- "Disabled in CrawlBar"
- default:
- "Waiting for status"
- }
- }
-
- private var databaseSummary: String {
- guard let status else { return "Unknown" }
- if let remoteStore = self.remoteStoreSummary {
- return remoteStore.databaseSummary
- }
- if self.app.id == BuiltInCrawlApps.gitcrawlID {
- return self.summaryText(label: "GitHub archives", bytes: self.totalDatabaseBytes)
- }
- if status.databases.count > 1 {
- return self.summaryText(label: "\(status.databases.count) databases", bytes: self.totalDatabaseBytes)
- }
- if let primaryDatabase = self.primaryDatabase ?? status.databases.first {
- let size = primaryDatabase.bytes ?? status.databaseBytes
- return self.summaryText(label: primaryDatabase.label, bytes: size)
- }
- return status.databasePath.map { URL(fileURLWithPath: $0).lastPathComponent } ?? "Unknown"
- }
-
- private var binarySummary: String {
- if self.isComingSoon { return "Coming soon" }
- return self.installation?.binaryPath == nil ? "Missing" : "Found"
- }
-
- private var lastSyncSummary: String {
- guard let date = self.lastSyncDate else { return "Never" }
- return "Synced \(CrawlBarDateText.relative(date))"
- }
-
- private var lastSyncDate: Date? {
- if let lastSyncAt = self.status?.lastSyncAt {
- return lastSyncAt
- }
- if let modifiedAt = self.primaryDatabase?.modifiedAt {
- return modifiedAt
- }
- return self.status?.databases.compactMap(\.modifiedAt).max()
- }
-
- private var primaryDatabase: CrawlDatabaseResource? {
- self.status?.databases.first(where: { $0.isPrimary })
- }
-
- private var totalDatabaseBytes: Int? {
- guard let status else { return nil }
- let total = status.databases.compactMap(\.bytes).reduce(0, +)
- if total > 0 { return total }
- return status.databaseBytes
- }
-
- private var overviewCounts: [CrawlCount] {
- if let primaryCounts = self.primaryDatabase?.counts, !primaryCounts.isEmpty {
- return primaryCounts
- }
- let databaseCounts = self.totalCountsAcrossDatabases()
- if !databaseCounts.isEmpty {
- return databaseCounts
- }
- return self.status?.counts ?? []
- }
-
- private var overviewDataScope: String {
- if let remoteStore = self.remoteStoreSummary {
- return remoteStore.dataScope
- }
- if self.primaryDatabase?.counts.isEmpty == false {
- return "Active database"
- }
- if let count = self.status?.databases.count, count > 1, !self.totalCountsAcrossDatabases().isEmpty {
- return "Total across \(count) databases"
- }
- return "Connected database"
- }
-
- private func totalCountsAcrossDatabases() -> [CrawlCount] {
- let counts = self.status?.databases.flatMap(\.counts) ?? []
- guard !counts.isEmpty else { return [] }
- var labels: [String: String] = [:]
- var values: [String: Int] = [:]
- for count in counts {
- labels[count.id] = labels[count.id] ?? count.label
- values[count.id, default: 0] += count.value
- }
- return values.keys.sorted().map { id in
- CrawlCount(id: id, label: labels[id] ?? id, value: values[id] ?? 0)
- }
- }
-
- private func summaryText(label: String, bytes: Int?) -> String {
- [
- label,
- bytes.map { CrawlBarFileSizeText.string(fromByteCount: Int64($0)) },
- ].compactMap { $0?.nilIfBlank }.joined(separator: " · ")
- }
-
- private func bundleSummary(_ bundle: CrawlSQLiteBundleStatus) -> String {
- [
- bundle.format?.nilIfBlank,
- bundle.compression?.nilIfBlank,
- bundle.compressedBytes.map { CrawlBarFileSizeText.string(fromByteCount: Int64($0)) },
- bundle.partCount.map { "\($0) part\($0 == 1 ? "" : "s")" },
- ].compactMap { $0 }.joined(separator: " · ")
- }
-
- private var isComingSoon: Bool {
- self.manifest?.availability == .comingSoon
- }
-
- private var isMissingBinary: Bool {
- self.manifest?.availability == .available && self.installation?.binaryPath == nil
- }
-
- private var usesRemoteStore: Bool {
- self.remoteStoreSummary != nil
- }
-
- private var remoteStoreSummary: CrawlBarRemoteStoreSummary? {
- if let remote = self.status?.remote, remote.enabled {
- let database = self.status?.databases.first(where: { $0.endpoint != nil || $0.archive != nil })
- let endpoint = remote.endpoint?.nilIfBlank ?? database?.endpoint?.nilIfBlank ?? "Cloudflare remote"
- let archive = remote.archive?.nilIfBlank ?? database?.archive?.nilIfBlank
- return CrawlBarRemoteStoreSummary(
- remote: endpoint,
- archive: archive,
- kind: .cloudflare,
- sqliteBundle: self.status?.sqliteBundle,
- sqliteObject: self.status?.sqliteObject,
- lastIngestAt: remote.lastIngestAt ?? remote.lastSyncAt)
- }
- if self.status?.share?.enabled == true, let remote = self.status?.share?.remote?.nilIfBlank {
- return CrawlBarRemoteStoreSummary(
- remote: remote,
- repoPath: self.status?.share?.repoPath?.nilIfBlank,
- branch: self.status?.share?.branch?.nilIfBlank,
- kind: .gitSnapshot)
- }
- guard self.app.id == BuiltInCrawlApps.gitcrawlID else { return nil }
- var paths = self.status?.databases.compactMap(\.path) ?? []
- if let databasePath = self.status?.databasePath {
- paths.append(databasePath)
- }
- guard let storePath = paths.first(where: { $0.contains("/gitcrawl-store/") || $0.contains("/gitcrawl-store-remote/") }) else {
- return nil
- }
- let repoPath = Self.repoPath(containing: "/data/", in: storePath)
- return CrawlBarRemoteStoreSummary(
- remote: "https://github.com/openclaw/gitcrawl-store.git",
- repoPath: repoPath,
- branch: nil,
- kind: .gitSnapshot)
- }
-
- private var usesGlobalRefreshBinding: Binding {
- Binding(
- get: { self.app.refreshFrequency == nil },
- set: {
- self.app.refreshFrequency = $0 ? nil : self.globalRefreshFrequency
- self.save()
- })
- }
-
- private var refreshFrequencyBinding: Binding {
- Binding(
- get: { self.app.refreshFrequency ?? self.globalRefreshFrequency },
- set: {
- self.app.refreshFrequency = $0
- self.save()
- })
- }
-
- private func commandAvailable(_ action: String) -> Bool {
- guard self.manifest?.availability == .available else { return false }
- return self.manifest?.commands[action] != nil && self.installation?.binaryPath != nil && self.app.enabled
- }
-
- private var nativeAppAvailable: Bool {
- guard let bundleIdentifier = self.manifest?.branding.bundleIdentifier?.nilIfBlank else { return false }
- return CrawlBarNativeAppLocator.url(for: bundleIdentifier) != nil
- }
-
- private func openNativeApp() {
- guard let bundleIdentifier = self.manifest?.branding.bundleIdentifier?.nilIfBlank,
- let url = CrawlBarNativeAppLocator.url(for: bundleIdentifier)
- else { return }
- NSWorkspace.shared.openApplication(at: url, configuration: NSWorkspace.OpenConfiguration())
- }
-
- private func optionalText(_ keyPath: WritableKeyPath) -> Binding {
- Binding(
- get: { self.app[keyPath: keyPath] ?? "" },
- set: {
- self.app[keyPath: keyPath] = $0.nilIfBlank
- self.saveDebounced()
- })
- }
-
- private func configValueBinding(for option: CrawlAppManifest.ConfigOption) -> Binding {
- Binding(
- get: { self.app.configValues[option.id] ?? option.defaultValue ?? "" },
- set: {
- let value = $0.nilIfBlank
- self.app.configValues[option.id] = value
- self.configValueChanged(option, value)
- })
- }
-
- private func configDisabledReason(for option: CrawlAppManifest.ConfigOption) -> String? {
- guard self.usesRemoteStore else { return nil }
- let optionText = [
- option.id,
- option.configKey,
- option.envVar,
- ].compactMap { $0?.lowercased() }.joined(separator: " ")
- guard optionText.contains("openai") || optionText.contains("embedding") else { return nil }
- return "Disabled while this crawler is using a remote store."
- }
-
- private static func repoPath(containing marker: String, in path: String) -> String? {
- guard let range = path.range(of: marker) else { return nil }
- return String(path[.. [CrawlBarConfigSection] {
- var optionsByID: [String: CrawlAppManifest.ConfigOption] = [:]
- for option in manifest.configOptions where optionsByID[option.id] == nil {
- optionsByID[option.id] = option
- }
- let sections = manifest.configSections.isEmpty
- ? [CrawlBarConfigSection(id: "config", title: "Configuration", optionIDs: manifest.configOptions.map(\.id))]
- : manifest.configSections.map {
- CrawlBarConfigSection(
- id: $0.id,
- title: $0.title,
- caption: $0.caption,
- optionIDs: $0.optionIDs)
- }
-
- let usedIDs = Set(sections.flatMap(\.optionIDs))
- let resolved = sections.compactMap { section -> CrawlBarConfigSection? in
- let options = section.optionIDs.compactMap { optionsByID[$0] }
- guard !options.isEmpty else { return nil }
- return section.resolved(options: options)
- }
- let extraOptions = manifest.configOptions.filter { !usedIDs.contains($0.id) }
- if extraOptions.isEmpty {
- return resolved
- }
- return resolved + [CrawlBarConfigSection(id: "advanced", title: "Advanced", optionIDs: [], options: extraOptions)]
- }
-}
-
-private struct CrawlBarConfigSection: Identifiable {
- var id: String
- var title: String
- var caption: String?
- var optionIDs: [String]
- var options: [CrawlAppManifest.ConfigOption]
-
- init(
- id: String,
- title: String,
- caption: String? = nil,
- optionIDs: [String],
- options: [CrawlAppManifest.ConfigOption] = [])
- {
- self.id = id
- self.title = title
- self.caption = caption
- self.optionIDs = optionIDs
- self.options = options
- }
-
- func resolved(options: [CrawlAppManifest.ConfigOption]) -> CrawlBarConfigSection {
- CrawlBarConfigSection(
- id: self.id,
- title: self.title,
- caption: self.caption,
- optionIDs: self.optionIDs,
- options: options)
- }
-}
-
-struct CrawlBarDetailSection: View {
- let title: String
- let content: Content
-
- init(title: String, @ViewBuilder content: () -> Content) {
- self.title = title
- self.content = content()
- }
-
- var body: some View {
- VStack(alignment: .leading, spacing: 10) {
- Text(self.title)
- .font(.headline.weight(.semibold))
- VStack(alignment: .leading, spacing: 14) {
- self.content
- }
- .frame(maxWidth: .infinity, alignment: .leading)
- }
- }
-}
-
-struct CrawlBarPanel: View {
- var title: String?
- var caption: String?
- @ViewBuilder var content: Content
-
- init(title: String? = nil, caption: String? = nil, @ViewBuilder content: () -> Content) {
- self.title = title
- self.caption = caption
- self.content = content()
- }
-
- var body: some View {
- VStack(alignment: .leading, spacing: 8) {
- if let title {
- Text(title)
- .font(.subheadline.weight(.semibold))
- .foregroundStyle(.secondary)
- }
- if let caption {
- Text(caption)
- .font(.footnote)
- .foregroundStyle(.tertiary)
- .fixedSize(horizontal: false, vertical: true)
- }
- VStack(alignment: .leading, spacing: 10) {
- self.content
- }
- .padding(14)
- .frame(maxWidth: .infinity, alignment: .leading)
- .background(.quaternary.opacity(0.38), in: RoundedRectangle(cornerRadius: 12, style: .continuous))
- .overlay {
- RoundedRectangle(cornerRadius: 12, style: .continuous)
- .strokeBorder(.white.opacity(0.055))
- }
- }
- }
-}
-
-struct CrawlBarStatusDot: View {
- let state: CrawlAppState
-
- var body: some View {
- Circle()
- .fill(self.color)
- .frame(width: 8, height: 8)
- .accessibilityLabel(CrawlBarStatusLabel.text(for: self.state))
- }
-
- private var color: Color {
- switch self.state {
- case .current:
- .green
- case .stale, .unknown:
- .yellow
- case .syncing:
- .blue
- case .needsConfig, .needsAuth, .error:
- .red
- case .disabled:
- .gray
- }
- }
-}
-
-struct CrawlBarStatusPill: View {
- let state: CrawlAppState
-
- var body: some View {
- HStack(spacing: 5) {
- CrawlBarStatusDot(state: self.state)
- Text(CrawlBarStatusLabel.text(for: self.state))
- .font(.caption.weight(.medium))
- }
- .padding(.horizontal, 8)
- .padding(.vertical, 4)
- .background(Color(nsColor: .quaternaryLabelColor).opacity(0.12))
- .clipShape(Capsule())
- }
-}
-
-struct CrawlBarFact: View {
- let label: String
- let value: String
-
- var body: some View {
- VStack(alignment: .leading, spacing: 3) {
- Text(self.label)
- .font(.caption)
- .foregroundStyle(.secondary)
- Text(self.value)
- .font(.callout)
- .lineLimit(2)
- .textSelection(.enabled)
- }
- .frame(maxWidth: .infinity, alignment: .leading)
- }
-}
-
-struct CrawlBarIssueBanner: View {
- let message: String
- let state: CrawlAppState
-
- var body: some View {
- HStack(alignment: .top, spacing: 8) {
- Image(systemName: "exclamationmark.triangle.fill")
- .font(.caption)
- .foregroundStyle(self.color)
- Text(self.message)
- .font(.caption)
- .foregroundStyle(self.color)
- .lineLimit(3)
- .textSelection(.enabled)
- }
- .padding(8)
- .frame(maxWidth: .infinity, alignment: .leading)
- .background(self.color.opacity(0.08))
- .clipShape(RoundedRectangle(cornerRadius: 6, style: .continuous))
- }
-
- private var color: Color {
- self.state == .stale ? .yellow : .red
- }
-}
-
-struct CrawlBarControlRow: View {
- let title: String
- let caption: String?
- @ViewBuilder var content: Content
-
- init(title: String, caption: String? = nil, @ViewBuilder content: () -> Content) {
- self.title = title
- self.caption = caption
- self.content = content()
- }
-
- var body: some View {
- HStack(alignment: .firstTextBaseline, spacing: 16) {
- VStack(alignment: .leading, spacing: 3) {
- Text(self.title)
- .font(.callout)
- if let caption {
- Text(caption)
- .font(.caption)
- .foregroundStyle(.secondary)
- .fixedSize(horizontal: false, vertical: true)
- }
- }
- .frame(maxWidth: .infinity, alignment: .leading)
- self.content
- }
- }
-}
-
-struct CrawlBarSwitchRow: View {
- let title: String
- let caption: String
- @Binding var isOn: Bool
-
- var body: some View {
- HStack(alignment: .center, spacing: 16) {
- CrawlBarOptionLabel(title: self.title, caption: self.caption)
- .frame(maxWidth: .infinity, alignment: .leading)
- Toggle(self.title, isOn: self.$isOn)
- .labelsHidden()
- .toggleStyle(.switch)
- .controlSize(.mini)
- }
- }
-}
-
-struct CrawlBarOptionLabel: View {
- let title: String
- let caption: String
-
- var body: some View {
- VStack(alignment: .leading, spacing: 3) {
- Text(self.title)
- .font(.callout)
- Text(self.caption)
- .font(.caption)
- .foregroundStyle(.secondary)
- .fixedSize(horizontal: false, vertical: true)
- }
- }
-}
-
-struct CrawlBarMetricRow: View {
- let label: String
- let value: String
-
- var body: some View {
- HStack(alignment: .firstTextBaseline, spacing: 8) {
- Text(self.label)
- .font(.callout)
- .foregroundStyle(.secondary)
- .lineLimit(1)
- Spacer(minLength: 8)
- Text(self.value)
- .font(.callout.weight(.semibold))
- .monospacedDigit()
- }
- .padding(.vertical, 7)
- .frame(maxWidth: .infinity, alignment: .leading)
- }
-}
-
-private struct CrawlBarRemoteStoreSummary {
- enum Kind {
- case gitSnapshot
- case cloudflare
- }
-
- var remote: String
- var archive: String? = nil
- var repoPath: String? = nil
- var branch: String? = nil
- var kind: Kind
- var sqliteBundle: CrawlSQLiteBundleStatus? = nil
- var sqliteObject: CrawlSQLiteObjectStatus? = nil
- var lastIngestAt: Date? = nil
-
- var title: String {
- switch self.kind {
- case .cloudflare:
- "Cloudflare Archive"
- case .gitSnapshot:
- "Remote Store"
- }
- }
-
- var shortName: String {
- if self.kind == .cloudflare {
- return self.archive?.nilIfBlank ?? "Cloudflare archive"
- }
- let trimmed = self.remote
- .replacingOccurrences(of: "https://github.com/", with: "")
- .replacingOccurrences(of: "git@github.com:", with: "")
- .replacingOccurrences(of: ".git", with: "")
- .nilIfBlank
- return trimmed ?? "Remote store"
- }
-
- var dataScope: String {
- switch self.kind {
- case .cloudflare:
- "Cloudflare remote"
- case .gitSnapshot:
- "Remote store"
- }
- }
-
- var databaseSummary: String {
- guard self.kind == .cloudflare else { return self.shortName }
- let pieces = [
- self.shortName,
- self.sqliteBundle?.compressedBytes.map { CrawlBarFileSizeText.string(fromByteCount: Int64($0)) },
- self.sqliteBundle?.compression?.nilIfBlank,
- ].compactMap { $0 }
- return pieces.isEmpty ? "Cloudflare archive" : pieces.joined(separator: " · ")
- }
-
- var bundle: String? {
- guard let sqliteBundle else { return nil }
- return [
- sqliteBundle.format?.nilIfBlank,
- sqliteBundle.compression?.nilIfBlank,
- ].compactMap { $0 }.joined(separator: " · ").nilIfBlank
- }
-
- var compressed: String? {
- let values = [
- sqliteBundle?.compressedBytes.map { CrawlBarFileSizeText.string(fromByteCount: Int64($0)) },
- sqliteBundle?.rawBytes.map { CrawlBarFileSizeText.string(fromByteCount: Int64($0)) + " raw" },
- sqliteObject?.bytes.map { CrawlBarFileSizeText.string(fromByteCount: Int64($0)) + " object" },
- ].compactMap { $0?.nilIfBlank }
- return values.isEmpty ? nil : values.joined(separator: " / ")
- }
-
- var parts: String? {
- sqliteBundle?.partCount.map { "\($0)" }
- }
-
- var lastIngest: String? {
- guard let lastIngestAt else { return nil }
- let formatter = RelativeDateTimeFormatter()
- formatter.unitsStyle = .abbreviated
- return formatter.localizedString(for: lastIngestAt, relativeTo: Date())
- }
-}
-
-struct CrawlBarDatabaseRow: View {
- let database: CrawlDatabaseResource
-
- var body: some View {
- HStack(alignment: .top, spacing: 10) {
- Image(systemName: self.iconName)
- .font(.body)
- .foregroundStyle(self.database.isPrimary ? .blue : .secondary)
- .frame(width: 18, height: 22)
- VStack(alignment: .leading, spacing: 5) {
- HStack(spacing: 6) {
- Text(self.database.label)
- .font(.callout.weight(.medium))
- .lineLimit(1)
- if self.database.isPrimary {
- Text("Primary")
- .font(.caption2.weight(.medium))
- .foregroundStyle(.blue)
- .padding(.horizontal, 5)
- .padding(.vertical, 2)
- .background(.blue.opacity(0.12))
- .clipShape(Capsule())
- }
- }
- Text(self.subtitle)
- .font(.caption)
- .foregroundStyle(.secondary)
- .lineLimit(1)
- .truncationMode(.middle)
- .textSelection(.enabled)
- if !self.database.counts.isEmpty {
- HStack(spacing: 8) {
- ForEach(self.database.counts.prefix(3)) { count in
- Text("\(count.value) \(count.label.lowercased())")
- .font(.caption2)
- .foregroundStyle(.tertiary)
- .monospacedDigit()
- .lineLimit(1)
- }
- }
- }
- }
- Spacer(minLength: 8)
- VStack(alignment: .trailing, spacing: 4) {
- if let bytes = self.database.bytes {
- Text(CrawlBarFileSizeText.string(fromByteCount: Int64(bytes)))
- .font(.caption)
- .foregroundStyle(.secondary)
- .monospacedDigit()
- }
- if let modifiedAt = self.database.modifiedAt {
- Text(CrawlBarDateText.relative(modifiedAt))
- .font(.caption2)
- .foregroundStyle(.tertiary)
- .lineLimit(1)
- }
- }
- }
- .padding(.vertical, 8)
- }
-
- private var subtitle: String {
- let pieces = [
- self.database.role,
- self.database.path,
- ].compactMap { $0?.nilIfBlank }
- return pieces.isEmpty ? self.database.kind.rawValue : pieces.joined(separator: " · ")
- }
-
- private var iconName: String {
- switch self.database.kind {
- case .sqlite:
- "internaldrive"
- case .cache:
- "externaldrive.connected.to.line.below"
- case .logical:
- "square.stack.3d.up"
- case .remote, .d1, .cloudflareD1:
- "cloud"
- case .sqliteBundle:
- "archivebox"
- }
- }
-}
-
-struct CrawlBarConfigOptionField: View {
- let option: CrawlAppManifest.ConfigOption
- @Binding var value: String
- var disabledReason: String?
-
- var body: some View {
- CrawlBarControlRow(title: self.option.label, caption: self.caption) {
- self.control
- }
- .disabled(self.disabledReason != nil)
- }
-
- @ViewBuilder
- private var control: some View {
- switch self.option.kind {
- case .secret:
- HStack(spacing: 8) {
- SecureField(self.option.placeholder ?? "Value", text: self.$value)
- .textFieldStyle(.roundedBorder)
- .frame(width: 300)
- Button {
- self.value = ""
- } label: {
- Image(systemName: "key.slash")
- }
- .buttonStyle(.borderless)
- .accessibilityLabel("Clear saved secret")
- }
- case .boolean:
- Toggle("", isOn: self.booleanBinding)
- .labelsHidden()
- .toggleStyle(.switch)
- .controlSize(.mini)
- case .choice:
- Picker("Value", selection: self.$value) {
- ForEach(self.choices, id: \.self) { choice in
- Text(choice).tag(choice)
- }
- }
- .labelsHidden()
- .frame(width: 220)
- case .string:
- TextField(self.option.placeholder ?? "Value", text: self.$value)
- .textFieldStyle(.roundedBorder)
- .frame(width: 300)
- case .number:
- TextField(self.option.placeholder ?? "0", text: self.$value)
- .textFieldStyle(.roundedBorder)
- .frame(width: 120)
- }
- }
-
- private var caption: String? {
- [
- self.disabledReason?.nilIfBlank,
- self.option.help?.nilIfBlank,
- self.metadata,
- ].compactMap { $0 }.joined(separator: "\n").nilIfBlank
- }
-
- private var metadata: String? {
- [
- self.option.envVar?.nilIfBlank,
- self.option.configKey?.nilIfBlank,
- ].compactMap { $0 }.joined(separator: " ").nilIfBlank
- }
-
- private var choices: [String] {
- var resolved = self.option.choices
- if let defaultValue = self.option.defaultValue?.nilIfBlank,
- !resolved.contains(defaultValue)
- {
- resolved.insert(defaultValue, at: 0)
- }
- if let currentValue = self.value.nilIfBlank,
- !resolved.contains(currentValue)
- {
- resolved.insert(currentValue, at: 0)
- }
- return resolved
- }
-
- private var booleanBinding: Binding {
- Binding(
- get: { ["1", "true", "yes", "on"].contains(self.value.lowercased()) },
- set: { self.value = $0 ? "true" : "false" })
- }
-}
-
-enum CrawlBarFrequencyLabel {
- static func text(for frequency: RefreshFrequency) -> String {
- switch frequency {
- case .manual:
- "Manual"
- case .fiveMinutes:
- "5 minutes"
- case .fifteenMinutes:
- "15 minutes"
- case .thirtyMinutes:
- "30 minutes"
- case .hourly:
- "Hourly"
- }
- }
-}
-
-enum CrawlBarStatusLabel {
- static func text(for state: CrawlAppState) -> String {
- switch state {
- case .current:
- "Current"
- case .stale:
- "Stale"
- case .syncing:
- "Syncing"
- case .needsConfig:
- "Needs Config"
- case .needsAuth:
- "Needs Auth"
- case .error:
- "Error"
- case .disabled:
- "Disabled"
- case .unknown:
- "Unknown"
- }
- }
-}
-
-@MainActor
-enum CrawlBarDateText {
- private static let relativeFormatter: RelativeDateTimeFormatter = {
- let formatter = RelativeDateTimeFormatter()
- formatter.unitsStyle = .abbreviated
- return formatter
- }()
-
- static func relative(_ date: Date) -> String {
- self.relativeFormatter.localizedString(for: date, relativeTo: Date())
- }
-}
-
-enum CrawlBarFileSizeText {
- private static let formatter = CrawlBarLockedByteCountFormatter()
-
- static func string(fromByteCount byteCount: Int64) -> String {
- self.formatter.string(fromByteCount: byteCount)
- }
-}
-
-private final class CrawlBarLockedByteCountFormatter: @unchecked Sendable {
- private let lock = NSLock()
- private let formatter: ByteCountFormatter = {
- let formatter = ByteCountFormatter()
- formatter.allowedUnits = [.useKB, .useMB, .useGB]
- formatter.countStyle = .file
- formatter.includesUnit = true
- formatter.includesCount = true
- return formatter
- }()
-
- func string(fromByteCount byteCount: Int64) -> String {
- self.lock.lock()
- defer { self.lock.unlock() }
- return self.formatter.string(fromByteCount: byteCount)
- }
-}
diff --git a/Sources/CrawlBar/SharedUI/StatusViews.swift b/Sources/CrawlBar/SharedUI/StatusViews.swift
new file mode 100644
index 0000000..4460ddc
--- /dev/null
+++ b/Sources/CrawlBar/SharedUI/StatusViews.swift
@@ -0,0 +1,68 @@
+import AppKit
+import CrawlBarCore
+import SwiftUI
+
+struct CrawlBarStatusDot: View {
+ let state: CrawlAppState
+
+ var body: some View {
+ Circle()
+ .fill(self.color)
+ .frame(width: 8, height: 8)
+ .accessibilityLabel(CrawlBarStatusLabel.text(for: self.state))
+ }
+
+ private var color: Color {
+ switch self.state {
+ case .current:
+ .green
+ case .stale, .unknown:
+ .yellow
+ case .syncing:
+ .blue
+ case .needsConfig, .needsAuth, .error:
+ .red
+ case .disabled:
+ .gray
+ }
+ }
+}
+
+struct CrawlBarStatusPill: View {
+ let state: CrawlAppState
+
+ var body: some View {
+ HStack(spacing: 5) {
+ CrawlBarStatusDot(state: self.state)
+ Text(CrawlBarStatusLabel.text(for: self.state))
+ .font(.caption.weight(.medium))
+ }
+ .padding(.horizontal, 8)
+ .padding(.vertical, 4)
+ .background(Color(nsColor: .quaternaryLabelColor).opacity(0.12))
+ .clipShape(Capsule())
+ }
+}
+
+enum CrawlBarStatusLabel {
+ static func text(for state: CrawlAppState) -> String {
+ switch state {
+ case .current:
+ "Current"
+ case .stale:
+ "Stale"
+ case .syncing:
+ "Syncing"
+ case .needsConfig:
+ "Needs Config"
+ case .needsAuth:
+ "Needs Auth"
+ case .error:
+ "Error"
+ case .disabled:
+ "Disabled"
+ case .unknown:
+ "Unknown"
+ }
+ }
+}
diff --git a/Sources/CrawlBar/SharedUI/TextFormatters.swift b/Sources/CrawlBar/SharedUI/TextFormatters.swift
new file mode 100644
index 0000000..d2fe8eb
--- /dev/null
+++ b/Sources/CrawlBar/SharedUI/TextFormatters.swift
@@ -0,0 +1,58 @@
+import CrawlBarCore
+import Foundation
+
+enum CrawlBarFrequencyLabel {
+ static func text(for frequency: RefreshFrequency) -> String {
+ switch frequency {
+ case .manual:
+ "Manual"
+ case .fiveMinutes:
+ "5 minutes"
+ case .fifteenMinutes:
+ "15 minutes"
+ case .thirtyMinutes:
+ "30 minutes"
+ case .hourly:
+ "Hourly"
+ }
+ }
+}
+
+@MainActor
+enum CrawlBarDateText {
+ private static let relativeFormatter: RelativeDateTimeFormatter = {
+ let formatter = RelativeDateTimeFormatter()
+ formatter.unitsStyle = .abbreviated
+ return formatter
+ }()
+
+ static func relative(_ date: Date) -> String {
+ self.relativeFormatter.localizedString(for: date, relativeTo: Date())
+ }
+}
+
+enum CrawlBarFileSizeText {
+ private static let formatter = CrawlBarLockedByteCountFormatter()
+
+ static func string(fromByteCount byteCount: Int64) -> String {
+ self.formatter.string(fromByteCount: byteCount)
+ }
+}
+
+private final class CrawlBarLockedByteCountFormatter: @unchecked Sendable {
+ private let lock = NSLock()
+ private let formatter: ByteCountFormatter = {
+ let formatter = ByteCountFormatter()
+ formatter.allowedUnits = [.useKB, .useMB, .useGB]
+ formatter.countStyle = .file
+ formatter.includesUnit = true
+ formatter.includesCount = true
+ return formatter
+ }()
+
+ func string(fromByteCount byteCount: Int64) -> String {
+ self.lock.lock()
+ defer { self.lock.unlock() }
+ return self.formatter.string(fromByteCount: byteCount)
+ }
+}
diff --git a/Sources/CrawlBarCLI/CLI.swift b/Sources/CrawlBarCLI/CLI.swift
index 0aec218..03ae41e 100644
--- a/Sources/CrawlBarCLI/CLI.swift
+++ b/Sources/CrawlBarCLI/CLI.swift
@@ -330,122 +330,6 @@ enum CrawlBarCLI {
return statusService.status(for: installation, timeoutSeconds: 30)
}
- private static func runConfig(_ options: CLIOptions, registry: CrawlAppRegistry) throws {
- let store = CrawlBarConfigStore()
- let nativeConfigStore = CrawlNativeConfigStore()
- switch options.positionals.first {
- case "path":
- print(store.fileURL.path)
- case "validate":
- _ = try store.loadOrCreateDefault()
- print("ok")
- case "init", nil:
- let config = try store.loadOrCreateDefault()
- try CLIOutput.writeJSON(config)
- case "get":
- let appID = try options.requiredAppID()
- let config = try store.loadOrCreateDefault(includeSecrets: options.revealSecrets)
- let installation = try registry.installation(for: appID)
- let baseAppConfig = config.appConfig(for: appID) ?? CrawlBarAppConfig(id: appID)
- let appConfig = installation.map {
- var copy = baseAppConfig
- copy.configValues = nativeConfigStore.resolvedConfigValues(appConfig: baseAppConfig, manifest: $0.manifest)
- return copy
- } ?? baseAppConfig
- let values = Self.configValues(
- appConfig: appConfig,
- manifest: installation?.manifest,
- key: options.key,
- revealSecrets: options.revealSecrets)
- if options.json {
- try CLIOutput.writeJSON(values)
- return
- }
- if let key = options.key {
- guard let value = values.first else {
- throw CLIError.usage("unknown config key for \(appID.rawValue): \(key)")
- }
- print(value.value ?? "")
- return
- }
- for value in values {
- print("\(value.id)\t\(value.value ?? "")")
- }
- case "set":
- let appID = try options.requiredAppID()
- guard let key = options.key?.nilIfBlank else {
- throw CLIError.usage("config set requires --key ")
- }
- guard let value = options.value else {
- throw CLIError.usage("config set requires --value ")
- }
- var config = try store.loadOrCreateDefault()
- guard let index = config.apps.firstIndex(where: { $0.id == appID }) else {
- throw CLIError.usage("unknown app: \(appID.rawValue)")
- }
- if value.nilIfBlank == nil {
- config.apps[index].configValues.removeValue(forKey: key)
- } else {
- config.apps[index].configValues[key] = value
- }
- let clearMissingSecretIDsByAppID: [CrawlAppID: Set] = value.nilIfBlank == nil ? [appID: [key]] : [:]
- try store.save(config, clearMissingSecretIDsByAppID: clearMissingSecretIDsByAppID)
- if let installation = try registry.installation(for: appID),
- let appConfig = config.appConfig(for: appID)
- {
- var nativeAppConfig = appConfig
- var resolvedValues = nativeConfigStore.resolvedConfigValues(
- appConfig: appConfig,
- manifest: installation.manifest)
- if value.nilIfBlank == nil {
- resolvedValues.removeValue(forKey: key)
- } else {
- resolvedValues[key] = value
- }
- nativeAppConfig.configValues = resolvedValues
- try nativeConfigStore.write(
- appConfig: nativeAppConfig,
- manifest: installation.manifest,
- clearMissingSecretIDs: clearMissingSecretIDsByAppID[appID] ?? [])
- }
- if options.json {
- try CLIOutput.writeJSON(["app_id": appID.rawValue, "key": key, "updated": "true"])
- return
- }
- print("ok")
- case let command?:
- throw CLIError.usage("unknown config command: \(command)")
- }
- }
-
- private static func configValues(
- appConfig: CrawlBarAppConfig,
- manifest: CrawlAppManifest?,
- key: String?,
- revealSecrets: Bool)
- -> [CLIConfigValue]
- {
- let options = manifest?.configOptions ?? []
- let knownIDs = Set(options.map(\.id))
- let extraOptions = appConfig.configValues.keys
- .filter { !knownIDs.contains($0) }
- .sorted()
- .map { CrawlAppManifest.ConfigOption(id: $0, label: $0) }
- return (options + extraOptions)
- .filter { key == nil || $0.id == key }
- .map { option in
- let rawValue = appConfig.configValues[option.id] ?? option.defaultValue
- let isSecret = option.kind == .secret
- return CLIConfigValue(
- id: option.id,
- label: option.label,
- value: isSecret && !revealSecrets && rawValue?.nilIfBlank != nil ? "********" : rawValue,
- secret: isSecret,
- envVar: option.envVar,
- configKey: option.configKey)
- }
- }
-
private static func printHelp() {
print("""
crawlbar commands:
@@ -466,107 +350,3 @@ enum CrawlBarCLI {
""")
}
}
-
-private struct CLIConfigValue: Encodable {
- var id: String
- var label: String
- var value: String?
- var secret: Bool
- var envVar: String?
- var configKey: String?
-
- private enum CodingKeys: String, CodingKey {
- case id
- case label
- case value
- case secret
- case envVar = "env_var"
- case configKey = "config_key"
- }
-}
-
-private struct CLIApp: Encodable {
- var id: String
- var displayName: String
- var enabled: Bool
- var available: Bool
- var availability: CrawlAppManifest.Availability
- var binaryPath: String?
- var configPath: String?
-
- init(_ installation: CrawlAppInstallation) {
- self.id = installation.id.rawValue
- self.displayName = installation.manifest.displayName
- self.enabled = installation.enabled
- self.available = installation.binaryPath != nil
- self.availability = installation.manifest.availability
- self.binaryPath = installation.binaryPath
- self.configPath = installation.configPathOverride
- }
-
- private enum CodingKeys: String, CodingKey {
- case id
- case displayName = "display_name"
- case enabled
- case available
- case availability
- case binaryPath = "binary_path"
- case configPath = "config_path"
- }
-}
-
-private struct CLIOptions {
- var json = false
- var appID: CrawlAppID?
- var key: String?
- var value: String?
- var revealSecrets = false
- var diagnostics = false
- var positionals: [String] = []
-
- init(_ arguments: ArraySlice) {
- var iterator = Array(arguments).makeIterator()
- while let argument = iterator.next() {
- switch argument {
- case "--json":
- self.json = true
- case "--app":
- if let value = iterator.next() {
- self.appID = CrawlAppID(rawValue: value)
- }
- case "--key":
- self.key = iterator.next()
- case "--value":
- self.value = iterator.next()
- case "--reveal":
- self.revealSecrets = true
- case "--diagnostics":
- self.diagnostics = true
- case "--":
- while let value = iterator.next() {
- self.positionals.append(value)
- }
- default:
- self.positionals.append(argument)
- }
- }
- }
-
- func requiredAppID() throws -> CrawlAppID {
- guard let appID else {
- throw CLIError.usage("--app is required")
- }
- return appID
- }
-}
-
-private enum CLIError: LocalizedError {
- case usage(String)
-
- var errorDescription: String? {
- switch self {
- case let .usage(message):
- message
- }
- }
-}
diff --git a/Sources/CrawlBarCLI/CLIConfigCommands.swift b/Sources/CrawlBarCLI/CLIConfigCommands.swift
new file mode 100644
index 0000000..f80c05a
--- /dev/null
+++ b/Sources/CrawlBarCLI/CLIConfigCommands.swift
@@ -0,0 +1,148 @@
+import CrawlBarCore
+import Foundation
+
+extension CrawlBarCLI {
+ static func runConfig(_ options: CLIOptions, registry: CrawlAppRegistry) throws {
+ let store = CrawlBarConfigStore()
+ let nativeConfigStore = CrawlNativeConfigStore()
+ switch options.positionals.first {
+ case "path":
+ print(store.fileURL.path)
+ case "validate":
+ _ = try store.loadOrCreateDefault()
+ print("ok")
+ case "init", nil:
+ let config = try store.loadOrCreateDefault()
+ try CLIOutput.writeJSON(config)
+ case "get":
+ try Self.printConfigValue(
+ options,
+ registry: registry,
+ store: store,
+ nativeConfigStore: nativeConfigStore)
+ case "set":
+ try Self.setConfigValue(
+ options,
+ registry: registry,
+ store: store,
+ nativeConfigStore: nativeConfigStore)
+ case let command?:
+ throw CLIError.usage("unknown config command: \(command)")
+ }
+ }
+
+ private static func printConfigValue(
+ _ options: CLIOptions,
+ registry: CrawlAppRegistry,
+ store: CrawlBarConfigStore,
+ nativeConfigStore: CrawlNativeConfigStore)
+ throws
+ {
+ let appID = try options.requiredAppID()
+ let config = try store.loadOrCreateDefault(includeSecrets: options.revealSecrets)
+ let installation = try registry.installation(for: appID)
+ let baseAppConfig = config.appConfig(for: appID) ?? CrawlBarAppConfig(id: appID)
+ let appConfig = installation.map {
+ var copy = baseAppConfig
+ copy.configValues = nativeConfigStore.resolvedConfigValues(appConfig: baseAppConfig, manifest: $0.manifest)
+ return copy
+ } ?? baseAppConfig
+ let values = Self.configValues(
+ appConfig: appConfig,
+ manifest: installation?.manifest,
+ key: options.key,
+ revealSecrets: options.revealSecrets)
+ if options.json {
+ try CLIOutput.writeJSON(values)
+ return
+ }
+ if let key = options.key {
+ guard let value = values.first else {
+ throw CLIError.usage("unknown config key for \(appID.rawValue): \(key)")
+ }
+ print(value.value ?? "")
+ return
+ }
+ for value in values {
+ print("\(value.id)\t\(value.value ?? "")")
+ }
+ }
+
+ private static func setConfigValue(
+ _ options: CLIOptions,
+ registry: CrawlAppRegistry,
+ store: CrawlBarConfigStore,
+ nativeConfigStore: CrawlNativeConfigStore)
+ throws
+ {
+ let appID = try options.requiredAppID()
+ guard let key = options.key?.nilIfBlank else {
+ throw CLIError.usage("config set requires --key ")
+ }
+ guard let value = options.value else {
+ throw CLIError.usage("config set requires --value ")
+ }
+ var config = try store.loadOrCreateDefault()
+ guard let index = config.apps.firstIndex(where: { $0.id == appID }) else {
+ throw CLIError.usage("unknown app: \(appID.rawValue)")
+ }
+ if value.nilIfBlank == nil {
+ config.apps[index].configValues.removeValue(forKey: key)
+ } else {
+ config.apps[index].configValues[key] = value
+ }
+ let clearMissingSecretIDsByAppID: [CrawlAppID: Set] = value.nilIfBlank == nil ? [appID: [key]] : [:]
+ try store.save(config, clearMissingSecretIDsByAppID: clearMissingSecretIDsByAppID)
+ if let installation = try registry.installation(for: appID),
+ let appConfig = config.appConfig(for: appID)
+ {
+ var nativeAppConfig = appConfig
+ var resolvedValues = nativeConfigStore.resolvedConfigValues(
+ appConfig: appConfig,
+ manifest: installation.manifest)
+ if value.nilIfBlank == nil {
+ resolvedValues.removeValue(forKey: key)
+ } else {
+ resolvedValues[key] = value
+ }
+ nativeAppConfig.configValues = resolvedValues
+ try nativeConfigStore.write(
+ appConfig: nativeAppConfig,
+ manifest: installation.manifest,
+ clearMissingSecretIDs: clearMissingSecretIDsByAppID[appID] ?? [])
+ }
+ if options.json {
+ try CLIOutput.writeJSON(["app_id": appID.rawValue, "key": key, "updated": "true"])
+ return
+ }
+ print("ok")
+ }
+
+ private static func configValues(
+ appConfig: CrawlBarAppConfig,
+ manifest: CrawlAppManifest?,
+ key: String?,
+ revealSecrets: Bool)
+ -> [CLIConfigValue]
+ {
+ let options = manifest?.configOptions ?? []
+ let knownIDs = Set(options.map(\.id))
+ let extraOptions = appConfig.configValues.keys
+ .filter { !knownIDs.contains($0) }
+ .sorted()
+ .map { CrawlAppManifest.ConfigOption(id: $0, label: $0) }
+ return (options + extraOptions)
+ .filter { key == nil || $0.id == key }
+ .map { option in
+ let rawValue = appConfig.configValues[option.id] ?? option.defaultValue
+ let isSecret = option.kind == .secret
+ return CLIConfigValue(
+ id: option.id,
+ label: option.label,
+ value: isSecret && !revealSecrets && rawValue?.nilIfBlank != nil ? "********" : rawValue,
+ secret: isSecret,
+ envVar: option.envVar,
+ configKey: option.configKey)
+ }
+ }
+}
diff --git a/Sources/CrawlBarCLI/CLIModels.swift b/Sources/CrawlBarCLI/CLIModels.swift
new file mode 100644
index 0000000..3f35a00
--- /dev/null
+++ b/Sources/CrawlBarCLI/CLIModels.swift
@@ -0,0 +1,106 @@
+import CrawlBarCore
+import Foundation
+
+struct CLIConfigValue: Encodable {
+ var id: String
+ var label: String
+ var value: String?
+ var secret: Bool
+ var envVar: String?
+ var configKey: String?
+
+ private enum CodingKeys: String, CodingKey {
+ case id
+ case label
+ case value
+ case secret
+ case envVar = "env_var"
+ case configKey = "config_key"
+ }
+}
+
+struct CLIApp: Encodable {
+ var id: String
+ var displayName: String
+ var enabled: Bool
+ var available: Bool
+ var availability: CrawlAppManifest.Availability
+ var binaryPath: String?
+ var configPath: String?
+
+ init(_ installation: CrawlAppInstallation) {
+ self.id = installation.id.rawValue
+ self.displayName = installation.manifest.displayName
+ self.enabled = installation.enabled
+ self.available = installation.binaryPath != nil
+ self.availability = installation.manifest.availability
+ self.binaryPath = installation.binaryPath
+ self.configPath = installation.configPathOverride
+ }
+
+ private enum CodingKeys: String, CodingKey {
+ case id
+ case displayName = "display_name"
+ case enabled
+ case available
+ case availability
+ case binaryPath = "binary_path"
+ case configPath = "config_path"
+ }
+}
+
+struct CLIOptions {
+ var json = false
+ var appID: CrawlAppID?
+ var key: String?
+ var value: String?
+ var revealSecrets = false
+ var diagnostics = false
+ var positionals: [String] = []
+
+ init(_ arguments: ArraySlice) {
+ var iterator = Array(arguments).makeIterator()
+ while let argument = iterator.next() {
+ switch argument {
+ case "--json":
+ self.json = true
+ case "--app":
+ if let value = iterator.next() {
+ self.appID = CrawlAppID(rawValue: value)
+ }
+ case "--key":
+ self.key = iterator.next()
+ case "--value":
+ self.value = iterator.next()
+ case "--reveal":
+ self.revealSecrets = true
+ case "--diagnostics":
+ self.diagnostics = true
+ case "--":
+ while let value = iterator.next() {
+ self.positionals.append(value)
+ }
+ default:
+ self.positionals.append(argument)
+ }
+ }
+ }
+
+ func requiredAppID() throws -> CrawlAppID {
+ guard let appID else {
+ throw CLIError.usage("--app is required")
+ }
+ return appID
+ }
+}
+
+enum CLIError: LocalizedError {
+ case usage(String)
+
+ var errorDescription: String? {
+ switch self {
+ case let .usage(message):
+ message
+ }
+ }
+}
diff --git a/Sources/CrawlBarCore/BuiltInApps/BuiltInCrawlApps+Birdclaw.swift b/Sources/CrawlBarCore/BuiltInApps/BuiltInCrawlApps+Birdclaw.swift
new file mode 100644
index 0000000..cad5938
--- /dev/null
+++ b/Sources/CrawlBarCore/BuiltInApps/BuiltInCrawlApps+Birdclaw.swift
@@ -0,0 +1,38 @@
+import Foundation
+
+public extension BuiltInCrawlApps {
+ static let birdclaw = CrawlAppManifest(
+ id: Self.birdclawID,
+ displayName: "X",
+ description: "X/Twitter connector through bird, with Birdclaw workspace support",
+ binary: .init(name: "bird"),
+ execution: .init(
+ kind: .local,
+ kindConfigID: "execution_mode",
+ targetConfigID: "remote_target",
+ runAsConfigID: "remote_run_as",
+ remoteBinary: "bird"),
+ branding: .init(symbolName: "xmark", accentColor: "#111111"),
+ paths: .init(
+ defaultConfig: "~/.birdclaw/config.json",
+ defaultDatabase: "~/.birdclaw/birdclaw.sqlite",
+ defaultCache: "~/.birdclaw/media"),
+ commands: [
+ "status": ["check", "--plain"],
+ "doctor": ["check", "--plain"],
+ "search": ["search", "-n", "10", "--json"],
+ ],
+ capabilities: [.status, .doctor, .search],
+ privacy: .init(containsPrivateMessages: true, exportsSecrets: false, localOnlyScopes: ["X archive", "DMs", "browser cookies"]),
+ configOptions: [
+ .init(id: "access_path", label: "Access path", kind: .choice, help: "Use bird first, or use Birdclaw when that host is authenticated through xurl.", defaultValue: "bird", choices: ["bird", "birdclaw"]),
+ .init(id: "execution_mode", label: "Run location", kind: .choice, help: "Run bird on this Mac or over SSH on another machine.", defaultValue: "local", choices: ["local", "remote"]),
+ .init(id: "remote_target", label: "SSH target", help: "SSH target that can run bird.", placeholder: "user@example-host"),
+ .init(id: "remote_run_as", label: "Run as user", help: "Optional remote Unix user for sudo -u, when bird is installed under a service account.", placeholder: "crawl"),
+ ],
+ configSections: [
+ .init(id: "execution", title: "Execution", optionIDs: ["access_path", "execution_mode"]),
+ .init(id: "remote", title: "Remote Host", optionIDs: ["remote_target", "remote_run_as"]),
+ ],
+ install: .init(method: .homebrew, package: "steipete/tap/bird"))
+}
diff --git a/Sources/CrawlBarCore/BuiltInApps/BuiltInCrawlApps+Discrawl.swift b/Sources/CrawlBarCore/BuiltInApps/BuiltInCrawlApps+Discrawl.swift
new file mode 100644
index 0000000..2ca024d
--- /dev/null
+++ b/Sources/CrawlBarCore/BuiltInApps/BuiltInCrawlApps+Discrawl.swift
@@ -0,0 +1,45 @@
+import Foundation
+
+public extension BuiltInCrawlApps {
+ static let discrawl = CrawlAppManifest(
+ id: Self.discrawlID,
+ displayName: "Discord",
+ description: "Local Discord guild and desktop-cache archive",
+ binary: .init(name: "discrawl"),
+ branding: .init(
+ symbolName: "antenna.radiowaves.left.and.right",
+ accentColor: "#5865F2",
+ bundleIdentifier: "com.hnc.Discord"),
+ paths: .init(
+ defaultConfig: "~/.discrawl/config.toml",
+ configEnv: "DISCRAWL_CONFIG",
+ defaultDatabase: "~/.discrawl/discrawl.db",
+ defaultCache: "~/.discrawl/cache",
+ defaultLogs: "~/.discrawl/logs",
+ defaultShare: "~/.discrawl/share"),
+ commands: [
+ "metadata": ["metadata", "--json"],
+ "status": ["status", "--json"],
+ "doctor": ["doctor", "--json"],
+ "refresh": ["--json", "cache-import"],
+ "desktop-cache-import": ["--json", "cache-import"],
+ "publish": ["--json", "publish"],
+ "update": ["--json", "update"],
+ "remote-status": ["remote", "status", "--json"],
+ "remote-archives": ["remote", "archives", "--json"],
+ "cloud-publish": ["cloud", "publish", "--sqlite-only", "--json"],
+ ],
+ capabilities: [.status, .doctor, .refresh, .publish, .subscribe, .update, .desktopCache, .remoteArchive, .cloudPublish],
+ statusRequiresSecrets: false,
+ privacy: .init(containsPrivateMessages: true, exportsSecrets: false, localOnlyScopes: ["@me"]),
+ configOptions: [
+ .init(id: "discord_token", label: "Discord token", kind: .secret, help: "Token for Discord API or desktop-cache assisted sync.", placeholder: "token", envVar: "DISCORD_TOKEN", configKey: "discord.token"),
+ .init(id: "openai_api_key", label: "OpenAI API key", kind: .secret, help: "Used when Discord Crawl generates embeddings.", placeholder: "sk-...", envVar: "OPENAI_API_KEY", configKey: "openai.api_key"),
+ .init(id: "embedding_model", label: "Embedding model", kind: .choice, defaultValue: "text-embedding-3-small", choices: ["text-embedding-3-small", "text-embedding-3-large"], envVar: "OPENAI_EMBEDDING_MODEL", configKey: "embeddings.model"),
+ ],
+ configSections: [
+ .init(id: "discord", title: "Discord Access", optionIDs: ["discord_token"]),
+ .init(id: "ai", title: "Embeddings", optionIDs: ["openai_api_key", "embedding_model"]),
+ ],
+ install: .init(method: .homebrew, package: "vincentkoc/tap/discrawl"))
+}
diff --git a/Sources/CrawlBarCore/BuiltInApps/BuiltInCrawlApps+Gitcrawl.swift b/Sources/CrawlBarCore/BuiltInApps/BuiltInCrawlApps+Gitcrawl.swift
new file mode 100644
index 0000000..73d4b31
--- /dev/null
+++ b/Sources/CrawlBarCore/BuiltInApps/BuiltInCrawlApps+Gitcrawl.swift
@@ -0,0 +1,43 @@
+import Foundation
+
+public extension BuiltInCrawlApps {
+ static let gitcrawl = CrawlAppManifest(
+ id: Self.gitcrawlID,
+ displayName: "GitHub",
+ description: "Local GitHub issue and pull request archive",
+ binary: .init(name: "gitcrawl"),
+ branding: .init(
+ symbolName: "point.3.connected.trianglepath.dotted",
+ accentColor: "#24292F",
+ bundleIdentifier: "com.github.GitHubClient"),
+ paths: .init(
+ defaultConfig: "~/.config/gitcrawl/config.toml",
+ configEnv: "GITCRAWL_CONFIG",
+ defaultDatabase: "~/.config/gitcrawl/gitcrawl.db",
+ defaultCache: "~/.config/gitcrawl/cache",
+ defaultLogs: "~/.config/gitcrawl/logs",
+ defaultShare: "~/.config/gitcrawl/share"),
+ commands: [
+ "metadata": ["metadata", "--json"],
+ "status": ["status", "--json"],
+ "doctor": ["doctor", "--json"],
+ "refresh": ["sync", "--json"],
+ "query": ["search", "--json"],
+ "remote-status": ["remote", "status", "--json"],
+ "remote-archives": ["remote", "archives", "--json"],
+ "cloud-publish": ["cloud", "publish", "--json"],
+ ],
+ capabilities: [.status, .doctor, .refresh, .search, .remoteArchive, .cloudPublish],
+ statusRequiresSecrets: false,
+ privacy: .init(exportsSecrets: false, localOnlyScopes: ["repositories", "issues", "pull requests"]),
+ configOptions: [
+ .init(id: "github_token", label: "GitHub token", kind: .secret, help: "Token used for GitHub API refreshes.", placeholder: "ghp_...", envVar: "GITHUB_TOKEN", configKey: "github.token"),
+ .init(id: "openai_api_key", label: "OpenAI API key", kind: .secret, help: "Used when Git Crawl generates embeddings.", placeholder: "sk-...", envVar: "OPENAI_API_KEY", configKey: "openai.api_key"),
+ .init(id: "embedding_model", label: "Embedding model", kind: .choice, help: "Model used for local semantic indexing.", defaultValue: "text-embedding-3-small", choices: ["text-embedding-3-small", "text-embedding-3-large"], envVar: "OPENAI_EMBEDDING_MODEL", configKey: "embeddings.model"),
+ ],
+ configSections: [
+ .init(id: "github", title: "GitHub Access", optionIDs: ["github_token"]),
+ .init(id: "ai", title: "Embeddings", optionIDs: ["openai_api_key", "embedding_model"]),
+ ],
+ install: .init(method: .homebrew, package: "vincentkoc/tap/gitcrawl"))
+}
diff --git a/Sources/CrawlBarCore/BuiltInApps/BuiltInCrawlApps+Gogcli.swift b/Sources/CrawlBarCore/BuiltInApps/BuiltInCrawlApps+Gogcli.swift
new file mode 100644
index 0000000..f8f6fd9
--- /dev/null
+++ b/Sources/CrawlBarCore/BuiltInApps/BuiltInCrawlApps+Gogcli.swift
@@ -0,0 +1,40 @@
+import Foundation
+
+public extension BuiltInCrawlApps {
+ static let gogcli = CrawlAppManifest(
+ id: Self.gogcliID,
+ displayName: "Google",
+ description: "Google Workspace and account automation",
+ binary: .init(name: "gog"),
+ execution: .init(
+ kind: .local,
+ kindConfigID: "execution_mode",
+ targetConfigID: "remote_target",
+ runAsConfigID: "remote_run_as",
+ remoteEnvFileConfigID: "remote_env_file",
+ remoteBinary: "gog"),
+ branding: .init(symbolName: "g.circle", accentColor: "#4285F4"),
+ paths: .init(
+ defaultConfig: "~/Library/Application Support/gogcli/config.json",
+ defaultCache: "~/Library/Application Support/gogcli",
+ defaultLogs: "~/Library/Logs/gogcli"),
+ commands: [
+ "status": ["auth", "list", "--check", "--json", "--no-input"],
+ "doctor": ["auth", "doctor", "--check", "--json", "--no-input"],
+ "search": ["--json", "--no-input", "search"],
+ ],
+ capabilities: [.status, .doctor, .search],
+ statusRequiresSecrets: false,
+ privacy: .init(exportsSecrets: false, localOnlyScopes: ["Google account config", "OAuth token metadata"]),
+ configOptions: [
+ .init(id: "execution_mode", label: "Run location", kind: .choice, help: "Run gog on this Mac or over SSH on another machine.", defaultValue: "local", choices: ["local", "remote"]),
+ .init(id: "remote_target", label: "SSH target", help: "SSH target that can run gog.", placeholder: "user@example-host"),
+ .init(id: "remote_run_as", label: "Run as user", help: "Optional remote Unix user for sudo -u, when gog is installed under a service account.", placeholder: "service-user"),
+ .init(id: "remote_env_file", label: "Remote env file", help: "Optional env file to source before running gog on the remote host.", placeholder: "/run/service/env"),
+ ],
+ configSections: [
+ .init(id: "execution", title: "Execution", optionIDs: ["execution_mode"]),
+ .init(id: "remote", title: "Remote Host", optionIDs: ["remote_target", "remote_run_as", "remote_env_file"]),
+ ],
+ install: .init(method: .homebrew, package: "openclaw/tap/gogcli"))
+}
diff --git a/Sources/CrawlBarCore/BuiltInApps/BuiltInCrawlApps+Graincrawl.swift b/Sources/CrawlBarCore/BuiltInApps/BuiltInCrawlApps+Graincrawl.swift
new file mode 100644
index 0000000..f861084
--- /dev/null
+++ b/Sources/CrawlBarCore/BuiltInApps/BuiltInCrawlApps+Graincrawl.swift
@@ -0,0 +1,45 @@
+import Foundation
+
+public extension BuiltInCrawlApps {
+ static let graincrawl = CrawlAppManifest(
+ id: Self.graincrawlID,
+ displayName: "Granola",
+ description: "Local-first archive for Granola notes, transcripts, summaries, and panels",
+ binary: .init(name: "graincrawl"),
+ branding: .init(
+ symbolName: "note.text",
+ accentColor: "#D4A017",
+ bundleIdentifier: "com.granola.app"),
+ paths: .init(
+ defaultConfig: "~/.config/graincrawl/config.toml",
+ configEnv: "GRAINCRAWL_CONFIG",
+ defaultDatabase: "~/.config/graincrawl/graincrawl.db",
+ defaultCache: "~/.config/graincrawl/cache",
+ defaultLogs: "~/.config/graincrawl/logs"),
+ commands: [
+ "metadata": ["metadata", "--json"],
+ "status": ["status", "--json"],
+ "doctor": ["doctor", "--json"],
+ "unlock": ["unlock", "--json"],
+ "refresh": ["sync", "--json"],
+ "desktop-cache-import": ["sync", "--source", "desktop-cache", "--json"],
+ "query": ["--json", "sql"],
+ "search": ["--json", "search"],
+ "export-md": ["export", "markdown", "--out", "./granola-notes"],
+ ],
+ capabilities: [.status, .doctor, .refresh, .search, .desktopCache, .exportMarkdown],
+ statusRequiresSecrets: false,
+ privacy: .init(containsPrivateMessages: true, exportsSecrets: false, localOnlyScopes: ["Granola profile", "graincrawl SQLite archive"]),
+ configOptions: [
+ .init(id: "granola_profile", label: "Granola profile", kind: .string, help: "Granola profile directory to inspect.", placeholder: "~/Library/Application Support/Granola", envVar: "GRAINCRAWL_GRANOLA_PROFILE", configKey: "granola.profile_path"),
+ .init(id: "preferred_source", label: "Preferred source", kind: .choice, help: "Source used by refresh.", defaultValue: "private-api", choices: ["private-api", "desktop-cache"], envVar: "GRAINCRAWL_SOURCE", configKey: "granola.preferred_source"),
+ .init(id: "allow_private_api", label: "Allow private API", kind: .boolean, defaultValue: "true", envVar: "GRAINCRAWL_ALLOW_PRIVATE_API", configKey: "granola.allow_private_api"),
+ .init(id: "allow_desktop_cache", label: "Allow desktop cache", kind: .boolean, defaultValue: "true", configKey: "granola.allow_desktop_cache"),
+ .init(id: "sync_limit", label: "Sync limit", kind: .number, help: "Maximum notes to import per sync run.", defaultValue: "100", configKey: "sync.default_limit"),
+ ],
+ configSections: [
+ .init(id: "granola", title: "Granola", optionIDs: ["granola_profile", "preferred_source", "allow_private_api", "allow_desktop_cache"]),
+ .init(id: "sync", title: "Sync", optionIDs: ["sync_limit"]),
+ ],
+ install: .init(method: .homebrew, package: "vincentkoc/tap/graincrawl"))
+}
diff --git a/Sources/CrawlBarCore/BuiltInApps/BuiltInCrawlApps+Notcrawl.swift b/Sources/CrawlBarCore/BuiltInApps/BuiltInCrawlApps+Notcrawl.swift
new file mode 100644
index 0000000..7b820d3
--- /dev/null
+++ b/Sources/CrawlBarCore/BuiltInApps/BuiltInCrawlApps+Notcrawl.swift
@@ -0,0 +1,45 @@
+import Foundation
+
+public extension BuiltInCrawlApps {
+ static let notcrawl = CrawlAppManifest(
+ id: Self.notcrawlID,
+ displayName: "Notion",
+ description: "Local Notion archive with Markdown and table exports",
+ binary: .init(name: "notcrawl"),
+ branding: .init(
+ symbolName: "doc.text.magnifyingglass",
+ accentColor: "#111111",
+ bundleIdentifier: "notion.id"),
+ paths: .init(
+ defaultConfig: "~/.notcrawl/config.toml",
+ configEnv: "NOTCRAWL_CONFIG",
+ defaultDatabase: "~/.notcrawl/notcrawl.db",
+ defaultCache: "~/.notcrawl/cache",
+ defaultLogs: "~/.notcrawl/logs",
+ defaultShare: "~/.notcrawl/share"),
+ commands: [
+ "metadata": ["metadata", "--json"],
+ "status": ["status", "--json"],
+ "doctor": ["doctor", "--json"],
+ "refresh": ["sync", "--source", "desktop"],
+ "desktop-cache-import": ["sync", "--source", "desktop"],
+ "query": ["sql"],
+ "search": ["search"],
+ "export-md": ["export-md"],
+ "publish": ["publish"],
+ "update": ["update"],
+ ],
+ capabilities: [.status, .doctor, .refresh, .search, .publish, .subscribe, .update, .desktopCache, .exportMarkdown, .exportDatabase, .maintain],
+ statusRequiresSecrets: false,
+ privacy: .init(containsPrivateMessages: true, exportsSecrets: false, localOnlyScopes: ["workspace pages", "comments", "exports"]),
+ configOptions: [
+ .init(id: "notion_token", label: "Notion token", kind: .secret, help: "Token or session credential for Notion sync.", placeholder: "secret_...", envVar: "NOTION_TOKEN", configKey: "notion.token"),
+ .init(id: "openai_api_key", label: "OpenAI API key", kind: .secret, help: "Used when Notion Crawl generates embeddings.", placeholder: "sk-...", envVar: "OPENAI_API_KEY", configKey: "openai.api_key"),
+ .init(id: "embedding_model", label: "Embedding model", kind: .choice, defaultValue: "text-embedding-3-small", choices: ["text-embedding-3-small", "text-embedding-3-large"], envVar: "OPENAI_EMBEDDING_MODEL", configKey: "embeddings.model"),
+ ],
+ configSections: [
+ .init(id: "notion", title: "Notion Access", optionIDs: ["notion_token"]),
+ .init(id: "ai", title: "Embeddings", optionIDs: ["openai_api_key", "embedding_model"]),
+ ],
+ install: .init(method: .homebrew, package: "vincentkoc/tap/notcrawl"))
+}
diff --git a/Sources/CrawlBarCore/BuiltInApps/BuiltInCrawlApps+Slacrawl.swift b/Sources/CrawlBarCore/BuiltInApps/BuiltInCrawlApps+Slacrawl.swift
new file mode 100644
index 0000000..80d4730
--- /dev/null
+++ b/Sources/CrawlBarCore/BuiltInApps/BuiltInCrawlApps+Slacrawl.swift
@@ -0,0 +1,44 @@
+import Foundation
+
+public extension BuiltInCrawlApps {
+ static let slacrawl = CrawlAppManifest(
+ id: Self.slacrawlID,
+ displayName: "Slack",
+ description: "Local-first Slack workspace archive",
+ binary: .init(name: "slacrawl"),
+ branding: .init(
+ symbolName: "bubble.left.and.bubble.right",
+ accentColor: "#4A154B",
+ bundleIdentifier: "com.tinyspeck.slackmacgap"),
+ paths: .init(
+ defaultConfig: "~/.slacrawl/config.toml",
+ configEnv: "SLACRAWL_CONFIG",
+ defaultDatabase: "~/.slacrawl/slacrawl.db",
+ defaultCache: "~/.slacrawl/cache",
+ defaultLogs: "~/.slacrawl/logs",
+ defaultShare: "~/.slacrawl/share"),
+ commands: [
+ "metadata": ["metadata", "--json"],
+ "status": ["status", "--json"],
+ "doctor": ["doctor", "--json"],
+ "refresh": ["--json", "sync", "--source", "desktop"],
+ "desktop-cache-import": ["--json", "sync", "--source", "desktop"],
+ "query": ["sql"],
+ "search": ["--json", "search"],
+ "publish": ["--json", "publish"],
+ "update": ["--json", "update"],
+ ],
+ capabilities: [.status, .doctor, .refresh, .search, .publish, .subscribe, .update, .desktopCache],
+ statusRequiresSecrets: false,
+ privacy: .init(containsPrivateMessages: true, exportsSecrets: false, localOnlyScopes: ["workspaces", "channels", "DMs"]),
+ configOptions: [
+ .init(id: "slack_token", label: "Slack token", kind: .secret, help: "User or bot token for Slack API sync.", placeholder: "xoxp- or xoxb-", envVar: "SLACK_TOKEN", configKey: "slack.token"),
+ .init(id: "openai_api_key", label: "OpenAI API key", kind: .secret, help: "Used when Slack Crawl generates embeddings.", placeholder: "sk-...", envVar: "OPENAI_API_KEY", configKey: "openai.api_key"),
+ .init(id: "embedding_model", label: "Embedding model", kind: .choice, defaultValue: "text-embedding-3-small", choices: ["text-embedding-3-small", "text-embedding-3-large"], envVar: "OPENAI_EMBEDDING_MODEL", configKey: "embeddings.model"),
+ ],
+ configSections: [
+ .init(id: "slack", title: "Slack Access", optionIDs: ["slack_token"]),
+ .init(id: "ai", title: "Embeddings", optionIDs: ["openai_api_key", "embedding_model"]),
+ ],
+ install: .init(method: .homebrew, package: "vincentkoc/tap/slacrawl"))
+}
diff --git a/Sources/CrawlBarCore/BuiltInApps/BuiltInCrawlApps+Telecrawl.swift b/Sources/CrawlBarCore/BuiltInApps/BuiltInCrawlApps+Telecrawl.swift
new file mode 100644
index 0000000..14f5471
--- /dev/null
+++ b/Sources/CrawlBarCore/BuiltInApps/BuiltInCrawlApps+Telecrawl.swift
@@ -0,0 +1,32 @@
+import Foundation
+
+public extension BuiltInCrawlApps {
+ static let telecrawl = CrawlAppManifest(
+ id: Self.telecrawlID,
+ displayName: "Telegram",
+ description: "Local-first Telegram Desktop archive crawler",
+ binary: .init(name: "telecrawl"),
+ branding: .init(
+ symbolName: "paperplane.fill",
+ accentColor: "#229ED9",
+ bundleIdentifier: "org.telegram.desktop"),
+ paths: .init(
+ defaultConfig: "~/.telecrawl/backup.json",
+ defaultDatabase: "~/.telecrawl/telecrawl.db",
+ defaultCache: "~/.telecrawl/cache",
+ defaultLogs: "~/.telecrawl/logs"),
+ commands: [
+ "metadata": ["metadata"],
+ "status": ["--json", "status"],
+ "doctor": ["--json", "doctor"],
+ "refresh": ["--json", "import"],
+ "search": ["--json", "search"],
+ ],
+ capabilities: [.status, .doctor, .refresh, .search],
+ statusRequiresSecrets: false,
+ privacy: .init(
+ containsPrivateMessages: true,
+ exportsSecrets: false,
+ localOnlyScopes: ["telegram-desktop", "sqlite", "encrypted-git-backup"]),
+ install: .init(method: .homebrew, package: "steipete/tap/telecrawl"))
+}
diff --git a/Sources/CrawlBarCore/BuiltInApps/BuiltInCrawlApps+Wacli.swift b/Sources/CrawlBarCore/BuiltInApps/BuiltInCrawlApps+Wacli.swift
new file mode 100644
index 0000000..2806086
--- /dev/null
+++ b/Sources/CrawlBarCore/BuiltInApps/BuiltInCrawlApps+Wacli.swift
@@ -0,0 +1,46 @@
+import Foundation
+
+public extension BuiltInCrawlApps {
+ static let wacli = CrawlAppManifest(
+ id: Self.wacliID,
+ displayName: "WhatsApp",
+ description: "WhatsApp linked-device message archive",
+ binary: .init(name: "wacli"),
+ execution: .init(
+ kind: .local,
+ kindConfigID: "execution_mode",
+ targetConfigID: "remote_target",
+ runAsConfigID: "remote_run_as",
+ remoteBinary: "wacli"),
+ branding: .init(
+ symbolName: "message.circle",
+ accentColor: "#25D366",
+ bundleIdentifier: "net.whatsapp.WhatsApp"),
+ paths: .init(
+ defaultConfig: "~/.wacli/config.yaml",
+ defaultDatabase: "~/.wacli/wacli.db",
+ defaultCache: "~/.wacli",
+ defaultLogs: "~/.wacli/logs",
+ defaultShare: "~/.wacli/share"),
+ commands: [
+ "status": ["--account", "{config:account}", "--read-only", "--json", "doctor"],
+ "doctor": ["--account", "{config:account}", "--read-only", "--json", "doctor"],
+ "refresh": ["--account", "{config:account}", "--json", "sync", "--once"],
+ "search": ["--account", "{config:account}", "--read-only", "--json", "messages", "search"],
+ ],
+ capabilities: [.status, .doctor, .refresh, .search],
+ statusRequiresSecrets: false,
+ privacy: .init(containsPrivateMessages: true, exportsSecrets: false, localOnlyScopes: ["WhatsApp chats", "contacts", "messages"]),
+ configOptions: [
+ .init(id: "execution_mode", label: "Run location", kind: .choice, help: "Run wacli on this Mac or over SSH on another machine.", defaultValue: "local", choices: ["local", "remote"]),
+ .init(id: "remote_target", label: "SSH target", help: "SSH target that can run wacli, for example user@example-host.", placeholder: "user@example-host"),
+ .init(id: "remote_run_as", label: "Run as user", help: "Optional remote Unix user for sudo -u, when wacli is installed under a service account.", placeholder: "crawl"),
+ .init(id: "account", label: "wacli account", help: "Optional named account from the wacli config.", placeholder: "personal"),
+ ],
+ configSections: [
+ .init(id: "execution", title: "Execution", optionIDs: ["execution_mode"]),
+ .init(id: "remote", title: "Remote Host", optionIDs: ["remote_target", "remote_run_as"]),
+ .init(id: "whatsapp", title: "WhatsApp Account", optionIDs: ["account"]),
+ ],
+ install: .init(method: .homebrew, package: "openclaw/tap/wacli"))
+}
diff --git a/Sources/CrawlBarCore/BuiltInApps/BuiltInCrawlApps.swift b/Sources/CrawlBarCore/BuiltInApps/BuiltInCrawlApps.swift
new file mode 100644
index 0000000..1b8144c
--- /dev/null
+++ b/Sources/CrawlBarCore/BuiltInApps/BuiltInCrawlApps.swift
@@ -0,0 +1,32 @@
+import Foundation
+
+public enum BuiltInCrawlApps {
+ public static let gitcrawlID = CrawlAppID(rawValue: "gitcrawl")
+ public static let slacrawlID = CrawlAppID(rawValue: "slacrawl")
+ public static let discrawlID = CrawlAppID(rawValue: "discrawl")
+ public static let telecrawlID = CrawlAppID(rawValue: "telecrawl")
+ public static let notcrawlID = CrawlAppID(rawValue: "notcrawl")
+ public static let gogcliID = CrawlAppID(rawValue: "gogcli")
+ public static let wacliID = CrawlAppID(rawValue: "wacli")
+ public static let birdclawID = CrawlAppID(rawValue: "birdclaw")
+ public static let graincrawlID = CrawlAppID(rawValue: "graincrawl")
+
+ public static let all: [CrawlAppManifest] = [
+ Self.gitcrawl,
+ Self.slacrawl,
+ Self.discrawl,
+ Self.telecrawl,
+ Self.notcrawl,
+ Self.gogcli,
+ Self.wacli,
+ Self.birdclaw,
+ Self.graincrawl,
+ ]
+
+ public static let allByID = Dictionary(uniqueKeysWithValues: Self.all.map { ($0.id, $0) })
+
+ public static func manifest(for id: CrawlAppID) -> CrawlAppManifest? {
+ self.allByID[id]
+ }
+
+}
diff --git a/Sources/CrawlBarCore/BuiltInCrawlApps.swift b/Sources/CrawlBarCore/BuiltInCrawlApps.swift
deleted file mode 100644
index 6566884..0000000
--- a/Sources/CrawlBarCore/BuiltInCrawlApps.swift
+++ /dev/null
@@ -1,382 +0,0 @@
-import Foundation
-
-public enum BuiltInCrawlApps {
- public static let gitcrawlID = CrawlAppID(rawValue: "gitcrawl")
- public static let slacrawlID = CrawlAppID(rawValue: "slacrawl")
- public static let discrawlID = CrawlAppID(rawValue: "discrawl")
- public static let telecrawlID = CrawlAppID(rawValue: "telecrawl")
- public static let notcrawlID = CrawlAppID(rawValue: "notcrawl")
- public static let gogcliID = CrawlAppID(rawValue: "gogcli")
- public static let wacliID = CrawlAppID(rawValue: "wacli")
- public static let birdclawID = CrawlAppID(rawValue: "birdclaw")
- public static let graincrawlID = CrawlAppID(rawValue: "graincrawl")
-
- public static let all: [CrawlAppManifest] = [
- Self.gitcrawl,
- Self.slacrawl,
- Self.discrawl,
- Self.telecrawl,
- Self.notcrawl,
- Self.gogcli,
- Self.wacli,
- Self.birdclaw,
- Self.graincrawl,
- ]
-
- public static let allByID = Dictionary(uniqueKeysWithValues: Self.all.map { ($0.id, $0) })
-
- public static func manifest(for id: CrawlAppID) -> CrawlAppManifest? {
- self.allByID[id]
- }
-
- public static let gitcrawl = CrawlAppManifest(
- id: Self.gitcrawlID,
- displayName: "GitHub",
- description: "Local GitHub issue and pull request archive",
- binary: .init(name: "gitcrawl"),
- branding: .init(
- symbolName: "point.3.connected.trianglepath.dotted",
- accentColor: "#24292F",
- bundleIdentifier: "com.github.GitHubClient"),
- paths: .init(
- defaultConfig: "~/.config/gitcrawl/config.toml",
- configEnv: "GITCRAWL_CONFIG",
- defaultDatabase: "~/.config/gitcrawl/gitcrawl.db",
- defaultCache: "~/.config/gitcrawl/cache",
- defaultLogs: "~/.config/gitcrawl/logs",
- defaultShare: "~/.config/gitcrawl/share"),
- commands: [
- "metadata": ["metadata", "--json"],
- "status": ["status", "--json"],
- "doctor": ["doctor", "--json"],
- "refresh": ["sync", "--json"],
- "query": ["search", "--json"],
- "remote-status": ["remote", "status", "--json"],
- "remote-archives": ["remote", "archives", "--json"],
- "cloud-publish": ["cloud", "publish", "--json"],
- ],
- capabilities: [.status, .doctor, .refresh, .search, .remoteArchive, .cloudPublish],
- statusRequiresSecrets: false,
- privacy: .init(exportsSecrets: false, localOnlyScopes: ["repositories", "issues", "pull requests"]),
- configOptions: [
- .init(id: "github_token", label: "GitHub token", kind: .secret, help: "Token used for GitHub API refreshes.", placeholder: "ghp_...", envVar: "GITHUB_TOKEN", configKey: "github.token"),
- .init(id: "openai_api_key", label: "OpenAI API key", kind: .secret, help: "Used when Git Crawl generates embeddings.", placeholder: "sk-...", envVar: "OPENAI_API_KEY", configKey: "openai.api_key"),
- .init(id: "embedding_model", label: "Embedding model", kind: .choice, help: "Model used for local semantic indexing.", defaultValue: "text-embedding-3-small", choices: ["text-embedding-3-small", "text-embedding-3-large"], envVar: "OPENAI_EMBEDDING_MODEL", configKey: "embeddings.model"),
- ],
- configSections: [
- .init(id: "github", title: "GitHub Access", optionIDs: ["github_token"]),
- .init(id: "ai", title: "Embeddings", optionIDs: ["openai_api_key", "embedding_model"]),
- ],
- install: .init(method: .homebrew, package: "vincentkoc/tap/gitcrawl"))
-
- public static let slacrawl = CrawlAppManifest(
- id: Self.slacrawlID,
- displayName: "Slack",
- description: "Local-first Slack workspace archive",
- binary: .init(name: "slacrawl"),
- branding: .init(
- symbolName: "bubble.left.and.bubble.right",
- accentColor: "#4A154B",
- bundleIdentifier: "com.tinyspeck.slackmacgap"),
- paths: .init(
- defaultConfig: "~/.slacrawl/config.toml",
- configEnv: "SLACRAWL_CONFIG",
- defaultDatabase: "~/.slacrawl/slacrawl.db",
- defaultCache: "~/.slacrawl/cache",
- defaultLogs: "~/.slacrawl/logs",
- defaultShare: "~/.slacrawl/share"),
- commands: [
- "metadata": ["metadata", "--json"],
- "status": ["status", "--json"],
- "doctor": ["doctor", "--json"],
- "refresh": ["--json", "sync", "--source", "desktop"],
- "desktop-cache-import": ["--json", "sync", "--source", "desktop"],
- "query": ["sql"],
- "search": ["--json", "search"],
- "publish": ["--json", "publish"],
- "update": ["--json", "update"],
- ],
- capabilities: [.status, .doctor, .refresh, .search, .publish, .subscribe, .update, .desktopCache],
- statusRequiresSecrets: false,
- privacy: .init(containsPrivateMessages: true, exportsSecrets: false, localOnlyScopes: ["workspaces", "channels", "DMs"]),
- configOptions: [
- .init(id: "slack_token", label: "Slack token", kind: .secret, help: "User or bot token for Slack API sync.", placeholder: "xoxp- or xoxb-", envVar: "SLACK_TOKEN", configKey: "slack.token"),
- .init(id: "openai_api_key", label: "OpenAI API key", kind: .secret, help: "Used when Slack Crawl generates embeddings.", placeholder: "sk-...", envVar: "OPENAI_API_KEY", configKey: "openai.api_key"),
- .init(id: "embedding_model", label: "Embedding model", kind: .choice, defaultValue: "text-embedding-3-small", choices: ["text-embedding-3-small", "text-embedding-3-large"], envVar: "OPENAI_EMBEDDING_MODEL", configKey: "embeddings.model"),
- ],
- configSections: [
- .init(id: "slack", title: "Slack Access", optionIDs: ["slack_token"]),
- .init(id: "ai", title: "Embeddings", optionIDs: ["openai_api_key", "embedding_model"]),
- ],
- install: .init(method: .homebrew, package: "vincentkoc/tap/slacrawl"))
-
- public static let discrawl = CrawlAppManifest(
- id: Self.discrawlID,
- displayName: "Discord",
- description: "Local Discord guild and desktop-cache archive",
- binary: .init(name: "discrawl"),
- branding: .init(
- symbolName: "antenna.radiowaves.left.and.right",
- accentColor: "#5865F2",
- bundleIdentifier: "com.hnc.Discord"),
- paths: .init(
- defaultConfig: "~/.discrawl/config.toml",
- configEnv: "DISCRAWL_CONFIG",
- defaultDatabase: "~/.discrawl/discrawl.db",
- defaultCache: "~/.discrawl/cache",
- defaultLogs: "~/.discrawl/logs",
- defaultShare: "~/.discrawl/share"),
- commands: [
- "metadata": ["metadata", "--json"],
- "status": ["status", "--json"],
- "doctor": ["doctor", "--json"],
- "refresh": ["--json", "cache-import"],
- "desktop-cache-import": ["--json", "cache-import"],
- "publish": ["--json", "publish"],
- "update": ["--json", "update"],
- "remote-status": ["remote", "status", "--json"],
- "remote-archives": ["remote", "archives", "--json"],
- "cloud-publish": ["cloud", "publish", "--sqlite-only", "--json"],
- ],
- capabilities: [.status, .doctor, .refresh, .publish, .subscribe, .update, .desktopCache, .remoteArchive, .cloudPublish],
- statusRequiresSecrets: false,
- privacy: .init(containsPrivateMessages: true, exportsSecrets: false, localOnlyScopes: ["@me"]),
- configOptions: [
- .init(id: "discord_token", label: "Discord token", kind: .secret, help: "Token for Discord API or desktop-cache assisted sync.", placeholder: "token", envVar: "DISCORD_TOKEN", configKey: "discord.token"),
- .init(id: "openai_api_key", label: "OpenAI API key", kind: .secret, help: "Used when Discord Crawl generates embeddings.", placeholder: "sk-...", envVar: "OPENAI_API_KEY", configKey: "openai.api_key"),
- .init(id: "embedding_model", label: "Embedding model", kind: .choice, defaultValue: "text-embedding-3-small", choices: ["text-embedding-3-small", "text-embedding-3-large"], envVar: "OPENAI_EMBEDDING_MODEL", configKey: "embeddings.model"),
- ],
- configSections: [
- .init(id: "discord", title: "Discord Access", optionIDs: ["discord_token"]),
- .init(id: "ai", title: "Embeddings", optionIDs: ["openai_api_key", "embedding_model"]),
- ],
- install: .init(method: .homebrew, package: "vincentkoc/tap/discrawl"))
-
- public static let telecrawl = CrawlAppManifest(
- id: Self.telecrawlID,
- displayName: "Telegram",
- description: "Local-first Telegram Desktop archive crawler",
- binary: .init(name: "telecrawl"),
- branding: .init(
- symbolName: "paperplane.fill",
- accentColor: "#229ED9",
- bundleIdentifier: "org.telegram.desktop"),
- paths: .init(
- defaultConfig: "~/.telecrawl/backup.json",
- defaultDatabase: "~/.telecrawl/telecrawl.db",
- defaultCache: "~/.telecrawl/cache",
- defaultLogs: "~/.telecrawl/logs"),
- commands: [
- "metadata": ["metadata"],
- "status": ["--json", "status"],
- "doctor": ["--json", "doctor"],
- "refresh": ["--json", "import"],
- "search": ["--json", "search"],
- ],
- capabilities: [.status, .doctor, .refresh, .search],
- statusRequiresSecrets: false,
- privacy: .init(
- containsPrivateMessages: true,
- exportsSecrets: false,
- localOnlyScopes: ["telegram-desktop", "sqlite", "encrypted-git-backup"]),
- install: .init(method: .homebrew, package: "steipete/tap/telecrawl"))
-
- public static let notcrawl = CrawlAppManifest(
- id: Self.notcrawlID,
- displayName: "Notion",
- description: "Local Notion archive with Markdown and table exports",
- binary: .init(name: "notcrawl"),
- branding: .init(
- symbolName: "doc.text.magnifyingglass",
- accentColor: "#111111",
- bundleIdentifier: "notion.id"),
- paths: .init(
- defaultConfig: "~/.notcrawl/config.toml",
- configEnv: "NOTCRAWL_CONFIG",
- defaultDatabase: "~/.notcrawl/notcrawl.db",
- defaultCache: "~/.notcrawl/cache",
- defaultLogs: "~/.notcrawl/logs",
- defaultShare: "~/.notcrawl/share"),
- commands: [
- "metadata": ["metadata", "--json"],
- "status": ["status", "--json"],
- "doctor": ["doctor", "--json"],
- "refresh": ["sync", "--source", "desktop"],
- "desktop-cache-import": ["sync", "--source", "desktop"],
- "query": ["sql"],
- "search": ["search"],
- "export-md": ["export-md"],
- "publish": ["publish"],
- "update": ["update"],
- ],
- capabilities: [.status, .doctor, .refresh, .search, .publish, .subscribe, .update, .desktopCache, .exportMarkdown, .exportDatabase, .maintain],
- statusRequiresSecrets: false,
- privacy: .init(containsPrivateMessages: true, exportsSecrets: false, localOnlyScopes: ["workspace pages", "comments", "exports"]),
- configOptions: [
- .init(id: "notion_token", label: "Notion token", kind: .secret, help: "Token or session credential for Notion sync.", placeholder: "secret_...", envVar: "NOTION_TOKEN", configKey: "notion.token"),
- .init(id: "openai_api_key", label: "OpenAI API key", kind: .secret, help: "Used when Notion Crawl generates embeddings.", placeholder: "sk-...", envVar: "OPENAI_API_KEY", configKey: "openai.api_key"),
- .init(id: "embedding_model", label: "Embedding model", kind: .choice, defaultValue: "text-embedding-3-small", choices: ["text-embedding-3-small", "text-embedding-3-large"], envVar: "OPENAI_EMBEDDING_MODEL", configKey: "embeddings.model"),
- ],
- configSections: [
- .init(id: "notion", title: "Notion Access", optionIDs: ["notion_token"]),
- .init(id: "ai", title: "Embeddings", optionIDs: ["openai_api_key", "embedding_model"]),
- ],
- install: .init(method: .homebrew, package: "vincentkoc/tap/notcrawl"))
-
- public static let gogcli = CrawlAppManifest(
- id: Self.gogcliID,
- displayName: "Google",
- description: "Google Workspace and account automation",
- binary: .init(name: "gog"),
- execution: .init(
- kind: .local,
- kindConfigID: "execution_mode",
- targetConfigID: "remote_target",
- runAsConfigID: "remote_run_as",
- remoteEnvFileConfigID: "remote_env_file",
- remoteBinary: "gog"),
- branding: .init(symbolName: "g.circle", accentColor: "#4285F4"),
- paths: .init(
- defaultConfig: "~/Library/Application Support/gogcli/config.json",
- defaultCache: "~/Library/Application Support/gogcli",
- defaultLogs: "~/Library/Logs/gogcli"),
- commands: [
- "status": ["auth", "list", "--check", "--json", "--no-input"],
- "doctor": ["auth", "doctor", "--check", "--json", "--no-input"],
- "search": ["--json", "--no-input", "search"],
- ],
- capabilities: [.status, .doctor, .search],
- statusRequiresSecrets: false,
- privacy: .init(exportsSecrets: false, localOnlyScopes: ["Google account config", "OAuth token metadata"]),
- configOptions: [
- .init(id: "execution_mode", label: "Run location", kind: .choice, help: "Run gog on this Mac or over SSH on another machine.", defaultValue: "local", choices: ["local", "remote"]),
- .init(id: "remote_target", label: "SSH target", help: "SSH target that can run gog.", placeholder: "user@example-host"),
- .init(id: "remote_run_as", label: "Run as user", help: "Optional remote Unix user for sudo -u, when gog is installed under a service account.", placeholder: "service-user"),
- .init(id: "remote_env_file", label: "Remote env file", help: "Optional env file to source before running gog on the remote host.", placeholder: "/run/service/env"),
- ],
- configSections: [
- .init(id: "execution", title: "Execution", optionIDs: ["execution_mode"]),
- .init(id: "remote", title: "Remote Host", optionIDs: ["remote_target", "remote_run_as", "remote_env_file"]),
- ],
- install: .init(method: .homebrew, package: "openclaw/tap/gogcli"))
-
- public static let wacli = CrawlAppManifest(
- id: Self.wacliID,
- displayName: "WhatsApp",
- description: "WhatsApp linked-device message archive",
- binary: .init(name: "wacli"),
- execution: .init(
- kind: .local,
- kindConfigID: "execution_mode",
- targetConfigID: "remote_target",
- runAsConfigID: "remote_run_as",
- remoteBinary: "wacli"),
- branding: .init(
- symbolName: "message.circle",
- accentColor: "#25D366",
- bundleIdentifier: "net.whatsapp.WhatsApp"),
- paths: .init(
- defaultConfig: "~/.wacli/config.yaml",
- defaultDatabase: "~/.wacli/wacli.db",
- defaultCache: "~/.wacli",
- defaultLogs: "~/.wacli/logs",
- defaultShare: "~/.wacli/share"),
- commands: [
- "status": ["--account", "{config:account}", "--read-only", "--json", "doctor"],
- "doctor": ["--account", "{config:account}", "--read-only", "--json", "doctor"],
- "refresh": ["--account", "{config:account}", "--json", "sync", "--once"],
- "search": ["--account", "{config:account}", "--read-only", "--json", "messages", "search"],
- ],
- capabilities: [.status, .doctor, .refresh, .search],
- statusRequiresSecrets: false,
- privacy: .init(containsPrivateMessages: true, exportsSecrets: false, localOnlyScopes: ["WhatsApp chats", "contacts", "messages"]),
- configOptions: [
- .init(id: "execution_mode", label: "Run location", kind: .choice, help: "Run wacli on this Mac or over SSH on another machine.", defaultValue: "local", choices: ["local", "remote"]),
- .init(id: "remote_target", label: "SSH target", help: "SSH target that can run wacli, for example user@example-host.", placeholder: "user@example-host"),
- .init(id: "remote_run_as", label: "Run as user", help: "Optional remote Unix user for sudo -u, when wacli is installed under a service account.", placeholder: "crawl"),
- .init(id: "account", label: "wacli account", help: "Optional named account from the wacli config.", placeholder: "personal"),
- ],
- configSections: [
- .init(id: "execution", title: "Execution", optionIDs: ["execution_mode"]),
- .init(id: "remote", title: "Remote Host", optionIDs: ["remote_target", "remote_run_as"]),
- .init(id: "whatsapp", title: "WhatsApp Account", optionIDs: ["account"]),
- ],
- install: .init(method: .homebrew, package: "openclaw/tap/wacli"))
-
- public static let birdclaw = CrawlAppManifest(
- id: Self.birdclawID,
- displayName: "X",
- description: "X/Twitter connector through bird, with Birdclaw workspace support",
- binary: .init(name: "bird"),
- execution: .init(
- kind: .local,
- kindConfigID: "execution_mode",
- targetConfigID: "remote_target",
- runAsConfigID: "remote_run_as",
- remoteBinary: "bird"),
- branding: .init(symbolName: "xmark", accentColor: "#111111"),
- paths: .init(
- defaultConfig: "~/.birdclaw/config.json",
- defaultDatabase: "~/.birdclaw/birdclaw.sqlite",
- defaultCache: "~/.birdclaw/media"),
- commands: [
- "status": ["check", "--plain"],
- "doctor": ["check", "--plain"],
- "search": ["search", "-n", "10", "--json"],
- ],
- capabilities: [.status, .doctor, .search],
- privacy: .init(containsPrivateMessages: true, exportsSecrets: false, localOnlyScopes: ["X archive", "DMs", "browser cookies"]),
- configOptions: [
- .init(id: "access_path", label: "Access path", kind: .choice, help: "Use bird first, or use Birdclaw when that host is authenticated through xurl.", defaultValue: "bird", choices: ["bird", "birdclaw"]),
- .init(id: "execution_mode", label: "Run location", kind: .choice, help: "Run bird on this Mac or over SSH on another machine.", defaultValue: "local", choices: ["local", "remote"]),
- .init(id: "remote_target", label: "SSH target", help: "SSH target that can run bird.", placeholder: "user@example-host"),
- .init(id: "remote_run_as", label: "Run as user", help: "Optional remote Unix user for sudo -u, when bird is installed under a service account.", placeholder: "crawl"),
- ],
- configSections: [
- .init(id: "execution", title: "Execution", optionIDs: ["access_path", "execution_mode"]),
- .init(id: "remote", title: "Remote Host", optionIDs: ["remote_target", "remote_run_as"]),
- ],
- install: .init(method: .homebrew, package: "steipete/tap/bird"))
-
- public static let graincrawl = CrawlAppManifest(
- id: Self.graincrawlID,
- displayName: "Granola",
- description: "Local-first archive for Granola notes, transcripts, summaries, and panels",
- binary: .init(name: "graincrawl"),
- branding: .init(
- symbolName: "note.text",
- accentColor: "#D4A017",
- bundleIdentifier: "com.granola.app"),
- paths: .init(
- defaultConfig: "~/.config/graincrawl/config.toml",
- configEnv: "GRAINCRAWL_CONFIG",
- defaultDatabase: "~/.config/graincrawl/graincrawl.db",
- defaultCache: "~/.config/graincrawl/cache",
- defaultLogs: "~/.config/graincrawl/logs"),
- commands: [
- "metadata": ["metadata", "--json"],
- "status": ["status", "--json"],
- "doctor": ["doctor", "--json"],
- "unlock": ["unlock", "--json"],
- "refresh": ["sync", "--json"],
- "desktop-cache-import": ["sync", "--source", "desktop-cache", "--json"],
- "query": ["--json", "sql"],
- "search": ["--json", "search"],
- "export-md": ["export", "markdown", "--out", "./granola-notes"],
- ],
- capabilities: [.status, .doctor, .refresh, .search, .desktopCache, .exportMarkdown],
- statusRequiresSecrets: false,
- privacy: .init(containsPrivateMessages: true, exportsSecrets: false, localOnlyScopes: ["Granola profile", "graincrawl SQLite archive"]),
- configOptions: [
- .init(id: "granola_profile", label: "Granola profile", kind: .string, help: "Granola profile directory to inspect.", placeholder: "~/Library/Application Support/Granola", envVar: "GRAINCRAWL_GRANOLA_PROFILE", configKey: "granola.profile_path"),
- .init(id: "preferred_source", label: "Preferred source", kind: .choice, help: "Source used by refresh.", defaultValue: "private-api", choices: ["private-api", "desktop-cache"], envVar: "GRAINCRAWL_SOURCE", configKey: "granola.preferred_source"),
- .init(id: "allow_private_api", label: "Allow private API", kind: .boolean, defaultValue: "true", envVar: "GRAINCRAWL_ALLOW_PRIVATE_API", configKey: "granola.allow_private_api"),
- .init(id: "allow_desktop_cache", label: "Allow desktop cache", kind: .boolean, defaultValue: "true", configKey: "granola.allow_desktop_cache"),
- .init(id: "sync_limit", label: "Sync limit", kind: .number, help: "Maximum notes to import per sync run.", defaultValue: "100", configKey: "sync.default_limit"),
- ],
- configSections: [
- .init(id: "granola", title: "Granola", optionIDs: ["granola_profile", "preferred_source", "allow_private_api", "allow_desktop_cache"]),
- .init(id: "sync", title: "Sync", optionIDs: ["sync_limit"]),
- ],
- install: .init(method: .homebrew, package: "vincentkoc/tap/graincrawl"))
-}
diff --git a/Sources/CrawlBarCore/CommandRunner.swift b/Sources/CrawlBarCore/CommandRunner.swift
index 3047569..debe46b 100644
--- a/Sources/CrawlBarCore/CommandRunner.swift
+++ b/Sources/CrawlBarCore/CommandRunner.swift
@@ -1,126 +1,12 @@
import Foundation
-#if os(macOS)
-import Darwin
-#elseif os(Linux)
-import Glibc
-#endif
-
-public enum CrawlCommandRunnerError: LocalizedError, Sendable {
- case executableNotFound(String)
- case commandUnavailable(appID: CrawlAppID, action: String)
- case missingRequiredConfig(appID: CrawlAppID, optionID: String)
- case invalidRemoteTarget(appID: CrawlAppID, target: String)
- case timedOut(appID: CrawlAppID, action: String, seconds: Int)
-
- public var errorDescription: String? {
- switch self {
- case let .executableNotFound(name):
- "Could not find executable: \(name)"
- case let .commandUnavailable(appID, action):
- "\(appID.rawValue) does not expose a \(action) command"
- case let .missingRequiredConfig(appID, optionID):
- "\(appID.rawValue) is missing required config: \(optionID)"
- case let .invalidRemoteTarget(appID, target):
- "\(appID.rawValue) has an invalid SSH target: \(target)"
- case let .timedOut(appID, action, seconds):
- "\(appID.rawValue) \(action) timed out after \(seconds)s"
- }
- }
-}
-
-public struct CrawlCommandRedactor: Sendable {
- public init() {}
-
- public func redact(_ text: String) -> String {
- var redacted = text
- let patterns: [(String, String)] = [
- (#"(?i)(Bearer[ \t]+)[^ \t\r\n"',}]+"#, "$1[REDACTED]"),
- (#"(?i)(api[_-]?key|token|secret|password|cookie|authorization)(["' \t:=]+)([^ \t\r\n"',}]+)"#, "$1$2[REDACTED]"),
- (#"(?i)\b(github_pat_)[A-Za-z0-9_]+\b"#, "$1[REDACTED]"),
- (#"(?i)\b(gh[pousr]_)[A-Za-z0-9_]+\b"#, "$1[REDACTED]"),
- (#"(?i)\b(sk-[A-Za-z0-9_-]{16,})\b"#, "[REDACTED]"),
- (#"(?i)\b(secret_)[A-Za-z0-9_]+\b"#, "$1[REDACTED]"),
- (#"(?i)(xox[aboprsxc]-)[A-Za-z0-9-]+"#, "$1[REDACTED]"),
- (#"(?i)\bmfa\.[A-Za-z0-9_-]+\b"#, "[REDACTED]"),
- (#"(?i)\b(ct0)(["' \t:=]+)([^ \t\r\n"',}]+)"#, "$1$2[REDACTED]"),
- (#"\b[A-Za-z0-9_-]{24}\.[A-Za-z0-9_-]{6}\.[A-Za-z0-9_-]{20,}\b"#, "[REDACTED]"),
- (#"(?i)(discord[_-]?token["' \t:=]+)([^ \t\r\n"',}]+)"#, "$1[REDACTED]"),
- ]
- for (pattern, template) in patterns {
- redacted = redacted.replacingOccurrences(
- of: pattern,
- with: template,
- options: [.regularExpression])
- }
- return redacted
- }
-}
-
-public final class CrawlExecutableResolver: @unchecked Sendable {
- private let fileManager: FileManager
- private let environment: [String: String]
- private let lock = NSLock()
- private var resolvedExecutables: [String: String] = [:]
-
- public init(fileManager: FileManager = .default, environment: [String: String] = ProcessInfo.processInfo.environment) {
- self.fileManager = fileManager
- self.environment = CrawlProcessEnvironment.normalized(environment)
- }
-
- public func resolve(_ requestedPathOrName: String) -> String? {
- self.lock.lock()
- if let cached = self.resolvedExecutables[requestedPathOrName] {
- self.lock.unlock()
- if self.isExecutable(cached) {
- return cached
- }
- self.lock.lock()
- self.resolvedExecutables.removeValue(forKey: requestedPathOrName)
- self.lock.unlock()
- } else {
- self.lock.unlock()
- }
-
- let resolved = self.resolveUncached(requestedPathOrName)
- self.lock.lock()
- if let resolved {
- self.resolvedExecutables[requestedPathOrName] = resolved
- } else {
- self.resolvedExecutables.removeValue(forKey: requestedPathOrName)
- }
- self.lock.unlock()
- return resolved
- }
-
- private func resolveUncached(_ requestedPathOrName: String) -> String? {
- let expanded = PathExpander.expandHome(requestedPathOrName)
- if expanded.contains("/") {
- return self.isExecutable(expanded) ? expanded : nil
- }
-
- for entry in CrawlProcessEnvironment.pathEntries(environment: self.environment) {
- let candidate = URL(fileURLWithPath: entry)
- .appendingPathComponent(expanded)
- .path
- if self.isExecutable(candidate) {
- return candidate
- }
- }
- return nil
- }
-
- private func isExecutable(_ path: String) -> Bool {
- self.fileManager.isExecutableFile(atPath: path)
- }
-}
public struct CrawlCommandRunner: @unchecked Sendable {
- private static let timeoutTerminationGrace: DispatchTimeInterval = .milliseconds(500)
+ static let timeoutTerminationGrace: DispatchTimeInterval = .milliseconds(500)
- private let resolver: CrawlExecutableResolver
- private let redactor: CrawlCommandRedactor
- private let fileManager: FileManager
- private let environment: [String: String]
+ let resolver: CrawlExecutableResolver
+ let redactor: CrawlCommandRedactor
+ let fileManager: FileManager
+ let environment: [String: String]
public init(
resolver: CrawlExecutableResolver = CrawlExecutableResolver(),
@@ -198,273 +84,4 @@ public struct CrawlCommandRunner: @unchecked Sendable {
environment: commandEnvironment,
timeoutSeconds: timeoutSeconds)
}
-
- private func runProcess(
- appID: CrawlAppID,
- action: String,
- executablePath: String,
- arguments: [String],
- environment: [String: String],
- timeoutSeconds: TimeInterval)
- throws -> CrawlCommandResult
- {
- let startedAt = Date()
- let tempDirectory = self.fileManager.temporaryDirectory
- .appendingPathComponent("crawlbar-\(UUID().uuidString)", isDirectory: true)
- try self.fileManager.createDirectory(at: tempDirectory, withIntermediateDirectories: true)
- defer { try? self.fileManager.removeItem(at: tempDirectory) }
-
- let stdoutURL = tempDirectory.appendingPathComponent("stdout.log")
- let stderrURL = tempDirectory.appendingPathComponent("stderr.log")
- self.fileManager.createFile(atPath: stdoutURL.path, contents: nil)
- self.fileManager.createFile(atPath: stderrURL.path, contents: nil)
-
- let stdoutHandle = try FileHandle(forWritingTo: stdoutURL)
- let stderrHandle = try FileHandle(forWritingTo: stderrURL)
- defer {
- try? stdoutHandle.close()
- try? stderrHandle.close()
- }
-
- let process = Process()
- process.executableURL = URL(fileURLWithPath: executablePath)
- process.arguments = arguments
- process.environment = environment
- process.standardOutput = stdoutHandle
- process.standardError = stderrHandle
-
- let semaphore = DispatchSemaphore(value: 0)
- process.terminationHandler = { _ in
- semaphore.signal()
- }
-
- try process.run()
- let waitResult = semaphore.wait(timeout: .now() + timeoutSeconds)
- if waitResult == .timedOut {
- process.terminate()
- #if os(macOS) || os(Linux)
- let pid = process.processIdentifier
- DispatchQueue.global().asyncAfter(deadline: .now() + Self.timeoutTerminationGrace) {
- if process.isRunning {
- kill(pid, SIGKILL)
- }
- }
- #endif
- process.waitUntilExit()
- throw CrawlCommandRunnerError.timedOut(
- appID: appID,
- action: action,
- seconds: Int(timeoutSeconds))
- }
-
- try? stdoutHandle.synchronize()
- try? stderrHandle.synchronize()
-
- let stdout = try String(contentsOf: stdoutURL, encoding: .utf8)
- let stderr = try String(contentsOf: stderrURL, encoding: .utf8)
- return CrawlCommandResult(
- appID: appID,
- action: action,
- exitCode: process.terminationStatus,
- stdout: self.redactor.redact(stdout),
- stderr: self.redactor.redact(stderr),
- startedAt: startedAt,
- finishedAt: Date())
- }
-
- private static func commandArguments(
- for installation: CrawlAppInstallation,
- action: String,
- commandArguments: [String],
- extraArguments: [String])
- -> [String]
- {
- if Self.wacliSearchNeedsJoinedQuery(installation: installation, action: action, extraArguments: extraArguments) {
- return commandArguments + Self.wacliSearchArguments(extraArguments)
- }
-
- guard installation.id == BuiltInCrawlApps.gitcrawlID,
- let repository = GitcrawlStatusSnapshot.repository(for: installation)
- else {
- return commandArguments + extraArguments
- }
-
- if Self.gitcrawlQueryNeedsRepository(action: action, extraArguments: extraArguments) {
- return commandArguments + [repository, "--query", extraArguments.joined(separator: " ")]
- }
-
- if Self.gitcrawlRefreshNeedsRepository(action: action, commandArguments: commandArguments) {
- return commandArguments + [repository] + extraArguments
- }
-
- return commandArguments + extraArguments
- }
-
- private static func interpolatedArguments(_ arguments: [String], installation: CrawlAppInstallation) throws -> [String] {
- var result: [String] = []
- for argument in arguments {
- if let optionID = Self.exactConfigTokenID(argument),
- Self.configValue(optionID, installation: installation) == nil,
- result.last == "--account"
- {
- result.removeLast()
- continue
- }
- result.append(try Self.interpolatedArgument(argument, installation: installation))
- }
- return result
- }
-
- private static func exactConfigTokenID(_ argument: String) -> String? {
- guard argument.hasPrefix("{config:"), argument.hasSuffix("}") else { return nil }
- let optionID = String(argument.dropFirst("{config:".count).dropLast())
- return optionID.isEmpty ? nil : optionID
- }
-
- private static func interpolatedArgument(_ argument: String, installation: CrawlAppInstallation) throws -> String {
- var value = argument
- while let range = value.range(of: #"\{config:([A-Za-z0-9_.-]+)\}"#, options: .regularExpression) {
- let token = String(value[range])
- let optionID = String(token.dropFirst("{config:".count).dropLast())
- guard let replacement = Self.configValue(optionID, installation: installation) else {
- throw CrawlCommandRunnerError.missingRequiredConfig(appID: installation.id, optionID: optionID)
- }
- value.replaceSubrange(range, with: replacement)
- }
- return value
- }
-
- private static func sshArguments(
- for installation: CrawlAppInstallation,
- remoteArguments: [String],
- remoteBinaryOverride: String? = nil)
- throws -> [String]
- {
- guard let execution = installation.manifest.execution else {
- return remoteArguments
- }
- let targetOptionID = execution.targetConfigID?.nilIfBlank ?? "remote_target"
- guard let target = Self.configValue(targetOptionID, installation: installation) else {
- throw CrawlCommandRunnerError.missingRequiredConfig(appID: installation.id, optionID: targetOptionID)
- }
- guard !target.hasPrefix("-"), !target.contains(where: { $0.isWhitespace }) else {
- throw CrawlCommandRunnerError.invalidRemoteTarget(appID: installation.id, target: target)
- }
-
- let remoteBinary = remoteBinaryOverride?.nilIfBlank
- ?? execution.remoteBinary?.nilIfBlank
- ?? installation.manifest.binary.name
- var commandParts = [remoteBinary] + remoteArguments
- let envFile = execution.remoteEnvFileConfigID
- .flatMap { Self.configValue($0, installation: installation) }
- let userCommand = Self.remoteShellCommand(commandParts: commandParts, envFile: envFile)
- if let runAsOptionID = execution.runAsConfigID?.nilIfBlank,
- let runAs = Self.configValue(runAsOptionID, installation: installation)
- {
- guard !runAs.hasPrefix("-"), !runAs.contains(where: { $0.isWhitespace }) else {
- throw CrawlCommandRunnerError.invalidRemoteTarget(appID: installation.id, target: runAs)
- }
- commandParts = ["sudo", "-u", runAs, "-H", "--", "sh", "-lc", userCommand]
- return ["--", target, commandParts.map(Self.shellQuoted).joined(separator: " ")]
- }
- if envFile?.nilIfBlank != nil {
- return ["--", target, ["sh", "-lc", userCommand].map(Self.shellQuoted).joined(separator: " ")]
- }
- return ["--", target, commandParts.map(Self.shellQuoted).joined(separator: " ")]
- }
-
- private static func remoteShellCommand(commandParts: [String], envFile: String?) -> String {
- let command = commandParts.map(Self.shellQuoted).joined(separator: " ")
- guard let envFile = envFile?.nilIfBlank else {
- return "cd ~ && exec " + command
- }
- return "cd ~ && set -a && . \(Self.shellQuoted(envFile)) && set +a && exec \(command)"
- }
-
- private static func shellQuoted(_ value: String) -> String {
- "'\(value.replacingOccurrences(of: "'", with: "'\\''"))'"
- }
-
- private static func configValue(_ optionID: String, installation: CrawlAppInstallation) -> String? {
- if let value = installation.configValues[optionID]?.nilIfBlank {
- return value
- }
- return installation.manifest.configOptions.first { $0.id == optionID }?.defaultValue?.nilIfBlank
- }
-
- private static func commandOverride(for installation: CrawlAppInstallation, action: String) -> [String]? {
- guard installation.id == BuiltInCrawlApps.birdclawID,
- Self.xAccessPath(for: installation) == "birdclaw"
- else { return nil }
- switch action {
- case "status", "doctor":
- return ["auth", "status", "--json"]
- case "search", "query":
- return ["--json", "search", "tweets"]
- default:
- return nil
- }
- }
-
- private static func effectiveBinaryName(for installation: CrawlAppInstallation) -> String {
- guard installation.id == BuiltInCrawlApps.birdclawID else {
- return installation.manifest.binary.name
- }
- return Self.xAccessPath(for: installation) == "birdclaw" ? "birdclaw" : "bird"
- }
-
- private static func xAccessPath(for installation: CrawlAppInstallation) -> String {
- (Self.configValue("access_path", installation: installation) ?? "bird")
- .trimmingCharacters(in: .whitespacesAndNewlines)
- .lowercased()
- }
-
- private static func wacliSearchNeedsJoinedQuery(
- installation: CrawlAppInstallation,
- action: String,
- extraArguments: [String])
- -> Bool
- {
- guard !extraArguments.isEmpty,
- action == "search" || action == "query",
- Self.isWacliInstallation(installation)
- else { return false }
- return true
- }
-
- private static func isWacliInstallation(_ installation: CrawlAppInstallation) -> Bool {
- installation.id == BuiltInCrawlApps.wacliID
- || installation.id.rawValue.hasPrefix("wacli-")
- || installation.manifest.binary.name == "wacli"
- }
-
- private static func wacliSearchArguments(_ extraArguments: [String]) -> [String] {
- var queryParts: [String] = []
- var optionArguments: [String] = []
- var reachedOptions = false
- for argument in extraArguments {
- if argument.hasPrefix("-") {
- reachedOptions = true
- }
- if reachedOptions {
- optionArguments.append(argument)
- } else {
- queryParts.append(argument)
- }
- }
- guard queryParts.count > 1 else { return extraArguments }
- return [queryParts.joined(separator: " ")] + optionArguments
- }
-
- private static func gitcrawlQueryNeedsRepository(action: String, extraArguments: [String]) -> Bool {
- (action == "query" || action == "search") && !extraArguments.isEmpty && !extraArguments.contains("--query")
- }
-
- private static func gitcrawlRefreshNeedsRepository(action: String, commandArguments: [String]) -> Bool {
- guard action == "refresh" || action == "sync",
- let command = commandArguments.first,
- command == "refresh" || command == "sync"
- else { return false }
- return !commandArguments.dropFirst().contains { !$0.hasPrefix("-") && $0.contains("/") }
- }
}
diff --git a/Sources/CrawlBarCore/Config.swift b/Sources/CrawlBarCore/Config.swift
deleted file mode 100644
index 2d7cba8..0000000
--- a/Sources/CrawlBarCore/Config.swift
+++ /dev/null
@@ -1,399 +0,0 @@
-import Foundation
-
-public enum RefreshFrequency: String, Codable, CaseIterable, Sendable {
- case manual
- case fiveMinutes = "5m"
- case fifteenMinutes = "15m"
- case thirtyMinutes = "30m"
- case hourly = "1h"
-
- public var seconds: TimeInterval? {
- switch self {
- case .manual:
- nil
- case .fiveMinutes:
- 300
- case .fifteenMinutes:
- 900
- case .thirtyMinutes:
- 1_800
- case .hourly:
- 3_600
- }
- }
-}
-
-public struct CrawlBarAppConfig: Codable, Equatable, Sendable, Identifiable {
- public var id: CrawlAppID
- public var enabled: Bool
- public var binaryPath: String?
- public var configPath: String?
- public var refreshFrequency: RefreshFrequency?
- public var preferredRefreshAction: String?
- public var autoRefreshEnabled: Bool
- public var shareEnabled: Bool
- public var shareAfterRefresh: Bool
- public var preferredShareAction: String?
- public var preferredUpdateAction: String?
- public var showInMenuBar: Bool
- public var configValues: [String: String]
-
- public init(
- id: CrawlAppID,
- enabled: Bool = true,
- binaryPath: String? = nil,
- configPath: String? = nil,
- refreshFrequency: RefreshFrequency? = nil,
- preferredRefreshAction: String? = "refresh",
- autoRefreshEnabled: Bool = false,
- shareEnabled: Bool = false,
- shareAfterRefresh: Bool = false,
- preferredShareAction: String? = "publish",
- preferredUpdateAction: String? = "update",
- showInMenuBar: Bool = true,
- configValues: [String: String] = [:])
- {
- self.id = id
- self.enabled = enabled
- self.binaryPath = binaryPath
- self.configPath = configPath
- self.refreshFrequency = refreshFrequency
- self.preferredRefreshAction = preferredRefreshAction
- self.autoRefreshEnabled = autoRefreshEnabled
- self.shareEnabled = shareEnabled
- self.shareAfterRefresh = shareAfterRefresh
- self.preferredShareAction = preferredShareAction
- self.preferredUpdateAction = preferredUpdateAction
- self.showInMenuBar = showInMenuBar
- self.configValues = configValues
- }
-
- private enum CodingKeys: String, CodingKey {
- case id
- case enabled
- case binaryPath = "binary_path"
- case configPath = "config_path"
- case refreshFrequency = "refresh_frequency"
- case preferredRefreshAction = "preferred_refresh_action"
- case autoRefreshEnabled = "auto_refresh_enabled"
- case shareEnabled = "share_enabled"
- case shareAfterRefresh = "share_after_refresh"
- case preferredShareAction = "preferred_share_action"
- case preferredUpdateAction = "preferred_update_action"
- case showInMenuBar = "show_in_menu_bar"
- case configValues = "config_values"
- }
-
- public init(from decoder: Decoder) throws {
- let container = try decoder.container(keyedBy: CodingKeys.self)
- self.id = try container.decode(CrawlAppID.self, forKey: .id)
- self.enabled = try container.decodeIfPresent(Bool.self, forKey: .enabled) ?? true
- self.binaryPath = try container.decodeIfPresent(String.self, forKey: .binaryPath)
- self.configPath = try container.decodeIfPresent(String.self, forKey: .configPath)
- self.refreshFrequency = try container.decodeIfPresent(RefreshFrequency.self, forKey: .refreshFrequency)
- self.preferredRefreshAction = try container.decodeIfPresent(String.self, forKey: .preferredRefreshAction) ?? "refresh"
- self.autoRefreshEnabled = try container.decodeIfPresent(Bool.self, forKey: .autoRefreshEnabled) ?? false
- self.shareEnabled = try container.decodeIfPresent(Bool.self, forKey: .shareEnabled) ?? false
- self.shareAfterRefresh = try container.decodeIfPresent(Bool.self, forKey: .shareAfterRefresh) ?? false
- self.preferredShareAction = try container.decodeIfPresent(String.self, forKey: .preferredShareAction) ?? "publish"
- self.preferredUpdateAction = try container.decodeIfPresent(String.self, forKey: .preferredUpdateAction) ?? "update"
- self.showInMenuBar = try container.decodeIfPresent(Bool.self, forKey: .showInMenuBar) ?? true
- self.configValues = try container.decodeIfPresent([String: String].self, forKey: .configValues) ?? [:]
- }
-
- public func encode(to encoder: Encoder) throws {
- var container = encoder.container(keyedBy: CodingKeys.self)
- try container.encode(self.id, forKey: .id)
- try container.encode(self.enabled, forKey: .enabled)
- try container.encodeIfPresent(self.binaryPath, forKey: .binaryPath)
- try container.encodeIfPresent(self.configPath, forKey: .configPath)
- try container.encodeIfPresent(self.refreshFrequency, forKey: .refreshFrequency)
- try container.encodeIfPresent(self.preferredRefreshAction, forKey: .preferredRefreshAction)
- try container.encode(self.autoRefreshEnabled, forKey: .autoRefreshEnabled)
- try container.encode(self.shareEnabled, forKey: .shareEnabled)
- try container.encode(self.shareAfterRefresh, forKey: .shareAfterRefresh)
- try container.encodeIfPresent(self.preferredShareAction, forKey: .preferredShareAction)
- try container.encodeIfPresent(self.preferredUpdateAction, forKey: .preferredUpdateAction)
- try container.encode(self.showInMenuBar, forKey: .showInMenuBar)
- if !self.configValues.isEmpty {
- try container.encode(self.configValues, forKey: .configValues)
- }
- }
-}
-
-public struct CrawlBarConfig: Codable, Equatable, Sendable {
- public static let currentVersion = 3
-
- public var version: Int
- public var refreshFrequency: RefreshFrequency
- public var manifestDirectories: [String]
- public var apps: [CrawlBarAppConfig]
-
- public init(
- version: Int = Self.currentVersion,
- refreshFrequency: RefreshFrequency = .fifteenMinutes,
- manifestDirectories: [String] = ["~/.crawlbar/apps"],
- apps: [CrawlBarAppConfig] = [])
- {
- self.version = version
- self.refreshFrequency = refreshFrequency
- self.manifestDirectories = manifestDirectories
- self.apps = apps
- }
-
- public func normalized(knownIDs: [CrawlAppID] = BuiltInCrawlApps.all.map(\.id)) -> CrawlBarConfig {
- var seen: Set = []
- var normalizedApps: [CrawlBarAppConfig] = []
- for var app in self.apps where !seen.contains(app.id) {
- seen.insert(app.id)
- if BuiltInCrawlApps.manifest(for: app.id)?.availability == .comingSoon {
- app.enabled = false
- app.showInMenuBar = false
- app.autoRefreshEnabled = false
- app.shareEnabled = false
- app.shareAfterRefresh = false
- } else if Self.shouldEnableNewlyAvailableApp(id: app.id, fromVersion: self.version),
- !app.enabled, !app.showInMenuBar
- {
- app.enabled = true
- app.showInMenuBar = true
- }
- normalizedApps.append(app)
- }
- for id in knownIDs where !seen.contains(id) {
- let enabled = BuiltInCrawlApps.manifest(for: id)?.availability != .comingSoon
- normalizedApps.append(CrawlBarAppConfig(id: id, enabled: enabled, showInMenuBar: enabled))
- }
- return CrawlBarConfig(
- version: Self.currentVersion,
- refreshFrequency: self.refreshFrequency,
- manifestDirectories: self.manifestDirectories.isEmpty ? ["~/.crawlbar/apps"] : self.manifestDirectories,
- apps: normalizedApps)
- }
-
- private static let newlyAvailableV2AppIDs: Set = [
- BuiltInCrawlApps.gogcliID,
- BuiltInCrawlApps.wacliID,
- ]
-
- private static let newlyAvailableV3AppIDs: Set = [
- BuiltInCrawlApps.birdclawID,
- ]
-
- private static func shouldEnableNewlyAvailableApp(id: CrawlAppID, fromVersion version: Int) -> Bool {
- (version < 2 && Self.newlyAvailableV2AppIDs.contains(id))
- || (version < 3 && Self.newlyAvailableV3AppIDs.contains(id))
- }
-
- public func appConfig(for id: CrawlAppID) -> CrawlBarAppConfig? {
- self.apps.first { $0.id == id }
- }
-
- private enum CodingKeys: String, CodingKey {
- case version
- case refreshFrequency = "refresh_frequency"
- case manifestDirectories = "manifest_directories"
- case apps
- }
-
- public init(from decoder: Decoder) throws {
- let container = try decoder.container(keyedBy: CodingKeys.self)
- self.version = try container.decodeIfPresent(Int.self, forKey: .version) ?? Self.currentVersion
- self.refreshFrequency = try container.decodeIfPresent(RefreshFrequency.self, forKey: .refreshFrequency) ?? .fifteenMinutes
- self.manifestDirectories = try container.decodeIfPresent([String].self, forKey: .manifestDirectories) ?? ["~/.crawlbar/apps"]
- self.apps = try container.decodeIfPresent([CrawlBarAppConfig].self, forKey: .apps) ?? []
- }
-
- public func encode(to encoder: Encoder) throws {
- var container = encoder.container(keyedBy: CodingKeys.self)
- try container.encode(self.version, forKey: .version)
- try container.encode(self.refreshFrequency, forKey: .refreshFrequency)
- try container.encode(self.manifestDirectories, forKey: .manifestDirectories)
- try container.encode(self.apps, forKey: .apps)
- }
-}
-
-public enum CrawlBarConfigStoreError: LocalizedError {
- case decodeFailed(String)
- case encodeFailed(String)
-
- public var errorDescription: String? {
- switch self {
- case let .decodeFailed(details):
- "Failed to decode CrawlBar config: \(details)"
- case let .encodeFailed(details):
- "Failed to encode CrawlBar config: \(details)"
- }
- }
-}
-
-public struct CrawlBarConfigStore: @unchecked Sendable {
- public var fileURL: URL
- private let fileManager: FileManager
- private let secretStore: CrawlSecretStore
- private let cache: CrawlBarConfigCache
-
- public init(
- fileURL: URL = Self.defaultURL(),
- fileManager: FileManager = .default,
- secretStore: CrawlSecretStore = CrawlSecretStore(),
- cache: CrawlBarConfigCache = .shared)
- {
- self.fileURL = fileURL
- self.fileManager = fileManager
- self.secretStore = secretStore
- self.cache = cache
- }
-
- public func load(includeSecrets: Bool = false) throws -> CrawlBarConfig? {
- guard self.fileManager.fileExists(atPath: self.fileURL.path) else { return nil }
- let modificationDate = self.modificationDate(for: self.fileURL)
- if !includeSecrets,
- let cached = self.cache.config(path: self.fileURL.path, modificationDate: modificationDate)
- {
- return cached
- }
- let data = try Data(contentsOf: self.fileURL)
- do {
- let config = try CrawlCoding.makeJSONDecoder().decode(CrawlBarConfig.self, from: data).normalized()
- if !includeSecrets {
- self.cache.set(config, path: self.fileURL.path, modificationDate: modificationDate)
- }
- return includeSecrets ? self.configWithSecrets(config) : config
- } catch {
- throw CrawlBarConfigStoreError.decodeFailed(error.localizedDescription)
- }
- }
-
- public func loadOrCreateDefault(includeSecrets: Bool = false) throws -> CrawlBarConfig {
- if let existing = try self.load(includeSecrets: includeSecrets) {
- return existing
- }
- let config = CrawlBarConfig().normalized()
- try self.save(config)
- return config
- }
-
- public func save(_ config: CrawlBarConfig, clearMissingSecretIDsByAppID: [CrawlAppID: Set] = [:]) throws {
- let normalized = config.normalized()
- let persisted = try self.configForDisk(normalized, clearMissingSecretIDsByAppID: clearMissingSecretIDsByAppID)
- let data: Data
- do {
- data = try CrawlCoding.makeJSONEncoder().encode(persisted)
- } catch {
- throw CrawlBarConfigStoreError.encodeFailed(error.localizedDescription)
- }
- let directory = self.fileURL.deletingLastPathComponent()
- if !self.fileManager.fileExists(atPath: directory.path) {
- try self.fileManager.createDirectory(at: directory, withIntermediateDirectories: true)
- }
- try data.write(to: self.fileURL, options: [.atomic])
- #if os(macOS) || os(Linux)
- try self.fileManager.setAttributes(
- [.posixPermissions: NSNumber(value: Int16(0o600))],
- ofItemAtPath: self.fileURL.path)
- #endif
- self.cache.set(persisted, path: self.fileURL.path, modificationDate: self.modificationDate(for: self.fileURL))
- }
-
- private func configWithSecrets(_ config: CrawlBarConfig) -> CrawlBarConfig {
- let manifests = self.manifestsByID(config: config)
- var copy = config
- for index in copy.apps.indices {
- guard let manifest = manifests[copy.apps[index].id] else { continue }
- copy.apps[index] = self.appConfigWithSecrets(copy.apps[index], manifest: manifest)
- }
- return copy
- }
-
- public func appConfigWithSecrets(_ appConfig: CrawlBarAppConfig, manifest: CrawlAppManifest) -> CrawlBarAppConfig {
- var copy = appConfig
- for option in manifest.configOptions where option.kind == .secret {
- guard copy.configValues[option.id]?.nilIfBlank == nil else { continue }
- do {
- if let value = try self.secretStore.value(appID: copy.id, optionID: option.id)?.nilIfBlank {
- copy.configValues[option.id] = value
- }
- } catch {
- CrawlBarLog.keychain.error(
- "Keychain read failed for \(copy.id.rawValue, privacy: .public).\(option.id, privacy: .public): \(error.localizedDescription, privacy: .public)")
- }
- }
- return copy
- }
-
- private func configForDisk(
- _ config: CrawlBarConfig,
- clearMissingSecretIDsByAppID: [CrawlAppID: Set] = [:])
- throws -> CrawlBarConfig
- {
- let manifests = self.manifestsByID(config: config)
- var copy = config
- for index in copy.apps.indices {
- guard let manifest = manifests[copy.apps[index].id] else { continue }
- for option in manifest.configOptions where option.kind == .secret {
- if let value = copy.apps[index].configValues.removeValue(forKey: option.id) {
- try self.secretStore.set(value.nilIfBlank, appID: copy.apps[index].id, optionID: option.id)
- } else if clearMissingSecretIDsByAppID[copy.apps[index].id]?.contains(option.id) == true {
- try self.secretStore.set(nil, appID: copy.apps[index].id, optionID: option.id)
- }
- }
- }
- return copy
- }
-
- private func manifestsByID(config: CrawlBarConfig) -> [CrawlAppID: CrawlAppManifest] {
- Dictionary(uniqueKeysWithValues: CrawlManifestCatalog(fileManager: self.fileManager)
- .manifests(config: config)
- .map { ($0.id, $0) })
- }
-
- private func modificationDate(for url: URL) -> Date? {
- (try? url.resourceValues(forKeys: [.contentModificationDateKey]))?.contentModificationDate
- }
-
- public static func defaultURL(home: URL = FileManager.default.homeDirectoryForCurrentUser) -> URL {
- home
- .appendingPathComponent(".crawlbar", isDirectory: true)
- .appendingPathComponent("config.json")
- }
-}
-
-public final class CrawlBarConfigCache: @unchecked Sendable {
- public static let shared = CrawlBarConfigCache()
-
- private struct Entry {
- var modificationDate: Date?
- var config: CrawlBarConfig
- }
-
- private let lock = NSLock()
- private var entries: [String: Entry] = [:]
-
- public init() {}
-
- func config(path: String, modificationDate: Date?) -> CrawlBarConfig? {
- self.lock.lock()
- defer { self.lock.unlock() }
- guard let entry = self.entries[path], entry.modificationDate == modificationDate else {
- return nil
- }
- return entry.config
- }
-
- func set(_ config: CrawlBarConfig, path: String, modificationDate: Date?) {
- self.lock.lock()
- self.entries[path] = Entry(modificationDate: modificationDate, config: config)
- self.lock.unlock()
- }
-}
-
-public enum PathExpander {
- public static func expandHome(_ path: String, home: String = NSHomeDirectory()) -> String {
- if path == "~" { return home }
- if path.hasPrefix("~/") {
- return URL(fileURLWithPath: home).appendingPathComponent(String(path.dropFirst(2))).path
- }
- return path
- }
-}
diff --git a/Sources/CrawlBarCore/Config/CrawlBarAppConfig.swift b/Sources/CrawlBarCore/Config/CrawlBarAppConfig.swift
new file mode 100644
index 0000000..3138783
--- /dev/null
+++ b/Sources/CrawlBarCore/Config/CrawlBarAppConfig.swift
@@ -0,0 +1,99 @@
+import Foundation
+
+public struct CrawlBarAppConfig: Codable, Equatable, Sendable, Identifiable {
+ public var id: CrawlAppID
+ public var enabled: Bool
+ public var binaryPath: String?
+ public var configPath: String?
+ public var refreshFrequency: RefreshFrequency?
+ public var preferredRefreshAction: String?
+ public var autoRefreshEnabled: Bool
+ public var shareEnabled: Bool
+ public var shareAfterRefresh: Bool
+ public var preferredShareAction: String?
+ public var preferredUpdateAction: String?
+ public var showInMenuBar: Bool
+ public var configValues: [String: String]
+
+ public init(
+ id: CrawlAppID,
+ enabled: Bool = true,
+ binaryPath: String? = nil,
+ configPath: String? = nil,
+ refreshFrequency: RefreshFrequency? = nil,
+ preferredRefreshAction: String? = "refresh",
+ autoRefreshEnabled: Bool = false,
+ shareEnabled: Bool = false,
+ shareAfterRefresh: Bool = false,
+ preferredShareAction: String? = "publish",
+ preferredUpdateAction: String? = "update",
+ showInMenuBar: Bool = true,
+ configValues: [String: String] = [:])
+ {
+ self.id = id
+ self.enabled = enabled
+ self.binaryPath = binaryPath
+ self.configPath = configPath
+ self.refreshFrequency = refreshFrequency
+ self.preferredRefreshAction = preferredRefreshAction
+ self.autoRefreshEnabled = autoRefreshEnabled
+ self.shareEnabled = shareEnabled
+ self.shareAfterRefresh = shareAfterRefresh
+ self.preferredShareAction = preferredShareAction
+ self.preferredUpdateAction = preferredUpdateAction
+ self.showInMenuBar = showInMenuBar
+ self.configValues = configValues
+ }
+
+ private enum CodingKeys: String, CodingKey {
+ case id
+ case enabled
+ case binaryPath = "binary_path"
+ case configPath = "config_path"
+ case refreshFrequency = "refresh_frequency"
+ case preferredRefreshAction = "preferred_refresh_action"
+ case autoRefreshEnabled = "auto_refresh_enabled"
+ case shareEnabled = "share_enabled"
+ case shareAfterRefresh = "share_after_refresh"
+ case preferredShareAction = "preferred_share_action"
+ case preferredUpdateAction = "preferred_update_action"
+ case showInMenuBar = "show_in_menu_bar"
+ case configValues = "config_values"
+ }
+
+ public init(from decoder: Decoder) throws {
+ let container = try decoder.container(keyedBy: CodingKeys.self)
+ self.id = try container.decode(CrawlAppID.self, forKey: .id)
+ self.enabled = try container.decodeIfPresent(Bool.self, forKey: .enabled) ?? true
+ self.binaryPath = try container.decodeIfPresent(String.self, forKey: .binaryPath)
+ self.configPath = try container.decodeIfPresent(String.self, forKey: .configPath)
+ self.refreshFrequency = try container.decodeIfPresent(RefreshFrequency.self, forKey: .refreshFrequency)
+ self.preferredRefreshAction = try container.decodeIfPresent(String.self, forKey: .preferredRefreshAction) ?? "refresh"
+ self.autoRefreshEnabled = try container.decodeIfPresent(Bool.self, forKey: .autoRefreshEnabled) ?? false
+ self.shareEnabled = try container.decodeIfPresent(Bool.self, forKey: .shareEnabled) ?? false
+ self.shareAfterRefresh = try container.decodeIfPresent(Bool.self, forKey: .shareAfterRefresh) ?? false
+ self.preferredShareAction = try container.decodeIfPresent(String.self, forKey: .preferredShareAction) ?? "publish"
+ self.preferredUpdateAction = try container.decodeIfPresent(String.self, forKey: .preferredUpdateAction) ?? "update"
+ self.showInMenuBar = try container.decodeIfPresent(Bool.self, forKey: .showInMenuBar) ?? true
+ self.configValues = try container.decodeIfPresent([String: String].self, forKey: .configValues) ?? [:]
+ }
+
+ public func encode(to encoder: Encoder) throws {
+ var container = encoder.container(keyedBy: CodingKeys.self)
+ try container.encode(self.id, forKey: .id)
+ try container.encode(self.enabled, forKey: .enabled)
+ try container.encodeIfPresent(self.binaryPath, forKey: .binaryPath)
+ try container.encodeIfPresent(self.configPath, forKey: .configPath)
+ try container.encodeIfPresent(self.refreshFrequency, forKey: .refreshFrequency)
+ try container.encodeIfPresent(self.preferredRefreshAction, forKey: .preferredRefreshAction)
+ try container.encode(self.autoRefreshEnabled, forKey: .autoRefreshEnabled)
+ try container.encode(self.shareEnabled, forKey: .shareEnabled)
+ try container.encode(self.shareAfterRefresh, forKey: .shareAfterRefresh)
+ try container.encodeIfPresent(self.preferredShareAction, forKey: .preferredShareAction)
+ try container.encodeIfPresent(self.preferredUpdateAction, forKey: .preferredUpdateAction)
+ try container.encode(self.showInMenuBar, forKey: .showInMenuBar)
+ if !self.configValues.isEmpty {
+ try container.encode(self.configValues, forKey: .configValues)
+ }
+ }
+}
diff --git a/Sources/CrawlBarCore/Config/CrawlBarConfig.swift b/Sources/CrawlBarCore/Config/CrawlBarConfig.swift
new file mode 100644
index 0000000..9b77903
--- /dev/null
+++ b/Sources/CrawlBarCore/Config/CrawlBarConfig.swift
@@ -0,0 +1,93 @@
+import Foundation
+
+public struct CrawlBarConfig: Codable, Equatable, Sendable {
+ public static let currentVersion = 3
+
+ public var version: Int
+ public var refreshFrequency: RefreshFrequency
+ public var manifestDirectories: [String]
+ public var apps: [CrawlBarAppConfig]
+
+ public init(
+ version: Int = Self.currentVersion,
+ refreshFrequency: RefreshFrequency = .fifteenMinutes,
+ manifestDirectories: [String] = ["~/.crawlbar/apps"],
+ apps: [CrawlBarAppConfig] = [])
+ {
+ self.version = version
+ self.refreshFrequency = refreshFrequency
+ self.manifestDirectories = manifestDirectories
+ self.apps = apps
+ }
+
+ public func normalized(knownIDs: [CrawlAppID] = BuiltInCrawlApps.all.map(\.id)) -> CrawlBarConfig {
+ var seen: Set = []
+ var normalizedApps: [CrawlBarAppConfig] = []
+ for var app in self.apps where !seen.contains(app.id) {
+ seen.insert(app.id)
+ if BuiltInCrawlApps.manifest(for: app.id)?.availability == .comingSoon {
+ app.enabled = false
+ app.showInMenuBar = false
+ app.autoRefreshEnabled = false
+ app.shareEnabled = false
+ app.shareAfterRefresh = false
+ } else if Self.shouldEnableNewlyAvailableApp(id: app.id, fromVersion: self.version),
+ !app.enabled, !app.showInMenuBar
+ {
+ app.enabled = true
+ app.showInMenuBar = true
+ }
+ normalizedApps.append(app)
+ }
+ for id in knownIDs where !seen.contains(id) {
+ let enabled = BuiltInCrawlApps.manifest(for: id)?.availability != .comingSoon
+ normalizedApps.append(CrawlBarAppConfig(id: id, enabled: enabled, showInMenuBar: enabled))
+ }
+ return CrawlBarConfig(
+ version: Self.currentVersion,
+ refreshFrequency: self.refreshFrequency,
+ manifestDirectories: self.manifestDirectories.isEmpty ? ["~/.crawlbar/apps"] : self.manifestDirectories,
+ apps: normalizedApps)
+ }
+
+ public func appConfig(for id: CrawlAppID) -> CrawlBarAppConfig? {
+ self.apps.first { $0.id == id }
+ }
+
+ private static let newlyAvailableV2AppIDs: Set = [
+ BuiltInCrawlApps.gogcliID,
+ BuiltInCrawlApps.wacliID,
+ ]
+
+ private static let newlyAvailableV3AppIDs: Set = [
+ BuiltInCrawlApps.birdclawID,
+ ]
+
+ private static func shouldEnableNewlyAvailableApp(id: CrawlAppID, fromVersion version: Int) -> Bool {
+ (version < 2 && Self.newlyAvailableV2AppIDs.contains(id))
+ || (version < 3 && Self.newlyAvailableV3AppIDs.contains(id))
+ }
+
+ private enum CodingKeys: String, CodingKey {
+ case version
+ case refreshFrequency = "refresh_frequency"
+ case manifestDirectories = "manifest_directories"
+ case apps
+ }
+
+ public init(from decoder: Decoder) throws {
+ let container = try decoder.container(keyedBy: CodingKeys.self)
+ self.version = try container.decodeIfPresent(Int.self, forKey: .version) ?? Self.currentVersion
+ self.refreshFrequency = try container.decodeIfPresent(RefreshFrequency.self, forKey: .refreshFrequency) ?? .fifteenMinutes
+ self.manifestDirectories = try container.decodeIfPresent([String].self, forKey: .manifestDirectories) ?? ["~/.crawlbar/apps"]
+ self.apps = try container.decodeIfPresent([CrawlBarAppConfig].self, forKey: .apps) ?? []
+ }
+
+ public func encode(to encoder: Encoder) throws {
+ var container = encoder.container(keyedBy: CodingKeys.self)
+ try container.encode(self.version, forKey: .version)
+ try container.encode(self.refreshFrequency, forKey: .refreshFrequency)
+ try container.encode(self.manifestDirectories, forKey: .manifestDirectories)
+ try container.encode(self.apps, forKey: .apps)
+ }
+}
diff --git a/Sources/CrawlBarCore/Config/CrawlBarConfigCache.swift b/Sources/CrawlBarCore/Config/CrawlBarConfigCache.swift
new file mode 100644
index 0000000..146d2d6
--- /dev/null
+++ b/Sources/CrawlBarCore/Config/CrawlBarConfigCache.swift
@@ -0,0 +1,30 @@
+import Foundation
+
+public final class CrawlBarConfigCache: @unchecked Sendable {
+ public static let shared = CrawlBarConfigCache()
+
+ private struct Entry {
+ var modificationDate: Date?
+ var config: CrawlBarConfig
+ }
+
+ private let lock = NSLock()
+ private var entries: [String: Entry] = [:]
+
+ public init() {}
+
+ func config(path: String, modificationDate: Date?) -> CrawlBarConfig? {
+ self.lock.lock()
+ defer { self.lock.unlock() }
+ guard let entry = self.entries[path], entry.modificationDate == modificationDate else {
+ return nil
+ }
+ return entry.config
+ }
+
+ func set(_ config: CrawlBarConfig, path: String, modificationDate: Date?) {
+ self.lock.lock()
+ self.entries[path] = Entry(modificationDate: modificationDate, config: config)
+ self.lock.unlock()
+ }
+}
diff --git a/Sources/CrawlBarCore/Config/CrawlBarConfigStore.swift b/Sources/CrawlBarCore/Config/CrawlBarConfigStore.swift
new file mode 100644
index 0000000..dd1529c
--- /dev/null
+++ b/Sources/CrawlBarCore/Config/CrawlBarConfigStore.swift
@@ -0,0 +1,133 @@
+import Foundation
+
+public struct CrawlBarConfigStore: @unchecked Sendable {
+ public var fileURL: URL
+ private let fileManager: FileManager
+ private let secretStore: CrawlSecretStore
+ private let cache: CrawlBarConfigCache
+
+ public init(
+ fileURL: URL = Self.defaultURL(),
+ fileManager: FileManager = .default,
+ secretStore: CrawlSecretStore = CrawlSecretStore(),
+ cache: CrawlBarConfigCache = .shared)
+ {
+ self.fileURL = fileURL
+ self.fileManager = fileManager
+ self.secretStore = secretStore
+ self.cache = cache
+ }
+
+ public func load(includeSecrets: Bool = false) throws -> CrawlBarConfig? {
+ guard self.fileManager.fileExists(atPath: self.fileURL.path) else { return nil }
+ let modificationDate = self.modificationDate(for: self.fileURL)
+ if !includeSecrets,
+ let cached = self.cache.config(path: self.fileURL.path, modificationDate: modificationDate)
+ {
+ return cached
+ }
+ let data = try Data(contentsOf: self.fileURL)
+ do {
+ let config = try CrawlCoding.makeJSONDecoder().decode(CrawlBarConfig.self, from: data).normalized()
+ if !includeSecrets {
+ self.cache.set(config, path: self.fileURL.path, modificationDate: modificationDate)
+ }
+ return includeSecrets ? self.configWithSecrets(config) : config
+ } catch {
+ throw CrawlBarConfigStoreError.decodeFailed(error.localizedDescription)
+ }
+ }
+
+ public func loadOrCreateDefault(includeSecrets: Bool = false) throws -> CrawlBarConfig {
+ if let existing = try self.load(includeSecrets: includeSecrets) {
+ return existing
+ }
+ let config = CrawlBarConfig().normalized()
+ try self.save(config)
+ return config
+ }
+
+ public func save(_ config: CrawlBarConfig, clearMissingSecretIDsByAppID: [CrawlAppID: Set] = [:]) throws {
+ let normalized = config.normalized()
+ let persisted = try self.configForDisk(normalized, clearMissingSecretIDsByAppID: clearMissingSecretIDsByAppID)
+ let data: Data
+ do {
+ data = try CrawlCoding.makeJSONEncoder().encode(persisted)
+ } catch {
+ throw CrawlBarConfigStoreError.encodeFailed(error.localizedDescription)
+ }
+ let directory = self.fileURL.deletingLastPathComponent()
+ if !self.fileManager.fileExists(atPath: directory.path) {
+ try self.fileManager.createDirectory(at: directory, withIntermediateDirectories: true)
+ }
+ try data.write(to: self.fileURL, options: [.atomic])
+ #if os(macOS) || os(Linux)
+ try self.fileManager.setAttributes(
+ [.posixPermissions: NSNumber(value: Int16(0o600))],
+ ofItemAtPath: self.fileURL.path)
+ #endif
+ self.cache.set(persisted, path: self.fileURL.path, modificationDate: self.modificationDate(for: self.fileURL))
+ }
+
+ public func appConfigWithSecrets(_ appConfig: CrawlBarAppConfig, manifest: CrawlAppManifest) -> CrawlBarAppConfig {
+ var copy = appConfig
+ for option in manifest.configOptions where option.kind == .secret {
+ guard copy.configValues[option.id]?.nilIfBlank == nil else { continue }
+ do {
+ if let value = try self.secretStore.value(appID: copy.id, optionID: option.id)?.nilIfBlank {
+ copy.configValues[option.id] = value
+ }
+ } catch {
+ CrawlBarLog.keychain.error(
+ "Keychain read failed for \(copy.id.rawValue, privacy: .public).\(option.id, privacy: .public): \(error.localizedDescription, privacy: .public)")
+ }
+ }
+ return copy
+ }
+
+ public static func defaultURL(home: URL = FileManager.default.homeDirectoryForCurrentUser) -> URL {
+ home
+ .appendingPathComponent(".crawlbar", isDirectory: true)
+ .appendingPathComponent("config.json")
+ }
+
+ private func configWithSecrets(_ config: CrawlBarConfig) -> CrawlBarConfig {
+ let manifests = self.manifestsByID(config: config)
+ var copy = config
+ for index in copy.apps.indices {
+ guard let manifest = manifests[copy.apps[index].id] else { continue }
+ copy.apps[index] = self.appConfigWithSecrets(copy.apps[index], manifest: manifest)
+ }
+ return copy
+ }
+
+ private func configForDisk(
+ _ config: CrawlBarConfig,
+ clearMissingSecretIDsByAppID: [CrawlAppID: Set] = [:])
+ throws -> CrawlBarConfig
+ {
+ let manifests = self.manifestsByID(config: config)
+ var copy = config
+ for index in copy.apps.indices {
+ guard let manifest = manifests[copy.apps[index].id] else { continue }
+ for option in manifest.configOptions where option.kind == .secret {
+ if let value = copy.apps[index].configValues.removeValue(forKey: option.id) {
+ try self.secretStore.set(value.nilIfBlank, appID: copy.apps[index].id, optionID: option.id)
+ } else if clearMissingSecretIDsByAppID[copy.apps[index].id]?.contains(option.id) == true {
+ try self.secretStore.set(nil, appID: copy.apps[index].id, optionID: option.id)
+ }
+ }
+ }
+ return copy
+ }
+
+ private func manifestsByID(config: CrawlBarConfig) -> [CrawlAppID: CrawlAppManifest] {
+ Dictionary(uniqueKeysWithValues: CrawlManifestCatalog(fileManager: self.fileManager)
+ .manifests(config: config)
+ .map { ($0.id, $0) })
+ }
+
+ private func modificationDate(for url: URL) -> Date? {
+ (try? url.resourceValues(forKeys: [.contentModificationDateKey]))?.contentModificationDate
+ }
+}
diff --git a/Sources/CrawlBarCore/Config/CrawlBarConfigStoreError.swift b/Sources/CrawlBarCore/Config/CrawlBarConfigStoreError.swift
new file mode 100644
index 0000000..1ab0fd3
--- /dev/null
+++ b/Sources/CrawlBarCore/Config/CrawlBarConfigStoreError.swift
@@ -0,0 +1,15 @@
+import Foundation
+
+public enum CrawlBarConfigStoreError: LocalizedError {
+ case decodeFailed(String)
+ case encodeFailed(String)
+
+ public var errorDescription: String? {
+ switch self {
+ case let .decodeFailed(details):
+ "Failed to decode CrawlBar config: \(details)"
+ case let .encodeFailed(details):
+ "Failed to encode CrawlBar config: \(details)"
+ }
+ }
+}
diff --git a/Sources/CrawlBarCore/Config/RefreshFrequency.swift b/Sources/CrawlBarCore/Config/RefreshFrequency.swift
new file mode 100644
index 0000000..9cad73b
--- /dev/null
+++ b/Sources/CrawlBarCore/Config/RefreshFrequency.swift
@@ -0,0 +1,24 @@
+import Foundation
+
+public enum RefreshFrequency: String, Codable, CaseIterable, Sendable {
+ case manual
+ case fiveMinutes = "5m"
+ case fifteenMinutes = "15m"
+ case thirtyMinutes = "30m"
+ case hourly = "1h"
+
+ public var seconds: TimeInterval? {
+ switch self {
+ case .manual:
+ nil
+ case .fiveMinutes:
+ 300
+ case .fifteenMinutes:
+ 900
+ case .thirtyMinutes:
+ 1_800
+ case .hourly:
+ 3_600
+ }
+ }
+}
diff --git a/Sources/CrawlBarCore/CrawlAppStatus+CommandFailures.swift b/Sources/CrawlBarCore/CrawlAppStatus+CommandFailures.swift
new file mode 100644
index 0000000..475b10f
--- /dev/null
+++ b/Sources/CrawlBarCore/CrawlAppStatus+CommandFailures.swift
@@ -0,0 +1,151 @@
+import Foundation
+
+public extension CrawlAppStatus {
+ var isRecoverableGraincrawlSourceFailure: Bool {
+ guard self.appID == BuiltInCrawlApps.graincrawlID, self.state == .error else { return false }
+ guard !Self.summaryLooksLikeActionFailure(self.summary) else { return false }
+ let text = ([self.summary] + self.errors + self.warnings)
+ .joined(separator: "\n")
+ .lowercased()
+ return text.contains("granola access token")
+ || text.contains("unsupported cache version")
+ || text.contains("private-api reports")
+ || text.contains("desktop-cache reports")
+ }
+
+ func mergingActionFailure(_ failure: CrawlAppStatus) -> CrawlAppStatus {
+ guard self.appID == failure.appID else { return failure }
+ return CrawlAppStatus(
+ schemaVersion: self.schemaVersion,
+ appID: self.appID,
+ generatedAt: failure.generatedAt,
+ state: failure.state,
+ summary: failure.summary,
+ configPath: self.configPath,
+ databasePath: self.databasePath,
+ databaseBytes: self.databaseBytes,
+ walBytes: self.walBytes,
+ lastSyncAt: self.lastSyncAt,
+ lastImportAt: self.lastImportAt,
+ lastExportAt: self.lastExportAt,
+ counts: self.counts,
+ databases: self.databases,
+ freshness: self.freshness,
+ share: self.share,
+ remote: self.remote,
+ sqliteObject: self.sqliteObject,
+ sqliteBundle: self.sqliteBundle,
+ warnings: Self.mergedMessages(failure.warnings, self.warnings),
+ errors: Self.mergedMessages(failure.errors, self.errors))
+ }
+
+ static func commandFailure(
+ appID: CrawlAppID,
+ action: String? = nil,
+ message: String?,
+ fallback: String)
+ -> CrawlAppStatus
+ {
+ let fullMessage = message?.nilIfBlank ?? fallback
+ let normalized = Self.normalizedCommandFailure(appID: appID, message: fullMessage)
+ let summary = [action?.nilIfBlank, normalized.summary].compactMap { $0 }.joined(separator: ": ")
+ return CrawlAppStatus(
+ appID: appID,
+ state: normalized.state,
+ summary: summary,
+ errors: [normalized.summary])
+ }
+
+ static func richestMetadataStatus(_ preferred: CrawlAppStatus?, fallback: CrawlAppStatus?) -> CrawlAppStatus? {
+ guard let preferred else { return fallback }
+ guard let fallback else { return preferred }
+ return preferred.metadataScore >= fallback.metadataScore ? preferred : fallback
+ }
+
+ private var metadataScore: Int {
+ var score = 0
+ score += self.configPath == nil ? 0 : 1
+ score += self.databasePath == nil ? 0 : 1
+ score += self.databaseBytes == nil ? 0 : 1
+ score += self.walBytes == nil ? 0 : 1
+ score += self.lastSyncAt == nil ? 0 : 1
+ score += self.lastImportAt == nil ? 0 : 1
+ score += self.lastExportAt == nil ? 0 : 1
+ score += self.counts.isEmpty ? 0 : 2
+ score += self.databases.isEmpty ? 0 : 3
+ score += self.freshness == nil ? 0 : 1
+ score += self.share == nil ? 0 : 2
+ score += self.remote == nil ? 0 : 3
+ score += self.sqliteObject == nil ? 0 : 2
+ score += self.sqliteBundle == nil ? 0 : 3
+ return score
+ }
+
+ private static func mergedMessages(_ primary: [String], _ secondary: [String]) -> [String] {
+ var seen = Set()
+ var messages: [String] = []
+ for message in primary + secondary {
+ guard !seen.contains(message) else { continue }
+ seen.insert(message)
+ messages.append(message)
+ }
+ return messages
+ }
+
+ private static func summaryLooksLikeActionFailure(_ summary: String) -> Bool {
+ let trimmed = summary.trimmingCharacters(in: .whitespacesAndNewlines).lowercased()
+ return [
+ "refresh:",
+ "sync:",
+ "desktop-cache-import:",
+ "doctor:",
+ "unlock:",
+ "query:",
+ "search:",
+ "export-md:",
+ ].contains { trimmed.hasPrefix($0) }
+ }
+
+ private static func normalizedCommandFailure(appID: CrawlAppID, message: String) -> (state: CrawlAppState, summary: String) {
+ let lowered = message.lowercased()
+ if appID == BuiltInCrawlApps.gitcrawlID,
+ lowered.contains("github"),
+ (lowered.contains("bad credentials") || lowered.contains("status 401") || lowered.contains("401"))
+ {
+ return (.needsAuth, "GitHub credentials rejected")
+ }
+ if appID == BuiltInCrawlApps.birdclawID,
+ lowered.contains("no twitter cookies")
+ || lowered.contains("no x cookies")
+ || lowered.contains("missing credentials")
+ || lowered.contains("missing auth_token")
+ || lowered.contains("missing ct0")
+ {
+ return (.needsAuth, "X browser cookies not found")
+ }
+ if appID == BuiltInCrawlApps.gogcliID,
+ lowered.contains("credentials") || lowered.contains("auth")
+ {
+ return (.needsAuth, "Google account needs auth")
+ }
+ return (.error, Self.firstUsefulFailureLine(in: message) ?? "Command failed")
+ }
+
+ private static func firstUsefulFailureLine(in message: String) -> String? {
+ message.split(separator: "\n")
+ .map { $0.trimmingCharacters(in: .whitespacesAndNewlines) }
+ .first { line in
+ guard !line.isEmpty else { return false }
+ return !Self.isRequestTraceLine(line)
+ }
+ }
+
+ private static func isRequestTraceLine(_ line: String) -> Bool {
+ let lowered = line.lowercased()
+ return lowered.hasPrefix("[github] request ")
+ || lowered.hasPrefix("[slack] request ")
+ || lowered.hasPrefix("[notion] request ")
+ || lowered.hasPrefix("[discord] request ")
+ || lowered.hasPrefix("[granola] request ")
+ }
+}
diff --git a/Sources/CrawlBarCore/CrawlCommandModels.swift b/Sources/CrawlBarCore/CrawlCommandModels.swift
new file mode 100644
index 0000000..075f1c2
--- /dev/null
+++ b/Sources/CrawlBarCore/CrawlCommandModels.swift
@@ -0,0 +1,96 @@
+import Foundation
+
+public struct CrawlAppInstallation: Codable, Equatable, Sendable, Identifiable {
+ public var manifest: CrawlAppManifest
+ public var binaryPath: String?
+ public var configPathOverride: String?
+ public var configValues: [String: String]
+ public var staleAfterSeconds: Int?
+ public var enabled: Bool
+
+ public var id: CrawlAppID {
+ self.manifest.id
+ }
+
+ public init(
+ manifest: CrawlAppManifest,
+ binaryPath: String? = nil,
+ configPathOverride: String? = nil,
+ configValues: [String: String] = [:],
+ staleAfterSeconds: Int? = nil,
+ enabled: Bool = true)
+ {
+ self.manifest = manifest
+ self.binaryPath = binaryPath
+ self.configPathOverride = configPathOverride
+ self.configValues = configValues
+ self.staleAfterSeconds = staleAfterSeconds
+ self.enabled = enabled
+ }
+}
+
+public enum CrawlActionID: String, Codable, Hashable, Sendable {
+ case status
+ case doctor
+ case refresh
+ case publish
+ case update
+ case desktopCacheImport = "desktop-cache-import"
+ case exportMarkdown = "export-md"
+}
+
+public struct CrawlCommandResult: Codable, Equatable, Sendable {
+ public var appID: CrawlAppID
+ public var action: String
+ public var exitCode: Int32
+ public var stdout: String
+ public var stderr: String
+ public var startedAt: Date
+ public var finishedAt: Date
+
+ public var succeeded: Bool {
+ self.exitCode == 0
+ }
+
+ public init(
+ appID: CrawlAppID,
+ action: String,
+ exitCode: Int32,
+ stdout: String,
+ stderr: String,
+ startedAt: Date,
+ finishedAt: Date)
+ {
+ self.appID = appID
+ self.action = action
+ self.exitCode = exitCode
+ self.stdout = stdout
+ self.stderr = stderr
+ self.startedAt = startedAt
+ self.finishedAt = finishedAt
+ }
+}
+
+public extension CrawlCommandResult {
+ var userFacingRunMessage: String? {
+ if self.succeeded {
+ return Self.firstLine(in: self.stderr)
+ }
+ return CrawlAppStatus.commandFailure(
+ appID: self.appID,
+ message: self.stderr.nilIfBlank ?? self.stdout.nilIfBlank,
+ fallback: "\(self.action) failed with exit \(self.exitCode)")
+ .summary
+ }
+
+ var shouldShowExitCode: Bool {
+ !self.succeeded
+ }
+
+ private static func firstLine(in output: String) -> String? {
+ output.nilIfBlank?
+ .split(separator: "\n", omittingEmptySubsequences: true)
+ .first
+ .map(String.init)
+ }
+}
diff --git a/Sources/CrawlBarCore/CrawlCommandRedactor.swift b/Sources/CrawlBarCore/CrawlCommandRedactor.swift
new file mode 100644
index 0000000..0548171
--- /dev/null
+++ b/Sources/CrawlBarCore/CrawlCommandRedactor.swift
@@ -0,0 +1,29 @@
+import Foundation
+
+public struct CrawlCommandRedactor: Sendable {
+ public init() {}
+
+ public func redact(_ text: String) -> String {
+ var redacted = text
+ let patterns: [(String, String)] = [
+ (#"(?i)(Bearer[ \t]+)[^ \t\r\n"',}]+"#, "$1[REDACTED]"),
+ (#"(?i)(api[_-]?key|token|secret|password|cookie|authorization)(["' \t:=]+)([^ \t\r\n"',}]+)"#, "$1$2[REDACTED]"),
+ (#"(?i)\b(github_pat_)[A-Za-z0-9_]+\b"#, "$1[REDACTED]"),
+ (#"(?i)\b(gh[pousr]_)[A-Za-z0-9_]+\b"#, "$1[REDACTED]"),
+ (#"(?i)\b(sk-[A-Za-z0-9_-]{16,})\b"#, "[REDACTED]"),
+ (#"(?i)\b(secret_)[A-Za-z0-9_]+\b"#, "$1[REDACTED]"),
+ (#"(?i)(xox[aboprsxc]-)[A-Za-z0-9-]+"#, "$1[REDACTED]"),
+ (#"(?i)\bmfa\.[A-Za-z0-9_-]+\b"#, "[REDACTED]"),
+ (#"(?i)\b(ct0)(["' \t:=]+)([^ \t\r\n"',}]+)"#, "$1$2[REDACTED]"),
+ (#"\b[A-Za-z0-9_-]{24}\.[A-Za-z0-9_-]{6}\.[A-Za-z0-9_-]{20,}\b"#, "[REDACTED]"),
+ (#"(?i)(discord[_-]?token["' \t:=]+)([^ \t\r\n"',}]+)"#, "$1[REDACTED]"),
+ ]
+ for (pattern, template) in patterns {
+ redacted = redacted.replacingOccurrences(
+ of: pattern,
+ with: template,
+ options: [.regularExpression])
+ }
+ return redacted
+ }
+}
diff --git a/Sources/CrawlBarCore/CrawlCommandRunnerArguments.swift b/Sources/CrawlBarCore/CrawlCommandRunnerArguments.swift
new file mode 100644
index 0000000..a21d99c
--- /dev/null
+++ b/Sources/CrawlBarCore/CrawlCommandRunnerArguments.swift
@@ -0,0 +1,199 @@
+import Foundation
+
+extension CrawlCommandRunner {
+ static func commandArguments(
+ for installation: CrawlAppInstallation,
+ action: String,
+ commandArguments: [String],
+ extraArguments: [String])
+ -> [String]
+ {
+ if Self.wacliSearchNeedsJoinedQuery(installation: installation, action: action, extraArguments: extraArguments) {
+ return commandArguments + Self.wacliSearchArguments(extraArguments)
+ }
+
+ guard installation.id == BuiltInCrawlApps.gitcrawlID,
+ let repository = GitcrawlStatusSnapshot.repository(for: installation)
+ else {
+ return commandArguments + extraArguments
+ }
+
+ if Self.gitcrawlQueryNeedsRepository(action: action, extraArguments: extraArguments) {
+ return commandArguments + [repository, "--query", extraArguments.joined(separator: " ")]
+ }
+
+ if Self.gitcrawlRefreshNeedsRepository(action: action, commandArguments: commandArguments) {
+ return commandArguments + [repository] + extraArguments
+ }
+
+ return commandArguments + extraArguments
+ }
+
+ static func interpolatedArguments(_ arguments: [String], installation: CrawlAppInstallation) throws -> [String] {
+ var result: [String] = []
+ for argument in arguments {
+ if let optionID = Self.exactConfigTokenID(argument),
+ Self.configValue(optionID, installation: installation) == nil,
+ result.last == "--account"
+ {
+ result.removeLast()
+ continue
+ }
+ result.append(try Self.interpolatedArgument(argument, installation: installation))
+ }
+ return result
+ }
+
+ static func sshArguments(
+ for installation: CrawlAppInstallation,
+ remoteArguments: [String],
+ remoteBinaryOverride: String? = nil)
+ throws -> [String]
+ {
+ guard let execution = installation.manifest.execution else {
+ return remoteArguments
+ }
+ let targetOptionID = execution.targetConfigID?.nilIfBlank ?? "remote_target"
+ guard let target = Self.configValue(targetOptionID, installation: installation) else {
+ throw CrawlCommandRunnerError.missingRequiredConfig(appID: installation.id, optionID: targetOptionID)
+ }
+ guard !target.hasPrefix("-"), !target.contains(where: { $0.isWhitespace }) else {
+ throw CrawlCommandRunnerError.invalidRemoteTarget(appID: installation.id, target: target)
+ }
+
+ let remoteBinary = remoteBinaryOverride?.nilIfBlank
+ ?? execution.remoteBinary?.nilIfBlank
+ ?? installation.manifest.binary.name
+ var commandParts = [remoteBinary] + remoteArguments
+ let envFile = execution.remoteEnvFileConfigID
+ .flatMap { Self.configValue($0, installation: installation) }
+ let userCommand = Self.remoteShellCommand(commandParts: commandParts, envFile: envFile)
+ if let runAsOptionID = execution.runAsConfigID?.nilIfBlank,
+ let runAs = Self.configValue(runAsOptionID, installation: installation)
+ {
+ guard !runAs.hasPrefix("-"), !runAs.contains(where: { $0.isWhitespace }) else {
+ throw CrawlCommandRunnerError.invalidRemoteTarget(appID: installation.id, target: runAs)
+ }
+ commandParts = ["sudo", "-u", runAs, "-H", "--", "sh", "-lc", userCommand]
+ return ["--", target, commandParts.map(Self.shellQuoted).joined(separator: " ")]
+ }
+ if envFile?.nilIfBlank != nil {
+ return ["--", target, ["sh", "-lc", userCommand].map(Self.shellQuoted).joined(separator: " ")]
+ }
+ return ["--", target, commandParts.map(Self.shellQuoted).joined(separator: " ")]
+ }
+
+ static func commandOverride(for installation: CrawlAppInstallation, action: String) -> [String]? {
+ guard installation.id == BuiltInCrawlApps.birdclawID,
+ Self.xAccessPath(for: installation) == "birdclaw"
+ else { return nil }
+ switch action {
+ case "status", "doctor":
+ return ["auth", "status", "--json"]
+ case "search", "query":
+ return ["--json", "search", "tweets"]
+ default:
+ return nil
+ }
+ }
+
+ static func effectiveBinaryName(for installation: CrawlAppInstallation) -> String {
+ guard installation.id == BuiltInCrawlApps.birdclawID else {
+ return installation.manifest.binary.name
+ }
+ return Self.xAccessPath(for: installation) == "birdclaw" ? "birdclaw" : "bird"
+ }
+
+ static func configValue(_ optionID: String, installation: CrawlAppInstallation) -> String? {
+ if let value = installation.configValues[optionID]?.nilIfBlank {
+ return value
+ }
+ return installation.manifest.configOptions.first { $0.id == optionID }?.defaultValue?.nilIfBlank
+ }
+
+ private static func exactConfigTokenID(_ argument: String) -> String? {
+ guard argument.hasPrefix("{config:"), argument.hasSuffix("}") else { return nil }
+ let optionID = String(argument.dropFirst("{config:".count).dropLast())
+ return optionID.isEmpty ? nil : optionID
+ }
+
+ private static func interpolatedArgument(_ argument: String, installation: CrawlAppInstallation) throws -> String {
+ var value = argument
+ while let range = value.range(of: #"\{config:([A-Za-z0-9_.-]+)\}"#, options: .regularExpression) {
+ let token = String(value[range])
+ let optionID = String(token.dropFirst("{config:".count).dropLast())
+ guard let replacement = Self.configValue(optionID, installation: installation) else {
+ throw CrawlCommandRunnerError.missingRequiredConfig(appID: installation.id, optionID: optionID)
+ }
+ value.replaceSubrange(range, with: replacement)
+ }
+ return value
+ }
+
+ private static func remoteShellCommand(commandParts: [String], envFile: String?) -> String {
+ let command = commandParts.map(Self.shellQuoted).joined(separator: " ")
+ guard let envFile = envFile?.nilIfBlank else {
+ return "cd ~ && exec " + command
+ }
+ return "cd ~ && set -a && . \(Self.shellQuoted(envFile)) && set +a && exec \(command)"
+ }
+
+ private static func shellQuoted(_ value: String) -> String {
+ "'\(value.replacingOccurrences(of: "'", with: "'\\''"))'"
+ }
+
+ private static func xAccessPath(for installation: CrawlAppInstallation) -> String {
+ (Self.configValue("access_path", installation: installation) ?? "bird")
+ .trimmingCharacters(in: .whitespacesAndNewlines)
+ .lowercased()
+ }
+
+ private static func wacliSearchNeedsJoinedQuery(
+ installation: CrawlAppInstallation,
+ action: String,
+ extraArguments: [String])
+ -> Bool
+ {
+ guard !extraArguments.isEmpty,
+ action == "search" || action == "query",
+ Self.isWacliInstallation(installation)
+ else { return false }
+ return true
+ }
+
+ private static func isWacliInstallation(_ installation: CrawlAppInstallation) -> Bool {
+ installation.id == BuiltInCrawlApps.wacliID
+ || installation.id.rawValue.hasPrefix("wacli-")
+ || installation.manifest.binary.name == "wacli"
+ }
+
+ private static func wacliSearchArguments(_ extraArguments: [String]) -> [String] {
+ var queryParts: [String] = []
+ var optionArguments: [String] = []
+ var reachedOptions = false
+ for argument in extraArguments {
+ if argument.hasPrefix("-") {
+ reachedOptions = true
+ }
+ if reachedOptions {
+ optionArguments.append(argument)
+ } else {
+ queryParts.append(argument)
+ }
+ }
+ guard queryParts.count > 1 else { return extraArguments }
+ return [queryParts.joined(separator: " ")] + optionArguments
+ }
+
+ private static func gitcrawlQueryNeedsRepository(action: String, extraArguments: [String]) -> Bool {
+ (action == "query" || action == "search") && !extraArguments.isEmpty && !extraArguments.contains("--query")
+ }
+
+ private static func gitcrawlRefreshNeedsRepository(action: String, commandArguments: [String]) -> Bool {
+ guard action == "refresh" || action == "sync",
+ let command = commandArguments.first,
+ command == "refresh" || command == "sync"
+ else { return false }
+ return !commandArguments.dropFirst().contains { !$0.hasPrefix("-") && $0.contains("/") }
+ }
+}
diff --git a/Sources/CrawlBarCore/CrawlCommandRunnerError.swift b/Sources/CrawlBarCore/CrawlCommandRunnerError.swift
new file mode 100644
index 0000000..1ed4b9b
--- /dev/null
+++ b/Sources/CrawlBarCore/CrawlCommandRunnerError.swift
@@ -0,0 +1,24 @@
+import Foundation
+
+public enum CrawlCommandRunnerError: LocalizedError, Sendable {
+ case executableNotFound(String)
+ case commandUnavailable(appID: CrawlAppID, action: String)
+ case missingRequiredConfig(appID: CrawlAppID, optionID: String)
+ case invalidRemoteTarget(appID: CrawlAppID, target: String)
+ case timedOut(appID: CrawlAppID, action: String, seconds: Int)
+
+ public var errorDescription: String? {
+ switch self {
+ case let .executableNotFound(name):
+ "Could not find executable: \(name)"
+ case let .commandUnavailable(appID, action):
+ "\(appID.rawValue) does not expose a \(action) command"
+ case let .missingRequiredConfig(appID, optionID):
+ "\(appID.rawValue) is missing required config: \(optionID)"
+ case let .invalidRemoteTarget(appID, target):
+ "\(appID.rawValue) has an invalid SSH target: \(target)"
+ case let .timedOut(appID, action, seconds):
+ "\(appID.rawValue) \(action) timed out after \(seconds)s"
+ }
+ }
+}
diff --git a/Sources/CrawlBarCore/CrawlCommandRunnerProcess.swift b/Sources/CrawlBarCore/CrawlCommandRunnerProcess.swift
new file mode 100644
index 0000000..7198f1f
--- /dev/null
+++ b/Sources/CrawlBarCore/CrawlCommandRunnerProcess.swift
@@ -0,0 +1,81 @@
+import Foundation
+#if os(macOS)
+import Darwin
+#elseif os(Linux)
+import Glibc
+#endif
+
+extension CrawlCommandRunner {
+ func runProcess(
+ appID: CrawlAppID,
+ action: String,
+ executablePath: String,
+ arguments: [String],
+ environment: [String: String],
+ timeoutSeconds: TimeInterval)
+ throws -> CrawlCommandResult
+ {
+ let startedAt = Date()
+ let tempDirectory = self.fileManager.temporaryDirectory
+ .appendingPathComponent("crawlbar-\(UUID().uuidString)", isDirectory: true)
+ try self.fileManager.createDirectory(at: tempDirectory, withIntermediateDirectories: true)
+ defer { try? self.fileManager.removeItem(at: tempDirectory) }
+
+ let stdoutURL = tempDirectory.appendingPathComponent("stdout.log")
+ let stderrURL = tempDirectory.appendingPathComponent("stderr.log")
+ self.fileManager.createFile(atPath: stdoutURL.path, contents: nil)
+ self.fileManager.createFile(atPath: stderrURL.path, contents: nil)
+
+ let stdoutHandle = try FileHandle(forWritingTo: stdoutURL)
+ let stderrHandle = try FileHandle(forWritingTo: stderrURL)
+ defer {
+ try? stdoutHandle.close()
+ try? stderrHandle.close()
+ }
+
+ let process = Process()
+ process.executableURL = URL(fileURLWithPath: executablePath)
+ process.arguments = arguments
+ process.environment = environment
+ process.standardOutput = stdoutHandle
+ process.standardError = stderrHandle
+
+ let semaphore = DispatchSemaphore(value: 0)
+ process.terminationHandler = { _ in
+ semaphore.signal()
+ }
+
+ try process.run()
+ let waitResult = semaphore.wait(timeout: .now() + timeoutSeconds)
+ if waitResult == .timedOut {
+ process.terminate()
+ #if os(macOS) || os(Linux)
+ let pid = process.processIdentifier
+ DispatchQueue.global().asyncAfter(deadline: .now() + Self.timeoutTerminationGrace) {
+ if process.isRunning {
+ kill(pid, SIGKILL)
+ }
+ }
+ #endif
+ process.waitUntilExit()
+ throw CrawlCommandRunnerError.timedOut(
+ appID: appID,
+ action: action,
+ seconds: Int(timeoutSeconds))
+ }
+
+ try? stdoutHandle.synchronize()
+ try? stderrHandle.synchronize()
+
+ let stdout = try String(contentsOf: stdoutURL, encoding: .utf8)
+ let stderr = try String(contentsOf: stderrURL, encoding: .utf8)
+ return CrawlCommandResult(
+ appID: appID,
+ action: action,
+ exitCode: process.terminationStatus,
+ stdout: self.redactor.redact(stdout),
+ stderr: self.redactor.redact(stderr),
+ startedAt: startedAt,
+ finishedAt: Date())
+ }
+}
diff --git a/Sources/CrawlBarCore/CrawlExecutableResolver.swift b/Sources/CrawlBarCore/CrawlExecutableResolver.swift
new file mode 100644
index 0000000..23fa921
--- /dev/null
+++ b/Sources/CrawlBarCore/CrawlExecutableResolver.swift
@@ -0,0 +1,59 @@
+import Foundation
+
+public final class CrawlExecutableResolver: @unchecked Sendable {
+ private let fileManager: FileManager
+ private let environment: [String: String]
+ private let lock = NSLock()
+ private var resolvedExecutables: [String: String] = [:]
+
+ public init(fileManager: FileManager = .default, environment: [String: String] = ProcessInfo.processInfo.environment) {
+ self.fileManager = fileManager
+ self.environment = CrawlProcessEnvironment.normalized(environment)
+ }
+
+ public func resolve(_ requestedPathOrName: String) -> String? {
+ self.lock.lock()
+ if let cached = self.resolvedExecutables[requestedPathOrName] {
+ self.lock.unlock()
+ if self.isExecutable(cached) {
+ return cached
+ }
+ self.lock.lock()
+ self.resolvedExecutables.removeValue(forKey: requestedPathOrName)
+ self.lock.unlock()
+ } else {
+ self.lock.unlock()
+ }
+
+ let resolved = self.resolveUncached(requestedPathOrName)
+ self.lock.lock()
+ if let resolved {
+ self.resolvedExecutables[requestedPathOrName] = resolved
+ } else {
+ self.resolvedExecutables.removeValue(forKey: requestedPathOrName)
+ }
+ self.lock.unlock()
+ return resolved
+ }
+
+ private func resolveUncached(_ requestedPathOrName: String) -> String? {
+ let expanded = PathExpander.expandHome(requestedPathOrName)
+ if expanded.contains("/") {
+ return self.isExecutable(expanded) ? expanded : nil
+ }
+
+ for entry in CrawlProcessEnvironment.pathEntries(environment: self.environment) {
+ let candidate = URL(fileURLWithPath: entry)
+ .appendingPathComponent(expanded)
+ .path
+ if self.isExecutable(candidate) {
+ return candidate
+ }
+ }
+ return nil
+ }
+
+ private func isExecutable(_ path: String) -> Bool {
+ self.fileManager.isExecutableFile(atPath: path)
+ }
+}
diff --git a/Sources/CrawlBarCore/Manifest/CrawlAppCapabilities.swift b/Sources/CrawlBarCore/Manifest/CrawlAppCapabilities.swift
new file mode 100644
index 0000000..7ad4ec8
--- /dev/null
+++ b/Sources/CrawlBarCore/Manifest/CrawlAppCapabilities.swift
@@ -0,0 +1,37 @@
+import Foundation
+
+public enum CrawlAppCapability: String, Codable, Equatable, Sendable, CaseIterable {
+ case status
+ case doctor
+ case refresh
+ case search
+ case publish
+ case subscribe
+ case update
+ case desktopCache = "desktop_cache"
+ case exportMarkdown = "export_markdown"
+ case exportDatabase = "export_database"
+ case remoteArchive = "remote_archive"
+ case cloudPublish = "cloud_publish"
+ case maintain
+}
+
+public enum CrawlQueryActionResolver {
+ public static func action(for manifest: CrawlAppManifest, queryArguments: [String]) -> String? {
+ if Self.queryLooksLikeSQL(queryArguments) {
+ return ["query", "sql"].first { manifest.commands[$0] != nil }
+ }
+ if manifest.commands["search"] != nil {
+ return "search"
+ }
+ if manifest.commands["query"] != nil {
+ return "query"
+ }
+ return nil
+ }
+
+ private static func queryLooksLikeSQL(_ arguments: [String]) -> Bool {
+ let query = arguments.joined(separator: " ").trimmingCharacters(in: .whitespacesAndNewlines).lowercased()
+ return ["select ", "with ", "pragma ", "explain "].contains { query.hasPrefix($0) }
+ }
+}
diff --git a/Sources/CrawlBarCore/Manifest/CrawlAppID.swift b/Sources/CrawlBarCore/Manifest/CrawlAppID.swift
new file mode 100644
index 0000000..d0bb282
--- /dev/null
+++ b/Sources/CrawlBarCore/Manifest/CrawlAppID.swift
@@ -0,0 +1,17 @@
+import Foundation
+
+public struct CrawlAppID: RawRepresentable, Codable, Hashable, Sendable, Comparable, CustomStringConvertible {
+ public var rawValue: String
+
+ public init(rawValue: String) {
+ self.rawValue = rawValue
+ }
+
+ public var description: String {
+ self.rawValue
+ }
+
+ public static func < (lhs: CrawlAppID, rhs: CrawlAppID) -> Bool {
+ lhs.rawValue < rhs.rawValue
+ }
+}
diff --git a/Sources/CrawlBarCore/Manifest/CrawlAppManifest.swift b/Sources/CrawlBarCore/Manifest/CrawlAppManifest.swift
new file mode 100644
index 0000000..121ccce
--- /dev/null
+++ b/Sources/CrawlBarCore/Manifest/CrawlAppManifest.swift
@@ -0,0 +1,250 @@
+import Foundation
+
+public struct CrawlAppManifest: Codable, Equatable, Sendable, Identifiable {
+ public var schemaVersion: Int
+ public var id: CrawlAppID
+ public var displayName: String
+ public var description: String
+ public var availability: CrawlAppManifest.Availability
+ public var binary: CrawlAppManifest.Binary
+ public var execution: CrawlAppManifest.Execution?
+ public var branding: CrawlAppManifest.Branding
+ public var paths: CrawlAppManifest.Paths
+ public var commands: [String: [String]]
+ public var capabilities: [CrawlAppCapability]
+ public var statusRequiresSecrets: Bool?
+ public var privacy: CrawlAppManifest.Privacy
+ public var configOptions: [CrawlAppManifest.ConfigOption]
+ public var configSections: [CrawlAppManifest.ConfigSection]
+ public var install: CrawlAppManifest.Install?
+
+ public init(
+ schemaVersion: Int = 1,
+ id: CrawlAppID,
+ displayName: String,
+ description: String,
+ availability: CrawlAppManifest.Availability = .available,
+ binary: CrawlAppManifest.Binary,
+ execution: CrawlAppManifest.Execution? = nil,
+ branding: CrawlAppManifest.Branding,
+ paths: CrawlAppManifest.Paths,
+ commands: [String: [String]],
+ capabilities: [CrawlAppCapability],
+ statusRequiresSecrets: Bool? = nil,
+ privacy: CrawlAppManifest.Privacy = CrawlAppManifest.Privacy(),
+ configOptions: [CrawlAppManifest.ConfigOption] = [],
+ configSections: [CrawlAppManifest.ConfigSection] = [],
+ install: CrawlAppManifest.Install? = nil)
+ {
+ self.schemaVersion = schemaVersion
+ self.id = id
+ self.displayName = displayName
+ self.description = description
+ self.availability = availability
+ self.binary = binary
+ self.execution = execution
+ self.branding = branding
+ self.paths = paths
+ self.commands = commands
+ self.capabilities = capabilities
+ self.statusRequiresSecrets = statusRequiresSecrets
+ self.privacy = privacy
+ self.configOptions = configOptions
+ self.configSections = configSections
+ self.install = install
+ }
+
+ private enum CodingKeys: String, CodingKey {
+ case schemaVersion = "schema_version"
+ case id
+ case displayName = "display_name"
+ case description
+ case availability
+ case binary
+ case execution
+ case branding
+ case paths
+ case commands
+ case capabilities
+ case statusRequiresSecrets = "status_requires_secrets"
+ case privacy
+ case configOptions = "config_options"
+ case configSections = "config_sections"
+ case install
+ }
+
+ public init(from decoder: Decoder) throws {
+ let container = try decoder.container(keyedBy: CodingKeys.self)
+ self.schemaVersion = Self.decodeSchemaVersion(from: container)
+ self.id = try container.decode(CrawlAppID.self, forKey: .id)
+ self.displayName = try container.decode(String.self, forKey: .displayName)
+ self.description = try container.decode(String.self, forKey: .description)
+ self.availability = try container.decodeIfPresent(CrawlAppManifest.Availability.self, forKey: .availability) ?? .available
+ self.binary = try container.decode(CrawlAppManifest.Binary.self, forKey: .binary)
+ self.execution = try container.decodeIfPresent(CrawlAppManifest.Execution.self, forKey: .execution)
+ self.branding = try container.decode(CrawlAppManifest.Branding.self, forKey: .branding)
+ self.paths = try container.decode(CrawlAppManifest.Paths.self, forKey: .paths)
+ self.commands = try Self.decodeCommands(from: container, binaryName: self.binary.name)
+ self.capabilities = Self.decodeCapabilities(from: container, commands: self.commands)
+ self.statusRequiresSecrets = try container.decodeIfPresent(Bool.self, forKey: .statusRequiresSecrets)
+ self.privacy = try container.decodeIfPresent(CrawlAppManifest.Privacy.self, forKey: .privacy) ?? CrawlAppManifest.Privacy()
+ self.configOptions = try container.decodeIfPresent([CrawlAppManifest.ConfigOption].self, forKey: .configOptions) ?? []
+ self.configSections = try container.decodeIfPresent([CrawlAppManifest.ConfigSection].self, forKey: .configSections) ?? []
+ self.install = try container.decodeIfPresent(CrawlAppManifest.Install.self, forKey: .install)
+ }
+
+ public var needsSecretsForStatus: Bool {
+ if let statusRequiresSecrets {
+ return statusRequiresSecrets
+ }
+ return self.commands["status"] != nil && self.configOptions.contains { option in
+ option.kind == .secret && option.envVar?.nilIfBlank != nil
+ }
+ }
+
+ public func executionKind(configValues: [String: String]) -> CrawlAppManifest.ExecutionKind {
+ guard let execution else { return .local }
+ guard let modeOptionID = execution.kindConfigID?.nilIfBlank else {
+ return execution.kind
+ }
+ let configuredMode = configValues[modeOptionID]?.nilIfBlank
+ ?? self.configOptions.first { $0.id == modeOptionID }?.defaultValue?.nilIfBlank
+ guard let configuredMode else { return execution.kind }
+ switch configuredMode.trimmingCharacters(in: .whitespacesAndNewlines).lowercased() {
+ case "local":
+ return .local
+ case "remote", "ssh":
+ return .ssh
+ default:
+ return execution.kind
+ }
+ }
+
+ private struct CommandEnvelope: Decodable {
+ var argv: [String]
+ }
+
+ private static func decodeSchemaVersion(from container: KeyedDecodingContainer) -> Int {
+ if let int = try? container.decode(Int.self, forKey: .schemaVersion) {
+ return int
+ }
+ guard let string = try? container.decode(String.self, forKey: .schemaVersion) else {
+ return 1
+ }
+ return Int(string) ?? 1
+ }
+
+ private static func decodeCommands(
+ from container: KeyedDecodingContainer,
+ binaryName: String)
+ throws -> [String: [String]]
+ {
+ if let commands = try? container.decode([String: [String]].self, forKey: .commands) {
+ return Self.normalizedCommands(commands)
+ }
+
+ let envelopes = try container.decode([String: CommandEnvelope].self, forKey: .commands)
+ let commands = envelopes.reduce(into: [String: [String]]()) { result, entry in
+ let arguments = entry.value.argv.first == binaryName
+ ? Array(entry.value.argv.dropFirst())
+ : entry.value.argv
+ result[entry.key] = Self.normalizedCommandArguments(arguments, action: entry.key)
+ }
+ return Self.normalizedCommands(commands)
+ }
+
+ private static func normalizedCommandArguments(_ arguments: [String], action: String) -> [String] {
+ guard (action == "sql" || action == "query"),
+ let last = arguments.last?.nilIfBlank,
+ Self.isSampleSQL(last)
+ else {
+ return arguments
+ }
+ return Array(arguments.dropLast())
+ }
+
+ private static func isSampleSQL(_ value: String) -> Bool {
+ let lowercased = value.lowercased()
+ return lowercased.hasPrefix("select ") || lowercased.hasPrefix("with ")
+ }
+
+ private static func normalizedCommands(_ commands: [String: [String]]) -> [String: [String]] {
+ var normalized = commands
+ if normalized["refresh"] == nil, let sync = normalized["sync"] {
+ normalized["refresh"] = sync
+ }
+ if normalized["desktop-cache-import"] == nil {
+ for alias in Self.desktopCacheCommandAliases where alias != "desktop-cache-import" {
+ if let command = normalized[alias] {
+ normalized["desktop-cache-import"] = command
+ break
+ }
+ }
+ }
+ return normalized
+ }
+
+ private static func decodeCapabilities(
+ from container: KeyedDecodingContainer,
+ commands: [String: [String]])
+ -> [CrawlAppCapability]
+ {
+ var capabilities: [CrawlAppCapability] = []
+ if let typed = try? container.decode([CrawlAppCapability].self, forKey: .capabilities) {
+ capabilities.append(contentsOf: typed)
+ } else if let rawValues = try? container.decode([String].self, forKey: .capabilities) {
+ capabilities.append(contentsOf: rawValues.flatMap(Self.capabilities(from:)))
+ }
+
+ for command in commands.keys.sorted() {
+ capabilities.append(contentsOf: Self.capabilities(from: command))
+ }
+
+ var seen = Set()
+ return capabilities.filter { seen.insert($0).inserted }
+ }
+
+ private static func capabilities(from rawValue: String) -> [CrawlAppCapability] {
+ switch rawValue {
+ case "status":
+ return [.status]
+ case "doctor":
+ return [.doctor]
+ case "refresh", "sync":
+ return [.refresh]
+ case "query", "search", "sql":
+ return [.search]
+ case "publish":
+ return [.publish]
+ case "subscribe":
+ return [.subscribe]
+ case "update":
+ return [.update]
+ case let value where Self.desktopCacheCommandAliases.contains(value):
+ return [.desktopCache]
+ case "markdown", "export", "export-md":
+ return [.exportMarkdown]
+ case "table-export", "export-db", "databases":
+ return [.exportDatabase]
+ case "maintain":
+ return [.maintain]
+ case "git-share":
+ return [.publish, .subscribe, .update]
+ case "remote", "remote-status", "remote-archives", "remote_archive":
+ return [.remoteArchive]
+ case "cloud", "cloud-publish", "cloud_publish":
+ return [.cloudPublish]
+ default:
+ return []
+ }
+ }
+
+ private static let desktopCacheCommandAliases = [
+ "desktop-cache-import",
+ "desktop-cache",
+ "desktop_cache",
+ "desktopcache",
+ "cache-import",
+ "tap",
+ ]
+}
diff --git a/Sources/CrawlBarCore/Manifest/CrawlAppManifestBinary.swift b/Sources/CrawlBarCore/Manifest/CrawlAppManifestBinary.swift
new file mode 100644
index 0000000..dca7675
--- /dev/null
+++ b/Sources/CrawlBarCore/Manifest/CrawlAppManifestBinary.swift
@@ -0,0 +1,23 @@
+import Foundation
+
+public extension CrawlAppManifest {
+ enum Availability: String, Codable, Equatable, Sendable {
+ case available
+ case comingSoon = "coming_soon"
+ }
+
+ struct Binary: Codable, Equatable, Sendable {
+ public var name: String
+ public var minVersion: String?
+
+ public init(name: String, minVersion: String? = nil) {
+ self.name = name
+ self.minVersion = minVersion
+ }
+
+ private enum CodingKeys: String, CodingKey {
+ case name
+ case minVersion = "min_version"
+ }
+ }
+}
diff --git a/Sources/CrawlBarCore/Manifest/CrawlAppManifestBranding.swift b/Sources/CrawlBarCore/Manifest/CrawlAppManifestBranding.swift
new file mode 100644
index 0000000..4360ae2
--- /dev/null
+++ b/Sources/CrawlBarCore/Manifest/CrawlAppManifestBranding.swift
@@ -0,0 +1,29 @@
+import Foundation
+
+public extension CrawlAppManifest {
+ struct Branding: Codable, Equatable, Sendable {
+ public var symbolName: String
+ public var accentColor: String
+ public var iconPath: String?
+ public var bundleIdentifier: String?
+
+ public init(
+ symbolName: String,
+ accentColor: String,
+ iconPath: String? = nil,
+ bundleIdentifier: String? = nil)
+ {
+ self.symbolName = symbolName
+ self.accentColor = accentColor
+ self.iconPath = iconPath
+ self.bundleIdentifier = bundleIdentifier
+ }
+
+ private enum CodingKeys: String, CodingKey {
+ case symbolName = "symbol_name"
+ case accentColor = "accent_color"
+ case iconPath = "icon_path"
+ case bundleIdentifier = "bundle_identifier"
+ }
+ }
+}
diff --git a/Sources/CrawlBarCore/Manifest/CrawlAppManifestConfig.swift b/Sources/CrawlBarCore/Manifest/CrawlAppManifestConfig.swift
new file mode 100644
index 0000000..733fa95
--- /dev/null
+++ b/Sources/CrawlBarCore/Manifest/CrawlAppManifestConfig.swift
@@ -0,0 +1,91 @@
+import Foundation
+
+public extension CrawlAppManifest {
+ enum ConfigOptionKind: String, Codable, Equatable, Sendable {
+ case string
+ case secret
+ case boolean
+ case number
+ case choice
+ }
+
+ struct ConfigOption: Codable, Equatable, Sendable, Identifiable {
+ public var id: String
+ public var label: String
+ public var kind: ConfigOptionKind
+ public var help: String?
+ public var placeholder: String?
+ public var defaultValue: String?
+ public var choices: [String]
+ public var envVar: String?
+ public var configKey: String?
+
+ public init(
+ id: String,
+ label: String,
+ kind: ConfigOptionKind = .string,
+ help: String? = nil,
+ placeholder: String? = nil,
+ defaultValue: String? = nil,
+ choices: [String] = [],
+ envVar: String? = nil,
+ configKey: String? = nil)
+ {
+ self.id = id
+ self.label = label
+ self.kind = kind
+ self.help = help
+ self.placeholder = placeholder
+ self.defaultValue = defaultValue
+ self.choices = choices
+ self.envVar = envVar
+ self.configKey = configKey
+ }
+
+ private enum CodingKeys: String, CodingKey {
+ case id
+ case label
+ case kind
+ case help
+ case placeholder
+ case defaultValue = "default_value"
+ case choices
+ case envVar = "env_var"
+ case configKey = "config_key"
+ }
+
+ public init(from decoder: Decoder) throws {
+ let container = try decoder.container(keyedBy: CodingKeys.self)
+ self.id = try container.decode(String.self, forKey: .id)
+ self.label = try container.decodeIfPresent(String.self, forKey: .label) ?? self.id
+ self.kind = try container.decodeIfPresent(ConfigOptionKind.self, forKey: .kind) ?? .string
+ self.help = try container.decodeIfPresent(String.self, forKey: .help)
+ self.placeholder = try container.decodeIfPresent(String.self, forKey: .placeholder)
+ self.defaultValue = try container.decodeIfPresent(String.self, forKey: .defaultValue)
+ self.choices = try container.decodeIfPresent([String].self, forKey: .choices) ?? []
+ self.envVar = try container.decodeIfPresent(String.self, forKey: .envVar)
+ self.configKey = try container.decodeIfPresent(String.self, forKey: .configKey)
+ }
+ }
+
+ struct ConfigSection: Codable, Equatable, Sendable, Identifiable {
+ public var id: String
+ public var title: String
+ public var caption: String?
+ public var optionIDs: [String]
+
+ public init(id: String, title: String, caption: String? = nil, optionIDs: [String]) {
+ self.id = id
+ self.title = title
+ self.caption = caption
+ self.optionIDs = optionIDs
+ }
+
+ private enum CodingKeys: String, CodingKey {
+ case id
+ case title
+ case caption
+ case optionIDs = "option_ids"
+ }
+ }
+}
diff --git a/Sources/CrawlBarCore/Manifest/CrawlAppManifestExecution.swift b/Sources/CrawlBarCore/Manifest/CrawlAppManifestExecution.swift
new file mode 100644
index 0000000..07cd02c
--- /dev/null
+++ b/Sources/CrawlBarCore/Manifest/CrawlAppManifestExecution.swift
@@ -0,0 +1,42 @@
+import Foundation
+
+public extension CrawlAppManifest {
+ enum ExecutionKind: String, Codable, Equatable, Sendable {
+ case local
+ case ssh
+ }
+
+ struct Execution: Codable, Equatable, Sendable {
+ public var kind: ExecutionKind
+ public var kindConfigID: String?
+ public var targetConfigID: String?
+ public var runAsConfigID: String?
+ public var remoteEnvFileConfigID: String?
+ public var remoteBinary: String?
+
+ public init(
+ kind: ExecutionKind = .local,
+ kindConfigID: String? = nil,
+ targetConfigID: String? = nil,
+ runAsConfigID: String? = nil,
+ remoteEnvFileConfigID: String? = nil,
+ remoteBinary: String? = nil)
+ {
+ self.kind = kind
+ self.kindConfigID = kindConfigID
+ self.targetConfigID = targetConfigID
+ self.runAsConfigID = runAsConfigID
+ self.remoteEnvFileConfigID = remoteEnvFileConfigID
+ self.remoteBinary = remoteBinary
+ }
+
+ private enum CodingKeys: String, CodingKey {
+ case kind
+ case kindConfigID = "kind_config_id"
+ case targetConfigID = "target_config_id"
+ case runAsConfigID = "run_as_config_id"
+ case remoteEnvFileConfigID = "remote_env_file_config_id"
+ case remoteBinary = "remote_binary"
+ }
+ }
+}
diff --git a/Sources/CrawlBarCore/Manifest/CrawlAppManifestInstall.swift b/Sources/CrawlBarCore/Manifest/CrawlAppManifestInstall.swift
new file mode 100644
index 0000000..1a431e6
--- /dev/null
+++ b/Sources/CrawlBarCore/Manifest/CrawlAppManifestInstall.swift
@@ -0,0 +1,17 @@
+import Foundation
+
+public extension CrawlAppManifest {
+ enum InstallMethod: String, Codable, Equatable, Sendable {
+ case homebrew
+ }
+
+ struct Install: Codable, Equatable, Sendable {
+ public var method: InstallMethod
+ public var package: String
+
+ public init(method: InstallMethod, package: String) {
+ self.method = method
+ self.package = package
+ }
+ }
+}
diff --git a/Sources/CrawlBarCore/Manifest/CrawlAppManifestPaths.swift b/Sources/CrawlBarCore/Manifest/CrawlAppManifestPaths.swift
new file mode 100644
index 0000000..f9c91ee
--- /dev/null
+++ b/Sources/CrawlBarCore/Manifest/CrawlAppManifestPaths.swift
@@ -0,0 +1,37 @@
+import Foundation
+
+public extension CrawlAppManifest {
+ struct Paths: Codable, Equatable, Sendable {
+ public var defaultConfig: String?
+ public var configEnv: String?
+ public var defaultDatabase: String?
+ public var defaultCache: String?
+ public var defaultLogs: String?
+ public var defaultShare: String?
+
+ public init(
+ defaultConfig: String? = nil,
+ configEnv: String? = nil,
+ defaultDatabase: String? = nil,
+ defaultCache: String? = nil,
+ defaultLogs: String? = nil,
+ defaultShare: String? = nil)
+ {
+ self.defaultConfig = defaultConfig
+ self.configEnv = configEnv
+ self.defaultDatabase = defaultDatabase
+ self.defaultCache = defaultCache
+ self.defaultLogs = defaultLogs
+ self.defaultShare = defaultShare
+ }
+
+ private enum CodingKeys: String, CodingKey {
+ case defaultConfig = "default_config"
+ case configEnv = "config_env"
+ case defaultDatabase = "default_database"
+ case defaultCache = "default_cache"
+ case defaultLogs = "default_logs"
+ case defaultShare = "default_share"
+ }
+ }
+}
diff --git a/Sources/CrawlBarCore/Manifest/CrawlAppManifestPrivacy.swift b/Sources/CrawlBarCore/Manifest/CrawlAppManifestPrivacy.swift
new file mode 100644
index 0000000..0cfdee2
--- /dev/null
+++ b/Sources/CrawlBarCore/Manifest/CrawlAppManifestPrivacy.swift
@@ -0,0 +1,32 @@
+import Foundation
+
+public extension CrawlAppManifest {
+ struct Privacy: Codable, Equatable, Sendable {
+ public var containsPrivateMessages: Bool
+ public var exportsSecrets: Bool
+ public var localOnlyScopes: [String]
+
+ public init(
+ containsPrivateMessages: Bool = false,
+ exportsSecrets: Bool = false,
+ localOnlyScopes: [String] = [])
+ {
+ self.containsPrivateMessages = containsPrivateMessages
+ self.exportsSecrets = exportsSecrets
+ self.localOnlyScopes = localOnlyScopes
+ }
+
+ private enum CodingKeys: String, CodingKey {
+ case containsPrivateMessages = "contains_private_messages"
+ case exportsSecrets = "exports_secrets"
+ case localOnlyScopes = "local_only_scopes"
+ }
+
+ public init(from decoder: Decoder) throws {
+ let container = try decoder.container(keyedBy: CodingKeys.self)
+ self.containsPrivateMessages = try container.decodeIfPresent(Bool.self, forKey: .containsPrivateMessages) ?? false
+ self.exportsSecrets = try container.decodeIfPresent(Bool.self, forKey: .exportsSecrets) ?? false
+ self.localOnlyScopes = try container.decodeIfPresent([String].self, forKey: .localOnlyScopes) ?? []
+ }
+ }
+}
diff --git a/Sources/CrawlBarCore/Models.swift b/Sources/CrawlBarCore/Models.swift
deleted file mode 100644
index de04b64..0000000
--- a/Sources/CrawlBarCore/Models.swift
+++ /dev/null
@@ -1,1129 +0,0 @@
-import Foundation
-
-public struct CrawlAppID: RawRepresentable, Codable, Hashable, Sendable, Comparable, CustomStringConvertible {
- public var rawValue: String
-
- public init(rawValue: String) {
- self.rawValue = rawValue
- }
-
- public var description: String {
- self.rawValue
- }
-
- public static func < (lhs: CrawlAppID, rhs: CrawlAppID) -> Bool {
- lhs.rawValue < rhs.rawValue
- }
-}
-
-public struct CrawlAppManifest: Codable, Equatable, Sendable, Identifiable {
- public enum Availability: String, Codable, Equatable, Sendable {
- case available
- case comingSoon = "coming_soon"
- }
-
- public struct Binary: Codable, Equatable, Sendable {
- public var name: String
- public var minVersion: String?
-
- public init(name: String, minVersion: String? = nil) {
- self.name = name
- self.minVersion = minVersion
- }
-
- private enum CodingKeys: String, CodingKey {
- case name
- case minVersion = "min_version"
- }
- }
-
- public enum ExecutionKind: String, Codable, Equatable, Sendable {
- case local
- case ssh
- }
-
- public struct Execution: Codable, Equatable, Sendable {
- public var kind: ExecutionKind
- public var kindConfigID: String?
- public var targetConfigID: String?
- public var runAsConfigID: String?
- public var remoteEnvFileConfigID: String?
- public var remoteBinary: String?
-
- public init(
- kind: ExecutionKind = .local,
- kindConfigID: String? = nil,
- targetConfigID: String? = nil,
- runAsConfigID: String? = nil,
- remoteEnvFileConfigID: String? = nil,
- remoteBinary: String? = nil)
- {
- self.kind = kind
- self.kindConfigID = kindConfigID
- self.targetConfigID = targetConfigID
- self.runAsConfigID = runAsConfigID
- self.remoteEnvFileConfigID = remoteEnvFileConfigID
- self.remoteBinary = remoteBinary
- }
-
- private enum CodingKeys: String, CodingKey {
- case kind
- case kindConfigID = "kind_config_id"
- case targetConfigID = "target_config_id"
- case runAsConfigID = "run_as_config_id"
- case remoteEnvFileConfigID = "remote_env_file_config_id"
- case remoteBinary = "remote_binary"
- }
- }
-
- public struct Branding: Codable, Equatable, Sendable {
- public var symbolName: String
- public var accentColor: String
- public var iconPath: String?
- public var bundleIdentifier: String?
-
- public init(
- symbolName: String,
- accentColor: String,
- iconPath: String? = nil,
- bundleIdentifier: String? = nil)
- {
- self.symbolName = symbolName
- self.accentColor = accentColor
- self.iconPath = iconPath
- self.bundleIdentifier = bundleIdentifier
- }
-
- private enum CodingKeys: String, CodingKey {
- case symbolName = "symbol_name"
- case accentColor = "accent_color"
- case iconPath = "icon_path"
- case bundleIdentifier = "bundle_identifier"
- }
- }
-
- public struct Paths: Codable, Equatable, Sendable {
- public var defaultConfig: String?
- public var configEnv: String?
- public var defaultDatabase: String?
- public var defaultCache: String?
- public var defaultLogs: String?
- public var defaultShare: String?
-
- public init(
- defaultConfig: String? = nil,
- configEnv: String? = nil,
- defaultDatabase: String? = nil,
- defaultCache: String? = nil,
- defaultLogs: String? = nil,
- defaultShare: String? = nil)
- {
- self.defaultConfig = defaultConfig
- self.configEnv = configEnv
- self.defaultDatabase = defaultDatabase
- self.defaultCache = defaultCache
- self.defaultLogs = defaultLogs
- self.defaultShare = defaultShare
- }
-
- private enum CodingKeys: String, CodingKey {
- case defaultConfig = "default_config"
- case configEnv = "config_env"
- case defaultDatabase = "default_database"
- case defaultCache = "default_cache"
- case defaultLogs = "default_logs"
- case defaultShare = "default_share"
- }
- }
-
- public struct Privacy: Codable, Equatable, Sendable {
- public var containsPrivateMessages: Bool
- public var exportsSecrets: Bool
- public var localOnlyScopes: [String]
-
- public init(
- containsPrivateMessages: Bool = false,
- exportsSecrets: Bool = false,
- localOnlyScopes: [String] = [])
- {
- self.containsPrivateMessages = containsPrivateMessages
- self.exportsSecrets = exportsSecrets
- self.localOnlyScopes = localOnlyScopes
- }
-
- private enum CodingKeys: String, CodingKey {
- case containsPrivateMessages = "contains_private_messages"
- case exportsSecrets = "exports_secrets"
- case localOnlyScopes = "local_only_scopes"
- }
-
- public init(from decoder: Decoder) throws {
- let container = try decoder.container(keyedBy: CodingKeys.self)
- self.containsPrivateMessages = try container.decodeIfPresent(Bool.self, forKey: .containsPrivateMessages) ?? false
- self.exportsSecrets = try container.decodeIfPresent(Bool.self, forKey: .exportsSecrets) ?? false
- self.localOnlyScopes = try container.decodeIfPresent([String].self, forKey: .localOnlyScopes) ?? []
- }
- }
-
- public enum InstallMethod: String, Codable, Equatable, Sendable {
- case homebrew
- }
-
- public struct Install: Codable, Equatable, Sendable {
- public var method: InstallMethod
- public var package: String
-
- public init(method: InstallMethod, package: String) {
- self.method = method
- self.package = package
- }
- }
-
- public enum ConfigOptionKind: String, Codable, Equatable, Sendable {
- case string
- case secret
- case boolean
- case number
- case choice
- }
-
- public struct ConfigOption: Codable, Equatable, Sendable, Identifiable {
- public var id: String
- public var label: String
- public var kind: ConfigOptionKind
- public var help: String?
- public var placeholder: String?
- public var defaultValue: String?
- public var choices: [String]
- public var envVar: String?
- public var configKey: String?
-
- public init(
- id: String,
- label: String,
- kind: ConfigOptionKind = .string,
- help: String? = nil,
- placeholder: String? = nil,
- defaultValue: String? = nil,
- choices: [String] = [],
- envVar: String? = nil,
- configKey: String? = nil)
- {
- self.id = id
- self.label = label
- self.kind = kind
- self.help = help
- self.placeholder = placeholder
- self.defaultValue = defaultValue
- self.choices = choices
- self.envVar = envVar
- self.configKey = configKey
- }
-
- private enum CodingKeys: String, CodingKey {
- case id
- case label
- case kind
- case help
- case placeholder
- case defaultValue = "default_value"
- case choices
- case envVar = "env_var"
- case configKey = "config_key"
- }
-
- public init(from decoder: Decoder) throws {
- let container = try decoder.container(keyedBy: CodingKeys.self)
- self.id = try container.decode(String.self, forKey: .id)
- self.label = try container.decodeIfPresent(String.self, forKey: .label) ?? self.id
- self.kind = try container.decodeIfPresent(ConfigOptionKind.self, forKey: .kind) ?? .string
- self.help = try container.decodeIfPresent(String.self, forKey: .help)
- self.placeholder = try container.decodeIfPresent(String.self, forKey: .placeholder)
- self.defaultValue = try container.decodeIfPresent(String.self, forKey: .defaultValue)
- self.choices = try container.decodeIfPresent([String].self, forKey: .choices) ?? []
- self.envVar = try container.decodeIfPresent(String.self, forKey: .envVar)
- self.configKey = try container.decodeIfPresent(String.self, forKey: .configKey)
- }
- }
-
- public struct ConfigSection: Codable, Equatable, Sendable, Identifiable {
- public var id: String
- public var title: String
- public var caption: String?
- public var optionIDs: [String]
-
- public init(id: String, title: String, caption: String? = nil, optionIDs: [String]) {
- self.id = id
- self.title = title
- self.caption = caption
- self.optionIDs = optionIDs
- }
-
- private enum CodingKeys: String, CodingKey {
- case id
- case title
- case caption
- case optionIDs = "option_ids"
- }
- }
-
- public var schemaVersion: Int
- public var id: CrawlAppID
- public var displayName: String
- public var description: String
- public var availability: Availability
- public var binary: Binary
- public var execution: Execution?
- public var branding: Branding
- public var paths: Paths
- public var commands: [String: [String]]
- public var capabilities: [CrawlAppCapability]
- public var statusRequiresSecrets: Bool?
- public var privacy: Privacy
- public var configOptions: [ConfigOption]
- public var configSections: [ConfigSection]
- public var install: Install?
-
- public init(
- schemaVersion: Int = 1,
- id: CrawlAppID,
- displayName: String,
- description: String,
- availability: Availability = .available,
- binary: Binary,
- execution: Execution? = nil,
- branding: Branding,
- paths: Paths,
- commands: [String: [String]],
- capabilities: [CrawlAppCapability],
- statusRequiresSecrets: Bool? = nil,
- privacy: Privacy = Privacy(),
- configOptions: [ConfigOption] = [],
- configSections: [ConfigSection] = [],
- install: Install? = nil)
- {
- self.schemaVersion = schemaVersion
- self.id = id
- self.displayName = displayName
- self.description = description
- self.availability = availability
- self.binary = binary
- self.execution = execution
- self.branding = branding
- self.paths = paths
- self.commands = commands
- self.capabilities = capabilities
- self.statusRequiresSecrets = statusRequiresSecrets
- self.privacy = privacy
- self.configOptions = configOptions
- self.configSections = configSections
- self.install = install
- }
-
- private enum CodingKeys: String, CodingKey {
- case schemaVersion = "schema_version"
- case id
- case displayName = "display_name"
- case description
- case availability
- case binary
- case execution
- case branding
- case paths
- case commands
- case capabilities
- case statusRequiresSecrets = "status_requires_secrets"
- case privacy
- case configOptions = "config_options"
- case configSections = "config_sections"
- case install
- }
-
- public init(from decoder: Decoder) throws {
- let container = try decoder.container(keyedBy: CodingKeys.self)
- self.schemaVersion = Self.decodeSchemaVersion(from: container)
- self.id = try container.decode(CrawlAppID.self, forKey: .id)
- self.displayName = try container.decode(String.self, forKey: .displayName)
- self.description = try container.decode(String.self, forKey: .description)
- self.availability = try container.decodeIfPresent(Availability.self, forKey: .availability) ?? .available
- self.binary = try container.decode(Binary.self, forKey: .binary)
- self.execution = try container.decodeIfPresent(Execution.self, forKey: .execution)
- self.branding = try container.decode(Branding.self, forKey: .branding)
- self.paths = try container.decode(Paths.self, forKey: .paths)
- self.commands = try Self.decodeCommands(from: container, binaryName: self.binary.name)
- self.capabilities = Self.decodeCapabilities(from: container, commands: self.commands)
- self.statusRequiresSecrets = try container.decodeIfPresent(Bool.self, forKey: .statusRequiresSecrets)
- self.privacy = try container.decodeIfPresent(Privacy.self, forKey: .privacy) ?? Privacy()
- self.configOptions = try container.decodeIfPresent([ConfigOption].self, forKey: .configOptions) ?? []
- self.configSections = try container.decodeIfPresent([ConfigSection].self, forKey: .configSections) ?? []
- self.install = try container.decodeIfPresent(Install.self, forKey: .install)
- }
-
- public var needsSecretsForStatus: Bool {
- if let statusRequiresSecrets {
- return statusRequiresSecrets
- }
- return self.commands["status"] != nil && self.configOptions.contains { option in
- option.kind == .secret && option.envVar?.nilIfBlank != nil
- }
- }
-
- public func executionKind(configValues: [String: String]) -> ExecutionKind {
- guard let execution else { return .local }
- guard let modeOptionID = execution.kindConfigID?.nilIfBlank else {
- return execution.kind
- }
- let configuredMode = configValues[modeOptionID]?.nilIfBlank
- ?? self.configOptions.first { $0.id == modeOptionID }?.defaultValue?.nilIfBlank
- guard let configuredMode else { return execution.kind }
- switch configuredMode.trimmingCharacters(in: .whitespacesAndNewlines).lowercased() {
- case "local":
- return .local
- case "remote", "ssh":
- return .ssh
- default:
- return execution.kind
- }
- }
-
- private struct CommandEnvelope: Decodable {
- var argv: [String]
- }
-
- private static func decodeSchemaVersion(from container: KeyedDecodingContainer) -> Int {
- if let int = try? container.decode(Int.self, forKey: .schemaVersion) {
- return int
- }
- guard let string = try? container.decode(String.self, forKey: .schemaVersion) else {
- return 1
- }
- return Int(string) ?? 1
- }
-
- private static func decodeCommands(
- from container: KeyedDecodingContainer,
- binaryName: String)
- throws -> [String: [String]]
- {
- if let commands = try? container.decode([String: [String]].self, forKey: .commands) {
- return Self.normalizedCommands(commands)
- }
-
- let envelopes = try container.decode([String: CommandEnvelope].self, forKey: .commands)
- let commands = envelopes.reduce(into: [String: [String]]()) { result, entry in
- let arguments = entry.value.argv.first == binaryName
- ? Array(entry.value.argv.dropFirst())
- : entry.value.argv
- result[entry.key] = Self.normalizedCommandArguments(arguments, action: entry.key)
- }
- return Self.normalizedCommands(commands)
- }
-
- private static func normalizedCommandArguments(_ arguments: [String], action: String) -> [String] {
- guard (action == "sql" || action == "query"),
- let last = arguments.last?.nilIfBlank,
- Self.isSampleSQL(last)
- else {
- return arguments
- }
- return Array(arguments.dropLast())
- }
-
- private static func isSampleSQL(_ value: String) -> Bool {
- let lowercased = value.lowercased()
- return lowercased.hasPrefix("select ") || lowercased.hasPrefix("with ")
- }
-
- private static func normalizedCommands(_ commands: [String: [String]]) -> [String: [String]] {
- var normalized = commands
- if normalized["refresh"] == nil, let sync = normalized["sync"] {
- normalized["refresh"] = sync
- }
- if normalized["desktop-cache-import"] == nil {
- for alias in Self.desktopCacheCommandAliases where alias != "desktop-cache-import" {
- if let command = normalized[alias] {
- normalized["desktop-cache-import"] = command
- break
- }
- }
- }
- return normalized
- }
-
- private static func decodeCapabilities(
- from container: KeyedDecodingContainer,
- commands: [String: [String]])
- -> [CrawlAppCapability]
- {
- var capabilities: [CrawlAppCapability] = []
- if let typed = try? container.decode([CrawlAppCapability].self, forKey: .capabilities) {
- capabilities.append(contentsOf: typed)
- } else if let rawValues = try? container.decode([String].self, forKey: .capabilities) {
- capabilities.append(contentsOf: rawValues.flatMap(Self.capabilities(from:)))
- }
-
- for command in commands.keys.sorted() {
- capabilities.append(contentsOf: Self.capabilities(from: command))
- }
-
- var seen = Set()
- return capabilities.filter { seen.insert($0).inserted }
- }
-
- private static func capabilities(from rawValue: String) -> [CrawlAppCapability] {
- switch rawValue {
- case "status":
- return [.status]
- case "doctor":
- return [.doctor]
- case "refresh", "sync":
- return [.refresh]
- case "query", "search", "sql":
- return [.search]
- case "publish":
- return [.publish]
- case "subscribe":
- return [.subscribe]
- case "update":
- return [.update]
- case let value where Self.desktopCacheCommandAliases.contains(value):
- return [.desktopCache]
- case "markdown", "export", "export-md":
- return [.exportMarkdown]
- case "table-export", "export-db", "databases":
- return [.exportDatabase]
- case "maintain":
- return [.maintain]
- case "git-share":
- return [.publish, .subscribe, .update]
- case "remote", "remote-status", "remote-archives", "remote_archive":
- return [.remoteArchive]
- case "cloud", "cloud-publish", "cloud_publish":
- return [.cloudPublish]
- default:
- return []
- }
- }
-
- private static let desktopCacheCommandAliases = [
- "desktop-cache-import",
- "desktop-cache",
- "desktop_cache",
- "desktopcache",
- "cache-import",
- "tap",
- ]
-}
-
-public enum CrawlAppCapability: String, Codable, Equatable, Sendable, CaseIterable {
- case status
- case doctor
- case refresh
- case search
- case publish
- case subscribe
- case update
- case desktopCache = "desktop_cache"
- case exportMarkdown = "export_markdown"
- case exportDatabase = "export_database"
- case remoteArchive = "remote_archive"
- case cloudPublish = "cloud_publish"
- case maintain
-}
-
-public enum CrawlQueryActionResolver {
- public static func action(for manifest: CrawlAppManifest, queryArguments: [String]) -> String? {
- if Self.queryLooksLikeSQL(queryArguments) {
- return ["query", "sql"].first { manifest.commands[$0] != nil }
- }
- if manifest.commands["search"] != nil {
- return "search"
- }
- if manifest.commands["query"] != nil {
- return "query"
- }
- return nil
- }
-
- private static func queryLooksLikeSQL(_ arguments: [String]) -> Bool {
- let query = arguments.joined(separator: " ").trimmingCharacters(in: .whitespacesAndNewlines).lowercased()
- return ["select ", "with ", "pragma ", "explain "].contains { query.hasPrefix($0) }
- }
-}
-
-public enum CrawlAppState: String, Codable, Equatable, Sendable {
- case current
- case stale
- case syncing
- case needsConfig = "needs_config"
- case needsAuth = "needs_auth"
- case error
- case disabled
- case unknown
-}
-
-public struct CrawlCount: Codable, Equatable, Sendable, Identifiable {
- public var id: String
- public var label: String
- public var value: Int
-
- public init(id: String, label: String, value: Int) {
- self.id = id
- self.label = label
- self.value = value
- }
-}
-
-public struct CrawlFreshness: Codable, Equatable, Sendable {
- public var status: CrawlAppState
- public var ageSeconds: Int?
- public var staleAfterSeconds: Int?
-
- public init(status: CrawlAppState, ageSeconds: Int? = nil, staleAfterSeconds: Int? = nil) {
- self.status = status
- self.ageSeconds = ageSeconds
- self.staleAfterSeconds = staleAfterSeconds
- }
-
- private enum CodingKeys: String, CodingKey {
- case status
- case ageSeconds = "age_seconds"
- case staleAfterSeconds = "stale_after_seconds"
- }
-}
-
-public struct CrawlShareStatus: Codable, Equatable, Sendable {
- public var enabled: Bool
- public var repoPath: String?
- public var remote: String?
- public var branch: String?
- public var needsUpdate: Bool?
-
- public init(enabled: Bool, repoPath: String? = nil, remote: String? = nil, branch: String? = nil, needsUpdate: Bool? = nil) {
- self.enabled = enabled
- self.repoPath = repoPath
- self.remote = remote
- self.branch = branch
- self.needsUpdate = needsUpdate
- }
-
- private enum CodingKeys: String, CodingKey {
- case enabled
- case repoPath = "repo_path"
- case remote
- case branch
- case needsUpdate = "needs_update"
- }
-}
-
-public struct CrawlRemoteStatus: Codable, Equatable, Sendable {
- public var enabled: Bool
- public var mode: String?
- public var endpoint: String?
- public var archive: String?
- public var lastIngestAt: Date?
- public var lastSyncAt: Date?
- public var needsUpdate: Bool?
-
- public init(
- enabled: Bool,
- mode: String? = nil,
- endpoint: String? = nil,
- archive: String? = nil,
- lastIngestAt: Date? = nil,
- lastSyncAt: Date? = nil,
- needsUpdate: Bool? = nil)
- {
- self.enabled = enabled
- self.mode = mode
- self.endpoint = endpoint
- self.archive = archive
- self.lastIngestAt = lastIngestAt
- self.lastSyncAt = lastSyncAt
- self.needsUpdate = needsUpdate
- }
-
- private enum CodingKeys: String, CodingKey {
- case enabled
- case mode
- case endpoint
- case archive
- case lastIngestAt = "last_ingest_at"
- case lastSyncAt = "last_sync_at"
- case needsUpdate = "needs_update"
- }
-}
-
-public struct CrawlSQLiteObjectStatus: Codable, Equatable, Sendable {
- public var key: String?
- public var contentType: String?
- public var bytes: Int?
- public var uploadedAt: Date?
-
- public init(key: String? = nil, contentType: String? = nil, bytes: Int? = nil, uploadedAt: Date? = nil) {
- self.key = key
- self.contentType = contentType
- self.bytes = bytes
- self.uploadedAt = uploadedAt
- }
-
- private enum CodingKeys: String, CodingKey {
- case key
- case contentType = "content_type"
- case bytes
- case uploadedAt = "uploaded_at"
- }
-}
-
-public struct CrawlSQLiteBundleStatus: Codable, Equatable, Sendable {
- public var key: String?
- public var contentType: String?
- public var format: String?
- public var compression: String?
- public var rawBytes: Int?
- public var compressedBytes: Int?
- public var partCount: Int?
- public var uploadedAt: Date?
- public var generatedAt: Date?
-
- public init(
- key: String? = nil,
- contentType: String? = nil,
- format: String? = nil,
- compression: String? = nil,
- rawBytes: Int? = nil,
- compressedBytes: Int? = nil,
- partCount: Int? = nil,
- uploadedAt: Date? = nil,
- generatedAt: Date? = nil)
- {
- self.key = key
- self.contentType = contentType
- self.format = format
- self.compression = compression
- self.rawBytes = rawBytes
- self.compressedBytes = compressedBytes
- self.partCount = partCount
- self.uploadedAt = uploadedAt
- self.generatedAt = generatedAt
- }
-
- private enum CodingKeys: String, CodingKey {
- case key
- case contentType = "content_type"
- case format
- case compression
- case rawBytes = "raw_bytes"
- case compressedBytes = "compressed_bytes"
- case partCount = "part_count"
- case uploadedAt = "uploaded_at"
- case generatedAt = "generated_at"
- }
-}
-
-public enum CrawlDatabaseKind: String, Codable, Equatable, Sendable {
- case sqlite
- case cache
- case logical
- case remote
- case d1
- case cloudflareD1 = "cloudflare-d1"
- case sqliteBundle = "sqlite_bundle"
-}
-
-public struct CrawlDatabaseResource: Codable, Equatable, Sendable, Identifiable {
- public var id: String
- public var label: String
- public var kind: CrawlDatabaseKind
- public var role: String?
- public var path: String?
- public var endpoint: String?
- public var archive: String?
- public var isPrimary: Bool
- public var bytes: Int?
- public var modifiedAt: Date?
- public var counts: [CrawlCount]
-
- public init(
- id: String,
- label: String,
- kind: CrawlDatabaseKind,
- role: String? = nil,
- path: String? = nil,
- endpoint: String? = nil,
- archive: String? = nil,
- isPrimary: Bool = false,
- bytes: Int? = nil,
- modifiedAt: Date? = nil,
- counts: [CrawlCount] = [])
- {
- self.id = id
- self.label = label
- self.kind = kind
- self.role = role
- self.path = path
- self.endpoint = endpoint
- self.archive = archive
- self.isPrimary = isPrimary
- self.bytes = bytes
- self.modifiedAt = modifiedAt
- self.counts = counts
- }
-
- private enum CodingKeys: String, CodingKey {
- case id
- case label
- case kind
- case role
- case path
- case endpoint
- case archive
- case isPrimary = "is_primary"
- case bytes
- case modifiedAt = "modified_at"
- case counts
- }
-}
-
-public struct CrawlAppStatus: Codable, Equatable, Sendable, Identifiable {
- public var schemaVersion: Int
- public var appID: CrawlAppID
- public var generatedAt: Date
- public var state: CrawlAppState
- public var summary: String
- public var configPath: String?
- public var databasePath: String?
- public var databaseBytes: Int?
- public var walBytes: Int?
- public var lastSyncAt: Date?
- public var lastImportAt: Date?
- public var lastExportAt: Date?
- public var counts: [CrawlCount]
- public var databases: [CrawlDatabaseResource]
- public var freshness: CrawlFreshness?
- public var share: CrawlShareStatus?
- public var remote: CrawlRemoteStatus?
- public var sqliteObject: CrawlSQLiteObjectStatus?
- public var sqliteBundle: CrawlSQLiteBundleStatus?
- public var warnings: [String]
- public var errors: [String]
-
- public var id: CrawlAppID {
- self.appID
- }
-
- public init(
- schemaVersion: Int = 1,
- appID: CrawlAppID,
- generatedAt: Date = Date(),
- state: CrawlAppState,
- summary: String,
- configPath: String? = nil,
- databasePath: String? = nil,
- databaseBytes: Int? = nil,
- walBytes: Int? = nil,
- lastSyncAt: Date? = nil,
- lastImportAt: Date? = nil,
- lastExportAt: Date? = nil,
- counts: [CrawlCount] = [],
- databases: [CrawlDatabaseResource] = [],
- freshness: CrawlFreshness? = nil,
- share: CrawlShareStatus? = nil,
- remote: CrawlRemoteStatus? = nil,
- sqliteObject: CrawlSQLiteObjectStatus? = nil,
- sqliteBundle: CrawlSQLiteBundleStatus? = nil,
- warnings: [String] = [],
- errors: [String] = [])
- {
- self.schemaVersion = schemaVersion
- self.appID = appID
- self.generatedAt = generatedAt
- self.state = state
- self.summary = summary
- self.configPath = configPath
- self.databasePath = databasePath
- self.databaseBytes = databaseBytes
- self.walBytes = walBytes
- self.lastSyncAt = lastSyncAt
- self.lastImportAt = lastImportAt
- self.lastExportAt = lastExportAt
- self.counts = counts
- self.databases = databases
- self.freshness = freshness
- self.share = share
- self.remote = remote
- self.sqliteObject = sqliteObject
- self.sqliteBundle = sqliteBundle
- self.warnings = warnings
- self.errors = errors
- }
-
- private enum CodingKeys: String, CodingKey {
- case schemaVersion = "schema_version"
- case appID = "app_id"
- case generatedAt = "generated_at"
- case state
- case summary
- case configPath = "config_path"
- case databasePath = "database_path"
- case databaseBytes = "database_bytes"
- case walBytes = "wal_bytes"
- case lastSyncAt = "last_sync_at"
- case lastImportAt = "last_import_at"
- case lastExportAt = "last_export_at"
- case counts
- case databases
- case freshness
- case share
- case remote
- case sqliteObject = "sqlite_object"
- case sqliteBundle = "sqlite_bundle"
- case warnings
- case errors
- }
-}
-
-public extension CrawlAppStatus {
- var isRecoverableGraincrawlSourceFailure: Bool {
- guard self.appID == BuiltInCrawlApps.graincrawlID, self.state == .error else { return false }
- guard !Self.summaryLooksLikeActionFailure(self.summary) else { return false }
- let text = ([self.summary] + self.errors + self.warnings)
- .joined(separator: "\n")
- .lowercased()
- return text.contains("granola access token")
- || text.contains("unsupported cache version")
- || text.contains("private-api reports")
- || text.contains("desktop-cache reports")
- }
-
- func mergingActionFailure(_ failure: CrawlAppStatus) -> CrawlAppStatus {
- guard self.appID == failure.appID else { return failure }
- return CrawlAppStatus(
- schemaVersion: self.schemaVersion,
- appID: self.appID,
- generatedAt: failure.generatedAt,
- state: failure.state,
- summary: failure.summary,
- configPath: self.configPath,
- databasePath: self.databasePath,
- databaseBytes: self.databaseBytes,
- walBytes: self.walBytes,
- lastSyncAt: self.lastSyncAt,
- lastImportAt: self.lastImportAt,
- lastExportAt: self.lastExportAt,
- counts: self.counts,
- databases: self.databases,
- freshness: self.freshness,
- share: self.share,
- remote: self.remote,
- sqliteObject: self.sqliteObject,
- sqliteBundle: self.sqliteBundle,
- warnings: Self.mergedMessages(failure.warnings, self.warnings),
- errors: Self.mergedMessages(failure.errors, self.errors))
- }
-
- static func commandFailure(
- appID: CrawlAppID,
- action: String? = nil,
- message: String?,
- fallback: String)
- -> CrawlAppStatus
- {
- let fullMessage = message?.nilIfBlank ?? fallback
- let normalized = Self.normalizedCommandFailure(appID: appID, message: fullMessage)
- let summary = [action?.nilIfBlank, normalized.summary].compactMap { $0 }.joined(separator: ": ")
- return CrawlAppStatus(
- appID: appID,
- state: normalized.state,
- summary: summary,
- errors: [normalized.summary])
- }
-
- static func richestMetadataStatus(_ preferred: CrawlAppStatus?, fallback: CrawlAppStatus?) -> CrawlAppStatus? {
- guard let preferred else { return fallback }
- guard let fallback else { return preferred }
- return preferred.metadataScore >= fallback.metadataScore ? preferred : fallback
- }
-
- private var metadataScore: Int {
- var score = 0
- score += self.configPath == nil ? 0 : 1
- score += self.databasePath == nil ? 0 : 1
- score += self.databaseBytes == nil ? 0 : 1
- score += self.walBytes == nil ? 0 : 1
- score += self.lastSyncAt == nil ? 0 : 1
- score += self.lastImportAt == nil ? 0 : 1
- score += self.lastExportAt == nil ? 0 : 1
- score += self.counts.isEmpty ? 0 : 2
- score += self.databases.isEmpty ? 0 : 3
- score += self.freshness == nil ? 0 : 1
- score += self.share == nil ? 0 : 2
- score += self.remote == nil ? 0 : 3
- score += self.sqliteObject == nil ? 0 : 2
- score += self.sqliteBundle == nil ? 0 : 3
- return score
- }
-
- private static func mergedMessages(_ primary: [String], _ secondary: [String]) -> [String] {
- var seen = Set()
- var messages: [String] = []
- for message in primary + secondary {
- guard !seen.contains(message) else { continue }
- seen.insert(message)
- messages.append(message)
- }
- return messages
- }
-
- private static func summaryLooksLikeActionFailure(_ summary: String) -> Bool {
- let trimmed = summary.trimmingCharacters(in: .whitespacesAndNewlines).lowercased()
- return [
- "refresh:",
- "sync:",
- "desktop-cache-import:",
- "doctor:",
- "unlock:",
- "query:",
- "search:",
- "export-md:",
- ].contains { trimmed.hasPrefix($0) }
- }
-
- private static func normalizedCommandFailure(appID: CrawlAppID, message: String) -> (state: CrawlAppState, summary: String) {
- let lowered = message.lowercased()
- if appID == BuiltInCrawlApps.gitcrawlID,
- lowered.contains("github"),
- (lowered.contains("bad credentials") || lowered.contains("status 401") || lowered.contains("401"))
- {
- return (.needsAuth, "GitHub credentials rejected")
- }
- if appID == BuiltInCrawlApps.birdclawID,
- lowered.contains("no twitter cookies")
- || lowered.contains("no x cookies")
- || lowered.contains("missing credentials")
- || lowered.contains("missing auth_token")
- || lowered.contains("missing ct0")
- {
- return (.needsAuth, "X browser cookies not found")
- }
- if appID == BuiltInCrawlApps.gogcliID,
- lowered.contains("credentials") || lowered.contains("auth")
- {
- return (.needsAuth, "Google account needs auth")
- }
- return (.error, Self.firstUsefulFailureLine(in: message) ?? "Command failed")
- }
-
- private static func firstUsefulFailureLine(in message: String) -> String? {
- message.split(separator: "\n")
- .map { $0.trimmingCharacters(in: .whitespacesAndNewlines) }
- .first { line in
- guard !line.isEmpty else { return false }
- return !Self.isRequestTraceLine(line)
- }
- }
-
- private static func isRequestTraceLine(_ line: String) -> Bool {
- let lowered = line.lowercased()
- return lowered.hasPrefix("[github] request ")
- || lowered.hasPrefix("[slack] request ")
- || lowered.hasPrefix("[notion] request ")
- || lowered.hasPrefix("[discord] request ")
- || lowered.hasPrefix("[granola] request ")
- }
-}
-
-public struct CrawlAppInstallation: Codable, Equatable, Sendable, Identifiable {
- public var manifest: CrawlAppManifest
- public var binaryPath: String?
- public var configPathOverride: String?
- public var configValues: [String: String]
- public var staleAfterSeconds: Int?
- public var enabled: Bool
-
- public var id: CrawlAppID {
- self.manifest.id
- }
-
- public init(
- manifest: CrawlAppManifest,
- binaryPath: String? = nil,
- configPathOverride: String? = nil,
- configValues: [String: String] = [:],
- staleAfterSeconds: Int? = nil,
- enabled: Bool = true)
- {
- self.manifest = manifest
- self.binaryPath = binaryPath
- self.configPathOverride = configPathOverride
- self.configValues = configValues
- self.staleAfterSeconds = staleAfterSeconds
- self.enabled = enabled
- }
-}
-
-public enum CrawlActionID: String, Codable, Hashable, Sendable {
- case status
- case doctor
- case refresh
- case publish
- case update
- case desktopCacheImport = "desktop-cache-import"
- case exportMarkdown = "export-md"
-}
-
-public struct CrawlCommandResult: Codable, Equatable, Sendable {
- public var appID: CrawlAppID
- public var action: String
- public var exitCode: Int32
- public var stdout: String
- public var stderr: String
- public var startedAt: Date
- public var finishedAt: Date
-
- public var succeeded: Bool {
- self.exitCode == 0
- }
-
- public init(
- appID: CrawlAppID,
- action: String,
- exitCode: Int32,
- stdout: String,
- stderr: String,
- startedAt: Date,
- finishedAt: Date)
- {
- self.appID = appID
- self.action = action
- self.exitCode = exitCode
- self.stdout = stdout
- self.stderr = stderr
- self.startedAt = startedAt
- self.finishedAt = finishedAt
- }
-}
-
-public extension CrawlCommandResult {
- var userFacingRunMessage: String? {
- if self.succeeded {
- return Self.firstLine(in: self.stderr)
- }
- return CrawlAppStatus.commandFailure(
- appID: self.appID,
- message: self.stderr.nilIfBlank ?? self.stdout.nilIfBlank,
- fallback: "\(self.action) failed with exit \(self.exitCode)")
- .summary
- }
-
- var shouldShowExitCode: Bool {
- !self.succeeded
- }
-
- private static func firstLine(in output: String) -> String? {
- output.nilIfBlank?
- .split(separator: "\n", omittingEmptySubsequences: true)
- .first
- .map(String.init)
- }
-}
diff --git a/Sources/CrawlBarCore/PathExpander.swift b/Sources/CrawlBarCore/PathExpander.swift
new file mode 100644
index 0000000..8067a5b
--- /dev/null
+++ b/Sources/CrawlBarCore/PathExpander.swift
@@ -0,0 +1,11 @@
+import Foundation
+
+public enum PathExpander {
+ public static func expandHome(_ path: String, home: String = NSHomeDirectory()) -> String {
+ if path == "~" { return home }
+ if path.hasPrefix("~/") {
+ return URL(fileURLWithPath: home).appendingPathComponent(String(path.dropFirst(2))).path
+ }
+ return path
+ }
+}
diff --git a/Sources/CrawlBarCore/Status/CrawlAppStatus.swift b/Sources/CrawlBarCore/Status/CrawlAppStatus.swift
new file mode 100644
index 0000000..b7231da
--- /dev/null
+++ b/Sources/CrawlBarCore/Status/CrawlAppStatus.swift
@@ -0,0 +1,99 @@
+import Foundation
+
+public struct CrawlAppStatus: Codable, Equatable, Sendable, Identifiable {
+ public var schemaVersion: Int
+ public var appID: CrawlAppID
+ public var generatedAt: Date
+ public var state: CrawlAppState
+ public var summary: String
+ public var configPath: String?
+ public var databasePath: String?
+ public var databaseBytes: Int?
+ public var walBytes: Int?
+ public var lastSyncAt: Date?
+ public var lastImportAt: Date?
+ public var lastExportAt: Date?
+ public var counts: [CrawlCount]
+ public var databases: [CrawlDatabaseResource]
+ public var freshness: CrawlFreshness?
+ public var share: CrawlShareStatus?
+ public var remote: CrawlRemoteStatus?
+ public var sqliteObject: CrawlSQLiteObjectStatus?
+ public var sqliteBundle: CrawlSQLiteBundleStatus?
+ public var warnings: [String]
+ public var errors: [String]
+
+ public var id: CrawlAppID {
+ self.appID
+ }
+
+ public init(
+ schemaVersion: Int = 1,
+ appID: CrawlAppID,
+ generatedAt: Date = Date(),
+ state: CrawlAppState,
+ summary: String,
+ configPath: String? = nil,
+ databasePath: String? = nil,
+ databaseBytes: Int? = nil,
+ walBytes: Int? = nil,
+ lastSyncAt: Date? = nil,
+ lastImportAt: Date? = nil,
+ lastExportAt: Date? = nil,
+ counts: [CrawlCount] = [],
+ databases: [CrawlDatabaseResource] = [],
+ freshness: CrawlFreshness? = nil,
+ share: CrawlShareStatus? = nil,
+ remote: CrawlRemoteStatus? = nil,
+ sqliteObject: CrawlSQLiteObjectStatus? = nil,
+ sqliteBundle: CrawlSQLiteBundleStatus? = nil,
+ warnings: [String] = [],
+ errors: [String] = [])
+ {
+ self.schemaVersion = schemaVersion
+ self.appID = appID
+ self.generatedAt = generatedAt
+ self.state = state
+ self.summary = summary
+ self.configPath = configPath
+ self.databasePath = databasePath
+ self.databaseBytes = databaseBytes
+ self.walBytes = walBytes
+ self.lastSyncAt = lastSyncAt
+ self.lastImportAt = lastImportAt
+ self.lastExportAt = lastExportAt
+ self.counts = counts
+ self.databases = databases
+ self.freshness = freshness
+ self.share = share
+ self.remote = remote
+ self.sqliteObject = sqliteObject
+ self.sqliteBundle = sqliteBundle
+ self.warnings = warnings
+ self.errors = errors
+ }
+
+ private enum CodingKeys: String, CodingKey {
+ case schemaVersion = "schema_version"
+ case appID = "app_id"
+ case generatedAt = "generated_at"
+ case state
+ case summary
+ case configPath = "config_path"
+ case databasePath = "database_path"
+ case databaseBytes = "database_bytes"
+ case walBytes = "wal_bytes"
+ case lastSyncAt = "last_sync_at"
+ case lastImportAt = "last_import_at"
+ case lastExportAt = "last_export_at"
+ case counts
+ case databases
+ case freshness
+ case share
+ case remote
+ case sqliteObject = "sqlite_object"
+ case sqliteBundle = "sqlite_bundle"
+ case warnings
+ case errors
+ }
+}
diff --git a/Sources/CrawlBarCore/Status/CrawlArchiveStatus.swift b/Sources/CrawlBarCore/Status/CrawlArchiveStatus.swift
new file mode 100644
index 0000000..bcc9453
--- /dev/null
+++ b/Sources/CrawlBarCore/Status/CrawlArchiveStatus.swift
@@ -0,0 +1,130 @@
+import Foundation
+
+public struct CrawlShareStatus: Codable, Equatable, Sendable {
+ public var enabled: Bool
+ public var repoPath: String?
+ public var remote: String?
+ public var branch: String?
+ public var needsUpdate: Bool?
+
+ public init(enabled: Bool, repoPath: String? = nil, remote: String? = nil, branch: String? = nil, needsUpdate: Bool? = nil) {
+ self.enabled = enabled
+ self.repoPath = repoPath
+ self.remote = remote
+ self.branch = branch
+ self.needsUpdate = needsUpdate
+ }
+
+ private enum CodingKeys: String, CodingKey {
+ case enabled
+ case repoPath = "repo_path"
+ case remote
+ case branch
+ case needsUpdate = "needs_update"
+ }
+}
+
+public struct CrawlRemoteStatus: Codable, Equatable, Sendable {
+ public var enabled: Bool
+ public var mode: String?
+ public var endpoint: String?
+ public var archive: String?
+ public var lastIngestAt: Date?
+ public var lastSyncAt: Date?
+ public var needsUpdate: Bool?
+
+ public init(
+ enabled: Bool,
+ mode: String? = nil,
+ endpoint: String? = nil,
+ archive: String? = nil,
+ lastIngestAt: Date? = nil,
+ lastSyncAt: Date? = nil,
+ needsUpdate: Bool? = nil)
+ {
+ self.enabled = enabled
+ self.mode = mode
+ self.endpoint = endpoint
+ self.archive = archive
+ self.lastIngestAt = lastIngestAt
+ self.lastSyncAt = lastSyncAt
+ self.needsUpdate = needsUpdate
+ }
+
+ private enum CodingKeys: String, CodingKey {
+ case enabled
+ case mode
+ case endpoint
+ case archive
+ case lastIngestAt = "last_ingest_at"
+ case lastSyncAt = "last_sync_at"
+ case needsUpdate = "needs_update"
+ }
+}
+
+public struct CrawlSQLiteObjectStatus: Codable, Equatable, Sendable {
+ public var key: String?
+ public var contentType: String?
+ public var bytes: Int?
+ public var uploadedAt: Date?
+
+ public init(key: String? = nil, contentType: String? = nil, bytes: Int? = nil, uploadedAt: Date? = nil) {
+ self.key = key
+ self.contentType = contentType
+ self.bytes = bytes
+ self.uploadedAt = uploadedAt
+ }
+
+ private enum CodingKeys: String, CodingKey {
+ case key
+ case contentType = "content_type"
+ case bytes
+ case uploadedAt = "uploaded_at"
+ }
+}
+
+public struct CrawlSQLiteBundleStatus: Codable, Equatable, Sendable {
+ public var key: String?
+ public var contentType: String?
+ public var format: String?
+ public var compression: String?
+ public var rawBytes: Int?
+ public var compressedBytes: Int?
+ public var partCount: Int?
+ public var uploadedAt: Date?
+ public var generatedAt: Date?
+
+ public init(
+ key: String? = nil,
+ contentType: String? = nil,
+ format: String? = nil,
+ compression: String? = nil,
+ rawBytes: Int? = nil,
+ compressedBytes: Int? = nil,
+ partCount: Int? = nil,
+ uploadedAt: Date? = nil,
+ generatedAt: Date? = nil)
+ {
+ self.key = key
+ self.contentType = contentType
+ self.format = format
+ self.compression = compression
+ self.rawBytes = rawBytes
+ self.compressedBytes = compressedBytes
+ self.partCount = partCount
+ self.uploadedAt = uploadedAt
+ self.generatedAt = generatedAt
+ }
+
+ private enum CodingKeys: String, CodingKey {
+ case key
+ case contentType = "content_type"
+ case format
+ case compression
+ case rawBytes = "raw_bytes"
+ case compressedBytes = "compressed_bytes"
+ case partCount = "part_count"
+ case uploadedAt = "uploaded_at"
+ case generatedAt = "generated_at"
+ }
+}
diff --git a/Sources/CrawlBarCore/Status/CrawlDatabaseResource.swift b/Sources/CrawlBarCore/Status/CrawlDatabaseResource.swift
new file mode 100644
index 0000000..f9d3bac
--- /dev/null
+++ b/Sources/CrawlBarCore/Status/CrawlDatabaseResource.swift
@@ -0,0 +1,65 @@
+import Foundation
+
+public enum CrawlDatabaseKind: String, Codable, Equatable, Sendable {
+ case sqlite
+ case cache
+ case logical
+ case remote
+ case d1
+ case cloudflareD1 = "cloudflare-d1"
+ case sqliteBundle = "sqlite_bundle"
+}
+
+public struct CrawlDatabaseResource: Codable, Equatable, Sendable, Identifiable {
+ public var id: String
+ public var label: String
+ public var kind: CrawlDatabaseKind
+ public var role: String?
+ public var path: String?
+ public var endpoint: String?
+ public var archive: String?
+ public var isPrimary: Bool
+ public var bytes: Int?
+ public var modifiedAt: Date?
+ public var counts: [CrawlCount]
+
+ public init(
+ id: String,
+ label: String,
+ kind: CrawlDatabaseKind,
+ role: String? = nil,
+ path: String? = nil,
+ endpoint: String? = nil,
+ archive: String? = nil,
+ isPrimary: Bool = false,
+ bytes: Int? = nil,
+ modifiedAt: Date? = nil,
+ counts: [CrawlCount] = [])
+ {
+ self.id = id
+ self.label = label
+ self.kind = kind
+ self.role = role
+ self.path = path
+ self.endpoint = endpoint
+ self.archive = archive
+ self.isPrimary = isPrimary
+ self.bytes = bytes
+ self.modifiedAt = modifiedAt
+ self.counts = counts
+ }
+
+ private enum CodingKeys: String, CodingKey {
+ case id
+ case label
+ case kind
+ case role
+ case path
+ case endpoint
+ case archive
+ case isPrimary = "is_primary"
+ case bytes
+ case modifiedAt = "modified_at"
+ case counts
+ }
+}
diff --git a/Sources/CrawlBarCore/Status/CrawlStatusState.swift b/Sources/CrawlBarCore/Status/CrawlStatusState.swift
new file mode 100644
index 0000000..624d64a
--- /dev/null
+++ b/Sources/CrawlBarCore/Status/CrawlStatusState.swift
@@ -0,0 +1,42 @@
+import Foundation
+
+public enum CrawlAppState: String, Codable, Equatable, Sendable {
+ case current
+ case stale
+ case syncing
+ case needsConfig = "needs_config"
+ case needsAuth = "needs_auth"
+ case error
+ case disabled
+ case unknown
+}
+
+public struct CrawlCount: Codable, Equatable, Sendable, Identifiable {
+ public var id: String
+ public var label: String
+ public var value: Int
+
+ public init(id: String, label: String, value: Int) {
+ self.id = id
+ self.label = label
+ self.value = value
+ }
+}
+
+public struct CrawlFreshness: Codable, Equatable, Sendable {
+ public var status: CrawlAppState
+ public var ageSeconds: Int?
+ public var staleAfterSeconds: Int?
+
+ public init(status: CrawlAppState, ageSeconds: Int? = nil, staleAfterSeconds: Int? = nil) {
+ self.status = status
+ self.ageSeconds = ageSeconds
+ self.staleAfterSeconds = staleAfterSeconds
+ }
+
+ private enum CodingKeys: String, CodingKey {
+ case status
+ case ageSeconds = "age_seconds"
+ case staleAfterSeconds = "stale_after_seconds"
+ }
+}
diff --git a/Sources/CrawlBarCore/StatusMapper.swift b/Sources/CrawlBarCore/StatusMapper.swift
index 19ef575..5b721a0 100644
--- a/Sources/CrawlBarCore/StatusMapper.swift
+++ b/Sources/CrawlBarCore/StatusMapper.swift
@@ -1,7 +1,7 @@
import Foundation
public struct CrawlStatusMapper: Sendable {
- private static let defaultStaleAfterSeconds = 86_400
+ static let defaultStaleAfterSeconds = 86_400
public init() {}
@@ -60,834 +60,4 @@ public struct CrawlStatusMapper: Sendable {
}
return CrawlDatabaseInventory.enrich(status, manifest: manifest)
}
-
- private func gitcrawlStatus(_ object: [String: Any], result: CrawlCommandResult, staleAfterSeconds: Int?) -> CrawlAppStatus {
- let counts = [
- self.count("threads", "Threads", ["thread_count", "threads"]),
- self.count("open_threads", "Open Threads", ["open_thread_count", "open_threads"]),
- self.count("clusters", "Clusters", ["cluster_count", "clusters"]),
- self.count("repositories", "Repositories", ["repo_count", "repository_count", "repositories"]),
- ].compactMap { self.value($0, in: object) }
-
- let remote = self.remoteStatus(in: object)
- let lastSyncAt = self.dateValue(["last_sync_at", "updated_at", "generated_at"], in: object)
- ?? remote?.lastSyncAt
- ?? remote?.lastIngestAt
- return CrawlAppStatus(
- appID: result.appID,
- state: self.state(lastSyncAt: lastSyncAt, fallback: .current, staleAfterSeconds: staleAfterSeconds),
- summary: self.summary(from: counts, fallback: "Git crawl status is current"),
- configPath: self.stringValue(["config_path", "config"], in: object),
- databasePath: self.stringValue(["db_path", "database_path", "database"], in: object),
- lastSyncAt: lastSyncAt,
- counts: counts,
- freshness: self.freshness(in: object, lastSyncAt: lastSyncAt, staleAfterSeconds: staleAfterSeconds),
- share: self.shareStatus(in: object),
- remote: remote,
- sqliteObject: self.sqliteObjectStatus(in: object),
- sqliteBundle: self.sqliteBundleStatus(in: object))
- }
-
- private func slacrawlStatus(_ object: [String: Any], result: CrawlCommandResult, staleAfterSeconds: Int?) -> CrawlAppStatus {
- let flatCounts = [
- self.count("workspaces", "Workspaces", ["workspace_count", "workspaces"]),
- self.count("channels", "Channels", ["channel_count", "channels"]),
- self.count("users", "Users", ["user_count", "users"]),
- self.count("messages", "Messages", ["message_count", "messages"]),
- ].compactMap { self.value($0, in: object) }
-
- let counts = self.statusCounts(in: object, fallback: flatCounts)
- let databases = self.databaseResources(in: object)
- let remote = self.remoteStatus(in: object)
- let lastSyncAt = self.dateValue(["last_sync_at", "latest_message_at", "updated_at"], in: object)
- ?? remote?.lastSyncAt
- ?? remote?.lastIngestAt
- ?? self.databaseModifiedAt(databases)
- let freshness = self.freshness(in: object, lastSyncAt: lastSyncAt, staleAfterSeconds: staleAfterSeconds)
- return CrawlAppStatus(
- appID: result.appID,
- state: self.statusState(in: object, lastSyncAt: lastSyncAt, freshness: freshness, fallback: .current, staleAfterSeconds: staleAfterSeconds),
- summary: self.stringValue(["summary", "message"], in: object) ?? self.summary(from: counts, fallback: "Slack crawl status is current"),
- configPath: self.stringValue(["config_path", "config"], in: object),
- databasePath: self.stringValue(["db_path", "database_path", "database"], in: object),
- databaseBytes: self.intValue(["db_bytes", "database_bytes"], in: object),
- lastSyncAt: lastSyncAt,
- counts: counts,
- databases: databases,
- freshness: freshness,
- share: self.shareStatus(in: object),
- remote: remote,
- sqliteObject: self.sqliteObjectStatus(in: object),
- sqliteBundle: self.sqliteBundleStatus(in: object))
- }
-
- private func gogStatus(_ object: [String: Any], result: CrawlCommandResult) -> CrawlAppStatus {
- if let checks = object["checks"] as? [[String: Any]] {
- return self.gogDoctorStatus(checks, result: result)
- }
-
- let account = self.firstObject(["account"], in: object) ?? [:]
- let config = self.firstObject(["config"], in: object) ?? [:]
- let credentialsExist = self.boolValue(["credentials_exists"], in: account) ?? false
- let serviceAccountConfigured = self.boolValue(["service_account_configured"], in: account) ?? false
- let email = self.stringValue(["email", "account"], in: account)
- let configPath = self.stringValue(["path"], in: config)
-
- let state: CrawlAppState = (credentialsExist || serviceAccountConfigured) ? .current : .needsAuth
- var warnings: [String] = []
- if self.boolValue(["exists"], in: config) == false {
- warnings.append("gog config file not found")
- }
- let summary = email.map { "Google account \($0) is ready" }
- ?? (state == .current ? "Google account is ready" : "Google account needs auth")
-
- return CrawlAppStatus(
- appID: result.appID,
- state: state,
- summary: summary,
- configPath: configPath,
- warnings: warnings)
- }
-
- private func gogDoctorStatus(_ checks: [[String: Any]], result: CrawlCommandResult) -> CrawlAppStatus {
- let readableTokens = checks.first { ($0["name"] as? String) == "tokens" && ($0["status"] as? String) == "ok" }
- let refreshErrors = checks.filter { check in
- guard let name = check["name"] as? String else { return false }
- return name.hasPrefix("refresh.") && (check["status"] as? String) == "error"
- }
- let warnings = checks.compactMap { check -> String? in
- guard let status = check["status"] as? String,
- status != "ok",
- let name = check["name"] as? String
- else { return nil }
- let detail = (check["detail"] as? String)?.nilIfBlank
- return [name, detail].compactMap { $0 }.joined(separator: ": ")
- }
-
- let state: CrawlAppState
- let summary: String
- if !refreshErrors.isEmpty {
- state = .error
- summary = "Google refresh token check failed"
- } else if let readableTokens {
- state = .current
- if let detail = (readableTokens["detail"] as? String)?.nilIfBlank,
- let count = detail.split(separator: " ").first
- {
- summary = "\(count) Google OAuth accounts readable"
- } else {
- summary = "Google OAuth accounts readable"
- }
- } else {
- state = .needsAuth
- summary = "Google account needs auth"
- }
-
- return CrawlAppStatus(
- appID: result.appID,
- state: state,
- summary: summary,
- warnings: warnings)
- }
-
- private func birdclawStatus(_ object: [String: Any], result: CrawlCommandResult) -> CrawlAppStatus {
- let transport = self.firstObject(["transport"], in: object) ?? object
- let installed = self.boolValue(["installed"], in: transport)
- let transportName = self.stringValue(["availableTransport"], in: transport)
- ?? self.stringValue(["available_transport"], in: transport)
- let statusText = self.stringValue(["statusText", "status_text", "summary", "message"], in: transport)
-
- let state: CrawlAppState = .current
- let summary = statusText ?? "birdclaw is ready"
- var warnings = transportName.map { ["Transport: \($0)"] } ?? []
- if installed == false, let statusText {
- warnings.append(statusText)
- }
-
- return CrawlAppStatus(
- appID: result.appID,
- state: state,
- summary: summary,
- warnings: warnings)
- }
-
- private func birdStatusText(_ result: CrawlCommandResult) -> CrawlAppStatus {
- let output = result.stdout.nilIfBlank ?? result.stderr.nilIfBlank ?? ""
- let lowercased = output.lowercased()
- let hasAuthToken = lowercased.contains("[ok] auth_token")
- || lowercased.contains("auth_token:")
- let hasCSRFToken = lowercased.contains("[ok] ct0")
- || lowercased.contains("ct0:")
- let ready = lowercased.contains("ready to tweet")
- || (hasAuthToken && hasCSRFToken)
- let source = output.split(separator: "\n")
- .map { $0.trimmingCharacters(in: .whitespacesAndNewlines) }
- .first { $0.lowercased().hasPrefix("source:") }
- let warningLines = output.split(separator: "\n")
- .map { $0.trimmingCharacters(in: .whitespacesAndNewlines) }
- .filter { line in
- let lowered = line.lowercased()
- return lowered.contains("[warn]") || lowered.hasPrefix("- ")
- }
- .map { line in
- line.replacingOccurrences(of: "[warn]", with: "")
- .trimmingCharacters(in: .whitespacesAndNewlines)
- }
- .map { line in
- line.hasPrefix("- ")
- ? String(line.dropFirst(2)).trimmingCharacters(in: .whitespacesAndNewlines)
- : line
- }
- .filter { !$0.isEmpty && $0.lowercased() != "warnings:" }
-
- if ready {
- return CrawlAppStatus(
- appID: result.appID,
- state: .current,
- summary: source.map { "X cookies available via bird (\($0.dropFirst("source:".count).trimmingCharacters(in: .whitespacesAndNewlines)))" }
- ?? "X cookies available via bird",
- warnings: warningLines)
- }
-
- return CrawlAppStatus(
- appID: result.appID,
- state: .needsAuth,
- summary: "X browser cookies not found",
- warnings: warningLines.isEmpty ? ["bird check did not find usable X cookies"] : warningLines)
- }
-
- private func discrawlStatus(_ object: [String: Any], result: CrawlCommandResult, staleAfterSeconds: Int?) -> CrawlAppStatus {
- let flatCounts = [
- self.count("guilds", "Guilds", ["guild_count", "guilds"]),
- self.count("channels", "Channels", ["channel_count", "channels"]),
- self.count("threads", "Threads", ["thread_count", "threads"]),
- self.count("messages", "Messages", ["message_count", "messages"]),
- self.count("members", "Members", ["member_count", "members"]),
- self.count("embedding_backlog", "Embedding Backlog", ["embedding_backlog"]),
- ].compactMap { self.value($0, in: object) }
-
- let counts = self.statusCounts(in: object, fallback: flatCounts)
- let databases = self.databaseResources(in: object)
- let remote = self.remoteStatus(in: object)
- let lastSyncAt = self.dateValue(["last_sync_at", "latest_message_at", "updated_at"], in: object)
- ?? remote?.lastSyncAt
- ?? remote?.lastIngestAt
- ?? self.databaseModifiedAt(databases)
- let freshness = self.freshness(in: object, lastSyncAt: lastSyncAt, staleAfterSeconds: staleAfterSeconds)
- return CrawlAppStatus(
- appID: result.appID,
- state: self.statusState(in: object, lastSyncAt: lastSyncAt, freshness: freshness, fallback: .current, staleAfterSeconds: staleAfterSeconds),
- summary: self.stringValue(["summary", "message"], in: object) ?? self.summary(from: counts, fallback: "Discord crawl status is current"),
- configPath: self.stringValue(["config_path", "config"], in: object),
- databasePath: self.stringValue(["db_path", "database_path", "database"], in: object),
- databaseBytes: self.intValue(["db_bytes", "database_bytes"], in: object),
- lastSyncAt: lastSyncAt,
- counts: counts,
- databases: databases,
- freshness: freshness,
- share: self.shareStatus(in: object),
- remote: remote,
- sqliteObject: self.sqliteObjectStatus(in: object),
- sqliteBundle: self.sqliteBundleStatus(in: object))
- }
-
- private func telecrawlStatus(_ object: [String: Any], result: CrawlCommandResult, staleAfterSeconds: Int?) -> CrawlAppStatus {
- let counts = [
- self.count("messages", "Messages", ["message_count", "messages"]),
- self.count("chats", "Chats", ["chat_count", "chats"]),
- self.count("folders", "Folders", ["folder_count", "folders"]),
- self.count("topics", "Topics", ["topic_count", "topics"]),
- self.count("unread_chats", "Unread Chats", ["unread_chat_count", "unread_chats"]),
- self.count("unread_messages", "Unread Messages", ["unread_message_count", "unread_messages"]),
- self.count("media_messages", "Media Messages", ["media_message_count", "media_messages"]),
- ].compactMap { self.value($0, in: object) }
-
- let lastSyncAt = self.dateValue(["last_sync_at", "last_import_at", "updated_at"], in: object)
- let freshness = self.freshness(in: object, lastSyncAt: lastSyncAt, staleAfterSeconds: staleAfterSeconds)
- return CrawlAppStatus(
- appID: result.appID,
- state: self.statusState(in: object, lastSyncAt: lastSyncAt, freshness: freshness, fallback: .current, staleAfterSeconds: staleAfterSeconds),
- summary: self.stringValue(["summary", "message"], in: object) ?? self.summary(from: counts, fallback: "Telegram crawl status is current"),
- configPath: self.stringValue(["config_path", "config"], in: object),
- databasePath: self.stringValue(["db_path", "database_path", "database"], in: object),
- databaseBytes: self.intValue(["db_bytes", "database_bytes"], in: object),
- lastSyncAt: lastSyncAt,
- lastImportAt: self.dateValue(["last_import_at"], in: object),
- counts: counts,
- freshness: freshness,
- share: self.shareStatus(in: object))
- }
-
- private func notcrawlStatus(_ object: [String: Any], result: CrawlCommandResult, staleAfterSeconds: Int?) -> CrawlAppStatus {
- let counts = [
- self.count("spaces", "Spaces", ["space_count", "spaces"]),
- self.count("users", "Users", ["user_count", "users"]),
- self.count("teams", "Teams", ["team_count", "teams"]),
- self.count("pages", "Pages", ["page_count", "pages"]),
- self.count("blocks", "Blocks", ["block_count", "blocks"]),
- self.count("collections", "Collections", ["collection_count", "collections"]),
- self.count("comments", "Comments", ["comment_count", "comments"]),
- self.count("raw_records", "Raw Records", ["raw_record_count", "raw_records"]),
- ].compactMap { self.value($0, in: object) }
-
- let remote = self.remoteStatus(in: object)
- let lastSyncAt = self.dateValue(["last_sync_at", "last_import_at", "updated_at"], in: object)
- ?? remote?.lastSyncAt
- ?? remote?.lastIngestAt
- return CrawlAppStatus(
- appID: result.appID,
- state: self.state(lastSyncAt: lastSyncAt, fallback: .current, staleAfterSeconds: staleAfterSeconds),
- summary: self.summary(from: counts, fallback: "Notion crawl status is current"),
- configPath: self.stringValue(["config_path", "config"], in: object),
- databasePath: self.stringValue(["db_path", "database_path", "database"], in: object),
- databaseBytes: self.intValue(["db_bytes", "database_bytes"], in: object),
- walBytes: self.intValue(["wal_bytes"], in: object),
- lastSyncAt: lastSyncAt,
- counts: counts,
- freshness: self.freshness(in: object, lastSyncAt: lastSyncAt, staleAfterSeconds: staleAfterSeconds),
- share: self.shareStatus(in: object),
- remote: remote,
- sqliteObject: self.sqliteObjectStatus(in: object),
- sqliteBundle: self.sqliteBundleStatus(in: object))
- }
-
- private func gogcliStatus(_ object: [String: Any], result: CrawlCommandResult, staleAfterSeconds _: Int?) -> CrawlAppStatus {
- if let accounts = object["accounts"] as? [[String: Any]] {
- let configuredAccount = accounts.first { account in
- self.boolValue(["valid"], in: account) != false
- && ["oauth", "service-account", "service_account", "oauth+service-account", "oauth+service_account"].contains(
- self.stringValue(["auth"], in: account)?.lowercased() ?? "")
- }
- let failedAccount = accounts.first { self.boolValue(["valid"], in: $0) == false }
- let failureSummary = failedAccount.flatMap { account in
- self.stringValue(["error", "hint", "email"], in: account)
- }
- return CrawlAppStatus(
- appID: result.appID,
- state: configuredAccount == nil ? .needsAuth : .current,
- summary: configuredAccount == nil ? failureSummary ?? "Google auth needs setup" : "Google auth configured")
- }
-
- if let status = self.statusValue(["status"], in: object), object["checks"] != nil {
- let checks = object["checks"] as? [[String: Any]]
- let readableTokens = checks?.first { check in
- self.stringValue(["name"], in: check) == "tokens"
- && self.statusValue(["status"], in: check) == .current
- }
- let refreshErrors = checks?.filter { check in
- guard let name = self.stringValue(["name"], in: check) else { return false }
- return name.hasPrefix("refresh.") && self.statusValue(["status"], in: check) == .error
- } ?? []
- let warnings = checks?.compactMap { check -> String? in
- guard self.statusValue(["status"], in: check) != .current,
- let name = self.stringValue(["name"], in: check)
- else { return nil }
- let detail = self.stringValue(["detail"], in: check)?.nilIfBlank
- return [name, detail].compactMap { $0 }.joined(separator: ": ")
- } ?? []
- if refreshErrors.isEmpty, let readableTokens {
- let detail = self.stringValue(["detail"], in: readableTokens)
- let summary = detail?.split(separator: " ").first.map { "\($0) Google OAuth accounts readable" }
- ?? "Google OAuth accounts readable"
- return CrawlAppStatus(
- appID: result.appID,
- state: .current,
- summary: summary,
- configPath: self.gogcliConfigPath(fromChecks: checks),
- warnings: warnings)
- }
- let failedCheck = checks?.first { check in
- self.statusValue(["status"], in: check) != .current
- }
- let failureSummary = failedCheck.flatMap { check in
- self.stringValue(["detail", "hint", "name"], in: check)
- }
- let mappedState = status == .current
- ? CrawlAppState.current
- : self.gogcliDoctorFailureState(failedCheck)
- return CrawlAppStatus(
- appID: result.appID,
- state: mappedState,
- summary: mappedState == .current ? "Google auth configured" : failureSummary ?? "Google auth needs setup",
- configPath: self.gogcliConfigPath(fromChecks: checks),
- warnings: warnings)
- }
-
- let account = self.firstObject(["account"], in: object) ?? [:]
- let config = self.firstObject(["config"], in: object) ?? [:]
- let serviceAccountConfigured = self.boolValue(["service_account_configured"], in: account) ?? false
- let state: CrawlAppState = serviceAccountConfigured ? .current : .needsAuth
- let summary = state == .current ? "Google service account configured" : "Google account needs auth"
- let warnings = self.boolValue(["exists"], in: config) == false
- ? ["gog config file not found"]
- : []
- return CrawlAppStatus(
- appID: result.appID,
- state: state,
- summary: summary,
- configPath: self.stringValue(["path"], in: config),
- warnings: warnings)
- }
-
- private func gogcliDoctorFailureState(_ check: [String: Any]?) -> CrawlAppState {
- let name = check.flatMap { self.stringValue(["name"], in: $0) }?.lowercased() ?? ""
- return name.contains("config") ? .needsConfig : .needsAuth
- }
-
- private func gogcliConfigPath(fromChecks checks: [[String: Any]]?) -> String? {
- checks?.first { self.stringValue(["name"], in: $0) == "config.path" }
- .flatMap { self.stringValue(["detail"], in: $0) }
- }
-
- private func wacliStatus(_ object: [String: Any], result: CrawlCommandResult, staleAfterSeconds: Int?) -> CrawlAppStatus {
- let data = self.firstObject(["data"], in: object) ?? object
- let isAuthenticated = self.boolValue(["authenticated"], in: data) ?? true
- let storeError = self.stringValue(["store_error"], in: data)?.nilIfBlank
- if let storeError, (isAuthenticated || !Self.isWacliFirstRunStoreError(storeError)) {
- return CrawlAppStatus(
- appID: result.appID,
- state: .error,
- summary: storeError,
- errors: [storeError])
- }
- if !isAuthenticated {
- return CrawlAppStatus(
- appID: result.appID,
- state: .needsAuth,
- summary: "WhatsApp auth needs setup")
- }
- let store = self.firstObject(["store"], in: data) ?? data
- let counts = [
- self.count("messages", "Messages", ["messages", "message_count"]),
- self.count("chats", "Chats", ["chats", "chat_count"]),
- self.count("contacts", "Contacts", ["contacts", "contact_count"]),
- self.count("groups", "Groups", ["groups", "group_count"]),
- ].compactMap { self.value($0, in: store) }
- let lastSyncAt = self.dateValue(["last_sync_at"], in: store)
- ?? self.dateValue(["last_sync_at", "updated_at"], in: data)
- let freshness = self.freshness(in: object, lastSyncAt: lastSyncAt, staleAfterSeconds: staleAfterSeconds)
- let storeDir = self.stringValue(["store_dir"], in: data)
- let state: CrawlAppState
- if self.boolValue(["success"], in: object) == false {
- state = .error
- } else {
- state = self.statusValue(["state", "status"], in: data)
- ?? self.statusValue(["state", "status"], in: object)
- ?? self.statusState(in: object, lastSyncAt: lastSyncAt, freshness: freshness, fallback: .current, staleAfterSeconds: staleAfterSeconds)
- }
- return CrawlAppStatus(
- appID: result.appID,
- state: state,
- summary: self.summary(from: counts, fallback: state == .error ? self.stringValue(["error"], in: object) ?? "WhatsApp diagnostics failed" : "WhatsApp archive is current"),
- configPath: storeDir.map(Self.wacliConfigPath(storeDir:)),
- databasePath: storeDir.map(Self.wacliDatabasePath(storeDir:)),
- lastSyncAt: lastSyncAt,
- counts: counts,
- databases: storeDir.map { Self.wacliDatabaseResources(storeDir: $0, counts: counts) } ?? [],
- freshness: freshness,
- warnings: self.wacliWarnings(in: data),
- errors: self.wacliErrors(in: object))
- }
-
- private static func wacliConfigPath(storeDir: String) -> String {
- let storeURL = URL(fileURLWithPath: storeDir)
- if storeURL.deletingLastPathComponent().lastPathComponent == "accounts" {
- return storeURL
- .deletingLastPathComponent()
- .deletingLastPathComponent()
- .appendingPathComponent("config.yaml")
- .path
- }
- return storeURL.appendingPathComponent("config.yaml").path
- }
-
- private static func wacliDatabasePath(storeDir: String) -> String {
- URL(fileURLWithPath: storeDir).appendingPathComponent("wacli.db").path
- }
-
- private static func wacliDatabaseResources(storeDir: String, counts: [CrawlCount]) -> [CrawlDatabaseResource] {
- let databasePath = Self.wacliDatabasePath(storeDir: storeDir)
- var resources = [
- CrawlDatabaseResource(
- id: databasePath,
- label: "WhatsApp SQLite database",
- kind: .sqlite,
- path: databasePath,
- isPrimary: true,
- counts: counts),
- ]
- if databasePath != storeDir {
- resources.append(CrawlDatabaseResource(
- id: storeDir,
- label: "WhatsApp store",
- kind: .logical,
- path: storeDir,
- counts: counts))
- }
- return resources
- }
-
- private static func isWacliFirstRunStoreError(_ error: String) -> Bool {
- let lowercased = error.lowercased()
- return lowercased.contains("no such file")
- || lowercased.contains("not found")
- || lowercased.contains("missing")
- || lowercased.contains("uninitialized")
- }
-
- private func genericStatus(_ object: [String: Any], result: CrawlCommandResult, staleAfterSeconds: Int?) -> CrawlAppStatus {
- let counts = self.statusCounts(in: object, fallback: self.counts(in: object))
- let databases = self.databaseResources(in: object)
- let remote = self.remoteStatus(in: object)
- let lastSyncAt = self.dateValue(["last_sync_at", "updated_at", "generated_at"], in: object)
- ?? remote?.lastSyncAt
- ?? remote?.lastIngestAt
- ?? self.databaseModifiedAt(databases)
- let freshness = self.freshness(in: object, lastSyncAt: lastSyncAt, staleAfterSeconds: staleAfterSeconds)
- return CrawlAppStatus(
- appID: result.appID,
- state: self.statusState(in: object, lastSyncAt: lastSyncAt, freshness: freshness, fallback: .current, staleAfterSeconds: staleAfterSeconds),
- summary: self.stringValue(["summary", "message"], in: object) ?? self.summary(from: counts, fallback: "Status is current"),
- configPath: self.stringValue(["config_path", "config"], in: object),
- databasePath: self.stringValue(["db_path", "database_path"], in: object),
- databaseBytes: self.intValue(["db_bytes", "database_bytes"], in: object),
- walBytes: self.intValue(["wal_bytes"], in: object),
- lastSyncAt: lastSyncAt,
- lastImportAt: self.dateValue(["last_import_at"], in: object),
- lastExportAt: self.dateValue(["last_export_at"], in: object),
- counts: counts,
- databases: databases,
- freshness: freshness,
- share: self.shareStatus(in: object),
- remote: remote,
- sqliteObject: self.sqliteObjectStatus(in: object),
- sqliteBundle: self.sqliteBundleStatus(in: object))
- }
-
- private func isWacliManifest(_ manifest: CrawlAppManifest) -> Bool {
- manifest.id == BuiltInCrawlApps.wacliID
- || manifest.id.rawValue.hasPrefix("wacli-")
- || manifest.binary.name == "wacli"
- }
-
- private func wacliWarnings(in object: [String: Any]) -> [String] {
- var warnings: [String] = []
- if self.boolValue(["lock_held"], in: object) == true,
- let state = self.stringValue(["connection_state"], in: object)
- {
- warnings.append("Store is locked by \(state)")
- }
- if self.boolValue(["fts_enabled"], in: object) == false {
- warnings.append("Full-text search is not enabled")
- }
- return warnings
- }
-
- private func wacliErrors(in object: [String: Any]) -> [String] {
- guard self.boolValue(["success"], in: object) == false else { return [] }
- if let error = self.stringValue(["error"], in: object) {
- return [error]
- }
- return ["wacli doctor reported failure"]
- }
-
- private func parseObject(_ text: String) -> [String: Any]? {
- guard let data = text.data(using: .utf8), !data.isEmpty else { return nil }
- return (try? JSONSerialization.jsonObject(with: data)) as? [String: Any]
- }
-
- private func isCrawlKitStatus(_ object: [String: Any]) -> Bool {
- if let schema = self.stringValue(["schema_version"], in: object), schema.hasPrefix("crawlkit.control.") {
- return true
- }
- return self.firstValue("databases", in: object) != nil && self.firstValue("counts", in: object) != nil
- }
-
- private func count(_ id: String, _ label: String, _ keys: [String]) -> (String, String, [String]) {
- (id, label, keys)
- }
-
- private func value(_ spec: (String, String, [String]), in object: [String: Any]) -> CrawlCount? {
- guard let value = self.intValue(spec.2, in: object) else { return nil }
- return CrawlCount(id: spec.0, label: spec.1, value: value)
- }
-
- private func counts(in object: [String: Any]) -> [CrawlCount] {
- guard let counts = self.firstObject(["counts", "stats"], in: object) else { return [] }
- return counts.compactMap { key, value in
- guard let int = self.int(value) else { return nil }
- return CrawlCount(id: key, label: self.label(from: key), value: int)
- }
- .sorted { $0.id < $1.id }
- }
-
- private func state(lastSyncAt: Date?, fallback: CrawlAppState, staleAfterSeconds: Int?) -> CrawlAppState {
- guard let lastSyncAt else { return fallback }
- let threshold = staleAfterSeconds ?? Self.defaultStaleAfterSeconds
- return Date().timeIntervalSince(lastSyncAt) > TimeInterval(threshold) ? .stale : .current
- }
-
- private func freshness(lastSyncAt: Date?, staleAfterSeconds: Int?) -> CrawlFreshness? {
- guard let lastSyncAt else { return nil }
- let threshold = staleAfterSeconds ?? Self.defaultStaleAfterSeconds
- let ageSeconds = max(0, Int(Date().timeIntervalSince(lastSyncAt)))
- return CrawlFreshness(
- status: ageSeconds > threshold ? .stale : .current,
- ageSeconds: ageSeconds,
- staleAfterSeconds: threshold)
- }
-
- private func freshness(in object: [String: Any], lastSyncAt: Date?, staleAfterSeconds: Int?) -> CrawlFreshness? {
- if let freshness = self.firstObject(["freshness"], in: object),
- let status = self.statusValue(["status", "state"], in: freshness)
- {
- return CrawlFreshness(
- status: status,
- ageSeconds: self.intValue(["age_seconds"], in: freshness),
- staleAfterSeconds: self.intValue(["stale_after_seconds"], in: freshness) ?? staleAfterSeconds)
- }
- return self.freshness(lastSyncAt: lastSyncAt, staleAfterSeconds: staleAfterSeconds)
- }
-
- private func summary(from counts: [CrawlCount], fallback: String) -> String {
- let visible = counts.prefix(2).map { "\($0.value) \($0.label.lowercased())" }
- return visible.isEmpty ? fallback : visible.joined(separator: ", ")
- }
-
- private func statusState(
- in object: [String: Any],
- lastSyncAt: Date?,
- freshness: CrawlFreshness?,
- fallback: CrawlAppState,
- staleAfterSeconds: Int?)
- -> CrawlAppState
- {
- if let state = self.statusValue(["state", "status"], in: object)
- {
- return state
- }
- return freshness?.status ?? self.state(lastSyncAt: lastSyncAt, fallback: fallback, staleAfterSeconds: staleAfterSeconds)
- }
-
- private func statusCounts(in object: [String: Any], fallback: [CrawlCount]) -> [CrawlCount] {
- let declared = self.countArray(["counts"], in: object)
- return declared.isEmpty ? fallback : declared
- }
-
- private func countArray(_ keys: [String], in object: [String: Any]) -> [CrawlCount] {
- for key in keys {
- guard let array = self.firstValue(key, in: object) as? [Any] else { continue }
- let counts = array.compactMap { item -> CrawlCount? in
- guard let item = item as? [String: Any],
- let id = self.stringValue(["id"], in: item),
- let value = self.intValue(["value", "count"], in: item)
- else { return nil }
- return CrawlCount(
- id: id,
- label: self.stringValue(["label"], in: item) ?? self.label(from: id),
- value: value)
- }
- if !counts.isEmpty { return counts }
- }
- return []
- }
-
- private func databaseResources(in object: [String: Any]) -> [CrawlDatabaseResource] {
- guard let array = self.firstValue("databases", in: object) as? [Any] else { return [] }
- return array.compactMap { item -> CrawlDatabaseResource? in
- guard let item = item as? [String: Any] else { return nil }
- let path = self.stringValue(["path"], in: item)
- guard let id = self.stringValue(["id"], in: item) ?? path else { return nil }
- let kindValue = self.stringValue(["kind"], in: item) ?? CrawlDatabaseKind.sqlite.rawValue
- return CrawlDatabaseResource(
- id: id,
- label: self.stringValue(["label"], in: item) ?? URL(fileURLWithPath: id).lastPathComponent,
- kind: CrawlDatabaseKind(rawValue: kindValue) ?? .remote,
- role: self.stringValue(["role"], in: item),
- path: path,
- endpoint: self.stringValue(["endpoint"], in: item),
- archive: self.stringValue(["archive"], in: item),
- isPrimary: self.boolValue(["is_primary", "primary"], in: item) ?? false,
- bytes: self.intValue(["bytes", "size_bytes"], in: item),
- modifiedAt: self.dateValue(["modified_at", "updated_at"], in: item),
- counts: self.countArray(["counts"], in: item))
- }
- }
-
- private func databaseModifiedAt(_ databases: [CrawlDatabaseResource]) -> Date? {
- databases.first(where: { $0.isPrimary })?.modifiedAt
- ?? databases.compactMap(\.modifiedAt).max()
- }
-
- private func intValue(_ keys: [String], in object: [String: Any]) -> Int? {
- for key in keys {
- if let value = self.firstValue(key, in: object), let int = self.int(value) {
- return int
- }
- }
- return nil
- }
-
- private func boolValue(_ keys: [String], in object: [String: Any]) -> Bool? {
- for key in keys {
- guard let value = self.firstValue(key, in: object) else { continue }
- if let bool = value as? Bool { return bool }
- if let number = value as? NSNumber { return number.boolValue }
- if let string = value as? String {
- switch string.lowercased() {
- case "true", "yes", "1":
- return true
- case "false", "no", "0":
- return false
- default:
- continue
- }
- }
- }
- return nil
- }
-
- private func stringValue(_ keys: [String], in object: [String: Any]) -> String? {
- for key in keys {
- if let value = self.firstValue(key, in: object) as? String, let string = value.nilIfBlank {
- return string
- }
- }
- return nil
- }
-
- private func statusValue(_ keys: [String], in object: [String: Any]) -> CrawlAppState? {
- guard let rawValue = self.stringValue(keys, in: object) else { return nil }
- if let state = CrawlAppState(rawValue: rawValue) {
- return state
- }
- switch rawValue.lowercased() {
- case "ok", "success", "healthy", "ready":
- return .current
- case "warn", "warning", "degraded":
- return .stale
- case "failed", "failure":
- return .error
- default:
- return nil
- }
- }
-
- private func dateValue(_ keys: [String], in object: [String: Any]) -> Date? {
- for key in keys {
- guard let value = self.firstValue(key, in: object) else { continue }
- if let date = self.date(value) {
- return date
- }
- }
- return nil
- }
-
- private func shareStatus(in object: [String: Any]) -> CrawlShareStatus? {
- guard let share = self.firstObject(["share", "sharing", "published"], in: object) else { return nil }
- return CrawlShareStatus(
- enabled: (share["enabled"] as? Bool) ?? (share["repo_path"] != nil),
- repoPath: share["repo_path"] as? String,
- remote: share["remote"] as? String,
- branch: share["branch"] as? String,
- needsUpdate: share["needs_update"] as? Bool)
- }
-
- private func remoteStatus(in object: [String: Any]) -> CrawlRemoteStatus? {
- let remote = self.firstObject(["remote"], in: object) ?? object
- let endpoint = self.stringValue(["endpoint"], in: remote)
- let archive = self.stringValue(["archive"], in: remote)
- let mode = self.stringValue(["mode"], in: remote)
- guard endpoint != nil || archive != nil || mode != nil else { return nil }
- return CrawlRemoteStatus(
- enabled: self.boolValue(["enabled"], in: remote) ?? true,
- mode: mode,
- endpoint: endpoint,
- archive: archive,
- lastIngestAt: self.dateValue(["last_ingest_at"], in: remote),
- lastSyncAt: self.dateValue(["last_sync_at"], in: remote),
- needsUpdate: self.boolValue(["needs_update"], in: remote))
- }
-
- private func sqliteObjectStatus(in object: [String: Any]) -> CrawlSQLiteObjectStatus? {
- guard let sqliteObject = self.firstObject(["sqlite_object"], in: object) else { return nil }
- return CrawlSQLiteObjectStatus(
- key: self.stringValue(["key"], in: sqliteObject),
- contentType: self.stringValue(["content_type"], in: sqliteObject),
- bytes: self.intValue(["bytes", "size"], in: sqliteObject),
- uploadedAt: self.dateValue(["uploaded_at", "uploaded", "modified_at"], in: sqliteObject))
- }
-
- private func sqliteBundleStatus(in object: [String: Any]) -> CrawlSQLiteBundleStatus? {
- guard let sqliteBundle = self.firstObject(["sqlite_bundle", "bundle"], in: object) else { return nil }
- let manifest = self.firstObject(["manifest"], in: sqliteBundle) ?? sqliteBundle
- let compression = self.firstObject(["compression"], in: manifest)
- let rawObject = self.firstObject(["object"], in: manifest)
- let compressedObject = self.firstObject(["compressed_object"], in: manifest)
- let parts = self.firstValue("parts", in: manifest) as? [Any]
- let compressedBytes = self.int(sqliteBundle["compressed_bytes"] as Any)
- ?? self.int(sqliteBundle["size"] as Any)
- ?? compressedObject.flatMap { self.intValue(["bytes", "size"], in: $0) }
- return CrawlSQLiteBundleStatus(
- key: self.stringValue(["key"], in: sqliteBundle),
- contentType: self.stringValue(["content_type"], in: sqliteBundle),
- format: self.stringValue(["format"], in: manifest),
- compression: compression.flatMap { self.stringValue(["algorithm"], in: $0) },
- rawBytes: self.int(sqliteBundle["raw_bytes"] as Any)
- ?? rawObject.flatMap { self.intValue(["bytes", "size"], in: $0) },
- compressedBytes: compressedBytes,
- partCount: self.intValue(["part_count"], in: sqliteBundle) ?? parts?.count,
- uploadedAt: self.dateValue(["uploaded_at", "uploaded", "modified_at"], in: sqliteBundle),
- generatedAt: self.dateValue(["generated_at"], in: manifest))
- }
-
- private func firstObject(_ keys: [String], in object: [String: Any]) -> [String: Any]? {
- for key in keys {
- if let object = self.firstValue(key, in: object) as? [String: Any] {
- return object
- }
- }
- return nil
- }
-
- private func firstValue(_ key: String, in object: [String: Any]) -> Any? {
- if let value = object[key] { return value }
- for value in object.values {
- if let nested = value as? [String: Any], let match = self.firstValue(key, in: nested) {
- return match
- }
- }
- return nil
- }
-
- private func int(_ value: Any) -> Int? {
- if let int = value as? Int { return int }
- if let number = value as? NSNumber { return number.intValue }
- if let string = value as? String { return Int(string) }
- return nil
- }
-
- private func date(_ value: Any) -> Date? {
- if let date = value as? Date { return date }
- if let number = value as? NSNumber {
- let seconds = number.doubleValue > 99_999_999_999 ? number.doubleValue / 1_000 : number.doubleValue
- return Date(timeIntervalSince1970: seconds)
- }
- guard let string = value as? String, let trimmed = string.nilIfBlank else { return nil }
- if let date = ISO8601DateFormatter.crawlBarDate(from: trimmed) {
- return date
- }
- if let seconds = Double(trimmed) {
- return Date(timeIntervalSince1970: seconds > 99_999_999_999 ? seconds / 1_000 : seconds)
- }
- return nil
- }
-
- private func label(from key: String) -> String {
- key
- .replacingOccurrences(of: "_", with: " ")
- .split(separator: " ")
- .map { $0.prefix(1).uppercased() + $0.dropFirst() }
- .joined(separator: " ")
- }
}
diff --git a/Sources/CrawlBarCore/StatusMapperAccountStatuses.swift b/Sources/CrawlBarCore/StatusMapperAccountStatuses.swift
new file mode 100644
index 0000000..fef767e
--- /dev/null
+++ b/Sources/CrawlBarCore/StatusMapperAccountStatuses.swift
@@ -0,0 +1,157 @@
+import Foundation
+
+extension CrawlStatusMapper {
+ func birdclawStatus(_ object: [String: Any], result: CrawlCommandResult) -> CrawlAppStatus {
+ let transport = self.firstObject(["transport"], in: object) ?? object
+ let installed = self.boolValue(["installed"], in: transport)
+ let transportName = self.stringValue(["availableTransport"], in: transport)
+ ?? self.stringValue(["available_transport"], in: transport)
+ let statusText = self.stringValue(["statusText", "status_text", "summary", "message"], in: transport)
+
+ let state: CrawlAppState = .current
+ let summary = statusText ?? "birdclaw is ready"
+ var warnings = transportName.map { ["Transport: \($0)"] } ?? []
+ if installed == false, let statusText {
+ warnings.append(statusText)
+ }
+
+ return CrawlAppStatus(
+ appID: result.appID,
+ state: state,
+ summary: summary,
+ warnings: warnings)
+ }
+
+ func birdStatusText(_ result: CrawlCommandResult) -> CrawlAppStatus {
+ let output = result.stdout.nilIfBlank ?? result.stderr.nilIfBlank ?? ""
+ let lowercased = output.lowercased()
+ let hasAuthToken = lowercased.contains("[ok] auth_token")
+ || lowercased.contains("auth_token:")
+ let hasCSRFToken = lowercased.contains("[ok] ct0")
+ || lowercased.contains("ct0:")
+ let ready = lowercased.contains("ready to tweet")
+ || (hasAuthToken && hasCSRFToken)
+ let source = output.split(separator: "\n")
+ .map { $0.trimmingCharacters(in: .whitespacesAndNewlines) }
+ .first { $0.lowercased().hasPrefix("source:") }
+ let warningLines = output.split(separator: "\n")
+ .map { $0.trimmingCharacters(in: .whitespacesAndNewlines) }
+ .filter { line in
+ let lowered = line.lowercased()
+ return lowered.contains("[warn]") || lowered.hasPrefix("- ")
+ }
+ .map { line in
+ line.replacingOccurrences(of: "[warn]", with: "")
+ .trimmingCharacters(in: .whitespacesAndNewlines)
+ }
+ .map { line in
+ line.hasPrefix("- ")
+ ? String(line.dropFirst(2)).trimmingCharacters(in: .whitespacesAndNewlines)
+ : line
+ }
+ .filter { !$0.isEmpty && $0.lowercased() != "warnings:" }
+
+ if ready {
+ return CrawlAppStatus(
+ appID: result.appID,
+ state: .current,
+ summary: source.map { "X cookies available via bird (\($0.dropFirst("source:".count).trimmingCharacters(in: .whitespacesAndNewlines)))" }
+ ?? "X cookies available via bird",
+ warnings: warningLines)
+ }
+
+ return CrawlAppStatus(
+ appID: result.appID,
+ state: .needsAuth,
+ summary: "X browser cookies not found",
+ warnings: warningLines.isEmpty ? ["bird check did not find usable X cookies"] : warningLines)
+ }
+
+ func gogcliStatus(_ object: [String: Any], result: CrawlCommandResult, staleAfterSeconds _: Int?) -> CrawlAppStatus {
+ if let accounts = object["accounts"] as? [[String: Any]] {
+ let configuredAccount = accounts.first { account in
+ self.boolValue(["valid"], in: account) != false
+ && ["oauth", "service-account", "service_account", "oauth+service-account", "oauth+service_account"].contains(
+ self.stringValue(["auth"], in: account)?.lowercased() ?? "")
+ }
+ let failedAccount = accounts.first { self.boolValue(["valid"], in: $0) == false }
+ let failureSummary = failedAccount.flatMap { account in
+ self.stringValue(["error", "hint", "email"], in: account)
+ }
+ return CrawlAppStatus(
+ appID: result.appID,
+ state: configuredAccount == nil ? .needsAuth : .current,
+ summary: configuredAccount == nil ? failureSummary ?? "Google auth needs setup" : "Google auth configured")
+ }
+
+ if let status = self.statusValue(["status"], in: object), object["checks"] != nil {
+ let checks = object["checks"] as? [[String: Any]]
+ let readableTokens = checks?.first { check in
+ self.stringValue(["name"], in: check) == "tokens"
+ && self.statusValue(["status"], in: check) == .current
+ }
+ let refreshErrors = checks?.filter { check in
+ guard let name = self.stringValue(["name"], in: check) else { return false }
+ return name.hasPrefix("refresh.") && self.statusValue(["status"], in: check) == .error
+ } ?? []
+ let warnings = checks?.compactMap { check -> String? in
+ guard self.statusValue(["status"], in: check) != .current,
+ let name = self.stringValue(["name"], in: check)
+ else { return nil }
+ let detail = self.stringValue(["detail"], in: check)?.nilIfBlank
+ return [name, detail].compactMap { $0 }.joined(separator: ": ")
+ } ?? []
+ if refreshErrors.isEmpty, let readableTokens {
+ let detail = self.stringValue(["detail"], in: readableTokens)
+ let summary = detail?.split(separator: " ").first.map { "\($0) Google OAuth accounts readable" }
+ ?? "Google OAuth accounts readable"
+ return CrawlAppStatus(
+ appID: result.appID,
+ state: .current,
+ summary: summary,
+ configPath: self.gogcliConfigPath(fromChecks: checks),
+ warnings: warnings)
+ }
+ let failedCheck = checks?.first { check in
+ self.statusValue(["status"], in: check) != .current
+ }
+ let failureSummary = failedCheck.flatMap { check in
+ self.stringValue(["detail", "hint", "name"], in: check)
+ }
+ let mappedState = status == .current
+ ? CrawlAppState.current
+ : self.gogcliDoctorFailureState(failedCheck)
+ return CrawlAppStatus(
+ appID: result.appID,
+ state: mappedState,
+ summary: mappedState == .current ? "Google auth configured" : failureSummary ?? "Google auth needs setup",
+ configPath: self.gogcliConfigPath(fromChecks: checks),
+ warnings: warnings)
+ }
+
+ let account = self.firstObject(["account"], in: object) ?? [:]
+ let config = self.firstObject(["config"], in: object) ?? [:]
+ let serviceAccountConfigured = self.boolValue(["service_account_configured"], in: account) ?? false
+ let state: CrawlAppState = serviceAccountConfigured ? .current : .needsAuth
+ let summary = state == .current ? "Google service account configured" : "Google account needs auth"
+ let warnings = self.boolValue(["exists"], in: config) == false
+ ? ["gog config file not found"]
+ : []
+ return CrawlAppStatus(
+ appID: result.appID,
+ state: state,
+ summary: summary,
+ configPath: self.stringValue(["path"], in: config),
+ warnings: warnings)
+ }
+
+ func gogcliDoctorFailureState(_ check: [String: Any]?) -> CrawlAppState {
+ let name = check.flatMap { self.stringValue(["name"], in: $0) }?.lowercased() ?? ""
+ return name.contains("config") ? .needsConfig : .needsAuth
+ }
+
+ func gogcliConfigPath(fromChecks checks: [[String: Any]]?) -> String? {
+ checks?.first { self.stringValue(["name"], in: $0) == "config.path" }
+ .flatMap { self.stringValue(["detail"], in: $0) }
+ }
+}
diff --git a/Sources/CrawlBarCore/StatusMapperCrawlKit.swift b/Sources/CrawlBarCore/StatusMapperCrawlKit.swift
new file mode 100644
index 0000000..19c5680
--- /dev/null
+++ b/Sources/CrawlBarCore/StatusMapperCrawlKit.swift
@@ -0,0 +1,39 @@
+import Foundation
+
+extension CrawlStatusMapper {
+ func genericStatus(_ object: [String: Any], result: CrawlCommandResult, staleAfterSeconds: Int?) -> CrawlAppStatus {
+ let counts = self.statusCounts(in: object, fallback: self.counts(in: object))
+ let databases = self.databaseResources(in: object)
+ let remote = self.remoteStatus(in: object)
+ let lastSyncAt = self.dateValue(["last_sync_at", "updated_at", "generated_at"], in: object)
+ ?? remote?.lastSyncAt
+ ?? remote?.lastIngestAt
+ ?? self.databaseModifiedAt(databases)
+ let freshness = self.freshness(in: object, lastSyncAt: lastSyncAt, staleAfterSeconds: staleAfterSeconds)
+ return CrawlAppStatus(
+ appID: result.appID,
+ state: self.statusState(in: object, lastSyncAt: lastSyncAt, freshness: freshness, fallback: .current, staleAfterSeconds: staleAfterSeconds),
+ summary: self.stringValue(["summary", "message"], in: object) ?? self.summary(from: counts, fallback: "Status is current"),
+ configPath: self.stringValue(["config_path", "config"], in: object),
+ databasePath: self.stringValue(["db_path", "database_path"], in: object),
+ databaseBytes: self.intValue(["db_bytes", "database_bytes"], in: object),
+ walBytes: self.intValue(["wal_bytes"], in: object),
+ lastSyncAt: lastSyncAt,
+ lastImportAt: self.dateValue(["last_import_at"], in: object),
+ lastExportAt: self.dateValue(["last_export_at"], in: object),
+ counts: counts,
+ databases: databases,
+ freshness: freshness,
+ share: self.shareStatus(in: object),
+ remote: remote,
+ sqliteObject: self.sqliteObjectStatus(in: object),
+ sqliteBundle: self.sqliteBundleStatus(in: object))
+ }
+
+ func isCrawlKitStatus(_ object: [String: Any]) -> Bool {
+ if let schema = self.stringValue(["schema_version"], in: object), schema.hasPrefix("crawlkit.control.") {
+ return true
+ }
+ return self.firstValue("databases", in: object) != nil && self.firstValue("counts", in: object) != nil
+ }
+}
diff --git a/Sources/CrawlBarCore/StatusMapperJSON.swift b/Sources/CrawlBarCore/StatusMapperJSON.swift
new file mode 100644
index 0000000..4d6e555
--- /dev/null
+++ b/Sources/CrawlBarCore/StatusMapperJSON.swift
@@ -0,0 +1,122 @@
+import Foundation
+
+extension CrawlStatusMapper {
+ func parseObject(_ text: String) -> [String: Any]? {
+ guard let data = text.data(using: .utf8), !data.isEmpty else { return nil }
+ return (try? JSONSerialization.jsonObject(with: data)) as? [String: Any]
+ }
+
+ func intValue(_ keys: [String], in object: [String: Any]) -> Int? {
+ for key in keys {
+ if let value = self.firstValue(key, in: object), let int = self.int(value) {
+ return int
+ }
+ }
+ return nil
+ }
+
+ func boolValue(_ keys: [String], in object: [String: Any]) -> Bool? {
+ for key in keys {
+ guard let value = self.firstValue(key, in: object) else { continue }
+ if let bool = value as? Bool { return bool }
+ if let number = value as? NSNumber { return number.boolValue }
+ if let string = value as? String {
+ switch string.lowercased() {
+ case "true", "yes", "1":
+ return true
+ case "false", "no", "0":
+ return false
+ default:
+ continue
+ }
+ }
+ }
+ return nil
+ }
+
+ func stringValue(_ keys: [String], in object: [String: Any]) -> String? {
+ for key in keys {
+ if let value = self.firstValue(key, in: object) as? String, let string = value.nilIfBlank {
+ return string
+ }
+ }
+ return nil
+ }
+
+ func statusValue(_ keys: [String], in object: [String: Any]) -> CrawlAppState? {
+ guard let rawValue = self.stringValue(keys, in: object) else { return nil }
+ if let state = CrawlAppState(rawValue: rawValue) {
+ return state
+ }
+ switch rawValue.lowercased() {
+ case "ok", "success", "healthy", "ready":
+ return .current
+ case "warn", "warning", "degraded":
+ return .stale
+ case "failed", "failure":
+ return .error
+ default:
+ return nil
+ }
+ }
+
+ func dateValue(_ keys: [String], in object: [String: Any]) -> Date? {
+ for key in keys {
+ guard let value = self.firstValue(key, in: object) else { continue }
+ if let date = self.date(value) {
+ return date
+ }
+ }
+ return nil
+ }
+
+ func firstObject(_ keys: [String], in object: [String: Any]) -> [String: Any]? {
+ for key in keys {
+ if let object = self.firstValue(key, in: object) as? [String: Any] {
+ return object
+ }
+ }
+ return nil
+ }
+
+ func firstValue(_ key: String, in object: [String: Any]) -> Any? {
+ if let value = object[key] { return value }
+ for value in object.values {
+ if let nested = value as? [String: Any], let match = self.firstValue(key, in: nested) {
+ return match
+ }
+ }
+ return nil
+ }
+
+ func int(_ value: Any) -> Int? {
+ if let int = value as? Int { return int }
+ if let number = value as? NSNumber { return number.intValue }
+ if let string = value as? String { return Int(string) }
+ return nil
+ }
+
+ func date(_ value: Any) -> Date? {
+ if let date = value as? Date { return date }
+ if let number = value as? NSNumber {
+ let seconds = number.doubleValue > 99_999_999_999 ? number.doubleValue / 1_000 : number.doubleValue
+ return Date(timeIntervalSince1970: seconds)
+ }
+ guard let string = value as? String, let trimmed = string.nilIfBlank else { return nil }
+ if let date = ISO8601DateFormatter.crawlBarDate(from: trimmed) {
+ return date
+ }
+ if let seconds = Double(trimmed) {
+ return Date(timeIntervalSince1970: seconds > 99_999_999_999 ? seconds / 1_000 : seconds)
+ }
+ return nil
+ }
+
+ func label(from key: String) -> String {
+ key
+ .replacingOccurrences(of: "_", with: " ")
+ .split(separator: " ")
+ .map { $0.prefix(1).uppercased() + $0.dropFirst() }
+ .joined(separator: " ")
+ }
+}
diff --git a/Sources/CrawlBarCore/StatusMapperResources.swift b/Sources/CrawlBarCore/StatusMapperResources.swift
new file mode 100644
index 0000000..d7dfdad
--- /dev/null
+++ b/Sources/CrawlBarCore/StatusMapperResources.swift
@@ -0,0 +1,177 @@
+import Foundation
+
+extension CrawlStatusMapper {
+ func count(_ id: String, _ label: String, _ keys: [String]) -> (String, String, [String]) {
+ (id, label, keys)
+ }
+
+ func value(_ spec: (String, String, [String]), in object: [String: Any]) -> CrawlCount? {
+ guard let value = self.intValue(spec.2, in: object) else { return nil }
+ return CrawlCount(id: spec.0, label: spec.1, value: value)
+ }
+
+ func counts(in object: [String: Any]) -> [CrawlCount] {
+ guard let counts = self.firstObject(["counts", "stats"], in: object) else { return [] }
+ return counts.compactMap { key, value in
+ guard let int = self.int(value) else { return nil }
+ return CrawlCount(id: key, label: self.label(from: key), value: int)
+ }
+ .sorted { $0.id < $1.id }
+ }
+
+ func state(lastSyncAt: Date?, fallback: CrawlAppState, staleAfterSeconds: Int?) -> CrawlAppState {
+ guard let lastSyncAt else { return fallback }
+ let threshold = staleAfterSeconds ?? Self.defaultStaleAfterSeconds
+ return Date().timeIntervalSince(lastSyncAt) > TimeInterval(threshold) ? .stale : .current
+ }
+
+ func freshness(lastSyncAt: Date?, staleAfterSeconds: Int?) -> CrawlFreshness? {
+ guard let lastSyncAt else { return nil }
+ let threshold = staleAfterSeconds ?? Self.defaultStaleAfterSeconds
+ let ageSeconds = max(0, Int(Date().timeIntervalSince(lastSyncAt)))
+ return CrawlFreshness(
+ status: ageSeconds > threshold ? .stale : .current,
+ ageSeconds: ageSeconds,
+ staleAfterSeconds: threshold)
+ }
+
+ func freshness(in object: [String: Any], lastSyncAt: Date?, staleAfterSeconds: Int?) -> CrawlFreshness? {
+ if let freshness = self.firstObject(["freshness"], in: object),
+ let status = self.statusValue(["status", "state"], in: freshness)
+ {
+ return CrawlFreshness(
+ status: status,
+ ageSeconds: self.intValue(["age_seconds"], in: freshness),
+ staleAfterSeconds: self.intValue(["stale_after_seconds"], in: freshness) ?? staleAfterSeconds)
+ }
+ return self.freshness(lastSyncAt: lastSyncAt, staleAfterSeconds: staleAfterSeconds)
+ }
+
+ func summary(from counts: [CrawlCount], fallback: String) -> String {
+ let visible = counts.prefix(2).map { "\($0.value) \($0.label.lowercased())" }
+ return visible.isEmpty ? fallback : visible.joined(separator: ", ")
+ }
+
+ func statusState(
+ in object: [String: Any],
+ lastSyncAt: Date?,
+ freshness: CrawlFreshness?,
+ fallback: CrawlAppState,
+ staleAfterSeconds: Int?)
+ -> CrawlAppState
+ {
+ if let state = self.statusValue(["state", "status"], in: object)
+ {
+ return state
+ }
+ return freshness?.status ?? self.state(lastSyncAt: lastSyncAt, fallback: fallback, staleAfterSeconds: staleAfterSeconds)
+ }
+
+ func statusCounts(in object: [String: Any], fallback: [CrawlCount]) -> [CrawlCount] {
+ let declared = self.countArray(["counts"], in: object)
+ return declared.isEmpty ? fallback : declared
+ }
+
+ func countArray(_ keys: [String], in object: [String: Any]) -> [CrawlCount] {
+ for key in keys {
+ guard let array = self.firstValue(key, in: object) as? [Any] else { continue }
+ let counts = array.compactMap { item -> CrawlCount? in
+ guard let item = item as? [String: Any],
+ let id = self.stringValue(["id"], in: item),
+ let value = self.intValue(["value", "count"], in: item)
+ else { return nil }
+ return CrawlCount(
+ id: id,
+ label: self.stringValue(["label"], in: item) ?? self.label(from: id),
+ value: value)
+ }
+ if !counts.isEmpty { return counts }
+ }
+ return []
+ }
+
+ func databaseResources(in object: [String: Any]) -> [CrawlDatabaseResource] {
+ guard let array = self.firstValue("databases", in: object) as? [Any] else { return [] }
+ return array.compactMap { item -> CrawlDatabaseResource? in
+ guard let item = item as? [String: Any] else { return nil }
+ let path = self.stringValue(["path"], in: item)
+ guard let id = self.stringValue(["id"], in: item) ?? path else { return nil }
+ let kindValue = self.stringValue(["kind"], in: item) ?? CrawlDatabaseKind.sqlite.rawValue
+ return CrawlDatabaseResource(
+ id: id,
+ label: self.stringValue(["label"], in: item) ?? URL(fileURLWithPath: id).lastPathComponent,
+ kind: CrawlDatabaseKind(rawValue: kindValue) ?? .remote,
+ role: self.stringValue(["role"], in: item),
+ path: path,
+ endpoint: self.stringValue(["endpoint"], in: item),
+ archive: self.stringValue(["archive"], in: item),
+ isPrimary: self.boolValue(["is_primary", "primary"], in: item) ?? false,
+ bytes: self.intValue(["bytes", "size_bytes"], in: item),
+ modifiedAt: self.dateValue(["modified_at", "updated_at"], in: item),
+ counts: self.countArray(["counts"], in: item))
+ }
+ }
+
+ func databaseModifiedAt(_ databases: [CrawlDatabaseResource]) -> Date? {
+ databases.first(where: { $0.isPrimary })?.modifiedAt
+ ?? databases.compactMap(\.modifiedAt).max()
+ }
+
+ func shareStatus(in object: [String: Any]) -> CrawlShareStatus? {
+ guard let share = self.firstObject(["share", "sharing", "published"], in: object) else { return nil }
+ return CrawlShareStatus(
+ enabled: (share["enabled"] as? Bool) ?? (share["repo_path"] != nil),
+ repoPath: share["repo_path"] as? String,
+ remote: share["remote"] as? String,
+ branch: share["branch"] as? String,
+ needsUpdate: share["needs_update"] as? Bool)
+ }
+
+ func remoteStatus(in object: [String: Any]) -> CrawlRemoteStatus? {
+ let remote = self.firstObject(["remote"], in: object) ?? object
+ let endpoint = self.stringValue(["endpoint"], in: remote)
+ let archive = self.stringValue(["archive"], in: remote)
+ let mode = self.stringValue(["mode"], in: remote)
+ guard endpoint != nil || archive != nil || mode != nil else { return nil }
+ return CrawlRemoteStatus(
+ enabled: self.boolValue(["enabled"], in: remote) ?? true,
+ mode: mode,
+ endpoint: endpoint,
+ archive: archive,
+ lastIngestAt: self.dateValue(["last_ingest_at"], in: remote),
+ lastSyncAt: self.dateValue(["last_sync_at"], in: remote),
+ needsUpdate: self.boolValue(["needs_update"], in: remote))
+ }
+
+ func sqliteObjectStatus(in object: [String: Any]) -> CrawlSQLiteObjectStatus? {
+ guard let sqliteObject = self.firstObject(["sqlite_object"], in: object) else { return nil }
+ return CrawlSQLiteObjectStatus(
+ key: self.stringValue(["key"], in: sqliteObject),
+ contentType: self.stringValue(["content_type"], in: sqliteObject),
+ bytes: self.intValue(["bytes", "size"], in: sqliteObject),
+ uploadedAt: self.dateValue(["uploaded_at", "uploaded", "modified_at"], in: sqliteObject))
+ }
+
+ func sqliteBundleStatus(in object: [String: Any]) -> CrawlSQLiteBundleStatus? {
+ guard let sqliteBundle = self.firstObject(["sqlite_bundle", "bundle"], in: object) else { return nil }
+ let manifest = self.firstObject(["manifest"], in: sqliteBundle) ?? sqliteBundle
+ let compression = self.firstObject(["compression"], in: manifest)
+ let rawObject = self.firstObject(["object"], in: manifest)
+ let compressedObject = self.firstObject(["compressed_object"], in: manifest)
+ let parts = self.firstValue("parts", in: manifest) as? [Any]
+ let compressedBytes = self.int(sqliteBundle["compressed_bytes"] as Any)
+ ?? self.int(sqliteBundle["size"] as Any)
+ ?? compressedObject.flatMap { self.intValue(["bytes", "size"], in: $0) }
+ return CrawlSQLiteBundleStatus(
+ key: self.stringValue(["key"], in: sqliteBundle),
+ contentType: self.stringValue(["content_type"], in: sqliteBundle),
+ format: self.stringValue(["format"], in: manifest),
+ compression: compression.flatMap { self.stringValue(["algorithm"], in: $0) },
+ rawBytes: self.int(sqliteBundle["raw_bytes"] as Any)
+ ?? rawObject.flatMap { self.intValue(["bytes", "size"], in: $0) },
+ compressedBytes: compressedBytes,
+ partCount: self.intValue(["part_count"], in: sqliteBundle) ?? parts?.count,
+ uploadedAt: self.dateValue(["uploaded_at", "uploaded", "modified_at"], in: sqliteBundle),
+ generatedAt: self.dateValue(["generated_at"], in: manifest))
+ }
+}
diff --git a/Sources/CrawlBarCore/StatusMapperSourceStatuses.swift b/Sources/CrawlBarCore/StatusMapperSourceStatuses.swift
new file mode 100644
index 0000000..0718d27
--- /dev/null
+++ b/Sources/CrawlBarCore/StatusMapperSourceStatuses.swift
@@ -0,0 +1,158 @@
+import Foundation
+
+extension CrawlStatusMapper {
+ func gitcrawlStatus(_ object: [String: Any], result: CrawlCommandResult, staleAfterSeconds: Int?) -> CrawlAppStatus {
+ let counts = [
+ self.count("threads", "Threads", ["thread_count", "threads"]),
+ self.count("open_threads", "Open Threads", ["open_thread_count", "open_threads"]),
+ self.count("clusters", "Clusters", ["cluster_count", "clusters"]),
+ self.count("repositories", "Repositories", ["repo_count", "repository_count", "repositories"]),
+ ].compactMap { self.value($0, in: object) }
+
+ let remote = self.remoteStatus(in: object)
+ let lastSyncAt = self.dateValue(["last_sync_at", "updated_at", "generated_at"], in: object)
+ ?? remote?.lastSyncAt
+ ?? remote?.lastIngestAt
+ return CrawlAppStatus(
+ appID: result.appID,
+ state: self.state(lastSyncAt: lastSyncAt, fallback: .current, staleAfterSeconds: staleAfterSeconds),
+ summary: self.summary(from: counts, fallback: "Git crawl status is current"),
+ configPath: self.stringValue(["config_path", "config"], in: object),
+ databasePath: self.stringValue(["db_path", "database_path", "database"], in: object),
+ lastSyncAt: lastSyncAt,
+ counts: counts,
+ freshness: self.freshness(in: object, lastSyncAt: lastSyncAt, staleAfterSeconds: staleAfterSeconds),
+ share: self.shareStatus(in: object),
+ remote: remote,
+ sqliteObject: self.sqliteObjectStatus(in: object),
+ sqliteBundle: self.sqliteBundleStatus(in: object))
+ }
+
+ func slacrawlStatus(_ object: [String: Any], result: CrawlCommandResult, staleAfterSeconds: Int?) -> CrawlAppStatus {
+ let flatCounts = [
+ self.count("workspaces", "Workspaces", ["workspace_count", "workspaces"]),
+ self.count("channels", "Channels", ["channel_count", "channels"]),
+ self.count("users", "Users", ["user_count", "users"]),
+ self.count("messages", "Messages", ["message_count", "messages"]),
+ ].compactMap { self.value($0, in: object) }
+
+ let counts = self.statusCounts(in: object, fallback: flatCounts)
+ let databases = self.databaseResources(in: object)
+ let remote = self.remoteStatus(in: object)
+ let lastSyncAt = self.dateValue(["last_sync_at", "latest_message_at", "updated_at"], in: object)
+ ?? remote?.lastSyncAt
+ ?? remote?.lastIngestAt
+ ?? self.databaseModifiedAt(databases)
+ let freshness = self.freshness(in: object, lastSyncAt: lastSyncAt, staleAfterSeconds: staleAfterSeconds)
+ return CrawlAppStatus(
+ appID: result.appID,
+ state: self.statusState(in: object, lastSyncAt: lastSyncAt, freshness: freshness, fallback: .current, staleAfterSeconds: staleAfterSeconds),
+ summary: self.stringValue(["summary", "message"], in: object) ?? self.summary(from: counts, fallback: "Slack crawl status is current"),
+ configPath: self.stringValue(["config_path", "config"], in: object),
+ databasePath: self.stringValue(["db_path", "database_path", "database"], in: object),
+ databaseBytes: self.intValue(["db_bytes", "database_bytes"], in: object),
+ lastSyncAt: lastSyncAt,
+ counts: counts,
+ databases: databases,
+ freshness: freshness,
+ share: self.shareStatus(in: object),
+ remote: remote,
+ sqliteObject: self.sqliteObjectStatus(in: object),
+ sqliteBundle: self.sqliteBundleStatus(in: object))
+ }
+
+ func discrawlStatus(_ object: [String: Any], result: CrawlCommandResult, staleAfterSeconds: Int?) -> CrawlAppStatus {
+ let flatCounts = [
+ self.count("guilds", "Guilds", ["guild_count", "guilds"]),
+ self.count("channels", "Channels", ["channel_count", "channels"]),
+ self.count("threads", "Threads", ["thread_count", "threads"]),
+ self.count("messages", "Messages", ["message_count", "messages"]),
+ self.count("members", "Members", ["member_count", "members"]),
+ self.count("embedding_backlog", "Embedding Backlog", ["embedding_backlog"]),
+ ].compactMap { self.value($0, in: object) }
+
+ let counts = self.statusCounts(in: object, fallback: flatCounts)
+ let databases = self.databaseResources(in: object)
+ let remote = self.remoteStatus(in: object)
+ let lastSyncAt = self.dateValue(["last_sync_at", "latest_message_at", "updated_at"], in: object)
+ ?? remote?.lastSyncAt
+ ?? remote?.lastIngestAt
+ ?? self.databaseModifiedAt(databases)
+ let freshness = self.freshness(in: object, lastSyncAt: lastSyncAt, staleAfterSeconds: staleAfterSeconds)
+ return CrawlAppStatus(
+ appID: result.appID,
+ state: self.statusState(in: object, lastSyncAt: lastSyncAt, freshness: freshness, fallback: .current, staleAfterSeconds: staleAfterSeconds),
+ summary: self.stringValue(["summary", "message"], in: object) ?? self.summary(from: counts, fallback: "Discord crawl status is current"),
+ configPath: self.stringValue(["config_path", "config"], in: object),
+ databasePath: self.stringValue(["db_path", "database_path", "database"], in: object),
+ databaseBytes: self.intValue(["db_bytes", "database_bytes"], in: object),
+ lastSyncAt: lastSyncAt,
+ counts: counts,
+ databases: databases,
+ freshness: freshness,
+ share: self.shareStatus(in: object),
+ remote: remote,
+ sqliteObject: self.sqliteObjectStatus(in: object),
+ sqliteBundle: self.sqliteBundleStatus(in: object))
+ }
+
+ func telecrawlStatus(_ object: [String: Any], result: CrawlCommandResult, staleAfterSeconds: Int?) -> CrawlAppStatus {
+ let counts = [
+ self.count("messages", "Messages", ["message_count", "messages"]),
+ self.count("chats", "Chats", ["chat_count", "chats"]),
+ self.count("folders", "Folders", ["folder_count", "folders"]),
+ self.count("topics", "Topics", ["topic_count", "topics"]),
+ self.count("unread_chats", "Unread Chats", ["unread_chat_count", "unread_chats"]),
+ self.count("unread_messages", "Unread Messages", ["unread_message_count", "unread_messages"]),
+ self.count("media_messages", "Media Messages", ["media_message_count", "media_messages"]),
+ ].compactMap { self.value($0, in: object) }
+
+ let lastSyncAt = self.dateValue(["last_sync_at", "last_import_at", "updated_at"], in: object)
+ let freshness = self.freshness(in: object, lastSyncAt: lastSyncAt, staleAfterSeconds: staleAfterSeconds)
+ return CrawlAppStatus(
+ appID: result.appID,
+ state: self.statusState(in: object, lastSyncAt: lastSyncAt, freshness: freshness, fallback: .current, staleAfterSeconds: staleAfterSeconds),
+ summary: self.stringValue(["summary", "message"], in: object) ?? self.summary(from: counts, fallback: "Telegram crawl status is current"),
+ configPath: self.stringValue(["config_path", "config"], in: object),
+ databasePath: self.stringValue(["db_path", "database_path", "database"], in: object),
+ databaseBytes: self.intValue(["db_bytes", "database_bytes"], in: object),
+ lastSyncAt: lastSyncAt,
+ lastImportAt: self.dateValue(["last_import_at"], in: object),
+ counts: counts,
+ freshness: freshness,
+ share: self.shareStatus(in: object))
+ }
+
+ func notcrawlStatus(_ object: [String: Any], result: CrawlCommandResult, staleAfterSeconds: Int?) -> CrawlAppStatus {
+ let counts = [
+ self.count("spaces", "Spaces", ["space_count", "spaces"]),
+ self.count("users", "Users", ["user_count", "users"]),
+ self.count("teams", "Teams", ["team_count", "teams"]),
+ self.count("pages", "Pages", ["page_count", "pages"]),
+ self.count("blocks", "Blocks", ["block_count", "blocks"]),
+ self.count("collections", "Collections", ["collection_count", "collections"]),
+ self.count("comments", "Comments", ["comment_count", "comments"]),
+ self.count("raw_records", "Raw Records", ["raw_record_count", "raw_records"]),
+ ].compactMap { self.value($0, in: object) }
+
+ let remote = self.remoteStatus(in: object)
+ let lastSyncAt = self.dateValue(["last_sync_at", "last_import_at", "updated_at"], in: object)
+ ?? remote?.lastSyncAt
+ ?? remote?.lastIngestAt
+ return CrawlAppStatus(
+ appID: result.appID,
+ state: self.state(lastSyncAt: lastSyncAt, fallback: .current, staleAfterSeconds: staleAfterSeconds),
+ summary: self.summary(from: counts, fallback: "Notion crawl status is current"),
+ configPath: self.stringValue(["config_path", "config"], in: object),
+ databasePath: self.stringValue(["db_path", "database_path", "database"], in: object),
+ databaseBytes: self.intValue(["db_bytes", "database_bytes"], in: object),
+ walBytes: self.intValue(["wal_bytes"], in: object),
+ lastSyncAt: lastSyncAt,
+ counts: counts,
+ freshness: self.freshness(in: object, lastSyncAt: lastSyncAt, staleAfterSeconds: staleAfterSeconds),
+ share: self.shareStatus(in: object),
+ remote: remote,
+ sqliteObject: self.sqliteObjectStatus(in: object),
+ sqliteBundle: self.sqliteBundleStatus(in: object))
+ }
+}
diff --git a/Sources/CrawlBarCore/StatusMapperWhatsApp.swift b/Sources/CrawlBarCore/StatusMapperWhatsApp.swift
new file mode 100644
index 0000000..1a22901
--- /dev/null
+++ b/Sources/CrawlBarCore/StatusMapperWhatsApp.swift
@@ -0,0 +1,126 @@
+import Foundation
+
+extension CrawlStatusMapper {
+ func wacliStatus(_ object: [String: Any], result: CrawlCommandResult, staleAfterSeconds: Int?) -> CrawlAppStatus {
+ let data = self.firstObject(["data"], in: object) ?? object
+ let isAuthenticated = self.boolValue(["authenticated"], in: data) ?? true
+ let storeError = self.stringValue(["store_error"], in: data)?.nilIfBlank
+ if let storeError, (isAuthenticated || !Self.isWacliFirstRunStoreError(storeError)) {
+ return CrawlAppStatus(
+ appID: result.appID,
+ state: .error,
+ summary: storeError,
+ errors: [storeError])
+ }
+ if !isAuthenticated {
+ return CrawlAppStatus(
+ appID: result.appID,
+ state: .needsAuth,
+ summary: "WhatsApp auth needs setup")
+ }
+ let store = self.firstObject(["store"], in: data) ?? data
+ let counts = [
+ self.count("messages", "Messages", ["messages", "message_count"]),
+ self.count("chats", "Chats", ["chats", "chat_count"]),
+ self.count("contacts", "Contacts", ["contacts", "contact_count"]),
+ self.count("groups", "Groups", ["groups", "group_count"]),
+ ].compactMap { self.value($0, in: store) }
+ let lastSyncAt = self.dateValue(["last_sync_at"], in: store)
+ ?? self.dateValue(["last_sync_at", "updated_at"], in: data)
+ let freshness = self.freshness(in: object, lastSyncAt: lastSyncAt, staleAfterSeconds: staleAfterSeconds)
+ let storeDir = self.stringValue(["store_dir"], in: data)
+ let state: CrawlAppState
+ if self.boolValue(["success"], in: object) == false {
+ state = .error
+ } else {
+ state = self.statusValue(["state", "status"], in: data)
+ ?? self.statusValue(["state", "status"], in: object)
+ ?? self.statusState(in: object, lastSyncAt: lastSyncAt, freshness: freshness, fallback: .current, staleAfterSeconds: staleAfterSeconds)
+ }
+ return CrawlAppStatus(
+ appID: result.appID,
+ state: state,
+ summary: self.summary(from: counts, fallback: state == .error ? self.stringValue(["error"], in: object) ?? "WhatsApp diagnostics failed" : "WhatsApp archive is current"),
+ configPath: storeDir.map(Self.wacliConfigPath(storeDir:)),
+ databasePath: storeDir.map(Self.wacliDatabasePath(storeDir:)),
+ lastSyncAt: lastSyncAt,
+ counts: counts,
+ databases: storeDir.map { Self.wacliDatabaseResources(storeDir: $0, counts: counts) } ?? [],
+ freshness: freshness,
+ warnings: self.wacliWarnings(in: data),
+ errors: self.wacliErrors(in: object))
+ }
+
+ func isWacliManifest(_ manifest: CrawlAppManifest) -> Bool {
+ manifest.id == BuiltInCrawlApps.wacliID
+ || manifest.id.rawValue.hasPrefix("wacli-")
+ || manifest.binary.name == "wacli"
+ }
+
+ func wacliWarnings(in object: [String: Any]) -> [String] {
+ var warnings: [String] = []
+ if self.boolValue(["lock_held"], in: object) == true,
+ let state = self.stringValue(["connection_state"], in: object)
+ {
+ warnings.append("Store is locked by \(state)")
+ }
+ if self.boolValue(["fts_enabled"], in: object) == false {
+ warnings.append("Full-text search is not enabled")
+ }
+ return warnings
+ }
+
+ func wacliErrors(in object: [String: Any]) -> [String] {
+ guard self.boolValue(["success"], in: object) == false else { return [] }
+ if let error = self.stringValue(["error"], in: object) {
+ return [error]
+ }
+ return ["wacli doctor reported failure"]
+ }
+
+ static func wacliConfigPath(storeDir: String) -> String {
+ let storeURL = URL(fileURLWithPath: storeDir)
+ if storeURL.deletingLastPathComponent().lastPathComponent == "accounts" {
+ return storeURL
+ .deletingLastPathComponent()
+ .deletingLastPathComponent()
+ .appendingPathComponent("config.yaml")
+ .path
+ }
+ return storeURL.appendingPathComponent("config.yaml").path
+ }
+
+ static func wacliDatabasePath(storeDir: String) -> String {
+ URL(fileURLWithPath: storeDir).appendingPathComponent("wacli.db").path
+ }
+
+ static func wacliDatabaseResources(storeDir: String, counts: [CrawlCount]) -> [CrawlDatabaseResource] {
+ let databasePath = Self.wacliDatabasePath(storeDir: storeDir)
+ var resources = [
+ CrawlDatabaseResource(
+ id: databasePath,
+ label: "WhatsApp SQLite database",
+ kind: .sqlite,
+ path: databasePath,
+ isPrimary: true,
+ counts: counts),
+ ]
+ if databasePath != storeDir {
+ resources.append(CrawlDatabaseResource(
+ id: storeDir,
+ label: "WhatsApp store",
+ kind: .logical,
+ path: storeDir,
+ counts: counts))
+ }
+ return resources
+ }
+
+ static func isWacliFirstRunStoreError(_ error: String) -> Bool {
+ let lowercased = error.lowercased()
+ return lowercased.contains("no such file")
+ || lowercased.contains("not found")
+ || lowercased.contains("missing")
+ || lowercased.contains("uninitialized")
+ }
+}
diff --git a/Sources/CrawlBarSelfTest/SelfTest.swift b/Sources/CrawlBarSelfTest/SelfTest.swift
index 2d7c91f..157cf74 100644
--- a/Sources/CrawlBarSelfTest/SelfTest.swift
+++ b/Sources/CrawlBarSelfTest/SelfTest.swift
@@ -30,1647 +30,4 @@ enum CrawlBarSelfTest {
try Self.testRedactorScrubsSecrets()
print("crawlbar selftest ok")
}
-
- private static func testAppIDSortsByRawValue() throws {
- try Self.expect(
- [CrawlAppID(rawValue: "b"), CrawlAppID(rawValue: "a")].sorted().map(\.rawValue) == ["a", "b"],
- "app ids sort by raw value")
- }
-
- private static func testDefaultConfigNormalizesBuiltInApps() throws {
- let config = CrawlBarConfig(apps: []).normalized()
- try Self.expect(config.version == CrawlBarConfig.currentVersion, "config version normalizes")
- try Self.expect(config.apps.map(\.id) == BuiltInCrawlApps.all.map(\.id), "built-in apps are present")
- try Self.expect(config.appConfig(for: BuiltInCrawlApps.gogcliID)?.enabled == true, "new Google app normalizes enabled")
- try Self.expect(config.appConfig(for: BuiltInCrawlApps.gogcliID)?.showInMenuBar == true, "new Google app appears in menu bar")
- let oldConfig = CrawlBarConfig(
- version: 1,
- apps: [CrawlBarAppConfig(id: BuiltInCrawlApps.wacliID, enabled: false, showInMenuBar: false)]).normalized()
- try Self.expect(oldConfig.appConfig(for: BuiltInCrawlApps.wacliID)?.enabled == true, "newly available apps migrate from forced disabled")
- let v2Config = CrawlBarConfig(
- version: 2,
- apps: [CrawlBarAppConfig(id: BuiltInCrawlApps.birdclawID, enabled: false, showInMenuBar: false)]).normalized()
- try Self.expect(v2Config.appConfig(for: BuiltInCrawlApps.birdclawID)?.enabled == true, "newly available Birdclaw migrates from forced disabled")
- try Self.expect(config.manifestDirectories == ["~/.crawlbar/apps"], "manifest directory default is present")
- }
-
- private static func testConfigStoreRoundTrips() throws {
- let directory = FileManager.default.temporaryDirectory
- .appendingPathComponent("crawlbar-config-\(UUID().uuidString)", isDirectory: true)
- try FileManager.default.createDirectory(at: directory, withIntermediateDirectories: true)
- defer { try? FileManager.default.removeItem(at: directory) }
-
- let url = directory.appendingPathComponent("config.json")
- let store = CrawlBarConfigStore(fileURL: url)
- let config = CrawlBarConfig(
- refreshFrequency: .hourly,
- apps: [CrawlBarAppConfig(
- id: BuiltInCrawlApps.gitcrawlID,
- enabled: false,
- configValues: ["embedding_model": "text-embedding-3-large"])])
-
- try store.save(config)
- guard let loaded = try store.load() else {
- throw SelfTestError.failed("config loads after save")
- }
-
- try Self.expect(loaded.refreshFrequency == .hourly, "refresh frequency round trips")
- try Self.expect(loaded.appConfig(for: BuiltInCrawlApps.gitcrawlID)?.enabled == false, "app enablement round trips")
- try Self.expect(loaded.appConfig(for: BuiltInCrawlApps.gitcrawlID)?.configValues["embedding_model"] == "text-embedding-3-large", "app config values round trip")
- try Self.expect(loaded.apps.count == BuiltInCrawlApps.all.count, "config store normalizes built-ins")
- }
-
- private static func testExternalManifestCatalog() throws {
- let directory = FileManager.default.temporaryDirectory
- .appendingPathComponent("crawlbar-manifest-\(UUID().uuidString)", isDirectory: true)
- try FileManager.default.createDirectory(at: directory, withIntermediateDirectories: true)
- defer { try? FileManager.default.removeItem(at: directory) }
-
- let manifest = CrawlAppManifest(
- id: CrawlAppID(rawValue: "customcrawl"),
- displayName: "Custom Crawl",
- description: "A custom crawl app",
- binary: .init(name: "customcrawl"),
- branding: .init(symbolName: "square.grid.2x2", accentColor: "#123456"),
- paths: .init(defaultConfig: "~/.customcrawl/config.toml"),
- commands: ["status": ["status", "--json"]],
- capabilities: [.status])
- let data = try CrawlCoding.makeJSONEncoder().encode(manifest)
- try data.write(to: directory.appendingPathComponent("customcrawl.json"))
- try Data("""
- {
- "schema_version": "crawlkit.control.v1",
- "id": "objectcrawl",
- "display_name": "Object Crawl",
- "description": "A crawlkit manifest",
- "binary": {"name": "objectcrawl"},
- "branding": {"symbol_name": "tray", "accent_color": "#123456"},
- "paths": {"default_config": "~/.objectcrawl/config.toml"},
- "commands": {
- "status": {"title": "Status", "argv": ["objectcrawl", "status", "--json"], "json": true},
- "sync": {"title": "Sync", "argv": ["objectcrawl", "sync", "--json"], "json": true, "mutates": true},
- "query": {"title": "Query", "argv": ["objectcrawl", "--json", "sql", "select count(*) from things"], "json": true},
- "tap": {"title": "Desktop", "argv": ["objectcrawl", "tap", "--json"], "json": true, "mutates": true}
- },
- "capabilities": ["metadata", "status", "sync", "tap", "git-share"],
- "privacy": {"exports_secrets": false}
- }
- """.utf8).write(to: directory.appendingPathComponent("objectcrawl.json"))
- try Data("""
- {
- "id": "cachecrawl",
- "display_name": "Cache Crawl",
- "description": "A desktop cache test manifest",
- "binary": {"name": "cachecrawl"},
- "branding": {"symbol_name": "tray", "accent_color": "#123456"},
- "paths": {"default_config": "~/.cachecrawl/config.toml"},
- "commands": {
- "desktop-cache-import": ["sync", "--source", "desktop-cache"]
- }
- }
- """.utf8).write(to: directory.appendingPathComponent("cachecrawl.json"))
- try Data("{ bad json".utf8).write(to: directory.appendingPathComponent("broken.json"))
-
- let config = CrawlBarConfig(manifestDirectories: [directory.path])
- let catalog = CrawlManifestCatalog()
- let manifests = catalog.manifests(config: config)
- let diagnostics = catalog.diagnostics(config: config)
- let configURL = directory.appendingPathComponent("config.json")
- let store = CrawlBarConfigStore(fileURL: configURL)
- try store.save(config)
- let registry = CrawlAppRegistry(configStore: store, catalog: catalog)
- let installations = try registry.installations(includeDisabled: true)
- try Self.expect(manifests.contains { $0.id == manifest.id }, "external manifests load from disk")
- guard let objectManifest = manifests.first(where: { $0.id.rawValue == "objectcrawl" }) else {
- throw SelfTestError.failed("crawlkit command-object manifests load from disk")
- }
- try Self.expect(objectManifest.commands["status"] == ["status", "--json"], "crawlkit command argv strips binary")
- try Self.expect(objectManifest.commands["query"] == ["--json", "sql"], "crawlkit query sample SQL is stripped")
- try Self.expect(objectManifest.commands["refresh"] == ["sync", "--json"], "crawlkit sync command aliases to refresh")
- try Self.expect(objectManifest.commands["desktop-cache-import"] == ["tap", "--json"], "crawlkit tap command aliases to desktop cache import")
- try Self.expect(objectManifest.capabilities.contains(.refresh), "crawlkit sync capability maps to refresh")
- try Self.expect(objectManifest.capabilities.contains(.search), "crawlkit SQL/query capability maps to search")
- try Self.expect(objectManifest.capabilities.contains(.desktopCache), "crawlkit tap capability maps to desktop cache")
- try Self.expect(objectManifest.capabilities.contains(.publish), "crawlkit git-share capability maps to publish")
- guard let cacheManifest = manifests.first(where: { $0.id.rawValue == "cachecrawl" }) else {
- throw SelfTestError.failed("desktop cache manifest loads from disk")
- }
- try Self.expect(cacheManifest.capabilities.contains(.desktopCache), "desktop-cache-import command maps to desktop cache")
- try Self.expect(diagnostics.contains { $0.path.hasSuffix("broken.json") }, "external manifest parse errors are reported")
- try Self.expect(installations.contains { $0.id == manifest.id }, "external manifests appear as installations")
- try Self.expect(BuiltInCrawlApps.gitcrawl.configOptions.contains { $0.id == "embedding_model" }, "built-in config options exist")
- try Self.expect(!BuiltInCrawlApps.gitcrawl.needsSecretsForStatus, "built-in status avoids launch keychain reads")
- let secretStatusManifest = CrawlAppManifest(
- id: CrawlAppID(rawValue: "secretstatus"),
- displayName: "Secret Status",
- description: "Status requires an env secret",
- binary: .init(name: "secretstatus"),
- branding: .init(symbolName: "lock", accentColor: "#123456"),
- paths: .init(),
- commands: ["status": ["status", "--json"]],
- capabilities: [.status],
- configOptions: [.init(id: "token", label: "Token", kind: .secret, envVar: "SECRET_STATUS_TOKEN")])
- try Self.expect(secretStatusManifest.needsSecretsForStatus, "external secret status can opt into keychain reads")
- try Self.expect(BuiltInCrawlApps.slacrawl.privacy.containsPrivateMessages, "Slack privacy metadata flags local messages")
- try Self.expect(BuiltInCrawlApps.notcrawl.privacy.localOnlyScopes.contains("workspace pages"), "Notion privacy metadata flags workspace pages")
- try Self.expect(BuiltInCrawlApps.slacrawl.install?.package == "vincentkoc/tap/slacrawl", "built-in install metadata exists")
- try Self.expect(BuiltInCrawlApps.gogcli.availability == .available, "Google manifest is available")
- try Self.expect(BuiltInCrawlApps.gogcli.binary.name == "gog", "Google manifest uses the installed gog binary")
- try Self.expect(BuiltInCrawlApps.gogcli.commands["status"] == ["auth", "list", "--check", "--json", "--no-input"], "Google status is wired")
- try Self.expect(BuiltInCrawlApps.wacli.availability == .available, "WhatsApp manifest is available")
- try Self.expect(BuiltInCrawlApps.wacli.commands["status"] == ["--account", "{config:account}", "--read-only", "--json", "doctor"], "WhatsApp status is wired")
- try Self.expect(BuiltInCrawlApps.birdclaw.binary.name == "bird", "X app id uses bird executable")
- try Self.expect(BuiltInCrawlApps.telecrawl.availability == .available, "telecrawl is available")
- try Self.expect(BuiltInCrawlApps.telecrawl.commands["status"] == ["--json", "status"], "telecrawl uses JSON status command")
- try Self.expect(BuiltInCrawlApps.telecrawl.commands["refresh"] == ["--json", "import"], "telecrawl imports through refresh")
- try Self.expect(BuiltInCrawlApps.telecrawl.capabilities.contains(.search), "telecrawl advertises search")
- try Self.expect(BuiltInCrawlApps.telecrawl.privacy.containsPrivateMessages, "telecrawl privacy metadata flags Telegram messages")
- try Self.expect(BuiltInCrawlApps.telecrawl.install?.package == "steipete/tap/telecrawl", "telecrawl install metadata exists")
- try Self.expect(BuiltInCrawlApps.telecrawl.paths.defaultConfig == "~/.telecrawl/backup.json", "telecrawl config path maps")
- try Self.expect(BuiltInCrawlApps.graincrawl.availability == .available, "graincrawl is available")
- try Self.expect(BuiltInCrawlApps.graincrawl.commands["status"] == ["status", "--json"], "graincrawl uses crawlkit status command")
- try Self.expect(
- BuiltInCrawlApps.graincrawl.commands["refresh"] == ["sync", "--json"],
- "graincrawl refresh honors configured source")
- try Self.expect(
- BuiltInCrawlApps.graincrawl.commands["desktop-cache-import"] == ["sync", "--source", "desktop-cache", "--json"],
- "graincrawl exposes explicit desktop cache import")
- try Self.expect(BuiltInCrawlApps.graincrawl.capabilities.contains(.desktopCache), "graincrawl advertises desktop cache capability")
- try Self.expect(
- BuiltInCrawlApps.graincrawl.commands["query"] == ["--json", "sql"],
- "graincrawl query emits JSON by default")
- try Self.expect(BuiltInCrawlApps.graincrawl.commands["unlock"] == ["unlock", "--json"], "graincrawl exposes unlock action")
- try Self.expect(BuiltInCrawlApps.graincrawl.branding.bundleIdentifier == "com.granola.app", "graincrawl uses native Granola icon")
- try Self.expect(BuiltInCrawlApps.gitcrawl.commands["status"] == ["status", "--json"], "gitcrawl uses fast status command")
- try Self.expect(BuiltInCrawlApps.gitcrawl.commands["refresh"] == ["sync", "--json"], "gitcrawl keeps refresh action wired")
- try Self.expect(BuiltInCrawlApps.gitcrawl.commands["remote-status"] == ["remote", "status", "--json"], "gitcrawl exposes remote status")
- try Self.expect(BuiltInCrawlApps.gitcrawl.commands["cloud-publish"] == ["cloud", "publish", "--json"], "gitcrawl exposes cloud publish")
- try Self.expect(BuiltInCrawlApps.gitcrawl.capabilities.contains(.remoteArchive), "gitcrawl advertises remote archive")
- try Self.expect(BuiltInCrawlApps.gitcrawl.capabilities.contains(.cloudPublish), "gitcrawl advertises cloud publish")
- try Self.expect(BuiltInCrawlApps.slacrawl.commands["query"] == ["sql"], "Slack exposes query action")
- try Self.expect(BuiltInCrawlApps.slacrawl.commands["search"] == ["--json", "search"], "Slack exposes text search action")
- try Self.expect(BuiltInCrawlApps.discrawl.commands["query"] == nil, "Discord does not advertise stale SQL action")
- try Self.expect(!BuiltInCrawlApps.discrawl.capabilities.contains(.search), "Discord search capability waits for upstream metadata")
- try Self.expect(BuiltInCrawlApps.discrawl.commands["remote-status"] == ["remote", "status", "--json"], "Discord exposes remote status")
- try Self.expect(BuiltInCrawlApps.discrawl.commands["cloud-publish"] == ["cloud", "publish", "--sqlite-only", "--json"], "Discord cloud publish defaults to sqlite-only")
- try Self.expect(BuiltInCrawlApps.discrawl.capabilities.contains(.remoteArchive), "Discord advertises remote archive")
- try Self.expect(BuiltInCrawlApps.discrawl.capabilities.contains(.cloudPublish), "Discord advertises cloud publish")
- try Self.expect(BuiltInCrawlApps.notcrawl.capabilities.contains(.desktopCache), "Notion advertises desktop cache capability")
- try Self.expect(BuiltInCrawlApps.gitcrawl.configSections.contains { $0.id == "github" }, "built-in config sections exist")
- }
-
- private static func testNativeConfigRoundTrips() throws {
- let directory = FileManager.default.temporaryDirectory
- .appendingPathComponent("crawlbar-native-config-\(UUID().uuidString)", isDirectory: true)
- try FileManager.default.createDirectory(at: directory, withIntermediateDirectories: true)
- defer { try? FileManager.default.removeItem(at: directory) }
-
- let configURL = directory.appendingPathComponent("config.toml")
- try Data("""
- [openai]
- api_key = "from-file"
- """.utf8).write(to: configURL)
-
- let manifest = CrawlAppManifest(
- id: CrawlAppID(rawValue: "tomlcrawl"),
- displayName: "TOML Crawl",
- description: "A TOML test crawler",
- binary: .init(name: "tomlcrawl"),
- branding: .init(symbolName: "terminal", accentColor: "#123456"),
- paths: .init(defaultConfig: configURL.path),
- commands: [:],
- capabilities: [],
- configOptions: [
- .init(id: "openai_api_key", label: "OpenAI API key", kind: .secret, configKey: "openai.api_key"),
- .init(id: "embedding_model", label: "Embedding model", kind: .choice, configKey: "embeddings.model"),
- .init(id: "sync_limit", label: "Sync limit", kind: .number, configKey: "sync.default_limit"),
- ])
- var appConfig = CrawlBarAppConfig(id: manifest.id)
- let nativeStore = CrawlNativeConfigStore()
- try Self.expect(
- nativeStore.resolvedConfigValues(appConfig: appConfig, manifest: manifest)["openai_api_key"] == "from-file",
- "native TOML config values load")
- try Self.expect(
- nativeStore.resolvedConfigValues(appConfig: appConfig, manifest: manifest, includeSecrets: false)["openai_api_key"] == nil,
- "native TOML secrets stay out of non-secret loads")
-
- appConfig.configValues = nativeStore.resolvedConfigValues(appConfig: appConfig, manifest: manifest)
- appConfig.configValues["embedding_model"] = "text-embedding-3-large"
- appConfig.configValues["sync_limit"] = "100"
- try nativeStore.write(appConfig: appConfig, manifest: manifest)
- let content = try String(contentsOf: configURL, encoding: .utf8)
- try Self.expect(content.contains("api_key = \"from-file\""), "native TOML values preserve existing keys")
- try Self.expect(content.contains("[embeddings]"), "native TOML section writes")
- try Self.expect(content.contains("model = \"text-embedding-3-large\""), "native TOML value writes")
- try Self.expect(content.contains("default_limit = 100"), "native TOML number writes without quotes")
-
- appConfig.configValues.removeValue(forKey: "openai_api_key")
- try nativeStore.write(appConfig: appConfig, manifest: manifest)
- let clearedContent = try String(contentsOf: configURL, encoding: .utf8)
- try Self.expect(clearedContent.contains("api_key = \"from-file\""), "native TOML secret keys preserve when omitted")
- try nativeStore.write(appConfig: appConfig, manifest: manifest, clearMissingSecretIDs: ["openai_api_key"])
- let explicitlyClearedContent = try String(contentsOf: configURL, encoding: .utf8)
- try Self.expect(!explicitlyClearedContent.contains("api_key ="), "native TOML secret keys clear when explicit")
- }
-
- private static func testStatusSecretsLoadFromNativeConfig() throws {
- let directory = FileManager.default.temporaryDirectory
- .appendingPathComponent("crawlbar-status-secret-\(UUID().uuidString)", isDirectory: true)
- try FileManager.default.createDirectory(at: directory, withIntermediateDirectories: true)
- defer { try? FileManager.default.removeItem(at: directory) }
-
- let scriptURL = directory.appendingPathComponent("secret-status.sh")
- try Data("""
- #!/bin/sh
- printf '{"state":"ok"}'
- """.utf8).write(to: scriptURL)
- try FileManager.default.setAttributes([.posixPermissions: 0o755], ofItemAtPath: scriptURL.path)
-
- let nativeConfigURL = directory.appendingPathComponent("status.toml")
- try Data("""
- [auth]
- token = "from-native"
- """.utf8).write(to: nativeConfigURL)
-
- let manifest = CrawlAppManifest(
- id: CrawlAppID(rawValue: "statussecret"),
- displayName: "Status Secret",
- description: "A status secret test crawler",
- binary: .init(name: scriptURL.path),
- branding: .init(symbolName: "lock", accentColor: "#123456"),
- paths: .init(defaultConfig: nativeConfigURL.path),
- commands: ["status": ["status", "--json"]],
- capabilities: [.status],
- statusRequiresSecrets: true,
- configOptions: [
- .init(id: "token", label: "Token", kind: .secret, envVar: "STATUS_SECRET_TOKEN", configKey: "auth.token"),
- ])
- try CrawlCoding.makeJSONEncoder().encode(manifest)
- .write(to: directory.appendingPathComponent("statussecret.json"))
- let configURL = directory.appendingPathComponent("config.json")
- let store = CrawlBarConfigStore(fileURL: configURL)
- try store.save(CrawlBarConfig(manifestDirectories: [directory.path]))
- let registry = CrawlAppRegistry(configStore: store)
-
- guard let plain = try registry.installation(for: manifest.id, includeSecrets: false),
- let status = try registry.installationForStatus(for: manifest.id)
- else {
- throw SelfTestError.failed("status secret crawler loads")
- }
- try Self.expect(plain.configValues["token"] == nil, "plain installation omits native secret")
- try Self.expect(status.configValues["token"] == "from-native", "status installation rehydrates native secret")
- }
-
- private static func testStatusMapperNormalizesCounts() throws {
- let output = """
- {"message_count":42,"channel_count":3,"last_sync_at":"2026-05-01T12:00:00Z","db_path":"/tmp/discrawl.db"}
- """
- let result = CrawlCommandResult(
- appID: BuiltInCrawlApps.discrawlID,
- action: "status",
- exitCode: 0,
- stdout: output,
- stderr: "",
- startedAt: Date(),
- finishedAt: Date())
-
- let status = CrawlStatusMapper().status(from: result, manifest: BuiltInCrawlApps.discrawl)
- try Self.expect(status.counts.contains(CrawlCount(id: "messages", label: "Messages", value: 42)), "discrawl messages map")
- try Self.expect(status.lastSyncAt != nil, "whole-second last sync dates map")
- try Self.expect(status.databasePath == "/tmp/discrawl.db", "database path maps")
- try Self.expect(status.databases.first?.label == "Discord archive", "database inventory maps")
- try Self.expect(status.databases.first?.counts.contains(CrawlCount(id: "messages", label: "Messages", value: 42)) == true, "database inventory carries counts")
-
- let gogResult = CrawlCommandResult(
- appID: BuiltInCrawlApps.gogcliID,
- action: "status",
- exitCode: 0,
- stdout: #"{"account":{"credentials_exists":true},"config":{"exists":true,"path":"/tmp/gog/config.json"}}"#,
- stderr: "",
- startedAt: Date(),
- finishedAt: Date())
- let gogStatus = CrawlStatusMapper().status(from: gogResult, manifest: BuiltInCrawlApps.gogcli)
- try Self.expect(gogStatus.state == .needsAuth, "gog raw status asks OAuth auth to be verified")
- try Self.expect(gogStatus.configPath == "/tmp/gog/config.json", "gog config path maps")
-
- let gogServiceAccountRawResult = CrawlCommandResult(
- appID: BuiltInCrawlApps.gogcliID,
- action: "status",
- exitCode: 0,
- stdout: #"{"account":{"service_account_configured":true},"config":{"exists":false}}"#,
- stderr: "",
- startedAt: Date(),
- finishedAt: Date())
- let gogServiceAccountRawStatus = CrawlStatusMapper().status(from: gogServiceAccountRawResult, manifest: BuiltInCrawlApps.gogcli)
- try Self.expect(gogServiceAccountRawStatus.state == .current, "gog raw status maps service account auth")
-
- let gogServiceAccountResult = CrawlCommandResult(
- appID: BuiltInCrawlApps.gogcliID,
- action: "status",
- exitCode: 0,
- stdout: #"{"status":"ok","checks":[{"name":"config.path","status":"ok","detail":"/tmp/gog/config.json"},{"name":"service_account","status":"ok"}]}"#,
- stderr: "",
- startedAt: Date(),
- finishedAt: Date())
- let gogServiceAccountStatus = CrawlStatusMapper().status(from: gogServiceAccountResult, manifest: BuiltInCrawlApps.gogcli)
- try Self.expect(gogServiceAccountStatus.state == .current, "gog doctor maps configured auth")
- try Self.expect(gogServiceAccountStatus.configPath == "/tmp/gog/config.json", "gog doctor config path maps")
-
- let gogDoctorFailureResult = CrawlCommandResult(
- appID: BuiltInCrawlApps.gogcliID,
- action: "status",
- exitCode: 0,
- stdout: #"{"status":"error","checks":[{"name":"tokens","status":"error","detail":"no readable OAuth tokens"}]}"#,
- stderr: "",
- startedAt: Date(),
- finishedAt: Date())
- let gogDoctorFailureStatus = CrawlStatusMapper().status(from: gogDoctorFailureResult, manifest: BuiltInCrawlApps.gogcli)
- try Self.expect(gogDoctorFailureStatus.state == .needsAuth, "gog doctor token failures map to auth setup")
- try Self.expect(gogDoctorFailureStatus.summary == "no readable OAuth tokens", "gog doctor failure detail maps")
-
- let gogDoctorConfigResult = CrawlCommandResult(
- appID: BuiltInCrawlApps.gogcliID,
- action: "status",
- exitCode: 0,
- stdout: #"{"status":"warn","checks":[{"name":"config.path","status":"warn","detail":"config missing"}]}"#,
- stderr: "",
- startedAt: Date(),
- finishedAt: Date())
- let gogDoctorConfigStatus = CrawlStatusMapper().status(from: gogDoctorConfigResult, manifest: BuiltInCrawlApps.gogcli)
- try Self.expect(gogDoctorConfigStatus.state == .needsConfig, "gog doctor config warnings map to config setup")
-
- let wacliResult = CrawlCommandResult(
- appID: BuiltInCrawlApps.wacliID,
- action: "status",
- exitCode: 0,
- stdout: #"{"success":true,"data":{"store_dir":"/tmp/wacli/accounts/me","authenticated":true,"store":{"messages":12,"chats":3,"last_sync_at":"2026-05-01T12:00:00Z"}}}"#,
- stderr: "",
- startedAt: Date(),
- finishedAt: Date())
- let wacliStatus = CrawlStatusMapper().status(from: wacliResult, manifest: BuiltInCrawlApps.wacli)
- try Self.expect(wacliStatus.counts.contains(CrawlCount(id: "messages", label: "Messages", value: 12)), "wacli message counts map")
- try Self.expect(wacliStatus.configPath == "/tmp/wacli/config.yaml", "wacli account config path maps")
- try Self.expect(wacliStatus.databasePath == "/tmp/wacli/accounts/me/wacli.db", "wacli database path maps")
- try Self.expect(wacliStatus.databases.contains { $0.kind == .sqlite && $0.path == "/tmp/wacli/accounts/me/wacli.db" }, "wacli database inventory keeps sqlite resource")
- try Self.expect(wacliStatus.databases.contains { $0.kind == .logical && $0.path == "/tmp/wacli/accounts/me" }, "wacli database inventory keeps logical store")
-
- let wacliStoreErrorResult = CrawlCommandResult(
- appID: BuiltInCrawlApps.wacliID,
- action: "status",
- exitCode: 0,
- stdout: #"{"success":true,"data":{"authenticated":true,"store_error":"database disk image is malformed"}}"#,
- stderr: "",
- startedAt: Date(),
- finishedAt: Date())
- let wacliStoreErrorStatus = CrawlStatusMapper().status(from: wacliStoreErrorResult, manifest: BuiltInCrawlApps.wacli)
- try Self.expect(wacliStoreErrorStatus.state == .error, "wacli store errors map to error status")
- try Self.expect(wacliStoreErrorStatus.errors.contains("database disk image is malformed"), "wacli store error is preserved")
-
- let wacliFirstRunResult = CrawlCommandResult(
- appID: BuiltInCrawlApps.wacliID,
- action: "status",
- exitCode: 0,
- stdout: #"{"success":true,"data":{"authenticated":false,"store_error":"open store: no such file"}}"#,
- stderr: "",
- startedAt: Date(),
- finishedAt: Date())
- let wacliFirstRunStatus = CrawlStatusMapper().status(from: wacliFirstRunResult, manifest: BuiltInCrawlApps.wacli)
- try Self.expect(wacliFirstRunStatus.state == .needsAuth, "wacli first-run store errors stay auth setup")
- try Self.expect(wacliFirstRunStatus.summary == "WhatsApp auth needs setup", "wacli first-run summary stays setup-oriented")
-
- let wacliCorruptUnauthedResult = CrawlCommandResult(
- appID: BuiltInCrawlApps.wacliID,
- action: "status",
- exitCode: 0,
- stdout: #"{"success":true,"data":{"authenticated":false,"store_error":"database disk image is malformed"}}"#,
- stderr: "",
- startedAt: Date(),
- finishedAt: Date())
- let wacliCorruptUnauthedStatus = CrawlStatusMapper().status(from: wacliCorruptUnauthedResult, manifest: BuiltInCrawlApps.wacli)
- try Self.expect(wacliCorruptUnauthedStatus.state == .error, "wacli corrupt unauthenticated stores stay errors")
-
- let crawlKitOutput = """
- {
- "schema_version": "crawlkit.control.v1",
- "app_id": "discrawl",
- "state": "current",
- "summary": "5052 messages across 293 channels",
- "database_path": "/tmp/discrawl.db",
- "database_bytes": 36397056,
- "counts": [
- {"id": "guilds", "label": "Guilds", "value": 56},
- {"id": "channels", "label": "Channels", "value": 293},
- {"id": "messages", "label": "Messages", "value": 5052}
- ],
- "databases": [
- {
- "id": "primary",
- "label": "Discord archive",
- "kind": "sqlite",
- "role": "archive",
- "path": "/tmp/discrawl.db",
- "is_primary": true,
- "bytes": 36397056,
- "modified_at": "2026-04-24T07:38:30Z",
- "counts": [
- {"id": "messages", "label": "Messages", "value": 5052}
- ]
- }
- ]
- }
- """
- let crawlKitResult = CrawlCommandResult(
- appID: BuiltInCrawlApps.discrawlID,
- action: "status",
- exitCode: 0,
- stdout: crawlKitOutput,
- stderr: "",
- startedAt: Date(),
- finishedAt: Date())
-
- let crawlKitStatus = CrawlStatusMapper().status(from: crawlKitResult, manifest: BuiltInCrawlApps.discrawl)
- try Self.expect(crawlKitStatus.summary == "5052 messages across 293 channels", "crawlkit status summary maps")
- try Self.expect(crawlKitStatus.state == .current, "crawlkit explicit state maps")
- try Self.expect(crawlKitStatus.databaseBytes == 36397056, "crawlkit database bytes map")
- try Self.expect(crawlKitStatus.counts.contains(CrawlCount(id: "messages", label: "Messages", value: 5052)), "crawlkit count array maps")
- try Self.expect(crawlKitStatus.databases.first?.id == "primary", "crawlkit databases map")
- try Self.expect(crawlKitStatus.databases.first?.modifiedAt != nil, "crawlkit database modified date maps")
- try Self.expect(crawlKitStatus.databases.first?.counts.contains(CrawlCount(id: "messages", label: "Messages", value: 5052)) == true, "crawlkit database counts map")
-
- let cloudOutput = """
- {
- "schema_version": "crawlkit.control.v1",
- "app_id": "discrawl",
- "state": "current",
- "summary": "1417329 messages in remote archive discrawl/openclaw",
- "config_path": "/tmp/discrawl.toml",
- "counts": [
- {"id": "channels", "label": "Channels", "value": 23956},
- {"id": "messages", "label": "Messages", "value": 1417329},
- {"id": "members", "label": "Members", "value": 173089}
- ],
- "remote": {
- "enabled": true,
- "mode": "cloud",
- "endpoint": "https://crawl.example.test",
- "archive": "discrawl/openclaw",
- "last_ingest_at": "2026-05-28T19:30:56.840Z"
- },
- "databases": [
- {
- "id": "remote",
- "label": "Discord cloud archive",
- "kind": "cloudflare-d1",
- "role": "archive",
- "endpoint": "https://crawl.example.test",
- "archive": "discrawl/openclaw",
- "is_primary": true,
- "counts": [
- {"id": "messages", "label": "Messages", "value": 1417329}
- ]
- }
- ],
- "sqlite_bundle": {
- "key": "v1/discrawl/discrawl%2Fopenclaw/sqlite/current.manifest.json",
- "content_type": "application/json",
- "uploaded_at": "2026-05-28T19:30:56.840Z",
- "manifest": {
- "format": "sqlite-gzip-chunked-v1",
- "generated_at": "2026-05-28T19:30:41Z",
- "compression": {"algorithm": "gzip"},
- "object": {"key": "v1/discrawl/discrawl%2Fopenclaw/sqlite/current.db", "size": 839589888, "sha256": "raw"},
- "compressed_object": {"key": "v1/discrawl/discrawl%2Fopenclaw/sqlite/current.db.gz", "size": 259315038, "sha256": "compressed"},
- "parts": [
- {"index": 0, "key": "part-0", "size": 67108864, "sha256": "a"},
- {"index": 1, "key": "part-1", "size": 67108864, "sha256": "b"},
- {"index": 2, "key": "part-2", "size": 67108864, "sha256": "c"},
- {"index": 3, "key": "part-3", "size": 57988446, "sha256": "d"}
- ]
- }
- }
- }
- """
- let cloudResult = CrawlCommandResult(
- appID: BuiltInCrawlApps.discrawlID,
- action: "status",
- exitCode: 0,
- stdout: cloudOutput,
- stderr: "",
- startedAt: Date(),
- finishedAt: Date())
- let cloudStatus = CrawlStatusMapper().status(from: cloudResult, manifest: BuiltInCrawlApps.discrawl)
- try Self.expect(cloudStatus.remote?.archive == "discrawl/openclaw", "remote archive maps")
- try Self.expect(cloudStatus.lastSyncAt != nil, "remote ingest maps as sync freshness")
- try Self.expect(cloudStatus.databases.first?.kind == .cloudflareD1, "remote database kind maps")
- try Self.expect(cloudStatus.databases.first?.endpoint == "https://crawl.example.test", "remote database endpoint maps")
- try Self.expect(cloudStatus.sqliteBundle?.format == "sqlite-gzip-chunked-v1", "sqlite bundle format maps")
- try Self.expect(cloudStatus.sqliteBundle?.compression == "gzip", "sqlite bundle compression maps")
- try Self.expect(cloudStatus.sqliteBundle?.rawBytes == 839589888, "sqlite bundle raw size maps")
- try Self.expect(cloudStatus.sqliteBundle?.compressedBytes == 259315038, "sqlite bundle compressed size maps")
- try Self.expect(cloudStatus.sqliteBundle?.partCount == 4, "sqlite bundle part count maps")
-
- let telecrawlOutput = """
- {
- "db_path": "/tmp/telecrawl.db",
- "chats": 3,
- "messages": 42,
- "unread_chats": 1,
- "unread_messages": 5,
- "media_messages": 6,
- "folders": 2,
- "topics": 4,
- "last_import_at": "2026-05-01T12:00:00Z"
- }
- """
- let telecrawlResult = CrawlCommandResult(
- appID: BuiltInCrawlApps.telecrawlID,
- action: "status",
- exitCode: 0,
- stdout: telecrawlOutput,
- stderr: "",
- startedAt: Date(),
- finishedAt: Date())
- let telecrawlStatus = CrawlStatusMapper().status(
- from: telecrawlResult,
- manifest: BuiltInCrawlApps.telecrawl,
- staleAfterSeconds: 60)
- try Self.expect(telecrawlStatus.counts.contains(CrawlCount(id: "messages", label: "Messages", value: 42)), "telecrawl messages map")
- try Self.expect(telecrawlStatus.counts.contains(CrawlCount(id: "chats", label: "Chats", value: 3)), "telecrawl chats map")
- try Self.expect(telecrawlStatus.lastSyncAt == telecrawlStatus.lastImportAt, "telecrawl import time maps to sync freshness")
- try Self.expect(telecrawlStatus.state == .stale, "telecrawl import freshness drives stale state")
- try Self.expect(telecrawlStatus.databases.first?.label == "Telegram archive", "telecrawl database inventory maps")
-
- let okOutput = """
- {
- "schema_version": "crawlkit.control.v1",
- "app_id": "graincrawl",
- "state": "ok",
- "summary": "1 notes",
- "counts": [{"id": "notes", "label": "Notes", "value": 1}]
- }
- """
- let okResult = CrawlCommandResult(
- appID: BuiltInCrawlApps.graincrawlID,
- action: "status",
- exitCode: 0,
- stdout: okOutput,
- stderr: "",
- startedAt: Date(),
- finishedAt: Date())
- let okStatus = CrawlStatusMapper().status(from: okResult, manifest: BuiltInCrawlApps.graincrawl)
- try Self.expect(okStatus.state == .current, "crawlkit ok state maps to current")
-
- let failedOutput = """
- {"schema_version":"crawlkit.control.v1","app_id":"graincrawl","state":"failed","summary":"broken"}
- """
- let failedResult = CrawlCommandResult(
- appID: BuiltInCrawlApps.graincrawlID,
- action: "status",
- exitCode: 0,
- stdout: failedOutput,
- stderr: "",
- startedAt: Date(),
- finishedAt: Date())
- let failedStatus = CrawlStatusMapper().status(from: failedResult, manifest: BuiltInCrawlApps.graincrawl)
- try Self.expect(failedStatus.state == .error, "crawlkit failed state maps to error")
-
- let githubAuthMessage = """
- [github] request GET /repos/openclaw/openclaw
- github GET /repos/openclaw/openclaw failed with status 401: {
- "message": "Bad credentials"
- }
- """
- let githubAuthResult = CrawlCommandResult(
- appID: BuiltInCrawlApps.gitcrawlID,
- action: "status",
- exitCode: 1,
- stdout: "",
- stderr: githubAuthMessage,
- startedAt: Date(),
- finishedAt: Date())
- let githubAuthStatus = CrawlStatusMapper().status(from: githubAuthResult, manifest: BuiltInCrawlApps.gitcrawl)
- try Self.expect(githubAuthStatus.state == .needsAuth, "gitcrawl 401 maps to auth state")
- try Self.expect(githubAuthStatus.summary == "GitHub credentials rejected", "gitcrawl 401 uses useful summary")
- try Self.expect(githubAuthStatus.errors == ["GitHub credentials rejected"], "gitcrawl 401 keeps request trace out of status errors")
-
- let githubServerMessage = """
- [github] request GET /repos/openclaw/openclaw
- github GET /repos/openclaw/openclaw failed with status 500
- """
- let githubServerStatus = CrawlAppStatus.commandFailure(
- appID: BuiltInCrawlApps.gitcrawlID,
- action: "refresh",
- message: githubServerMessage,
- fallback: "refresh failed")
- try Self.expect(
- githubServerStatus.summary == "refresh: github GET /repos/openclaw/openclaw failed with status 500",
- "gitcrawl request trace is skipped in failure summaries")
- }
-
- private static func testConfigValuesReachCommandEnvironment() throws {
- let directory = FileManager.default.temporaryDirectory
- .appendingPathComponent("crawlbar-env-\(UUID().uuidString)", isDirectory: true)
- try FileManager.default.createDirectory(at: directory, withIntermediateDirectories: true)
- defer { try? FileManager.default.removeItem(at: directory) }
-
- let scriptURL = directory.appendingPathComponent("print-env.sh")
- try Data("""
- #!/bin/sh
- if [ "$#" -gt 0 ]; then
- printf '%s|%s' "$CRAWLBAR_TEST_VALUE" "$*"
- else
- printf '%s' "$CRAWLBAR_TEST_VALUE"
- fi
- """.utf8).write(to: scriptURL)
- try FileManager.default.setAttributes([.posixPermissions: 0o755], ofItemAtPath: scriptURL.path)
-
- let manifest = CrawlAppManifest(
- id: CrawlAppID(rawValue: "envcrawl"),
- displayName: "Env Crawl",
- description: "A test crawler",
- binary: .init(name: scriptURL.path),
- branding: .init(symbolName: "terminal", accentColor: "#123456"),
- paths: .init(),
- commands: ["status": [], "query": ["query"]],
- capabilities: [.status],
- configOptions: [
- .init(id: "test_value", label: "Test Value", envVar: "CRAWLBAR_TEST_VALUE"),
- ])
- let installation = CrawlAppInstallation(
- manifest: manifest,
- binaryPath: scriptURL.path,
- configValues: ["test_value": "from-config"])
- let result = try CrawlCommandRunner().run(installation: installation, action: "status", timeoutSeconds: 5)
- try Self.expect(result.stdout == "from-config", "config values reach crawler command environment")
- let queryResult = try CrawlCommandRunner().run(
- installation: installation,
- action: "query",
- extraArguments: ["select count(*) from items"],
- timeoutSeconds: 5)
- try Self.expect(queryResult.stdout == "from-config|query select count(*) from items", "query arguments reach crawler commands")
- }
-
- private static func testExecutableResolverUsesMacCliFallbackPaths() throws {
- let directory = FileManager.default.temporaryDirectory
- .appendingPathComponent("crawlbar-path-\(UUID().uuidString)", isDirectory: true)
- let localBinURL = directory.appendingPathComponent(".local/bin", isDirectory: true)
- try FileManager.default.createDirectory(at: localBinURL, withIntermediateDirectories: true)
- defer { try? FileManager.default.removeItem(at: directory) }
-
- let scriptURL = localBinURL.appendingPathComponent("fallbackcrawl")
- try Data("""
- #!/bin/sh
- printf '%s' "$PATH"
- """.utf8).write(to: scriptURL)
- try FileManager.default.setAttributes([.posixPermissions: 0o755], ofItemAtPath: scriptURL.path)
-
- let environment = [
- "HOME": directory.path,
- "PATH": "/usr/bin:/bin:/usr/sbin:/sbin",
- ]
- let resolver = CrawlExecutableResolver(environment: environment)
- try Self.expect(
- resolver.resolve("fallbackcrawl") == scriptURL.path,
- "resolver checks user CLI fallback paths")
-
- let manifest = CrawlAppManifest(
- id: CrawlAppID(rawValue: "fallbackcrawl"),
- displayName: "Fallback Crawl",
- description: "A fallback PATH crawler",
- binary: .init(name: "fallbackcrawl"),
- branding: .init(symbolName: "terminal", accentColor: "#123456"),
- paths: .init(),
- commands: ["status": []],
- capabilities: [.status])
- let installation = CrawlAppInstallation(manifest: manifest, binaryPath: "fallbackcrawl")
- let result = try CrawlCommandRunner(resolver: resolver, environment: environment)
- .run(installation: installation, action: "status", timeoutSeconds: 5)
-
- try Self.expect(
- result.stdout.split(separator: ":").contains(Substring(localBinURL.path)),
- "runner passes normalized fallback PATH to crawlers")
- try Self.expect(
- CrawlProcessEnvironment.normalized(["PATH": "/usr/bin"])["HOME"]?.isEmpty == false,
- "normalized environment supplies HOME for launchd crawler commands")
- }
-
- private static func testRegistryResolvesBirdclawAccessPathBinary() throws {
- let directory = FileManager.default.temporaryDirectory
- .appendingPathComponent("crawlbar-birdclaw-\(UUID().uuidString)", isDirectory: true)
- try FileManager.default.createDirectory(at: directory, withIntermediateDirectories: true)
- defer { try? FileManager.default.removeItem(at: directory) }
-
- let binaryURL = directory.appendingPathComponent("birdclaw")
- try Data("#!/bin/sh\n".utf8).write(to: binaryURL)
- try FileManager.default.setAttributes([.posixPermissions: 0o755], ofItemAtPath: binaryURL.path)
-
- let configURL = directory.appendingPathComponent("config.json")
- let store = CrawlBarConfigStore(fileURL: configURL)
- try store.save(CrawlBarConfig(apps: [
- CrawlBarAppConfig(
- id: BuiltInCrawlApps.birdclawID,
- configValues: ["access_path": "birdclaw"]),
- ]))
- let registry = CrawlAppRegistry(
- configStore: store,
- resolver: CrawlExecutableResolver(environment: [
- "HOME": directory.path,
- "PATH": directory.path,
- ]))
- let installation = try registry.installation(for: BuiltInCrawlApps.birdclawID)
- try Self.expect(installation?.binaryPath == binaryURL.path, "Birdclaw access path resolves birdclaw binary")
- }
-
- private static func testRemoteSshExecutionBuildsCommand() throws {
- let directory = FileManager.default.temporaryDirectory
- .appendingPathComponent("crawlbar-remote-\(UUID().uuidString)", isDirectory: true)
- try FileManager.default.createDirectory(at: directory, withIntermediateDirectories: true)
- defer { try? FileManager.default.removeItem(at: directory) }
-
- let scriptURL = directory.appendingPathComponent("ssh")
- try Data("""
- #!/bin/sh
- printf '%s' "$*"
- """.utf8).write(to: scriptURL)
- try FileManager.default.setAttributes([.posixPermissions: 0o755], ofItemAtPath: scriptURL.path)
- let environment = [
- "HOME": directory.path,
- "PATH": directory.path,
- ]
- let runner = CrawlCommandRunner(
- resolver: CrawlExecutableResolver(environment: environment),
- environment: environment)
-
- let manifest = CrawlAppManifest(
- id: CrawlAppID(rawValue: "wacli-test"),
- displayName: "WhatsApp Test",
- description: "A remote WhatsApp crawler",
- binary: .init(name: "wacli"),
- execution: .init(
- kind: .local,
- kindConfigID: "execution_mode",
- targetConfigID: "remote_target",
- runAsConfigID: "remote_run_as",
- remoteBinary: "wacli"),
- branding: .init(symbolName: "message.circle", accentColor: "#25D366"),
- paths: .init(),
- commands: ["search": ["--account", "{config:account}", "--read-only", "--json", "messages", "search"]],
- capabilities: [.search],
- configOptions: [
- .init(id: "execution_mode", label: "Run Location", kind: .choice, defaultValue: "local", choices: ["local", "remote"]),
- .init(id: "account", label: "Account", defaultValue: "personal"),
- ])
- let installation = CrawlAppInstallation(
- manifest: manifest,
- binaryPath: scriptURL.path,
- configValues: [
- "execution_mode": "remote",
- "remote_target": "user@example-host",
- "remote_run_as": "crawl",
- ])
- let result = try runner.run(
- installation: installation,
- action: "search",
- extraArguments: ["hello", "world"],
- timeoutSeconds: 5)
-
- try Self.expect(
- result.stdout == #"-- user@example-host 'sudo' '-u' 'crawl' '-H' '--' 'sh' '-lc' 'cd ~ && exec '\''wacli'\'' '\''--account'\'' '\''personal'\'' '\''--read-only'\'' '\''--json'\'' '\''messages'\'' '\''search'\'' '\''hello world'\'''"#,
- "remote SSH execution builds a quoted remote command with config defaults")
-
- let optionTargetInstallation = CrawlAppInstallation(
- manifest: manifest,
- binaryPath: scriptURL.path,
- configValues: [
- "execution_mode": "remote",
- "remote_target": "-oProxyCommand=/tmp/hook",
- ])
- do {
- _ = try runner.run(
- installation: optionTargetInstallation,
- action: "search",
- extraArguments: ["hello"],
- timeoutSeconds: 5)
- throw SelfTestError.failed("remote SSH target rejects option-looking values")
- } catch CrawlCommandRunnerError.invalidRemoteTarget {
- }
-
- let localInstallation = CrawlAppInstallation(
- manifest: manifest,
- binaryPath: scriptURL.path,
- configValues: ["execution_mode": "local"])
- let localResult = try runner.run(
- installation: localInstallation,
- action: "search",
- extraArguments: ["hello", "world"],
- timeoutSeconds: 5)
- try Self.expect(
- localResult.stdout == "--account personal --read-only --json messages search hello world",
- "local execution mode bypasses SSH and uses the crawler binary")
-
- let remoteBirdclawInstallation = CrawlAppInstallation(
- manifest: BuiltInCrawlApps.birdclaw,
- binaryPath: scriptURL.path,
- configValues: [
- "access_path": "birdclaw",
- "execution_mode": "remote",
- "remote_target": "user@example-host",
- "remote_run_as": "crawl",
- ])
- let birdclawResult = try runner.run(
- installation: remoteBirdclawInstallation,
- action: "status",
- timeoutSeconds: 5)
- try Self.expect(
- birdclawResult.stdout == #"-- user@example-host 'sudo' '-u' 'crawl' '-H' '--' 'sh' '-lc' 'cd ~ && exec '\''birdclaw'\'' '\''auth'\'' '\''status'\'' '\''--json'\'''"#,
- "X remote execution can use the Birdclaw/xurl access path")
-
- let customBirdclawURL = directory.appendingPathComponent("custom-birdclaw")
- try Data("""
- #!/bin/sh
- printf 'custom:%s' "$*"
- """.utf8).write(to: customBirdclawURL)
- try FileManager.default.setAttributes([.posixPermissions: 0o755], ofItemAtPath: customBirdclawURL.path)
- let localBirdclawOverride = CrawlAppInstallation(
- manifest: BuiltInCrawlApps.birdclaw,
- binaryPath: customBirdclawURL.path,
- configValues: ["access_path": "birdclaw"])
- let localBirdclawResult = try runner.run(
- installation: localBirdclawOverride,
- action: "status",
- timeoutSeconds: 5)
- try Self.expect(
- localBirdclawResult.stdout == "custom:auth status --json",
- "Birdclaw access path preserves binary overrides")
-
- let envManifest = CrawlAppManifest(
- id: CrawlAppID(rawValue: "envcrawl-test"),
- displayName: "Env Crawl Test",
- description: "A remote crawler that needs an env file",
- binary: .init(name: "envcrawl-local"),
- execution: .init(
- kind: .local,
- kindConfigID: "execution_mode",
- targetConfigID: "remote_target",
- runAsConfigID: "remote_run_as",
- remoteEnvFileConfigID: "remote_env_file",
- remoteBinary: "envcrawl"),
- branding: .init(symbolName: "terminal", accentColor: "#123456"),
- paths: .init(),
- commands: ["status": ["status", "--json"]],
- capabilities: [.status],
- configOptions: [
- .init(id: "execution_mode", label: "Run Location", kind: .choice, defaultValue: "local", choices: ["local", "remote"]),
- .init(id: "remote_env_file", label: "Remote Env File"),
- ])
- let envInstallation = CrawlAppInstallation(
- manifest: envManifest,
- binaryPath: scriptURL.path,
- configValues: [
- "execution_mode": "remote",
- "remote_target": "user@example-host",
- "remote_run_as": "crawl",
- "remote_env_file": "/run/example/env",
- ])
- let envResult = try runner.run(
- installation: envInstallation,
- action: "status",
- timeoutSeconds: 5)
- try Self.expect(
- envResult.stdout == #"-- user@example-host 'sudo' '-u' 'crawl' '-H' '--' 'sh' '-lc' 'cd ~ && set -a && . '\''/run/example/env'\'' && set +a && exec '\''envcrawl'\'' '\''status'\'' '\''--json'\'''"#,
- "remote SSH execution can source an env file before exec")
- }
-
- private static func testQueryActionResolverSkipsSQLForPlainText() throws {
- let sqlOnlyManifest = CrawlAppManifest(
- id: CrawlAppID(rawValue: "sqlonly"),
- displayName: "SQL Only",
- description: "A SQL-only crawler",
- binary: .init(name: "sqlonly"),
- branding: .init(symbolName: "terminal", accentColor: "#123456"),
- paths: .init(),
- commands: ["sql": ["sql"]],
- capabilities: [.search])
- try Self.expect(
- CrawlQueryActionResolver.action(for: sqlOnlyManifest, queryArguments: ["select count(*) from rows"]) == "sql",
- "SQL-looking queries can use SQL-only crawler commands")
- try Self.expect(
- CrawlQueryActionResolver.action(for: sqlOnlyManifest, queryArguments: ["manifest"]) == nil,
- "plain text queries do not fall through to SQL-only crawler commands")
-
- let textManifest = CrawlAppManifest(
- id: CrawlAppID(rawValue: "textcrawl"),
- displayName: "Text Crawl",
- description: "A text search crawler",
- binary: .init(name: "textcrawl"),
- branding: .init(symbolName: "terminal", accentColor: "#123456"),
- paths: .init(),
- commands: ["search": ["search"], "query": ["query"]],
- capabilities: [.search])
- try Self.expect(
- CrawlQueryActionResolver.action(for: textManifest, queryArguments: ["manifest"]) == "search",
- "plain text queries prefer explicit search commands")
- }
-
- private static func testWacliSearchJoinsQueryArguments() throws {
- let directory = FileManager.default.temporaryDirectory
- .appendingPathComponent("crawlbar-wacli-search-\(UUID().uuidString)", isDirectory: true)
- try FileManager.default.createDirectory(at: directory, withIntermediateDirectories: true)
- defer { try? FileManager.default.removeItem(at: directory) }
-
- let scriptURL = directory.appendingPathComponent("wacli")
- try Data("""
- #!/bin/sh
- printf '%s\\n' "$#"
- for arg in "$@"; do printf '<%s>\\n' "$arg"; done
- """.utf8).write(to: scriptURL)
- try FileManager.default.setAttributes([.posixPermissions: 0o755], ofItemAtPath: scriptURL.path)
-
- let manifest = CrawlAppManifest(
- id: BuiltInCrawlApps.wacliID,
- displayName: "WhatsApp",
- description: "A WhatsApp crawler",
- binary: .init(name: scriptURL.path),
- branding: .init(symbolName: "message", accentColor: "#25D366"),
- paths: .init(),
- commands: ["search": ["messages", "search"]],
- capabilities: [.search])
- let installation = CrawlAppInstallation(manifest: manifest, binaryPath: scriptURL.path)
- let result = try CrawlCommandRunner()
- .run(installation: installation, action: "search", extraArguments: ["hello", "world"], timeoutSeconds: 5)
-
- try Self.expect(
- result.stdout == "3\n\n\n\n",
- "wacli search receives multi-word query as one argument")
-
- let flaggedResult = try CrawlCommandRunner()
- .run(
- installation: installation,
- action: "search",
- extraArguments: ["hello", "world", "--limit", "5"],
- timeoutSeconds: 5)
- try Self.expect(
- flaggedResult.stdout == "5\n\n\n\n<--limit>\n<5>\n",
- "wacli search preserves flags after joined query")
-
- let builtInDefault = CrawlAppInstallation(manifest: BuiltInCrawlApps.wacli, binaryPath: scriptURL.path)
- let defaultAccountResult = try CrawlCommandRunner()
- .run(installation: builtInDefault, action: "search", extraArguments: ["hello", "world"], timeoutSeconds: 5)
- try Self.expect(
- defaultAccountResult.stdout == "5\n<--read-only>\n<--json>\n\n\n\n",
- "built-in wacli omits account flag until configured")
-
- let builtInNamed = CrawlAppInstallation(
- manifest: BuiltInCrawlApps.wacli,
- binaryPath: scriptURL.path,
- configValues: ["account": "personal"])
- let namedAccountResult = try CrawlCommandRunner()
- .run(installation: builtInNamed, action: "search", extraArguments: ["hello", "world"], timeoutSeconds: 5)
- try Self.expect(
- namedAccountResult.stdout == "7\n<--account>\n\n<--read-only>\n<--json>\n\n\n\n",
- "built-in wacli applies configured account")
-
- let literalConfigQuery = try CrawlCommandRunner()
- .run(
- installation: builtInNamed,
- action: "search",
- extraArguments: ["{config:account}"],
- timeoutSeconds: 5)
- try Self.expect(
- literalConfigQuery.stdout == "7\n<--account>\n\n<--read-only>\n<--json>\n\n\n<{config:account}>\n",
- "user query text is not config-interpolated")
- }
-
- private static func testGitcrawlCommandArgumentsInferRepository() throws {
- let directory = FileManager.default.temporaryDirectory
- .appendingPathComponent("crawlbar-gitcrawl-\(UUID().uuidString)", isDirectory: true)
- try FileManager.default.createDirectory(at: directory, withIntermediateDirectories: true)
- defer { try? FileManager.default.removeItem(at: directory) }
-
- let scriptURL = directory.appendingPathComponent("print-args.sh")
- try Data("""
- #!/bin/sh
- printf '%s' "$*"
- """.utf8).write(to: scriptURL)
- try FileManager.default.setAttributes([.posixPermissions: 0o755], ofItemAtPath: scriptURL.path)
-
- let configURL = directory.appendingPathComponent("gitcrawl.toml")
- let databaseURL = directory.appendingPathComponent("openclaw__openclaw.sync.db")
- try Data("db_path = \"\(databaseURL.path)\"\n".utf8).write(to: configURL)
-
- let manifest = CrawlAppManifest(
- id: BuiltInCrawlApps.gitcrawlID,
- displayName: "Git Crawl",
- description: "A git crawler",
- binary: .init(name: scriptURL.path),
- branding: .init(symbolName: "tray", accentColor: "#123456"),
- paths: .init(defaultConfig: configURL.path),
- commands: ["refresh": ["sync", "--json"], "query": ["search", "--json"]],
- capabilities: [.refresh, .search])
- let installation = CrawlAppInstallation(manifest: manifest, binaryPath: scriptURL.path)
-
- let refreshResult = try CrawlCommandRunner().run(installation: installation, action: "refresh", timeoutSeconds: 5)
- try Self.expect(refreshResult.stdout == "sync --json openclaw/openclaw", "gitcrawl refresh infers repository")
-
- let queryResult = try CrawlCommandRunner().run(
- installation: installation,
- action: "query",
- extraArguments: ["stale", "branches"],
- timeoutSeconds: 5)
- try Self.expect(
- queryResult.stdout == "search --json openclaw/openclaw --query stale branches",
- "gitcrawl query infers repository and joins query text")
-
- let storeURL = directory.appendingPathComponent("stores/generic-store", isDirectory: true)
- let genericConfigURL = directory.appendingPathComponent("gitcrawl-generic.toml")
- let genericDatabaseURL = storeURL.appendingPathComponent("data/gitcrawl.db")
- let reportURL = storeURL.appendingPathComponent("reports/latest-status.json")
- try FileManager.default.createDirectory(
- at: genericDatabaseURL.deletingLastPathComponent(),
- withIntermediateDirectories: true)
- try FileManager.default.createDirectory(
- at: reportURL.deletingLastPathComponent(),
- withIntermediateDirectories: true)
- try Data("db_path = \"\(genericDatabaseURL.path)\"\n".utf8).write(to: genericConfigURL)
- try Data(#"{"repository":{"owner":"openclaw","name":"openclaw"}}"#.utf8).write(to: reportURL)
- let genericManifest = CrawlAppManifest(
- id: BuiltInCrawlApps.gitcrawlID,
- displayName: "Git Crawl",
- description: "A git crawler",
- binary: .init(name: scriptURL.path),
- branding: .init(symbolName: "tray", accentColor: "#123456"),
- paths: .init(defaultConfig: genericConfigURL.path),
- commands: ["query": ["search", "--json"]],
- capabilities: [.search])
- let genericInstallation = CrawlAppInstallation(manifest: genericManifest, binaryPath: scriptURL.path)
- let genericQueryResult = try CrawlCommandRunner().run(
- installation: genericInstallation,
- action: "query",
- extraArguments: ["manifest"],
- timeoutSeconds: 5)
- try Self.expect(
- genericQueryResult.stdout == "search --json openclaw/openclaw --query manifest",
- "gitcrawl query infers repository from latest report when db filename is generic")
- let genericStatus = GitcrawlStatusSnapshot.status(for: genericInstallation)
- try Self.expect(
- genericStatus?.databasePath == genericDatabaseURL.path,
- "gitcrawl status keeps the database path that supplied its adjacent report")
- let nativeSearchManifest = CrawlAppManifest(
- id: BuiltInCrawlApps.gitcrawlID,
- displayName: "Git Crawl",
- description: "A metadata-derived git crawler",
- binary: .init(name: scriptURL.path),
- branding: .init(symbolName: "tray", accentColor: "#123456"),
- paths: .init(defaultConfig: genericConfigURL.path),
- commands: ["search": ["search", "--json"]],
- capabilities: [.search])
- let nativeSearchInstallation = CrawlAppInstallation(manifest: nativeSearchManifest, binaryPath: scriptURL.path)
- let nativeSearchResult = try CrawlCommandRunner().run(
- installation: nativeSearchInstallation,
- action: "search",
- extraArguments: ["manifest"],
- timeoutSeconds: 5)
- try Self.expect(
- nativeSearchResult.stdout == "search --json openclaw/openclaw --query manifest",
- "metadata-derived gitcrawl search infers repository and query flag")
-
- let missingReportConfigURL = directory.appendingPathComponent("gitcrawl-missing-report.toml")
- let missingReportDatabaseURL = directory.appendingPathComponent("other-store/data/gitcrawl.db")
- try FileManager.default.createDirectory(
- at: missingReportDatabaseURL.deletingLastPathComponent(),
- withIntermediateDirectories: true)
- try Data("db_path = \"\(missingReportDatabaseURL.path)\"\n".utf8).write(to: missingReportConfigURL)
- let missingReportManifest = CrawlAppManifest(
- id: BuiltInCrawlApps.gitcrawlID,
- displayName: "Git Crawl",
- description: "A git crawler",
- binary: .init(name: scriptURL.path),
- branding: .init(symbolName: "tray", accentColor: "#123456"),
- paths: .init(
- defaultConfig: missingReportConfigURL.path,
- defaultDatabase: directory.appendingPathComponent("wrong__repo.sync.db").path),
- commands: ["query": ["search", "--json"]],
- capabilities: [.search])
- let missingReportInstallation = CrawlAppInstallation(manifest: missingReportManifest, binaryPath: scriptURL.path)
- let missingReportQueryResult = try CrawlCommandRunner().run(
- installation: missingReportInstallation,
- action: "query",
- extraArguments: ["manifest"],
- timeoutSeconds: 5)
- try Self.expect(
- missingReportQueryResult.stdout == "search --json manifest",
- "gitcrawl query does not infer repository from unrelated global reports")
- try Self.expect(
- GitcrawlStatusSnapshot.status(for: missingReportInstallation) == nil,
- "gitcrawl status does not read unrelated global reports for explicit database configs")
- }
-
- private static func testGogStatusServiceVerifiesOAuthOrServiceAccount() throws {
- let directory = FileManager.default.temporaryDirectory
- .appendingPathComponent("crawlbar-gog-status-\(UUID().uuidString)", isDirectory: true)
- try FileManager.default.createDirectory(at: directory, withIntermediateDirectories: true)
- defer { try? FileManager.default.removeItem(at: directory) }
-
- let scriptURL = directory.appendingPathComponent("gog")
- try Data("""
- #!/bin/sh
- if [ "$2" = "list" ]; then
- printf '%s' '{"accounts":[{"email":"user@example.com","auth":"oauth","valid":true}]}'
- else
- printf '%s' '{"status":"ok","checks":[{"name":"tokens","status":"ok","detail":"1 readable OAuth token"}]}'
- fi
- """.utf8).write(to: scriptURL)
- try FileManager.default.setAttributes([.posixPermissions: 0o755], ofItemAtPath: scriptURL.path)
-
- let manifest = CrawlAppManifest(
- id: BuiltInCrawlApps.gogcliID,
- displayName: "Google",
- description: "A Google crawler",
- binary: .init(name: scriptURL.path),
- branding: .init(symbolName: "g.circle", accentColor: "#4285F4"),
- paths: .init(),
- commands: [
- "status": ["auth", "list", "--check", "--json", "--no-input"],
- "doctor": ["auth", "doctor", "--check", "--json", "--no-input"],
- ],
- capabilities: [.status, .doctor])
- let installation = CrawlAppInstallation(manifest: manifest, binaryPath: scriptURL.path)
- let status = CrawlStatusService().status(for: installation, timeoutSeconds: 5)
- try Self.expect(status.state == .current, "gog status service verifies OAuth with doctor")
-
- let serviceScriptURL = directory.appendingPathComponent("gog-service")
- try Data("""
- #!/bin/sh
- if [ "$2" = "list" ]; then
- printf '%s' '{"accounts":[{"email":"admin@example.com","auth":"service_account","valid":true,"error":"service account (not checked)"}]}'
- else
- printf '%s' '{"status":"warn","checks":[{"name":"tokens","status":"warn","detail":"no OAuth tokens"}]}'
- fi
- """.utf8).write(to: serviceScriptURL)
- try FileManager.default.setAttributes([.posixPermissions: 0o755], ofItemAtPath: serviceScriptURL.path)
-
- let serviceManifest = CrawlAppManifest(
- id: BuiltInCrawlApps.gogcliID,
- displayName: "Google",
- description: "A Google crawler",
- binary: .init(name: serviceScriptURL.path),
- branding: .init(symbolName: "g.circle", accentColor: "#4285F4"),
- paths: .init(),
- commands: manifest.commands,
- capabilities: [.status, .doctor])
- let serviceInstallation = CrawlAppInstallation(manifest: serviceManifest, binaryPath: serviceScriptURL.path)
- let serviceStatus = CrawlStatusService().status(for: serviceInstallation, timeoutSeconds: 5)
- try Self.expect(serviceStatus.state == .current, "gog status service accepts service-account-only auth")
- }
-
- private static func testActionFailuresPreserveStatusMetadata() throws {
- let lastSyncAt = Date(timeIntervalSince1970: 1_775_000_000)
- let metadata = CrawlAppStatus(
- appID: BuiltInCrawlApps.discrawlID,
- state: .current,
- summary: "5052 messages",
- configPath: "/tmp/discrawl.toml",
- databasePath: "/tmp/discrawl.db",
- databaseBytes: 36_397_056,
- lastSyncAt: lastSyncAt,
- counts: [CrawlCount(id: "messages", label: "Messages", value: 5052)],
- databases: [
- CrawlDatabaseResource(
- id: "primary",
- label: "Discord archive",
- kind: .sqlite,
- path: "/tmp/discrawl.db",
- isPrimary: true,
- bytes: 36_397_056),
- ],
- freshness: CrawlFreshness(status: .current, ageSeconds: 12, staleAfterSeconds: 86_400),
- share: CrawlShareStatus(enabled: true, repoPath: "/tmp/share", remote: "origin", branch: "main"),
- warnings: ["old warning"])
- let failure = CrawlAppStatus(
- appID: BuiltInCrawlApps.discrawlID,
- state: .error,
- summary: "refresh: network timed out",
- errors: ["network timed out"])
-
- let merged = metadata.mergingActionFailure(failure)
- try Self.expect(merged.state == .error, "action failure state is visible")
- try Self.expect(merged.summary == "refresh: network timed out", "action failure summary is visible")
- try Self.expect(merged.databasePath == metadata.databasePath, "action failure preserves database path")
- try Self.expect(merged.databases == metadata.databases, "action failure preserves databases")
- try Self.expect(merged.counts == metadata.counts, "action failure preserves counts")
- try Self.expect(merged.lastSyncAt == lastSyncAt, "action failure preserves last sync")
- try Self.expect(merged.share == metadata.share, "action failure preserves share metadata")
- try Self.expect(merged.warnings == ["old warning"], "action failure preserves existing warnings")
- try Self.expect(merged.errors == ["network timed out"], "action failure keeps failure errors")
-
- let githubAuthMessage = """
- [github] request GET /repos/openclaw/openclaw
- github GET /repos/openclaw/openclaw failed with status 401: {
- "message": "Bad credentials"
- }
- """
- let githubAuthFailure = CrawlAppStatus.commandFailure(
- appID: BuiltInCrawlApps.gitcrawlID,
- action: "refresh",
- message: githubAuthMessage,
- fallback: "refresh failed")
- try Self.expect(githubAuthFailure.state == .needsAuth, "gitcrawl auth failure action is recoverable")
- try Self.expect(
- githubAuthFailure.summary == "refresh: GitHub credentials rejected",
- "gitcrawl auth failure action summary is useful")
-
- let githubMetadata = CrawlAppStatus(
- appID: BuiltInCrawlApps.gitcrawlID,
- state: .current,
- summary: "1 repo",
- databasePath: "/tmp/gitcrawl.db",
- counts: [CrawlCount(id: "repositories", label: "Repositories", value: 1)])
- let mergedGithub = githubMetadata.mergingActionFailure(githubAuthFailure)
- try Self.expect(mergedGithub.state == .needsAuth, "action failure merge preserves auth state")
- try Self.expect(mergedGithub.summary == "refresh: GitHub credentials rejected", "action failure merge preserves auth summary")
- try Self.expect(mergedGithub.databasePath == githubMetadata.databasePath, "action failure merge preserves github metadata")
-
- let emptyRefreshed = CrawlAppStatus(
- appID: BuiltInCrawlApps.discrawlID,
- state: .error,
- summary: "status failed")
- let richest = CrawlAppStatus.richestMetadataStatus(emptyRefreshed, fallback: metadata)
- try Self.expect(richest == metadata, "previous rich metadata wins over empty refreshed status")
-
- let recoverableGraincrawlStatus = CrawlAppStatus(
- appID: BuiltInCrawlApps.graincrawlID,
- state: .error,
- summary: "private-api reports expired token, desktop-cache reports unsupported cache version 8")
- try Self.expect(
- recoverableGraincrawlStatus.isRecoverableGraincrawlSourceFailure,
- "graincrawl source status failures can render as stale")
- let graincrawlActionFailure = recoverableGraincrawlStatus.mergingActionFailure(CrawlAppStatus(
- appID: BuiltInCrawlApps.graincrawlID,
- state: .error,
- summary: "refresh: Granola access token expired",
- errors: ["Granola access token expired"]))
- try Self.expect(
- !graincrawlActionFailure.isRecoverableGraincrawlSourceFailure,
- "graincrawl action failures stay visible as errors")
- }
-
- private static func testStatusMapperNormalizesWacliDoctorOutput() throws {
- let result = CrawlCommandResult(
- appID: CrawlAppID(rawValue: "wacli-test"),
- action: "status",
- exitCode: 0,
- stdout: """
- {"success":true,"data":{"state":"current","store_dir":"/tmp/wacli-store","lock_held":true,"connection_state":"locked_by_other_process","authenticated":true,"fts_enabled":false,"store":{"messages":6991,"chats":677,"contacts":514,"groups":250,"last_sync_at":"2026-05-09T05:45:44Z"}},"error":null}
- """,
- stderr: "",
- startedAt: Date(timeIntervalSince1970: 1_775_000_000),
- finishedAt: Date(timeIntervalSince1970: 1_775_000_001))
- let manifest = CrawlAppManifest(
- id: result.appID,
- displayName: "WhatsApp Test",
- description: "Remote WhatsApp archive",
- binary: .init(name: "ssh"),
- branding: .init(symbolName: "message.circle", accentColor: "#25D366"),
- paths: .init(),
- commands: ["status": ["host", "wacli --account test --read-only doctor --json"]],
- capabilities: [.status])
- let status = CrawlStatusMapper().status(from: result, manifest: manifest, staleAfterSeconds: 900)
-
- try Self.expect(status.state == .current, "wacli doctor honors explicit current state over stale timestamps")
- try Self.expect(status.freshness?.status == .stale, "wacli doctor still exposes stale freshness metadata")
- try Self.expect(status.summary == "6991 messages, 677 chats", "wacli doctor maps store counts")
- try Self.expect(status.databasePath == "/tmp/wacli-store/wacli.db", "wacli doctor maps database path")
- try Self.expect(status.lastSyncAt != nil, "wacli doctor maps last sync")
- try Self.expect(status.warnings.contains("Store is locked by locked_by_other_process"), "wacli lock is a warning")
- try Self.expect(status.warnings.contains("Full-text search is not enabled"), "wacli FTS state is a warning")
- }
-
- private static func testStatusMapperNormalizesGogAuthStatus() throws {
- let needsAuthResult = CrawlCommandResult(
- appID: BuiltInCrawlApps.gogcliID,
- action: "status",
- exitCode: 0,
- stdout: """
- {"account":{"credentials_exists":false,"service_account_configured":false,"email":""},"config":{"exists":false,"path":"/tmp/gog/config.json"},"keyring":{"backend":"auto","source":"default"}}
- """,
- stderr: "",
- startedAt: Date(),
- finishedAt: Date())
- let needsAuth = CrawlStatusMapper().status(from: needsAuthResult, manifest: BuiltInCrawlApps.gogcli)
- try Self.expect(needsAuth.state == .needsAuth, "gog auth status without credentials maps to needs auth")
- try Self.expect(needsAuth.summary == "Google account needs auth", "gog auth status has a useful setup summary")
- try Self.expect(needsAuth.configPath == "/tmp/gog/config.json", "gog auth status maps config path")
-
- let readyResult = CrawlCommandResult(
- appID: BuiltInCrawlApps.gogcliID,
- action: "status",
- exitCode: 0,
- stdout: """
- {"account":{"credentials_exists":true,"service_account_configured":false,"email":"user@example.com"},"config":{"exists":true,"path":"/tmp/gog/config.json"}}
- """,
- stderr: "",
- startedAt: Date(),
- finishedAt: Date())
- let ready = CrawlStatusMapper().status(from: readyResult, manifest: BuiltInCrawlApps.gogcli)
- try Self.expect(ready.state == .needsAuth, "gog raw credentials still require verified token auth")
- try Self.expect(ready.summary == "Google account needs auth", "gog raw credentials keep setup summary")
-
- let doctorResult = CrawlCommandResult(
- appID: BuiltInCrawlApps.gogcliID,
- action: "status",
- exitCode: 0,
- stdout: """
- {"checks":[{"name":"config.path","status":"warn","detail":"/tmp/gog/config.json (missing)"},{"name":"keyring.open","status":"ok","detail":"opened"},{"name":"tokens","status":"ok","detail":"4 readable OAuth tokens of 4 stored token accounts"},{"name":"refresh.default.user@example.com","status":"ok","detail":"refresh token exchange succeeded"}],"status":"warn"}
- """,
- stderr: "",
- startedAt: Date(),
- finishedAt: Date())
- let doctor = CrawlStatusMapper().status(from: doctorResult, manifest: BuiltInCrawlApps.gogcli)
- try Self.expect(doctor.state == .current, "gog doctor maps readable refreshable tokens to current")
- try Self.expect(doctor.summary == "4 Google OAuth accounts readable", "gog doctor summarizes readable tokens")
- try Self.expect(doctor.warnings.contains("config.path: /tmp/gog/config.json (missing)"), "gog doctor preserves non-auth warnings")
- }
-
- private static func testStatusMapperNormalizesBirdclawAuthStatus() throws {
- let result = CrawlCommandResult(
- appID: BuiltInCrawlApps.birdclawID,
- action: "status",
- exitCode: 0,
- stdout: """
- {"installed":false,"availableTransport":"local","statusText":"xurl not installed. local mode active."}
- """,
- stderr: "",
- startedAt: Date(),
- finishedAt: Date())
- let status = CrawlStatusMapper().status(from: result, manifest: BuiltInCrawlApps.birdclaw)
- try Self.expect(status.state == .current, "birdclaw auth status keeps local mode usable")
- try Self.expect(status.summary == "xurl not installed. local mode active.", "birdclaw auth status has a useful summary")
- try Self.expect(status.warnings.contains("Transport: local"), "birdclaw auth status exposes transport")
- try Self.expect(status.warnings.contains("xurl not installed. local mode active."), "birdclaw auth status preserves transport warning")
-
- let birdResult = CrawlCommandResult(
- appID: BuiltInCrawlApps.birdclawID,
- action: "status",
- exitCode: 0,
- stdout: """
- [info] Credential check
- [ok] auth_token: abc...
- [ok] ct0: def...
- source: Chrome default profile
- [warn] Warnings:
- - No Twitter cookies found in Safari.
- [ok] Ready to tweet!
- """,
- stderr: "",
- startedAt: Date(),
- finishedAt: Date())
- let birdStatus = CrawlStatusMapper().status(from: birdResult, manifest: BuiltInCrawlApps.birdclaw)
- try Self.expect(birdStatus.state == .current, "bird check text maps to current when cookies are usable")
- try Self.expect(birdStatus.summary == "X cookies available via bird (Chrome default profile)", "bird check text exposes cookie source")
- try Self.expect(birdStatus.warnings.contains("No Twitter cookies found in Safari."), "bird check warnings are preserved")
- }
-
- private static func testStatusMapperTrustsCrawlerState() throws {
- let result = CrawlCommandResult(
- appID: BuiltInCrawlApps.discrawlID,
- action: "status",
- exitCode: 0,
- stdout: """
- {"schema_version":"crawlkit.control.v1","state":"current","summary":"ok","last_sync_at":"2026-05-09T05:45:44Z","counts":[{"id":"messages","label":"Messages","value":10}]}
- """,
- stderr: "",
- startedAt: Date(timeIntervalSince1970: 1_775_000_000),
- finishedAt: Date(timeIntervalSince1970: 1_775_000_001))
- let status = CrawlStatusMapper().status(
- from: result,
- manifest: BuiltInCrawlApps.discrawl,
- staleAfterSeconds: 900)
-
- try Self.expect(status.state == .current, "explicit crawler state wins over stale timestamp heuristics")
- try Self.expect(status.freshness?.status == .stale, "stale timestamp can still be shown as metadata")
- }
-
- private static func testActionLogStoreReadsRecentResults() throws {
- let directory = FileManager.default.temporaryDirectory
- .appendingPathComponent("crawlbar-logs-\(UUID().uuidString)", isDirectory: true)
- let store = CrawlActionLogStore(directoryURL: directory)
- defer { try? FileManager.default.removeItem(at: directory) }
-
- let result = CrawlCommandResult(
- appID: BuiltInCrawlApps.graincrawlID,
- action: "refresh",
- exitCode: 1,
- stdout: "",
- stderr: "Granola access token expired",
- startedAt: Date(timeIntervalSince1970: 1_775_000_000),
- finishedAt: Date(timeIntervalSince1970: 1_775_000_001))
- _ = try store.save(result)
-
- let recent = store.recentResults(limit: 5)
- try Self.expect(recent.first == result, "action logs decode back into recent command results")
-
- let successfulJSONResult = CrawlCommandResult(
- appID: BuiltInCrawlApps.graincrawlID,
- action: "refresh",
- exitCode: 0,
- stdout: """
- {"notes":1}
- """,
- stderr: "",
- startedAt: Date(timeIntervalSince1970: 1_775_000_002),
- finishedAt: Date(timeIntervalSince1970: 1_775_000_003))
- try Self.expect(successfulJSONResult.userFacingRunMessage == nil, "successful stdout is not shown as a run message")
- try Self.expect(!successfulJSONResult.shouldShowExitCode, "successful runs do not show exit code")
-
- let warningResult = CrawlCommandResult(
- appID: BuiltInCrawlApps.graincrawlID,
- action: "refresh",
- exitCode: 0,
- stdout: """
- {"notes":1}
- """,
- stderr: "Used cached Granola data",
- startedAt: Date(timeIntervalSince1970: 1_775_000_004),
- finishedAt: Date(timeIntervalSince1970: 1_775_000_005))
- try Self.expect(warningResult.userFacingRunMessage == "Used cached Granola data", "successful stderr can still surface a warning")
-
- let failedGitHubResult = CrawlCommandResult(
- appID: BuiltInCrawlApps.gitcrawlID,
- action: "refresh",
- exitCode: 1,
- stdout: "",
- stderr: """
- [github] request GET /repos/openclaw/openclaw
- github GET /repos/openclaw/openclaw failed with status 401: {
- "message": "Bad credentials"
- }
- """,
- startedAt: Date(timeIntervalSince1970: 1_775_000_006),
- finishedAt: Date(timeIntervalSince1970: 1_775_000_007))
- try Self.expect(failedGitHubResult.userFacingRunMessage == "GitHub credentials rejected", "failed gitcrawl run message is normalized")
-
- let failedBirdResult = CrawlCommandResult(
- appID: BuiltInCrawlApps.birdclawID,
- action: "status",
- exitCode: 1,
- stdout: "",
- stderr: "Missing auth_token",
- startedAt: Date(timeIntervalSince1970: 1_775_000_006),
- finishedAt: Date(timeIntervalSince1970: 1_775_000_007))
- try Self.expect(failedBirdResult.userFacingRunMessage == "X browser cookies not found", "failed X credential check maps to auth setup")
-
- let failedStdoutResult = CrawlCommandResult(
- appID: BuiltInCrawlApps.graincrawlID,
- action: "refresh",
- exitCode: 1,
- stdout: "Granola refresh failed",
- stderr: "",
- startedAt: Date(timeIntervalSince1970: 1_775_000_008),
- finishedAt: Date(timeIntervalSince1970: 1_775_000_009))
- try Self.expect(failedStdoutResult.userFacingRunMessage == "Granola refresh failed", "failed stdout is shown as a run message")
- try Self.expect(failedStdoutResult.shouldShowExitCode, "failed runs show exit code")
- }
-
- private static func testCommandTimeoutEscalates() throws {
- let directory = FileManager.default.temporaryDirectory
- .appendingPathComponent("crawlbar-timeout-\(UUID().uuidString)", isDirectory: true)
- try FileManager.default.createDirectory(at: directory, withIntermediateDirectories: true)
- defer { try? FileManager.default.removeItem(at: directory) }
-
- let scriptURL = directory.appendingPathComponent("ignore-term.sh")
- try Data("""
- #!/bin/sh
- trap '' TERM
- sleep 5
- """.utf8).write(to: scriptURL)
- try FileManager.default.setAttributes([.posixPermissions: 0o755], ofItemAtPath: scriptURL.path)
-
- let manifest = CrawlAppManifest(
- id: CrawlAppID(rawValue: "timeoutcrawl"),
- displayName: "Timeout Crawl",
- description: "A timeout test crawler",
- binary: .init(name: scriptURL.path),
- branding: .init(symbolName: "timer", accentColor: "#123456"),
- paths: .init(),
- commands: ["status": []],
- capabilities: [.status])
- let installation = CrawlAppInstallation(manifest: manifest, binaryPath: scriptURL.path)
- let startedAt = Date()
- do {
- _ = try CrawlCommandRunner().run(installation: installation, action: "status", timeoutSeconds: 0.1)
- throw SelfTestError.failed("timeout command should not complete")
- } catch CrawlCommandRunnerError.timedOut {
- try Self.expect(Date().timeIntervalSince(startedAt) < 2.5, "timed-out commands are killed promptly")
- }
- }
-
- private static func testDatabaseBackupCopiesFiles() throws {
- let directory = FileManager.default.temporaryDirectory
- .appendingPathComponent("crawlbar-backup-\(UUID().uuidString)", isDirectory: true)
- try FileManager.default.createDirectory(at: directory, withIntermediateDirectories: true)
- defer { try? FileManager.default.removeItem(at: directory) }
-
- let firstDirectory = directory.appendingPathComponent("first", isDirectory: true)
- let secondDirectory = directory.appendingPathComponent("second", isDirectory: true)
- try FileManager.default.createDirectory(at: firstDirectory, withIntermediateDirectories: true)
- try FileManager.default.createDirectory(at: secondDirectory, withIntermediateDirectories: true)
- let firstDatabaseURL = firstDirectory.appendingPathComponent("sample.db")
- let secondDatabaseURL = secondDirectory.appendingPathComponent("sample.db")
- try Self.createSQLiteDatabase(firstDatabaseURL, value: "sqlite-one")
- try Self.createSQLiteDatabase(secondDatabaseURL, value: "sqlite-two")
- let status = CrawlAppStatus(
- appID: BuiltInCrawlApps.notcrawlID,
- state: .current,
- summary: "ok",
- databases: [
- CrawlDatabaseResource(
- id: firstDatabaseURL.path,
- label: "Workspace One",
- kind: .sqlite,
- path: firstDatabaseURL.path,
- isPrimary: true),
- CrawlDatabaseResource(
- id: secondDatabaseURL.path,
- label: "Workspace Two",
- kind: .sqlite,
- path: secondDatabaseURL.path,
- isPrimary: false),
- ])
-
- let backup = try CrawlDatabaseBackupStore.backup(status: status, root: directory.appendingPathComponent("backups", isDirectory: true))
- try Self.expect(backup.files.count == 2, "backup copies duplicate-named files")
- try Self.expect(Set(backup.files.map { URL(fileURLWithPath: $0).lastPathComponent }).count == 2, "backup destination names are unique")
- let copiedContents = try backup.files.map { try Self.sqliteValue(URL(fileURLWithPath: $0)) }
- try Self.expect(copiedContents.contains("sqlite-one"), "backup preserves first duplicate file")
- try Self.expect(copiedContents.contains("sqlite-two"), "backup preserves second duplicate file")
- }
-
- private static func testRedactorScrubsSecrets() throws {
- let redacted = CrawlCommandRedactor().redact("""
- token=abc123
- Authorization: Bearer secret-token
- discord_token=discord-secret
- github_pat_1234567890abcdef
- ghp_1234567890abcdef
- sk-proj-1234567890abcdef
- xoxc-1234567890abcdef
- secret_notion123
- mfa.discordsecret
- ct0: csrf-secret
- label=Discord archive
- """)
- try Self.expect(!redacted.contains("abc123"), "token value redacts")
- try Self.expect(!redacted.contains("secret-token"), "bearer value redacts")
- try Self.expect(!redacted.contains("discord-secret"), "discord token value redacts")
- try Self.expect(!redacted.contains("1234567890abcdef"), "bare tokens redact")
- try Self.expect(!redacted.contains("notion123"), "notion secrets redact")
- try Self.expect(!redacted.contains("csrf-secret"), "ct0 cookies redact")
- try Self.expect(redacted.contains("Discord archive"), "discord labels are not redacted")
- }
-
- private static func expect(_ condition: Bool, _ message: String) throws {
- if !condition {
- throw SelfTestError.failed(message)
- }
- }
-
- private static func createSQLiteDatabase(_ url: URL, value: String) throws {
- try Self.runSQLite(url, sql: "create table sample(value text); insert into sample(value) values('\(value)');")
- }
-
- private static func sqliteValue(_ url: URL) throws -> String {
- try Self.runSQLite(url, sql: "select value from sample limit 1;")
- }
-
- @discardableResult
- private static func runSQLite(_ url: URL, sql: String) throws -> String {
- guard let sqlitePath = CrawlExecutableResolver().resolve("sqlite3") else {
- throw SelfTestError.failed("sqlite3 is available")
- }
- let process = Process()
- process.executableURL = URL(fileURLWithPath: sqlitePath)
- process.arguments = [url.path, sql]
- let output = Pipe()
- process.standardOutput = output
- process.standardError = output
- try process.run()
- process.waitUntilExit()
- let data = output.fileHandleForReading.readDataToEndOfFile()
- let text = String(data: data, encoding: .utf8) ?? ""
- guard process.terminationStatus == 0 else {
- throw SelfTestError.failed("sqlite3 failed: \(text)")
- }
- return text.trimmingCharacters(in: .whitespacesAndNewlines)
- }
-}
-
-private enum SelfTestError: LocalizedError {
- case failed(String)
-
- var errorDescription: String? {
- switch self {
- case let .failed(message):
- "selftest failed: \(message)"
- }
- }
}
diff --git a/Sources/CrawlBarSelfTest/SelfTestCommandRunner.swift b/Sources/CrawlBarSelfTest/SelfTestCommandRunner.swift
new file mode 100644
index 0000000..e99712a
--- /dev/null
+++ b/Sources/CrawlBarSelfTest/SelfTestCommandRunner.swift
@@ -0,0 +1,278 @@
+import CrawlBarCore
+import Foundation
+
+extension CrawlBarSelfTest {
+ static func testConfigValuesReachCommandEnvironment() throws {
+ let directory = FileManager.default.temporaryDirectory
+ .appendingPathComponent("crawlbar-env-\(UUID().uuidString)", isDirectory: true)
+ try FileManager.default.createDirectory(at: directory, withIntermediateDirectories: true)
+ defer { try? FileManager.default.removeItem(at: directory) }
+
+ let scriptURL = directory.appendingPathComponent("print-env.sh")
+ try Data("""
+ #!/bin/sh
+ if [ "$#" -gt 0 ]; then
+ printf '%s|%s' "$CRAWLBAR_TEST_VALUE" "$*"
+ else
+ printf '%s' "$CRAWLBAR_TEST_VALUE"
+ fi
+ """.utf8).write(to: scriptURL)
+ try FileManager.default.setAttributes([.posixPermissions: 0o755], ofItemAtPath: scriptURL.path)
+
+ let manifest = CrawlAppManifest(
+ id: CrawlAppID(rawValue: "envcrawl"),
+ displayName: "Env Crawl",
+ description: "A test crawler",
+ binary: .init(name: scriptURL.path),
+ branding: .init(symbolName: "terminal", accentColor: "#123456"),
+ paths: .init(),
+ commands: ["status": [], "query": ["query"]],
+ capabilities: [.status],
+ configOptions: [
+ .init(id: "test_value", label: "Test Value", envVar: "CRAWLBAR_TEST_VALUE"),
+ ])
+ let installation = CrawlAppInstallation(
+ manifest: manifest,
+ binaryPath: scriptURL.path,
+ configValues: ["test_value": "from-config"])
+ let result = try CrawlCommandRunner().run(installation: installation, action: "status", timeoutSeconds: 5)
+ try Self.expect(result.stdout == "from-config", "config values reach crawler command environment")
+ let queryResult = try CrawlCommandRunner().run(
+ installation: installation,
+ action: "query",
+ extraArguments: ["select count(*) from items"],
+ timeoutSeconds: 5)
+ try Self.expect(queryResult.stdout == "from-config|query select count(*) from items", "query arguments reach crawler commands")
+ }
+
+ static func testExecutableResolverUsesMacCliFallbackPaths() throws {
+ let directory = FileManager.default.temporaryDirectory
+ .appendingPathComponent("crawlbar-path-\(UUID().uuidString)", isDirectory: true)
+ let localBinURL = directory.appendingPathComponent(".local/bin", isDirectory: true)
+ try FileManager.default.createDirectory(at: localBinURL, withIntermediateDirectories: true)
+ defer { try? FileManager.default.removeItem(at: directory) }
+
+ let scriptURL = localBinURL.appendingPathComponent("fallbackcrawl")
+ try Data("""
+ #!/bin/sh
+ printf '%s' "$PATH"
+ """.utf8).write(to: scriptURL)
+ try FileManager.default.setAttributes([.posixPermissions: 0o755], ofItemAtPath: scriptURL.path)
+
+ let environment = [
+ "HOME": directory.path,
+ "PATH": "/usr/bin:/bin:/usr/sbin:/sbin",
+ ]
+ let resolver = CrawlExecutableResolver(environment: environment)
+ try Self.expect(
+ resolver.resolve("fallbackcrawl") == scriptURL.path,
+ "resolver checks user CLI fallback paths")
+
+ let manifest = CrawlAppManifest(
+ id: CrawlAppID(rawValue: "fallbackcrawl"),
+ displayName: "Fallback Crawl",
+ description: "A fallback PATH crawler",
+ binary: .init(name: "fallbackcrawl"),
+ branding: .init(symbolName: "terminal", accentColor: "#123456"),
+ paths: .init(),
+ commands: ["status": []],
+ capabilities: [.status])
+ let installation = CrawlAppInstallation(manifest: manifest, binaryPath: "fallbackcrawl")
+ let result = try CrawlCommandRunner(resolver: resolver, environment: environment)
+ .run(installation: installation, action: "status", timeoutSeconds: 5)
+
+ try Self.expect(
+ result.stdout.split(separator: ":").contains(Substring(localBinURL.path)),
+ "runner passes normalized fallback PATH to crawlers")
+ try Self.expect(
+ CrawlProcessEnvironment.normalized(["PATH": "/usr/bin"])["HOME"]?.isEmpty == false,
+ "normalized environment supplies HOME for launchd crawler commands")
+ }
+
+ static func testRegistryResolvesBirdclawAccessPathBinary() throws {
+ let directory = FileManager.default.temporaryDirectory
+ .appendingPathComponent("crawlbar-birdclaw-\(UUID().uuidString)", isDirectory: true)
+ try FileManager.default.createDirectory(at: directory, withIntermediateDirectories: true)
+ defer { try? FileManager.default.removeItem(at: directory) }
+
+ let binaryURL = directory.appendingPathComponent("birdclaw")
+ try Data("#!/bin/sh\n".utf8).write(to: binaryURL)
+ try FileManager.default.setAttributes([.posixPermissions: 0o755], ofItemAtPath: binaryURL.path)
+
+ let configURL = directory.appendingPathComponent("config.json")
+ let store = CrawlBarConfigStore(fileURL: configURL)
+ try store.save(CrawlBarConfig(apps: [
+ CrawlBarAppConfig(
+ id: BuiltInCrawlApps.birdclawID,
+ configValues: ["access_path": "birdclaw"]),
+ ]))
+ let registry = CrawlAppRegistry(
+ configStore: store,
+ resolver: CrawlExecutableResolver(environment: [
+ "HOME": directory.path,
+ "PATH": directory.path,
+ ]))
+ let installation = try registry.installation(for: BuiltInCrawlApps.birdclawID)
+ try Self.expect(installation?.binaryPath == binaryURL.path, "Birdclaw access path resolves birdclaw binary")
+ }
+
+ static func testRemoteSshExecutionBuildsCommand() throws {
+ let directory = FileManager.default.temporaryDirectory
+ .appendingPathComponent("crawlbar-remote-\(UUID().uuidString)", isDirectory: true)
+ try FileManager.default.createDirectory(at: directory, withIntermediateDirectories: true)
+ defer { try? FileManager.default.removeItem(at: directory) }
+
+ let scriptURL = directory.appendingPathComponent("ssh")
+ try Data("""
+ #!/bin/sh
+ printf '%s' "$*"
+ """.utf8).write(to: scriptURL)
+ try FileManager.default.setAttributes([.posixPermissions: 0o755], ofItemAtPath: scriptURL.path)
+ let environment = [
+ "HOME": directory.path,
+ "PATH": directory.path,
+ ]
+ let runner = CrawlCommandRunner(
+ resolver: CrawlExecutableResolver(environment: environment),
+ environment: environment)
+
+ let manifest = CrawlAppManifest(
+ id: CrawlAppID(rawValue: "wacli-test"),
+ displayName: "WhatsApp Test",
+ description: "A remote WhatsApp crawler",
+ binary: .init(name: "wacli"),
+ execution: .init(
+ kind: .local,
+ kindConfigID: "execution_mode",
+ targetConfigID: "remote_target",
+ runAsConfigID: "remote_run_as",
+ remoteBinary: "wacli"),
+ branding: .init(symbolName: "message.circle", accentColor: "#25D366"),
+ paths: .init(),
+ commands: ["search": ["--account", "{config:account}", "--read-only", "--json", "messages", "search"]],
+ capabilities: [.search],
+ configOptions: [
+ .init(id: "execution_mode", label: "Run Location", kind: .choice, defaultValue: "local", choices: ["local", "remote"]),
+ .init(id: "account", label: "Account", defaultValue: "personal"),
+ ])
+ let installation = CrawlAppInstallation(
+ manifest: manifest,
+ binaryPath: scriptURL.path,
+ configValues: [
+ "execution_mode": "remote",
+ "remote_target": "user@example-host",
+ "remote_run_as": "crawl",
+ ])
+ let result = try runner.run(
+ installation: installation,
+ action: "search",
+ extraArguments: ["hello", "world"],
+ timeoutSeconds: 5)
+
+ try Self.expect(
+ result.stdout == #"-- user@example-host 'sudo' '-u' 'crawl' '-H' '--' 'sh' '-lc' 'cd ~ && exec '\''wacli'\'' '\''--account'\'' '\''personal'\'' '\''--read-only'\'' '\''--json'\'' '\''messages'\'' '\''search'\'' '\''hello world'\'''"#,
+ "remote SSH execution builds a quoted remote command with config defaults")
+
+ let optionTargetInstallation = CrawlAppInstallation(
+ manifest: manifest,
+ binaryPath: scriptURL.path,
+ configValues: [
+ "execution_mode": "remote",
+ "remote_target": "-oProxyCommand=/tmp/hook",
+ ])
+ do {
+ _ = try runner.run(
+ installation: optionTargetInstallation,
+ action: "search",
+ extraArguments: ["hello"],
+ timeoutSeconds: 5)
+ throw SelfTestError.failed("remote SSH target rejects option-looking values")
+ } catch CrawlCommandRunnerError.invalidRemoteTarget {
+ }
+
+ let localInstallation = CrawlAppInstallation(
+ manifest: manifest,
+ binaryPath: scriptURL.path,
+ configValues: ["execution_mode": "local"])
+ let localResult = try runner.run(
+ installation: localInstallation,
+ action: "search",
+ extraArguments: ["hello", "world"],
+ timeoutSeconds: 5)
+ try Self.expect(
+ localResult.stdout == "--account personal --read-only --json messages search hello world",
+ "local execution mode bypasses SSH and uses the crawler binary")
+
+ let remoteBirdclawInstallation = CrawlAppInstallation(
+ manifest: BuiltInCrawlApps.birdclaw,
+ binaryPath: scriptURL.path,
+ configValues: [
+ "access_path": "birdclaw",
+ "execution_mode": "remote",
+ "remote_target": "user@example-host",
+ "remote_run_as": "crawl",
+ ])
+ let birdclawResult = try runner.run(
+ installation: remoteBirdclawInstallation,
+ action: "status",
+ timeoutSeconds: 5)
+ try Self.expect(
+ birdclawResult.stdout == #"-- user@example-host 'sudo' '-u' 'crawl' '-H' '--' 'sh' '-lc' 'cd ~ && exec '\''birdclaw'\'' '\''auth'\'' '\''status'\'' '\''--json'\'''"#,
+ "X remote execution can use the Birdclaw/xurl access path")
+
+ let customBirdclawURL = directory.appendingPathComponent("custom-birdclaw")
+ try Data("""
+ #!/bin/sh
+ printf 'custom:%s' "$*"
+ """.utf8).write(to: customBirdclawURL)
+ try FileManager.default.setAttributes([.posixPermissions: 0o755], ofItemAtPath: customBirdclawURL.path)
+ let localBirdclawOverride = CrawlAppInstallation(
+ manifest: BuiltInCrawlApps.birdclaw,
+ binaryPath: customBirdclawURL.path,
+ configValues: ["access_path": "birdclaw"])
+ let localBirdclawResult = try runner.run(
+ installation: localBirdclawOverride,
+ action: "status",
+ timeoutSeconds: 5)
+ try Self.expect(
+ localBirdclawResult.stdout == "custom:auth status --json",
+ "Birdclaw access path preserves binary overrides")
+
+ let envManifest = CrawlAppManifest(
+ id: CrawlAppID(rawValue: "envcrawl-test"),
+ displayName: "Env Crawl Test",
+ description: "A remote crawler that needs an env file",
+ binary: .init(name: "envcrawl-local"),
+ execution: .init(
+ kind: .local,
+ kindConfigID: "execution_mode",
+ targetConfigID: "remote_target",
+ runAsConfigID: "remote_run_as",
+ remoteEnvFileConfigID: "remote_env_file",
+ remoteBinary: "envcrawl"),
+ branding: .init(symbolName: "terminal", accentColor: "#123456"),
+ paths: .init(),
+ commands: ["status": ["status", "--json"]],
+ capabilities: [.status],
+ configOptions: [
+ .init(id: "execution_mode", label: "Run Location", kind: .choice, defaultValue: "local", choices: ["local", "remote"]),
+ .init(id: "remote_env_file", label: "Remote Env File"),
+ ])
+ let envInstallation = CrawlAppInstallation(
+ manifest: envManifest,
+ binaryPath: scriptURL.path,
+ configValues: [
+ "execution_mode": "remote",
+ "remote_target": "user@example-host",
+ "remote_run_as": "crawl",
+ "remote_env_file": "/run/example/env",
+ ])
+ let envResult = try runner.run(
+ installation: envInstallation,
+ action: "status",
+ timeoutSeconds: 5)
+ try Self.expect(
+ envResult.stdout == #"-- user@example-host 'sudo' '-u' 'crawl' '-H' '--' 'sh' '-lc' 'cd ~ && set -a && . '\''/run/example/env'\'' && set +a && exec '\''envcrawl'\'' '\''status'\'' '\''--json'\'''"#,
+ "remote SSH execution can source an env file before exec")
+ }
+}
diff --git a/Sources/CrawlBarSelfTest/SelfTestConfig.swift b/Sources/CrawlBarSelfTest/SelfTestConfig.swift
new file mode 100644
index 0000000..b5d5e4c
--- /dev/null
+++ b/Sources/CrawlBarSelfTest/SelfTestConfig.swift
@@ -0,0 +1,294 @@
+import CrawlBarCore
+import Foundation
+
+extension CrawlBarSelfTest {
+ static func testAppIDSortsByRawValue() throws {
+ try Self.expect(
+ [CrawlAppID(rawValue: "b"), CrawlAppID(rawValue: "a")].sorted().map(\.rawValue) == ["a", "b"],
+ "app ids sort by raw value")
+ }
+
+ static func testDefaultConfigNormalizesBuiltInApps() throws {
+ let config = CrawlBarConfig(apps: []).normalized()
+ try Self.expect(config.version == CrawlBarConfig.currentVersion, "config version normalizes")
+ try Self.expect(config.apps.map(\.id) == BuiltInCrawlApps.all.map(\.id), "built-in apps are present")
+ try Self.expect(config.appConfig(for: BuiltInCrawlApps.gogcliID)?.enabled == true, "new Google app normalizes enabled")
+ try Self.expect(config.appConfig(for: BuiltInCrawlApps.gogcliID)?.showInMenuBar == true, "new Google app appears in menu bar")
+ let oldConfig = CrawlBarConfig(
+ version: 1,
+ apps: [CrawlBarAppConfig(id: BuiltInCrawlApps.wacliID, enabled: false, showInMenuBar: false)]).normalized()
+ try Self.expect(oldConfig.appConfig(for: BuiltInCrawlApps.wacliID)?.enabled == true, "newly available apps migrate from forced disabled")
+ let v2Config = CrawlBarConfig(
+ version: 2,
+ apps: [CrawlBarAppConfig(id: BuiltInCrawlApps.birdclawID, enabled: false, showInMenuBar: false)]).normalized()
+ try Self.expect(v2Config.appConfig(for: BuiltInCrawlApps.birdclawID)?.enabled == true, "newly available Birdclaw migrates from forced disabled")
+ try Self.expect(config.manifestDirectories == ["~/.crawlbar/apps"], "manifest directory default is present")
+ }
+
+ static func testConfigStoreRoundTrips() throws {
+ let directory = FileManager.default.temporaryDirectory
+ .appendingPathComponent("crawlbar-config-\(UUID().uuidString)", isDirectory: true)
+ try FileManager.default.createDirectory(at: directory, withIntermediateDirectories: true)
+ defer { try? FileManager.default.removeItem(at: directory) }
+
+ let url = directory.appendingPathComponent("config.json")
+ let store = CrawlBarConfigStore(fileURL: url)
+ let config = CrawlBarConfig(
+ refreshFrequency: .hourly,
+ apps: [CrawlBarAppConfig(
+ id: BuiltInCrawlApps.gitcrawlID,
+ enabled: false,
+ configValues: ["embedding_model": "text-embedding-3-large"])])
+
+ try store.save(config)
+ guard let loaded = try store.load() else {
+ throw SelfTestError.failed("config loads after save")
+ }
+
+ try Self.expect(loaded.refreshFrequency == .hourly, "refresh frequency round trips")
+ try Self.expect(loaded.appConfig(for: BuiltInCrawlApps.gitcrawlID)?.enabled == false, "app enablement round trips")
+ try Self.expect(loaded.appConfig(for: BuiltInCrawlApps.gitcrawlID)?.configValues["embedding_model"] == "text-embedding-3-large", "app config values round trip")
+ try Self.expect(loaded.apps.count == BuiltInCrawlApps.all.count, "config store normalizes built-ins")
+ }
+
+ static func testExternalManifestCatalog() throws {
+ let directory = FileManager.default.temporaryDirectory
+ .appendingPathComponent("crawlbar-manifest-\(UUID().uuidString)", isDirectory: true)
+ try FileManager.default.createDirectory(at: directory, withIntermediateDirectories: true)
+ defer { try? FileManager.default.removeItem(at: directory) }
+
+ let manifest = CrawlAppManifest(
+ id: CrawlAppID(rawValue: "customcrawl"),
+ displayName: "Custom Crawl",
+ description: "A custom crawl app",
+ binary: .init(name: "customcrawl"),
+ branding: .init(symbolName: "square.grid.2x2", accentColor: "#123456"),
+ paths: .init(defaultConfig: "~/.customcrawl/config.toml"),
+ commands: ["status": ["status", "--json"]],
+ capabilities: [.status])
+ let data = try CrawlCoding.makeJSONEncoder().encode(manifest)
+ try data.write(to: directory.appendingPathComponent("customcrawl.json"))
+ try Data("""
+ {
+ "schema_version": "crawlkit.control.v1",
+ "id": "objectcrawl",
+ "display_name": "Object Crawl",
+ "description": "A crawlkit manifest",
+ "binary": {"name": "objectcrawl"},
+ "branding": {"symbol_name": "tray", "accent_color": "#123456"},
+ "paths": {"default_config": "~/.objectcrawl/config.toml"},
+ "commands": {
+ "status": {"title": "Status", "argv": ["objectcrawl", "status", "--json"], "json": true},
+ "sync": {"title": "Sync", "argv": ["objectcrawl", "sync", "--json"], "json": true, "mutates": true},
+ "query": {"title": "Query", "argv": ["objectcrawl", "--json", "sql", "select count(*) from things"], "json": true},
+ "tap": {"title": "Desktop", "argv": ["objectcrawl", "tap", "--json"], "json": true, "mutates": true}
+ },
+ "capabilities": ["metadata", "status", "sync", "tap", "git-share"],
+ "privacy": {"exports_secrets": false}
+ }
+ """.utf8).write(to: directory.appendingPathComponent("objectcrawl.json"))
+ try Data("""
+ {
+ "id": "cachecrawl",
+ "display_name": "Cache Crawl",
+ "description": "A desktop cache test manifest",
+ "binary": {"name": "cachecrawl"},
+ "branding": {"symbol_name": "tray", "accent_color": "#123456"},
+ "paths": {"default_config": "~/.cachecrawl/config.toml"},
+ "commands": {
+ "desktop-cache-import": ["sync", "--source", "desktop-cache"]
+ }
+ }
+ """.utf8).write(to: directory.appendingPathComponent("cachecrawl.json"))
+ try Data("{ bad json".utf8).write(to: directory.appendingPathComponent("broken.json"))
+
+ let config = CrawlBarConfig(manifestDirectories: [directory.path])
+ let catalog = CrawlManifestCatalog()
+ let manifests = catalog.manifests(config: config)
+ let diagnostics = catalog.diagnostics(config: config)
+ let configURL = directory.appendingPathComponent("config.json")
+ let store = CrawlBarConfigStore(fileURL: configURL)
+ try store.save(config)
+ let registry = CrawlAppRegistry(configStore: store, catalog: catalog)
+ let installations = try registry.installations(includeDisabled: true)
+ try Self.expect(manifests.contains { $0.id == manifest.id }, "external manifests load from disk")
+ guard let objectManifest = manifests.first(where: { $0.id.rawValue == "objectcrawl" }) else {
+ throw SelfTestError.failed("crawlkit command-object manifests load from disk")
+ }
+ try Self.expect(objectManifest.commands["status"] == ["status", "--json"], "crawlkit command argv strips binary")
+ try Self.expect(objectManifest.commands["query"] == ["--json", "sql"], "crawlkit query sample SQL is stripped")
+ try Self.expect(objectManifest.commands["refresh"] == ["sync", "--json"], "crawlkit sync command aliases to refresh")
+ try Self.expect(objectManifest.commands["desktop-cache-import"] == ["tap", "--json"], "crawlkit tap command aliases to desktop cache import")
+ try Self.expect(objectManifest.capabilities.contains(.refresh), "crawlkit sync capability maps to refresh")
+ try Self.expect(objectManifest.capabilities.contains(.search), "crawlkit SQL/query capability maps to search")
+ try Self.expect(objectManifest.capabilities.contains(.desktopCache), "crawlkit tap capability maps to desktop cache")
+ try Self.expect(objectManifest.capabilities.contains(.publish), "crawlkit git-share capability maps to publish")
+ guard let cacheManifest = manifests.first(where: { $0.id.rawValue == "cachecrawl" }) else {
+ throw SelfTestError.failed("desktop cache manifest loads from disk")
+ }
+ try Self.expect(cacheManifest.capabilities.contains(.desktopCache), "desktop-cache-import command maps to desktop cache")
+ try Self.expect(diagnostics.contains { $0.path.hasSuffix("broken.json") }, "external manifest parse errors are reported")
+ try Self.expect(installations.contains { $0.id == manifest.id }, "external manifests appear as installations")
+ try Self.expect(BuiltInCrawlApps.gitcrawl.configOptions.contains { $0.id == "embedding_model" }, "built-in config options exist")
+ try Self.expect(!BuiltInCrawlApps.gitcrawl.needsSecretsForStatus, "built-in status avoids launch keychain reads")
+ let secretStatusManifest = CrawlAppManifest(
+ id: CrawlAppID(rawValue: "secretstatus"),
+ displayName: "Secret Status",
+ description: "Status requires an env secret",
+ binary: .init(name: "secretstatus"),
+ branding: .init(symbolName: "lock", accentColor: "#123456"),
+ paths: .init(),
+ commands: ["status": ["status", "--json"]],
+ capabilities: [.status],
+ configOptions: [.init(id: "token", label: "Token", kind: .secret, envVar: "SECRET_STATUS_TOKEN")])
+ try Self.expect(secretStatusManifest.needsSecretsForStatus, "external secret status can opt into keychain reads")
+ try Self.expect(BuiltInCrawlApps.slacrawl.privacy.containsPrivateMessages, "Slack privacy metadata flags local messages")
+ try Self.expect(BuiltInCrawlApps.notcrawl.privacy.localOnlyScopes.contains("workspace pages"), "Notion privacy metadata flags workspace pages")
+ try Self.expect(BuiltInCrawlApps.slacrawl.install?.package == "vincentkoc/tap/slacrawl", "built-in install metadata exists")
+ try Self.expect(BuiltInCrawlApps.gogcli.availability == .available, "Google manifest is available")
+ try Self.expect(BuiltInCrawlApps.gogcli.binary.name == "gog", "Google manifest uses the installed gog binary")
+ try Self.expect(BuiltInCrawlApps.gogcli.commands["status"] == ["auth", "list", "--check", "--json", "--no-input"], "Google status is wired")
+ try Self.expect(BuiltInCrawlApps.wacli.availability == .available, "WhatsApp manifest is available")
+ try Self.expect(BuiltInCrawlApps.wacli.commands["status"] == ["--account", "{config:account}", "--read-only", "--json", "doctor"], "WhatsApp status is wired")
+ try Self.expect(BuiltInCrawlApps.birdclaw.binary.name == "bird", "X app id uses bird executable")
+ try Self.expect(BuiltInCrawlApps.telecrawl.availability == .available, "telecrawl is available")
+ try Self.expect(BuiltInCrawlApps.telecrawl.commands["status"] == ["--json", "status"], "telecrawl uses JSON status command")
+ try Self.expect(BuiltInCrawlApps.telecrawl.commands["refresh"] == ["--json", "import"], "telecrawl imports through refresh")
+ try Self.expect(BuiltInCrawlApps.telecrawl.capabilities.contains(.search), "telecrawl advertises search")
+ try Self.expect(BuiltInCrawlApps.telecrawl.privacy.containsPrivateMessages, "telecrawl privacy metadata flags Telegram messages")
+ try Self.expect(BuiltInCrawlApps.telecrawl.install?.package == "steipete/tap/telecrawl", "telecrawl install metadata exists")
+ try Self.expect(BuiltInCrawlApps.telecrawl.paths.defaultConfig == "~/.telecrawl/backup.json", "telecrawl config path maps")
+ try Self.expect(BuiltInCrawlApps.graincrawl.availability == .available, "graincrawl is available")
+ try Self.expect(BuiltInCrawlApps.graincrawl.commands["status"] == ["status", "--json"], "graincrawl uses crawlkit status command")
+ try Self.expect(
+ BuiltInCrawlApps.graincrawl.commands["refresh"] == ["sync", "--json"],
+ "graincrawl refresh honors configured source")
+ try Self.expect(
+ BuiltInCrawlApps.graincrawl.commands["desktop-cache-import"] == ["sync", "--source", "desktop-cache", "--json"],
+ "graincrawl exposes explicit desktop cache import")
+ try Self.expect(BuiltInCrawlApps.graincrawl.capabilities.contains(.desktopCache), "graincrawl advertises desktop cache capability")
+ try Self.expect(
+ BuiltInCrawlApps.graincrawl.commands["query"] == ["--json", "sql"],
+ "graincrawl query emits JSON by default")
+ try Self.expect(BuiltInCrawlApps.graincrawl.commands["unlock"] == ["unlock", "--json"], "graincrawl exposes unlock action")
+ try Self.expect(BuiltInCrawlApps.graincrawl.branding.bundleIdentifier == "com.granola.app", "graincrawl uses native Granola icon")
+ try Self.expect(BuiltInCrawlApps.gitcrawl.commands["status"] == ["status", "--json"], "gitcrawl uses fast status command")
+ try Self.expect(BuiltInCrawlApps.gitcrawl.commands["refresh"] == ["sync", "--json"], "gitcrawl keeps refresh action wired")
+ try Self.expect(BuiltInCrawlApps.gitcrawl.commands["remote-status"] == ["remote", "status", "--json"], "gitcrawl exposes remote status")
+ try Self.expect(BuiltInCrawlApps.gitcrawl.commands["cloud-publish"] == ["cloud", "publish", "--json"], "gitcrawl exposes cloud publish")
+ try Self.expect(BuiltInCrawlApps.gitcrawl.capabilities.contains(.remoteArchive), "gitcrawl advertises remote archive")
+ try Self.expect(BuiltInCrawlApps.gitcrawl.capabilities.contains(.cloudPublish), "gitcrawl advertises cloud publish")
+ try Self.expect(BuiltInCrawlApps.slacrawl.commands["query"] == ["sql"], "Slack exposes query action")
+ try Self.expect(BuiltInCrawlApps.slacrawl.commands["search"] == ["--json", "search"], "Slack exposes text search action")
+ try Self.expect(BuiltInCrawlApps.discrawl.commands["query"] == nil, "Discord does not advertise stale SQL action")
+ try Self.expect(!BuiltInCrawlApps.discrawl.capabilities.contains(.search), "Discord search capability waits for upstream metadata")
+ try Self.expect(BuiltInCrawlApps.discrawl.commands["remote-status"] == ["remote", "status", "--json"], "Discord exposes remote status")
+ try Self.expect(BuiltInCrawlApps.discrawl.commands["cloud-publish"] == ["cloud", "publish", "--sqlite-only", "--json"], "Discord cloud publish defaults to sqlite-only")
+ try Self.expect(BuiltInCrawlApps.discrawl.capabilities.contains(.remoteArchive), "Discord advertises remote archive")
+ try Self.expect(BuiltInCrawlApps.discrawl.capabilities.contains(.cloudPublish), "Discord advertises cloud publish")
+ try Self.expect(BuiltInCrawlApps.notcrawl.capabilities.contains(.desktopCache), "Notion advertises desktop cache capability")
+ try Self.expect(BuiltInCrawlApps.gitcrawl.configSections.contains { $0.id == "github" }, "built-in config sections exist")
+ }
+
+ static func testNativeConfigRoundTrips() throws {
+ let directory = FileManager.default.temporaryDirectory
+ .appendingPathComponent("crawlbar-native-config-\(UUID().uuidString)", isDirectory: true)
+ try FileManager.default.createDirectory(at: directory, withIntermediateDirectories: true)
+ defer { try? FileManager.default.removeItem(at: directory) }
+
+ let configURL = directory.appendingPathComponent("config.toml")
+ try Data("""
+ [openai]
+ api_key = "from-file"
+ """.utf8).write(to: configURL)
+
+ let manifest = CrawlAppManifest(
+ id: CrawlAppID(rawValue: "tomlcrawl"),
+ displayName: "TOML Crawl",
+ description: "A TOML test crawler",
+ binary: .init(name: "tomlcrawl"),
+ branding: .init(symbolName: "terminal", accentColor: "#123456"),
+ paths: .init(defaultConfig: configURL.path),
+ commands: [:],
+ capabilities: [],
+ configOptions: [
+ .init(id: "openai_api_key", label: "OpenAI API key", kind: .secret, configKey: "openai.api_key"),
+ .init(id: "embedding_model", label: "Embedding model", kind: .choice, configKey: "embeddings.model"),
+ .init(id: "sync_limit", label: "Sync limit", kind: .number, configKey: "sync.default_limit"),
+ ])
+ var appConfig = CrawlBarAppConfig(id: manifest.id)
+ let nativeStore = CrawlNativeConfigStore()
+ try Self.expect(
+ nativeStore.resolvedConfigValues(appConfig: appConfig, manifest: manifest)["openai_api_key"] == "from-file",
+ "native TOML config values load")
+ try Self.expect(
+ nativeStore.resolvedConfigValues(appConfig: appConfig, manifest: manifest, includeSecrets: false)["openai_api_key"] == nil,
+ "native TOML secrets stay out of non-secret loads")
+
+ appConfig.configValues = nativeStore.resolvedConfigValues(appConfig: appConfig, manifest: manifest)
+ appConfig.configValues["embedding_model"] = "text-embedding-3-large"
+ appConfig.configValues["sync_limit"] = "100"
+ try nativeStore.write(appConfig: appConfig, manifest: manifest)
+ let content = try String(contentsOf: configURL, encoding: .utf8)
+ try Self.expect(content.contains("api_key = \"from-file\""), "native TOML values preserve existing keys")
+ try Self.expect(content.contains("[embeddings]"), "native TOML section writes")
+ try Self.expect(content.contains("model = \"text-embedding-3-large\""), "native TOML value writes")
+ try Self.expect(content.contains("default_limit = 100"), "native TOML number writes without quotes")
+
+ appConfig.configValues.removeValue(forKey: "openai_api_key")
+ try nativeStore.write(appConfig: appConfig, manifest: manifest)
+ let clearedContent = try String(contentsOf: configURL, encoding: .utf8)
+ try Self.expect(clearedContent.contains("api_key = \"from-file\""), "native TOML secret keys preserve when omitted")
+ try nativeStore.write(appConfig: appConfig, manifest: manifest, clearMissingSecretIDs: ["openai_api_key"])
+ let explicitlyClearedContent = try String(contentsOf: configURL, encoding: .utf8)
+ try Self.expect(!explicitlyClearedContent.contains("api_key ="), "native TOML secret keys clear when explicit")
+ }
+
+ static func testStatusSecretsLoadFromNativeConfig() throws {
+ let directory = FileManager.default.temporaryDirectory
+ .appendingPathComponent("crawlbar-status-secret-\(UUID().uuidString)", isDirectory: true)
+ try FileManager.default.createDirectory(at: directory, withIntermediateDirectories: true)
+ defer { try? FileManager.default.removeItem(at: directory) }
+
+ let scriptURL = directory.appendingPathComponent("secret-status.sh")
+ try Data("""
+ #!/bin/sh
+ printf '{"state":"ok"}'
+ """.utf8).write(to: scriptURL)
+ try FileManager.default.setAttributes([.posixPermissions: 0o755], ofItemAtPath: scriptURL.path)
+
+ let nativeConfigURL = directory.appendingPathComponent("status.toml")
+ try Data("""
+ [auth]
+ token = "from-native"
+ """.utf8).write(to: nativeConfigURL)
+
+ let manifest = CrawlAppManifest(
+ id: CrawlAppID(rawValue: "statussecret"),
+ displayName: "Status Secret",
+ description: "A status secret test crawler",
+ binary: .init(name: scriptURL.path),
+ branding: .init(symbolName: "lock", accentColor: "#123456"),
+ paths: .init(defaultConfig: nativeConfigURL.path),
+ commands: ["status": ["status", "--json"]],
+ capabilities: [.status],
+ statusRequiresSecrets: true,
+ configOptions: [
+ .init(id: "token", label: "Token", kind: .secret, envVar: "STATUS_SECRET_TOKEN", configKey: "auth.token"),
+ ])
+ try CrawlCoding.makeJSONEncoder().encode(manifest)
+ .write(to: directory.appendingPathComponent("statussecret.json"))
+ let configURL = directory.appendingPathComponent("config.json")
+ let store = CrawlBarConfigStore(fileURL: configURL)
+ try store.save(CrawlBarConfig(manifestDirectories: [directory.path]))
+ let registry = CrawlAppRegistry(configStore: store)
+
+ guard let plain = try registry.installation(for: manifest.id, includeSecrets: false),
+ let status = try registry.installationForStatus(for: manifest.id)
+ else {
+ throw SelfTestError.failed("status secret crawler loads")
+ }
+ try Self.expect(plain.configValues["token"] == nil, "plain installation omits native secret")
+ try Self.expect(status.configValues["token"] == "from-native", "status installation rehydrates native secret")
+ }
+}
diff --git a/Sources/CrawlBarSelfTest/SelfTestQueryAndGitcrawl.swift b/Sources/CrawlBarSelfTest/SelfTestQueryAndGitcrawl.swift
new file mode 100644
index 0000000..40e872b
--- /dev/null
+++ b/Sources/CrawlBarSelfTest/SelfTestQueryAndGitcrawl.swift
@@ -0,0 +1,228 @@
+import CrawlBarCore
+import Foundation
+
+extension CrawlBarSelfTest {
+ static func testQueryActionResolverSkipsSQLForPlainText() throws {
+ let sqlOnlyManifest = CrawlAppManifest(
+ id: CrawlAppID(rawValue: "sqlonly"),
+ displayName: "SQL Only",
+ description: "A SQL-only crawler",
+ binary: .init(name: "sqlonly"),
+ branding: .init(symbolName: "terminal", accentColor: "#123456"),
+ paths: .init(),
+ commands: ["sql": ["sql"]],
+ capabilities: [.search])
+ try Self.expect(
+ CrawlQueryActionResolver.action(for: sqlOnlyManifest, queryArguments: ["select count(*) from rows"]) == "sql",
+ "SQL-looking queries can use SQL-only crawler commands")
+ try Self.expect(
+ CrawlQueryActionResolver.action(for: sqlOnlyManifest, queryArguments: ["manifest"]) == nil,
+ "plain text queries do not fall through to SQL-only crawler commands")
+
+ let textManifest = CrawlAppManifest(
+ id: CrawlAppID(rawValue: "textcrawl"),
+ displayName: "Text Crawl",
+ description: "A text search crawler",
+ binary: .init(name: "textcrawl"),
+ branding: .init(symbolName: "terminal", accentColor: "#123456"),
+ paths: .init(),
+ commands: ["search": ["search"], "query": ["query"]],
+ capabilities: [.search])
+ try Self.expect(
+ CrawlQueryActionResolver.action(for: textManifest, queryArguments: ["manifest"]) == "search",
+ "plain text queries prefer explicit search commands")
+ }
+
+ static func testWacliSearchJoinsQueryArguments() throws {
+ let directory = FileManager.default.temporaryDirectory
+ .appendingPathComponent("crawlbar-wacli-search-\(UUID().uuidString)", isDirectory: true)
+ try FileManager.default.createDirectory(at: directory, withIntermediateDirectories: true)
+ defer { try? FileManager.default.removeItem(at: directory) }
+
+ let scriptURL = directory.appendingPathComponent("wacli")
+ try Data("""
+ #!/bin/sh
+ printf '%s\\n' "$#"
+ for arg in "$@"; do printf '<%s>\\n' "$arg"; done
+ """.utf8).write(to: scriptURL)
+ try FileManager.default.setAttributes([.posixPermissions: 0o755], ofItemAtPath: scriptURL.path)
+
+ let manifest = CrawlAppManifest(
+ id: BuiltInCrawlApps.wacliID,
+ displayName: "WhatsApp",
+ description: "A WhatsApp crawler",
+ binary: .init(name: scriptURL.path),
+ branding: .init(symbolName: "message", accentColor: "#25D366"),
+ paths: .init(),
+ commands: ["search": ["messages", "search"]],
+ capabilities: [.search])
+ let installation = CrawlAppInstallation(manifest: manifest, binaryPath: scriptURL.path)
+ let result = try CrawlCommandRunner()
+ .run(installation: installation, action: "search", extraArguments: ["hello", "world"], timeoutSeconds: 5)
+
+ try Self.expect(
+ result.stdout == "3\n\n\n\n",
+ "wacli search receives multi-word query as one argument")
+
+ let flaggedResult = try CrawlCommandRunner()
+ .run(
+ installation: installation,
+ action: "search",
+ extraArguments: ["hello", "world", "--limit", "5"],
+ timeoutSeconds: 5)
+ try Self.expect(
+ flaggedResult.stdout == "5\n\n\n\n<--limit>\n<5>\n",
+ "wacli search preserves flags after joined query")
+
+ let builtInDefault = CrawlAppInstallation(manifest: BuiltInCrawlApps.wacli, binaryPath: scriptURL.path)
+ let defaultAccountResult = try CrawlCommandRunner()
+ .run(installation: builtInDefault, action: "search", extraArguments: ["hello", "world"], timeoutSeconds: 5)
+ try Self.expect(
+ defaultAccountResult.stdout == "5\n<--read-only>\n<--json>\n\n\n\n",
+ "built-in wacli omits account flag until configured")
+
+ let builtInNamed = CrawlAppInstallation(
+ manifest: BuiltInCrawlApps.wacli,
+ binaryPath: scriptURL.path,
+ configValues: ["account": "personal"])
+ let namedAccountResult = try CrawlCommandRunner()
+ .run(installation: builtInNamed, action: "search", extraArguments: ["hello", "world"], timeoutSeconds: 5)
+ try Self.expect(
+ namedAccountResult.stdout == "7\n<--account>\n\n<--read-only>\n<--json>\n\n\n\n",
+ "built-in wacli applies configured account")
+
+ let literalConfigQuery = try CrawlCommandRunner()
+ .run(
+ installation: builtInNamed,
+ action: "search",
+ extraArguments: ["{config:account}"],
+ timeoutSeconds: 5)
+ try Self.expect(
+ literalConfigQuery.stdout == "7\n<--account>\n\n<--read-only>\n<--json>\n\n\n<{config:account}>\n",
+ "user query text is not config-interpolated")
+ }
+
+ static func testGitcrawlCommandArgumentsInferRepository() throws {
+ let directory = FileManager.default.temporaryDirectory
+ .appendingPathComponent("crawlbar-gitcrawl-\(UUID().uuidString)", isDirectory: true)
+ try FileManager.default.createDirectory(at: directory, withIntermediateDirectories: true)
+ defer { try? FileManager.default.removeItem(at: directory) }
+
+ let scriptURL = directory.appendingPathComponent("print-args.sh")
+ try Data("""
+ #!/bin/sh
+ printf '%s' "$*"
+ """.utf8).write(to: scriptURL)
+ try FileManager.default.setAttributes([.posixPermissions: 0o755], ofItemAtPath: scriptURL.path)
+
+ let configURL = directory.appendingPathComponent("gitcrawl.toml")
+ let databaseURL = directory.appendingPathComponent("openclaw__openclaw.sync.db")
+ try Data("db_path = \"\(databaseURL.path)\"\n".utf8).write(to: configURL)
+
+ let manifest = CrawlAppManifest(
+ id: BuiltInCrawlApps.gitcrawlID,
+ displayName: "Git Crawl",
+ description: "A git crawler",
+ binary: .init(name: scriptURL.path),
+ branding: .init(symbolName: "tray", accentColor: "#123456"),
+ paths: .init(defaultConfig: configURL.path),
+ commands: ["refresh": ["sync", "--json"], "query": ["search", "--json"]],
+ capabilities: [.refresh, .search])
+ let installation = CrawlAppInstallation(manifest: manifest, binaryPath: scriptURL.path)
+
+ let refreshResult = try CrawlCommandRunner().run(installation: installation, action: "refresh", timeoutSeconds: 5)
+ try Self.expect(refreshResult.stdout == "sync --json openclaw/openclaw", "gitcrawl refresh infers repository")
+
+ let queryResult = try CrawlCommandRunner().run(
+ installation: installation,
+ action: "query",
+ extraArguments: ["stale", "branches"],
+ timeoutSeconds: 5)
+ try Self.expect(
+ queryResult.stdout == "search --json openclaw/openclaw --query stale branches",
+ "gitcrawl query infers repository and joins query text")
+
+ let storeURL = directory.appendingPathComponent("stores/generic-store", isDirectory: true)
+ let genericConfigURL = directory.appendingPathComponent("gitcrawl-generic.toml")
+ let genericDatabaseURL = storeURL.appendingPathComponent("data/gitcrawl.db")
+ let reportURL = storeURL.appendingPathComponent("reports/latest-status.json")
+ try FileManager.default.createDirectory(
+ at: genericDatabaseURL.deletingLastPathComponent(),
+ withIntermediateDirectories: true)
+ try FileManager.default.createDirectory(
+ at: reportURL.deletingLastPathComponent(),
+ withIntermediateDirectories: true)
+ try Data("db_path = \"\(genericDatabaseURL.path)\"\n".utf8).write(to: genericConfigURL)
+ try Data(#"{"repository":{"owner":"openclaw","name":"openclaw"}}"#.utf8).write(to: reportURL)
+ let genericManifest = CrawlAppManifest(
+ id: BuiltInCrawlApps.gitcrawlID,
+ displayName: "Git Crawl",
+ description: "A git crawler",
+ binary: .init(name: scriptURL.path),
+ branding: .init(symbolName: "tray", accentColor: "#123456"),
+ paths: .init(defaultConfig: genericConfigURL.path),
+ commands: ["query": ["search", "--json"]],
+ capabilities: [.search])
+ let genericInstallation = CrawlAppInstallation(manifest: genericManifest, binaryPath: scriptURL.path)
+ let genericQueryResult = try CrawlCommandRunner().run(
+ installation: genericInstallation,
+ action: "query",
+ extraArguments: ["manifest"],
+ timeoutSeconds: 5)
+ try Self.expect(
+ genericQueryResult.stdout == "search --json openclaw/openclaw --query manifest",
+ "gitcrawl query infers repository from latest report when db filename is generic")
+ let genericStatus = GitcrawlStatusSnapshot.status(for: genericInstallation)
+ try Self.expect(
+ genericStatus?.databasePath == genericDatabaseURL.path,
+ "gitcrawl status keeps the database path that supplied its adjacent report")
+ let nativeSearchManifest = CrawlAppManifest(
+ id: BuiltInCrawlApps.gitcrawlID,
+ displayName: "Git Crawl",
+ description: "A metadata-derived git crawler",
+ binary: .init(name: scriptURL.path),
+ branding: .init(symbolName: "tray", accentColor: "#123456"),
+ paths: .init(defaultConfig: genericConfigURL.path),
+ commands: ["search": ["search", "--json"]],
+ capabilities: [.search])
+ let nativeSearchInstallation = CrawlAppInstallation(manifest: nativeSearchManifest, binaryPath: scriptURL.path)
+ let nativeSearchResult = try CrawlCommandRunner().run(
+ installation: nativeSearchInstallation,
+ action: "search",
+ extraArguments: ["manifest"],
+ timeoutSeconds: 5)
+ try Self.expect(
+ nativeSearchResult.stdout == "search --json openclaw/openclaw --query manifest",
+ "metadata-derived gitcrawl search infers repository and query flag")
+
+ let missingReportConfigURL = directory.appendingPathComponent("gitcrawl-missing-report.toml")
+ let missingReportDatabaseURL = directory.appendingPathComponent("other-store/data/gitcrawl.db")
+ try FileManager.default.createDirectory(
+ at: missingReportDatabaseURL.deletingLastPathComponent(),
+ withIntermediateDirectories: true)
+ try Data("db_path = \"\(missingReportDatabaseURL.path)\"\n".utf8).write(to: missingReportConfigURL)
+ let missingReportManifest = CrawlAppManifest(
+ id: BuiltInCrawlApps.gitcrawlID,
+ displayName: "Git Crawl",
+ description: "A git crawler",
+ binary: .init(name: scriptURL.path),
+ branding: .init(symbolName: "tray", accentColor: "#123456"),
+ paths: .init(
+ defaultConfig: missingReportConfigURL.path,
+ defaultDatabase: directory.appendingPathComponent("wrong__repo.sync.db").path),
+ commands: ["query": ["search", "--json"]],
+ capabilities: [.search])
+ let missingReportInstallation = CrawlAppInstallation(manifest: missingReportManifest, binaryPath: scriptURL.path)
+ let missingReportQueryResult = try CrawlCommandRunner().run(
+ installation: missingReportInstallation,
+ action: "query",
+ extraArguments: ["manifest"],
+ timeoutSeconds: 5)
+ try Self.expect(
+ missingReportQueryResult.stdout == "search --json manifest",
+ "gitcrawl query does not infer repository from unrelated global reports")
+ try Self.expect(
+ GitcrawlStatusSnapshot.status(for: missingReportInstallation) == nil,
+ "gitcrawl status does not read unrelated global reports for explicit database configs")
+ }
+}
diff --git a/Sources/CrawlBarSelfTest/SelfTestServices.swift b/Sources/CrawlBarSelfTest/SelfTestServices.swift
new file mode 100644
index 0000000..5dd0087
--- /dev/null
+++ b/Sources/CrawlBarSelfTest/SelfTestServices.swift
@@ -0,0 +1,153 @@
+import CrawlBarCore
+import Foundation
+
+extension CrawlBarSelfTest {
+ static func testGogStatusServiceVerifiesOAuthOrServiceAccount() throws {
+ let directory = FileManager.default.temporaryDirectory
+ .appendingPathComponent("crawlbar-gog-status-\(UUID().uuidString)", isDirectory: true)
+ try FileManager.default.createDirectory(at: directory, withIntermediateDirectories: true)
+ defer { try? FileManager.default.removeItem(at: directory) }
+
+ let scriptURL = directory.appendingPathComponent("gog")
+ try Data("""
+ #!/bin/sh
+ if [ "$2" = "list" ]; then
+ printf '%s' '{"accounts":[{"email":"user@example.com","auth":"oauth","valid":true}]}'
+ else
+ printf '%s' '{"status":"ok","checks":[{"name":"tokens","status":"ok","detail":"1 readable OAuth token"}]}'
+ fi
+ """.utf8).write(to: scriptURL)
+ try FileManager.default.setAttributes([.posixPermissions: 0o755], ofItemAtPath: scriptURL.path)
+
+ let manifest = CrawlAppManifest(
+ id: BuiltInCrawlApps.gogcliID,
+ displayName: "Google",
+ description: "A Google crawler",
+ binary: .init(name: scriptURL.path),
+ branding: .init(symbolName: "g.circle", accentColor: "#4285F4"),
+ paths: .init(),
+ commands: [
+ "status": ["auth", "list", "--check", "--json", "--no-input"],
+ "doctor": ["auth", "doctor", "--check", "--json", "--no-input"],
+ ],
+ capabilities: [.status, .doctor])
+ let installation = CrawlAppInstallation(manifest: manifest, binaryPath: scriptURL.path)
+ let status = CrawlStatusService().status(for: installation, timeoutSeconds: 5)
+ try Self.expect(status.state == .current, "gog status service verifies OAuth with doctor")
+
+ let serviceScriptURL = directory.appendingPathComponent("gog-service")
+ try Data("""
+ #!/bin/sh
+ if [ "$2" = "list" ]; then
+ printf '%s' '{"accounts":[{"email":"admin@example.com","auth":"service_account","valid":true,"error":"service account (not checked)"}]}'
+ else
+ printf '%s' '{"status":"warn","checks":[{"name":"tokens","status":"warn","detail":"no OAuth tokens"}]}'
+ fi
+ """.utf8).write(to: serviceScriptURL)
+ try FileManager.default.setAttributes([.posixPermissions: 0o755], ofItemAtPath: serviceScriptURL.path)
+
+ let serviceManifest = CrawlAppManifest(
+ id: BuiltInCrawlApps.gogcliID,
+ displayName: "Google",
+ description: "A Google crawler",
+ binary: .init(name: serviceScriptURL.path),
+ branding: .init(symbolName: "g.circle", accentColor: "#4285F4"),
+ paths: .init(),
+ commands: manifest.commands,
+ capabilities: [.status, .doctor])
+ let serviceInstallation = CrawlAppInstallation(manifest: serviceManifest, binaryPath: serviceScriptURL.path)
+ let serviceStatus = CrawlStatusService().status(for: serviceInstallation, timeoutSeconds: 5)
+ try Self.expect(serviceStatus.state == .current, "gog status service accepts service-account-only auth")
+ }
+
+ static func testActionFailuresPreserveStatusMetadata() throws {
+ let lastSyncAt = Date(timeIntervalSince1970: 1_775_000_000)
+ let metadata = CrawlAppStatus(
+ appID: BuiltInCrawlApps.discrawlID,
+ state: .current,
+ summary: "5052 messages",
+ configPath: "/tmp/discrawl.toml",
+ databasePath: "/tmp/discrawl.db",
+ databaseBytes: 36_397_056,
+ lastSyncAt: lastSyncAt,
+ counts: [CrawlCount(id: "messages", label: "Messages", value: 5052)],
+ databases: [
+ CrawlDatabaseResource(
+ id: "primary",
+ label: "Discord archive",
+ kind: .sqlite,
+ path: "/tmp/discrawl.db",
+ isPrimary: true,
+ bytes: 36_397_056),
+ ],
+ freshness: CrawlFreshness(status: .current, ageSeconds: 12, staleAfterSeconds: 86_400),
+ share: CrawlShareStatus(enabled: true, repoPath: "/tmp/share", remote: "origin", branch: "main"),
+ warnings: ["old warning"])
+ let failure = CrawlAppStatus(
+ appID: BuiltInCrawlApps.discrawlID,
+ state: .error,
+ summary: "refresh: network timed out",
+ errors: ["network timed out"])
+
+ let merged = metadata.mergingActionFailure(failure)
+ try Self.expect(merged.state == .error, "action failure state is visible")
+ try Self.expect(merged.summary == "refresh: network timed out", "action failure summary is visible")
+ try Self.expect(merged.databasePath == metadata.databasePath, "action failure preserves database path")
+ try Self.expect(merged.databases == metadata.databases, "action failure preserves databases")
+ try Self.expect(merged.counts == metadata.counts, "action failure preserves counts")
+ try Self.expect(merged.lastSyncAt == lastSyncAt, "action failure preserves last sync")
+ try Self.expect(merged.share == metadata.share, "action failure preserves share metadata")
+ try Self.expect(merged.warnings == ["old warning"], "action failure preserves existing warnings")
+ try Self.expect(merged.errors == ["network timed out"], "action failure keeps failure errors")
+
+ let githubAuthMessage = """
+ [github] request GET /repos/openclaw/openclaw
+ github GET /repos/openclaw/openclaw failed with status 401: {
+ "message": "Bad credentials"
+ }
+ """
+ let githubAuthFailure = CrawlAppStatus.commandFailure(
+ appID: BuiltInCrawlApps.gitcrawlID,
+ action: "refresh",
+ message: githubAuthMessage,
+ fallback: "refresh failed")
+ try Self.expect(githubAuthFailure.state == .needsAuth, "gitcrawl auth failure action is recoverable")
+ try Self.expect(
+ githubAuthFailure.summary == "refresh: GitHub credentials rejected",
+ "gitcrawl auth failure action summary is useful")
+
+ let githubMetadata = CrawlAppStatus(
+ appID: BuiltInCrawlApps.gitcrawlID,
+ state: .current,
+ summary: "1 repo",
+ databasePath: "/tmp/gitcrawl.db",
+ counts: [CrawlCount(id: "repositories", label: "Repositories", value: 1)])
+ let mergedGithub = githubMetadata.mergingActionFailure(githubAuthFailure)
+ try Self.expect(mergedGithub.state == .needsAuth, "action failure merge preserves auth state")
+ try Self.expect(mergedGithub.summary == "refresh: GitHub credentials rejected", "action failure merge preserves auth summary")
+ try Self.expect(mergedGithub.databasePath == githubMetadata.databasePath, "action failure merge preserves github metadata")
+
+ let emptyRefreshed = CrawlAppStatus(
+ appID: BuiltInCrawlApps.discrawlID,
+ state: .error,
+ summary: "status failed")
+ let richest = CrawlAppStatus.richestMetadataStatus(emptyRefreshed, fallback: metadata)
+ try Self.expect(richest == metadata, "previous rich metadata wins over empty refreshed status")
+
+ let recoverableGraincrawlStatus = CrawlAppStatus(
+ appID: BuiltInCrawlApps.graincrawlID,
+ state: .error,
+ summary: "private-api reports expired token, desktop-cache reports unsupported cache version 8")
+ try Self.expect(
+ recoverableGraincrawlStatus.isRecoverableGraincrawlSourceFailure,
+ "graincrawl source status failures can render as stale")
+ let graincrawlActionFailure = recoverableGraincrawlStatus.mergingActionFailure(CrawlAppStatus(
+ appID: BuiltInCrawlApps.graincrawlID,
+ state: .error,
+ summary: "refresh: Granola access token expired",
+ errors: ["Granola access token expired"]))
+ try Self.expect(
+ !graincrawlActionFailure.isRecoverableGraincrawlSourceFailure,
+ "graincrawl action failures stay visible as errors")
+ }
+}
diff --git a/Sources/CrawlBarSelfTest/SelfTestStatusAuth.swift b/Sources/CrawlBarSelfTest/SelfTestStatusAuth.swift
new file mode 100644
index 0000000..2a6fab5
--- /dev/null
+++ b/Sources/CrawlBarSelfTest/SelfTestStatusAuth.swift
@@ -0,0 +1,140 @@
+import CrawlBarCore
+import Foundation
+
+extension CrawlBarSelfTest {
+ static func testStatusMapperNormalizesWacliDoctorOutput() throws {
+ let result = CrawlCommandResult(
+ appID: CrawlAppID(rawValue: "wacli-test"),
+ action: "status",
+ exitCode: 0,
+ stdout: """
+ {"success":true,"data":{"state":"current","store_dir":"/tmp/wacli-store","lock_held":true,"connection_state":"locked_by_other_process","authenticated":true,"fts_enabled":false,"store":{"messages":6991,"chats":677,"contacts":514,"groups":250,"last_sync_at":"2026-05-09T05:45:44Z"}},"error":null}
+ """,
+ stderr: "",
+ startedAt: Date(timeIntervalSince1970: 1_775_000_000),
+ finishedAt: Date(timeIntervalSince1970: 1_775_000_001))
+ let manifest = CrawlAppManifest(
+ id: result.appID,
+ displayName: "WhatsApp Test",
+ description: "Remote WhatsApp archive",
+ binary: .init(name: "ssh"),
+ branding: .init(symbolName: "message.circle", accentColor: "#25D366"),
+ paths: .init(),
+ commands: ["status": ["host", "wacli --account test --read-only doctor --json"]],
+ capabilities: [.status])
+ let status = CrawlStatusMapper().status(from: result, manifest: manifest, staleAfterSeconds: 900)
+
+ try Self.expect(status.state == .current, "wacli doctor honors explicit current state over stale timestamps")
+ try Self.expect(status.freshness?.status == .stale, "wacli doctor still exposes stale freshness metadata")
+ try Self.expect(status.summary == "6991 messages, 677 chats", "wacli doctor maps store counts")
+ try Self.expect(status.databasePath == "/tmp/wacli-store/wacli.db", "wacli doctor maps database path")
+ try Self.expect(status.lastSyncAt != nil, "wacli doctor maps last sync")
+ try Self.expect(status.warnings.contains("Store is locked by locked_by_other_process"), "wacli lock is a warning")
+ try Self.expect(status.warnings.contains("Full-text search is not enabled"), "wacli FTS state is a warning")
+ }
+
+ static func testStatusMapperNormalizesGogAuthStatus() throws {
+ let needsAuthResult = CrawlCommandResult(
+ appID: BuiltInCrawlApps.gogcliID,
+ action: "status",
+ exitCode: 0,
+ stdout: """
+ {"account":{"credentials_exists":false,"service_account_configured":false,"email":""},"config":{"exists":false,"path":"/tmp/gog/config.json"},"keyring":{"backend":"auto","source":"default"}}
+ """,
+ stderr: "",
+ startedAt: Date(),
+ finishedAt: Date())
+ let needsAuth = CrawlStatusMapper().status(from: needsAuthResult, manifest: BuiltInCrawlApps.gogcli)
+ try Self.expect(needsAuth.state == .needsAuth, "gog auth status without credentials maps to needs auth")
+ try Self.expect(needsAuth.summary == "Google account needs auth", "gog auth status has a useful setup summary")
+ try Self.expect(needsAuth.configPath == "/tmp/gog/config.json", "gog auth status maps config path")
+
+ let readyResult = CrawlCommandResult(
+ appID: BuiltInCrawlApps.gogcliID,
+ action: "status",
+ exitCode: 0,
+ stdout: """
+ {"account":{"credentials_exists":true,"service_account_configured":false,"email":"user@example.com"},"config":{"exists":true,"path":"/tmp/gog/config.json"}}
+ """,
+ stderr: "",
+ startedAt: Date(),
+ finishedAt: Date())
+ let ready = CrawlStatusMapper().status(from: readyResult, manifest: BuiltInCrawlApps.gogcli)
+ try Self.expect(ready.state == .needsAuth, "gog raw credentials still require verified token auth")
+ try Self.expect(ready.summary == "Google account needs auth", "gog raw credentials keep setup summary")
+
+ let doctorResult = CrawlCommandResult(
+ appID: BuiltInCrawlApps.gogcliID,
+ action: "status",
+ exitCode: 0,
+ stdout: """
+ {"checks":[{"name":"config.path","status":"warn","detail":"/tmp/gog/config.json (missing)"},{"name":"keyring.open","status":"ok","detail":"opened"},{"name":"tokens","status":"ok","detail":"4 readable OAuth tokens of 4 stored token accounts"},{"name":"refresh.default.user@example.com","status":"ok","detail":"refresh token exchange succeeded"}],"status":"warn"}
+ """,
+ stderr: "",
+ startedAt: Date(),
+ finishedAt: Date())
+ let doctor = CrawlStatusMapper().status(from: doctorResult, manifest: BuiltInCrawlApps.gogcli)
+ try Self.expect(doctor.state == .current, "gog doctor maps readable refreshable tokens to current")
+ try Self.expect(doctor.summary == "4 Google OAuth accounts readable", "gog doctor summarizes readable tokens")
+ try Self.expect(doctor.warnings.contains("config.path: /tmp/gog/config.json (missing)"), "gog doctor preserves non-auth warnings")
+ }
+
+ static func testStatusMapperNormalizesBirdclawAuthStatus() throws {
+ let result = CrawlCommandResult(
+ appID: BuiltInCrawlApps.birdclawID,
+ action: "status",
+ exitCode: 0,
+ stdout: """
+ {"installed":false,"availableTransport":"local","statusText":"xurl not installed. local mode active."}
+ """,
+ stderr: "",
+ startedAt: Date(),
+ finishedAt: Date())
+ let status = CrawlStatusMapper().status(from: result, manifest: BuiltInCrawlApps.birdclaw)
+ try Self.expect(status.state == .current, "birdclaw auth status keeps local mode usable")
+ try Self.expect(status.summary == "xurl not installed. local mode active.", "birdclaw auth status has a useful summary")
+ try Self.expect(status.warnings.contains("Transport: local"), "birdclaw auth status exposes transport")
+ try Self.expect(status.warnings.contains("xurl not installed. local mode active."), "birdclaw auth status preserves transport warning")
+
+ let birdResult = CrawlCommandResult(
+ appID: BuiltInCrawlApps.birdclawID,
+ action: "status",
+ exitCode: 0,
+ stdout: """
+ [info] Credential check
+ [ok] auth_token: abc...
+ [ok] ct0: def...
+ source: Chrome default profile
+ [warn] Warnings:
+ - No Twitter cookies found in Safari.
+ [ok] Ready to tweet!
+ """,
+ stderr: "",
+ startedAt: Date(),
+ finishedAt: Date())
+ let birdStatus = CrawlStatusMapper().status(from: birdResult, manifest: BuiltInCrawlApps.birdclaw)
+ try Self.expect(birdStatus.state == .current, "bird check text maps to current when cookies are usable")
+ try Self.expect(birdStatus.summary == "X cookies available via bird (Chrome default profile)", "bird check text exposes cookie source")
+ try Self.expect(birdStatus.warnings.contains("No Twitter cookies found in Safari."), "bird check warnings are preserved")
+ }
+
+ static func testStatusMapperTrustsCrawlerState() throws {
+ let result = CrawlCommandResult(
+ appID: BuiltInCrawlApps.discrawlID,
+ action: "status",
+ exitCode: 0,
+ stdout: """
+ {"schema_version":"crawlkit.control.v1","state":"current","summary":"ok","last_sync_at":"2026-05-09T05:45:44Z","counts":[{"id":"messages","label":"Messages","value":10}]}
+ """,
+ stderr: "",
+ startedAt: Date(timeIntervalSince1970: 1_775_000_000),
+ finishedAt: Date(timeIntervalSince1970: 1_775_000_001))
+ let status = CrawlStatusMapper().status(
+ from: result,
+ manifest: BuiltInCrawlApps.discrawl,
+ staleAfterSeconds: 900)
+
+ try Self.expect(status.state == .current, "explicit crawler state wins over stale timestamp heuristics")
+ try Self.expect(status.freshness?.status == .stale, "stale timestamp can still be shown as metadata")
+ }
+}
diff --git a/Sources/CrawlBarSelfTest/SelfTestStatusMapping.swift b/Sources/CrawlBarSelfTest/SelfTestStatusMapping.swift
new file mode 100644
index 0000000..b23060b
--- /dev/null
+++ b/Sources/CrawlBarSelfTest/SelfTestStatusMapping.swift
@@ -0,0 +1,350 @@
+import CrawlBarCore
+import Foundation
+
+extension CrawlBarSelfTest {
+ static func testStatusMapperNormalizesCounts() throws {
+ let output = """
+ {"message_count":42,"channel_count":3,"last_sync_at":"2026-05-01T12:00:00Z","db_path":"/tmp/discrawl.db"}
+ """
+ let result = CrawlCommandResult(
+ appID: BuiltInCrawlApps.discrawlID,
+ action: "status",
+ exitCode: 0,
+ stdout: output,
+ stderr: "",
+ startedAt: Date(),
+ finishedAt: Date())
+
+ let status = CrawlStatusMapper().status(from: result, manifest: BuiltInCrawlApps.discrawl)
+ try Self.expect(status.counts.contains(CrawlCount(id: "messages", label: "Messages", value: 42)), "discrawl messages map")
+ try Self.expect(status.lastSyncAt != nil, "whole-second last sync dates map")
+ try Self.expect(status.databasePath == "/tmp/discrawl.db", "database path maps")
+ try Self.expect(status.databases.first?.label == "Discord archive", "database inventory maps")
+ try Self.expect(status.databases.first?.counts.contains(CrawlCount(id: "messages", label: "Messages", value: 42)) == true, "database inventory carries counts")
+
+ let gogResult = CrawlCommandResult(
+ appID: BuiltInCrawlApps.gogcliID,
+ action: "status",
+ exitCode: 0,
+ stdout: #"{"account":{"credentials_exists":true},"config":{"exists":true,"path":"/tmp/gog/config.json"}}"#,
+ stderr: "",
+ startedAt: Date(),
+ finishedAt: Date())
+ let gogStatus = CrawlStatusMapper().status(from: gogResult, manifest: BuiltInCrawlApps.gogcli)
+ try Self.expect(gogStatus.state == .needsAuth, "gog raw status asks OAuth auth to be verified")
+ try Self.expect(gogStatus.configPath == "/tmp/gog/config.json", "gog config path maps")
+
+ let gogServiceAccountRawResult = CrawlCommandResult(
+ appID: BuiltInCrawlApps.gogcliID,
+ action: "status",
+ exitCode: 0,
+ stdout: #"{"account":{"service_account_configured":true},"config":{"exists":false}}"#,
+ stderr: "",
+ startedAt: Date(),
+ finishedAt: Date())
+ let gogServiceAccountRawStatus = CrawlStatusMapper().status(from: gogServiceAccountRawResult, manifest: BuiltInCrawlApps.gogcli)
+ try Self.expect(gogServiceAccountRawStatus.state == .current, "gog raw status maps service account auth")
+
+ let gogServiceAccountResult = CrawlCommandResult(
+ appID: BuiltInCrawlApps.gogcliID,
+ action: "status",
+ exitCode: 0,
+ stdout: #"{"status":"ok","checks":[{"name":"config.path","status":"ok","detail":"/tmp/gog/config.json"},{"name":"service_account","status":"ok"}]}"#,
+ stderr: "",
+ startedAt: Date(),
+ finishedAt: Date())
+ let gogServiceAccountStatus = CrawlStatusMapper().status(from: gogServiceAccountResult, manifest: BuiltInCrawlApps.gogcli)
+ try Self.expect(gogServiceAccountStatus.state == .current, "gog doctor maps configured auth")
+ try Self.expect(gogServiceAccountStatus.configPath == "/tmp/gog/config.json", "gog doctor config path maps")
+
+ let gogDoctorFailureResult = CrawlCommandResult(
+ appID: BuiltInCrawlApps.gogcliID,
+ action: "status",
+ exitCode: 0,
+ stdout: #"{"status":"error","checks":[{"name":"tokens","status":"error","detail":"no readable OAuth tokens"}]}"#,
+ stderr: "",
+ startedAt: Date(),
+ finishedAt: Date())
+ let gogDoctorFailureStatus = CrawlStatusMapper().status(from: gogDoctorFailureResult, manifest: BuiltInCrawlApps.gogcli)
+ try Self.expect(gogDoctorFailureStatus.state == .needsAuth, "gog doctor token failures map to auth setup")
+ try Self.expect(gogDoctorFailureStatus.summary == "no readable OAuth tokens", "gog doctor failure detail maps")
+
+ let gogDoctorConfigResult = CrawlCommandResult(
+ appID: BuiltInCrawlApps.gogcliID,
+ action: "status",
+ exitCode: 0,
+ stdout: #"{"status":"warn","checks":[{"name":"config.path","status":"warn","detail":"config missing"}]}"#,
+ stderr: "",
+ startedAt: Date(),
+ finishedAt: Date())
+ let gogDoctorConfigStatus = CrawlStatusMapper().status(from: gogDoctorConfigResult, manifest: BuiltInCrawlApps.gogcli)
+ try Self.expect(gogDoctorConfigStatus.state == .needsConfig, "gog doctor config warnings map to config setup")
+
+ let wacliResult = CrawlCommandResult(
+ appID: BuiltInCrawlApps.wacliID,
+ action: "status",
+ exitCode: 0,
+ stdout: #"{"success":true,"data":{"store_dir":"/tmp/wacli/accounts/me","authenticated":true,"store":{"messages":12,"chats":3,"last_sync_at":"2026-05-01T12:00:00Z"}}}"#,
+ stderr: "",
+ startedAt: Date(),
+ finishedAt: Date())
+ let wacliStatus = CrawlStatusMapper().status(from: wacliResult, manifest: BuiltInCrawlApps.wacli)
+ try Self.expect(wacliStatus.counts.contains(CrawlCount(id: "messages", label: "Messages", value: 12)), "wacli message counts map")
+ try Self.expect(wacliStatus.configPath == "/tmp/wacli/config.yaml", "wacli account config path maps")
+ try Self.expect(wacliStatus.databasePath == "/tmp/wacli/accounts/me/wacli.db", "wacli database path maps")
+ try Self.expect(wacliStatus.databases.contains { $0.kind == .sqlite && $0.path == "/tmp/wacli/accounts/me/wacli.db" }, "wacli database inventory keeps sqlite resource")
+ try Self.expect(wacliStatus.databases.contains { $0.kind == .logical && $0.path == "/tmp/wacli/accounts/me" }, "wacli database inventory keeps logical store")
+
+ let wacliStoreErrorResult = CrawlCommandResult(
+ appID: BuiltInCrawlApps.wacliID,
+ action: "status",
+ exitCode: 0,
+ stdout: #"{"success":true,"data":{"authenticated":true,"store_error":"database disk image is malformed"}}"#,
+ stderr: "",
+ startedAt: Date(),
+ finishedAt: Date())
+ let wacliStoreErrorStatus = CrawlStatusMapper().status(from: wacliStoreErrorResult, manifest: BuiltInCrawlApps.wacli)
+ try Self.expect(wacliStoreErrorStatus.state == .error, "wacli store errors map to error status")
+ try Self.expect(wacliStoreErrorStatus.errors.contains("database disk image is malformed"), "wacli store error is preserved")
+
+ let wacliFirstRunResult = CrawlCommandResult(
+ appID: BuiltInCrawlApps.wacliID,
+ action: "status",
+ exitCode: 0,
+ stdout: #"{"success":true,"data":{"authenticated":false,"store_error":"open store: no such file"}}"#,
+ stderr: "",
+ startedAt: Date(),
+ finishedAt: Date())
+ let wacliFirstRunStatus = CrawlStatusMapper().status(from: wacliFirstRunResult, manifest: BuiltInCrawlApps.wacli)
+ try Self.expect(wacliFirstRunStatus.state == .needsAuth, "wacli first-run store errors stay auth setup")
+ try Self.expect(wacliFirstRunStatus.summary == "WhatsApp auth needs setup", "wacli first-run summary stays setup-oriented")
+
+ let wacliCorruptUnauthedResult = CrawlCommandResult(
+ appID: BuiltInCrawlApps.wacliID,
+ action: "status",
+ exitCode: 0,
+ stdout: #"{"success":true,"data":{"authenticated":false,"store_error":"database disk image is malformed"}}"#,
+ stderr: "",
+ startedAt: Date(),
+ finishedAt: Date())
+ let wacliCorruptUnauthedStatus = CrawlStatusMapper().status(from: wacliCorruptUnauthedResult, manifest: BuiltInCrawlApps.wacli)
+ try Self.expect(wacliCorruptUnauthedStatus.state == .error, "wacli corrupt unauthenticated stores stay errors")
+
+ let crawlKitOutput = """
+ {
+ "schema_version": "crawlkit.control.v1",
+ "app_id": "discrawl",
+ "state": "current",
+ "summary": "5052 messages across 293 channels",
+ "database_path": "/tmp/discrawl.db",
+ "database_bytes": 36397056,
+ "counts": [
+ {"id": "guilds", "label": "Guilds", "value": 56},
+ {"id": "channels", "label": "Channels", "value": 293},
+ {"id": "messages", "label": "Messages", "value": 5052}
+ ],
+ "databases": [
+ {
+ "id": "primary",
+ "label": "Discord archive",
+ "kind": "sqlite",
+ "role": "archive",
+ "path": "/tmp/discrawl.db",
+ "is_primary": true,
+ "bytes": 36397056,
+ "modified_at": "2026-04-24T07:38:30Z",
+ "counts": [
+ {"id": "messages", "label": "Messages", "value": 5052}
+ ]
+ }
+ ]
+ }
+ """
+ let crawlKitResult = CrawlCommandResult(
+ appID: BuiltInCrawlApps.discrawlID,
+ action: "status",
+ exitCode: 0,
+ stdout: crawlKitOutput,
+ stderr: "",
+ startedAt: Date(),
+ finishedAt: Date())
+
+ let crawlKitStatus = CrawlStatusMapper().status(from: crawlKitResult, manifest: BuiltInCrawlApps.discrawl)
+ try Self.expect(crawlKitStatus.summary == "5052 messages across 293 channels", "crawlkit status summary maps")
+ try Self.expect(crawlKitStatus.state == .current, "crawlkit explicit state maps")
+ try Self.expect(crawlKitStatus.databaseBytes == 36397056, "crawlkit database bytes map")
+ try Self.expect(crawlKitStatus.counts.contains(CrawlCount(id: "messages", label: "Messages", value: 5052)), "crawlkit count array maps")
+ try Self.expect(crawlKitStatus.databases.first?.id == "primary", "crawlkit databases map")
+ try Self.expect(crawlKitStatus.databases.first?.modifiedAt != nil, "crawlkit database modified date maps")
+ try Self.expect(crawlKitStatus.databases.first?.counts.contains(CrawlCount(id: "messages", label: "Messages", value: 5052)) == true, "crawlkit database counts map")
+
+ let cloudOutput = """
+ {
+ "schema_version": "crawlkit.control.v1",
+ "app_id": "discrawl",
+ "state": "current",
+ "summary": "1417329 messages in remote archive discrawl/openclaw",
+ "config_path": "/tmp/discrawl.toml",
+ "counts": [
+ {"id": "channels", "label": "Channels", "value": 23956},
+ {"id": "messages", "label": "Messages", "value": 1417329},
+ {"id": "members", "label": "Members", "value": 173089}
+ ],
+ "remote": {
+ "enabled": true,
+ "mode": "cloud",
+ "endpoint": "https://crawl.example.test",
+ "archive": "discrawl/openclaw",
+ "last_ingest_at": "2026-05-28T19:30:56.840Z"
+ },
+ "databases": [
+ {
+ "id": "remote",
+ "label": "Discord cloud archive",
+ "kind": "cloudflare-d1",
+ "role": "archive",
+ "endpoint": "https://crawl.example.test",
+ "archive": "discrawl/openclaw",
+ "is_primary": true,
+ "counts": [
+ {"id": "messages", "label": "Messages", "value": 1417329}
+ ]
+ }
+ ],
+ "sqlite_bundle": {
+ "key": "v1/discrawl/discrawl%2Fopenclaw/sqlite/current.manifest.json",
+ "content_type": "application/json",
+ "uploaded_at": "2026-05-28T19:30:56.840Z",
+ "manifest": {
+ "format": "sqlite-gzip-chunked-v1",
+ "generated_at": "2026-05-28T19:30:41Z",
+ "compression": {"algorithm": "gzip"},
+ "object": {"key": "v1/discrawl/discrawl%2Fopenclaw/sqlite/current.db", "size": 839589888, "sha256": "raw"},
+ "compressed_object": {"key": "v1/discrawl/discrawl%2Fopenclaw/sqlite/current.db.gz", "size": 259315038, "sha256": "compressed"},
+ "parts": [
+ {"index": 0, "key": "part-0", "size": 67108864, "sha256": "a"},
+ {"index": 1, "key": "part-1", "size": 67108864, "sha256": "b"},
+ {"index": 2, "key": "part-2", "size": 67108864, "sha256": "c"},
+ {"index": 3, "key": "part-3", "size": 57988446, "sha256": "d"}
+ ]
+ }
+ }
+ }
+ """
+ let cloudResult = CrawlCommandResult(
+ appID: BuiltInCrawlApps.discrawlID,
+ action: "status",
+ exitCode: 0,
+ stdout: cloudOutput,
+ stderr: "",
+ startedAt: Date(),
+ finishedAt: Date())
+ let cloudStatus = CrawlStatusMapper().status(from: cloudResult, manifest: BuiltInCrawlApps.discrawl)
+ try Self.expect(cloudStatus.remote?.archive == "discrawl/openclaw", "remote archive maps")
+ try Self.expect(cloudStatus.lastSyncAt != nil, "remote ingest maps as sync freshness")
+ try Self.expect(cloudStatus.databases.first?.kind == .cloudflareD1, "remote database kind maps")
+ try Self.expect(cloudStatus.databases.first?.endpoint == "https://crawl.example.test", "remote database endpoint maps")
+ try Self.expect(cloudStatus.sqliteBundle?.format == "sqlite-gzip-chunked-v1", "sqlite bundle format maps")
+ try Self.expect(cloudStatus.sqliteBundle?.compression == "gzip", "sqlite bundle compression maps")
+ try Self.expect(cloudStatus.sqliteBundle?.rawBytes == 839589888, "sqlite bundle raw size maps")
+ try Self.expect(cloudStatus.sqliteBundle?.compressedBytes == 259315038, "sqlite bundle compressed size maps")
+ try Self.expect(cloudStatus.sqliteBundle?.partCount == 4, "sqlite bundle part count maps")
+
+ let telecrawlOutput = """
+ {
+ "db_path": "/tmp/telecrawl.db",
+ "chats": 3,
+ "messages": 42,
+ "unread_chats": 1,
+ "unread_messages": 5,
+ "media_messages": 6,
+ "folders": 2,
+ "topics": 4,
+ "last_import_at": "2026-05-01T12:00:00Z"
+ }
+ """
+ let telecrawlResult = CrawlCommandResult(
+ appID: BuiltInCrawlApps.telecrawlID,
+ action: "status",
+ exitCode: 0,
+ stdout: telecrawlOutput,
+ stderr: "",
+ startedAt: Date(),
+ finishedAt: Date())
+ let telecrawlStatus = CrawlStatusMapper().status(
+ from: telecrawlResult,
+ manifest: BuiltInCrawlApps.telecrawl,
+ staleAfterSeconds: 60)
+ try Self.expect(telecrawlStatus.counts.contains(CrawlCount(id: "messages", label: "Messages", value: 42)), "telecrawl messages map")
+ try Self.expect(telecrawlStatus.counts.contains(CrawlCount(id: "chats", label: "Chats", value: 3)), "telecrawl chats map")
+ try Self.expect(telecrawlStatus.lastSyncAt == telecrawlStatus.lastImportAt, "telecrawl import time maps to sync freshness")
+ try Self.expect(telecrawlStatus.state == .stale, "telecrawl import freshness drives stale state")
+ try Self.expect(telecrawlStatus.databases.first?.label == "Telegram archive", "telecrawl database inventory maps")
+
+ let okOutput = """
+ {
+ "schema_version": "crawlkit.control.v1",
+ "app_id": "graincrawl",
+ "state": "ok",
+ "summary": "1 notes",
+ "counts": [{"id": "notes", "label": "Notes", "value": 1}]
+ }
+ """
+ let okResult = CrawlCommandResult(
+ appID: BuiltInCrawlApps.graincrawlID,
+ action: "status",
+ exitCode: 0,
+ stdout: okOutput,
+ stderr: "",
+ startedAt: Date(),
+ finishedAt: Date())
+ let okStatus = CrawlStatusMapper().status(from: okResult, manifest: BuiltInCrawlApps.graincrawl)
+ try Self.expect(okStatus.state == .current, "crawlkit ok state maps to current")
+
+ let failedOutput = """
+ {"schema_version":"crawlkit.control.v1","app_id":"graincrawl","state":"failed","summary":"broken"}
+ """
+ let failedResult = CrawlCommandResult(
+ appID: BuiltInCrawlApps.graincrawlID,
+ action: "status",
+ exitCode: 0,
+ stdout: failedOutput,
+ stderr: "",
+ startedAt: Date(),
+ finishedAt: Date())
+ let failedStatus = CrawlStatusMapper().status(from: failedResult, manifest: BuiltInCrawlApps.graincrawl)
+ try Self.expect(failedStatus.state == .error, "crawlkit failed state maps to error")
+
+ let githubAuthMessage = """
+ [github] request GET /repos/openclaw/openclaw
+ github GET /repos/openclaw/openclaw failed with status 401: {
+ "message": "Bad credentials"
+ }
+ """
+ let githubAuthResult = CrawlCommandResult(
+ appID: BuiltInCrawlApps.gitcrawlID,
+ action: "status",
+ exitCode: 1,
+ stdout: "",
+ stderr: githubAuthMessage,
+ startedAt: Date(),
+ finishedAt: Date())
+ let githubAuthStatus = CrawlStatusMapper().status(from: githubAuthResult, manifest: BuiltInCrawlApps.gitcrawl)
+ try Self.expect(githubAuthStatus.state == .needsAuth, "gitcrawl 401 maps to auth state")
+ try Self.expect(githubAuthStatus.summary == "GitHub credentials rejected", "gitcrawl 401 uses useful summary")
+ try Self.expect(githubAuthStatus.errors == ["GitHub credentials rejected"], "gitcrawl 401 keeps request trace out of status errors")
+
+ let githubServerMessage = """
+ [github] request GET /repos/openclaw/openclaw
+ github GET /repos/openclaw/openclaw failed with status 500
+ """
+ let githubServerStatus = CrawlAppStatus.commandFailure(
+ appID: BuiltInCrawlApps.gitcrawlID,
+ action: "refresh",
+ message: githubServerMessage,
+ fallback: "refresh failed")
+ try Self.expect(
+ githubServerStatus.summary == "refresh: github GET /repos/openclaw/openclaw failed with status 500",
+ "gitcrawl request trace is skipped in failure summaries")
+ }
+}
diff --git a/Sources/CrawlBarSelfTest/SelfTestStorageAndRedaction.swift b/Sources/CrawlBarSelfTest/SelfTestStorageAndRedaction.swift
new file mode 100644
index 0000000..2496521
--- /dev/null
+++ b/Sources/CrawlBarSelfTest/SelfTestStorageAndRedaction.swift
@@ -0,0 +1,182 @@
+import CrawlBarCore
+import Foundation
+
+extension CrawlBarSelfTest {
+ static func testActionLogStoreReadsRecentResults() throws {
+ let directory = FileManager.default.temporaryDirectory
+ .appendingPathComponent("crawlbar-logs-\(UUID().uuidString)", isDirectory: true)
+ let store = CrawlActionLogStore(directoryURL: directory)
+ defer { try? FileManager.default.removeItem(at: directory) }
+
+ let result = CrawlCommandResult(
+ appID: BuiltInCrawlApps.graincrawlID,
+ action: "refresh",
+ exitCode: 1,
+ stdout: "",
+ stderr: "Granola access token expired",
+ startedAt: Date(timeIntervalSince1970: 1_775_000_000),
+ finishedAt: Date(timeIntervalSince1970: 1_775_000_001))
+ _ = try store.save(result)
+
+ let recent = store.recentResults(limit: 5)
+ try Self.expect(recent.first == result, "action logs decode back into recent command results")
+
+ let successfulJSONResult = CrawlCommandResult(
+ appID: BuiltInCrawlApps.graincrawlID,
+ action: "refresh",
+ exitCode: 0,
+ stdout: """
+ {"notes":1}
+ """,
+ stderr: "",
+ startedAt: Date(timeIntervalSince1970: 1_775_000_002),
+ finishedAt: Date(timeIntervalSince1970: 1_775_000_003))
+ try Self.expect(successfulJSONResult.userFacingRunMessage == nil, "successful stdout is not shown as a run message")
+ try Self.expect(!successfulJSONResult.shouldShowExitCode, "successful runs do not show exit code")
+
+ let warningResult = CrawlCommandResult(
+ appID: BuiltInCrawlApps.graincrawlID,
+ action: "refresh",
+ exitCode: 0,
+ stdout: """
+ {"notes":1}
+ """,
+ stderr: "Used cached Granola data",
+ startedAt: Date(timeIntervalSince1970: 1_775_000_004),
+ finishedAt: Date(timeIntervalSince1970: 1_775_000_005))
+ try Self.expect(warningResult.userFacingRunMessage == "Used cached Granola data", "successful stderr can still surface a warning")
+
+ let failedGitHubResult = CrawlCommandResult(
+ appID: BuiltInCrawlApps.gitcrawlID,
+ action: "refresh",
+ exitCode: 1,
+ stdout: "",
+ stderr: """
+ [github] request GET /repos/openclaw/openclaw
+ github GET /repos/openclaw/openclaw failed with status 401: {
+ "message": "Bad credentials"
+ }
+ """,
+ startedAt: Date(timeIntervalSince1970: 1_775_000_006),
+ finishedAt: Date(timeIntervalSince1970: 1_775_000_007))
+ try Self.expect(failedGitHubResult.userFacingRunMessage == "GitHub credentials rejected", "failed gitcrawl run message is normalized")
+
+ let failedBirdResult = CrawlCommandResult(
+ appID: BuiltInCrawlApps.birdclawID,
+ action: "status",
+ exitCode: 1,
+ stdout: "",
+ stderr: "Missing auth_token",
+ startedAt: Date(timeIntervalSince1970: 1_775_000_006),
+ finishedAt: Date(timeIntervalSince1970: 1_775_000_007))
+ try Self.expect(failedBirdResult.userFacingRunMessage == "X browser cookies not found", "failed X credential check maps to auth setup")
+
+ let failedStdoutResult = CrawlCommandResult(
+ appID: BuiltInCrawlApps.graincrawlID,
+ action: "refresh",
+ exitCode: 1,
+ stdout: "Granola refresh failed",
+ stderr: "",
+ startedAt: Date(timeIntervalSince1970: 1_775_000_008),
+ finishedAt: Date(timeIntervalSince1970: 1_775_000_009))
+ try Self.expect(failedStdoutResult.userFacingRunMessage == "Granola refresh failed", "failed stdout is shown as a run message")
+ try Self.expect(failedStdoutResult.shouldShowExitCode, "failed runs show exit code")
+ }
+
+ static func testCommandTimeoutEscalates() throws {
+ let directory = FileManager.default.temporaryDirectory
+ .appendingPathComponent("crawlbar-timeout-\(UUID().uuidString)", isDirectory: true)
+ try FileManager.default.createDirectory(at: directory, withIntermediateDirectories: true)
+ defer { try? FileManager.default.removeItem(at: directory) }
+
+ let scriptURL = directory.appendingPathComponent("ignore-term.sh")
+ try Data("""
+ #!/bin/sh
+ trap '' TERM
+ sleep 5
+ """.utf8).write(to: scriptURL)
+ try FileManager.default.setAttributes([.posixPermissions: 0o755], ofItemAtPath: scriptURL.path)
+
+ let manifest = CrawlAppManifest(
+ id: CrawlAppID(rawValue: "timeoutcrawl"),
+ displayName: "Timeout Crawl",
+ description: "A timeout test crawler",
+ binary: .init(name: scriptURL.path),
+ branding: .init(symbolName: "timer", accentColor: "#123456"),
+ paths: .init(),
+ commands: ["status": []],
+ capabilities: [.status])
+ let installation = CrawlAppInstallation(manifest: manifest, binaryPath: scriptURL.path)
+ let startedAt = Date()
+ do {
+ _ = try CrawlCommandRunner().run(installation: installation, action: "status", timeoutSeconds: 0.1)
+ throw SelfTestError.failed("timeout command should not complete")
+ } catch CrawlCommandRunnerError.timedOut {
+ try Self.expect(Date().timeIntervalSince(startedAt) < 2.5, "timed-out commands are killed promptly")
+ }
+ }
+
+ static func testDatabaseBackupCopiesFiles() throws {
+ let directory = FileManager.default.temporaryDirectory
+ .appendingPathComponent("crawlbar-backup-\(UUID().uuidString)", isDirectory: true)
+ try FileManager.default.createDirectory(at: directory, withIntermediateDirectories: true)
+ defer { try? FileManager.default.removeItem(at: directory) }
+
+ let firstDirectory = directory.appendingPathComponent("first", isDirectory: true)
+ let secondDirectory = directory.appendingPathComponent("second", isDirectory: true)
+ try FileManager.default.createDirectory(at: firstDirectory, withIntermediateDirectories: true)
+ try FileManager.default.createDirectory(at: secondDirectory, withIntermediateDirectories: true)
+ let firstDatabaseURL = firstDirectory.appendingPathComponent("sample.db")
+ let secondDatabaseURL = secondDirectory.appendingPathComponent("sample.db")
+ try Self.createSQLiteDatabase(firstDatabaseURL, value: "sqlite-one")
+ try Self.createSQLiteDatabase(secondDatabaseURL, value: "sqlite-two")
+ let status = CrawlAppStatus(
+ appID: BuiltInCrawlApps.notcrawlID,
+ state: .current,
+ summary: "ok",
+ databases: [
+ CrawlDatabaseResource(
+ id: firstDatabaseURL.path,
+ label: "Workspace One",
+ kind: .sqlite,
+ path: firstDatabaseURL.path,
+ isPrimary: true),
+ CrawlDatabaseResource(
+ id: secondDatabaseURL.path,
+ label: "Workspace Two",
+ kind: .sqlite,
+ path: secondDatabaseURL.path,
+ isPrimary: false),
+ ])
+
+ let backup = try CrawlDatabaseBackupStore.backup(status: status, root: directory.appendingPathComponent("backups", isDirectory: true))
+ try Self.expect(backup.files.count == 2, "backup copies duplicate-named files")
+ try Self.expect(Set(backup.files.map { URL(fileURLWithPath: $0).lastPathComponent }).count == 2, "backup destination names are unique")
+ let copiedContents = try backup.files.map { try Self.sqliteValue(URL(fileURLWithPath: $0)) }
+ try Self.expect(copiedContents.contains("sqlite-one"), "backup preserves first duplicate file")
+ try Self.expect(copiedContents.contains("sqlite-two"), "backup preserves second duplicate file")
+ }
+
+ static func testRedactorScrubsSecrets() throws {
+ let redacted = CrawlCommandRedactor().redact("""
+ token=abc123
+ Authorization: Bearer secret-token
+ discord_token=discord-secret
+ github_pat_1234567890abcdef
+ ghp_1234567890abcdef
+ sk-proj-1234567890abcdef
+ xoxc-1234567890abcdef
+ secret_notion123
+ mfa.discordsecret
+ ct0: csrf-secret
+ label=Discord archive
+ """)
+ try Self.expect(!redacted.contains("abc123"), "token value redacts")
+ try Self.expect(!redacted.contains("secret-token"), "bearer value redacts")
+ try Self.expect(!redacted.contains("discord-secret"), "discord token value redacts")
+ try Self.expect(!redacted.contains("1234567890abcdef"), "bare tokens redact")
+ try Self.expect(!redacted.contains("notion123"), "notion secrets redact")
+ try Self.expect(!redacted.contains("csrf-secret"), "ct0 cookies redact")
+ try Self.expect(redacted.contains("Discord archive"), "discord labels are not redacted")
+ }
+}
diff --git a/Sources/CrawlBarSelfTest/SelfTestSupport.swift b/Sources/CrawlBarSelfTest/SelfTestSupport.swift
new file mode 100644
index 0000000..60d4ccb
--- /dev/null
+++ b/Sources/CrawlBarSelfTest/SelfTestSupport.swift
@@ -0,0 +1,50 @@
+import CrawlBarCore
+import Foundation
+
+extension CrawlBarSelfTest {
+ static func expect(_ condition: Bool, _ message: String) throws {
+ if !condition {
+ throw SelfTestError.failed(message)
+ }
+ }
+
+ static func createSQLiteDatabase(_ url: URL, value: String) throws {
+ try Self.runSQLite(url, sql: "create table sample(value text); insert into sample(value) values('\(value)');")
+ }
+
+ static func sqliteValue(_ url: URL) throws -> String {
+ try Self.runSQLite(url, sql: "select value from sample limit 1;")
+ }
+
+ @discardableResult
+ static func runSQLite(_ url: URL, sql: String) throws -> String {
+ guard let sqlitePath = CrawlExecutableResolver().resolve("sqlite3") else {
+ throw SelfTestError.failed("sqlite3 is available")
+ }
+ let process = Process()
+ process.executableURL = URL(fileURLWithPath: sqlitePath)
+ process.arguments = [url.path, sql]
+ let output = Pipe()
+ process.standardOutput = output
+ process.standardError = output
+ try process.run()
+ process.waitUntilExit()
+ let data = output.fileHandleForReading.readDataToEndOfFile()
+ let text = String(data: data, encoding: .utf8) ?? ""
+ guard process.terminationStatus == 0 else {
+ throw SelfTestError.failed("sqlite3 failed: \(text)")
+ }
+ return text.trimmingCharacters(in: .whitespacesAndNewlines)
+ }
+}
+
+enum SelfTestError: LocalizedError {
+ case failed(String)
+
+ var errorDescription: String? {
+ switch self {
+ case let .failed(message):
+ "selftest failed: \(message)"
+ }
+ }
+}
diff --git a/docs/quality-rubric.md b/docs/quality-rubric.md
new file mode 100644
index 0000000..678dc89
--- /dev/null
+++ b/docs/quality-rubric.md
@@ -0,0 +1,467 @@
+---
+written_by: ai
+---
+
+# CrawlBar Quality Rubric
+
+This is the engineering review contract for CrawlBar. Use it with
+`docs/ui-rules.md`: this file covers app quality, architecture, API boundaries,
+proof, privacy, and complexity reduction; `ui-rules.md` covers the smaller UI
+design-system contract.
+
+CrawlBar's product boundary is simple: a professional native macOS control
+plane for local source crawlers. Source crawlers own source parsing, archives,
+auth/session handling, search/status/open/evidence, and privacy policy.
+CrawlBar discovers crawler contracts, displays status, routes actions, and
+keeps local configuration legible.
+
+## How Reviewers Use This
+
+Review against axes and evidence, not vibes. A finding should include:
+
+```text
+axis: one rubric axis below
+severity: P0, P1, P2, or P3
+evidence: file/line, command output, screenshot, AX dump, or exact behavior
+recommended_fix: smallest change that improves the axis
+proof_needed: command, runtime observation, screenshot, or reviewer check
+```
+
+Severity:
+
+- `P0`: breaks the product boundary, privacy, build/run, or basic usability.
+- `P1`: makes the app materially harder to maintain, observe, or trust.
+- `P2`: unclear responsibility, one-off UI, avoidable API surface, or optional
+ complexity that will compound.
+- `P3`: polish, naming, or cleanup that is useful but not blocking.
+
+A good stopping state has no accepted P0/P1 findings left. P2 findings are
+either fixed, explicitly deferred with a reason, or named as the next slice. P3
+findings usually do not block when the proof gates pass.
+
+## Measurement Protocol
+
+Start substantial review or refactor work with:
+
+```sh
+Scripts/quality_baseline.sh
+```
+
+The baseline is evidence, not a pass/fail gate. Do not optimize a single metric.
+File size, type count, screenshots, build success, and pretty UI are proxies.
+Progress means reduced cognitive load, clearer ownership, fewer concepts,
+better native behavior, stronger proof, or a smaller real API.
+
+Measure at least:
+
+- Swift LOC by target and by file.
+- Files over 400 LOC.
+- Top-level type count per file.
+- Public and package API declarations.
+- SwiftPM products.
+- Settings panels, toggles, fields, and command buttons.
+- Single-use UI wrappers and helper types.
+- AppKit, process, filesystem, and task references by target.
+
+Suggested thresholds:
+
+- No new Swift file over 400 LOC.
+- Prefer 12 or fewer top-level types per file.
+- Treat 20 top-level types in one file as a hard review stop.
+- Keep shared UI primitives scarce; do not create one for a single call site.
+- Keep `CrawlBarCore` free of SwiftUI and AppKit imports.
+- Keep public/package API intentional and compatible with shipped SwiftPM
+ products.
+
+Metrics only matter when the qualitative review agrees that a concept,
+responsibility, interface, or workflow became easier to understand.
+
+## Review Scorecard
+
+For non-trivial reviews or handoffs, this compact scorecard keeps the result
+auditable without turning the rubric into ceremony:
+
+```text
+baseline: command/path/date used
+build_proof: pass/fail/blocker
+runtime_proof: AX, screenshot, log, or blocker
+P0/P1: none, or listed with owner/fix
+P2: fixed, deferred, or next slice
+axis_changed: rubric axis improved
+anti_gaming_check: why this is not just LOC/files/build/screenshot progress
+next_slice: highest-leverage remaining accepted finding
+```
+
+## Planning Notes
+
+For multi-step CrawlBar work, derive success criteria from the rubric instead
+of from a generic cleanup target. A useful work note names:
+
+- the product behavior or rubric axis being improved;
+- current evidence and known failures;
+- unacceptable drift and bad proxies;
+- proof commands or runtime observations;
+- review lenses to run before handoff;
+- exact completion proof.
+
+Keep volatile implementation state out of the rubric. Keep architecture
+rationale in repository-local docs only when the decision should survive beyond
+the current branch or PR. Keep handoffs concise and decision-grade.
+
+## Axis 1: Product Boundary
+
+Good:
+
+- CrawlBar is a menu/settings control plane for crawler commands.
+- It uses source-native crawler names and command contracts.
+- It shows counts, status, freshness, capability, paths, and actions without
+ becoming a higher-level synthesis surface.
+
+Bad signals:
+
+- New concepts that belong outside the crawler control plane.
+- UI or API names that hide source-native crawler names.
+- CrawlBar interpreting private source data beyond status/control needs.
+
+Measure:
+
+- Search new names for invented umbrella terms.
+- Check action/status fields against manifests and crawler CLIs.
+- Verify UI/log output contains no raw private content.
+
+## Axis 2: Native macOS Behavior
+
+Good:
+
+- Menu bar is for quick state and frequent actions.
+- Settings is the full management surface.
+- `Settings...` and Command-Comma open settings.
+- Menu labels are concise, title case, grouped, and use ellipsis when more
+ input or a window is needed.
+- Settings avoid asking for setup the app can detect.
+
+Bad signals:
+
+- Menu becomes a dashboard.
+- Settings is unreachable or behaves like an arbitrary custom window.
+- Native app availability is treated as a hidden preference instead of
+ detected state.
+- Long or unclear menu labels.
+
+Measure:
+
+- AX dump for menu names, roles, enabled state, and order.
+- AX or visual proof that Settings opens from menu and keyboard.
+- Visual proof across the main settings window.
+
+Sources:
+
+- Apple documentation for menu bar extras, settings scenes, menus, and
+ accessibility.
+- Build macOS Apps guidance for packaged app launch, AppKit interop, window
+ management, and runtime proof.
+
+## Axis 3: Observability And Proof
+
+Good:
+
+- A reviewer can prove build, launch, menu, settings, and core actions from
+ commands and observations.
+- AX can read names, roles, values, enabled states, and trigger key actions.
+- Screenshots or Computer Use verify rendering when available.
+- Logs explain menu/settings/action events without leaking private data.
+
+Bad signals:
+
+- "Looks fine" without screenshot or AX evidence.
+- Screenshots without behavior proof.
+- AX item names are missing or custom rows are unreadable.
+- Logs are silent during user-visible actions that need diagnosis.
+
+Measure:
+
+```sh
+swift build
+swift run crawlbar-selftest
+Scripts/package_app.sh
+codesign --verify --deep --strict --verbose=2 dist/CrawlBar.app
+dist/CrawlBar.app/Contents/Helpers/crawlbar config validate
+```
+
+Use packaged-app AX scripts to press the menu extra, dump menu item
+names/roles/enabled states, open Settings, and dump the settings window
+hierarchy. Raw SwiftPM GUI executable launch is not sufficient product proof.
+
+## Axis 4: Architecture And DAG
+
+Good:
+
+- `CrawlBarCore` owns data contracts, manifest/config/status/action models,
+ runners, registries, redaction, and source-neutral services.
+- `CrawlBar` owns AppKit, SwiftUI, windows, menu bar, settings UI, and
+ presentation state.
+- `CrawlBarCLI` owns argument parsing and command output only.
+- `CrawlBarSelfTest` owns executable contract checks only.
+- Dependencies point inward to Core; Core does not import AppKit or SwiftUI.
+
+Bad signals:
+
+- AppKit or SwiftUI in Core.
+- Process, filesystem, config persistence, or crawler command policy embedded
+ in view bodies.
+- Menu and settings features importing each other without a clear reason.
+- Services exposed publicly because a boundary is leaky.
+
+Measure:
+
+```sh
+rg -n 'import (AppKit|SwiftUI)' Sources/CrawlBarCore
+rg -n 'Process\(|FileManager\.|NSWorkspace|NSApp|NSWindow|NSMenu|Task\s*\{' Sources
+rg -n '^(public|package) ' Sources/CrawlBarCore
+```
+
+Review whether every import and every public/package declaration has a real
+caller and a real contract.
+
+## Axis 5: Deep Modules And API Surface
+
+Good:
+
+- Public/package API exposes stable contracts, not implementation convenience.
+- Interfaces are small, explicit, and source-native.
+- Sequencing and policy are hidden inside deep modules.
+- Callers should not need to know how status, manifests, redaction, or command
+ arguments are assembled.
+
+Bad signals:
+
+- Boolean-heavy shallow helpers.
+- Many tiny services that merely pass through to each other.
+- Package visibility used to avoid designing a boundary.
+- Optional extras exposed as first-class knobs before repeated pain exists.
+- Removing a shipped SwiftPM product or public type inside a cleanup PR.
+
+Measure:
+
+- Count public/package declarations and justify them.
+- Compare SwiftPM products against the released package surface.
+- For each service, ask what complexity it hides from callers.
+- Review call sites for repeated policy or command assembly.
+
+## Axis 6: State And Effects Ownership
+
+Good:
+
+- One scene or feature model owns scene state.
+- Rows/panels receive explicit values, bindings, and callbacks.
+- Effects live in model/service/action layers, not low-level UI components.
+- Async tasks have clear owner, cancellation, and stale-result behavior.
+
+Bad signals:
+
+- A view owns process execution, config writes, filesystem operations, or
+ crawler policy.
+- Many tiny row view models.
+- Notification, timer, task, and reload logic scattered across unrelated files.
+- State mutation hidden behind broad closures with unclear side effects.
+
+Measure:
+
+- Search for effects in SwiftUI files.
+- Review async tasks for ownership and cancellation.
+- Check whether a new behavior changes one file or many unrelated files.
+
+## Axis 7: UI System
+
+Use `docs/ui-rules.md` as the detailed UI contract.
+
+Good:
+
+- 8-12 shared UI primitives total.
+- New UI files stay under about 400 LOC.
+- Prefer 12 or fewer top-level UI types per file; 20 is a hard smell.
+- Feature views compose shared primitives instead of redefining visual grammar.
+
+Bad signals:
+
+- A "design system" where every object is its own primitive.
+- Feature files recreating rows, cards, panels, issue banners, or status dots.
+- Custom chrome fighting native macOS sidebar/detail behavior.
+
+Measure:
+
+```sh
+find Sources -name '*.swift' -print0 | xargs -0 wc -l | sort -nr | head
+find Sources -name '*.swift' -print0 | xargs -0 rg -n '^(struct|class|enum|protocol) '
+```
+
+## Axis 8: Manifest And Crawler Contracts
+
+Good:
+
+- Built-in and external manifests are the primary app model.
+- Crawler-specific differences live in manifests or narrow adapters.
+- CrawlBar can explain whether a crawler is available, configured, suggested,
+ or unsupported from source-backed signals.
+
+Bad signals:
+
+- Hardcoded UI branching for a crawler when a manifest field would express it.
+- Manifest fields added speculatively for one crawler without repeated need.
+- CrawlBar assuming command names that drift from crawler CLIs.
+
+Measure:
+
+- Compare manifests against current crawler CLI help/status/metadata commands.
+- Validate config and metadata through `crawlbarctl`.
+- Review any crawler-specific code path for whether it should be manifest data.
+
+## Axis 9: Reliability And Dev Lifecycle
+
+Good:
+
+- One obvious build/package/run path exists for the app.
+- Dev does not require replacing Homebrew/system binaries.
+- The packaged app contains resources, helper binary, bundle metadata, and
+ signing expected by macOS.
+- Runtime proof uses the packaged app, not only raw SwiftPM executables.
+
+Bad signals:
+
+- Manual command chains that differ per agent.
+- Raw GUI executable launch used as product proof.
+- Stale packaged app or helper binary.
+- Build passes but packaged app cannot be observed.
+
+Measure:
+
+- `swift build`
+- `swift run crawlbar-selftest`
+- `Scripts/package_app.sh`
+- `codesign --verify --deep --strict --verbose=2 dist/CrawlBar.app`
+- `pgrep -fl CrawlBar`
+- Helper CLI `--help` and `config validate`.
+
+## Axis 10: Privacy And Redaction
+
+Good:
+
+- UI, logs, screenshots, and CLI output use paths, counts, status, capability,
+ and aggregate summaries.
+- Command output is redacted before persistence or UI display.
+- Secrets are kept out of config and scrubbed from settings state.
+
+Bad signals:
+
+- Raw contacts, phone numbers, message bodies, mail bodies, GPS coordinates,
+ tokens, or private reports in durable docs/logs/screenshots.
+- Review output that includes private source content when capability/status
+ would be enough.
+
+Measure:
+
+- Inspect action logs and screenshot/AX artifacts before sharing.
+- Search for token-like/private fields in new outputs.
+- Review secret config handling for both save and window close paths.
+
+## Axis 11: Tests And Contract Checks
+
+Good:
+
+- Self-test covers manifest loading, command mapping, redaction, config, status
+ mapping, and edge cases that are easy to regress.
+- Runtime proof covers menu/settings behavior self-test cannot cover.
+- Tests make behavior clearer instead of duplicating implementation.
+
+Bad signals:
+
+- Large tests that are hard to localize.
+- Tests pass because they no longer cover the risky behavior.
+- UI refactors without runtime proof.
+
+Measure:
+
+- `swift run crawlbar-selftest`
+- Review self-test file size and cohesion.
+- Add or split tests when a change touches shared contracts.
+
+## Axis 12: Docs And Agent Operability
+
+Good:
+
+- Docs explain durable contracts, not volatile state.
+- Agent guidance points contributors to repository-local contracts.
+- Reviewers know what to measure and what counts as proof.
+- Implementation state stays out of the rubric.
+
+Bad signals:
+
+- Duplicate overviews.
+- Pointer stubs with no contract.
+- Markdown created to look productive.
+- Docs that freeze tactical guesses as requirements.
+- References to user-local files, private skills, or unreproducible process.
+
+Measure:
+
+- Check whether a new doc has one purpose and a real consumer.
+- Check whether future agents can use it without thread context.
+- Delete or consolidate stale docs only when they lose purpose.
+
+## Axis 13: Dead Code And Optional Complexity
+
+Good:
+
+- Unused types, buttons, config fields, and status branches are measured before
+ being kept.
+- Feature removal is explicit when it changes user behavior.
+- Optional capability UI appears only when the manifest/status proves it is
+ relevant.
+- The common path is visible without reading optional feature code.
+
+Bad signals:
+
+- Keeping code because it might be useful later.
+- Hiding a feature instead of deciding whether it belongs.
+- Deleting behavior only to improve LOC metrics.
+- Adding settings for rare maintainer workflows without proving they belong in
+ the app.
+
+Measure:
+
+- Low-reference type candidates from `Scripts/quality_baseline.sh`.
+- Settings surface count from `Scripts/quality_baseline.sh`.
+- Search for unreachable status branches and stale command mappings.
+- Review each optional feature concept for current user value, risk, and
+ removal approval needed.
+
+## Complexity Reduction Order
+
+When the code still feels too large, investigate in this order:
+
+1. Dead code: unused types, unused helpers, unreachable branches, stale command
+ mappings, stale status mappers.
+2. User-facing settings: toggles, fields, panels, and buttons that are not part
+ of the simple current path.
+3. API surface: public/package declarations and SwiftPM products.
+4. Feature concepts: install flows, remote execution, scheduling, snapshot
+ publishing, cloud archive controls, and other optional capabilities.
+5. UI primitives: one-off rows, panels, wrappers, and style variants.
+
+Stop before removing functionality unless the behavior is dead, unreachable, or
+explicitly approved for removal.
+
+## Pre-Handoff Review Lenses
+
+Before handoff, run these lenses and accept only findings grounded in evidence:
+
+- Principal engineer: does the structure reduce change amplification?
+- Ousterhout: are modules deep enough and interfaces narrow enough?
+- Zen of Python: is there one obvious path and explicit naming?
+- Native macOS: does it behave like a Mac menu/settings app?
+- Accessibility: can AX read and drive key UI?
+- Source discipline: are claims source-backed and source-native?
+- Privacy: are private data and secrets absent from outputs?
+- Complexity: did the change reduce concepts instead of moving lines?
+
+Stop reviewing when no accepted severe findings remain and proof is current.
diff --git a/docs/ui-rules.md b/docs/ui-rules.md
new file mode 100644
index 0000000..c54173c
--- /dev/null
+++ b/docs/ui-rules.md
@@ -0,0 +1,94 @@
+---
+written_by: ai
+---
+
+# CrawlBar UI Rules
+
+These rules keep CrawlBar's SwiftUI code native-first, small, and composable.
+They are a review rubric for UI refactor work, not a new framework.
+
+Use this together with `docs/quality-rubric.md`. That file covers architecture,
+API boundaries, native macOS behavior, proof, privacy, and review loops. This
+file covers the smaller UI design-system contract.
+
+Metrics are necessary but not sufficient. A refactor slice only counts as
+progress if it removes duplicated visual grammar or makes a product boundary
+clearer. LOC and type-count improvements alone do not count.
+
+## Design System Contract
+
+CrawlBar should have 8-12 shared UI primitives total. Shared primitives may
+encode stable visual vocabulary:
+
+- app icon
+- status dot/pill
+- panel
+- detail section
+- fact row
+- control row
+- switch row
+- metric row
+- empty state
+- inline issue
+- text formatters
+
+Do not add a shared UI primitive unless it replaces repeated styling in 2+
+places or encodes a stable CrawlBar-wide concept.
+
+Shared UI must not mention crawler names, command names, settings workflows,
+menu workflows, installation logic, or app-specific behavior.
+
+Shared UI primitives should stay deep, not broad. Do not hide feature-specific
+logic behind a primitive with many boolean flags or style variants. If a
+primitive needs more than 2-3 visual variants, stop and review the boundary.
+
+Shared UI primitives must not accept `CrawlAppConfig`, `CrawlAppManifest`,
+crawler IDs, or command names unless the primitive is explicitly CrawlBar-wide,
+such as an app icon or status view.
+
+## Feature Composition
+
+Feature views compose shared primitives. They may arrange crawler-specific
+content, but should not define their own panel, card, row, status, or issue
+styling.
+
+Feature-local components live under their feature folder, such as `Settings/`
+or `Menu/`. They are allowed when they hide cohesive product complexity and
+keep the root scene easier to read.
+
+Do not reward-hack the shared primitive cap by recreating panel, card, row,
+status, or issue styling inside feature files. Feature files arrange product
+content; shared primitives own visual grammar.
+
+## File Metrics
+
+- New UI files stay under 400 LOC.
+- Prefer 12 or fewer top-level UI types per file.
+- Treat 20 top-level types per file as a hard cap.
+- Large existing UI files should shrink over time unless there is a stated
+ reason.
+- A maintainer should be able to name what each file owns in one sentence.
+- Avoid giant computed view fragments. If a `body` or view helper grows beyond
+ roughly 80-120 LOC, extract a named feature component instead of hiding the
+ complexity inside one type.
+
+## State Boundaries
+
+- One scene or feature model owns scene state.
+- Rows and panels receive explicit values, bindings, and callbacks.
+- Do not create a view model per tiny row.
+- Views do not own process, filesystem, config, or command effects.
+- AppKit bridges stay narrow and do not leak through unrelated SwiftUI layers.
+
+## Review Checklist
+
+Before finishing a UI refactor slice:
+
+- Did this reduce concepts, or only move lines?
+- Did any new type become a one-off wrapper?
+- Would this change still look good if LOC and type count were hidden?
+- What boundary did this extraction improve in one sentence?
+- Are feature views composing shared primitives?
+- Is visual grammar centralized?
+- Can a maintainer find panel, row, and status styling immediately?
+- Would adding a crawler require product composition, not new visual primitives?