From 839d9a39c07c36d5d5dfae9fed4cabe2d14622b2 Mon Sep 17 00:00:00 2001 From: Joshua Pham Date: Sun, 8 Mar 2026 22:08:49 -0400 Subject: [PATCH 01/23] Add Inbox Capture template and ML Kit handwriting recognition Replace quick pages with an Inbox Capture template that renders a frontmatter zone (auto-filled created date, handwritable tags area), a divider line, and a lined content area below. This is the first step toward syncing handwritten notes to Obsidian as markdown files. - Add ML Kit Digital Ink Recognition dependency (19.0.0) - Add "inbox" native background type with drawInboxBg() renderer - Change quick page creation to use inbox background - Add InkTestView debug screen for testing recognition (~111ms on NoteAir5C) - Add CLAUDE.md with project setup and build instructions - Add docs/inbox-capture.md with full feature specification - Add IS_NEXT=false to gradle.properties for local builds Co-Authored-By: Claude Opus 4.6 --- CLAUDE.md | 189 ++++++++++++ app/build.gradle | 3 + .../com/ethran/notable/data/AppRepository.kt | 2 +- .../notable/editor/drawing/backgrounds.kt | 85 ++++++ .../notable/navigation/NotableNavHost.kt | 11 +- .../notable/navigation/NotableNavigator.kt | 5 + .../notable/ui/components/DebugSettings.kt | 10 +- .../ethran/notable/ui/views/InkTestView.kt | 273 ++++++++++++++++++ .../com/ethran/notable/ui/views/Settings.kt | 5 +- docs/inbox-capture.md | 109 +++++++ gradle.properties | 1 + 11 files changed, 689 insertions(+), 4 deletions(-) create mode 100644 CLAUDE.md create mode 100644 app/src/main/java/com/ethran/notable/ui/views/InkTestView.kt create mode 100644 docs/inbox-capture.md diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 00000000..a6c64d1d --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,189 @@ +# Notable - Onyx Boox Note-Taking App + +## Project Overview +Fork of [Ethran/notable](https://github.com/Ethran/notable) — an alternative note-taking app for Onyx Boox e-ink devices. Written in Kotlin with Jetpack Compose. Uses the Onyx Pen SDK for low-latency stylus input. + +## Prerequisites + +### Required tools (install via Homebrew) +```bash +brew install openjdk@17 +brew install --cask android-commandlinetools +brew install --cask android-platform-tools +``` + +### Shell config (~/.zshrc) +```bash +export JAVA_HOME=$(/usr/libexec/java_home -v 17) +export ANDROID_HOME=/opt/homebrew/share/android-commandlinetools +export PATH=$PATH:$ANDROID_HOME/cmdline-tools/latest/bin +export PATH=$PATH:$ANDROID_HOME/platform-tools +``` + +### Android SDK setup (one-time) +```bash +sdkmanager "platform-tools" "platforms;android-36" "build-tools;36.0.0" +sdkmanager --licenses +``` + +## Building + +### First-time setup +The `IS_NEXT` property must exist in `gradle.properties`. If missing, add: +``` +IS_NEXT=false +``` + +### Build commands +```bash +# Build debug APK +./gradlew assembleDebug + +# Build + install to connected Boox device +./gradlew installDebug + +# Build + install + launch in one shot +./gradlew installDebug && adb shell am start -n com.ethran.notable/.MainActivity + +# Run unit tests +./gradlew test + +# Clean +./gradlew clean +``` + +### Build output +Debug APK is at: `app/build/outputs/apk/debug/app-debug.apk` + +## Deploying to Onyx Boox + +### Enable USB debugging on the Boox +1. Settings > Apps > App Management > enable "USB Debug Mode" + - OR: Settings > About Tablet > tap "Build Number" 7 times > Developer Options > USB Debugging + +### ADB commands +```bash +# Verify device is connected +adb devices + +# Install APK manually +adb install -r app/build/outputs/apk/debug/app-debug.apk + +# View app logs (filter to this app) +adb logcat --pid=$(adb shell pidof com.ethran.notable) + +# View all logs (useful when app crashes on launch) +adb logcat -d | tail -100 + +# Take a screenshot from the Boox +adb exec-out screencap -p > screenshot.png + +# Force stop the app +adb shell am force-stop com.ethran.notable + +# Uninstall +adb uninstall com.ethran.notable +``` + +## Project Structure + +``` +app/src/main/java/com/ethran/notable/ +├── MainActivity.kt # Entry point, Hilt setup, fullscreen +├── NotableApp.kt # Application class +├── data/ +│ ├── AppRepository.kt # Main data repository +│ ├── PageDataManager.kt # Page data operations +│ ├── datastore/ # App settings, editor settings cache +│ └── db/ # Room database: Db.kt, Migrations.kt +│ ├── Stroke.kt # Stroke entity (polyline-encoded points) +│ ├── Page.kt, Notebook.kt, Folder.kt, Image.kt, Kv.kt +│ └── EncodePolyline.kt # Stroke point compression +├── editor/ +│ ├── EditorControlTower.kt # Central editor logic coordinator +│ ├── EditorView.kt # Main editor composable +│ ├── PageView.kt # Page rendering +│ ├── canvas/ +│ │ ├── DrawCanvas.kt # SurfaceView-based canvas +│ │ ├── OnyxInputHandler.kt # Onyx Pen SDK integration (TouchHelper) +│ │ ├── CanvasEventBus.kt # Event system for canvas +│ │ └── CanvasRefreshManager.kt # E-ink refresh coordination +│ ├── drawing/ # Stroke rendering, pen styles, backgrounds +│ ├── state/ # EditorState, SelectionState, history (undo/redo) +│ ├── ui/ # Toolbar, menus, selection UI +│ └── utils/ # Geometry, eraser, pen, draw helpers +├── gestures/ # Gesture detection and handling +├── io/ # Import/export: PDF, XOPP, PNG, share +├── navigation/ # Compose Navigation setup +└── ui/ # App-level UI: settings, dialogs, themes +``` + +## Key Technical Details + +### Onyx Pen SDK +- Uses `TouchHelper` + `RawInputCallback` for low-latency stylus input +- SDK deps: `onyxsdk-pen:1.5.1`, `onyxsdk-device:1.3.2`, `onyxsdk-base:1.8.3` +- Maven repo: `http://repo.boox.com/repository/maven-public/` (insecure HTTP, required) +- Pen input only works on Onyx hardware — cannot test in emulator +- Key files: `OnyxInputHandler.kt`, `DrawCanvas.kt` + +### E-ink considerations +- No animations or transitions — every frame causes ghosting +- Use `EpdController` for refresh mode control (see `einkHelper.kt`) +- Batch UI updates; minimize unnecessary recompositions +- High contrast, no gradients + +### Architecture +- **Dependency injection**: Dagger Hilt (`@AndroidEntryPoint`, `@Inject`) +- **Database**: Room with KSP code generation, schema migrations in `app/schemas/` +- **UI**: Jetpack Compose (Material, not Material3) +- **Navigation**: Compose Navigation +- **Logging**: ShipBook SDK (remote logging, needs API keys or uses defaults) +- **Firebase**: Analytics only (google-services.json is committed) + +### Important properties +- `applicationId`: `com.ethran.notable` +- `minSdk`: 29 (Android 10) +- `targetSdk`: 35 +- `compileSdk`: 36 +- `JVM target`: 17 +- `Gradle`: 9.1.0 +- `Kotlin`: 2.3.10 +- `AGP`: 9.0.0 + +## Debugging Tips + +### App crashes on launch +```bash +# Get the crash stacktrace +adb logcat -d | grep -A 20 "FATAL EXCEPTION" +``` + +### Build fails +```bash +# Full error output +./gradlew assembleDebug --stacktrace + +# Dependency issues +./gradlew dependencies --configuration debugRuntimeClasspath +``` + +### Testing without a Boox device +- The Android emulator can test general UI layout and navigation +- Pen/stylus features will NOT work (Onyx SDK is device-specific) +- The emulator cannot simulate e-ink refresh behavior + +## Workflow for Claude Code + +When making changes to this project: +1. Read the relevant source files before editing +2. Build with `./gradlew assembleDebug` to check compilation +3. If a Boox device is connected, deploy with `./gradlew installDebug` +4. Check logs with `adb logcat` after deployment +5. Run `./gradlew test` for unit tests when touching data/logic code + +### Common gotchas +- The `IS_NEXT` gradle property must be set (add `IS_NEXT=false` to `gradle.properties`) +- The Onyx Maven repo uses HTTP, not HTTPS — `allowInsecureProtocol = true` is required +- Room schema changes require a migration in `Migrations.kt` and incrementing the DB version in `Db.kt` +- Compose recomposition can be expensive on e-ink — use `remember`, `derivedStateOf`, and avoid unnecessary state changes diff --git a/app/build.gradle b/app/build.gradle index 9fe65cc1..dae88d3f 100644 --- a/app/build.gradle +++ b/app/build.gradle @@ -202,6 +202,9 @@ dependencies { // for PDF support: implementation("com.artifex.mupdf:fitz:1.26.10") + // ML Kit Digital Ink Recognition + implementation 'com.google.mlkit:digital-ink-recognition:19.0.0' + // Hilt implementation "com.google.dagger:hilt-android:2.59.2" ksp "com.google.dagger:hilt-compiler:2.59.2" diff --git a/app/src/main/java/com/ethran/notable/data/AppRepository.kt b/app/src/main/java/com/ethran/notable/data/AppRepository.kt index 3b9ec99f..98d7a68c 100644 --- a/app/src/main/java/com/ethran/notable/data/AppRepository.kt +++ b/app/src/main/java/com/ethran/notable/data/AppRepository.kt @@ -128,7 +128,7 @@ class AppRepository @Inject constructor( suspend fun createNewQuickPage(parentFolderId: String? = null) : String? { val page = Page( notebookId = null, - background = GlobalAppSettings.current.defaultNativeTemplate, + background = "inbox", backgroundType = BackgroundType.Native.key, parentFolderId = parentFolderId ) diff --git a/app/src/main/java/com/ethran/notable/editor/drawing/backgrounds.kt b/app/src/main/java/com/ethran/notable/editor/drawing/backgrounds.kt index 5ab4b688..e0906d35 100644 --- a/app/src/main/java/com/ethran/notable/editor/drawing/backgrounds.kt +++ b/app/src/main/java/com/ethran/notable/editor/drawing/backgrounds.kt @@ -185,6 +185,90 @@ fun drawHexagon(canvas: Canvas, centerX: Float, centerY: Float, r: Float) { canvas.drawPath(path, defaultPaintStroke) } +// Inbox capture template zones (in page coordinates, before scroll) +// Note: toolbar occupies ~80px at top of screen +const val INBOX_CREATED_Y = 80f +const val INBOX_TAGS_LABEL_Y = 170f +const val INBOX_TAGS_ZONE_BOTTOM = 350f +const val INBOX_DIVIDER_Y = 370f +const val INBOX_CONTENT_START_Y = 400f +const val INBOX_LEFT_MARGIN = 40f +const val INBOX_LABEL_TEXT_SIZE = 40f + +private val inboxLabelPaint = Paint().apply { + color = Color.DKGRAY + textSize = INBOX_LABEL_TEXT_SIZE + isAntiAlias = true + typeface = android.graphics.Typeface.create("sans-serif-light", android.graphics.Typeface.NORMAL) +} + +private val inboxValuePaint = Paint().apply { + color = Color.BLACK + textSize = INBOX_LABEL_TEXT_SIZE + isAntiAlias = true + typeface = android.graphics.Typeface.create("sans-serif", android.graphics.Typeface.NORMAL) +} + +private val inboxDividerPaint = Paint().apply { + color = Color.DKGRAY + strokeWidth = 2f + isAntiAlias = true +} + +private val inboxZonePaint = Paint().apply { + color = Color.argb(12, 0, 0, 0) + style = Paint.Style.FILL +} + +fun drawInboxBg(canvas: Canvas, scroll: Offset, scale: Float) { + val width = (canvas.width / scale) + val canvasHeight = (canvas.height / scale) + + // White background + canvas.drawColor(Color.WHITE) + + val scrollY = scroll.y + + // Frontmatter zone background (light gray tint) + val zoneTop = -scrollY + val zoneBottom = INBOX_DIVIDER_Y - scrollY + if (zoneBottom > 0 && zoneTop < canvasHeight) { + canvas.drawRect(0f, maxOf(0f, zoneTop), width, minOf(canvasHeight, zoneBottom), inboxZonePaint) + } + + // "created:" label + date + val createdY = INBOX_CREATED_Y - scrollY + INBOX_LABEL_TEXT_SIZE + if (createdY > -INBOX_LABEL_TEXT_SIZE && createdY < canvasHeight) { + canvas.drawText("created:", INBOX_LEFT_MARGIN, createdY, inboxLabelPaint) + val dateStr = java.text.SimpleDateFormat("yyyy-MM-dd", java.util.Locale.US) + .format(java.util.Date()) + val labelWidth = inboxLabelPaint.measureText("created: ") + canvas.drawText(dateStr, INBOX_LEFT_MARGIN + labelWidth, createdY, inboxValuePaint) + } + + // "tags:" label — user handwrites tags to the right of this + val tagsY = INBOX_TAGS_LABEL_Y - scrollY + INBOX_LABEL_TEXT_SIZE + if (tagsY > -INBOX_LABEL_TEXT_SIZE && tagsY < canvasHeight) { + canvas.drawText("tags:", INBOX_LEFT_MARGIN, tagsY, inboxLabelPaint) + } + + // Divider line (thicker, full width) + val dividerY = INBOX_DIVIDER_Y - scrollY + if (dividerY > 0 && dividerY < canvasHeight) { + canvas.drawLine(0f, dividerY, width, dividerY, inboxDividerPaint) + } + + // Lined content area below the divider + val firstContentLine = INBOX_CONTENT_START_Y + lineHeight + var lineY = firstContentLine - scrollY + while (lineY < canvasHeight) { + if (lineY > 0) { + canvas.drawLine(INBOX_LEFT_MARGIN, lineY, width - INBOX_LEFT_MARGIN, lineY, defaultPaint) + } + lineY += lineHeight + } +} + fun drawBackgroundImages( context: Context, canvas: Canvas, @@ -392,6 +476,7 @@ fun drawBg( "lined" -> drawLinedBg(canvas, scroll, scale) "squared" -> drawSquaredBg(canvas, scroll, scale) "hexed" -> drawHexedBg(canvas, scroll, scale) + "inbox" -> drawInboxBg(canvas, scroll, scale) else -> { throw IllegalArgumentException("Unknown background type: $background") } diff --git a/app/src/main/java/com/ethran/notable/navigation/NotableNavHost.kt b/app/src/main/java/com/ethran/notable/navigation/NotableNavHost.kt index 5f955df1..a86a754c 100644 --- a/app/src/main/java/com/ethran/notable/navigation/NotableNavHost.kt +++ b/app/src/main/java/com/ethran/notable/navigation/NotableNavHost.kt @@ -22,6 +22,8 @@ import com.ethran.notable.ui.views.Library import com.ethran.notable.ui.views.LibraryDestination import com.ethran.notable.ui.views.PagesDestination import com.ethran.notable.ui.views.PagesView +import com.ethran.notable.ui.views.InkTestDestination +import com.ethran.notable.ui.views.InkTestView import com.ethran.notable.ui.views.SettingsDestination import com.ethran.notable.ui.views.SettingsView import com.ethran.notable.ui.views.SystemInformationDestination @@ -133,7 +135,8 @@ fun NotableNavHost( SettingsView( onBack = { appNavigator.goBack() }, goToWelcome = { appNavigator.goToWelcome() }, - goToSystemInfo = { appNavigator.goToSystemInfo() } + goToSystemInfo = { appNavigator.goToSystemInfo() }, + goToInkTest = { appNavigator.goToInkTest() } ) appNavigator.cleanCurrentPageId() } @@ -143,6 +146,12 @@ fun NotableNavHost( BugReportScreen(goBack = { appNavigator.goBack() }) appNavigator.cleanCurrentPageId() } + composable( + route = InkTestDestination.route, + ) { + InkTestView(onBack = { appNavigator.goBack() }) + appNavigator.cleanCurrentPageId() + } } } } \ No newline at end of file diff --git a/app/src/main/java/com/ethran/notable/navigation/NotableNavigator.kt b/app/src/main/java/com/ethran/notable/navigation/NotableNavigator.kt index 133de57b..a629c756 100644 --- a/app/src/main/java/com/ethran/notable/navigation/NotableNavigator.kt +++ b/app/src/main/java/com/ethran/notable/navigation/NotableNavigator.kt @@ -18,6 +18,7 @@ import com.ethran.notable.editor.canvas.CanvasEventBus import com.ethran.notable.editor.utils.refreshScreen import com.ethran.notable.ui.views.LibraryDestination import com.ethran.notable.ui.views.SystemInformationDestination +import com.ethran.notable.ui.views.InkTestDestination import com.ethran.notable.ui.views.WelcomeDestination import com.ethran.notable.utils.hasFilePermission import io.shipbook.shipbooksdk.ShipBook @@ -127,6 +128,10 @@ class NotableNavigator( navController.navigate(SystemInformationDestination.route) } + fun goToInkTest() { + navController.navigate(InkTestDestination.route) + } + fun goToPage(appRepository: AppRepository, pageId: String) { coroutineScope.launch { val bookId = runCatching { diff --git a/app/src/main/java/com/ethran/notable/ui/components/DebugSettings.kt b/app/src/main/java/com/ethran/notable/ui/components/DebugSettings.kt index 394a2739..6a032bf5 100644 --- a/app/src/main/java/com/ethran/notable/ui/components/DebugSettings.kt +++ b/app/src/main/java/com/ethran/notable/ui/components/DebugSettings.kt @@ -9,7 +9,8 @@ fun DebugSettings( settings: AppSettings, onSettingsChange: (AppSettings) -> Unit, goToWelcome: () -> Unit, - goToSystemInfo: () -> Unit + goToSystemInfo: () -> Unit, + goToInkTest: () -> Unit = {} ) { Column { SettingToggleRow( @@ -61,5 +62,12 @@ fun DebugSettings( onSettingsChange(settings.copy(destructiveMigrations = isChecked)) } ) + SettingToggleRow( + label = "Ink Recognition Test", + value = false, + onToggle = { + goToInkTest() + } + ) } } diff --git a/app/src/main/java/com/ethran/notable/ui/views/InkTestView.kt b/app/src/main/java/com/ethran/notable/ui/views/InkTestView.kt new file mode 100644 index 00000000..f72ba4a7 --- /dev/null +++ b/app/src/main/java/com/ethran/notable/ui/views/InkTestView.kt @@ -0,0 +1,273 @@ +package com.ethran.notable.ui.views + +import android.graphics.Path +import android.view.MotionEvent +import androidx.compose.foundation.Canvas +import androidx.compose.foundation.background +import androidx.compose.foundation.border +import androidx.compose.foundation.clickable +import androidx.compose.foundation.layout.Arrangement +import androidx.compose.foundation.layout.Box +import androidx.compose.foundation.layout.Column +import androidx.compose.foundation.layout.Row +import androidx.compose.foundation.layout.Spacer +import androidx.compose.foundation.layout.fillMaxSize +import androidx.compose.foundation.layout.fillMaxWidth +import androidx.compose.foundation.layout.height +import androidx.compose.foundation.layout.padding +import androidx.compose.foundation.rememberScrollState +import androidx.compose.foundation.verticalScroll +import androidx.compose.material.Text +import androidx.compose.runtime.Composable +import androidx.compose.runtime.DisposableEffect +import androidx.compose.runtime.getValue +import androidx.compose.runtime.mutableLongStateOf +import androidx.compose.runtime.mutableStateListOf +import androidx.compose.runtime.mutableStateOf +import androidx.compose.runtime.remember +import androidx.compose.runtime.setValue +import androidx.compose.ui.Alignment +import androidx.compose.ui.ExperimentalComposeUiApi +import androidx.compose.ui.Modifier +import androidx.compose.ui.graphics.Color +import androidx.compose.ui.graphics.nativeCanvas +import androidx.compose.ui.input.pointer.pointerInteropFilter +import androidx.compose.ui.unit.dp +import androidx.compose.ui.unit.sp +import com.google.mlkit.common.model.DownloadConditions +import com.google.mlkit.common.model.RemoteModelManager +import com.google.mlkit.vision.digitalink.recognition.DigitalInkRecognition +import com.google.mlkit.vision.digitalink.recognition.DigitalInkRecognitionModel +import com.google.mlkit.vision.digitalink.recognition.DigitalInkRecognitionModelIdentifier +import com.google.mlkit.vision.digitalink.recognition.DigitalInkRecognizerOptions +import com.google.mlkit.vision.digitalink.recognition.Ink +import com.ethran.notable.navigation.NavigationDestination + +object InkTestDestination : NavigationDestination { + override val route = "ink_test" +} + +@OptIn(ExperimentalComposeUiApi::class) +@Composable +fun InkTestView(onBack: () -> Unit) { + val inkBuilder = remember { Ink.builder() } + var currentStrokeBuilder by remember { mutableStateOf(null) } + val paths = remember { mutableStateListOf() } + var currentPath by remember { mutableStateOf(null) } + + var recognizedText by remember { mutableStateOf("(write something and tap Recognize)") } + var modelStatus by remember { mutableStateOf("Checking model...") } + var recognitionTimeMs by remember { mutableLongStateOf(0L) } + + val modelIdentifier = remember { + DigitalInkRecognitionModelIdentifier.fromLanguageTag("en-US") + } + val model = remember { + modelIdentifier?.let { DigitalInkRecognitionModel.builder(it).build() } + } + val recognizer = remember { + model?.let { + DigitalInkRecognition.getClient( + DigitalInkRecognizerOptions.builder(it).build() + ) + } + } + + // Download model on first load + DisposableEffect(model) { + if (model != null) { + val remoteModelManager = RemoteModelManager.getInstance() + remoteModelManager.isModelDownloaded(model).addOnSuccessListener { isDownloaded -> + if (isDownloaded) { + modelStatus = "Model ready" + } else { + modelStatus = "Downloading model..." + remoteModelManager.download(model, DownloadConditions.Builder().build()) + .addOnSuccessListener { modelStatus = "Model ready" } + .addOnFailureListener { modelStatus = "Download failed: ${it.message}" } + } + } + } else { + modelStatus = "Model identifier not found" + } + onDispose { } + } + + Column( + modifier = Modifier + .fillMaxSize() + .background(Color.White) + .padding(16.dp) + ) { + // Header + Row( + modifier = Modifier.fillMaxWidth(), + horizontalArrangement = Arrangement.SpaceBetween, + verticalAlignment = Alignment.CenterVertically + ) { + Text("ML Kit Ink Test", fontSize = 20.sp, color = Color.Black) + Text( + "< Back", + fontSize = 16.sp, + color = Color.Black, + modifier = Modifier.clickable { onBack() } + ) + } + + Text(modelStatus, fontSize = 12.sp, color = Color.Gray) + Spacer(modifier = Modifier.height(8.dp)) + + // Drawing area + Box( + modifier = Modifier + .fillMaxWidth() + .height(300.dp) + .border(1.dp, Color.Black) + .background(Color.White) + ) { + Canvas( + modifier = Modifier + .fillMaxSize() + .pointerInteropFilter { event -> + when (event.action) { + MotionEvent.ACTION_DOWN -> { + val strokeBuilder = Ink.Stroke.builder() + strokeBuilder.addPoint( + Ink.Point.create( + event.x, + event.y, + event.eventTime + ) + ) + currentStrokeBuilder = strokeBuilder + val path = Path() + path.moveTo(event.x, event.y) + currentPath = path + true + } + + MotionEvent.ACTION_MOVE -> { + currentStrokeBuilder?.addPoint( + Ink.Point.create( + event.x, + event.y, + event.eventTime + ) + ) + currentPath?.lineTo(event.x, event.y) + // Force recomposition + val p = currentPath + currentPath = null + currentPath = p + true + } + + MotionEvent.ACTION_UP -> { + currentStrokeBuilder?.addPoint( + Ink.Point.create( + event.x, + event.y, + event.eventTime + ) + ) + currentStrokeBuilder?.let { inkBuilder.addStroke(it.build()) } + currentStrokeBuilder = null + currentPath?.let { paths.add(it) } + currentPath = null + true + } + + else -> false + } + } + ) { + val paint = android.graphics.Paint().apply { + color = android.graphics.Color.BLACK + style = android.graphics.Paint.Style.STROKE + strokeWidth = 4f + isAntiAlias = true + } + paths.forEach { path -> + drawContext.canvas.nativeCanvas.drawPath(path, paint) + } + currentPath?.let { path -> + drawContext.canvas.nativeCanvas.drawPath(path, paint) + } + } + } + + Spacer(modifier = Modifier.height(12.dp)) + + // Buttons + Row( + modifier = Modifier.fillMaxWidth(), + horizontalArrangement = Arrangement.spacedBy(12.dp) + ) { + Box( + modifier = Modifier + .border(1.dp, Color.Black) + .clickable { + if (recognizer != null && modelStatus == "Model ready") { + recognizedText = "Recognizing..." + val ink = inkBuilder.build() + val start = System.currentTimeMillis() + recognizer.recognize(ink) + .addOnSuccessListener { result -> + val elapsed = System.currentTimeMillis() - start + recognitionTimeMs = elapsed + val candidates = result.candidates + recognizedText = if (candidates.isNotEmpty()) { + candidates.joinToString("\n") { candidate -> "\"${candidate.text}\"" } + } else { + "(no results)" + } + } + .addOnFailureListener { ex -> + recognizedText = "Error: ${ex.localizedMessage}" + } + } + } + .padding(horizontal = 16.dp, vertical = 8.dp) + ) { + Text("Recognize", fontSize = 16.sp, color = Color.Black) + } + + Box( + modifier = Modifier + .border(1.dp, Color.Black) + .clickable { + paths.clear() + currentPath = null + currentStrokeBuilder = null + // Reset ink builder by creating a fresh one — we'll + // just rebuild in the recognizer call + recognizedText = "(cleared)" + } + .padding(horizontal = 16.dp, vertical = 8.dp) + ) { + Text("Clear", fontSize = 16.sp, color = Color.Black) + } + } + + Spacer(modifier = Modifier.height(12.dp)) + + if (recognitionTimeMs > 0) { + Text("Recognition time: ${recognitionTimeMs}ms", fontSize = 12.sp, color = Color.Gray) + } + + Spacer(modifier = Modifier.height(4.dp)) + + // Results + Text("Results:", fontSize = 14.sp, color = Color.Black) + Box( + modifier = Modifier + .fillMaxWidth() + .weight(1f) + .border(1.dp, Color.LightGray) + .padding(8.dp) + .verticalScroll(rememberScrollState()) + ) { + Text(recognizedText, fontSize = 16.sp, color = Color.Black) + } + } +} diff --git a/app/src/main/java/com/ethran/notable/ui/views/Settings.kt b/app/src/main/java/com/ethran/notable/ui/views/Settings.kt index 2c9a99ee..5fa51b64 100644 --- a/app/src/main/java/com/ethran/notable/ui/views/Settings.kt +++ b/app/src/main/java/com/ethran/notable/ui/views/Settings.kt @@ -77,6 +77,7 @@ fun SettingsView( onBack: () -> Unit, goToWelcome: () -> Unit, goToSystemInfo: () -> Unit, + goToInkTest: () -> Unit = {}, viewModel: SettingsViewModel = hiltViewModel() ) { val context = LocalContext.current @@ -97,6 +98,7 @@ fun SettingsView( onBack = onBack, goToWelcome = goToWelcome, goToSystemInfo = goToSystemInfo, + goToInkTest = goToInkTest, onCheckUpdate = { force -> viewModel.checkUpdate(context, force) }, @@ -114,6 +116,7 @@ fun SettingsContent( onBack: () -> Unit, goToWelcome: () -> Unit, goToSystemInfo: () -> Unit, + goToInkTest: () -> Unit = {}, onCheckUpdate: (Boolean) -> Unit, onUpdateSettings: (AppSettings) -> Unit, selectedTabInitial: Int = 0, @@ -153,7 +156,7 @@ fun SettingsContent( settings, onUpdateSettings, listOfGestures, availableGestures ) - 2 -> DebugSettings(settings, onUpdateSettings, goToWelcome, goToSystemInfo) + 2 -> DebugSettings(settings, onUpdateSettings, goToWelcome, goToSystemInfo, goToInkTest) } } diff --git a/docs/inbox-capture.md b/docs/inbox-capture.md new file mode 100644 index 00000000..e02ba5f3 --- /dev/null +++ b/docs/inbox-capture.md @@ -0,0 +1,109 @@ +# Inbox Capture + +Inbox Capture replaces Quick Pages with a templated note-taking surface that syncs handwritten notes to an Obsidian vault as markdown files. + +## Concept + +The user creates an Inbox Capture page, writes tags in the frontmatter zone and note content below the divider. On page exit, the handwriting is recognized via ML Kit Digital Ink Recognition and written as a markdown file to the configured Obsidian inbox folder. + +## Template Layout + +The inbox page has three zones, rendered as a native background type (`"inbox"`): + +``` ++------------------------------------------+ +| created: 2026-03-08 (auto-filled)| +| tags: [handwrite tags here] | +| ~~~~~~~~~ gray background ~~~~~~~~~~~~ | ++==========================================+ <-- divider (Y=370) +| | +| [handwrite note content here] | +| ______________________________________ | +| ______________________________________ | <-- lined area +| ______________________________________ | ++------------------------------------------+ +``` + +### Zone constants (page coordinates) + +| Constant | Value | Purpose | +|---|---|---| +| `INBOX_CREATED_Y` | 80 | Y position of "created:" label | +| `INBOX_TAGS_LABEL_Y` | 170 | Y position of "tags:" label | +| `INBOX_DIVIDER_Y` | 370 | Y position of the divider line | +| `INBOX_CONTENT_START_Y` | 400 | Where content lines begin | + +### Stroke classification + +Strokes are classified by their bounding box position relative to the divider: +- **Tags zone**: strokes with `top < INBOX_DIVIDER_Y` +- **Content zone**: strokes with `top >= INBOX_CONTENT_START_Y` + +## On-Exit Sync Flow + +When the user exits an inbox capture page (navigates back): + +1. **Collect strokes** from the page database +2. **Classify strokes** into tags zone vs content zone by Y position +3. **Recognize handwriting** using ML Kit Digital Ink Recognition: + - Tags zone strokes -> recognized as tag text + - Content zone strokes -> recognized as note body text +4. **Generate markdown** with YAML frontmatter: + ```markdown + --- + created: 2026-03-08 + tags: + - relationships + - pkm + --- + + some content here + ``` +5. **Write the file** to the Obsidian inbox folder: + - Default path: `Documents/primary/inbox/` + - Filename: `YYYY-MM-DD-HH-mm-ss.md` + - Configurable via folder picker in Notable settings +6. **Delete the Notable page** after successful sync + +## Multi-Page Support + +An inbox capture can span multiple pages (via notebook). All pages are processed in order, with content strokes from each page appended to the markdown body. + +## Technical Dependencies + +- **ML Kit Digital Ink Recognition** (`com.google.mlkit:digital-ink-recognition:19.0.0`) + - Package: `com.google.mlkit.vision.digitalink.recognition` + - ~20MB model download (one-time, on-device, offline after download) + - ~100ms recognition per line of text on NoteAir5C +- **Onyx Pen SDK** for stroke capture (existing) +- **Android file system** for writing to Documents folder (existing MANAGE_EXTERNAL_STORAGE permission) + +## Settings + +- **Obsidian vault path**: folder picker, defaults to `Documents/primary/inbox/` +- Stored in `AppSettings` / `GlobalAppSettings` + +## Files Modified/Created + +### New files +- `app/src/main/java/com/ethran/notable/ui/views/InkTestView.kt` — ML Kit test screen (debug) + +### Modified files +- `app/build.gradle` — added ML Kit dependency +- `app/src/main/java/com/ethran/notable/editor/drawing/backgrounds.kt` — added `drawInboxBg()` and inbox zone constants +- `app/src/main/java/com/ethran/notable/data/AppRepository.kt` — quick page creation uses `"inbox"` background +- `app/src/main/java/com/ethran/notable/data/model/BackgroundType.kt` — (no changes yet, uses existing Native type) +- `app/src/main/java/com/ethran/notable/navigation/NotableNavHost.kt` — added ink test route +- `app/src/main/java/com/ethran/notable/navigation/NotableNavigator.kt` — added ink test navigation +- `app/src/main/java/com/ethran/notable/ui/views/Settings.kt` — threaded ink test button +- `app/src/main/java/com/ethran/notable/ui/components/DebugSettings.kt` — added ink test toggle + +### TODO (next steps) +- [ ] Implement on-exit sync flow in EditorView.kt +- [ ] Add stroke classification by zone (tags vs content) +- [ ] Integrate ML Kit recognition with existing stroke data (convert Onyx TouchPoints to ML Kit Ink) +- [ ] Generate markdown with YAML frontmatter +- [ ] Write markdown file to Obsidian inbox folder +- [ ] Add Obsidian vault folder picker in settings +- [ ] Delete Notable page after successful sync +- [ ] Handle multi-page inbox captures diff --git a/gradle.properties b/gradle.properties index 385c564d..697ed4ca 100644 --- a/gradle.properties +++ b/gradle.properties @@ -29,6 +29,7 @@ android.enableAppCompileTimeRClass=false android.usesSdkInManifest.disallowed=false android.uniquePackageNames=false android.dependency.useConstraints=true +IS_NEXT=false android.r8.strictFullModeForKeepRules=false android.r8.optimizedResourceShrinking=false android.builtInKotlin=false From 2facb6cdcc060240ca5347c4c4d7939b3d08c1e0 Mon Sep 17 00:00:00 2001 From: Joshua Pham Date: Sun, 8 Mar 2026 22:22:04 -0400 Subject: [PATCH 02/23] Add on-exit sync: recognize inbox handwriting and write to Obsidian vault When leaving an inbox capture page, strokes are classified by zone (tags vs content), recognized via ML Kit Digital Ink, and written as a markdown file with YAML frontmatter to the Obsidian inbox folder. The Notable page is deleted after successful sync. Co-Authored-By: Claude Opus 4.6 --- .../notable/data/datastore/AppSettings.kt | 3 + .../com/ethran/notable/editor/EditorView.kt | 19 ++ .../com/ethran/notable/io/InboxSyncEngine.kt | 182 ++++++++++++++++++ 3 files changed, 204 insertions(+) create mode 100644 app/src/main/java/com/ethran/notable/io/InboxSyncEngine.kt diff --git a/app/src/main/java/com/ethran/notable/data/datastore/AppSettings.kt b/app/src/main/java/com/ethran/notable/data/datastore/AppSettings.kt index 29851b96..c65b5cf6 100644 --- a/app/src/main/java/com/ethran/notable/data/datastore/AppSettings.kt +++ b/app/src/main/java/com/ethran/notable/data/datastore/AppSettings.kt @@ -49,6 +49,9 @@ data class AppSettings( val enableQuickNav: Boolean = true, + // Inbox Capture + val obsidianInboxPath: String = "Documents/primary/inbox", + // Debug val showWelcome: Boolean = true, // [system information -- does not have a setting] diff --git a/app/src/main/java/com/ethran/notable/editor/EditorView.kt b/app/src/main/java/com/ethran/notable/editor/EditorView.kt index 4b09bb0d..5738f2a1 100644 --- a/app/src/main/java/com/ethran/notable/editor/EditorView.kt +++ b/app/src/main/java/com/ethran/notable/editor/EditorView.kt @@ -30,6 +30,7 @@ import com.ethran.notable.editor.ui.SelectedBitmap import com.ethran.notable.editor.ui.toolbar.Toolbar import com.ethran.notable.gestures.EditorGestureReceiver import com.ethran.notable.io.ExportEngine +import com.ethran.notable.io.InboxSyncEngine import com.ethran.notable.io.exportToLinkedFile import com.ethran.notable.navigation.NavigationDestination import com.ethran.notable.ui.LocalSnackContext @@ -39,6 +40,7 @@ import com.ethran.notable.ui.convertDpToPixel import com.ethran.notable.ui.theme.InkaTheme import com.ethran.notable.ui.views.LibraryDestination import io.shipbook.shipbooksdk.ShipBook +import kotlinx.coroutines.CoroutineScope import kotlinx.coroutines.Dispatchers import kotlinx.coroutines.launch import kotlinx.coroutines.withContext @@ -151,6 +153,23 @@ fun EditorView( if (bookId != null) exportToLinkedFile(exportEngine, bookId, appRepository.bookRepository) page.disposeOldPage() + + // Inbox capture: sync handwriting to Obsidian markdown on exit + if (page.pageFromDb?.background == "inbox") { + CoroutineScope(Dispatchers.IO).launch { + try { + InboxSyncEngine.syncInboxPage(appRepository, pageId) + } catch (e: Exception) { + log.e("Inbox sync failed: ${e.message}", e) + SnackState.globalSnackFlow.tryEmit( + SnackConf( + text = "Inbox sync failed: ${e.message}", + duration = 4000 + ) + ) + } + } + } } } diff --git a/app/src/main/java/com/ethran/notable/io/InboxSyncEngine.kt b/app/src/main/java/com/ethran/notable/io/InboxSyncEngine.kt new file mode 100644 index 00000000..842941b6 --- /dev/null +++ b/app/src/main/java/com/ethran/notable/io/InboxSyncEngine.kt @@ -0,0 +1,182 @@ +package com.ethran.notable.io + +import android.os.Environment +import com.ethran.notable.data.AppRepository +import com.ethran.notable.data.datastore.GlobalAppSettings +import com.ethran.notable.data.db.Stroke +import com.ethran.notable.editor.drawing.INBOX_CONTENT_START_Y +import com.ethran.notable.editor.drawing.INBOX_DIVIDER_Y +import com.google.mlkit.common.model.DownloadConditions +import com.google.mlkit.common.model.RemoteModelManager +import com.google.mlkit.vision.digitalink.recognition.DigitalInkRecognition +import com.google.mlkit.vision.digitalink.recognition.DigitalInkRecognitionModel +import com.google.mlkit.vision.digitalink.recognition.DigitalInkRecognitionModelIdentifier +import com.google.mlkit.vision.digitalink.recognition.DigitalInkRecognizerOptions +import com.google.mlkit.vision.digitalink.recognition.Ink +import io.shipbook.shipbooksdk.ShipBook +import kotlinx.coroutines.suspendCancellableCoroutine +import java.io.File +import java.text.SimpleDateFormat +import java.util.Date +import java.util.Locale +import kotlin.coroutines.resume +import kotlin.coroutines.resumeWithException + +private val log = ShipBook.getLogger("InboxSyncEngine") + +object InboxSyncEngine { + + private val modelIdentifier = + DigitalInkRecognitionModelIdentifier.fromLanguageTag("en-US") + private val model = + modelIdentifier?.let { DigitalInkRecognitionModel.builder(it).build() } + private val recognizer = model?.let { + DigitalInkRecognition.getClient( + DigitalInkRecognizerOptions.builder(it).build() + ) + } + + suspend fun syncInboxPage(appRepository: AppRepository, pageId: String) { + log.i("Starting inbox sync for page $pageId") + + val page = appRepository.pageRepository.getById(pageId) + if (page == null) { + log.e("Page $pageId not found") + return + } + + val pageWithStrokes = appRepository.pageRepository.getWithStrokeById(pageId) + val strokes = pageWithStrokes.strokes + + if (strokes.isEmpty()) { + log.i("No strokes on inbox page, deleting") + appRepository.pageRepository.delete(pageId) + return + } + + ensureModelDownloaded() + + // Classify strokes by zone using bounding box position + val tagStrokes = strokes.filter { it.top < INBOX_DIVIDER_Y } + val contentStrokes = strokes.filter { it.top >= INBOX_CONTENT_START_Y } + + log.i("Classified ${tagStrokes.size} tag strokes, ${contentStrokes.size} content strokes") + + val tagText = if (tagStrokes.isNotEmpty()) recognizeStrokes(tagStrokes) else "" + val contentText = if (contentStrokes.isNotEmpty()) recognizeStrokes(contentStrokes) else "" + + log.i("Recognized tags: '$tagText', content: '${contentText.take(100)}'") + + val tags = parseTags(tagText) + val createdDate = SimpleDateFormat("yyyy-MM-dd", Locale.US).format(page.createdAt) + val markdown = generateMarkdown(createdDate, tags, contentText) + + val inboxPath = GlobalAppSettings.current.obsidianInboxPath + writeMarkdownFile(markdown, page.createdAt, inboxPath) + + appRepository.pageRepository.delete(pageId) + log.i("Inbox sync complete, page $pageId deleted") + } + + private suspend fun ensureModelDownloaded() { + val m = model ?: throw IllegalStateException("ML Kit model identifier not found") + val manager = RemoteModelManager.getInstance() + + val isDownloaded = suspendCancellableCoroutine { cont -> + manager.isModelDownloaded(m) + .addOnSuccessListener { cont.resume(it) } + .addOnFailureListener { cont.resumeWithException(it) } + } + + if (!isDownloaded) { + log.i("Downloading ML Kit model...") + suspendCancellableCoroutine { cont -> + manager.download(m, DownloadConditions.Builder().build()) + .addOnSuccessListener { cont.resume(null) } + .addOnFailureListener { cont.resumeWithException(it) } + } + log.i("Model downloaded") + } + } + + private suspend fun recognizeStrokes(strokes: List): String { + val rec = recognizer + ?: throw IllegalStateException("ML Kit recognizer not initialized") + + val inkBuilder = Ink.builder() + + // Sort strokes by creation time, then by position for deterministic order + val sortedStrokes = strokes.sortedWith( + compareBy { it.createdAt.time }.thenBy { it.top }.thenBy { it.left } + ) + + // Use actual stroke creation timestamps as base, with synthetic + // per-point timing (~10ms between points, simulating natural writing) + for (stroke in sortedStrokes) { + val strokeBuilder = Ink.Stroke.builder() + val baseTime = stroke.createdAt.time + for ((i, point) in stroke.points.withIndex()) { + val t = if (point.dt != null) { + baseTime + point.dt.toLong() + } else { + baseTime + (i * 10L) // 10ms between points + } + strokeBuilder.addPoint(Ink.Point.create(point.x, point.y, t)) + } + inkBuilder.addStroke(strokeBuilder.build()) + } + + val ink = inkBuilder.build() + return suspendCancellableCoroutine { cont -> + rec.recognize(ink) + .addOnSuccessListener { result -> + val text = result.candidates.firstOrNull()?.text ?: "" + cont.resume(text) + } + .addOnFailureListener { cont.resumeWithException(it) } + } + } + + private fun parseTags(rawText: String): List { + return rawText + .replace("#", "") + .replace(",", " ") + .split("\\s+".toRegex()) + .map { it.trim().lowercase() } + .filter { it.isNotEmpty() } + } + + private fun generateMarkdown( + createdDate: String, + tags: List, + content: String + ): String { + val sb = StringBuilder() + sb.appendLine("---") + sb.appendLine("created: $createdDate") + if (tags.isNotEmpty()) { + sb.appendLine("tags:") + tags.forEach { sb.appendLine(" - $it") } + } + sb.appendLine("---") + sb.appendLine() + sb.appendLine(content.trim()) + return sb.toString() + } + + private fun writeMarkdownFile(markdown: String, createdAt: Date, inboxPath: String) { + val timestamp = SimpleDateFormat("yyyy-MM-dd-HH-mm-ss", Locale.US).format(createdAt) + val fileName = "$timestamp.md" + + val dir = if (inboxPath.startsWith("/")) { + File(inboxPath) + } else { + File(Environment.getExternalStorageDirectory(), inboxPath) + } + + dir.mkdirs() + val file = File(dir, fileName) + file.writeText(markdown) + log.i("Written inbox note to ${file.absolutePath}") + } +} From 07748bc2512bc474b744306afb3af00dbb0038f9 Mon Sep 17 00:00:00 2001 From: Joshua Pham Date: Sun, 8 Mar 2026 22:56:37 -0400 Subject: [PATCH 03/23] Add inbox capture UI: tag selection, vault scanning, settings, and explicit save MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Replace the handwritten frontmatter zones with a Compose UI toolbar at top: tag pills from vault (ranked by frequency×recency), text input with autocomplete filtering, and explicit Save & Exit / Back buttons. Pen toolbar moves to the bottom on inbox pages. - VaultTagScanner parses tags from inbox folder markdown frontmatter - Tags are cached on app start for instant loading - InboxSyncEngine now accepts tags from UI instead of stroke recognition - Frontmatter uses Obsidian format: created: "[[YYYY-MM-DD]]" - Settings page shows Inbox Capture section with configurable folder path - Double-sync guard prevents duplicate saves Co-Authored-By: Claude Opus 4.6 --- .../java/com/ethran/notable/MainActivity.kt | 4 + .../com/ethran/notable/editor/EditorView.kt | 87 ++++-- .../notable/editor/canvas/OnyxInputHandler.kt | 7 +- .../notable/editor/drawing/backgrounds.kt | 50 +--- .../notable/editor/state/EditorState.kt | 2 + .../ethran/notable/editor/ui/InboxToolbar.kt | 264 ++++++++++++++++++ .../com/ethran/notable/io/InboxSyncEngine.kt | 47 ++-- .../com/ethran/notable/io/VaultTagScanner.kt | 127 +++++++++ .../notable/ui/components/GeneralSettings.kt | 106 +++++++ 9 files changed, 608 insertions(+), 86 deletions(-) create mode 100644 app/src/main/java/com/ethran/notable/editor/ui/InboxToolbar.kt create mode 100644 app/src/main/java/com/ethran/notable/io/VaultTagScanner.kt diff --git a/app/src/main/java/com/ethran/notable/MainActivity.kt b/app/src/main/java/com/ethran/notable/MainActivity.kt index fa6b6943..2011d1ea 100644 --- a/app/src/main/java/com/ethran/notable/MainActivity.kt +++ b/app/src/main/java/com/ethran/notable/MainActivity.kt @@ -34,6 +34,7 @@ import com.ethran.notable.data.db.KvProxy import com.ethran.notable.data.db.StrokeMigrationHelper import com.ethran.notable.editor.canvas.CanvasEventBus import com.ethran.notable.io.ExportEngine +import com.ethran.notable.io.VaultTagScanner import com.ethran.notable.ui.LocalSnackContext import com.ethran.notable.ui.SnackState import com.ethran.notable.ui.components.NotableApp @@ -109,6 +110,9 @@ class MainActivity : ComponentActivity() { editorSettingCacheManager.get().init() strokeMigrationHelper.get().reencodeStrokePointsToSB1() + + // Pre-populate inbox tag cache + VaultTagScanner.refreshCache(savedSettings.obsidianInboxPath) } } isInitialized = true diff --git a/app/src/main/java/com/ethran/notable/editor/EditorView.kt b/app/src/main/java/com/ethran/notable/editor/EditorView.kt index 5738f2a1..a04b90e5 100644 --- a/app/src/main/java/com/ethran/notable/editor/EditorView.kt +++ b/app/src/main/java/com/ethran/notable/editor/EditorView.kt @@ -10,6 +10,7 @@ import androidx.compose.runtime.Composable import androidx.compose.runtime.DisposableEffect import androidx.compose.runtime.LaunchedEffect import androidx.compose.runtime.getValue +import androidx.compose.runtime.mutableStateListOf import androidx.compose.runtime.mutableStateOf import androidx.compose.runtime.remember import androidx.compose.runtime.rememberCoroutineScope @@ -25,12 +26,14 @@ import com.ethran.notable.editor.state.EditorState import com.ethran.notable.editor.state.History import com.ethran.notable.editor.ui.EditorSurface import com.ethran.notable.editor.ui.HorizontalScrollIndicator +import com.ethran.notable.editor.ui.InboxToolbar import com.ethran.notable.editor.ui.ScrollIndicator import com.ethran.notable.editor.ui.SelectedBitmap import com.ethran.notable.editor.ui.toolbar.Toolbar import com.ethran.notable.gestures.EditorGestureReceiver import com.ethran.notable.io.ExportEngine import com.ethran.notable.io.InboxSyncEngine +import com.ethran.notable.io.VaultTagScanner import com.ethran.notable.io.exportToLinkedFile import com.ethran.notable.navigation.NavigationDestination import com.ethran.notable.ui.LocalSnackContext @@ -146,6 +149,25 @@ fun EditorView( } + // Inbox mode detection — query DB since pageFromDb loads async + var isInboxPage by remember { mutableStateOf(false) } + var isSyncing by remember { mutableStateOf(false) } + val selectedTags = remember { mutableStateListOf() } + val suggestedTags = remember { mutableStateListOf() } + + LaunchedEffect(pageId) { + val pageData = withContext(Dispatchers.IO) { + appRepository.pageRepository.getById(pageId) + } + val inbox = pageData?.background == "inbox" + isInboxPage = inbox + editorState.isInboxPage = inbox + if (inbox) { + // Use pre-cached tags (scanned on app start) + suggestedTags.addAll(VaultTagScanner.cachedTags) + } + } + DisposableEffect(Unit) { onDispose { // finish selection operation @@ -153,23 +175,6 @@ fun EditorView( if (bookId != null) exportToLinkedFile(exportEngine, bookId, appRepository.bookRepository) page.disposeOldPage() - - // Inbox capture: sync handwriting to Obsidian markdown on exit - if (page.pageFromDb?.background == "inbox") { - CoroutineScope(Dispatchers.IO).launch { - try { - InboxSyncEngine.syncInboxPage(appRepository, pageId) - } catch (e: Exception) { - log.e("Inbox sync failed: ${e.message}", e) - SnackState.globalSnackFlow.tryEmit( - SnackConf( - text = "Inbox sync failed: ${e.message}", - duration = 4000 - ) - ) - } - } - } } } @@ -216,7 +221,53 @@ fun EditorView( Spacer(modifier = Modifier.weight(1f)) ScrollIndicator(state = editorState) } - PositionedToolbar(exportEngine,navController, appRepository, editorState, editorControlTower) + if (isInboxPage) { + // Inbox toolbar at top + InboxToolbar( + selectedTags = selectedTags, + suggestedTags = suggestedTags, + onTagAdd = { tag -> + if (tag !in selectedTags) selectedTags.add(tag) + }, + onTagRemove = { tag -> selectedTags.remove(tag) }, + onSave = { + if (!isSyncing) { + isSyncing = true + scope.launch(Dispatchers.IO) { + try { + InboxSyncEngine.syncInboxPage( + appRepository, pageId, selectedTags.toList() + ) + withContext(Dispatchers.Main) { + navController.popBackStack() + } + } catch (e: Exception) { + isSyncing = false + log.e("Inbox sync failed: ${e.message}", e) + SnackState.globalSnackFlow.tryEmit( + SnackConf( + text = "Inbox sync failed: ${e.message}", + duration = 4000 + ) + ) + } + } + } + }, + onDiscard = { navController.popBackStack() } + ) + // Pen toolbar at bottom for inbox pages + Column( + Modifier + .fillMaxWidth() + .fillMaxHeight() + ) { + Spacer(modifier = Modifier.weight(1f)) + Toolbar(exportEngine, navController, appRepository, editorState, editorControlTower) + } + } else { + PositionedToolbar(exportEngine, navController, appRepository, editorState, editorControlTower) + } HorizontalScrollIndicator(state = editorState) } } diff --git a/app/src/main/java/com/ethran/notable/editor/canvas/OnyxInputHandler.kt b/app/src/main/java/com/ethran/notable/editor/canvas/OnyxInputHandler.kt index c07f6d4c..1356e1f5 100644 --- a/app/src/main/java/com/ethran/notable/editor/canvas/OnyxInputHandler.kt +++ b/app/src/main/java/com/ethran/notable/editor/canvas/OnyxInputHandler.kt @@ -181,8 +181,11 @@ class OnyxInputHandler( log.i("Update editable surface") coroutineScope.launch { onSurfaceInit(drawCanvas) - val toolbarHeight = - if (state.isToolbarOpen) convertDpToPixel(40.dp, drawCanvas.context).toInt() else 0 + val toolbarHeight = when { + state.isInboxPage -> convertDpToPixel(210.dp, drawCanvas.context).toInt() + state.isToolbarOpen -> convertDpToPixel(40.dp, drawCanvas.context).toInt() + else -> 0 + } setupSurface( drawCanvas, touchHelper, diff --git a/app/src/main/java/com/ethran/notable/editor/drawing/backgrounds.kt b/app/src/main/java/com/ethran/notable/editor/drawing/backgrounds.kt index e0906d35..dc7058de 100644 --- a/app/src/main/java/com/ethran/notable/editor/drawing/backgrounds.kt +++ b/app/src/main/java/com/ethran/notable/editor/drawing/backgrounds.kt @@ -227,45 +227,19 @@ fun drawInboxBg(canvas: Canvas, scroll: Offset, scale: Float) { // White background canvas.drawColor(Color.WHITE) - val scrollY = scroll.y - - // Frontmatter zone background (light gray tint) - val zoneTop = -scrollY - val zoneBottom = INBOX_DIVIDER_Y - scrollY - if (zoneBottom > 0 && zoneTop < canvasHeight) { - canvas.drawRect(0f, maxOf(0f, zoneTop), width, minOf(canvasHeight, zoneBottom), inboxZonePaint) - } - - // "created:" label + date - val createdY = INBOX_CREATED_Y - scrollY + INBOX_LABEL_TEXT_SIZE - if (createdY > -INBOX_LABEL_TEXT_SIZE && createdY < canvasHeight) { - canvas.drawText("created:", INBOX_LEFT_MARGIN, createdY, inboxLabelPaint) - val dateStr = java.text.SimpleDateFormat("yyyy-MM-dd", java.util.Locale.US) - .format(java.util.Date()) - val labelWidth = inboxLabelPaint.measureText("created: ") - canvas.drawText(dateStr, INBOX_LEFT_MARGIN + labelWidth, createdY, inboxValuePaint) - } - - // "tags:" label — user handwrites tags to the right of this - val tagsY = INBOX_TAGS_LABEL_Y - scrollY + INBOX_LABEL_TEXT_SIZE - if (tagsY > -INBOX_LABEL_TEXT_SIZE && tagsY < canvasHeight) { - canvas.drawText("tags:", INBOX_LEFT_MARGIN, tagsY, inboxLabelPaint) - } - - // Divider line (thicker, full width) - val dividerY = INBOX_DIVIDER_Y - scrollY - if (dividerY > 0 && dividerY < canvasHeight) { - canvas.drawLine(0f, dividerY, width, dividerY, inboxDividerPaint) - } + // Lined content area — starts from the top since tags are handled by Compose UI + val offset = IntOffset(lineHeight, lineHeight) - IntOffset( + scroll.x.toInt() % lineHeight, scroll.y.toInt() % lineHeight + ) - // Lined content area below the divider - val firstContentLine = INBOX_CONTENT_START_Y + lineHeight - var lineY = firstContentLine - scrollY - while (lineY < canvasHeight) { - if (lineY > 0) { - canvas.drawLine(INBOX_LEFT_MARGIN, lineY, width - INBOX_LEFT_MARGIN, lineY, defaultPaint) - } - lineY += lineHeight + for (y in 0..(canvasHeight.toInt()) step lineHeight) { + canvas.drawLine( + INBOX_LEFT_MARGIN, + y.toFloat() + offset.y, + width - INBOX_LEFT_MARGIN, + y.toFloat() + offset.y, + defaultPaint + ) } } diff --git a/app/src/main/java/com/ethran/notable/editor/state/EditorState.kt b/app/src/main/java/com/ethran/notable/editor/state/EditorState.kt index 0abb8c1c..06aad1e8 100644 --- a/app/src/main/java/com/ethran/notable/editor/state/EditorState.kt +++ b/app/src/main/java/com/ethran/notable/editor/state/EditorState.kt @@ -103,6 +103,8 @@ class EditorState( // } // } + var isInboxPage by mutableStateOf(false) + var isToolbarOpen by mutableStateOf( persistedEditorSettings?.isToolbarOpen ?: false ) // should save diff --git a/app/src/main/java/com/ethran/notable/editor/ui/InboxToolbar.kt b/app/src/main/java/com/ethran/notable/editor/ui/InboxToolbar.kt new file mode 100644 index 00000000..5801d3b5 --- /dev/null +++ b/app/src/main/java/com/ethran/notable/editor/ui/InboxToolbar.kt @@ -0,0 +1,264 @@ +package com.ethran.notable.editor.ui + +import androidx.compose.foundation.background +import androidx.compose.foundation.border +import androidx.compose.foundation.clickable +import androidx.compose.foundation.horizontalScroll +import androidx.compose.foundation.layout.Arrangement +import androidx.compose.foundation.layout.Box +import androidx.compose.foundation.layout.Column +import androidx.compose.foundation.layout.Row +import androidx.compose.foundation.layout.Spacer +import androidx.compose.foundation.layout.fillMaxWidth +import androidx.compose.foundation.layout.height +import androidx.compose.foundation.layout.padding +import androidx.compose.foundation.layout.size +import androidx.compose.foundation.layout.width +import androidx.compose.foundation.rememberScrollState +import androidx.compose.foundation.shape.RoundedCornerShape +import androidx.compose.foundation.text.BasicTextField +import androidx.compose.foundation.text.KeyboardActions +import androidx.compose.foundation.text.KeyboardOptions +import androidx.compose.material.Divider +import androidx.compose.material.Icon +import androidx.compose.material.Text +import androidx.compose.material.icons.Icons +import androidx.compose.material.icons.filled.Add +import androidx.compose.material.icons.filled.Close +import androidx.compose.runtime.Composable +import androidx.compose.runtime.getValue +import androidx.compose.runtime.mutableStateOf +import androidx.compose.runtime.remember +import androidx.compose.runtime.setValue +import androidx.compose.ui.Alignment +import androidx.compose.ui.Modifier +import androidx.compose.ui.graphics.Color +import androidx.compose.ui.graphics.SolidColor +import androidx.compose.ui.text.TextStyle +import androidx.compose.ui.text.font.FontWeight +import androidx.compose.ui.text.input.ImeAction +import androidx.compose.ui.unit.dp +import androidx.compose.ui.unit.sp + +val INBOX_TOOLBAR_HEIGHT = 210.dp + +@Composable +fun InboxToolbar( + selectedTags: List, + suggestedTags: List, + onTagAdd: (String) -> Unit, + onTagRemove: (String) -> Unit, + onSave: () -> Unit, + onDiscard: () -> Unit +) { + var tagInput by remember { mutableStateOf("") } + + fun submitTag() { + val tags = tagInput + .replace("#", "") + .replace(",", " ") + .split("\\s+".toRegex()) + .map { it.trim().lowercase() } + .filter { it.isNotEmpty() } + tags.forEach { onTagAdd(it) } + tagInput = "" + } + + // Filter suggestions based on input + val unselected = suggestedTags.filter { it !in selectedTags } + val filtered = if (tagInput.isNotEmpty()) { + val query = tagInput.lowercase().replace("#", "").trim() + unselected.filter { it.contains(query) } + } else { + unselected + } + + Column( + modifier = Modifier + .fillMaxWidth() + .background(Color.White) + ) { + // Action bar: Discard + Save + Row( + modifier = Modifier + .fillMaxWidth() + .padding(horizontal = 16.dp, vertical = 10.dp), + horizontalArrangement = Arrangement.SpaceBetween, + verticalAlignment = Alignment.CenterVertically + ) { + Box( + modifier = Modifier + .border(1.5.dp, Color.DarkGray, RoundedCornerShape(8.dp)) + .clickable { onDiscard() } + .padding(horizontal = 20.dp, vertical = 8.dp) + ) { + Text( + "← Back", + fontSize = 18.sp, + color = Color.DarkGray + ) + } + + Text( + "Inbox Capture", + fontSize = 18.sp, + fontWeight = FontWeight.Medium, + color = Color.Gray + ) + + Box( + modifier = Modifier + .background(Color.Black, RoundedCornerShape(8.dp)) + .clickable { onSave() } + .padding(horizontal = 24.dp, vertical = 8.dp) + ) { + Text( + "Save & Exit", + fontSize = 18.sp, + fontWeight = FontWeight.Bold, + color = Color.White + ) + } + } + + // Search / add tag input + Row( + modifier = Modifier + .fillMaxWidth() + .padding(horizontal = 16.dp), + verticalAlignment = Alignment.CenterVertically, + horizontalArrangement = Arrangement.spacedBy(10.dp) + ) { + Box( + modifier = Modifier + .weight(1f) + .height(46.dp) + .border(1.5.dp, Color.Gray, RoundedCornerShape(8.dp)) + .background(Color(0xFFFAFAFA), RoundedCornerShape(8.dp)) + .padding(horizontal = 14.dp), + contentAlignment = Alignment.CenterStart + ) { + if (tagInput.isEmpty()) { + Text( + "search or add tags...", + fontSize = 17.sp, + color = Color(0xFF999999) + ) + } + BasicTextField( + value = tagInput, + onValueChange = { tagInput = it }, + textStyle = TextStyle( + fontSize = 17.sp, + color = Color.Black + ), + singleLine = true, + cursorBrush = SolidColor(Color.Black), + keyboardOptions = KeyboardOptions(imeAction = ImeAction.Done), + keyboardActions = KeyboardActions(onDone = { submitTag() }), + modifier = Modifier.fillMaxWidth() + ) + } + + Box( + modifier = Modifier + .size(46.dp) + .background(Color.Black, RoundedCornerShape(8.dp)) + .clickable { submitTag() }, + contentAlignment = Alignment.Center + ) { + Icon( + Icons.Default.Add, + contentDescription = "Add tag", + tint = Color.White, + modifier = Modifier.size(26.dp) + ) + } + } + + Spacer(modifier = Modifier.height(10.dp)) + + // Tag pills row + Row( + modifier = Modifier + .fillMaxWidth() + .padding(horizontal = 16.dp) + .horizontalScroll(rememberScrollState()), + horizontalArrangement = Arrangement.spacedBy(8.dp), + verticalAlignment = Alignment.CenterVertically + ) { + // Selected tags with × button + selectedTags.forEach { tag -> + TagPill( + tag = tag, + selected = true, + onTap = { onTagRemove(tag) } + ) + } + + // Divider between selected and suggestions + if (filtered.isNotEmpty() && selectedTags.isNotEmpty()) { + Box( + modifier = Modifier + .width(1.5.dp) + .height(28.dp) + .background(Color.LightGray) + ) + } + + filtered.forEach { tag -> + TagPill( + tag = tag, + selected = false, + onTap = { + onTagAdd(tag) + tagInput = "" + } + ) + } + } + + Spacer(modifier = Modifier.height(10.dp)) + + // Divider + Divider(color = Color.DarkGray, thickness = 2.dp) + } +} + +@Composable +private fun TagPill( + tag: String, + selected: Boolean, + onTap: () -> Unit +) { + val bgColor = if (selected) Color.Black else Color.White + val textColor = if (selected) Color.White else Color.Black + val borderColor = if (selected) Color.Black else Color.Gray + + Box( + modifier = Modifier + .border(1.5.dp, borderColor, RoundedCornerShape(20.dp)) + .background(bgColor, RoundedCornerShape(20.dp)) + .clickable { onTap() } + .padding(horizontal = 14.dp, vertical = 6.dp) + ) { + Row( + verticalAlignment = Alignment.CenterVertically, + horizontalArrangement = Arrangement.spacedBy(6.dp) + ) { + Text( + "#$tag", + fontSize = 16.sp, + color = textColor + ) + if (selected) { + Icon( + Icons.Default.Close, + contentDescription = "Remove tag", + tint = textColor, + modifier = Modifier.size(16.dp) + ) + } + } + } +} diff --git a/app/src/main/java/com/ethran/notable/io/InboxSyncEngine.kt b/app/src/main/java/com/ethran/notable/io/InboxSyncEngine.kt index 842941b6..507b3edd 100644 --- a/app/src/main/java/com/ethran/notable/io/InboxSyncEngine.kt +++ b/app/src/main/java/com/ethran/notable/io/InboxSyncEngine.kt @@ -4,8 +4,6 @@ import android.os.Environment import com.ethran.notable.data.AppRepository import com.ethran.notable.data.datastore.GlobalAppSettings import com.ethran.notable.data.db.Stroke -import com.ethran.notable.editor.drawing.INBOX_CONTENT_START_Y -import com.ethran.notable.editor.drawing.INBOX_DIVIDER_Y import com.google.mlkit.common.model.DownloadConditions import com.google.mlkit.common.model.RemoteModelManager import com.google.mlkit.vision.digitalink.recognition.DigitalInkRecognition @@ -36,8 +34,16 @@ object InboxSyncEngine { ) } - suspend fun syncInboxPage(appRepository: AppRepository, pageId: String) { - log.i("Starting inbox sync for page $pageId") + /** + * Sync an inbox page to Obsidian. Tags come from the UI (pill selection), + * content is recognized from all strokes on the page. + */ + suspend fun syncInboxPage( + appRepository: AppRepository, + pageId: String, + tags: List + ) { + log.i("Starting inbox sync for page $pageId with tags: $tags") val page = appRepository.pageRepository.getById(pageId) if (page == null) { @@ -48,26 +54,20 @@ object InboxSyncEngine { val pageWithStrokes = appRepository.pageRepository.getWithStrokeById(pageId) val strokes = pageWithStrokes.strokes - if (strokes.isEmpty()) { - log.i("No strokes on inbox page, deleting") + if (strokes.isEmpty() && tags.isEmpty()) { + log.i("No strokes and no tags on inbox page, deleting") appRepository.pageRepository.delete(pageId) return } - ensureModelDownloaded() + val contentText = if (strokes.isNotEmpty()) { + ensureModelDownloaded() + log.i("Recognizing ${strokes.size} content strokes") + recognizeStrokes(strokes) + } else "" - // Classify strokes by zone using bounding box position - val tagStrokes = strokes.filter { it.top < INBOX_DIVIDER_Y } - val contentStrokes = strokes.filter { it.top >= INBOX_CONTENT_START_Y } + log.i("Recognized content: '${contentText.take(100)}'") - log.i("Classified ${tagStrokes.size} tag strokes, ${contentStrokes.size} content strokes") - - val tagText = if (tagStrokes.isNotEmpty()) recognizeStrokes(tagStrokes) else "" - val contentText = if (contentStrokes.isNotEmpty()) recognizeStrokes(contentStrokes) else "" - - log.i("Recognized tags: '$tagText', content: '${contentText.take(100)}'") - - val tags = parseTags(tagText) val createdDate = SimpleDateFormat("yyyy-MM-dd", Locale.US).format(page.createdAt) val markdown = generateMarkdown(createdDate, tags, contentText) @@ -137,15 +137,6 @@ object InboxSyncEngine { } } - private fun parseTags(rawText: String): List { - return rawText - .replace("#", "") - .replace(",", " ") - .split("\\s+".toRegex()) - .map { it.trim().lowercase() } - .filter { it.isNotEmpty() } - } - private fun generateMarkdown( createdDate: String, tags: List, @@ -153,7 +144,7 @@ object InboxSyncEngine { ): String { val sb = StringBuilder() sb.appendLine("---") - sb.appendLine("created: $createdDate") + sb.appendLine("created: \"[[$createdDate]]\"") if (tags.isNotEmpty()) { sb.appendLine("tags:") tags.forEach { sb.appendLine(" - $it") } diff --git a/app/src/main/java/com/ethran/notable/io/VaultTagScanner.kt b/app/src/main/java/com/ethran/notable/io/VaultTagScanner.kt new file mode 100644 index 00000000..71141c85 --- /dev/null +++ b/app/src/main/java/com/ethran/notable/io/VaultTagScanner.kt @@ -0,0 +1,127 @@ +package com.ethran.notable.io + +import android.os.Environment +import io.shipbook.shipbooksdk.ShipBook +import java.io.File + +private val log = ShipBook.getLogger("VaultTagScanner") + +data class TagScore( + val tag: String, + val frequency: Int, + val lastSeenMs: Long +) + +object VaultTagScanner { + + // Cached tags — pre-populated on app start + @Volatile + var cachedTags: List = emptyList() + private set + + /** + * Refresh the tag cache in the background. Call from app initialization. + */ + fun refreshCache(inboxPath: String) { + cachedTags = scanTags(inboxPath) + log.i("Tag cache refreshed: ${cachedTags.size} tags") + } + + /** + * Scans markdown files in the inbox folder, parses YAML frontmatter tags, + * and returns tags ranked by frequency × recency. + */ + fun scanTags(inboxPath: String, limit: Int = 30): List { + val dir = if (inboxPath.startsWith("/")) { + File(inboxPath) + } else { + File(Environment.getExternalStorageDirectory(), inboxPath) + } + + if (!dir.exists() || !dir.isDirectory) { + log.i("Inbox directory not found: ${dir.absolutePath}") + return emptyList() + } + + val tagScores = mutableMapOf() + + val mdFiles = dir.listFiles { f -> f.extension == "md" } ?: emptyArray() + log.i("Scanning ${mdFiles.size} markdown files for tags") + + for (file in mdFiles) { + val tags = parseFrontmatterTags(file) + val fileTime = file.lastModified() + for (tag in tags) { + val normalized = tag.lowercase().trim() + if (normalized.isEmpty()) continue + val existing = tagScores[normalized] + tagScores[normalized] = TagScore( + tag = normalized, + frequency = (existing?.frequency ?: 0) + 1, + lastSeenMs = maxOf(existing?.lastSeenMs ?: 0L, fileTime) + ) + } + } + + if (tagScores.isEmpty()) return emptyList() + + // Score: frequency * recency weight (more recent = higher weight) + val now = System.currentTimeMillis() + val maxAge = 90.0 * 24 * 60 * 60 * 1000 // 90 days in ms + + return tagScores.values + .sortedByDescending { score -> + val ageMs = (now - score.lastSeenMs).coerceAtLeast(0) + val recencyWeight = 1.0 - (ageMs / maxAge).coerceAtMost(1.0) + score.frequency * (0.3 + 0.7 * recencyWeight) + } + .take(limit) + .map { it.tag } + } + + private fun parseFrontmatterTags(file: File): List { + try { + val lines = file.readLines() + if (lines.isEmpty() || lines[0].trim() != "---") return emptyList() + + var inFrontmatter = true + var inTagsBlock = false + val tags = mutableListOf() + + for (i in 1 until lines.size) { + val line = lines[i] + if (line.trim() == "---") break // end of frontmatter + + if (line.startsWith("tags:")) { + // Inline tags: tags: [a, b, c] or tags: a, b + val inline = line.substringAfter("tags:").trim() + if (inline.isNotEmpty()) { + // Handle [a, b, c] or a, b formats + val cleaned = inline.trimStart('[').trimEnd(']') + tags.addAll( + cleaned.split(",") + .map { it.trim().trimStart('#') } + .filter { it.isNotEmpty() } + ) + } + inTagsBlock = true + continue + } + + if (inTagsBlock) { + val trimmed = line.trim() + if (trimmed.startsWith("- ")) { + tags.add(trimmed.removePrefix("- ").trim().trimStart('#')) + } else { + inTagsBlock = false + } + } + } + + return tags + } catch (e: Exception) { + log.e("Failed to parse tags from ${file.name}: ${e.message}") + return emptyList() + } + } +} diff --git a/app/src/main/java/com/ethran/notable/ui/components/GeneralSettings.kt b/app/src/main/java/com/ethran/notable/ui/components/GeneralSettings.kt index 3c0f6714..1b4bf038 100644 --- a/app/src/main/java/com/ethran/notable/ui/components/GeneralSettings.kt +++ b/app/src/main/java/com/ethran/notable/ui/components/GeneralSettings.kt @@ -1,10 +1,37 @@ package com.ethran.notable.ui.components +import androidx.compose.foundation.border import androidx.compose.foundation.layout.Column +import androidx.compose.foundation.layout.Row +import androidx.compose.foundation.layout.Spacer +import androidx.compose.foundation.layout.fillMaxWidth +import androidx.compose.foundation.layout.height +import androidx.compose.foundation.layout.padding +import androidx.compose.foundation.shape.RoundedCornerShape +import androidx.compose.foundation.text.BasicTextField +import androidx.compose.foundation.text.KeyboardActions +import androidx.compose.foundation.text.KeyboardOptions +import androidx.compose.material.MaterialTheme +import androidx.compose.material.Text import androidx.compose.runtime.Composable +import androidx.compose.runtime.getValue +import androidx.compose.runtime.mutableStateOf +import androidx.compose.runtime.remember +import androidx.compose.runtime.setValue +import androidx.compose.ui.Alignment +import androidx.compose.ui.Modifier +import androidx.compose.ui.graphics.Color +import androidx.compose.ui.graphics.SolidColor +import androidx.compose.ui.platform.LocalFocusManager import androidx.compose.ui.res.stringResource +import androidx.compose.ui.text.TextStyle +import androidx.compose.ui.text.font.FontWeight +import androidx.compose.ui.text.input.ImeAction +import androidx.compose.ui.unit.dp +import androidx.compose.ui.unit.sp import com.ethran.notable.R import com.ethran.notable.data.datastore.AppSettings +import com.ethran.notable.io.VaultTagScanner @Composable @@ -12,6 +39,11 @@ fun GeneralSettings( settings: AppSettings, onSettingsChange: (AppSettings) -> Unit ) { Column { + // Inbox Capture section + InboxCaptureSettings(settings, onSettingsChange) + + Spacer(modifier = Modifier.height(8.dp)) + SelectorRow( label = stringResource(R.string.default_page_background_template), options = listOf( "blank" to stringResource(R.string.blank_page), @@ -87,3 +119,77 @@ fun GeneralSettings( }) } } + +@Composable +private fun InboxCaptureSettings( + settings: AppSettings, + onSettingsChange: (AppSettings) -> Unit +) { + val focusManager = LocalFocusManager.current + var pathInput by remember { mutableStateOf(settings.obsidianInboxPath) } + + Column( + modifier = Modifier + .fillMaxWidth() + .border(1.dp, Color.LightGray, RoundedCornerShape(8.dp)) + .padding(16.dp) + ) { + Text( + "Inbox Capture", + style = MaterialTheme.typography.h6, + fontWeight = FontWeight.Bold + ) + + Spacer(modifier = Modifier.height(4.dp)) + + Text( + "Handwritten notes are recognized and saved as markdown to this folder. " + + "Tags are also loaded from existing notes in this folder.", + style = MaterialTheme.typography.body2, + color = Color.Gray + ) + + Spacer(modifier = Modifier.height(12.dp)) + + Text( + "Inbox folder path", + style = MaterialTheme.typography.body1, + fontWeight = FontWeight.Medium + ) + + Spacer(modifier = Modifier.height(4.dp)) + + Row( + modifier = Modifier.fillMaxWidth(), + verticalAlignment = Alignment.CenterVertically + ) { + BasicTextField( + value = pathInput, + onValueChange = { pathInput = it }, + textStyle = TextStyle(fontSize = 16.sp, color = Color.Black), + singleLine = true, + cursorBrush = SolidColor(Color.Black), + keyboardOptions = KeyboardOptions(imeAction = ImeAction.Done), + keyboardActions = KeyboardActions(onDone = { + onSettingsChange(settings.copy(obsidianInboxPath = pathInput)) + VaultTagScanner.refreshCache(pathInput) + focusManager.clearFocus() + }), + modifier = Modifier + .weight(1f) + .border(1.dp, Color.Gray, RoundedCornerShape(6.dp)) + .padding(horizontal = 12.dp, vertical = 10.dp) + ) + } + + Spacer(modifier = Modifier.height(4.dp)) + + Text( + "Relative to /storage/emulated/0/. Press Done to save.", + style = MaterialTheme.typography.caption, + color = Color.Gray + ) + } + + SettingsDivider() +} From 3a8f26bf6dcd84c8404524792863824d16b95236 Mon Sep 17 00:00:00 2001 From: Joshua Pham Date: Sun, 8 Mar 2026 23:33:18 -0400 Subject: [PATCH 04/23] Simplify UI to atomic capture workflow: collapsible tags, fix dead zone, clean library - Fix non-writable area: setupSurface now excludes both top (inbox toolbar) and bottom (pen toolbar) for capture pages, preventing pen input dead zones - Make tag section collapsible: starts collapsed showing "Capture" toggle with tag count, expands to show search + tag pills. Drawing surface updates dynamically - Simplify library: remove folders/notebooks/import, show clean grid of captures with prominent "New Capture" card - Rename "Inbox Capture" to "Capture" throughout UI and settings - Remove background template selector (all pages are captures now) - Add sync overlay ("Saving to vault...") during capture save - Default pen changed to fountain Co-Authored-By: Claude Opus 4.6 --- .../com/ethran/notable/editor/EditorView.kt | 27 +++ .../editor/canvas/CanvasObserverRegistry.kt | 8 + .../notable/editor/canvas/OnyxInputHandler.kt | 39 +++- .../notable/editor/state/EditorState.kt | 3 +- .../ethran/notable/editor/ui/InboxToolbar.kt | 212 ++++++++++-------- .../ethran/notable/editor/utils/einkHelper.kt | 32 ++- .../com/ethran/notable/io/InboxSyncEngine.kt | 25 ++- .../notable/ui/components/GeneralSettings.kt | 20 +- .../com/ethran/notable/ui/views/HomeView.kt | 138 +++++++----- 9 files changed, 311 insertions(+), 193 deletions(-) diff --git a/app/src/main/java/com/ethran/notable/editor/EditorView.kt b/app/src/main/java/com/ethran/notable/editor/EditorView.kt index a04b90e5..9bc4d873 100644 --- a/app/src/main/java/com/ethran/notable/editor/EditorView.kt +++ b/app/src/main/java/com/ethran/notable/editor/EditorView.kt @@ -1,11 +1,18 @@ package com.ethran.notable.editor +import androidx.compose.foundation.background +import androidx.compose.foundation.layout.Box import androidx.compose.foundation.layout.BoxWithConstraints import androidx.compose.foundation.layout.Column import androidx.compose.foundation.layout.Row import androidx.compose.foundation.layout.Spacer import androidx.compose.foundation.layout.fillMaxHeight +import androidx.compose.foundation.layout.fillMaxSize import androidx.compose.foundation.layout.fillMaxWidth +import androidx.compose.material.Text +import androidx.compose.ui.Alignment +import androidx.compose.ui.graphics.Color +import androidx.compose.ui.unit.sp import androidx.compose.runtime.Composable import androidx.compose.runtime.DisposableEffect import androidx.compose.runtime.LaunchedEffect @@ -226,6 +233,10 @@ fun EditorView( InboxToolbar( selectedTags = selectedTags, suggestedTags = suggestedTags, + isExpanded = editorState.isInboxTagsExpanded, + onToggleExpanded = { + editorState.isInboxTagsExpanded = !editorState.isInboxTagsExpanded + }, onTagAdd = { tag -> if (tag !in selectedTags) selectedTags.add(tag) }, @@ -269,6 +280,22 @@ fun EditorView( PositionedToolbar(exportEngine, navController, appRepository, editorState, editorControlTower) } HorizontalScrollIndicator(state = editorState) + + // Full-screen overlay while inbox sync is in progress + if (isSyncing) { + Box( + modifier = Modifier + .fillMaxSize() + .background(Color.White), + contentAlignment = Alignment.Center + ) { + Text( + text = "Saving to vault...", + fontSize = 24.sp, + color = Color.Black + ) + } + } } } } diff --git a/app/src/main/java/com/ethran/notable/editor/canvas/CanvasObserverRegistry.kt b/app/src/main/java/com/ethran/notable/editor/canvas/CanvasObserverRegistry.kt index b73e08e1..76f520d8 100644 --- a/app/src/main/java/com/ethran/notable/editor/canvas/CanvasObserverRegistry.kt +++ b/app/src/main/java/com/ethran/notable/editor/canvas/CanvasObserverRegistry.kt @@ -214,6 +214,14 @@ class CanvasObserverRegistry( refreshManager.refreshUi(null) } } + coroutineScope.launch { + snapshotFlow { state.isInboxTagsExpanded }.drop(1).collect { + logCanvasObserver.v("inbox tags expanded change: ${state.isInboxTagsExpanded}") + inputHandler.updateActiveSurface() + inputHandler.updatePenAndStroke() + refreshManager.refreshUi(null) + } + } } private fun observeMode() { diff --git a/app/src/main/java/com/ethran/notable/editor/canvas/OnyxInputHandler.kt b/app/src/main/java/com/ethran/notable/editor/canvas/OnyxInputHandler.kt index 1356e1f5..d87efd6c 100644 --- a/app/src/main/java/com/ethran/notable/editor/canvas/OnyxInputHandler.kt +++ b/app/src/main/java/com/ethran/notable/editor/canvas/OnyxInputHandler.kt @@ -6,8 +6,11 @@ import android.graphics.RectF import android.util.Log import androidx.compose.ui.unit.dp import androidx.core.graphics.toRect +import com.ethran.notable.data.datastore.AppSettings import com.ethran.notable.data.datastore.GlobalAppSettings import com.ethran.notable.editor.PageView +import com.ethran.notable.editor.ui.INBOX_TOOLBAR_COLLAPSED_HEIGHT +import com.ethran.notable.editor.ui.INBOX_TOOLBAR_EXPANDED_HEIGHT import com.ethran.notable.editor.state.EditorState import com.ethran.notable.editor.state.History import com.ethran.notable.editor.state.Mode @@ -176,21 +179,35 @@ class OnyxInputHandler( } fun updateActiveSurface() { - // Takes at least 50ms on Note 4c, - // and I don't think that we need it immediately log.i("Update editable surface") coroutineScope.launch { onSurfaceInit(drawCanvas) - val toolbarHeight = when { - state.isInboxPage -> convertDpToPixel(210.dp, drawCanvas.context).toInt() - state.isToolbarOpen -> convertDpToPixel(40.dp, drawCanvas.context).toInt() - else -> 0 + if (state.isInboxPage) { + val inboxToolbarHeight = if (state.isInboxTagsExpanded) { + convertDpToPixel(INBOX_TOOLBAR_EXPANDED_HEIGHT, drawCanvas.context).toInt() + } else { + convertDpToPixel(INBOX_TOOLBAR_COLLAPSED_HEIGHT, drawCanvas.context).toInt() + } + val penToolbarHeight = convertDpToPixel(40.dp, drawCanvas.context).toInt() + setupSurface( + drawCanvas, + touchHelper, + topExcludeHeight = inboxToolbarHeight, + bottomExcludeHeight = penToolbarHeight + ) + } else { + val toolbarHeight = if (state.isToolbarOpen) { + convertDpToPixel(40.dp, drawCanvas.context).toInt() + } else 0 + val topExclude = if (GlobalAppSettings.current.toolbarPosition == AppSettings.Position.Top) toolbarHeight else 0 + val bottomExclude = if (GlobalAppSettings.current.toolbarPosition == AppSettings.Position.Bottom) toolbarHeight else 0 + setupSurface( + drawCanvas, + touchHelper, + topExcludeHeight = topExclude, + bottomExcludeHeight = bottomExclude + ) } - setupSurface( - drawCanvas, - touchHelper, - toolbarHeight - ) } } private fun onRawDrawingList(plist: TouchPointList) { diff --git a/app/src/main/java/com/ethran/notable/editor/state/EditorState.kt b/app/src/main/java/com/ethran/notable/editor/state/EditorState.kt index 06aad1e8..36c8fabb 100644 --- a/app/src/main/java/com/ethran/notable/editor/state/EditorState.kt +++ b/app/src/main/java/com/ethran/notable/editor/state/EditorState.kt @@ -89,7 +89,7 @@ class EditorState( private val log = ShipBook.getLogger("EditorState") var mode by mutableStateOf(persistedEditorSettings?.mode ?: Mode.Draw) // should save - var pen by mutableStateOf(persistedEditorSettings?.pen ?: Pen.BALLPEN) // should save + var pen by mutableStateOf(persistedEditorSettings?.pen ?: Pen.FOUNTAIN) // should save var eraser by mutableStateOf(persistedEditorSettings?.eraser ?: Eraser.PEN) // should save var isDrawing by mutableStateOf(true) // gives information if pen touch will be drawn or not // For debugging: @@ -104,6 +104,7 @@ class EditorState( // } var isInboxPage by mutableStateOf(false) + var isInboxTagsExpanded by mutableStateOf(false) var isToolbarOpen by mutableStateOf( persistedEditorSettings?.isToolbarOpen ?: false diff --git a/app/src/main/java/com/ethran/notable/editor/ui/InboxToolbar.kt b/app/src/main/java/com/ethran/notable/editor/ui/InboxToolbar.kt index 5801d3b5..84bae98c 100644 --- a/app/src/main/java/com/ethran/notable/editor/ui/InboxToolbar.kt +++ b/app/src/main/java/com/ethran/notable/editor/ui/InboxToolbar.kt @@ -25,6 +25,8 @@ import androidx.compose.material.Text import androidx.compose.material.icons.Icons import androidx.compose.material.icons.filled.Add import androidx.compose.material.icons.filled.Close +import androidx.compose.material.icons.filled.KeyboardArrowDown +import androidx.compose.material.icons.filled.KeyboardArrowUp import androidx.compose.runtime.Composable import androidx.compose.runtime.getValue import androidx.compose.runtime.mutableStateOf @@ -37,15 +39,19 @@ import androidx.compose.ui.graphics.SolidColor import androidx.compose.ui.text.TextStyle import androidx.compose.ui.text.font.FontWeight import androidx.compose.ui.text.input.ImeAction +import androidx.compose.ui.unit.Dp import androidx.compose.ui.unit.dp import androidx.compose.ui.unit.sp -val INBOX_TOOLBAR_HEIGHT = 210.dp +val INBOX_TOOLBAR_COLLAPSED_HEIGHT: Dp = 55.dp +val INBOX_TOOLBAR_EXPANDED_HEIGHT: Dp = 170.dp @Composable fun InboxToolbar( selectedTags: List, suggestedTags: List, + isExpanded: Boolean, + onToggleExpanded: () -> Unit, onTagAdd: (String) -> Unit, onTagRemove: (String) -> Unit, onSave: () -> Unit, @@ -64,7 +70,6 @@ fun InboxToolbar( tagInput = "" } - // Filter suggestions based on input val unselected = suggestedTags.filter { it !in selectedTags } val filtered = if (tagInput.isNotEmpty()) { val query = tagInput.lowercase().replace("#", "").trim() @@ -78,7 +83,7 @@ fun InboxToolbar( .fillMaxWidth() .background(Color.White) ) { - // Action bar: Discard + Save + // Action bar: Back + toggle/tag count + Save & Exit Row( modifier = Modifier .fillMaxWidth() @@ -99,12 +104,31 @@ fun InboxToolbar( ) } - Text( - "Inbox Capture", - fontSize = 18.sp, - fontWeight = FontWeight.Medium, - color = Color.Gray - ) + // Title + tag toggle + Row( + modifier = Modifier + .clickable { onToggleExpanded() } + .border(1.dp, Color.Gray, RoundedCornerShape(8.dp)) + .padding(horizontal = 14.dp, vertical = 6.dp), + verticalAlignment = Alignment.CenterVertically, + horizontalArrangement = Arrangement.spacedBy(8.dp) + ) { + val label = if (selectedTags.isEmpty()) "Capture" + else "Capture · ${selectedTags.size} tag${if (selectedTags.size != 1) "s" else ""}" + Text( + label, + fontSize = 17.sp, + fontWeight = FontWeight.Medium, + color = Color.DarkGray + ) + Icon( + if (isExpanded) Icons.Default.KeyboardArrowUp + else Icons.Default.KeyboardArrowDown, + contentDescription = if (isExpanded) "Collapse tags" else "Expand tags", + tint = Color.DarkGray, + modifier = Modifier.size(22.dp) + ) + } Box( modifier = Modifier @@ -121,105 +145,105 @@ fun InboxToolbar( } } - // Search / add tag input - Row( - modifier = Modifier - .fillMaxWidth() - .padding(horizontal = 16.dp), - verticalAlignment = Alignment.CenterVertically, - horizontalArrangement = Arrangement.spacedBy(10.dp) - ) { - Box( + if (isExpanded) { + // Search / add tag input + Row( modifier = Modifier - .weight(1f) - .height(46.dp) - .border(1.5.dp, Color.Gray, RoundedCornerShape(8.dp)) - .background(Color(0xFFFAFAFA), RoundedCornerShape(8.dp)) - .padding(horizontal = 14.dp), - contentAlignment = Alignment.CenterStart + .fillMaxWidth() + .padding(horizontal = 16.dp), + verticalAlignment = Alignment.CenterVertically, + horizontalArrangement = Arrangement.spacedBy(10.dp) ) { - if (tagInput.isEmpty()) { - Text( - "search or add tags...", - fontSize = 17.sp, - color = Color(0xFF999999) + Box( + modifier = Modifier + .weight(1f) + .height(46.dp) + .border(1.5.dp, Color.Gray, RoundedCornerShape(8.dp)) + .background(Color(0xFFFAFAFA), RoundedCornerShape(8.dp)) + .padding(horizontal = 14.dp), + contentAlignment = Alignment.CenterStart + ) { + if (tagInput.isEmpty()) { + Text( + "search or add tags...", + fontSize = 17.sp, + color = Color(0xFF999999) + ) + } + BasicTextField( + value = tagInput, + onValueChange = { tagInput = it }, + textStyle = TextStyle( + fontSize = 17.sp, + color = Color.Black + ), + singleLine = true, + cursorBrush = SolidColor(Color.Black), + keyboardOptions = KeyboardOptions(imeAction = ImeAction.Done), + keyboardActions = KeyboardActions(onDone = { submitTag() }), + modifier = Modifier.fillMaxWidth() + ) + } + + Box( + modifier = Modifier + .size(46.dp) + .background(Color.Black, RoundedCornerShape(8.dp)) + .clickable { submitTag() }, + contentAlignment = Alignment.Center + ) { + Icon( + Icons.Default.Add, + contentDescription = "Add tag", + tint = Color.White, + modifier = Modifier.size(26.dp) ) } - BasicTextField( - value = tagInput, - onValueChange = { tagInput = it }, - textStyle = TextStyle( - fontSize = 17.sp, - color = Color.Black - ), - singleLine = true, - cursorBrush = SolidColor(Color.Black), - keyboardOptions = KeyboardOptions(imeAction = ImeAction.Done), - keyboardActions = KeyboardActions(onDone = { submitTag() }), - modifier = Modifier.fillMaxWidth() - ) } - Box( + Spacer(modifier = Modifier.height(10.dp)) + + // Tag pills row + Row( modifier = Modifier - .size(46.dp) - .background(Color.Black, RoundedCornerShape(8.dp)) - .clickable { submitTag() }, - contentAlignment = Alignment.Center + .fillMaxWidth() + .padding(horizontal = 16.dp) + .horizontalScroll(rememberScrollState()), + horizontalArrangement = Arrangement.spacedBy(8.dp), + verticalAlignment = Alignment.CenterVertically ) { - Icon( - Icons.Default.Add, - contentDescription = "Add tag", - tint = Color.White, - modifier = Modifier.size(26.dp) - ) - } - } - - Spacer(modifier = Modifier.height(10.dp)) + selectedTags.forEach { tag -> + TagPill( + tag = tag, + selected = true, + onTap = { onTagRemove(tag) } + ) + } - // Tag pills row - Row( - modifier = Modifier - .fillMaxWidth() - .padding(horizontal = 16.dp) - .horizontalScroll(rememberScrollState()), - horizontalArrangement = Arrangement.spacedBy(8.dp), - verticalAlignment = Alignment.CenterVertically - ) { - // Selected tags with × button - selectedTags.forEach { tag -> - TagPill( - tag = tag, - selected = true, - onTap = { onTagRemove(tag) } - ) - } + if (filtered.isNotEmpty() && selectedTags.isNotEmpty()) { + Box( + modifier = Modifier + .width(1.5.dp) + .height(28.dp) + .background(Color.LightGray) + ) + } - // Divider between selected and suggestions - if (filtered.isNotEmpty() && selectedTags.isNotEmpty()) { - Box( - modifier = Modifier - .width(1.5.dp) - .height(28.dp) - .background(Color.LightGray) - ) + filtered.forEach { tag -> + TagPill( + tag = tag, + selected = false, + onTap = { + onTagAdd(tag) + tagInput = "" + } + ) + } } - filtered.forEach { tag -> - TagPill( - tag = tag, - selected = false, - onTap = { - onTagAdd(tag) - tagInput = "" - } - ) - } + Spacer(modifier = Modifier.height(10.dp)) } - Spacer(modifier = Modifier.height(10.dp)) - // Divider Divider(color = Color.DarkGray, thickness = 2.dp) } diff --git a/app/src/main/java/com/ethran/notable/editor/utils/einkHelper.kt b/app/src/main/java/com/ethran/notable/editor/utils/einkHelper.kt index 262d2a2e..24d27b41 100644 --- a/app/src/main/java/com/ethran/notable/editor/utils/einkHelper.kt +++ b/app/src/main/java/com/ethran/notable/editor/utils/einkHelper.kt @@ -236,34 +236,32 @@ fun onSurfaceDestroy(view: View, touchHelper: TouchHelper?) { } -fun setupSurface(view: View, touchHelper: TouchHelper?, toolbarHeight: Int) { +fun setupSurface( + view: View, + touchHelper: TouchHelper?, + topExcludeHeight: Int = 0, + bottomExcludeHeight: Int = 0 +) { if(touchHelper == null) return - // Takes at least 50ms on Note 4c, - // and I don't think that we need it immediately log.i("Setup editable surface") touchHelper.debugLog(false) touchHelper.setRawDrawingEnabled(false) touchHelper.closeRawDrawing() - // Store view dimensions locally before using in Rect val viewWidth = view.width val viewHeight = view.height - // Determine the exclusion area based on toolbar position - val excludeRect: Rect = - if (GlobalAppSettings.current.toolbarPosition == AppSettings.Position.Top) { - Rect(0, 0, viewWidth, toolbarHeight) - } else { - Rect(0, viewHeight - toolbarHeight, viewWidth, viewHeight) - } + val excludeRects = mutableListOf() + if (topExcludeHeight > 0) { + excludeRects.add(Rect(0, 0, viewWidth, topExcludeHeight)) + } + if (bottomExcludeHeight > 0) { + excludeRects.add(Rect(0, viewHeight - bottomExcludeHeight, viewWidth, viewHeight)) + } - val limitRect = - if (GlobalAppSettings.current.toolbarPosition == AppSettings.Position.Top) - Rect(0, toolbarHeight, viewWidth, viewHeight) - else - Rect(0, 0, viewWidth, viewHeight - toolbarHeight) + val limitRect = Rect(0, topExcludeHeight, viewWidth, viewHeight - bottomExcludeHeight) - touchHelper.setLimitRect(mutableListOf(limitRect)).setExcludeRect(listOf(excludeRect)) + touchHelper.setLimitRect(mutableListOf(limitRect)).setExcludeRect(excludeRects) .openRawDrawing() touchHelper.setRawDrawingEnabled(true) diff --git a/app/src/main/java/com/ethran/notable/io/InboxSyncEngine.kt b/app/src/main/java/com/ethran/notable/io/InboxSyncEngine.kt index 507b3edd..162f631b 100644 --- a/app/src/main/java/com/ethran/notable/io/InboxSyncEngine.kt +++ b/app/src/main/java/com/ethran/notable/io/InboxSyncEngine.kt @@ -63,7 +63,8 @@ object InboxSyncEngine { val contentText = if (strokes.isNotEmpty()) { ensureModelDownloaded() log.i("Recognizing ${strokes.size} content strokes") - recognizeStrokes(strokes) + val raw = recognizeStrokes(strokes) + postProcessRecognition(raw) } else "" log.i("Recognized content: '${contentText.take(100)}'") @@ -137,6 +138,28 @@ object InboxSyncEngine { } } + /** + * Post-process ML Kit recognition output: + * - Normalize any bracket/paren wrapping to [[wiki links]] + * - Collapse space between # and the following word into a proper #tag + */ + private fun postProcessRecognition(text: String): String { + var result = text + + // Normalize bracket/paren wrapping to [[wiki links]] + // Handles: [text], ((text)), ([text]), [(text)], [[text]], etc. + result = result.replace(Regex("""[(\[]{1,2}([^)\]\n]+?)[)\]]{1,2}""")) { match -> + "[[${match.groupValues[1].trim()}]]" + } + + // Collapse space between # and the word following it + result = result.replace(Regex("""#\s+(\w+)""")) { match -> + "#${match.groupValues[1]}" + } + + return result + } + private fun generateMarkdown( createdDate: String, tags: List, diff --git a/app/src/main/java/com/ethran/notable/ui/components/GeneralSettings.kt b/app/src/main/java/com/ethran/notable/ui/components/GeneralSettings.kt index 1b4bf038..d0fb4529 100644 --- a/app/src/main/java/com/ethran/notable/ui/components/GeneralSettings.kt +++ b/app/src/main/java/com/ethran/notable/ui/components/GeneralSettings.kt @@ -39,21 +39,11 @@ fun GeneralSettings( settings: AppSettings, onSettingsChange: (AppSettings) -> Unit ) { Column { - // Inbox Capture section + // Capture settings InboxCaptureSettings(settings, onSettingsChange) Spacer(modifier = Modifier.height(8.dp)) - SelectorRow( - label = stringResource(R.string.default_page_background_template), options = listOf( - "blank" to stringResource(R.string.blank_page), - "dotted" to stringResource(R.string.dot_grid), - "lined" to stringResource(R.string.lines), - "squared" to stringResource(R.string.small_squares_grid), - "hexed" to stringResource(R.string.hexagon_grid), - ), value = settings.defaultNativeTemplate, onValueChange = { - onSettingsChange(settings.copy(defaultNativeTemplate = it)) - }) SelectorRow( label = stringResource(R.string.toolbar_position), options = listOf( AppSettings.Position.Top to stringResource(R.string.toolbar_position_top), @@ -135,7 +125,7 @@ private fun InboxCaptureSettings( .padding(16.dp) ) { Text( - "Inbox Capture", + "Capture", style = MaterialTheme.typography.h6, fontWeight = FontWeight.Bold ) @@ -143,8 +133,8 @@ private fun InboxCaptureSettings( Spacer(modifier = Modifier.height(4.dp)) Text( - "Handwritten notes are recognized and saved as markdown to this folder. " + - "Tags are also loaded from existing notes in this folder.", + "Handwritten captures are recognized and saved as markdown to your Obsidian vault. " + + "Tags are loaded from existing notes in this folder.", style = MaterialTheme.typography.body2, color = Color.Gray ) @@ -152,7 +142,7 @@ private fun InboxCaptureSettings( Spacer(modifier = Modifier.height(12.dp)) Text( - "Inbox folder path", + "Vault inbox folder", style = MaterialTheme.typography.body1, fontWeight = FontWeight.Medium ) diff --git a/app/src/main/java/com/ethran/notable/ui/views/HomeView.kt b/app/src/main/java/com/ethran/notable/ui/views/HomeView.kt index 785c01d9..55bde8ab 100644 --- a/app/src/main/java/com/ethran/notable/ui/views/HomeView.kt +++ b/app/src/main/java/com/ethran/notable/ui/views/HomeView.kt @@ -60,6 +60,7 @@ import com.ethran.notable.ui.SnackConf import com.ethran.notable.ui.SnackState import com.ethran.notable.ui.components.BreadCrumb import com.ethran.notable.ui.components.NotebookCard +import com.ethran.notable.ui.components.PagePreview import com.ethran.notable.ui.components.ShowPagesRow import com.ethran.notable.ui.dialogs.EmptyBookWarningHandler import com.ethran.notable.ui.dialogs.FolderConfigDialog @@ -141,68 +142,97 @@ fun LibraryContent( onImportXopp: (Uri) -> Unit ) { Column(Modifier.fillMaxSize()) { - Topbar { - Row(Modifier.fillMaxWidth()) { - Spacer(modifier = Modifier.weight(1f)) - BadgedBox( - badge = { - if (!uiState.isLatestVersion) Badge( - backgroundColor = Color.Black, - modifier = Modifier.offset((-12).dp, 10.dp) - ) - }) { - Icon( - imageVector = FeatherIcons.Settings, contentDescription = "Settings", - Modifier - .padding(8.dp) - .noRippleClickable(onClick = onNavigateToSettings) + // Slim header + Row( + Modifier + .fillMaxWidth() + .padding(horizontal = 16.dp, vertical = 12.dp), + horizontalArrangement = Arrangement.SpaceBetween, + verticalAlignment = Alignment.CenterVertically + ) { + Text( + text = "Notable", + style = androidx.compose.material.MaterialTheme.typography.h5, + fontWeight = androidx.compose.ui.text.font.FontWeight.Bold + ) + BadgedBox( + badge = { + if (!uiState.isLatestVersion) Badge( + backgroundColor = Color.Black, + modifier = Modifier.offset((-12).dp, 10.dp) ) - } - } - Row(Modifier.padding(10.dp)) { - BreadCrumb( - folders = uiState.breadcrumbFolders, onSelectFolderId = onNavigateToFolder + }) { + Icon( + imageVector = FeatherIcons.Settings, contentDescription = "Settings", + Modifier + .padding(8.dp) + .noRippleClickable(onClick = onNavigateToSettings) ) } - } - Column(Modifier.padding(10.dp)) { - Spacer(Modifier.height(10.dp)) - - FolderList( - appRepository = appRepository, - folders = uiState.folders, - onNavigateToFolder = onNavigateToFolder, - onCreateNewFolder = onCreateNewFolder - ) - - Spacer(Modifier.height(10.dp)) - ShowPagesRow( - appRepository = appRepository, - pages = uiState.singlePages, - currentPageId = null, - title = stringResource(R.string.home_quick_pages), onSelectPage = goToPage, - showAddQuickPage = true, onCreateNewQuickPage = onCreateNewQuickPage - ) - - Spacer(Modifier.height(10.dp)) + // Page grid + val pages = uiState.singlePages?.reversed() ?: emptyList() + LazyVerticalGrid( + columns = GridCells.Adaptive(140.dp), + verticalArrangement = Arrangement.spacedBy(16.dp), + horizontalArrangement = Arrangement.spacedBy(16.dp), + modifier = Modifier + .padding(horizontal = 16.dp) + .autoEInkAnimationOnScroll() + ) { + // New capture card + item { + Box( + contentAlignment = Alignment.Center, + modifier = Modifier + .aspectRatio(3f / 4f) + .border(2.dp, Color.Black, RectangleShape) + .noRippleClickable(onClick = onCreateNewQuickPage) + ) { + Column( + horizontalAlignment = Alignment.CenterHorizontally, + verticalArrangement = Arrangement.spacedBy(8.dp) + ) { + Icon( + imageVector = FeatherIcons.FilePlus, + contentDescription = "New Capture", + tint = Color.Black, + modifier = Modifier.size(48.dp) + ) + Text( + "New Capture", + style = androidx.compose.material.MaterialTheme.typography.body2, + color = Color.DarkGray + ) + } + } + } - NotebookGrid( - appRepository = appRepository, - exportEngine = exportEngine, - books = uiState.books, - isImporting = uiState.isImporting, - onNavigateToEditor = onNavigateToEditor, - onDeleteEmptyBook = onDeleteEmptyBook, - onCreateNewNotebook = onCreateNewNotebook, - onImportPdf = onImportPdf, - onImportXopp = onImportXopp - ) + // Existing pages + items(pages) { page -> + var isPageSelected by remember { mutableStateOf(false) } + Box { + PagePreview( + modifier = Modifier + .combinedClickable( + onClick = { goToPage(page.id) }, + onLongClick = { isPageSelected = true } + ) + .aspectRatio(3f / 4f) + .border(1.dp, Color.Gray, RectangleShape), + pageId = page.id + ) + if (isPageSelected) com.ethran.notable.editor.ui.PageMenu( + appRepository = appRepository, + pageId = page.id, + canDelete = true, + onClose = { isPageSelected = false } + ) + } + } } } - - } @Composable From 658bc743a0547ef844647acc6d04e05ae1ce57c2 Mon Sep 17 00:00:00 2001 From: Joshua Pham Date: Sun, 8 Mar 2026 23:39:23 -0400 Subject: [PATCH 05/23] Improve handwriting recognition: line segmentation, WritingArea, and pre-context ML Kit assumes single-line input, so multi-line captures were poorly recognized. Now strokes are clustered into lines by vertical position, each line is recognized independently with its bounding box as the WritingArea, and the last 20 chars feed forward as pre-context. Co-Authored-By: Claude Opus 4.6 --- .../com/ethran/notable/io/InboxSyncEngine.kt | 119 +++++++++++++++--- 1 file changed, 102 insertions(+), 17 deletions(-) diff --git a/app/src/main/java/com/ethran/notable/io/InboxSyncEngine.kt b/app/src/main/java/com/ethran/notable/io/InboxSyncEngine.kt index 162f631b..9c9d0488 100644 --- a/app/src/main/java/com/ethran/notable/io/InboxSyncEngine.kt +++ b/app/src/main/java/com/ethran/notable/io/InboxSyncEngine.kt @@ -11,6 +11,8 @@ import com.google.mlkit.vision.digitalink.recognition.DigitalInkRecognitionModel import com.google.mlkit.vision.digitalink.recognition.DigitalInkRecognitionModelIdentifier import com.google.mlkit.vision.digitalink.recognition.DigitalInkRecognizerOptions import com.google.mlkit.vision.digitalink.recognition.Ink +import com.google.mlkit.vision.digitalink.recognition.RecognitionContext +import com.google.mlkit.vision.digitalink.recognition.WritingArea import io.shipbook.shipbooksdk.ShipBook import kotlinx.coroutines.suspendCancellableCoroutine import java.io.File @@ -100,42 +102,125 @@ object InboxSyncEngine { } } + /** + * Segment strokes into lines based on vertical position, recognize each + * line independently with WritingArea and pre-context, then join results. + */ private suspend fun recognizeStrokes(strokes: List): String { val rec = recognizer ?: throw IllegalStateException("ML Kit recognizer not initialized") - val inkBuilder = Ink.builder() + val lines = segmentIntoLines(strokes) + log.i("Segmented ${strokes.size} strokes into ${lines.size} lines") + + val recognizedLines = mutableListOf() + var preContext = "" + + for (lineStrokes in lines) { + val ink = buildInk(lineStrokes) + val writingArea = computeWritingArea(lineStrokes) + val context = RecognitionContext.builder() + .setWritingArea(writingArea) + .apply { if (preContext.isNotEmpty()) setPreContext(preContext) } + .build() - // Sort strokes by creation time, then by position for deterministic order - val sortedStrokes = strokes.sortedWith( - compareBy { it.createdAt.time }.thenBy { it.top }.thenBy { it.left } + val text = suspendCancellableCoroutine { cont -> + rec.recognize(ink, context) + .addOnSuccessListener { result -> + cont.resume(result.candidates.firstOrNull()?.text ?: "") + } + .addOnFailureListener { cont.resumeWithException(it) } + } + + if (text.isNotBlank()) { + recognizedLines.add(text) + // Use last ~20 chars as pre-context for the next line + preContext = text.takeLast(20) + } + } + + return recognizedLines.joinToString("\n") + } + + /** + * Group strokes into horizontal lines by clustering on vertical midpoint. + * Strokes whose vertical centers are within half a line-height of each + * other belong to the same line. + */ + private fun segmentIntoLines(strokes: List): List> { + if (strokes.isEmpty()) return emptyList() + + // Sort by vertical midpoint, then left edge + val sorted = strokes.sortedWith( + compareBy { (it.top + it.bottom) / 2f }.thenBy { it.left } ) - // Use actual stroke creation timestamps as base, with synthetic - // per-point timing (~10ms between points, simulating natural writing) - for (stroke in sortedStrokes) { + // Estimate typical line height from median stroke height + val strokeHeights = sorted.map { it.bottom - it.top }.filter { it > 0 }.sorted() + val medianHeight = if (strokeHeights.isNotEmpty()) { + strokeHeights[strokeHeights.size / 2] + } else { + 50f // fallback + } + // Strokes within 0.75x median height of each other are on the same line + val lineGapThreshold = medianHeight * 0.75f + + val lines = mutableListOf>() + var currentLine = mutableListOf(sorted.first()) + var currentLineCenter = (sorted.first().top + sorted.first().bottom) / 2f + + for (stroke in sorted.drop(1)) { + val strokeCenter = (stroke.top + stroke.bottom) / 2f + if (strokeCenter - currentLineCenter > lineGapThreshold) { + // New line + lines.add(currentLine) + currentLine = mutableListOf(stroke) + currentLineCenter = strokeCenter + } else { + currentLine.add(stroke) + // Update running average of line center + currentLineCenter = currentLine.map { (it.top + it.bottom) / 2f }.average().toFloat() + } + } + lines.add(currentLine) + + // Sort strokes within each line left-to-right by creation time + return lines.map { line -> + line.sortedWith(compareBy { it.createdAt.time }.thenBy { it.left }) + } + } + + private fun buildInk(strokes: List): Ink { + val inkBuilder = Ink.builder() + for (stroke in strokes) { val strokeBuilder = Ink.Stroke.builder() val baseTime = stroke.createdAt.time for ((i, point) in stroke.points.withIndex()) { val t = if (point.dt != null) { baseTime + point.dt.toLong() } else { - baseTime + (i * 10L) // 10ms between points + baseTime + (i * 10L) } strokeBuilder.addPoint(Ink.Point.create(point.x, point.y, t)) } inkBuilder.addStroke(strokeBuilder.build()) } + return inkBuilder.build() + } - val ink = inkBuilder.build() - return suspendCancellableCoroutine { cont -> - rec.recognize(ink) - .addOnSuccessListener { result -> - val text = result.candidates.firstOrNull()?.text ?: "" - cont.resume(text) - } - .addOnFailureListener { cont.resumeWithException(it) } - } + /** + * Compute a WritingArea from the bounding box of a set of strokes. + * Uses the line's full width and height so ML Kit can judge relative + * character sizes (e.g. uppercase vs lowercase). + */ + private fun computeWritingArea(strokes: List): WritingArea { + val minLeft = strokes.minOf { it.left } + val maxRight = strokes.maxOf { it.right } + val minTop = strokes.minOf { it.top } + val maxBottom = strokes.maxOf { it.bottom } + val width = (maxRight - minLeft).coerceAtLeast(1f) + val height = (maxBottom - minTop).coerceAtLeast(1f) + return WritingArea(width, height) } /** From c7c21d957feba42169bd6de03c2313a104a4a0d2 Mon Sep 17 00:00:00 2001 From: Joshua Pham Date: Sun, 8 Mar 2026 23:44:26 -0400 Subject: [PATCH 06/23] Fix RecognitionContext: preContext is required, not optional RecognitionContext.builder() requires setPreContext() to be called. Always pass it (empty string for the first line). Co-Authored-By: Claude Opus 4.6 --- app/src/main/java/com/ethran/notable/io/InboxSyncEngine.kt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/app/src/main/java/com/ethran/notable/io/InboxSyncEngine.kt b/app/src/main/java/com/ethran/notable/io/InboxSyncEngine.kt index 9c9d0488..a1702144 100644 --- a/app/src/main/java/com/ethran/notable/io/InboxSyncEngine.kt +++ b/app/src/main/java/com/ethran/notable/io/InboxSyncEngine.kt @@ -120,8 +120,8 @@ object InboxSyncEngine { val ink = buildInk(lineStrokes) val writingArea = computeWritingArea(lineStrokes) val context = RecognitionContext.builder() + .setPreContext(preContext) .setWritingArea(writingArea) - .apply { if (preContext.isNotEmpty()) setPreContext(preContext) } .build() val text = suspendCancellableCoroutine { cont -> From 3573fce17f76925574ea4eceefcac7dbb7d01618 Mon Sep 17 00:00:00 2001 From: Joshua Pham Date: Sun, 8 Mar 2026 23:48:18 -0400 Subject: [PATCH 07/23] Add release signing docs and gitignore keystore files Document the signing process with env vars in CLAUDE.md. Add *.jks to .gitignore to prevent committing keystores. Co-Authored-By: Claude Opus 4.6 --- .gitignore | 1 + CLAUDE.md | 20 +++++++++++++++++++- 2 files changed, 20 insertions(+), 1 deletion(-) diff --git a/.gitignore b/.gitignore index aa724b77..698a2419 100644 --- a/.gitignore +++ b/.gitignore @@ -13,3 +13,4 @@ .externalNativeBuild .cxx local.properties +*.jks diff --git a/CLAUDE.md b/CLAUDE.md index a6c64d1d..2bfc4cd4 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -53,7 +53,25 @@ IS_NEXT=false ``` ### Build output -Debug APK is at: `app/build/outputs/apk/debug/app-debug.apk` +- Debug APK: `app/build/outputs/apk/debug/app-debug.apk` +- Release APK: `app/build/outputs/apk/release/app-release.apk` + +### Release signing +The release build uses env vars for signing (configured in `app/build.gradle`). The keystore is at `notable-release.jks` (gitignored). + +```bash +# Build signed release APK +STORE_FILE=/Users/joshuapham/Hacks/notable/notable-release.jks \ +STORE_PASSWORD=notable123 \ +KEY_ALIAS=notable \ +KEY_PASSWORD=notable123 \ +./gradlew assembleRelease + +# Install release APK (must uninstall debug build first if switching signing keys) +adb install -r app/build/outputs/apk/release/app-release.apk +``` + +**Note:** Switching between debug and release signing requires uninstalling first (`adb uninstall com.ethran.notable`) which wipes app data. ## Deploying to Onyx Boox From 9044c8b6d28753370662211c90faa55e26ff7073 Mon Sep 17 00:00:00 2001 From: Joshua Pham Date: Sun, 8 Mar 2026 23:58:30 -0400 Subject: [PATCH 08/23] Make tag suggestions reactive and add release build notes VaultTagScanner.cachedTags is now Compose mutableStateOf so the InboxToolbar automatically picks up tags when refreshCache() runs (either at startup or after changing the vault path in settings). Also add release build best practices to CLAUDE.md. Co-Authored-By: Claude Opus 4.6 --- CLAUDE.md | 5 ++++- .../java/com/ethran/notable/editor/EditorView.kt | 7 ++----- .../java/com/ethran/notable/io/VaultTagScanner.kt | 13 +++++++------ 3 files changed, 13 insertions(+), 12 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 2bfc4cd4..66860269 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -71,7 +71,10 @@ KEY_PASSWORD=notable123 \ adb install -r app/build/outputs/apk/release/app-release.apk ``` -**Note:** Switching between debug and release signing requires uninstalling first (`adb uninstall com.ethran.notable`) which wipes app data. +**Important notes:** +- Switching between debug and release signing requires uninstalling first (`adb uninstall com.ethran.notable`) which wipes app data (settings, pages, etc.) +- After installing a release APK, wait ~30s for dex2oat compilation to finish before launching. The device may show "install" prompt if the app hasn't finished compiling. A reboot can help. +- Prefer `./gradlew installDebug` for development iteration — it avoids signing key conflicts ## Deploying to Onyx Boox diff --git a/app/src/main/java/com/ethran/notable/editor/EditorView.kt b/app/src/main/java/com/ethran/notable/editor/EditorView.kt index 9bc4d873..a63d67ad 100644 --- a/app/src/main/java/com/ethran/notable/editor/EditorView.kt +++ b/app/src/main/java/com/ethran/notable/editor/EditorView.kt @@ -160,7 +160,8 @@ fun EditorView( var isInboxPage by remember { mutableStateOf(false) } var isSyncing by remember { mutableStateOf(false) } val selectedTags = remember { mutableStateListOf() } - val suggestedTags = remember { mutableStateListOf() } + // Read tags reactively — updates when VaultTagScanner.refreshCache() runs + val suggestedTags = VaultTagScanner.cachedTags LaunchedEffect(pageId) { val pageData = withContext(Dispatchers.IO) { @@ -169,10 +170,6 @@ fun EditorView( val inbox = pageData?.background == "inbox" isInboxPage = inbox editorState.isInboxPage = inbox - if (inbox) { - // Use pre-cached tags (scanned on app start) - suggestedTags.addAll(VaultTagScanner.cachedTags) - } } DisposableEffect(Unit) { diff --git a/app/src/main/java/com/ethran/notable/io/VaultTagScanner.kt b/app/src/main/java/com/ethran/notable/io/VaultTagScanner.kt index 71141c85..7c328d34 100644 --- a/app/src/main/java/com/ethran/notable/io/VaultTagScanner.kt +++ b/app/src/main/java/com/ethran/notable/io/VaultTagScanner.kt @@ -1,6 +1,7 @@ package com.ethran.notable.io import android.os.Environment +import androidx.compose.runtime.mutableStateOf import io.shipbook.shipbooksdk.ShipBook import java.io.File @@ -14,17 +15,17 @@ data class TagScore( object VaultTagScanner { - // Cached tags — pre-populated on app start - @Volatile - var cachedTags: List = emptyList() - private set + // Observable tag cache — Compose UI recomposes when this changes + private val _cachedTags = mutableStateOf>(emptyList()) + val cachedTags: List + get() = _cachedTags.value /** * Refresh the tag cache in the background. Call from app initialization. */ fun refreshCache(inboxPath: String) { - cachedTags = scanTags(inboxPath) - log.i("Tag cache refreshed: ${cachedTags.size} tags") + _cachedTags.value = scanTags(inboxPath) + log.i("Tag cache refreshed: ${_cachedTags.value.size} tags") } /** From 9153d4fff6f97f7d618929c771126ccbad7c0528 Mon Sep 17 00:00:00 2001 From: Joshua Pham Date: Mon, 9 Mar 2026 00:17:19 -0400 Subject: [PATCH 09/23] Fix fountain pen default, pressure curve, and stroke rendering consistency Three interconnected bugs prevented the fountain pen from working correctly as the default nib: 1. Race condition in pen initialization: setupSurface() calls openRawDrawing() which resets the Onyx SDK stroke style to its default. Because updatePenAndStroke() ran synchronously in surfaceCreated while updateActiveSurface() ran in a coroutine, the pen style was always overwritten before the first stroke. Fix: updatePenAndStroke() now runs inside updateActiveSurface() after setupSurface() completes. 2. Fountain pen bitmap replay rendered as ballpoint: The NeoFountainPenV2 SDK wrapper returned null PenPathResults during offline replay, producing uniform-width strokes that lost all pressure variation on any screen redraw (menu open, undo, dropdown). Fix: switched fountain pen replay to the custom drawFountainPenStroke renderer which explicitly varies stroke width per-point based on pressure. Deleted the now-unused NeoFountainPenV2Wrapper. 3. Light pressure produced invisible strokes: The linear pressure normalization (pressure/maxPressure) meant a gentle touch at ~200/4096 mapped to 0.05, barely visible. Fix: applied sqrt curve so the same touch maps to ~0.22, making light strokes clearly visible while preserving full range at heavy pressure. Refactoring: - Centralized pen defaults into Pen.DEFAULT and Pen.DEFAULT_SETTINGS - Added Pen.strokeStyle property, replacing standalone penToStroke() function - Extracted calculateExcludeHeights() from updateActiveSurface() - Removed redundant updatePenAndStroke() calls from observers that already call updateActiveSurface() - Pen.fromString() now falls back to Pen.DEFAULT instead of hardcoded BALLPEN Also adds a "Clear all pages" button in settings with confirmation dialog. Co-Authored-By: Claude Opus 4.6 --- .../editor/canvas/CanvasObserverRegistry.kt | 3 +- .../notable/editor/canvas/DrawCanvas.kt | 4 +- .../notable/editor/canvas/OnyxInputHandler.kt | 99 +++++++++---------- .../editor/drawing/NeoFountainPenV2Wrapper.kt | 93 ----------------- .../notable/editor/drawing/drawStroke.kt | 10 +- .../notable/editor/drawing/penStrokes.kt | 7 +- .../notable/editor/state/EditorState.kt | 36 ++----- .../com/ethran/notable/editor/utils/pen.kt | 43 +++++--- .../notable/ui/components/GeneralSettings.kt | 70 ++++++++++++- .../ui/viewmodels/SettingsViewModel.kt | 11 +++ .../com/ethran/notable/ui/views/Settings.kt | 4 +- 11 files changed, 169 insertions(+), 211 deletions(-) delete mode 100644 app/src/main/java/com/ethran/notable/editor/drawing/NeoFountainPenV2Wrapper.kt diff --git a/app/src/main/java/com/ethran/notable/editor/canvas/CanvasObserverRegistry.kt b/app/src/main/java/com/ethran/notable/editor/canvas/CanvasObserverRegistry.kt index 76f520d8..3659a60b 100644 --- a/app/src/main/java/com/ethran/notable/editor/canvas/CanvasObserverRegistry.kt +++ b/app/src/main/java/com/ethran/notable/editor/canvas/CanvasObserverRegistry.kt @@ -209,8 +209,8 @@ class CanvasObserverRegistry( coroutineScope.launch { snapshotFlow { state.isToolbarOpen }.drop(1).collect { logCanvasObserver.v("istoolbaropen change: ${state.isToolbarOpen}") + // updateActiveSurface reconfigures exclude rects and restores pen/stroke style inputHandler.updateActiveSurface() - inputHandler.updatePenAndStroke() refreshManager.refreshUi(null) } } @@ -218,7 +218,6 @@ class CanvasObserverRegistry( snapshotFlow { state.isInboxTagsExpanded }.drop(1).collect { logCanvasObserver.v("inbox tags expanded change: ${state.isInboxTagsExpanded}") inputHandler.updateActiveSurface() - inputHandler.updatePenAndStroke() refreshManager.refreshUi(null) } } diff --git a/app/src/main/java/com/ethran/notable/editor/canvas/DrawCanvas.kt b/app/src/main/java/com/ethran/notable/editor/canvas/DrawCanvas.kt index d816d3b1..7a75be75 100644 --- a/app/src/main/java/com/ethran/notable/editor/canvas/DrawCanvas.kt +++ b/app/src/main/java/com/ethran/notable/editor/canvas/DrawCanvas.kt @@ -106,10 +106,8 @@ class DrawCanvas( val surfaceCallback: SurfaceHolder.Callback = object : SurfaceHolder.Callback { override fun surfaceCreated(holder: SurfaceHolder) { log.i("surface created $holder") - // set up the drawing surface + // set up the drawing surface (also sets pen/stroke style after setup completes) inputHandler.updateActiveSurface() - // Restore the correct stroke size and style. - inputHandler.updatePenAndStroke() } override fun surfaceChanged( diff --git a/app/src/main/java/com/ethran/notable/editor/canvas/OnyxInputHandler.kt b/app/src/main/java/com/ethran/notable/editor/canvas/OnyxInputHandler.kt index d87efd6c..937e0321 100644 --- a/app/src/main/java/com/ethran/notable/editor/canvas/OnyxInputHandler.kt +++ b/app/src/main/java/com/ethran/notable/editor/canvas/OnyxInputHandler.kt @@ -27,7 +27,6 @@ import com.ethran.notable.editor.utils.handleScribbleToErase import com.ethran.notable.editor.utils.handleSelect import com.ethran.notable.editor.utils.onSurfaceInit import com.ethran.notable.editor.utils.partialRefreshRegionOnce -import com.ethran.notable.editor.utils.penToStroke import com.ethran.notable.editor.utils.prepareForPartialUpdate import com.ethran.notable.editor.utils.restoreDefaults import com.ethran.notable.editor.utils.setupSurface @@ -130,36 +129,33 @@ class OnyxInputHandler( fun updatePenAndStroke() { if(touchHelper == null) return - // it takes around 11 ms to run on Note 4c. - log.i("Update pen and stroke") + log.i("Update pen and stroke: pen=${state.pen}, mode=${state.mode}") when (state.mode) { - // we need to change size according to zoom level before drawing on screen - Mode.Draw, Mode.Line -> touchHelper!!.setStrokeStyle(penToStroke(state.pen)) - ?.setStrokeWidth(state.penSettings[state.pen.penName]!!.strokeSize * page.zoomLevel.value) - ?.setStrokeColor(state.penSettings[state.pen.penName]!!.color) - - Mode.Erase -> { - when (state.eraser) { - Eraser.PEN -> touchHelper!!.setStrokeStyle(penToStroke(Pen.MARKER)) - ?.setStrokeWidth(30f) - ?.setStrokeColor(Color.GRAY) - - Eraser.SELECT -> { - val dashStyleID = penToStroke(Pen.DASHED) - touchHelper!!.setStrokeStyle(dashStyleID) - ?.setStrokeWidth(3f) - ?.setStrokeColor(Color.BLACK) - val params = FloatArray(4) - params[0] = 5f // thickness - params[1] = 9f // no idea - params[2] = 9f // no idea - params[3] = 0f // no idea - Device.currentDevice().setStrokeParameters(dashStyleID, params) - } + Mode.Draw, Mode.Line -> { + val setting = state.penSettings[state.pen.penName] ?: return + touchHelper!!.setStrokeStyle(state.pen.strokeStyle) + ?.setStrokeWidth(setting.strokeSize * page.zoomLevel.value) + ?.setStrokeColor(setting.color) + } + + Mode.Erase -> when (state.eraser) { + Eraser.PEN -> touchHelper!!.setStrokeStyle(Pen.MARKER.strokeStyle) + ?.setStrokeWidth(30f) + ?.setStrokeColor(Color.GRAY) + + Eraser.SELECT -> { + touchHelper!!.setStrokeStyle(Pen.DASHED.strokeStyle) + ?.setStrokeWidth(3f) + ?.setStrokeColor(Color.BLACK) + Device.currentDevice().setStrokeParameters( + Pen.DASHED.strokeStyle, + floatArrayOf(5f, 9f, 9f, 0f) + ) } } - Mode.Select -> touchHelper?.setStrokeStyle(penToStroke(Pen.BALLPEN))?.setStrokeWidth(3f) + Mode.Select -> touchHelper?.setStrokeStyle(Pen.BALLPEN.strokeStyle) + ?.setStrokeWidth(3f) ?.setStrokeColor(Color.GRAY) } } @@ -182,32 +178,33 @@ class OnyxInputHandler( log.i("Update editable surface") coroutineScope.launch { onSurfaceInit(drawCanvas) - if (state.isInboxPage) { - val inboxToolbarHeight = if (state.isInboxTagsExpanded) { - convertDpToPixel(INBOX_TOOLBAR_EXPANDED_HEIGHT, drawCanvas.context).toInt() - } else { - convertDpToPixel(INBOX_TOOLBAR_COLLAPSED_HEIGHT, drawCanvas.context).toInt() - } - val penToolbarHeight = convertDpToPixel(40.dp, drawCanvas.context).toInt() - setupSurface( - drawCanvas, - touchHelper, - topExcludeHeight = inboxToolbarHeight, - bottomExcludeHeight = penToolbarHeight - ) + + val (topExclude, bottomExclude) = calculateExcludeHeights() + setupSurface(drawCanvas, touchHelper, topExclude, bottomExclude) + + // Must set pen/stroke AFTER setupSurface, because openRawDrawing() resets the stroke style + updatePenAndStroke() + } + } + + private fun calculateExcludeHeights(): Pair { + if (state.isInboxPage) { + val toolbarHeight = if (state.isInboxTagsExpanded) { + convertDpToPixel(INBOX_TOOLBAR_EXPANDED_HEIGHT, drawCanvas.context).toInt() } else { - val toolbarHeight = if (state.isToolbarOpen) { - convertDpToPixel(40.dp, drawCanvas.context).toInt() - } else 0 - val topExclude = if (GlobalAppSettings.current.toolbarPosition == AppSettings.Position.Top) toolbarHeight else 0 - val bottomExclude = if (GlobalAppSettings.current.toolbarPosition == AppSettings.Position.Bottom) toolbarHeight else 0 - setupSurface( - drawCanvas, - touchHelper, - topExcludeHeight = topExclude, - bottomExcludeHeight = bottomExclude - ) + convertDpToPixel(INBOX_TOOLBAR_COLLAPSED_HEIGHT, drawCanvas.context).toInt() } + val penToolbarHeight = convertDpToPixel(40.dp, drawCanvas.context).toInt() + return toolbarHeight to penToolbarHeight + } + + val toolbarHeight = if (state.isToolbarOpen) { + convertDpToPixel(40.dp, drawCanvas.context).toInt() + } else 0 + + return when (GlobalAppSettings.current.toolbarPosition) { + AppSettings.Position.Top -> toolbarHeight to 0 + AppSettings.Position.Bottom -> 0 to toolbarHeight } } private fun onRawDrawingList(plist: TouchPointList) { diff --git a/app/src/main/java/com/ethran/notable/editor/drawing/NeoFountainPenV2Wrapper.kt b/app/src/main/java/com/ethran/notable/editor/drawing/NeoFountainPenV2Wrapper.kt deleted file mode 100644 index 7a36f731..00000000 --- a/app/src/main/java/com/ethran/notable/editor/drawing/NeoFountainPenV2Wrapper.kt +++ /dev/null @@ -1,93 +0,0 @@ -package com.ethran.notable.editor.drawing - -import android.graphics.Canvas -import android.graphics.Paint -import com.onyx.android.sdk.data.note.TouchPoint -import com.onyx.android.sdk.pen.NeoFountainPenV2 -import com.onyx.android.sdk.pen.NeoPenConfig -import com.onyx.android.sdk.pen.PenPathResult -import com.onyx.android.sdk.pen.PenResult -import io.shipbook.shipbooksdk.ShipBook - -private val logger = ShipBook.getLogger("NeoFountainPenV2Wrapper") - - -object NeoFountainPenV2Wrapper { - - fun drawStroke( - canvas: Canvas, - paint: Paint, - points: List, - strokeWidth: Float, - maxTouchPressure: Float, - ) { - - if (points.size < 2) { - logger.e("Drawing strokes failed: Not enough points") - return - } - - // Normalize pressure to [0, 1] using provided maxTouchPressure - if (maxTouchPressure > 0f) { - for (i in points.indices) { - points[i].pressure /= maxTouchPressure - } - } - - val neoPenConfig = NeoPenConfig().apply { - setWidth(strokeWidth) - setTiltEnabled(true) - setMaxTouchPressure(maxTouchPressure) - } - val neoPen = NeoFountainPenV2.create(neoPenConfig) - if (neoPen == null) { - logger.e("Drawing strokes failed: Pen creation failed") - return - } - - try { - // Pen down - drawResult( - neoPen.onPenDown(points.first(), repaint = true), - canvas, - paint - ) - - // Moves (exclude first and last) - if (points.size > 2) { - drawResult( - neoPen.onPenMove( - points.subList(1, points.size - 1), - prediction = null, - repaint = true - ), - canvas, - paint - ) - } - - // Pen up - drawResult( - neoPen.onPenUp(points.last(), repaint = true), - canvas, - paint - ) - } finally { - neoPen.destroy() - } - } - - - private fun drawResult( - result: Pair?, - canvas: Canvas, - paint: Paint - ) { - val first = result?.first - if (first !is PenPathResult) { - logger.d("Expected PenPathResult but got ${first?.javaClass?.simpleName ?: "null"}") - return - } - first.draw(canvas, paint = paint) - } -} diff --git a/app/src/main/java/com/ethran/notable/editor/drawing/drawStroke.kt b/app/src/main/java/com/ethran/notable/editor/drawing/drawStroke.kt index 4269edc1..bfb0c0e6 100644 --- a/app/src/main/java/com/ethran/notable/editor/drawing/drawStroke.kt +++ b/app/src/main/java/com/ethran/notable/editor/drawing/drawStroke.kt @@ -40,15 +40,7 @@ fun drawStroke(canvas: Canvas, stroke: Stroke, offset: Offset) { Pen.GREENBALLPEN -> drawBallPenStroke(canvas, paint, stroke.size, points) Pen.BLUEBALLPEN -> drawBallPenStroke(canvas, paint, stroke.size, points) - Pen.FOUNTAIN -> { - NeoFountainPenV2Wrapper.drawStroke( - /* canvas = */ canvas, - /* paint = */ paint, - /* points = */ points, - /* strokeWidth = */ stroke.size, - /* maxTouchPressure = */ stroke.maxPressure.toFloat(), - ) - } + Pen.FOUNTAIN -> drawFountainPenStroke(canvas, paint, stroke.size, points) Pen.BRUSH -> { NeoBrushPenWrapper.drawStroke( diff --git a/app/src/main/java/com/ethran/notable/editor/drawing/penStrokes.kt b/app/src/main/java/com/ethran/notable/editor/drawing/penStrokes.kt index ee599f05..cc59e029 100644 --- a/app/src/main/java/com/ethran/notable/editor/drawing/penStrokes.kt +++ b/app/src/main/java/com/ethran/notable/editor/drawing/penStrokes.kt @@ -124,12 +124,9 @@ fun drawFountainPenStroke( path.quadTo(prePoint.x, prePoint.y, point.x, point.y) prePoint.x = point.x prePoint.y = point.y + val normalizedPressure = kotlin.math.sqrt((point.pressure / pressure).coerceIn(0f, 1f)) copyPaint.strokeWidth = - (1.5f - strokeSize / 40f) * strokeSize * (1 - cos(0.5f * 3.14f * point.pressure / pressure)) - point.tiltX - point.tiltY - point.timestamp - + (1.5f - strokeSize / 40f) * strokeSize * (1 - cos(0.5f * 3.14f * normalizedPressure)) canvas.drawPath(path, copyPaint) path.reset() path.moveTo(point.x, point.y) diff --git a/app/src/main/java/com/ethran/notable/editor/state/EditorState.kt b/app/src/main/java/com/ethran/notable/editor/state/EditorState.kt index 36c8fabb..00f99201 100644 --- a/app/src/main/java/com/ethran/notable/editor/state/EditorState.kt +++ b/app/src/main/java/com/ethran/notable/editor/state/EditorState.kt @@ -1,6 +1,5 @@ package com.ethran.notable.editor.state -import android.graphics.Color import androidx.compose.runtime.Stable import androidx.compose.runtime.getValue import androidx.compose.runtime.mutableStateOf @@ -88,39 +87,16 @@ class EditorState( private val log = ShipBook.getLogger("EditorState") - var mode by mutableStateOf(persistedEditorSettings?.mode ?: Mode.Draw) // should save - var pen by mutableStateOf(persistedEditorSettings?.pen ?: Pen.FOUNTAIN) // should save - var eraser by mutableStateOf(persistedEditorSettings?.eraser ?: Eraser.PEN) // should save - var isDrawing by mutableStateOf(true) // gives information if pen touch will be drawn or not - // For debugging: -// var isDrawing: Boolean -// get() = _isDrawing -// set(value) { -// if (_isDrawing != value) { -// Log.d(TAG, "isDrawing modified from ${_isDrawing} to $value") -// logCallStack("isDrawing modification") -// _isDrawing = value -// } -// } + var mode by mutableStateOf(persistedEditorSettings?.mode ?: Mode.Draw) + var pen by mutableStateOf(persistedEditorSettings?.pen ?: Pen.DEFAULT) + var eraser by mutableStateOf(persistedEditorSettings?.eraser ?: Eraser.PEN) + var isDrawing by mutableStateOf(true) var isInboxPage by mutableStateOf(false) var isInboxTagsExpanded by mutableStateOf(false) - var isToolbarOpen by mutableStateOf( - persistedEditorSettings?.isToolbarOpen ?: false - ) // should save - var penSettings by mutableStateOf( - persistedEditorSettings?.penSettings ?: mapOf( - Pen.BALLPEN.penName to PenSetting(5f, Color.BLACK), - Pen.REDBALLPEN.penName to PenSetting(5f, Color.RED), - Pen.BLUEBALLPEN.penName to PenSetting(5f, Color.BLUE), - Pen.GREENBALLPEN.penName to PenSetting(5f, Color.GREEN), - Pen.PENCIL.penName to PenSetting(5f, Color.BLACK), - Pen.BRUSH.penName to PenSetting(5f, Color.BLACK), - Pen.MARKER.penName to PenSetting(40f, Color.LTGRAY), - Pen.FOUNTAIN.penName to PenSetting(5f, Color.BLACK) - ) - ) + var isToolbarOpen by mutableStateOf(persistedEditorSettings?.isToolbarOpen ?: false) + var penSettings by mutableStateOf(persistedEditorSettings?.penSettings ?: Pen.DEFAULT_SETTINGS) val selectionState = SelectionState() diff --git a/app/src/main/java/com/ethran/notable/editor/utils/pen.kt b/app/src/main/java/com/ethran/notable/editor/utils/pen.kt index 359e307a..6270360b 100644 --- a/app/src/main/java/com/ethran/notable/editor/utils/pen.kt +++ b/app/src/main/java/com/ethran/notable/editor/utils/pen.kt @@ -1,6 +1,7 @@ package com.ethran.notable.editor.utils +import android.graphics.Color import com.onyx.android.sdk.pen.style.StrokeStyle import kotlinx.serialization.Serializable @@ -16,28 +17,38 @@ enum class Pen(val penName: String) { FOUNTAIN("FOUNTAIN"), DASHED("DASHED"); + val strokeStyle: Int + get() = when (this) { + BALLPEN, REDBALLPEN, GREENBALLPEN, BLUEBALLPEN -> StrokeStyle.PENCIL + PENCIL -> StrokeStyle.CHARCOAL + BRUSH -> StrokeStyle.NEO_BRUSH + MARKER -> StrokeStyle.MARKER + FOUNTAIN -> StrokeStyle.FOUNTAIN + DASHED -> StrokeStyle.DASH + } + companion object { + /** The pen selected by default on fresh install */ + val DEFAULT = FOUNTAIN + + /** Default pen settings for each pen type (fresh install) */ + val DEFAULT_SETTINGS: NamedSettings = mapOf( + BALLPEN.penName to PenSetting(5f, Color.BLACK), + REDBALLPEN.penName to PenSetting(5f, Color.RED), + BLUEBALLPEN.penName to PenSetting(5f, Color.BLUE), + GREENBALLPEN.penName to PenSetting(5f, Color.GREEN), + PENCIL.penName to PenSetting(5f, Color.BLACK), + BRUSH.penName to PenSetting(5f, Color.BLACK), + MARKER.penName to PenSetting(40f, Color.LTGRAY), + FOUNTAIN.penName to PenSetting(5f, Color.BLACK), + ) + fun fromString(name: String?): Pen { - return entries.find { it.penName.equals(name, ignoreCase = true) } ?: BALLPEN + return entries.find { it.penName.equals(name, ignoreCase = true) } ?: DEFAULT } } } -fun penToStroke(pen: Pen): Int { - return when (pen) { - Pen.BALLPEN -> StrokeStyle.PENCIL - Pen.REDBALLPEN -> StrokeStyle.PENCIL - Pen.GREENBALLPEN -> StrokeStyle.PENCIL - Pen.BLUEBALLPEN -> StrokeStyle.PENCIL - Pen.PENCIL -> StrokeStyle.CHARCOAL - Pen.BRUSH -> StrokeStyle.NEO_BRUSH - Pen.MARKER -> StrokeStyle.MARKER - Pen.FOUNTAIN -> StrokeStyle.FOUNTAIN - Pen.DASHED -> StrokeStyle.DASH - } -} - - @Serializable data class PenSetting( var strokeSize: Float, diff --git a/app/src/main/java/com/ethran/notable/ui/components/GeneralSettings.kt b/app/src/main/java/com/ethran/notable/ui/components/GeneralSettings.kt index d0fb4529..cb42ea3d 100644 --- a/app/src/main/java/com/ethran/notable/ui/components/GeneralSettings.kt +++ b/app/src/main/java/com/ethran/notable/ui/components/GeneralSettings.kt @@ -1,6 +1,9 @@ package com.ethran.notable.ui.components +import androidx.compose.foundation.background import androidx.compose.foundation.border +import androidx.compose.foundation.clickable +import androidx.compose.foundation.layout.Box import androidx.compose.foundation.layout.Column import androidx.compose.foundation.layout.Row import androidx.compose.foundation.layout.Spacer @@ -36,7 +39,9 @@ import com.ethran.notable.io.VaultTagScanner @Composable fun GeneralSettings( - settings: AppSettings, onSettingsChange: (AppSettings) -> Unit + settings: AppSettings, + onSettingsChange: (AppSettings) -> Unit, + onClearAllPages: ((onComplete: () -> Unit) -> Unit)? = null ) { Column { // Capture settings @@ -107,6 +112,69 @@ fun GeneralSettings( onToggle = { isChecked -> onSettingsChange(settings.copy(visualizePdfPagination = isChecked)) }) + + if (onClearAllPages != null) { + Spacer(modifier = Modifier.height(16.dp)) + ClearAllPagesButton(onClearAllPages) + } + } +} + +@Composable +private fun ClearAllPagesButton(onClearAllPages: (onComplete: () -> Unit) -> Unit) { + var confirmState by remember { mutableStateOf(false) } + var isClearing by remember { mutableStateOf(false) } + + if (isClearing) { + Text( + "Clearing...", + style = MaterialTheme.typography.body1, + color = Color.Gray, + modifier = Modifier.padding(vertical = 12.dp) + ) + } else if (confirmState) { + Row( + modifier = Modifier.fillMaxWidth(), + verticalAlignment = Alignment.CenterVertically + ) { + Text( + "Delete all pages, notebooks, and folders?", + style = MaterialTheme.typography.body1, + modifier = Modifier.weight(1f) + ) + Box( + modifier = Modifier + .background(Color.Black, RoundedCornerShape(6.dp)) + .clickable { + isClearing = true + onClearAllPages { + isClearing = false + confirmState = false + } + } + .padding(horizontal = 16.dp, vertical = 8.dp) + ) { + Text("Yes, delete all", color = Color.White, fontSize = 14.sp) + } + Spacer(modifier = Modifier.padding(horizontal = 4.dp)) + Box( + modifier = Modifier + .border(1.dp, Color.Gray, RoundedCornerShape(6.dp)) + .clickable { confirmState = false } + .padding(horizontal = 16.dp, vertical = 8.dp) + ) { + Text("Cancel", fontSize = 14.sp) + } + } + } else { + Box( + modifier = Modifier + .border(1.dp, Color.Red, RoundedCornerShape(6.dp)) + .clickable { confirmState = true } + .padding(horizontal = 16.dp, vertical = 10.dp) + ) { + Text("Clear all pages", color = Color.Red, fontSize = 14.sp) + } } } diff --git a/app/src/main/java/com/ethran/notable/ui/viewmodels/SettingsViewModel.kt b/app/src/main/java/com/ethran/notable/ui/viewmodels/SettingsViewModel.kt index b49e5dfe..ddd0e52d 100644 --- a/app/src/main/java/com/ethran/notable/ui/viewmodels/SettingsViewModel.kt +++ b/app/src/main/java/com/ethran/notable/ui/viewmodels/SettingsViewModel.kt @@ -10,6 +10,7 @@ import com.ethran.notable.APP_SETTINGS_KEY import com.ethran.notable.R import com.ethran.notable.data.datastore.AppSettings import com.ethran.notable.data.datastore.GlobalAppSettings +import com.ethran.notable.data.db.AppDatabase import com.ethran.notable.data.db.KvProxy import com.ethran.notable.utils.isLatestVersion import dagger.hilt.android.lifecycle.HiltViewModel @@ -30,6 +31,7 @@ data class GestureRowModel( @HiltViewModel class SettingsViewModel @Inject constructor( private val kvProxy: KvProxy, + private val db: AppDatabase, ) : ViewModel() { companion object {} @@ -68,6 +70,15 @@ class SettingsViewModel @Inject constructor( } } + fun clearAllPages(onComplete: () -> Unit) { + viewModelScope.launch(Dispatchers.IO) { + db.clearAllTables() + withContext(Dispatchers.Main) { + onComplete() + } + } + } + // ----------------- // // Gesture Settings // ----------------- // diff --git a/app/src/main/java/com/ethran/notable/ui/views/Settings.kt b/app/src/main/java/com/ethran/notable/ui/views/Settings.kt index 5fa51b64..a50d76c8 100644 --- a/app/src/main/java/com/ethran/notable/ui/views/Settings.kt +++ b/app/src/main/java/com/ethran/notable/ui/views/Settings.kt @@ -103,6 +103,7 @@ fun SettingsView( viewModel.checkUpdate(context, force) }, onUpdateSettings = { viewModel.updateSettings(it) }, + onClearAllPages = { onComplete -> viewModel.clearAllPages(onComplete) }, listOfGestures = viewModel.getGestureRows(), availableGestures = viewModel.availableGestures ) @@ -119,6 +120,7 @@ fun SettingsContent( goToInkTest: () -> Unit = {}, onCheckUpdate: (Boolean) -> Unit, onUpdateSettings: (AppSettings) -> Unit, + onClearAllPages: ((onComplete: () -> Unit) -> Unit)? = null, selectedTabInitial: Int = 0, listOfGestures: List = emptyList(), availableGestures: List> = emptyList() @@ -151,7 +153,7 @@ fun SettingsContent( .verticalScroll(rememberScrollState()) ) { when (selectedTab) { - 0 -> GeneralSettings(settings, onUpdateSettings) + 0 -> GeneralSettings(settings, onUpdateSettings, onClearAllPages) 1 -> GesturesSettings( settings, onUpdateSettings, listOfGestures, availableGestures ) From 3fc9f0848a215889afbf6b2a630f6cb65408d2b9 Mon Sep 17 00:00:00 2001 From: Joshua Pham Date: Mon, 9 Mar 2026 00:33:01 -0400 Subject: [PATCH 10/23] Capture real pen timestamps for handwriting recognition Stroke points now store the actual delta time (dt) from the Onyx SDK's TouchPoint.timestamp instead of leaving it null. Previously, the ML Kit recognizer fell back to synthetic 10ms-apart timestamps, losing all real pen dynamics (writing speed, pauses between letters, stroke duration). This data is critical for ML Kit to disambiguate characters. Co-Authored-By: Claude Opus 4.6 --- .../com/ethran/notable/editor/utils/GeometryExtensions.kt | 6 +++++- .../java/com/ethran/notable/editor/utils/handleInput.kt | 7 +++---- 2 files changed, 8 insertions(+), 5 deletions(-) diff --git a/app/src/main/java/com/ethran/notable/editor/utils/GeometryExtensions.kt b/app/src/main/java/com/ethran/notable/editor/utils/GeometryExtensions.kt index a0d0c0e3..b3a6305e 100644 --- a/app/src/main/java/com/ethran/notable/editor/utils/GeometryExtensions.kt +++ b/app/src/main/java/com/ethran/notable/editor/utils/GeometryExtensions.kt @@ -89,12 +89,16 @@ fun strokeToTouchPoints(stroke: Stroke): List { } -fun TouchPoint.toStrokePoint(scroll: Offset, scale: Float): StrokePoint { +fun TouchPoint.toStrokePoint(scroll: Offset, scale: Float, baseTimestamp: Long = 0L): StrokePoint { + val dt = if (baseTimestamp > 0L && timestamp > 0L) { + (timestamp - baseTimestamp).coerceIn(0, UShort.MAX_VALUE.toLong()).toUShort() + } else null return StrokePoint( x = x / scale + scroll.x, y = y / scale + scroll.y, pressure = pressure, tiltX = tiltX, tiltY = tiltY, + dt = dt, ) } diff --git a/app/src/main/java/com/ethran/notable/editor/utils/handleInput.kt b/app/src/main/java/com/ethran/notable/editor/utils/handleInput.kt index 4be84445..3a547a62 100644 --- a/app/src/main/java/com/ethran/notable/editor/utils/handleInput.kt +++ b/app/src/main/java/com/ethran/notable/editor/utils/handleInput.kt @@ -7,10 +7,9 @@ import com.onyx.android.sdk.data.note.TouchPoint fun copyInput(touchPoints: List, scroll: Offset, scale: Float): List { - val points = touchPoints.map { - it.toStrokePoint(scroll, scale) - } - return points + if (touchPoints.isEmpty()) return emptyList() + val baseTime = touchPoints.first().timestamp + return touchPoints.map { it.toStrokePoint(scroll, scale, baseTime) } } From eb061a2b8bc2661e99e75efb2879fb86da5c633d Mon Sep 17 00:00:00 2001 From: Joshua Pham Date: Mon, 9 Mar 2026 00:34:42 -0400 Subject: [PATCH 11/23] Fix accidental toolbar button hits on capture pages Move pen toolbar toggle to InboxToolbar at the top of the screen so it's away from the writing hand. When hidden on inbox pages, the bottom toolbar is fully removed (no leftover button to ghost-tap). Co-Authored-By: Claude Opus 4.6 --- .../com/ethran/notable/editor/EditorView.kt | 22 ++++++++++------ .../ethran/notable/editor/ui/InboxToolbar.kt | 26 +++++++++++++++++++ 2 files changed, 40 insertions(+), 8 deletions(-) diff --git a/app/src/main/java/com/ethran/notable/editor/EditorView.kt b/app/src/main/java/com/ethran/notable/editor/EditorView.kt index a63d67ad..c1e1ba55 100644 --- a/app/src/main/java/com/ethran/notable/editor/EditorView.kt +++ b/app/src/main/java/com/ethran/notable/editor/EditorView.kt @@ -231,9 +231,13 @@ fun EditorView( selectedTags = selectedTags, suggestedTags = suggestedTags, isExpanded = editorState.isInboxTagsExpanded, + isToolbarOpen = editorState.isToolbarOpen, onToggleExpanded = { editorState.isInboxTagsExpanded = !editorState.isInboxTagsExpanded }, + onToggleToolbar = { + editorState.isToolbarOpen = !editorState.isToolbarOpen + }, onTagAdd = { tag -> if (tag !in selectedTags) selectedTags.add(tag) }, @@ -264,14 +268,16 @@ fun EditorView( }, onDiscard = { navController.popBackStack() } ) - // Pen toolbar at bottom for inbox pages - Column( - Modifier - .fillMaxWidth() - .fillMaxHeight() - ) { - Spacer(modifier = Modifier.weight(1f)) - Toolbar(exportEngine, navController, appRepository, editorState, editorControlTower) + // Pen toolbar at bottom for inbox pages — only when open + if (editorState.isToolbarOpen) { + Column( + Modifier + .fillMaxWidth() + .fillMaxHeight() + ) { + Spacer(modifier = Modifier.weight(1f)) + Toolbar(exportEngine, navController, appRepository, editorState, editorControlTower) + } } } else { PositionedToolbar(exportEngine, navController, appRepository, editorState, editorControlTower) diff --git a/app/src/main/java/com/ethran/notable/editor/ui/InboxToolbar.kt b/app/src/main/java/com/ethran/notable/editor/ui/InboxToolbar.kt index 84bae98c..e5cb58b4 100644 --- a/app/src/main/java/com/ethran/notable/editor/ui/InboxToolbar.kt +++ b/app/src/main/java/com/ethran/notable/editor/ui/InboxToolbar.kt @@ -25,6 +25,7 @@ import androidx.compose.material.Text import androidx.compose.material.icons.Icons import androidx.compose.material.icons.filled.Add import androidx.compose.material.icons.filled.Close +import androidx.compose.material.icons.filled.Edit import androidx.compose.material.icons.filled.KeyboardArrowDown import androidx.compose.material.icons.filled.KeyboardArrowUp import androidx.compose.runtime.Composable @@ -51,7 +52,9 @@ fun InboxToolbar( selectedTags: List, suggestedTags: List, isExpanded: Boolean, + isToolbarOpen: Boolean, onToggleExpanded: () -> Unit, + onToggleToolbar: () -> Unit, onTagAdd: (String) -> Unit, onTagRemove: (String) -> Unit, onSave: () -> Unit, @@ -130,6 +133,29 @@ fun InboxToolbar( ) } + // Pen toolbar toggle + Box( + modifier = Modifier + .border( + 1.5.dp, + if (isToolbarOpen) Color.Black else Color.DarkGray, + RoundedCornerShape(8.dp) + ) + .background( + if (isToolbarOpen) Color.Black else Color.Transparent, + RoundedCornerShape(8.dp) + ) + .clickable { onToggleToolbar() } + .padding(horizontal = 14.dp, vertical = 8.dp) + ) { + Icon( + Icons.Default.Edit, + contentDescription = if (isToolbarOpen) "Hide pen toolbar" else "Show pen toolbar", + tint = if (isToolbarOpen) Color.White else Color.DarkGray, + modifier = Modifier.size(22.dp) + ) + } + Box( modifier = Modifier .background(Color.Black, RoundedCornerShape(8.dp)) From 3941b50771ddfb1e5e192c2b0384f3bdc0699969 Mon Sep 17 00:00:00 2001 From: Joshua Pham Date: Mon, 9 Mar 2026 00:43:13 -0400 Subject: [PATCH 12/23] Navigate back instantly on save, sync in background MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Save & Exit now returns to the library immediately instead of blocking while sync runs. A SyncState singleton with its own coroutine scope handles background sync, and the library shows a "Syncing..." overlay on pages still processing. Pages are no longer deleted after sync — they stay in the library. Co-Authored-By: Claude Opus 4.6 --- .../com/ethran/notable/editor/EditorView.kt | 43 +++--------------- .../com/ethran/notable/io/InboxSyncEngine.kt | 6 +-- .../java/com/ethran/notable/io/SyncState.kt | 44 +++++++++++++++++++ .../com/ethran/notable/ui/views/HomeView.kt | 16 +++++++ 4 files changed, 67 insertions(+), 42 deletions(-) create mode 100644 app/src/main/java/com/ethran/notable/io/SyncState.kt diff --git a/app/src/main/java/com/ethran/notable/editor/EditorView.kt b/app/src/main/java/com/ethran/notable/editor/EditorView.kt index c1e1ba55..402f6d32 100644 --- a/app/src/main/java/com/ethran/notable/editor/EditorView.kt +++ b/app/src/main/java/com/ethran/notable/editor/EditorView.kt @@ -40,6 +40,7 @@ import com.ethran.notable.editor.ui.toolbar.Toolbar import com.ethran.notable.gestures.EditorGestureReceiver import com.ethran.notable.io.ExportEngine import com.ethran.notable.io.InboxSyncEngine +import com.ethran.notable.io.SyncState import com.ethran.notable.io.VaultTagScanner import com.ethran.notable.io.exportToLinkedFile import com.ethran.notable.navigation.NavigationDestination @@ -158,7 +159,6 @@ fun EditorView( // Inbox mode detection — query DB since pageFromDb loads async var isInboxPage by remember { mutableStateOf(false) } - var isSyncing by remember { mutableStateOf(false) } val selectedTags = remember { mutableStateListOf() } // Read tags reactively — updates when VaultTagScanner.refreshCache() runs val suggestedTags = VaultTagScanner.cachedTags @@ -243,28 +243,10 @@ fun EditorView( }, onTagRemove = { tag -> selectedTags.remove(tag) }, onSave = { - if (!isSyncing) { - isSyncing = true - scope.launch(Dispatchers.IO) { - try { - InboxSyncEngine.syncInboxPage( - appRepository, pageId, selectedTags.toList() - ) - withContext(Dispatchers.Main) { - navController.popBackStack() - } - } catch (e: Exception) { - isSyncing = false - log.e("Inbox sync failed: ${e.message}", e) - SnackState.globalSnackFlow.tryEmit( - SnackConf( - text = "Inbox sync failed: ${e.message}", - duration = 4000 - ) - ) - } - } - } + SyncState.launchSync( + appRepository, pageId, selectedTags.toList() + ) + navController.popBackStack() }, onDiscard = { navController.popBackStack() } ) @@ -284,21 +266,6 @@ fun EditorView( } HorizontalScrollIndicator(state = editorState) - // Full-screen overlay while inbox sync is in progress - if (isSyncing) { - Box( - modifier = Modifier - .fillMaxSize() - .background(Color.White), - contentAlignment = Alignment.Center - ) { - Text( - text = "Saving to vault...", - fontSize = 24.sp, - color = Color.Black - ) - } - } } } } diff --git a/app/src/main/java/com/ethran/notable/io/InboxSyncEngine.kt b/app/src/main/java/com/ethran/notable/io/InboxSyncEngine.kt index a1702144..35de9660 100644 --- a/app/src/main/java/com/ethran/notable/io/InboxSyncEngine.kt +++ b/app/src/main/java/com/ethran/notable/io/InboxSyncEngine.kt @@ -57,8 +57,7 @@ object InboxSyncEngine { val strokes = pageWithStrokes.strokes if (strokes.isEmpty() && tags.isEmpty()) { - log.i("No strokes and no tags on inbox page, deleting") - appRepository.pageRepository.delete(pageId) + log.i("No strokes and no tags on inbox page, skipping sync") return } @@ -77,8 +76,7 @@ object InboxSyncEngine { val inboxPath = GlobalAppSettings.current.obsidianInboxPath writeMarkdownFile(markdown, page.createdAt, inboxPath) - appRepository.pageRepository.delete(pageId) - log.i("Inbox sync complete, page $pageId deleted") + log.i("Inbox sync complete for page $pageId") } private suspend fun ensureModelDownloaded() { diff --git a/app/src/main/java/com/ethran/notable/io/SyncState.kt b/app/src/main/java/com/ethran/notable/io/SyncState.kt new file mode 100644 index 00000000..cb6ca23d --- /dev/null +++ b/app/src/main/java/com/ethran/notable/io/SyncState.kt @@ -0,0 +1,44 @@ +package com.ethran.notable.io + +import androidx.compose.runtime.mutableStateListOf +import com.ethran.notable.data.AppRepository +import com.ethran.notable.ui.SnackConf +import com.ethran.notable.ui.SnackState +import io.shipbook.shipbooksdk.ShipBook +import kotlinx.coroutines.CoroutineScope +import kotlinx.coroutines.Dispatchers +import kotlinx.coroutines.SupervisorJob +import kotlinx.coroutines.launch + +private val log = ShipBook.getLogger("SyncState") + +object SyncState { + private val scope = CoroutineScope(SupervisorJob() + Dispatchers.IO) + val syncingPageIds = mutableStateListOf() + + fun launchSync( + appRepository: AppRepository, + pageId: String, + tags: List + ) { + if (pageId in syncingPageIds) return + syncingPageIds.add(pageId) + + scope.launch { + try { + InboxSyncEngine.syncInboxPage(appRepository, pageId, tags) + log.i("Background sync complete for page $pageId") + } catch (e: Exception) { + log.e("Background sync failed for page $pageId: ${e.message}", e) + SnackState.globalSnackFlow.tryEmit( + SnackConf( + text = "Sync failed: ${e.message}", + duration = 4000 + ) + ) + } finally { + syncingPageIds.remove(pageId) + } + } + } +} diff --git a/app/src/main/java/com/ethran/notable/ui/views/HomeView.kt b/app/src/main/java/com/ethran/notable/ui/views/HomeView.kt index 55bde8ab..c56f9085 100644 --- a/app/src/main/java/com/ethran/notable/ui/views/HomeView.kt +++ b/app/src/main/java/com/ethran/notable/ui/views/HomeView.kt @@ -55,6 +55,7 @@ import com.ethran.notable.editor.EditorDestination import com.ethran.notable.editor.ui.toolbar.Topbar import com.ethran.notable.editor.utils.autoEInkAnimationOnScroll import com.ethran.notable.io.ExportEngine +import com.ethran.notable.io.SyncState import com.ethran.notable.navigation.NavigationDestination import com.ethran.notable.ui.SnackConf import com.ethran.notable.ui.SnackState @@ -212,6 +213,7 @@ fun LibraryContent( // Existing pages items(pages) { page -> var isPageSelected by remember { mutableStateOf(false) } + val isSyncing = page.id in SyncState.syncingPageIds Box { PagePreview( modifier = Modifier @@ -223,6 +225,20 @@ fun LibraryContent( .border(1.dp, Color.Gray, RectangleShape), pageId = page.id ) + if (isSyncing) { + Box( + modifier = Modifier + .aspectRatio(3f / 4f) + .background(Color.White.copy(alpha = 0.7f)), + contentAlignment = Alignment.Center + ) { + Text( + "Syncing...", + style = androidx.compose.material.MaterialTheme.typography.caption, + color = Color.DarkGray + ) + } + } if (isPageSelected) com.ethran.notable.editor.ui.PageMenu( appRepository = appRepository, pageId = page.id, From ba4586926c0a866d8214759df4e93700ce69f548 Mon Sep 17 00:00:00 2001 From: Joshua Pham Date: Mon, 9 Mar 2026 02:19:15 -0400 Subject: [PATCH 13/23] Replace ML Kit with Onyx HWR (MyScript) for handwriting recognition Remove ML Kit Digital Ink Recognition dependency and all related code. Handwriting recognition now uses the Boox firmware's built-in MyScript engine via AIDL IPC, which produces significantly better results. - Add AIDL interface for com.onyx.android.ksync HWR service - Add OnyxHWREngine with protobuf encoding and service binding - Simplify InboxSyncEngine to use only OnyxHWR (no fallback) - Remove InkTestView debug screen and its navigation wiring - Fix CountDownLatch reuse bug on service reconnection - Make Context non-nullable in sync path (always available) - Remove redundant DB query in sync flow Co-Authored-By: Claude Opus 4.6 --- app/build.gradle | 4 +- .../sdk/hwr/service/HWRCommandArgs.aidl | 3 + .../android/sdk/hwr/service/HWRInputArgs.aidl | 3 + .../sdk/hwr/service/HWROutputArgs.aidl | 3 + .../sdk/hwr/service/HWROutputCallback.aidl | 7 + .../android/sdk/hwr/service/IHWRService.aidl | 19 + .../com/ethran/notable/editor/EditorView.kt | 2 +- .../com/ethran/notable/io/InboxSyncEngine.kt | 207 ++------- .../com/ethran/notable/io/OnyxHWREngine.kt | 400 ++++++++++++++++++ .../java/com/ethran/notable/io/SyncState.kt | 6 +- .../notable/navigation/NotableNavHost.kt | 9 - .../notable/navigation/NotableNavigator.kt | 5 - .../notable/ui/components/DebugSettings.kt | 10 +- .../ethran/notable/ui/views/InkTestView.kt | 273 ------------ .../com/ethran/notable/ui/views/Settings.kt | 5 +- .../android/sdk/hwr/service/HWRCommandArgs.kt | 17 + .../android/sdk/hwr/service/HWRInputArgs.kt | 87 ++++ .../android/sdk/hwr/service/HWROutputArgs.kt | 47 ++ 18 files changed, 623 insertions(+), 484 deletions(-) create mode 100644 app/src/main/aidl/com/onyx/android/sdk/hwr/service/HWRCommandArgs.aidl create mode 100644 app/src/main/aidl/com/onyx/android/sdk/hwr/service/HWRInputArgs.aidl create mode 100644 app/src/main/aidl/com/onyx/android/sdk/hwr/service/HWROutputArgs.aidl create mode 100644 app/src/main/aidl/com/onyx/android/sdk/hwr/service/HWROutputCallback.aidl create mode 100644 app/src/main/aidl/com/onyx/android/sdk/hwr/service/IHWRService.aidl create mode 100644 app/src/main/java/com/ethran/notable/io/OnyxHWREngine.kt delete mode 100644 app/src/main/java/com/ethran/notable/ui/views/InkTestView.kt create mode 100644 app/src/main/java/com/onyx/android/sdk/hwr/service/HWRCommandArgs.kt create mode 100644 app/src/main/java/com/onyx/android/sdk/hwr/service/HWRInputArgs.kt create mode 100644 app/src/main/java/com/onyx/android/sdk/hwr/service/HWROutputArgs.kt diff --git a/app/build.gradle b/app/build.gradle index dae88d3f..aff10d85 100644 --- a/app/build.gradle +++ b/app/build.gradle @@ -103,6 +103,7 @@ android { buildFeatures { compose true buildConfig true + aidl true } composeOptions { kotlinCompilerExtensionVersion compose_version @@ -202,9 +203,6 @@ dependencies { // for PDF support: implementation("com.artifex.mupdf:fitz:1.26.10") - // ML Kit Digital Ink Recognition - implementation 'com.google.mlkit:digital-ink-recognition:19.0.0' - // Hilt implementation "com.google.dagger:hilt-android:2.59.2" ksp "com.google.dagger:hilt-compiler:2.59.2" diff --git a/app/src/main/aidl/com/onyx/android/sdk/hwr/service/HWRCommandArgs.aidl b/app/src/main/aidl/com/onyx/android/sdk/hwr/service/HWRCommandArgs.aidl new file mode 100644 index 00000000..008e543e --- /dev/null +++ b/app/src/main/aidl/com/onyx/android/sdk/hwr/service/HWRCommandArgs.aidl @@ -0,0 +1,3 @@ +package com.onyx.android.sdk.hwr.service; + +parcelable HWRCommandArgs; diff --git a/app/src/main/aidl/com/onyx/android/sdk/hwr/service/HWRInputArgs.aidl b/app/src/main/aidl/com/onyx/android/sdk/hwr/service/HWRInputArgs.aidl new file mode 100644 index 00000000..c47a0d80 --- /dev/null +++ b/app/src/main/aidl/com/onyx/android/sdk/hwr/service/HWRInputArgs.aidl @@ -0,0 +1,3 @@ +package com.onyx.android.sdk.hwr.service; + +parcelable HWRInputArgs; diff --git a/app/src/main/aidl/com/onyx/android/sdk/hwr/service/HWROutputArgs.aidl b/app/src/main/aidl/com/onyx/android/sdk/hwr/service/HWROutputArgs.aidl new file mode 100644 index 00000000..d325ae08 --- /dev/null +++ b/app/src/main/aidl/com/onyx/android/sdk/hwr/service/HWROutputArgs.aidl @@ -0,0 +1,3 @@ +package com.onyx.android.sdk.hwr.service; + +parcelable HWROutputArgs; diff --git a/app/src/main/aidl/com/onyx/android/sdk/hwr/service/HWROutputCallback.aidl b/app/src/main/aidl/com/onyx/android/sdk/hwr/service/HWROutputCallback.aidl new file mode 100644 index 00000000..92592596 --- /dev/null +++ b/app/src/main/aidl/com/onyx/android/sdk/hwr/service/HWROutputCallback.aidl @@ -0,0 +1,7 @@ +package com.onyx.android.sdk.hwr.service; + +import com.onyx.android.sdk.hwr.service.HWROutputArgs; + +interface HWROutputCallback { + void read(in HWROutputArgs args); +} diff --git a/app/src/main/aidl/com/onyx/android/sdk/hwr/service/IHWRService.aidl b/app/src/main/aidl/com/onyx/android/sdk/hwr/service/IHWRService.aidl new file mode 100644 index 00000000..f7ec6552 --- /dev/null +++ b/app/src/main/aidl/com/onyx/android/sdk/hwr/service/IHWRService.aidl @@ -0,0 +1,19 @@ +package com.onyx.android.sdk.hwr.service; + +import android.os.ParcelFileDescriptor; +import com.onyx.android.sdk.hwr.service.HWROutputCallback; +import com.onyx.android.sdk.hwr.service.HWRInputArgs; +import com.onyx.android.sdk.hwr.service.HWRCommandArgs; + +// Method order determines transaction codes — must match the service exactly. +// init=1, compileRecognizeText=2, batchRecognize=3, openIncrementalRecognizer=4, +// execCommand=5, closeRecognizer=6 +// All methods are oneway (async) to match the service's original interface. +oneway interface IHWRService { + void init(in HWRInputArgs args, boolean forceReinit, HWROutputCallback callback); + void compileRecognizeText(String text, String language, HWROutputCallback callback); + void batchRecognize(in ParcelFileDescriptor pfd, HWROutputCallback callback); + void openIncrementalRecognizer(in HWRInputArgs args, HWROutputCallback callback); + void execCommand(in HWRInputArgs args, in HWRCommandArgs cmdArgs, HWROutputCallback callback); + void closeRecognizer(); +} diff --git a/app/src/main/java/com/ethran/notable/editor/EditorView.kt b/app/src/main/java/com/ethran/notable/editor/EditorView.kt index 402f6d32..d860bdf8 100644 --- a/app/src/main/java/com/ethran/notable/editor/EditorView.kt +++ b/app/src/main/java/com/ethran/notable/editor/EditorView.kt @@ -244,7 +244,7 @@ fun EditorView( onTagRemove = { tag -> selectedTags.remove(tag) }, onSave = { SyncState.launchSync( - appRepository, pageId, selectedTags.toList() + appRepository, pageId, selectedTags.toList(), context ) navController.popBackStack() }, diff --git a/app/src/main/java/com/ethran/notable/io/InboxSyncEngine.kt b/app/src/main/java/com/ethran/notable/io/InboxSyncEngine.kt index 35de9660..f7b2a76f 100644 --- a/app/src/main/java/com/ethran/notable/io/InboxSyncEngine.kt +++ b/app/src/main/java/com/ethran/notable/io/InboxSyncEngine.kt @@ -1,59 +1,33 @@ package com.ethran.notable.io +import android.content.Context import android.os.Environment import com.ethran.notable.data.AppRepository import com.ethran.notable.data.datastore.GlobalAppSettings -import com.ethran.notable.data.db.Stroke -import com.google.mlkit.common.model.DownloadConditions -import com.google.mlkit.common.model.RemoteModelManager -import com.google.mlkit.vision.digitalink.recognition.DigitalInkRecognition -import com.google.mlkit.vision.digitalink.recognition.DigitalInkRecognitionModel -import com.google.mlkit.vision.digitalink.recognition.DigitalInkRecognitionModelIdentifier -import com.google.mlkit.vision.digitalink.recognition.DigitalInkRecognizerOptions -import com.google.mlkit.vision.digitalink.recognition.Ink -import com.google.mlkit.vision.digitalink.recognition.RecognitionContext -import com.google.mlkit.vision.digitalink.recognition.WritingArea import io.shipbook.shipbooksdk.ShipBook -import kotlinx.coroutines.suspendCancellableCoroutine import java.io.File import java.text.SimpleDateFormat import java.util.Date import java.util.Locale -import kotlin.coroutines.resume -import kotlin.coroutines.resumeWithException private val log = ShipBook.getLogger("InboxSyncEngine") object InboxSyncEngine { - private val modelIdentifier = - DigitalInkRecognitionModelIdentifier.fromLanguageTag("en-US") - private val model = - modelIdentifier?.let { DigitalInkRecognitionModel.builder(it).build() } - private val recognizer = model?.let { - DigitalInkRecognition.getClient( - DigitalInkRecognizerOptions.builder(it).build() - ) - } - /** * Sync an inbox page to Obsidian. Tags come from the UI (pill selection), - * content is recognized from all strokes on the page. + * content is recognized from all strokes on the page via Onyx HWR (MyScript). */ suspend fun syncInboxPage( appRepository: AppRepository, pageId: String, - tags: List + tags: List, + context: Context ) { log.i("Starting inbox sync for page $pageId with tags: $tags") - val page = appRepository.pageRepository.getById(pageId) - if (page == null) { - log.e("Page $pageId not found") - return - } - val pageWithStrokes = appRepository.pageRepository.getWithStrokeById(pageId) + val page = pageWithStrokes.page val strokes = pageWithStrokes.strokes if (strokes.isEmpty() && tags.isEmpty()) { @@ -62,10 +36,30 @@ object InboxSyncEngine { } val contentText = if (strokes.isNotEmpty()) { - ensureModelDownloaded() log.i("Recognizing ${strokes.size} content strokes") - val raw = recognizeStrokes(strokes) - postProcessRecognition(raw) + try { + val serviceReady = OnyxHWREngine.bindAndAwait(context) + if (serviceReady) { + val result = OnyxHWREngine.recognizeStrokes( + strokes, + viewWidth = 1404f, + viewHeight = 1872f + ) + if (result != null) { + log.i("OnyxHWR succeeded, ${result.length} chars") + postProcessRecognition(result) + } else { + log.w("OnyxHWR returned null") + "" + } + } else { + log.w("OnyxHWR service not available") + "" + } + } catch (e: Exception) { + log.e("OnyxHWR failed: ${e.message}") + "" + } } else "" log.i("Recognized content: '${contentText.take(100)}'") @@ -79,150 +73,8 @@ object InboxSyncEngine { log.i("Inbox sync complete for page $pageId") } - private suspend fun ensureModelDownloaded() { - val m = model ?: throw IllegalStateException("ML Kit model identifier not found") - val manager = RemoteModelManager.getInstance() - - val isDownloaded = suspendCancellableCoroutine { cont -> - manager.isModelDownloaded(m) - .addOnSuccessListener { cont.resume(it) } - .addOnFailureListener { cont.resumeWithException(it) } - } - - if (!isDownloaded) { - log.i("Downloading ML Kit model...") - suspendCancellableCoroutine { cont -> - manager.download(m, DownloadConditions.Builder().build()) - .addOnSuccessListener { cont.resume(null) } - .addOnFailureListener { cont.resumeWithException(it) } - } - log.i("Model downloaded") - } - } - - /** - * Segment strokes into lines based on vertical position, recognize each - * line independently with WritingArea and pre-context, then join results. - */ - private suspend fun recognizeStrokes(strokes: List): String { - val rec = recognizer - ?: throw IllegalStateException("ML Kit recognizer not initialized") - - val lines = segmentIntoLines(strokes) - log.i("Segmented ${strokes.size} strokes into ${lines.size} lines") - - val recognizedLines = mutableListOf() - var preContext = "" - - for (lineStrokes in lines) { - val ink = buildInk(lineStrokes) - val writingArea = computeWritingArea(lineStrokes) - val context = RecognitionContext.builder() - .setPreContext(preContext) - .setWritingArea(writingArea) - .build() - - val text = suspendCancellableCoroutine { cont -> - rec.recognize(ink, context) - .addOnSuccessListener { result -> - cont.resume(result.candidates.firstOrNull()?.text ?: "") - } - .addOnFailureListener { cont.resumeWithException(it) } - } - - if (text.isNotBlank()) { - recognizedLines.add(text) - // Use last ~20 chars as pre-context for the next line - preContext = text.takeLast(20) - } - } - - return recognizedLines.joinToString("\n") - } - - /** - * Group strokes into horizontal lines by clustering on vertical midpoint. - * Strokes whose vertical centers are within half a line-height of each - * other belong to the same line. - */ - private fun segmentIntoLines(strokes: List): List> { - if (strokes.isEmpty()) return emptyList() - - // Sort by vertical midpoint, then left edge - val sorted = strokes.sortedWith( - compareBy { (it.top + it.bottom) / 2f }.thenBy { it.left } - ) - - // Estimate typical line height from median stroke height - val strokeHeights = sorted.map { it.bottom - it.top }.filter { it > 0 }.sorted() - val medianHeight = if (strokeHeights.isNotEmpty()) { - strokeHeights[strokeHeights.size / 2] - } else { - 50f // fallback - } - // Strokes within 0.75x median height of each other are on the same line - val lineGapThreshold = medianHeight * 0.75f - - val lines = mutableListOf>() - var currentLine = mutableListOf(sorted.first()) - var currentLineCenter = (sorted.first().top + sorted.first().bottom) / 2f - - for (stroke in sorted.drop(1)) { - val strokeCenter = (stroke.top + stroke.bottom) / 2f - if (strokeCenter - currentLineCenter > lineGapThreshold) { - // New line - lines.add(currentLine) - currentLine = mutableListOf(stroke) - currentLineCenter = strokeCenter - } else { - currentLine.add(stroke) - // Update running average of line center - currentLineCenter = currentLine.map { (it.top + it.bottom) / 2f }.average().toFloat() - } - } - lines.add(currentLine) - - // Sort strokes within each line left-to-right by creation time - return lines.map { line -> - line.sortedWith(compareBy { it.createdAt.time }.thenBy { it.left }) - } - } - - private fun buildInk(strokes: List): Ink { - val inkBuilder = Ink.builder() - for (stroke in strokes) { - val strokeBuilder = Ink.Stroke.builder() - val baseTime = stroke.createdAt.time - for ((i, point) in stroke.points.withIndex()) { - val t = if (point.dt != null) { - baseTime + point.dt.toLong() - } else { - baseTime + (i * 10L) - } - strokeBuilder.addPoint(Ink.Point.create(point.x, point.y, t)) - } - inkBuilder.addStroke(strokeBuilder.build()) - } - return inkBuilder.build() - } - - /** - * Compute a WritingArea from the bounding box of a set of strokes. - * Uses the line's full width and height so ML Kit can judge relative - * character sizes (e.g. uppercase vs lowercase). - */ - private fun computeWritingArea(strokes: List): WritingArea { - val minLeft = strokes.minOf { it.left } - val maxRight = strokes.maxOf { it.right } - val minTop = strokes.minOf { it.top } - val maxBottom = strokes.maxOf { it.bottom } - val width = (maxRight - minLeft).coerceAtLeast(1f) - val height = (maxBottom - minTop).coerceAtLeast(1f) - return WritingArea(width, height) - } - /** - * Post-process ML Kit recognition output: + * Post-process recognition output: * - Normalize any bracket/paren wrapping to [[wiki links]] * - Collapse space between # and the following word into a proper #tag */ @@ -230,7 +82,6 @@ object InboxSyncEngine { var result = text // Normalize bracket/paren wrapping to [[wiki links]] - // Handles: [text], ((text)), ([text]), [(text)], [[text]], etc. result = result.replace(Regex("""[(\[]{1,2}([^)\]\n]+?)[)\]]{1,2}""")) { match -> "[[${match.groupValues[1].trim()}]]" } diff --git a/app/src/main/java/com/ethran/notable/io/OnyxHWREngine.kt b/app/src/main/java/com/ethran/notable/io/OnyxHWREngine.kt new file mode 100644 index 00000000..28d9db06 --- /dev/null +++ b/app/src/main/java/com/ethran/notable/io/OnyxHWREngine.kt @@ -0,0 +1,400 @@ +package com.ethran.notable.io + +import android.content.ComponentName +import android.content.Context +import android.content.Intent +import android.content.ServiceConnection +import android.os.IBinder +import android.os.MemoryFile +import android.os.ParcelFileDescriptor +import com.ethran.notable.data.db.Stroke +import com.onyx.android.sdk.hwr.service.HWRInputArgs +import com.onyx.android.sdk.hwr.service.HWROutputArgs +import com.onyx.android.sdk.hwr.service.HWROutputCallback +import com.onyx.android.sdk.hwr.service.IHWRService +import io.shipbook.shipbooksdk.ShipBook +import kotlinx.coroutines.suspendCancellableCoroutine +import org.json.JSONObject +import java.io.ByteArrayOutputStream +import java.io.FileDescriptor +import kotlin.coroutines.resume +import kotlin.coroutines.resumeWithException + +private val log = ShipBook.getLogger("OnyxHWREngine") + +object OnyxHWREngine { + + @Volatile private var service: IHWRService? = null + @Volatile private var bound = false + @Volatile private var connectLatch = java.util.concurrent.CountDownLatch(1) + + private val connection = object : ServiceConnection { + override fun onServiceConnected(name: ComponentName?, binder: IBinder?) { + service = IHWRService.Stub.asInterface(binder) + bound = true + log.i("OnyxHWR service connected") + connectLatch.countDown() + } + + override fun onServiceDisconnected(name: ComponentName?) { + service = null + bound = false + initialized = false + log.w("OnyxHWR service disconnected") + } + } + + /** + * Bind to the service and wait for connection (up to timeout). + * Returns true if service is ready. + */ + suspend fun bindAndAwait(context: Context, timeoutMs: Long = 2000): Boolean { + if (bound && service != null) return true + + // Create a fresh latch for this bind attempt + connectLatch = java.util.concurrent.CountDownLatch(1) + + val intent = Intent().apply { + component = ComponentName( + "com.onyx.android.ksync", + "com.onyx.android.ksync.service.KHwrService" + ) + } + val bindStarted = try { + context.bindService(intent, connection, Context.BIND_AUTO_CREATE) + } catch (e: Exception) { + log.w("Failed to bind OnyxHWR service: ${e.message}") + return false + } + if (!bindStarted) return false + + // Wait for onServiceConnected on IO dispatcher + return kotlinx.coroutines.withContext(kotlinx.coroutines.Dispatchers.IO) { + connectLatch.await(timeoutMs, java.util.concurrent.TimeUnit.MILLISECONDS) + } && service != null + } + + fun unbind(context: Context) { + if (bound) { + try { + context.unbindService(connection) + } catch (_: Exception) {} + bound = false + service = null + initialized = false + } + } + + @Volatile private var initialized = false + + /** + * Initialize the MyScript recognizer via the service's init method. + */ + private suspend fun ensureInitialized(svc: IHWRService, viewWidth: Float, viewHeight: Float) { + if (initialized) return + log.i("Initializing OnyxHWR recognizer...") + + val inputArgs = HWRInputArgs().apply { + lang = "en_US" + contentType = "Text" + recognizerType = "MS_ON_SCREEN" + this.viewWidth = viewWidth + this.viewHeight = viewHeight + isTextEnable = true + } + + suspendCancellableCoroutine { cont -> + svc.init(inputArgs, true, object : HWROutputCallback.Stub() { + override fun read(args: HWROutputArgs?) { + log.i("OnyxHWR init: recognizerActivated=${args?.recognizerActivated}, compileSuccess=${args?.compileSuccess}") + initialized = args?.recognizerActivated == true + cont.resume(Unit) + } + }) + } + log.i("OnyxHWR initialized=$initialized") + } + + /** + * Recognize strokes using the Onyx HWR (MyScript) service. + * Returns recognized text, or null if service is unavailable. + */ + suspend fun recognizeStrokes( + strokes: List, + viewWidth: Float, + viewHeight: Float + ): String? { + val svc = service ?: return null + if (strokes.isEmpty()) return "" + + ensureInitialized(svc, viewWidth, viewHeight) + + val protoBytes = buildProtobuf(strokes, viewWidth, viewHeight) + val pfd = createMemoryFilePfd(protoBytes) + ?: throw IllegalStateException("Failed to create MemoryFile PFD") + + return try { + suspendCancellableCoroutine { cont -> + svc.batchRecognize(pfd, object : HWROutputCallback.Stub() { + override fun read(args: HWROutputArgs?) { + try { + // Check for error in hwrResult first + val errorJson = args?.hwrResult + if (!errorJson.isNullOrBlank()) { + log.e("OnyxHWR error: ${errorJson.take(300)}") + cont.resume("") + return + } + + // Success: result is in PFD as JSON + val resultPfd = args?.pfd + if (resultPfd == null) { + log.w("OnyxHWR returned no PFD and no hwrResult") + cont.resume("") + return + } + + val json = readPfdAsString(resultPfd) + resultPfd.close() + val text = parseHwrResult(json) + log.i("OnyxHWR recognized ${text.length} chars") + cont.resume(text) + } catch (e: Exception) { + log.e("Error parsing OnyxHWR result: ${e.message}") + cont.resumeWithException(e) + } + } + }) + } + } finally { + pfd.close() + } + } + + /** + * Read a PFD's contents as a UTF-8 string. + */ + private fun readPfdAsString(pfd: ParcelFileDescriptor): String { + val input = java.io.FileInputStream(pfd.fileDescriptor) + val buffered = java.io.BufferedInputStream(input) + val baos = ByteArrayOutputStream() + val buf = ByteArray(4096) + var n: Int + while (buffered.read(buf).also { n = it } != -1) { + baos.write(buf, 0, n) + } + return baos.toString("UTF-8") + } + + /** + * Parse the JSON result from the HWR service. + * Success response: HWROutputData JSON with result.label containing recognized text. + */ + private fun parseHwrResult(json: String): String { + return try { + val obj = JSONObject(json) + // Check for error + if (obj.has("exception")) { + val exc = obj.optJSONObject("exception") + val cause = exc?.optJSONObject("cause") + val msg = cause?.optString("message") ?: exc?.optString("message") ?: "unknown" + log.e("OnyxHWR error response: $msg") + return "" + } + // Success: result is HWRConvertBean with label field + val result = obj.optJSONObject("result") + if (result != null) { + return result.optString("label", "") + } + // Fallback: try top-level label + obj.optString("label", "") + } catch (e: Exception) { + log.w("Failed to parse HWR JSON: ${e.message}") + "" + } + } + + // --- Protobuf encoding (hand-rolled, no library needed) --- + + /** + * Build the HWRInputProto protobuf bytes. + * Field numbers from HWRInputDataProto.HWRInputProto: + * 1: lang (string), 2: contentType (string), 3: editorType (string), + * 4: recognizerType (string), 5: viewWidth (float), 6: viewHeight (float), + * 7: offsetX (float), 8: offsetY (float), 9: gestureEnable (bool), + * 10: recognizeText (bool), 11: recognizeShape (bool), 12: isIncremental (bool), + * 15: repeated pointerEvents (HWRPointerProto) + */ + private fun buildProtobuf( + strokes: List, + viewWidth: Float, + viewHeight: Float + ): ByteArray { + val out = ByteArrayOutputStream() + + // Field 1: lang (string) + writeTag(out, 1, 2) + writeString(out, "en_US") + + // Field 2: contentType (string) + writeTag(out, 2, 2) + writeString(out, "Text") + + // Field 4: recognizerType (string) + writeTag(out, 4, 2) + writeString(out, "MS_ON_SCREEN") + + // Field 5: viewWidth (float, wire type 5 = fixed32) + writeTag(out, 5, 5) + writeFixed32(out, viewWidth) + + // Field 6: viewHeight (float, wire type 5 = fixed32) + writeTag(out, 6, 5) + writeFixed32(out, viewHeight) + + // Field 10: recognizeText = true (varint 1) + writeTag(out, 10, 0) + writeVarint(out, 1) + + // Field 15: repeated pointer events (wire type 2 = length-delimited) + for (stroke in strokes) { + val points = stroke.points + if (points.isEmpty()) continue + + val strokeEpoch = stroke.createdAt.time + + for ((i, point) in points.withIndex()) { + val eventType = when (i) { + 0 -> 0 // DOWN + points.size - 1 -> 2 // UP + else -> 1 // MOVE + } + + val timestamp = if (point.dt != null) { + strokeEpoch + point.dt.toLong() + } else { + strokeEpoch + (i * 10L) + } + + val pressure = point.pressure ?: 0.5f + + val pointerBytes = encodePointerProto( + x = point.x, + y = point.y, + t = timestamp, + f = pressure, + pointerId = 0, + eventType = eventType, + pointerType = 0 // PEN (0=PEN, 1=TOUCH, 2=ERASER, 3=MOUSE) + ) + + writeTag(out, 15, 2) + writeBytes(out, pointerBytes) + } + } + + return out.toByteArray() + } + + /** + * Encode a single HWRPointerProto message. + * Fields: float x(1), float y(2), sint64 t(3), float f(4), + * sint32 pointerId(5), enum eventType(6), enum pointerType(7) + */ + private fun encodePointerProto( + x: Float, y: Float, t: Long, f: Float, + pointerId: Int, eventType: Int, pointerType: Int + ): ByteArray { + val out = ByteArrayOutputStream() + + // Field 1: x (wire type 5 = fixed32) + writeTag(out, 1, 5) + writeFixed32(out, x) + + // Field 2: y (wire type 5 = fixed32) + writeTag(out, 2, 5) + writeFixed32(out, y) + + // Field 3: t (wire type 0, sint64 = ZigZag encoded) + writeTag(out, 3, 0) + writeVarint(out, (t shl 1) xor (t shr 63)) + + // Field 4: f / pressure (wire type 5 = fixed32) + writeTag(out, 4, 5) + writeFixed32(out, f) + + // Field 5: pointerId (wire type 0, sint32 = ZigZag encoded) + writeTag(out, 5, 0) + val zigzagPid = (pointerId shl 1) xor (pointerId shr 31) + writeVarint(out, zigzagPid.toLong()) + + // Field 6: eventType (wire type 0 = enum/varint) + writeTag(out, 6, 0) + writeVarint(out, eventType.toLong()) + + // Field 7: pointerType (wire type 0 = enum/varint) + writeTag(out, 7, 0) + writeVarint(out, pointerType.toLong()) + + return out.toByteArray() + } + + // --- Low-level protobuf primitives --- + + private fun writeTag(out: ByteArrayOutputStream, fieldNumber: Int, wireType: Int) { + writeVarint(out, ((fieldNumber shl 3) or wireType).toLong()) + } + + private fun writeVarint(out: ByteArrayOutputStream, value: Long) { + var v = value + while (v and 0x7FL.inv() != 0L) { + out.write(((v.toInt() and 0x7F) or 0x80)) + v = v ushr 7 + } + out.write(v.toInt() and 0x7F) + } + + private fun writeFixed32(out: ByteArrayOutputStream, value: Float) { + val bits = java.lang.Float.floatToIntBits(value) + out.write(bits and 0xFF) + out.write((bits shr 8) and 0xFF) + out.write((bits shr 16) and 0xFF) + out.write((bits shr 24) and 0xFF) + } + + private fun writeString(out: ByteArrayOutputStream, value: String) { + val bytes = value.toByteArray(Charsets.UTF_8) + writeVarint(out, bytes.size.toLong()) + out.write(bytes) + } + + private fun writeBytes(out: ByteArrayOutputStream, bytes: ByteArray) { + writeVarint(out, bytes.size.toLong()) + out.write(bytes) + } + + // --- MemoryFile → ParcelFileDescriptor --- + + /** + * Write bytes to a MemoryFile and return a ParcelFileDescriptor. + * Uses reflection to access MemoryFile.getFileDescriptor() (hidden API). + * This matches the approach used by Onyx's own MemoryFileUtils. + */ + private fun createMemoryFilePfd(data: ByteArray): ParcelFileDescriptor? { + return try { + val memFile = MemoryFile("hwr_input", data.size) + memFile.writeBytes(data, 0, 0, data.size) + + val method = MemoryFile::class.java.getDeclaredMethod("getFileDescriptor") + method.isAccessible = true + val fd = method.invoke(memFile) as FileDescriptor + + val pfd = ParcelFileDescriptor.dup(fd) + memFile.close() + pfd + } catch (e: Exception) { + log.e("Failed to create MemoryFile PFD: ${e.message}") + null + } + } +} diff --git a/app/src/main/java/com/ethran/notable/io/SyncState.kt b/app/src/main/java/com/ethran/notable/io/SyncState.kt index cb6ca23d..314fca27 100644 --- a/app/src/main/java/com/ethran/notable/io/SyncState.kt +++ b/app/src/main/java/com/ethran/notable/io/SyncState.kt @@ -1,5 +1,6 @@ package com.ethran.notable.io +import android.content.Context import androidx.compose.runtime.mutableStateListOf import com.ethran.notable.data.AppRepository import com.ethran.notable.ui.SnackConf @@ -19,14 +20,15 @@ object SyncState { fun launchSync( appRepository: AppRepository, pageId: String, - tags: List + tags: List, + context: Context ) { if (pageId in syncingPageIds) return syncingPageIds.add(pageId) scope.launch { try { - InboxSyncEngine.syncInboxPage(appRepository, pageId, tags) + InboxSyncEngine.syncInboxPage(appRepository, pageId, tags, context) log.i("Background sync complete for page $pageId") } catch (e: Exception) { log.e("Background sync failed for page $pageId: ${e.message}", e) diff --git a/app/src/main/java/com/ethran/notable/navigation/NotableNavHost.kt b/app/src/main/java/com/ethran/notable/navigation/NotableNavHost.kt index a86a754c..7774e67d 100644 --- a/app/src/main/java/com/ethran/notable/navigation/NotableNavHost.kt +++ b/app/src/main/java/com/ethran/notable/navigation/NotableNavHost.kt @@ -22,8 +22,6 @@ import com.ethran.notable.ui.views.Library import com.ethran.notable.ui.views.LibraryDestination import com.ethran.notable.ui.views.PagesDestination import com.ethran.notable.ui.views.PagesView -import com.ethran.notable.ui.views.InkTestDestination -import com.ethran.notable.ui.views.InkTestView import com.ethran.notable.ui.views.SettingsDestination import com.ethran.notable.ui.views.SettingsView import com.ethran.notable.ui.views.SystemInformationDestination @@ -136,7 +134,6 @@ fun NotableNavHost( onBack = { appNavigator.goBack() }, goToWelcome = { appNavigator.goToWelcome() }, goToSystemInfo = { appNavigator.goToSystemInfo() }, - goToInkTest = { appNavigator.goToInkTest() } ) appNavigator.cleanCurrentPageId() } @@ -146,12 +143,6 @@ fun NotableNavHost( BugReportScreen(goBack = { appNavigator.goBack() }) appNavigator.cleanCurrentPageId() } - composable( - route = InkTestDestination.route, - ) { - InkTestView(onBack = { appNavigator.goBack() }) - appNavigator.cleanCurrentPageId() - } } } } \ No newline at end of file diff --git a/app/src/main/java/com/ethran/notable/navigation/NotableNavigator.kt b/app/src/main/java/com/ethran/notable/navigation/NotableNavigator.kt index a629c756..133de57b 100644 --- a/app/src/main/java/com/ethran/notable/navigation/NotableNavigator.kt +++ b/app/src/main/java/com/ethran/notable/navigation/NotableNavigator.kt @@ -18,7 +18,6 @@ import com.ethran.notable.editor.canvas.CanvasEventBus import com.ethran.notable.editor.utils.refreshScreen import com.ethran.notable.ui.views.LibraryDestination import com.ethran.notable.ui.views.SystemInformationDestination -import com.ethran.notable.ui.views.InkTestDestination import com.ethran.notable.ui.views.WelcomeDestination import com.ethran.notable.utils.hasFilePermission import io.shipbook.shipbooksdk.ShipBook @@ -128,10 +127,6 @@ class NotableNavigator( navController.navigate(SystemInformationDestination.route) } - fun goToInkTest() { - navController.navigate(InkTestDestination.route) - } - fun goToPage(appRepository: AppRepository, pageId: String) { coroutineScope.launch { val bookId = runCatching { diff --git a/app/src/main/java/com/ethran/notable/ui/components/DebugSettings.kt b/app/src/main/java/com/ethran/notable/ui/components/DebugSettings.kt index 6a032bf5..394a2739 100644 --- a/app/src/main/java/com/ethran/notable/ui/components/DebugSettings.kt +++ b/app/src/main/java/com/ethran/notable/ui/components/DebugSettings.kt @@ -9,8 +9,7 @@ fun DebugSettings( settings: AppSettings, onSettingsChange: (AppSettings) -> Unit, goToWelcome: () -> Unit, - goToSystemInfo: () -> Unit, - goToInkTest: () -> Unit = {} + goToSystemInfo: () -> Unit ) { Column { SettingToggleRow( @@ -62,12 +61,5 @@ fun DebugSettings( onSettingsChange(settings.copy(destructiveMigrations = isChecked)) } ) - SettingToggleRow( - label = "Ink Recognition Test", - value = false, - onToggle = { - goToInkTest() - } - ) } } diff --git a/app/src/main/java/com/ethran/notable/ui/views/InkTestView.kt b/app/src/main/java/com/ethran/notable/ui/views/InkTestView.kt deleted file mode 100644 index f72ba4a7..00000000 --- a/app/src/main/java/com/ethran/notable/ui/views/InkTestView.kt +++ /dev/null @@ -1,273 +0,0 @@ -package com.ethran.notable.ui.views - -import android.graphics.Path -import android.view.MotionEvent -import androidx.compose.foundation.Canvas -import androidx.compose.foundation.background -import androidx.compose.foundation.border -import androidx.compose.foundation.clickable -import androidx.compose.foundation.layout.Arrangement -import androidx.compose.foundation.layout.Box -import androidx.compose.foundation.layout.Column -import androidx.compose.foundation.layout.Row -import androidx.compose.foundation.layout.Spacer -import androidx.compose.foundation.layout.fillMaxSize -import androidx.compose.foundation.layout.fillMaxWidth -import androidx.compose.foundation.layout.height -import androidx.compose.foundation.layout.padding -import androidx.compose.foundation.rememberScrollState -import androidx.compose.foundation.verticalScroll -import androidx.compose.material.Text -import androidx.compose.runtime.Composable -import androidx.compose.runtime.DisposableEffect -import androidx.compose.runtime.getValue -import androidx.compose.runtime.mutableLongStateOf -import androidx.compose.runtime.mutableStateListOf -import androidx.compose.runtime.mutableStateOf -import androidx.compose.runtime.remember -import androidx.compose.runtime.setValue -import androidx.compose.ui.Alignment -import androidx.compose.ui.ExperimentalComposeUiApi -import androidx.compose.ui.Modifier -import androidx.compose.ui.graphics.Color -import androidx.compose.ui.graphics.nativeCanvas -import androidx.compose.ui.input.pointer.pointerInteropFilter -import androidx.compose.ui.unit.dp -import androidx.compose.ui.unit.sp -import com.google.mlkit.common.model.DownloadConditions -import com.google.mlkit.common.model.RemoteModelManager -import com.google.mlkit.vision.digitalink.recognition.DigitalInkRecognition -import com.google.mlkit.vision.digitalink.recognition.DigitalInkRecognitionModel -import com.google.mlkit.vision.digitalink.recognition.DigitalInkRecognitionModelIdentifier -import com.google.mlkit.vision.digitalink.recognition.DigitalInkRecognizerOptions -import com.google.mlkit.vision.digitalink.recognition.Ink -import com.ethran.notable.navigation.NavigationDestination - -object InkTestDestination : NavigationDestination { - override val route = "ink_test" -} - -@OptIn(ExperimentalComposeUiApi::class) -@Composable -fun InkTestView(onBack: () -> Unit) { - val inkBuilder = remember { Ink.builder() } - var currentStrokeBuilder by remember { mutableStateOf(null) } - val paths = remember { mutableStateListOf() } - var currentPath by remember { mutableStateOf(null) } - - var recognizedText by remember { mutableStateOf("(write something and tap Recognize)") } - var modelStatus by remember { mutableStateOf("Checking model...") } - var recognitionTimeMs by remember { mutableLongStateOf(0L) } - - val modelIdentifier = remember { - DigitalInkRecognitionModelIdentifier.fromLanguageTag("en-US") - } - val model = remember { - modelIdentifier?.let { DigitalInkRecognitionModel.builder(it).build() } - } - val recognizer = remember { - model?.let { - DigitalInkRecognition.getClient( - DigitalInkRecognizerOptions.builder(it).build() - ) - } - } - - // Download model on first load - DisposableEffect(model) { - if (model != null) { - val remoteModelManager = RemoteModelManager.getInstance() - remoteModelManager.isModelDownloaded(model).addOnSuccessListener { isDownloaded -> - if (isDownloaded) { - modelStatus = "Model ready" - } else { - modelStatus = "Downloading model..." - remoteModelManager.download(model, DownloadConditions.Builder().build()) - .addOnSuccessListener { modelStatus = "Model ready" } - .addOnFailureListener { modelStatus = "Download failed: ${it.message}" } - } - } - } else { - modelStatus = "Model identifier not found" - } - onDispose { } - } - - Column( - modifier = Modifier - .fillMaxSize() - .background(Color.White) - .padding(16.dp) - ) { - // Header - Row( - modifier = Modifier.fillMaxWidth(), - horizontalArrangement = Arrangement.SpaceBetween, - verticalAlignment = Alignment.CenterVertically - ) { - Text("ML Kit Ink Test", fontSize = 20.sp, color = Color.Black) - Text( - "< Back", - fontSize = 16.sp, - color = Color.Black, - modifier = Modifier.clickable { onBack() } - ) - } - - Text(modelStatus, fontSize = 12.sp, color = Color.Gray) - Spacer(modifier = Modifier.height(8.dp)) - - // Drawing area - Box( - modifier = Modifier - .fillMaxWidth() - .height(300.dp) - .border(1.dp, Color.Black) - .background(Color.White) - ) { - Canvas( - modifier = Modifier - .fillMaxSize() - .pointerInteropFilter { event -> - when (event.action) { - MotionEvent.ACTION_DOWN -> { - val strokeBuilder = Ink.Stroke.builder() - strokeBuilder.addPoint( - Ink.Point.create( - event.x, - event.y, - event.eventTime - ) - ) - currentStrokeBuilder = strokeBuilder - val path = Path() - path.moveTo(event.x, event.y) - currentPath = path - true - } - - MotionEvent.ACTION_MOVE -> { - currentStrokeBuilder?.addPoint( - Ink.Point.create( - event.x, - event.y, - event.eventTime - ) - ) - currentPath?.lineTo(event.x, event.y) - // Force recomposition - val p = currentPath - currentPath = null - currentPath = p - true - } - - MotionEvent.ACTION_UP -> { - currentStrokeBuilder?.addPoint( - Ink.Point.create( - event.x, - event.y, - event.eventTime - ) - ) - currentStrokeBuilder?.let { inkBuilder.addStroke(it.build()) } - currentStrokeBuilder = null - currentPath?.let { paths.add(it) } - currentPath = null - true - } - - else -> false - } - } - ) { - val paint = android.graphics.Paint().apply { - color = android.graphics.Color.BLACK - style = android.graphics.Paint.Style.STROKE - strokeWidth = 4f - isAntiAlias = true - } - paths.forEach { path -> - drawContext.canvas.nativeCanvas.drawPath(path, paint) - } - currentPath?.let { path -> - drawContext.canvas.nativeCanvas.drawPath(path, paint) - } - } - } - - Spacer(modifier = Modifier.height(12.dp)) - - // Buttons - Row( - modifier = Modifier.fillMaxWidth(), - horizontalArrangement = Arrangement.spacedBy(12.dp) - ) { - Box( - modifier = Modifier - .border(1.dp, Color.Black) - .clickable { - if (recognizer != null && modelStatus == "Model ready") { - recognizedText = "Recognizing..." - val ink = inkBuilder.build() - val start = System.currentTimeMillis() - recognizer.recognize(ink) - .addOnSuccessListener { result -> - val elapsed = System.currentTimeMillis() - start - recognitionTimeMs = elapsed - val candidates = result.candidates - recognizedText = if (candidates.isNotEmpty()) { - candidates.joinToString("\n") { candidate -> "\"${candidate.text}\"" } - } else { - "(no results)" - } - } - .addOnFailureListener { ex -> - recognizedText = "Error: ${ex.localizedMessage}" - } - } - } - .padding(horizontal = 16.dp, vertical = 8.dp) - ) { - Text("Recognize", fontSize = 16.sp, color = Color.Black) - } - - Box( - modifier = Modifier - .border(1.dp, Color.Black) - .clickable { - paths.clear() - currentPath = null - currentStrokeBuilder = null - // Reset ink builder by creating a fresh one — we'll - // just rebuild in the recognizer call - recognizedText = "(cleared)" - } - .padding(horizontal = 16.dp, vertical = 8.dp) - ) { - Text("Clear", fontSize = 16.sp, color = Color.Black) - } - } - - Spacer(modifier = Modifier.height(12.dp)) - - if (recognitionTimeMs > 0) { - Text("Recognition time: ${recognitionTimeMs}ms", fontSize = 12.sp, color = Color.Gray) - } - - Spacer(modifier = Modifier.height(4.dp)) - - // Results - Text("Results:", fontSize = 14.sp, color = Color.Black) - Box( - modifier = Modifier - .fillMaxWidth() - .weight(1f) - .border(1.dp, Color.LightGray) - .padding(8.dp) - .verticalScroll(rememberScrollState()) - ) { - Text(recognizedText, fontSize = 16.sp, color = Color.Black) - } - } -} diff --git a/app/src/main/java/com/ethran/notable/ui/views/Settings.kt b/app/src/main/java/com/ethran/notable/ui/views/Settings.kt index a50d76c8..68630c0b 100644 --- a/app/src/main/java/com/ethran/notable/ui/views/Settings.kt +++ b/app/src/main/java/com/ethran/notable/ui/views/Settings.kt @@ -77,7 +77,6 @@ fun SettingsView( onBack: () -> Unit, goToWelcome: () -> Unit, goToSystemInfo: () -> Unit, - goToInkTest: () -> Unit = {}, viewModel: SettingsViewModel = hiltViewModel() ) { val context = LocalContext.current @@ -98,7 +97,6 @@ fun SettingsView( onBack = onBack, goToWelcome = goToWelcome, goToSystemInfo = goToSystemInfo, - goToInkTest = goToInkTest, onCheckUpdate = { force -> viewModel.checkUpdate(context, force) }, @@ -117,7 +115,6 @@ fun SettingsContent( onBack: () -> Unit, goToWelcome: () -> Unit, goToSystemInfo: () -> Unit, - goToInkTest: () -> Unit = {}, onCheckUpdate: (Boolean) -> Unit, onUpdateSettings: (AppSettings) -> Unit, onClearAllPages: ((onComplete: () -> Unit) -> Unit)? = null, @@ -158,7 +155,7 @@ fun SettingsContent( settings, onUpdateSettings, listOfGestures, availableGestures ) - 2 -> DebugSettings(settings, onUpdateSettings, goToWelcome, goToSystemInfo, goToInkTest) + 2 -> DebugSettings(settings, onUpdateSettings, goToWelcome, goToSystemInfo) } } diff --git a/app/src/main/java/com/onyx/android/sdk/hwr/service/HWRCommandArgs.kt b/app/src/main/java/com/onyx/android/sdk/hwr/service/HWRCommandArgs.kt new file mode 100644 index 00000000..ce6546f8 --- /dev/null +++ b/app/src/main/java/com/onyx/android/sdk/hwr/service/HWRCommandArgs.kt @@ -0,0 +1,17 @@ +package com.onyx.android.sdk.hwr.service + +import android.os.Parcel +import android.os.Parcelable + +/** Stub parcelable required by AIDL — not used in batchRecognize path. */ +class HWRCommandArgs() : Parcelable { + constructor(parcel: Parcel) : this() + + override fun writeToParcel(parcel: Parcel, flags: Int) {} + override fun describeContents(): Int = 0 + + companion object CREATOR : Parcelable.Creator { + override fun createFromParcel(parcel: Parcel) = HWRCommandArgs(parcel) + override fun newArray(size: Int) = arrayOfNulls(size) + } +} diff --git a/app/src/main/java/com/onyx/android/sdk/hwr/service/HWRInputArgs.kt b/app/src/main/java/com/onyx/android/sdk/hwr/service/HWRInputArgs.kt new file mode 100644 index 00000000..53cd0157 --- /dev/null +++ b/app/src/main/java/com/onyx/android/sdk/hwr/service/HWRInputArgs.kt @@ -0,0 +1,87 @@ +package com.onyx.android.sdk.hwr.service + +import android.os.Parcel +import android.os.ParcelFileDescriptor +import android.os.Parcelable + +/** + * Parcelable matching the ksync service's HWRInputArgs. + * Field order: inputData (Parcelable), pfd (Parcelable), content (String). + * + * inputData is written as a Parcelable with class name + * "com.onyx.android.sdk.hwr.bean.HWRInputData" so the service's classloader + * can deserialize it. We manually write the same fields. + */ +class HWRInputArgs() : Parcelable { + // HWRInputData fields + var lang: String = "en_US" + var contentType: String = "Text" + var recognizerType: String = "Text" + var viewWidth: Float = 0f + var viewHeight: Float = 0f + var offsetX: Float = 0f + var offsetY: Float = 0f + var isGestureEnable: Boolean = false + var isTextEnable: Boolean = true + var isShapeEnable: Boolean = false + var isIncremental: Boolean = false + + // HWRInputArgs fields + var pfd: ParcelFileDescriptor? = null + var content: String? = null + + constructor(parcel: Parcel) : this() { + // Read inputData as Parcelable (skip class name + fields) + val className = parcel.readString() + if (className != null) { + lang = parcel.readString() ?: "en_US" + contentType = parcel.readString() ?: "Text" + recognizerType = parcel.readString() ?: "Text" + viewWidth = parcel.readFloat() + viewHeight = parcel.readFloat() + offsetX = parcel.readFloat() + offsetY = parcel.readFloat() + isGestureEnable = parcel.readByte() != 0.toByte() + isTextEnable = parcel.readByte() != 0.toByte() + isShapeEnable = parcel.readByte() != 0.toByte() + isIncremental = parcel.readByte() != 0.toByte() + } + pfd = parcel.readParcelable(ParcelFileDescriptor::class.java.classLoader) + content = parcel.readString() + } + + override fun writeToParcel(parcel: Parcel, flags: Int) { + // Write inputData as a Parcelable: class name then fields + // This matches Parcel.writeParcelable format + parcel.writeString("com.onyx.android.sdk.hwr.bean.HWRInputData") + parcel.writeString(lang) + parcel.writeString(contentType) + parcel.writeString(recognizerType) + parcel.writeFloat(viewWidth) + parcel.writeFloat(viewHeight) + parcel.writeFloat(offsetX) + parcel.writeFloat(offsetY) + parcel.writeByte(if (isGestureEnable) 1 else 0) + parcel.writeByte(if (isTextEnable) 1 else 0) + parcel.writeByte(if (isShapeEnable) 1 else 0) + parcel.writeByte(if (isIncremental) 1 else 0) + + // Write pfd as Parcelable + if (pfd != null) { + parcel.writeString("android.os.ParcelFileDescriptor") + pfd!!.writeToParcel(parcel, flags) + } else { + parcel.writeString(null) + } + + // Write content + parcel.writeString(content) + } + + override fun describeContents(): Int = 0 + + companion object CREATOR : Parcelable.Creator { + override fun createFromParcel(parcel: Parcel) = HWRInputArgs(parcel) + override fun newArray(size: Int) = arrayOfNulls(size) + } +} diff --git a/app/src/main/java/com/onyx/android/sdk/hwr/service/HWROutputArgs.kt b/app/src/main/java/com/onyx/android/sdk/hwr/service/HWROutputArgs.kt new file mode 100644 index 00000000..612c9180 --- /dev/null +++ b/app/src/main/java/com/onyx/android/sdk/hwr/service/HWROutputArgs.kt @@ -0,0 +1,47 @@ +package com.onyx.android.sdk.hwr.service + +import android.os.Parcel +import android.os.ParcelFileDescriptor +import android.os.Parcelable + +/** + * Parcelable matching the KHwrService output format. + * Field order must match the service's writeToParcel exactly: + * pfd, recognizerActivated, compileSuccess, hwrResult, gesture(null), outputType, itemIdMap + */ +class HWROutputArgs() : Parcelable { + var pfd: ParcelFileDescriptor? = null + var recognizerActivated: Boolean = false + var compileSuccess: Boolean = false + var hwrResult: String? = null + var gesture: String? = null + var outputType: Int = 0 + var itemIdMap: String? = null + + constructor(parcel: Parcel) : this() { + pfd = parcel.readParcelable(ParcelFileDescriptor::class.java.classLoader) + recognizerActivated = parcel.readByte() != 0.toByte() + compileSuccess = parcel.readByte() != 0.toByte() + hwrResult = parcel.readString() + gesture = parcel.readString() + outputType = parcel.readInt() + itemIdMap = parcel.readString() + } + + override fun writeToParcel(parcel: Parcel, flags: Int) { + parcel.writeParcelable(pfd, flags) + parcel.writeByte(if (recognizerActivated) 1 else 0) + parcel.writeByte(if (compileSuccess) 1 else 0) + parcel.writeString(hwrResult) + parcel.writeString(gesture) + parcel.writeInt(outputType) + parcel.writeString(itemIdMap) + } + + override fun describeContents(): Int = 0 + + companion object CREATOR : Parcelable.Creator { + override fun createFromParcel(parcel: Parcel) = HWROutputArgs(parcel) + override fun newArray(size: Int) = arrayOfNulls(size) + } +} From e20d0a54ca140c7dc90905cb936181150fd17ae1 Mon Sep 17 00:00:00 2001 From: Joshua Pham Date: Mon, 9 Mar 2026 17:54:54 -0400 Subject: [PATCH 14/23] Replace custom stroke rendering with Jetpack Ink API Swap out manual Path-based pen rendering (ballpen, fountain, brush, marker, pencil) and Onyx Neo*PenWrapper calls with androidx.ink's CanvasStrokeRenderer. Removes ~200 lines of custom drawing code. Onyx SDK live input path is unchanged. Co-Authored-By: Claude Opus 4.6 --- app/build.gradle | 8 ++ .../notable/editor/drawing/InkBrushes.kt | 87 ++++++++++++ .../notable/editor/drawing/drawStroke.kt | 83 ++--------- .../notable/editor/drawing/penStrokes.kt | 133 +----------------- 4 files changed, 108 insertions(+), 203 deletions(-) create mode 100644 app/src/main/java/com/ethran/notable/editor/drawing/InkBrushes.kt diff --git a/app/build.gradle b/app/build.gradle index aff10d85..75b90ae5 100644 --- a/app/build.gradle +++ b/app/build.gradle @@ -203,6 +203,14 @@ dependencies { // for PDF support: implementation("com.artifex.mupdf:fitz:1.26.10") + // Jetpack Ink API — stroke rendering + geometry + def ink_version = "1.0.0" + implementation "androidx.ink:ink-nativeloader:$ink_version" + implementation "androidx.ink:ink-brush:$ink_version" + implementation "androidx.ink:ink-geometry:$ink_version" + implementation "androidx.ink:ink-rendering:$ink_version" + implementation "androidx.ink:ink-strokes:$ink_version" + // Hilt implementation "com.google.dagger:hilt-android:2.59.2" ksp "com.google.dagger:hilt-compiler:2.59.2" diff --git a/app/src/main/java/com/ethran/notable/editor/drawing/InkBrushes.kt b/app/src/main/java/com/ethran/notable/editor/drawing/InkBrushes.kt new file mode 100644 index 00000000..d7a52890 --- /dev/null +++ b/app/src/main/java/com/ethran/notable/editor/drawing/InkBrushes.kt @@ -0,0 +1,87 @@ +package com.ethran.notable.editor.drawing + +import androidx.compose.ui.geometry.Offset +import androidx.ink.brush.Brush +import androidx.ink.brush.InputToolType +import androidx.ink.brush.StockBrushes +import androidx.ink.rendering.android.canvas.CanvasStrokeRenderer +import androidx.ink.strokes.MutableStrokeInputBatch +import androidx.ink.strokes.Stroke as InkStroke +import com.ethran.notable.data.db.Stroke +import com.ethran.notable.editor.utils.Pen +import com.ethran.notable.editor.utils.offsetStroke + +/** + * Singleton renderer — thread-safe, reuse across draw calls. + */ +val inkRenderer: CanvasStrokeRenderer by lazy { CanvasStrokeRenderer.create() } + +// Cache brush families (they're lazy-computed internally, but let's not re-invoke every stroke) +private val markerFamily by lazy { StockBrushes.marker() } +private val pressurePenFamily by lazy { StockBrushes.pressurePen() } +private val highlighterFamily by lazy { StockBrushes.highlighter() } +private val dashedLineFamily by lazy { StockBrushes.dashedLine() } + +/** + * Map our Pen types to Ink API BrushFamily + create a Brush with the stroke's color/size. + */ +fun brushForStroke(stroke: Stroke): Brush { + val family = when (stroke.pen) { + Pen.BALLPEN, Pen.REDBALLPEN, Pen.GREENBALLPEN, Pen.BLUEBALLPEN -> + markerFamily + Pen.FOUNTAIN -> + pressurePenFamily + Pen.BRUSH -> + pressurePenFamily + Pen.PENCIL -> + markerFamily + Pen.MARKER -> + highlighterFamily + Pen.DASHED -> + dashedLineFamily + } + return Brush.createWithColorIntArgb( + family = family, + colorIntArgb = stroke.color, + size = stroke.size, + epsilon = 0.1f + ) +} + +// Spacing between synthetic timestamps when dt is missing (ms per point). +// 5ms ≈ 200Hz stylus sample rate — realistic enough for Ink API's stroke smoothing. +private const val SYNTHETIC_DT_MS = 5L + +/** + * Convert our Stroke data model to an Ink API Stroke for rendering. + */ +fun Stroke.toInkStroke(offset: Offset = Offset.Zero): InkStroke { + val src = if (offset != Offset.Zero) offsetStroke(this, offset) else this + val batch = MutableStrokeInputBatch() + val points = src.points + if (points.isEmpty()) { + return InkStroke(brushForStroke(this), batch.toImmutable()) + } + + for (i in points.indices) { + val pt = points[i] + // Use real dt if available, otherwise synthesize realistic timing + val elapsedMs = pt.dt?.toLong() ?: (i * SYNTHETIC_DT_MS) + val pressure = pt.pressure?.let { it / maxPressure.toFloat() } ?: 0.5f + // tiltRadians must be in [0, π/2] or -1 (unset). Our tiltX is degrees [-90, 90]. + val tiltRad = pt.tiltX?.let { + Math.toRadians(it.toDouble()).toFloat().coerceIn(0f, Math.PI.toFloat() / 2f) + } ?: -1f + + batch.add( + type = InputToolType.STYLUS, + x = pt.x, + y = pt.y, + elapsedTimeMillis = elapsedMs, + pressure = pressure.coerceIn(0f, 1f), + tiltRadians = tiltRad, + orientationRadians = -1f // not captured by our hardware + ) + } + return InkStroke(brushForStroke(this), batch) +} diff --git a/app/src/main/java/com/ethran/notable/editor/drawing/drawStroke.kt b/app/src/main/java/com/ethran/notable/editor/drawing/drawStroke.kt index bfb0c0e6..45689d00 100644 --- a/app/src/main/java/com/ethran/notable/editor/drawing/drawStroke.kt +++ b/app/src/main/java/com/ethran/notable/editor/drawing/drawStroke.kt @@ -2,88 +2,29 @@ package com.ethran.notable.editor.drawing import android.graphics.Canvas import android.graphics.Matrix -import android.graphics.Paint import androidx.compose.ui.geometry.Offset import com.ethran.notable.data.db.Stroke -import com.ethran.notable.editor.utils.Pen -import com.ethran.notable.editor.utils.offsetStroke -import com.ethran.notable.editor.utils.strokeToTouchPoints -import com.ethran.notable.ui.SnackState -import com.onyx.android.sdk.data.note.ShapeCreateArgs -import com.onyx.android.sdk.pen.NeoBrushPenWrapper -import com.onyx.android.sdk.pen.NeoCharcoalPenWrapper -import com.onyx.android.sdk.pen.NeoMarkerPenWrapper -import com.onyx.android.sdk.pen.PenRenderArgs import io.shipbook.shipbooksdk.ShipBook private val strokeDrawingLogger = ShipBook.getLogger("drawStroke") - +private val identityMatrix = Matrix() fun drawStroke(canvas: Canvas, stroke: Stroke, offset: Offset) { - //canvas.save() - //canvas.translate(offset.x.toFloat(), offset.y.toFloat()) - - val paint = Paint().apply { - color = stroke.color - this.strokeWidth = stroke.size - } - - val points = strokeToTouchPoints(offsetStroke(stroke, offset)) + if (stroke.points.isEmpty()) return - // Trying to find what throws error when drawing quickly try { - when (stroke.pen) { - Pen.BALLPEN -> drawBallPenStroke(canvas, paint, stroke.size, points) - Pen.REDBALLPEN -> drawBallPenStroke(canvas, paint, stroke.size, points) - Pen.GREENBALLPEN -> drawBallPenStroke(canvas, paint, stroke.size, points) - Pen.BLUEBALLPEN -> drawBallPenStroke(canvas, paint, stroke.size, points) - - Pen.FOUNTAIN -> drawFountainPenStroke(canvas, paint, stroke.size, points) - - Pen.BRUSH -> { - NeoBrushPenWrapper.drawStroke( - canvas, - paint, - points, - stroke.size, - stroke.maxPressure.toFloat(), - false - ) - } - - Pen.MARKER -> { - NeoMarkerPenWrapper.drawStroke( - canvas, - paint, - points, - stroke.size, - false - ) - } - - Pen.PENCIL -> { - val shapeArg = ShapeCreateArgs() - val arg = PenRenderArgs() - .setCanvas(canvas) - .setPaint(paint) - .setPoints(points) - .setColor(stroke.color) - .setStrokeWidth(stroke.size) - .setTiltEnabled(true) - .setErase(false) - .setCreateArgs(shapeArg) - .setRenderMatrix(Matrix()) - .setScreenMatrix(Matrix()) - NeoCharcoalPenWrapper.drawNormalStroke(arg) - } - else -> { - SnackState.logAndShowError("drawStroke", "Unknown pen type: ${stroke.pen}") - } - } + val inkStroke = stroke.toInkStroke(offset) + inkRenderer.draw( + canvas = canvas, + stroke = inkStroke, + strokeToScreenTransform = identityMatrix + ) } catch (e: Exception) { - strokeDrawingLogger.e("Drawing strokes failed: ${e.message}") + strokeDrawingLogger.e( + "Drawing stroke failed: id=${stroke.id} pen=${stroke.pen} " + + "points=${stroke.points.size} size=${stroke.size}: ${e.message}" + ) } - //canvas.restore() } diff --git a/app/src/main/java/com/ethran/notable/editor/drawing/penStrokes.kt b/app/src/main/java/com/ethran/notable/editor/drawing/penStrokes.kt index cc59e029..f52a5123 100644 --- a/app/src/main/java/com/ethran/notable/editor/drawing/penStrokes.kt +++ b/app/src/main/java/com/ethran/notable/editor/drawing/penStrokes.kt @@ -1,139 +1,8 @@ package com.ethran.notable.editor.drawing -import android.graphics.Canvas import android.graphics.Color import android.graphics.DashPathEffect import android.graphics.Paint -import android.graphics.Path -import android.graphics.PointF -import android.graphics.PorterDuff -import android.graphics.PorterDuffXfermode -import com.ethran.notable.data.db.StrokePoint -import com.ethran.notable.data.model.SimplePointF -import com.ethran.notable.editor.canvas.pressure -import com.ethran.notable.editor.utils.pointsToPath -import com.onyx.android.sdk.data.note.TouchPoint -import io.shipbook.shipbooksdk.ShipBook -import kotlin.math.abs -import kotlin.math.cos - -private val penStrokesLog = ShipBook.getLogger("PenStrokesLog") - - -fun drawBallPenStroke( - canvas: Canvas, paint: Paint, strokeSize: Float, points: List -) { - val copyPaint = Paint(paint).apply { - this.strokeWidth = strokeSize - this.style = Paint.Style.STROKE - this.strokeCap = Paint.Cap.ROUND - this.strokeJoin = Paint.Join.ROUND - - this.isAntiAlias = true - } - - val path = Path() - val prePoint = PointF(points[0].x, points[0].y) - path.moveTo(prePoint.x, prePoint.y) - - for (point in points) { - // skip strange jump point. - if (abs(prePoint.y - point.y) >= 30) continue - path.quadTo(prePoint.x, prePoint.y, point.x, point.y) - prePoint.x = point.x - prePoint.y = point.y - } - try { - canvas.drawPath(path, copyPaint) - } catch (e: Exception) { - penStrokesLog.e("Exception during draw", e) - } -} - -val eraserPaint = Paint().apply { - style = Paint.Style.STROKE - strokeCap = Paint.Cap.ROUND - strokeJoin = Paint.Join.ROUND - color = Color.BLACK - xfermode = PorterDuffXfermode(PorterDuff.Mode.SRC) - isAntiAlias = false -} -private val reusablePath = Path() -fun drawEraserStroke(canvas: Canvas, points: List, strokeSize: Float) { - eraserPaint.strokeWidth = strokeSize - - reusablePath.reset() - if (points.isEmpty()) return - - val prePoint = PointF(points[0].x, points[0].y) - reusablePath.moveTo(prePoint.x, prePoint.y) - - for (i in 1 until points.size) { - val point = points[i] - if (abs(prePoint.y - point.y) >= 30) continue - reusablePath.quadTo(prePoint.x, prePoint.y, point.x, point.y) - prePoint.x = point.x - prePoint.y = point.y - } - - try { - canvas.drawPath(reusablePath, eraserPaint) - } catch (e: Exception) { - penStrokesLog.e("Exception during draw", e) - } -} - - -fun drawMarkerStroke( - canvas: Canvas, paint: Paint, strokeSize: Float, points: List -) { - val copyPaint = Paint(paint).apply { - this.strokeWidth = strokeSize - this.style = Paint.Style.STROKE - this.strokeCap = Paint.Cap.ROUND - this.strokeJoin = Paint.Join.ROUND - this.isAntiAlias = true - this.alpha = 100 - - } - - val path = pointsToPath(points.map { SimplePointF(it.x, it.y) }) - - canvas.drawPath(path, copyPaint) -} - -fun drawFountainPenStroke( - canvas: Canvas, paint: Paint, strokeSize: Float, points: List -) { - val copyPaint = Paint(paint).apply { - this.strokeWidth = strokeSize - this.style = Paint.Style.STROKE - this.strokeCap = Paint.Cap.ROUND - this.strokeJoin = Paint.Join.ROUND -// this.blendMode = BlendMode.OVERLAY - this.isAntiAlias = true - } - - val path = Path() - val prePoint = PointF(points[0].x, points[0].y) - path.moveTo(prePoint.x, prePoint.y) - - for (point in points) { - // skip strange jump point. - if (abs(prePoint.y - point.y) >= 30) continue - path.quadTo(prePoint.x, prePoint.y, point.x, point.y) - prePoint.x = point.x - prePoint.y = point.y - val normalizedPressure = kotlin.math.sqrt((point.pressure / pressure).coerceIn(0f, 1f)) - copyPaint.strokeWidth = - (1.5f - strokeSize / 40f) * strokeSize * (1 - cos(0.5f * 3.14f * normalizedPressure)) - canvas.drawPath(path, copyPaint) - path.reset() - path.moveTo(point.x, point.y) - } -} - - val selectPaint = Paint().apply { strokeWidth = 5f @@ -141,4 +10,4 @@ val selectPaint = Paint().apply { pathEffect = DashPathEffect(floatArrayOf(20f, 10f), 0f) isAntiAlias = true color = Color.GRAY -} \ No newline at end of file +} From e8de6393804a9abb33094ad2f6350598ca4719c8 Mon Sep 17 00:00:00 2001 From: Joshua Pham Date: Mon, 9 Mar 2026 18:36:21 -0400 Subject: [PATCH 15/23] Replace bottom toolbar with left-edge sidebar, add annotation box system - Move all editor tools (pen, eraser, lasso, line, undo/redo, image, home, menu) from bottom toolbar to a vertically-centered left-edge sidebar - Pen picker flyout with size/color options appears to the right of sidebar - Eraser flyout with pen/select eraser and scribble-to-erase toggle - Add wiki link [[]] and tag # annotation buttons to sidebar - Annotation mode: tap [[ or #, then draw a box over text to create an annotation - Annotations render as semi-transparent colored overlays (blue for wikilink, green for tag) - New Annotation entity in Room DB (v35) with type, bounding box, and page FK - Annotation data flows through PageDataManager, PageView, and AppRepository - One-shot annotation: mode resets after each box is drawn - Remove PositionedToolbar and bottom toolbar from EditorView Co-Authored-By: Claude Opus 4.6 --- .../35.json | 596 +++++++++++++++ .../com/ethran/notable/data/AppRepository.kt | 2 + .../ethran/notable/data/PageDataManager.kt | 18 + .../com/ethran/notable/data/db/Annotation.kt | 74 ++ .../java/com/ethran/notable/data/db/Db.kt | 12 +- .../com/ethran/notable/editor/EditorView.kt | 61 +- .../com/ethran/notable/editor/PageView.kt | 21 + .../notable/editor/canvas/OnyxInputHandler.kt | 36 +- .../notable/editor/drawing/pageDrawing.kt | 53 ++ .../notable/editor/state/EditorState.kt | 5 + .../notable/editor/ui/AnnotationSidebar.kt | 112 +++ .../ethran/notable/editor/ui/EditorSidebar.kt | 718 ++++++++++++++++++ .../notable/editor/ui/toolbar/Toolbar.kt | 27 - .../com/ethran/notable/editor/utils/draw.kt | 37 + 14 files changed, 1674 insertions(+), 98 deletions(-) create mode 100644 app/schemas/com.ethran.notable.data.db.AppDatabase/35.json create mode 100644 app/src/main/java/com/ethran/notable/data/db/Annotation.kt create mode 100644 app/src/main/java/com/ethran/notable/editor/ui/AnnotationSidebar.kt create mode 100644 app/src/main/java/com/ethran/notable/editor/ui/EditorSidebar.kt diff --git a/app/schemas/com.ethran.notable.data.db.AppDatabase/35.json b/app/schemas/com.ethran.notable.data.db.AppDatabase/35.json new file mode 100644 index 00000000..fed1cd4b --- /dev/null +++ b/app/schemas/com.ethran.notable.data.db.AppDatabase/35.json @@ -0,0 +1,596 @@ +{ + "formatVersion": 1, + "database": { + "version": 35, + "identityHash": "e833f66629726de6b438770c17eb6f83", + "entities": [ + { + "tableName": "Folder", + "createSql": "CREATE TABLE IF NOT EXISTS `${TABLE_NAME}` (`id` TEXT NOT NULL, `title` TEXT NOT NULL, `parentFolderId` TEXT, `createdAt` INTEGER NOT NULL, `updatedAt` INTEGER NOT NULL, PRIMARY KEY(`id`), FOREIGN KEY(`parentFolderId`) REFERENCES `Folder`(`id`) ON UPDATE NO ACTION ON DELETE CASCADE )", + "fields": [ + { + "fieldPath": "id", + "columnName": "id", + "affinity": "TEXT", + "notNull": true + }, + { + "fieldPath": "title", + "columnName": "title", + "affinity": "TEXT", + "notNull": true + }, + { + "fieldPath": "parentFolderId", + "columnName": "parentFolderId", + "affinity": "TEXT" + }, + { + "fieldPath": "createdAt", + "columnName": "createdAt", + "affinity": "INTEGER", + "notNull": true + }, + { + "fieldPath": "updatedAt", + "columnName": "updatedAt", + "affinity": "INTEGER", + "notNull": true + } + ], + "primaryKey": { + "autoGenerate": false, + "columnNames": [ + "id" + ] + }, + "indices": [ + { + "name": "index_Folder_parentFolderId", + "unique": false, + "columnNames": [ + "parentFolderId" + ], + "orders": [], + "createSql": "CREATE INDEX IF NOT EXISTS `index_Folder_parentFolderId` ON `${TABLE_NAME}` (`parentFolderId`)" + } + ], + "foreignKeys": [ + { + "table": "Folder", + "onDelete": "CASCADE", + "onUpdate": "NO ACTION", + "columns": [ + "parentFolderId" + ], + "referencedColumns": [ + "id" + ] + } + ] + }, + { + "tableName": "Notebook", + "createSql": "CREATE TABLE IF NOT EXISTS `${TABLE_NAME}` (`id` TEXT NOT NULL, `title` TEXT NOT NULL, `openPageId` TEXT, `pageIds` TEXT NOT NULL, `parentFolderId` TEXT, `defaultBackground` TEXT NOT NULL DEFAULT 'blank', `defaultBackgroundType` TEXT NOT NULL DEFAULT 'native', `linkedExternalUri` TEXT, `createdAt` INTEGER NOT NULL, `updatedAt` INTEGER NOT NULL, PRIMARY KEY(`id`), FOREIGN KEY(`parentFolderId`) REFERENCES `Folder`(`id`) ON UPDATE NO ACTION ON DELETE CASCADE )", + "fields": [ + { + "fieldPath": "id", + "columnName": "id", + "affinity": "TEXT", + "notNull": true + }, + { + "fieldPath": "title", + "columnName": "title", + "affinity": "TEXT", + "notNull": true + }, + { + "fieldPath": "openPageId", + "columnName": "openPageId", + "affinity": "TEXT" + }, + { + "fieldPath": "pageIds", + "columnName": "pageIds", + "affinity": "TEXT", + "notNull": true + }, + { + "fieldPath": "parentFolderId", + "columnName": "parentFolderId", + "affinity": "TEXT" + }, + { + "fieldPath": "defaultBackground", + "columnName": "defaultBackground", + "affinity": "TEXT", + "notNull": true, + "defaultValue": "'blank'" + }, + { + "fieldPath": "defaultBackgroundType", + "columnName": "defaultBackgroundType", + "affinity": "TEXT", + "notNull": true, + "defaultValue": "'native'" + }, + { + "fieldPath": "linkedExternalUri", + "columnName": "linkedExternalUri", + "affinity": "TEXT" + }, + { + "fieldPath": "createdAt", + "columnName": "createdAt", + "affinity": "INTEGER", + "notNull": true + }, + { + "fieldPath": "updatedAt", + "columnName": "updatedAt", + "affinity": "INTEGER", + "notNull": true + } + ], + "primaryKey": { + "autoGenerate": false, + "columnNames": [ + "id" + ] + }, + "indices": [ + { + "name": "index_Notebook_parentFolderId", + "unique": false, + "columnNames": [ + "parentFolderId" + ], + "orders": [], + "createSql": "CREATE INDEX IF NOT EXISTS `index_Notebook_parentFolderId` ON `${TABLE_NAME}` (`parentFolderId`)" + } + ], + "foreignKeys": [ + { + "table": "Folder", + "onDelete": "CASCADE", + "onUpdate": "NO ACTION", + "columns": [ + "parentFolderId" + ], + "referencedColumns": [ + "id" + ] + } + ] + }, + { + "tableName": "Page", + "createSql": "CREATE TABLE IF NOT EXISTS `${TABLE_NAME}` (`id` TEXT NOT NULL, `scroll` INTEGER NOT NULL, `notebookId` TEXT, `background` TEXT NOT NULL DEFAULT 'blank', `backgroundType` TEXT NOT NULL DEFAULT 'native', `parentFolderId` TEXT, `createdAt` INTEGER NOT NULL, `updatedAt` INTEGER NOT NULL, PRIMARY KEY(`id`), FOREIGN KEY(`parentFolderId`) REFERENCES `Folder`(`id`) ON UPDATE NO ACTION ON DELETE CASCADE , FOREIGN KEY(`notebookId`) REFERENCES `Notebook`(`id`) ON UPDATE NO ACTION ON DELETE CASCADE )", + "fields": [ + { + "fieldPath": "id", + "columnName": "id", + "affinity": "TEXT", + "notNull": true + }, + { + "fieldPath": "scroll", + "columnName": "scroll", + "affinity": "INTEGER", + "notNull": true + }, + { + "fieldPath": "notebookId", + "columnName": "notebookId", + "affinity": "TEXT" + }, + { + "fieldPath": "background", + "columnName": "background", + "affinity": "TEXT", + "notNull": true, + "defaultValue": "'blank'" + }, + { + "fieldPath": "backgroundType", + "columnName": "backgroundType", + "affinity": "TEXT", + "notNull": true, + "defaultValue": "'native'" + }, + { + "fieldPath": "parentFolderId", + "columnName": "parentFolderId", + "affinity": "TEXT" + }, + { + "fieldPath": "createdAt", + "columnName": "createdAt", + "affinity": "INTEGER", + "notNull": true + }, + { + "fieldPath": "updatedAt", + "columnName": "updatedAt", + "affinity": "INTEGER", + "notNull": true + } + ], + "primaryKey": { + "autoGenerate": false, + "columnNames": [ + "id" + ] + }, + "indices": [ + { + "name": "index_Page_notebookId", + "unique": false, + "columnNames": [ + "notebookId" + ], + "orders": [], + "createSql": "CREATE INDEX IF NOT EXISTS `index_Page_notebookId` ON `${TABLE_NAME}` (`notebookId`)" + }, + { + "name": "index_Page_parentFolderId", + "unique": false, + "columnNames": [ + "parentFolderId" + ], + "orders": [], + "createSql": "CREATE INDEX IF NOT EXISTS `index_Page_parentFolderId` ON `${TABLE_NAME}` (`parentFolderId`)" + } + ], + "foreignKeys": [ + { + "table": "Folder", + "onDelete": "CASCADE", + "onUpdate": "NO ACTION", + "columns": [ + "parentFolderId" + ], + "referencedColumns": [ + "id" + ] + }, + { + "table": "Notebook", + "onDelete": "CASCADE", + "onUpdate": "NO ACTION", + "columns": [ + "notebookId" + ], + "referencedColumns": [ + "id" + ] + } + ] + }, + { + "tableName": "Stroke", + "createSql": "CREATE TABLE IF NOT EXISTS `${TABLE_NAME}` (`id` TEXT NOT NULL, `size` REAL NOT NULL, `pen` TEXT NOT NULL, `color` INTEGER NOT NULL DEFAULT 0xFF000000, `maxPressure` INTEGER NOT NULL DEFAULT 4096, `top` REAL NOT NULL, `bottom` REAL NOT NULL, `left` REAL NOT NULL, `right` REAL NOT NULL, `points` BLOB NOT NULL, `pageId` TEXT NOT NULL, `createdAt` INTEGER NOT NULL, `updatedAt` INTEGER NOT NULL, PRIMARY KEY(`id`), FOREIGN KEY(`pageId`) REFERENCES `Page`(`id`) ON UPDATE NO ACTION ON DELETE CASCADE )", + "fields": [ + { + "fieldPath": "id", + "columnName": "id", + "affinity": "TEXT", + "notNull": true + }, + { + "fieldPath": "size", + "columnName": "size", + "affinity": "REAL", + "notNull": true + }, + { + "fieldPath": "pen", + "columnName": "pen", + "affinity": "TEXT", + "notNull": true + }, + { + "fieldPath": "color", + "columnName": "color", + "affinity": "INTEGER", + "notNull": true, + "defaultValue": "0xFF000000" + }, + { + "fieldPath": "maxPressure", + "columnName": "maxPressure", + "affinity": "INTEGER", + "notNull": true, + "defaultValue": "4096" + }, + { + "fieldPath": "top", + "columnName": "top", + "affinity": "REAL", + "notNull": true + }, + { + "fieldPath": "bottom", + "columnName": "bottom", + "affinity": "REAL", + "notNull": true + }, + { + "fieldPath": "left", + "columnName": "left", + "affinity": "REAL", + "notNull": true + }, + { + "fieldPath": "right", + "columnName": "right", + "affinity": "REAL", + "notNull": true + }, + { + "fieldPath": "points", + "columnName": "points", + "affinity": "BLOB", + "notNull": true + }, + { + "fieldPath": "pageId", + "columnName": "pageId", + "affinity": "TEXT", + "notNull": true + }, + { + "fieldPath": "createdAt", + "columnName": "createdAt", + "affinity": "INTEGER", + "notNull": true + }, + { + "fieldPath": "updatedAt", + "columnName": "updatedAt", + "affinity": "INTEGER", + "notNull": true + } + ], + "primaryKey": { + "autoGenerate": false, + "columnNames": [ + "id" + ] + }, + "indices": [ + { + "name": "index_Stroke_pageId", + "unique": false, + "columnNames": [ + "pageId" + ], + "orders": [], + "createSql": "CREATE INDEX IF NOT EXISTS `index_Stroke_pageId` ON `${TABLE_NAME}` (`pageId`)" + } + ], + "foreignKeys": [ + { + "table": "Page", + "onDelete": "CASCADE", + "onUpdate": "NO ACTION", + "columns": [ + "pageId" + ], + "referencedColumns": [ + "id" + ] + } + ] + }, + { + "tableName": "Image", + "createSql": "CREATE TABLE IF NOT EXISTS `${TABLE_NAME}` (`id` TEXT NOT NULL, `x` INTEGER NOT NULL, `y` INTEGER NOT NULL, `height` INTEGER NOT NULL, `width` INTEGER NOT NULL, `uri` TEXT, `pageId` TEXT NOT NULL, `createdAt` INTEGER NOT NULL, `updatedAt` INTEGER NOT NULL, PRIMARY KEY(`id`), FOREIGN KEY(`pageId`) REFERENCES `Page`(`id`) ON UPDATE NO ACTION ON DELETE CASCADE )", + "fields": [ + { + "fieldPath": "id", + "columnName": "id", + "affinity": "TEXT", + "notNull": true + }, + { + "fieldPath": "x", + "columnName": "x", + "affinity": "INTEGER", + "notNull": true + }, + { + "fieldPath": "y", + "columnName": "y", + "affinity": "INTEGER", + "notNull": true + }, + { + "fieldPath": "height", + "columnName": "height", + "affinity": "INTEGER", + "notNull": true + }, + { + "fieldPath": "width", + "columnName": "width", + "affinity": "INTEGER", + "notNull": true + }, + { + "fieldPath": "uri", + "columnName": "uri", + "affinity": "TEXT" + }, + { + "fieldPath": "pageId", + "columnName": "pageId", + "affinity": "TEXT", + "notNull": true + }, + { + "fieldPath": "createdAt", + "columnName": "createdAt", + "affinity": "INTEGER", + "notNull": true + }, + { + "fieldPath": "updatedAt", + "columnName": "updatedAt", + "affinity": "INTEGER", + "notNull": true + } + ], + "primaryKey": { + "autoGenerate": false, + "columnNames": [ + "id" + ] + }, + "indices": [ + { + "name": "index_Image_pageId", + "unique": false, + "columnNames": [ + "pageId" + ], + "orders": [], + "createSql": "CREATE INDEX IF NOT EXISTS `index_Image_pageId` ON `${TABLE_NAME}` (`pageId`)" + } + ], + "foreignKeys": [ + { + "table": "Page", + "onDelete": "CASCADE", + "onUpdate": "NO ACTION", + "columns": [ + "pageId" + ], + "referencedColumns": [ + "id" + ] + } + ] + }, + { + "tableName": "Kv", + "createSql": "CREATE TABLE IF NOT EXISTS `${TABLE_NAME}` (`key` TEXT NOT NULL, `value` TEXT NOT NULL, PRIMARY KEY(`key`))", + "fields": [ + { + "fieldPath": "key", + "columnName": "key", + "affinity": "TEXT", + "notNull": true + }, + { + "fieldPath": "value", + "columnName": "value", + "affinity": "TEXT", + "notNull": true + } + ], + "primaryKey": { + "autoGenerate": false, + "columnNames": [ + "key" + ] + } + }, + { + "tableName": "Annotation", + "createSql": "CREATE TABLE IF NOT EXISTS `${TABLE_NAME}` (`id` TEXT NOT NULL, `type` TEXT NOT NULL, `x` REAL NOT NULL, `y` REAL NOT NULL, `width` REAL NOT NULL, `height` REAL NOT NULL, `pageId` TEXT NOT NULL, `createdAt` INTEGER NOT NULL, `updatedAt` INTEGER NOT NULL, PRIMARY KEY(`id`), FOREIGN KEY(`pageId`) REFERENCES `Page`(`id`) ON UPDATE NO ACTION ON DELETE CASCADE )", + "fields": [ + { + "fieldPath": "id", + "columnName": "id", + "affinity": "TEXT", + "notNull": true + }, + { + "fieldPath": "type", + "columnName": "type", + "affinity": "TEXT", + "notNull": true + }, + { + "fieldPath": "x", + "columnName": "x", + "affinity": "REAL", + "notNull": true + }, + { + "fieldPath": "y", + "columnName": "y", + "affinity": "REAL", + "notNull": true + }, + { + "fieldPath": "width", + "columnName": "width", + "affinity": "REAL", + "notNull": true + }, + { + "fieldPath": "height", + "columnName": "height", + "affinity": "REAL", + "notNull": true + }, + { + "fieldPath": "pageId", + "columnName": "pageId", + "affinity": "TEXT", + "notNull": true + }, + { + "fieldPath": "createdAt", + "columnName": "createdAt", + "affinity": "INTEGER", + "notNull": true + }, + { + "fieldPath": "updatedAt", + "columnName": "updatedAt", + "affinity": "INTEGER", + "notNull": true + } + ], + "primaryKey": { + "autoGenerate": false, + "columnNames": [ + "id" + ] + }, + "indices": [ + { + "name": "index_Annotation_pageId", + "unique": false, + "columnNames": [ + "pageId" + ], + "orders": [], + "createSql": "CREATE INDEX IF NOT EXISTS `index_Annotation_pageId` ON `${TABLE_NAME}` (`pageId`)" + } + ], + "foreignKeys": [ + { + "table": "Page", + "onDelete": "CASCADE", + "onUpdate": "NO ACTION", + "columns": [ + "pageId" + ], + "referencedColumns": [ + "id" + ] + } + ] + } + ], + "setupQueries": [ + "CREATE TABLE IF NOT EXISTS room_master_table (id INTEGER PRIMARY KEY,identity_hash TEXT)", + "INSERT OR REPLACE INTO room_master_table (id,identity_hash) VALUES(42, 'e833f66629726de6b438770c17eb6f83')" + ] + } +} \ No newline at end of file diff --git a/app/src/main/java/com/ethran/notable/data/AppRepository.kt b/app/src/main/java/com/ethran/notable/data/AppRepository.kt index 98d7a68c..0eace832 100644 --- a/app/src/main/java/com/ethran/notable/data/AppRepository.kt +++ b/app/src/main/java/com/ethran/notable/data/AppRepository.kt @@ -1,6 +1,7 @@ package com.ethran.notable.data import com.ethran.notable.data.datastore.GlobalAppSettings +import com.ethran.notable.data.db.AnnotationRepository import com.ethran.notable.data.db.BookRepository import com.ethran.notable.data.db.FolderRepository import com.ethran.notable.data.db.ImageRepository @@ -24,6 +25,7 @@ class AppRepository @Inject constructor( val pageRepository: PageRepository, val strokeRepository: StrokeRepository, val imageRepository: ImageRepository, + val annotationRepository: AnnotationRepository, val folderRepository: FolderRepository, val kvProxy: KvProxy ) { diff --git a/app/src/main/java/com/ethran/notable/data/PageDataManager.kt b/app/src/main/java/com/ethran/notable/data/PageDataManager.kt index 49115a2f..b2785194 100644 --- a/app/src/main/java/com/ethran/notable/data/PageDataManager.kt +++ b/app/src/main/java/com/ethran/notable/data/PageDataManager.kt @@ -10,6 +10,7 @@ import androidx.compose.runtime.mutableStateMapOf import androidx.compose.ui.geometry.Offset import com.ethran.notable.SCREEN_HEIGHT import com.ethran.notable.SCREEN_WIDTH +import com.ethran.notable.data.db.Annotation import com.ethran.notable.data.db.Image import com.ethran.notable.data.db.Stroke import com.ethran.notable.data.db.getBackgroundType @@ -75,6 +76,8 @@ object PageDataManager { private val images = LinkedHashMap>() private var imagesById = LinkedHashMap>() + private val annotations = LinkedHashMap>() + private val backgroundCache = LinkedHashMap() private val pageToBackgroundKey = HashMap() private val bitmapCache = LinkedHashMap>() @@ -274,6 +277,8 @@ object PageDataManager { cacheStrokes(pageId, pageWithStrokes.strokes) val pageWithImages = appRepository.pageRepository.getWithImageById(pageId) cacheImages(pageId, pageWithImages.images) + val pageAnnotations = appRepository.annotationRepository.getByPageId(pageId) + cacheAnnotations(pageId, pageAnnotations) recomputeHeight(pageId) indexImages(coroutineScope, pageId) indexStrokes(coroutineScope, pageId) @@ -486,6 +491,12 @@ object PageDataManager { return imageIds.map { i -> imagesById[pageId]?.get(i) } } + fun getAnnotations(pageId: String): List = annotations[pageId] ?: emptyList() + + fun setAnnotations(pageId: String, annotations: List) { + this.annotations[pageId] = annotations.toMutableList() + } + // Assuming Rect uses 'left', 'top', 'right', 'bottom' fun getImagesInRectangle(inPageCoordinates: Rect, id: String): List? { @@ -531,6 +542,12 @@ object PageDataManager { } } + private fun cacheAnnotations(pageId: String, annotations: List) { + synchronized(accessLock) { + this.annotations[pageId] = annotations.toMutableList() + } + } + fun setBackground(pageId: String, background: CachedBackground, observe: Boolean) { synchronized(accessLock) { @@ -719,6 +736,7 @@ object PageDataManager { synchronized(accessLock) { strokes.remove(pageId) images.remove(pageId) + annotations.remove(pageId) pageHigh.remove(pageId) pageZoom.remove(pageId) pageScroll.remove(pageId) diff --git a/app/src/main/java/com/ethran/notable/data/db/Annotation.kt b/app/src/main/java/com/ethran/notable/data/db/Annotation.kt new file mode 100644 index 00000000..36497f9b --- /dev/null +++ b/app/src/main/java/com/ethran/notable/data/db/Annotation.kt @@ -0,0 +1,74 @@ +package com.ethran.notable.data.db + +import androidx.room.ColumnInfo +import androidx.room.Dao +import androidx.room.Entity +import androidx.room.ForeignKey +import androidx.room.Insert +import androidx.room.PrimaryKey +import androidx.room.Query +import androidx.room.Transaction +import java.util.Date +import java.util.UUID +import javax.inject.Inject + +enum class AnnotationType { + WIKILINK, + TAG +} + +@Entity( + foreignKeys = [ForeignKey( + entity = Page::class, + parentColumns = arrayOf("id"), + childColumns = arrayOf("pageId"), + onDelete = ForeignKey.CASCADE + )] +) +data class Annotation( + @PrimaryKey val id: String = UUID.randomUUID().toString(), + + val type: String, // "WIKILINK" or "TAG" + + // Bounding box in page coordinates + val x: Float, + val y: Float, + val width: Float, + val height: Float, + + @ColumnInfo(index = true) + val pageId: String, + + val createdAt: Date = Date(), + val updatedAt: Date = Date() +) + +@Dao +interface AnnotationDao { + @Insert + suspend fun create(annotation: Annotation): Long + + @Insert + suspend fun create(annotations: List) + + @Query("DELETE FROM Annotation WHERE id IN (:ids)") + suspend fun deleteAll(ids: List) + + @Transaction + @Query("SELECT * FROM Annotation WHERE pageId = :pageId") + suspend fun getByPageId(pageId: String): List + + @Transaction + @Query("SELECT * FROM Annotation WHERE id = :annotationId") + suspend fun getById(annotationId: String): Annotation? +} + +class AnnotationRepository @Inject constructor( + private val db: AnnotationDao +) { + suspend fun create(annotation: Annotation): Long = db.create(annotation) + suspend fun create(annotations: List) = db.create(annotations) + suspend fun deleteAll(ids: List) = db.deleteAll(ids) + suspend fun getByPageId(pageId: String): List = db.getByPageId(pageId) + suspend fun getById(annotationId: String): Annotation? = db.getById(annotationId) +} diff --git a/app/src/main/java/com/ethran/notable/data/db/Db.kt b/app/src/main/java/com/ethran/notable/data/db/Db.kt index 99e76ce8..7afd0af3 100644 --- a/app/src/main/java/com/ethran/notable/data/db/Db.kt +++ b/app/src/main/java/com/ethran/notable/data/db/Db.kt @@ -53,8 +53,8 @@ class Converters { @Database( - entities = [Folder::class, Notebook::class, Page::class, Stroke::class, Image::class, Kv::class], - version = 34, + entities = [Folder::class, Notebook::class, Page::class, Stroke::class, Image::class, Kv::class, Annotation::class], + version = 35, autoMigrations = [ AutoMigration(19, 20), AutoMigration(20, 21), @@ -69,7 +69,8 @@ class Converters { AutoMigration(30, 31, spec = AutoMigration30to31::class), AutoMigration(31, 32, spec = AutoMigration31to32::class), AutoMigration(32, 33), - AutoMigration(33, 34) + AutoMigration(33, 34), + AutoMigration(34, 35) ], exportSchema = true ) @TypeConverters(Converters::class) @@ -81,6 +82,7 @@ abstract class AppDatabase : RoomDatabase() { abstract fun pageDao(): PageDao abstract fun strokeDao(): StrokeDao abstract fun ImageDao(): ImageDao + abstract fun annotationDao(): AnnotationDao // companion object { // private var INSTANCE: AppDatabase? = null @@ -161,6 +163,8 @@ object DatabaseModule { fun provideImageDao(db: AppDatabase): ImageDao = db.ImageDao() - + @Provides + fun provideAnnotationDao(db: AppDatabase): AnnotationDao = + db.annotationDao() } \ No newline at end of file diff --git a/app/src/main/java/com/ethran/notable/editor/EditorView.kt b/app/src/main/java/com/ethran/notable/editor/EditorView.kt index d860bdf8..80a47f99 100644 --- a/app/src/main/java/com/ethran/notable/editor/EditorView.kt +++ b/app/src/main/java/com/ethran/notable/editor/EditorView.kt @@ -1,18 +1,10 @@ package com.ethran.notable.editor -import androidx.compose.foundation.background -import androidx.compose.foundation.layout.Box import androidx.compose.foundation.layout.BoxWithConstraints -import androidx.compose.foundation.layout.Column import androidx.compose.foundation.layout.Row import androidx.compose.foundation.layout.Spacer import androidx.compose.foundation.layout.fillMaxHeight -import androidx.compose.foundation.layout.fillMaxSize import androidx.compose.foundation.layout.fillMaxWidth -import androidx.compose.material.Text -import androidx.compose.ui.Alignment -import androidx.compose.ui.graphics.Color -import androidx.compose.ui.unit.sp import androidx.compose.runtime.Composable import androidx.compose.runtime.DisposableEffect import androidx.compose.runtime.LaunchedEffect @@ -26,20 +18,17 @@ import androidx.compose.ui.Modifier import androidx.compose.ui.platform.LocalContext import androidx.navigation.NavController import com.ethran.notable.data.AppRepository -import com.ethran.notable.data.datastore.AppSettings import com.ethran.notable.data.datastore.EditorSettingCacheManager -import com.ethran.notable.data.datastore.GlobalAppSettings import com.ethran.notable.editor.state.EditorState import com.ethran.notable.editor.state.History +import com.ethran.notable.editor.ui.EditorSidebar import com.ethran.notable.editor.ui.EditorSurface import com.ethran.notable.editor.ui.HorizontalScrollIndicator import com.ethran.notable.editor.ui.InboxToolbar import com.ethran.notable.editor.ui.ScrollIndicator import com.ethran.notable.editor.ui.SelectedBitmap -import com.ethran.notable.editor.ui.toolbar.Toolbar import com.ethran.notable.gestures.EditorGestureReceiver import com.ethran.notable.io.ExportEngine -import com.ethran.notable.io.InboxSyncEngine import com.ethran.notable.io.SyncState import com.ethran.notable.io.VaultTagScanner import com.ethran.notable.io.exportToLinkedFile @@ -51,7 +40,6 @@ import com.ethran.notable.ui.convertDpToPixel import com.ethran.notable.ui.theme.InkaTheme import com.ethran.notable.ui.views.LibraryDestination import io.shipbook.shipbooksdk.ShipBook -import kotlinx.coroutines.CoroutineScope import kotlinx.coroutines.Dispatchers import kotlinx.coroutines.launch import kotlinx.coroutines.withContext @@ -226,7 +214,7 @@ fun EditorView( ScrollIndicator(state = editorState) } if (isInboxPage) { - // Inbox toolbar at top + // Inbox toolbar at top (Back, Capture, Save & Exit) InboxToolbar( selectedTags = selectedTags, suggestedTags = suggestedTags, @@ -250,20 +238,9 @@ fun EditorView( }, onDiscard = { navController.popBackStack() } ) - // Pen toolbar at bottom for inbox pages — only when open - if (editorState.isToolbarOpen) { - Column( - Modifier - .fillMaxWidth() - .fillMaxHeight() - ) { - Spacer(modifier = Modifier.weight(1f)) - Toolbar(exportEngine, navController, appRepository, editorState, editorControlTower) - } - } - } else { - PositionedToolbar(exportEngine, navController, appRepository, editorState, editorControlTower) } + // Left-edge sidebar with all tools + EditorSidebar(exportEngine, navController, appRepository, editorState, editorControlTower) HorizontalScrollIndicator(state = editorState) } @@ -271,33 +248,3 @@ fun EditorView( } -@Composable -fun PositionedToolbar( - exportEngine: ExportEngine, - navController: NavController, - appRepository: AppRepository, - editorState: EditorState, - editorControlTower: EditorControlTower -) { - val position = GlobalAppSettings.current.toolbarPosition - - when (position) { - AppSettings.Position.Top -> { - Toolbar( - exportEngine, - navController, appRepository, editorState, editorControlTower - ) - } - - AppSettings.Position.Bottom -> { - Column( - Modifier - .fillMaxWidth() - .fillMaxHeight() - ) { - Spacer(modifier = Modifier.weight(1f)) - Toolbar(exportEngine, navController, appRepository, editorState, editorControlTower) - } - } - } -} \ No newline at end of file diff --git a/app/src/main/java/com/ethran/notable/editor/PageView.kt b/app/src/main/java/com/ethran/notable/editor/PageView.kt index 47fe8d0f..121d8d03 100644 --- a/app/src/main/java/com/ethran/notable/editor/PageView.kt +++ b/app/src/main/java/com/ethran/notable/editor/PageView.kt @@ -20,6 +20,7 @@ import com.ethran.notable.data.AppRepository import com.ethran.notable.data.CachedBackground import com.ethran.notable.data.PageDataManager import com.ethran.notable.data.datastore.GlobalAppSettings +import com.ethran.notable.data.db.Annotation import com.ethran.notable.data.db.Image import com.ethran.notable.data.db.Page import com.ethran.notable.data.db.Stroke @@ -95,6 +96,10 @@ class PageView( get() = PageDataManager.getImages(currentPageId) set(value) = PageDataManager.setImages(currentPageId, value) + var annotations: List + get() = PageDataManager.getAnnotations(currentPageId) + set(value) = PageDataManager.setAnnotations(currentPageId, value) + private var currentBackground: CachedBackground get() = PageDataManager.getBackground(currentPageId) set(value) { @@ -346,6 +351,22 @@ class PageView( return PageDataManager.getStrokes(strokeIds, currentPageId) } + fun addAnnotations(annotationsToAdd: List) { + annotations = annotations + annotationsToAdd + coroutineScope.launch(Dispatchers.IO) { + appRepository.annotationRepository.create(annotationsToAdd) + } + persistBitmapDebounced() + } + + fun removeAnnotations(annotationIds: List) { + annotations = annotations.filter { a -> a.id !in annotationIds } + coroutineScope.launch(Dispatchers.IO) { + appRepository.annotationRepository.deleteAll(annotationIds) + } + persistBitmapDebounced() + } + fun updateHeightForChange(strokesChanged: List) { strokesChanged.forEach { val bottomPlusPadding = it.bottom + 50 diff --git a/app/src/main/java/com/ethran/notable/editor/canvas/OnyxInputHandler.kt b/app/src/main/java/com/ethran/notable/editor/canvas/OnyxInputHandler.kt index 937e0321..d72f74d0 100644 --- a/app/src/main/java/com/ethran/notable/editor/canvas/OnyxInputHandler.kt +++ b/app/src/main/java/com/ethran/notable/editor/canvas/OnyxInputHandler.kt @@ -21,6 +21,8 @@ import com.ethran.notable.editor.utils.calculateBoundingBox import com.ethran.notable.editor.utils.copyInput import com.ethran.notable.editor.utils.copyInputToSimplePointF import com.ethran.notable.editor.utils.getModifiedStrokeEndpoints +import com.ethran.notable.editor.state.AnnotationMode +import com.ethran.notable.editor.utils.handleAnnotation import com.ethran.notable.editor.utils.handleDraw import com.ethran.notable.editor.utils.handleErase import com.ethran.notable.editor.utils.handleScribbleToErase @@ -305,16 +307,30 @@ class OnyxInputHandler( firstPointTime ) if (erasedByScribbleDirtyRect.isNullOrEmpty()) { - log.d("Drawing...") - // draw the stroke - handleDraw( - drawCanvas.page, - strokeHistoryBatch, - drawCanvas.getActualState().penSettings[drawCanvas.getActualState().pen.penName]!!.strokeSize, - drawCanvas.getActualState().penSettings[drawCanvas.getActualState().pen.penName]!!.color, - drawCanvas.getActualState().pen, - scaledPoints - ) + val annotMode = drawCanvas.getActualState().annotationMode + if (annotMode != AnnotationMode.None) { + log.d("Creating annotation...") + handleAnnotation( + drawCanvas.page, + annotMode, + scaledPoints + ) + // Reset to one-shot: annotation mode turns off after one box + drawCanvas.getActualState().annotationMode = AnnotationMode.None + // Refresh canvas to show the annotation overlay + drawCanvas.drawCanvasToView(null) + } else { + log.d("Drawing...") + // draw the stroke + handleDraw( + drawCanvas.page, + strokeHistoryBatch, + drawCanvas.getActualState().penSettings[drawCanvas.getActualState().pen.penName]!!.strokeSize, + drawCanvas.getActualState().penSettings[drawCanvas.getActualState().pen.penName]!!.color, + drawCanvas.getActualState().pen, + scaledPoints + ) + } } else { log.d("Erased by scribble, $erasedByScribbleDirtyRect") drawCanvas.drawCanvasToView(erasedByScribbleDirtyRect) diff --git a/app/src/main/java/com/ethran/notable/editor/drawing/pageDrawing.kt b/app/src/main/java/com/ethran/notable/editor/drawing/pageDrawing.kt index 9726240a..7da56a4e 100644 --- a/app/src/main/java/com/ethran/notable/editor/drawing/pageDrawing.kt +++ b/app/src/main/java/com/ethran/notable/editor/drawing/pageDrawing.kt @@ -14,6 +14,8 @@ import androidx.core.graphics.toRect import androidx.core.graphics.withClip import androidx.core.net.toUri import com.ethran.notable.data.datastore.GlobalAppSettings +import com.ethran.notable.data.db.Annotation +import com.ethran.notable.data.db.AnnotationType import com.ethran.notable.data.db.Image import com.ethran.notable.data.db.getBackgroundType import com.ethran.notable.data.model.BackgroundType @@ -28,6 +30,44 @@ import io.shipbook.shipbooksdk.ShipBook private val pageDrawingLog = ShipBook.getLogger("PageDrawingLog") +// Annotation overlay paints +private val wikiLinkPaint = Paint().apply { + color = Color.argb(50, 0, 100, 255) // semi-transparent blue fill + style = Paint.Style.FILL +} +private val wikiLinkBorderPaint = Paint().apply { + color = Color.argb(180, 0, 100, 255) // blue border + style = Paint.Style.STROKE + strokeWidth = 2f + isAntiAlias = true +} +private val tagPaint = Paint().apply { + color = Color.argb(50, 0, 180, 0) // semi-transparent green fill + style = Paint.Style.FILL +} +private val tagBorderPaint = Paint().apply { + color = Color.argb(180, 0, 180, 0) // green border + style = Paint.Style.STROKE + strokeWidth = 2f + isAntiAlias = true +} + +fun drawAnnotation(canvas: Canvas, annotation: Annotation, offset: Offset) { + val rect = RectF( + annotation.x + offset.x, + annotation.y + offset.y, + annotation.x + annotation.width + offset.x, + annotation.y + annotation.height + offset.y + ) + val (fillPaint, borderPaint) = if (annotation.type == AnnotationType.WIKILINK.name) { + wikiLinkPaint to wikiLinkBorderPaint + } else { + tagPaint to tagBorderPaint + } + canvas.drawRect(rect, fillPaint) + canvas.drawRect(rect, borderPaint) +} + /** * Draws an image onto the provided Canvas at a specified location and size, using its URI. @@ -176,5 +216,18 @@ fun drawOnCanvasFromPage( pageDrawingLog.e("PageView.kt: Drawing strokes failed: ${e.message}", e) showHint("Error drawing strokes", page.coroutineScope) } + // Draw annotation overlays on top of strokes + try { + page.annotations.forEach { annotation -> + val annotRect = RectF( + annotation.x, annotation.y, + annotation.x + annotation.width, annotation.y + annotation.height + ) + if (!annotRect.toRect().intersect(pageArea)) return@forEach + drawAnnotation(this, annotation, -page.scroll) + } + } catch (e: Exception) { + pageDrawingLog.e("Drawing annotations failed: ${e.message}", e) + } } } \ No newline at end of file diff --git a/app/src/main/java/com/ethran/notable/editor/state/EditorState.kt b/app/src/main/java/com/ethran/notable/editor/state/EditorState.kt index 00f99201..43bec198 100644 --- a/app/src/main/java/com/ethran/notable/editor/state/EditorState.kt +++ b/app/src/main/java/com/ethran/notable/editor/state/EditorState.kt @@ -19,6 +19,10 @@ enum class Mode { Draw, Erase, Select, Line } +enum class AnnotationMode { + None, WikiLink, Tag +} + @Stable class MenuStates { var isStrokeSelectionOpen by mutableStateOf(false) @@ -94,6 +98,7 @@ class EditorState( var isInboxPage by mutableStateOf(false) var isInboxTagsExpanded by mutableStateOf(false) + var annotationMode by mutableStateOf(AnnotationMode.None) var isToolbarOpen by mutableStateOf(persistedEditorSettings?.isToolbarOpen ?: false) var penSettings by mutableStateOf(persistedEditorSettings?.penSettings ?: Pen.DEFAULT_SETTINGS) diff --git a/app/src/main/java/com/ethran/notable/editor/ui/AnnotationSidebar.kt b/app/src/main/java/com/ethran/notable/editor/ui/AnnotationSidebar.kt new file mode 100644 index 00000000..18a80452 --- /dev/null +++ b/app/src/main/java/com/ethran/notable/editor/ui/AnnotationSidebar.kt @@ -0,0 +1,112 @@ +package com.ethran.notable.editor.ui + +import androidx.compose.foundation.background +import androidx.compose.foundation.border +import androidx.compose.foundation.layout.Arrangement +import androidx.compose.foundation.layout.Box +import androidx.compose.foundation.layout.Column +import androidx.compose.foundation.layout.padding +import androidx.compose.foundation.layout.size +import androidx.compose.foundation.layout.width +import androidx.compose.foundation.shape.RoundedCornerShape +import androidx.compose.material.Icon +import androidx.compose.material.Text +import androidx.compose.runtime.Composable +import androidx.compose.ui.Alignment +import androidx.compose.ui.Modifier +import androidx.compose.ui.graphics.Color +import androidx.compose.ui.res.painterResource +import androidx.compose.ui.text.font.FontWeight +import androidx.compose.ui.unit.dp +import androidx.compose.ui.unit.sp +import com.ethran.notable.R +import com.ethran.notable.ui.noRippleClickable + +@Composable +fun AnnotationSidebar( + onUndo: () -> Unit, + onRedo: () -> Unit, + onWikiLink: () -> Unit, + onTag: () -> Unit, + modifier: Modifier = Modifier +) { + Column( + modifier = modifier + .width(44.dp) + .padding(vertical = 80.dp, horizontal = 4.dp), + verticalArrangement = Arrangement.spacedBy(6.dp), + horizontalAlignment = Alignment.CenterHorizontally + ) { + // Undo + SidebarButton( + iconId = R.drawable.undo, + contentDescription = "Undo", + onClick = onUndo + ) + + // Redo + SidebarButton( + iconId = R.drawable.redo, + contentDescription = "Redo", + onClick = onRedo + ) + + // Spacer/divider + Box( + Modifier + .size(width = 28.dp, height = 1.dp) + .background(Color.LightGray) + ) + + // Wiki link annotation + SidebarButton( + text = "[[", + contentDescription = "Wiki link", + onClick = onWikiLink + ) + + // Tag annotation + SidebarButton( + text = "#", + contentDescription = "Tag", + onClick = onTag + ) + } +} + +@Composable +private fun SidebarButton( + iconId: Int? = null, + text: String? = null, + contentDescription: String, + isSelected: Boolean = false, + onClick: () -> Unit +) { + val bgColor = if (isSelected) Color.Black else Color.White + val fgColor = if (isSelected) Color.White else Color.Black + val borderColor = if (isSelected) Color.Black else Color.Gray + + Box( + modifier = Modifier + .size(36.dp) + .border(1.dp, borderColor, RoundedCornerShape(6.dp)) + .background(bgColor, RoundedCornerShape(6.dp)) + .noRippleClickable { onClick() }, + contentAlignment = Alignment.Center + ) { + when { + iconId != null -> Icon( + painterResource(iconId), + contentDescription = contentDescription, + tint = fgColor, + modifier = Modifier.size(20.dp) + ) + text != null -> Text( + text, + fontSize = if (text.length > 2) 10.sp else 18.sp, + fontWeight = FontWeight.Bold, + color = fgColor + ) + } + } +} diff --git a/app/src/main/java/com/ethran/notable/editor/ui/EditorSidebar.kt b/app/src/main/java/com/ethran/notable/editor/ui/EditorSidebar.kt new file mode 100644 index 00000000..08036c3d --- /dev/null +++ b/app/src/main/java/com/ethran/notable/editor/ui/EditorSidebar.kt @@ -0,0 +1,718 @@ +package com.ethran.notable.editor.ui + +import androidx.activity.compose.rememberLauncherForActivityResult +import androidx.activity.result.PickVisualMediaRequest +import androidx.activity.result.contract.ActivityResultContracts.PickVisualMedia +import androidx.compose.foundation.background +import androidx.compose.foundation.border +import androidx.compose.foundation.clickable +import androidx.compose.foundation.layout.Arrangement +import androidx.compose.foundation.layout.Box +import androidx.compose.foundation.layout.Column +import androidx.compose.foundation.layout.IntrinsicSize +import androidx.compose.foundation.layout.Row +import androidx.compose.foundation.layout.Spacer +import androidx.compose.foundation.layout.fillMaxHeight +import androidx.compose.foundation.layout.height +import androidx.compose.foundation.layout.padding +import androidx.compose.foundation.layout.size +import androidx.compose.foundation.layout.width +import androidx.compose.foundation.shape.RoundedCornerShape +import androidx.compose.foundation.text.BasicText +import androidx.compose.material.Icon +import androidx.compose.material.Text +import androidx.compose.runtime.Composable +import androidx.compose.runtime.LaunchedEffect +import androidx.compose.runtime.collectAsState +import androidx.compose.runtime.getValue +import androidx.compose.runtime.mutableStateOf +import androidx.compose.runtime.remember +import androidx.compose.runtime.rememberCoroutineScope +import androidx.compose.runtime.setValue +import androidx.compose.ui.Alignment +import androidx.compose.ui.Modifier +import androidx.compose.ui.graphics.Color +import androidx.compose.ui.graphics.vector.ImageVector +import androidx.compose.ui.platform.LocalContext +import androidx.compose.ui.res.painterResource +import androidx.compose.ui.res.stringResource +import androidx.compose.ui.text.TextStyle +import androidx.compose.ui.text.font.FontWeight +import androidx.compose.ui.unit.IntOffset +import androidx.compose.ui.unit.dp +import androidx.compose.ui.unit.sp +import androidx.compose.ui.window.Popup +import androidx.compose.ui.window.PopupProperties +import androidx.core.net.toUri +import androidx.navigation.NavController +import com.ethran.notable.R +import com.ethran.notable.data.AppRepository +import com.ethran.notable.data.copyImageToDatabase +import com.ethran.notable.data.db.getParentFolder +import com.ethran.notable.data.datastore.GlobalAppSettings +import com.ethran.notable.editor.EditorControlTower +import com.ethran.notable.editor.canvas.CanvasEventBus +import com.ethran.notable.editor.state.AnnotationMode +import com.ethran.notable.editor.state.EditorState +import com.ethran.notable.editor.state.Mode +import com.ethran.notable.editor.ui.toolbar.ToolbarMenu +import com.ethran.notable.editor.ui.toolbar.presentlyUsedToolIcon +import com.ethran.notable.editor.utils.Eraser +import com.ethran.notable.editor.utils.Pen +import com.ethran.notable.editor.utils.PenSetting +import com.ethran.notable.io.ExportEngine +import com.ethran.notable.ui.convertDpToPixel +import com.ethran.notable.ui.dialogs.BackgroundSelector +import com.ethran.notable.ui.noRippleClickable +import com.ethran.notable.ui.views.BugReportDestination +import com.ethran.notable.ui.views.LibraryDestination +import compose.icons.FeatherIcons +import compose.icons.feathericons.EyeOff +import compose.icons.feathericons.RefreshCcw +import compose.icons.feathericons.Clipboard +import io.shipbook.shipbooksdk.ShipBook +import kotlinx.coroutines.Dispatchers +import kotlinx.coroutines.launch +import kotlinx.coroutines.withContext + +private val log = ShipBook.getLogger("EditorSidebar") + +private val SIZES_STROKES_DEFAULT = listOf("S" to 3f, "M" to 5f, "L" to 10f, "XL" to 20f) +private val SIZES_MARKER_DEFAULT = listOf("M" to 25f, "L" to 40f, "XL" to 60f, "XXL" to 80f) + +private const val SIDEBAR_WIDTH = 56 +private const val BUTTON_SIZE = 46 +private const val ICON_SIZE = 26 + +@Composable +fun EditorSidebar( + exportEngine: ExportEngine, + navController: NavController, + appRepository: AppRepository, + state: EditorState, + controlTower: EditorControlTower, + topPadding: Int = 0 +) { + val scope = rememberCoroutineScope() + val context = LocalContext.current + val zoomLevel by state.pageView.zoomLevel.collectAsState() + + val pickMedia = rememberLauncherForActivityResult(contract = PickVisualMedia()) { uri -> + if (uri == null) { + log.w("PickVisualMedia: uri is null") + return@rememberLauncherForActivityResult + } + scope.launch(Dispatchers.IO) { + try { + val copiedFile = copyImageToDatabase(context, uri) + log.i("Image copied to: ${copiedFile.toUri()}") + CanvasEventBus.addImageByUri.value = copiedFile.toUri() + } catch (e: Exception) { + log.e("ImagePicker: copy failed: ${e.message}", e) + } + } + } + + // Background selector dialog + if (state.menuStates.isBackgroundSelectorModalOpen) { + BackgroundSelector( + initialPageBackgroundType = state.pageView.pageFromDb?.backgroundType ?: "native", + initialPageBackground = state.pageView.pageFromDb?.background ?: "blank", + initialPageNumberInPdf = state.pageView.getBackgroundPageNumber(), + notebookId = state.pageView.pageFromDb?.notebookId, + pageNumberInBook = state.pageView.currentPageNumber, + onChange = { backgroundType, background -> + val updatedPage = if (background == null) + state.pageView.pageFromDb!!.copy(backgroundType = backgroundType) + else state.pageView.pageFromDb!!.copy( + background = background, + backgroundType = backgroundType + ) + state.pageView.updatePageSettings(updatedPage) + scope.launch { CanvasEventBus.refreshUi.emit(Unit) } + } + ) { + state.menuStates.isBackgroundSelectorModalOpen = false + } + } + + LaunchedEffect(state.menuStates.isBackgroundSelectorModalOpen, state.menuStates.isMenuOpen) { + state.checkForSelectionsAndMenus() + } + + if (!state.isToolbarOpen) { + // Collapsed: single icon button in top-left to reopen + Box(Modifier.padding(top = (topPadding + 4).dp, start = 4.dp)) { + SidebarIconButton( + iconId = presentlyUsedToolIcon(state.mode, state.pen), + contentDescription = "open toolbar", + onClick = { state.isToolbarOpen = true } + ) + } + return + } + + // --- Expanded sidebar --- + var isPenPickerOpen by remember { mutableStateOf(false) } + var isEraserMenuOpen by remember { mutableStateOf(false) } + + // Pause drawing when popups are open + LaunchedEffect(isPenPickerOpen, isEraserMenuOpen) { + state.isDrawing = !(isPenPickerOpen || isEraserMenuOpen) + } + + fun handleChangePen(pen: Pen) { + if (state.mode == Mode.Draw && state.pen == pen) { + // Already on this pen — toggle stroke picker + isPenPickerOpen = !isPenPickerOpen + } else { + state.mode = Mode.Draw + state.pen = pen + isPenPickerOpen = false + } + } + + fun onChangeStrokeSetting(penName: String, setting: PenSetting) { + val settings = state.penSettings.toMutableMap() + settings[penName] = setting.copy() + state.penSettings = settings + } + + Column( + modifier = Modifier + .width(SIDEBAR_WIDTH.dp) + .fillMaxHeight() + .padding(horizontal = 4.dp), + verticalArrangement = Arrangement.Center, + horizontalAlignment = Alignment.CenterHorizontally + ) { + // Close sidebar + SidebarIconButton( + vectorIcon = FeatherIcons.EyeOff, + contentDescription = "close toolbar", + onClick = { state.isToolbarOpen = false } + ) + + SidebarDivider() + + // --- Drawing tools --- + + // Pen button (shows current pen, tap to open picker) + Box { + SidebarIconButton( + iconId = presentlyUsedToolIcon(Mode.Draw, state.pen), + contentDescription = "pen", + isSelected = state.mode == Mode.Draw, + onClick = { + if (state.mode == Mode.Draw) { + isPenPickerOpen = !isPenPickerOpen + isEraserMenuOpen = false + } else { + state.mode = Mode.Draw + isPenPickerOpen = false + } + } + ) + if (isPenPickerOpen) { + PenPickerFlyout( + state = state, + onSelectPen = { pen -> handleChangePen(pen) }, + onChangeSetting = { penName, setting -> onChangeStrokeSetting(penName, setting) }, + onClose = { isPenPickerOpen = false } + ) + } + } + + // Eraser + Box { + SidebarIconButton( + iconId = if (state.eraser == Eraser.PEN) R.drawable.eraser else R.drawable.eraser_select, + contentDescription = "eraser", + isSelected = state.mode == Mode.Erase, + onClick = { + if (state.mode == Mode.Erase) { + isEraserMenuOpen = !isEraserMenuOpen + isPenPickerOpen = false + } else { + state.mode = Mode.Erase + isEraserMenuOpen = false + } + } + ) + if (isEraserMenuOpen) { + EraserFlyout( + value = state.eraser, + onChange = { state.eraser = it }, + toggleScribbleToErase = { enabled -> + scope.launch(Dispatchers.IO) { + appRepository.kvProxy.setAppSettings( + GlobalAppSettings.current.copy(scribbleToEraseEnabled = enabled) + ) + } + }, + onClose = { isEraserMenuOpen = false } + ) + } + } + + // Lasso / Select + SidebarIconButton( + iconId = R.drawable.lasso, + contentDescription = "lasso", + isSelected = state.mode == Mode.Select, + onClick = { + state.mode = Mode.Select + isPenPickerOpen = false + isEraserMenuOpen = false + } + ) + + // Line + SidebarIconButton( + iconId = R.drawable.line, + contentDescription = "line", + isSelected = state.mode == Mode.Line, + onClick = { + if (state.mode == Mode.Line) state.mode = Mode.Draw + else state.mode = Mode.Line + isPenPickerOpen = false + isEraserMenuOpen = false + } + ) + + SidebarDivider() + + // --- Actions --- + + // Undo + SidebarIconButton( + iconId = R.drawable.undo, + contentDescription = "undo", + onClick = { scope.launch { controlTower.undo() } } + ) + + // Redo + SidebarIconButton( + iconId = R.drawable.redo, + contentDescription = "redo", + onClick = { scope.launch { controlTower.redo() } } + ) + + SidebarDivider() + + // --- Annotations --- + + // Wiki link + SidebarTextButton( + text = "[[", + contentDescription = "Wiki link", + isSelected = state.annotationMode == AnnotationMode.WikiLink, + onClick = { + state.annotationMode = if (state.annotationMode == AnnotationMode.WikiLink) + AnnotationMode.None else AnnotationMode.WikiLink + // Ensure we're in draw mode so the stylus gesture gets captured + if (state.annotationMode != AnnotationMode.None) state.mode = Mode.Draw + isPenPickerOpen = false + isEraserMenuOpen = false + } + ) + + // Tag + SidebarTextButton( + text = "#", + contentDescription = "Tag", + isSelected = state.annotationMode == AnnotationMode.Tag, + onClick = { + state.annotationMode = if (state.annotationMode == AnnotationMode.Tag) + AnnotationMode.None else AnnotationMode.Tag + if (state.annotationMode != AnnotationMode.None) state.mode = Mode.Draw + isPenPickerOpen = false + isEraserMenuOpen = false + } + ) + + SidebarDivider() + + // --- Utilities --- + + // Clipboard paste (conditional) + if (state.clipboard != null) { + SidebarIconButton( + vectorIcon = FeatherIcons.Clipboard, + contentDescription = "paste", + onClick = { controlTower.pasteFromClipboard() } + ) + } + + // Reset zoom (conditional) + val showResetView = state.pageView.scroll.x != 0f || zoomLevel != 1.0f + if (showResetView) { + SidebarIconButton( + vectorIcon = FeatherIcons.RefreshCcw, + contentDescription = "reset view", + onClick = { controlTower.resetZoomAndScroll() } + ) + } + + // Image insert + SidebarIconButton( + iconId = R.drawable.image, + contentDescription = "insert image", + onClick = { + log.i("Launching image picker...") + pickMedia.launch(PickVisualMediaRequest(PickVisualMedia.ImageOnly)) + } + ) + + // Home + SidebarIconButton( + iconId = R.drawable.home, + contentDescription = "home", + onClick = { navController.navigate("library") } + ) + + // Menu + Box { + SidebarIconButton( + iconId = R.drawable.menu, + contentDescription = "menu", + onClick = { + state.menuStates.isMenuOpen = !state.menuStates.isMenuOpen + } + ) + if (state.menuStates.isMenuOpen) { + ToolbarMenu( + exportEngine = exportEngine, + goToBugReport = { navController.navigate(BugReportDestination.route) }, + goToLibrary = { + scope.launch { + val page = withContext(Dispatchers.IO) { + appRepository.pageRepository.getById(state.currentPageId) + } + val parentFolder = withContext(Dispatchers.IO) { + page?.getParentFolder(appRepository.bookRepository) + } + navController.navigate(LibraryDestination.createRoute(parentFolder)) + } + }, + currentPageId = state.currentPageId, + currentBookId = state.bookId, + onClose = { state.menuStates.isMenuOpen = false }, + onBackgroundSelectorModalOpen = { + state.menuStates.isBackgroundSelectorModalOpen = true + } + ) + } + } + } +} + +// --- Pen Picker Flyout --- + +@Composable +private fun PenPickerFlyout( + state: EditorState, + onSelectPen: (Pen) -> Unit, + onChangeSetting: (String, PenSetting) -> Unit, + onClose: () -> Unit +) { + val context = LocalContext.current + var selectedPenForSize by remember { mutableStateOf(null) } + + Popup( + alignment = Alignment.TopStart, + offset = IntOffset( + convertDpToPixel((SIDEBAR_WIDTH + 4).dp, context).toInt(), + 0 + ), + onDismissRequest = { onClose() }, + properties = PopupProperties(focusable = true) + ) { + Column( + modifier = Modifier + .background(Color.White) + .border(1.dp, Color.Black) + .padding(6.dp), + verticalArrangement = Arrangement.spacedBy(4.dp) + ) { + // Pen grid — 2 columns + val pens = buildList { + add(Pen.BALLPEN to R.drawable.ballpen) + if (!GlobalAppSettings.current.monochromeMode) { + add(Pen.REDBALLPEN to R.drawable.ballpenred) + add(Pen.BLUEBALLPEN to R.drawable.ballpenblue) + add(Pen.GREENBALLPEN to R.drawable.ballpengreen) + } + if (GlobalAppSettings.current.neoTools) { + add(Pen.PENCIL to R.drawable.pencil) + add(Pen.BRUSH to R.drawable.brush) + } + add(Pen.FOUNTAIN to R.drawable.fountain) + add(Pen.MARKER to R.drawable.marker) + } + + // Show pens in rows of 4 + pens.chunked(4).forEach { row -> + Row(horizontalArrangement = Arrangement.spacedBy(2.dp)) { + row.forEach { (pen, iconId) -> + val isSelected = state.mode == Mode.Draw && state.pen == pen + val penSetting = state.penSettings[pen.penName] + val penColor = penSetting?.let { Color(it.color) } + + SidebarIconButton( + iconId = iconId, + contentDescription = pen.penName, + isSelected = isSelected, + penColor = penColor, + onClick = { + if (isSelected) { + // Toggle size picker for this pen + selectedPenForSize = if (selectedPenForSize == pen) null else pen + } else { + onSelectPen(pen) + selectedPenForSize = null + } + } + ) + } + } + } + + // Size/color picker for the selected pen + val penForSize = selectedPenForSize ?: (if (state.mode == Mode.Draw) state.pen else null) + if (penForSize != null) { + val penSetting = state.penSettings[penForSize.penName] ?: return@Column + val sizes = if (penForSize == Pen.MARKER) SIZES_MARKER_DEFAULT else SIZES_STROKES_DEFAULT + + Box( + Modifier + .size(width = 28.dp, height = 1.dp) + .background(Color.LightGray) + .align(Alignment.CenterHorizontally) + ) + + // Size buttons + Row( + modifier = Modifier + .background(Color.White) + .border(1.dp, Color.Black), + horizontalArrangement = Arrangement.Center + ) { + sizes.forEach { (label, size) -> + SidebarIconButton( + text = label, + contentDescription = "size $label", + isSelected = penSetting.strokeSize == size, + onClick = { + onChangeSetting(penForSize.penName, PenSetting(strokeSize = size, color = penSetting.color)) + } + ) + } + } + + // Color options + val colorOptions = if (GlobalAppSettings.current.monochromeMode) listOf( + Color.Black, Color.DarkGray, Color.Gray, Color.LightGray + ) else listOf( + Color.Red, Color.Green, Color.Blue, + Color.Cyan, Color.Magenta, Color.Yellow, + Color.Gray, Color.DarkGray, Color.Black, + ) + // Color grid in rows of 5 + colorOptions.chunked(5).forEach { row -> + Row(horizontalArrangement = Arrangement.spacedBy(1.dp)) { + row.forEach { color -> + Box( + modifier = Modifier + .size(32.dp) + .background(color) + .border( + 2.dp, + if (color == Color(penSetting.color)) Color.Black else Color.Transparent + ) + .clickable { + onChangeSetting( + penForSize.penName, + PenSetting( + strokeSize = penSetting.strokeSize, + color = android.graphics.Color.argb( + (color.alpha * 255).toInt(), + (color.red * 255).toInt(), + (color.green * 255).toInt(), + (color.blue * 255).toInt() + ) + ) + ) + } + ) + } + } + } + } + } + } +} + +// --- Eraser Flyout --- + +@Composable +private fun EraserFlyout( + value: Eraser, + onChange: (Eraser) -> Unit, + toggleScribbleToErase: (Boolean) -> Unit, + onClose: () -> Unit +) { + val context = LocalContext.current + + Popup( + alignment = Alignment.TopStart, + offset = IntOffset( + convertDpToPixel((SIDEBAR_WIDTH + 4).dp, context).toInt(), + 0 + ), + onDismissRequest = { onClose() }, + properties = PopupProperties(focusable = true) + ) { + Column( + modifier = Modifier + .background(Color.White) + .border(1.dp, Color.Black) + .padding(6.dp) + ) { + Row( + Modifier + .height(IntrinsicSize.Max) + .border(1.dp, Color.Black) + ) { + SidebarIconButton( + iconId = R.drawable.eraser, + contentDescription = "pen eraser", + isSelected = value == Eraser.PEN, + onClick = { onChange(Eraser.PEN) } + ) + SidebarIconButton( + iconId = R.drawable.eraser_select, + contentDescription = "select eraser", + isSelected = value == Eraser.SELECT, + onClick = { onChange(Eraser.SELECT) } + ) + } + Row( + modifier = Modifier + .padding(4.dp) + .height(26.dp), + verticalAlignment = Alignment.CenterVertically + ) { + BasicText( + text = stringResource(R.string.toolbar_scribble_to_erase_two_lined_short), + modifier = Modifier.padding(end = 6.dp), + style = TextStyle(color = Color.Black, fontSize = 13.sp) + ) + val initialState = GlobalAppSettings.current.scribbleToEraseEnabled + var isChecked by remember { mutableStateOf(initialState) } + Box( + modifier = Modifier + .size(15.dp) + .border(1.dp, Color.Black) + .background(if (isChecked) Color.Black else Color.White) + .clickable { + isChecked = !isChecked + toggleScribbleToErase(isChecked) + } + ) + } + } + } +} + +// --- Reusable sidebar button components --- + +@Composable +private fun SidebarIconButton( + iconId: Int? = null, + vectorIcon: ImageVector? = null, + text: String? = null, + contentDescription: String, + isSelected: Boolean = false, + penColor: Color? = null, + onClick: () -> Unit +) { + val bgColor = when { + isSelected && penColor != null -> penColor + isSelected -> Color.Black + else -> Color.White + } + val fgColor = when { + isSelected && penColor != null && penColor != Color.Black && penColor != Color.DarkGray -> Color.Black + isSelected -> Color.White + else -> Color.Black + } + val borderColor = if (isSelected) Color.Black else Color.Gray + + Box( + modifier = Modifier + .size(BUTTON_SIZE.dp) + .border(1.dp, borderColor, RoundedCornerShape(6.dp)) + .background(bgColor, RoundedCornerShape(6.dp)) + .noRippleClickable { onClick() }, + contentAlignment = Alignment.Center + ) { + when { + iconId != null -> Icon( + painterResource(iconId), + contentDescription = contentDescription, + tint = fgColor, + modifier = Modifier.size(ICON_SIZE.dp) + ) + vectorIcon != null -> Icon( + vectorIcon, + contentDescription = contentDescription, + tint = fgColor, + modifier = Modifier.size(ICON_SIZE.dp) + ) + text != null -> Text( + text, + fontSize = 14.sp, + fontWeight = FontWeight.Bold, + color = fgColor + ) + } + } +} + +@Composable +private fun SidebarTextButton( + text: String, + contentDescription: String, + isSelected: Boolean = false, + onClick: () -> Unit +) { + val bgColor = if (isSelected) Color.Black else Color.White + val fgColor = if (isSelected) Color.White else Color.Black + val borderColor = if (isSelected) Color.Black else Color.Gray + + Box( + modifier = Modifier + .size(BUTTON_SIZE.dp) + .border(1.dp, borderColor, RoundedCornerShape(6.dp)) + .background(bgColor, RoundedCornerShape(6.dp)) + .noRippleClickable { onClick() }, + contentAlignment = Alignment.Center + ) { + Text( + text, + fontSize = if (text.length > 2) 12.sp else 22.sp, + fontWeight = FontWeight.Bold, + color = fgColor + ) + } +} + +@Composable +private fun SidebarDivider() { + Box( + Modifier + .size(width = 28.dp, height = 1.dp) + .background(Color.LightGray) + ) +} diff --git a/app/src/main/java/com/ethran/notable/editor/ui/toolbar/Toolbar.kt b/app/src/main/java/com/ethran/notable/editor/ui/toolbar/Toolbar.kt index 418911fd..3396198e 100644 --- a/app/src/main/java/com/ethran/notable/editor/ui/toolbar/Toolbar.kt +++ b/app/src/main/java/com/ethran/notable/editor/ui/toolbar/Toolbar.kt @@ -422,33 +422,6 @@ fun Toolbar( Spacer(Modifier.weight(1f)) - Box( - Modifier - .fillMaxHeight() - .width(0.5.dp) - .background(Color.Black) - ) - - ToolbarButton( - onSelect = { - scope.launch { - controlTower.undo() - } - }, - iconId = R.drawable.undo, - contentDescription = "undo" - ) - - ToolbarButton( - onSelect = { - scope.launch { - controlTower.redo() - } - }, - iconId = R.drawable.redo, - contentDescription = "redo" - ) - Box( Modifier .fillMaxHeight() diff --git a/app/src/main/java/com/ethran/notable/editor/utils/draw.kt b/app/src/main/java/com/ethran/notable/editor/utils/draw.kt index 9fdda4ca..80790471 100644 --- a/app/src/main/java/com/ethran/notable/editor/utils/draw.kt +++ b/app/src/main/java/com/ethran/notable/editor/utils/draw.kt @@ -1,9 +1,12 @@ package com.ethran.notable.editor.utils import androidx.core.graphics.toRect +import com.ethran.notable.data.db.Annotation +import com.ethran.notable.data.db.AnnotationType import com.ethran.notable.data.db.Stroke import com.ethran.notable.data.db.StrokePoint import com.ethran.notable.editor.PageView +import com.ethran.notable.editor.state.AnnotationMode import com.onyx.android.sdk.api.device.epd.EpdController import io.shipbook.shipbooksdk.ShipBook @@ -45,3 +48,37 @@ fun handleDraw( log.e("Handle Draw: An error occurred while handling the drawing: ${e.message}") } } + +fun handleAnnotation( + page: PageView, + annotationMode: AnnotationMode, + touchPoints: List +): String? { + try { + if (touchPoints.isEmpty() || annotationMode == AnnotationMode.None) return null + + val boundingBox = calculateBoundingBox(touchPoints) { Pair(it.x, it.y) } + + val type = when (annotationMode) { + AnnotationMode.WikiLink -> AnnotationType.WIKILINK.name + AnnotationMode.Tag -> AnnotationType.TAG.name + else -> return null + } + + val annotation = Annotation( + type = type, + x = boundingBox.left, + y = boundingBox.top, + width = boundingBox.right - boundingBox.left, + height = boundingBox.bottom - boundingBox.top, + pageId = page.currentPageId + ) + + page.addAnnotations(listOf(annotation)) + page.drawAreaPageCoordinates(boundingBox.toRect()) + return annotation.id + } catch (e: Exception) { + log.e("Handle Annotation: An error occurred: ${e.message}") + return null + } +} From 706f9c8e0b9526c94bb3b5620607a13dad427ec4 Mon Sep 17 00:00:00 2001 From: Joshua Pham Date: Mon, 9 Mar 2026 19:45:28 -0400 Subject: [PATCH 16/23] Fix sidebar touch, annotation rendering, eraser, and HWR integration - Move sidebar outside canvas SurfaceView (Row layout) so finger taps work even when Onyx SDK raw drawing is active - Add dashed stroke preview in annotation color (blue/green) when annotation mode is active, with e-ink refresh after box creation - Increase annotation overlay visibility (alpha 130, border 4px) - Add annotation mode observer to update pen stroke style immediately - Eraser now removes annotation boxes in addition to strokes - HWR sync recognizes annotated text inline: strokes inside [[]] boxes become [[wiki links]], strokes inside # boxes become #tags - Add e-ink refresh helper to sidebar for button state updates - Skip scribble-to-erase when annotation mode is active Co-Authored-By: Claude Opus 4.6 --- .../com/ethran/notable/editor/EditorView.kt | 103 ++++++++++-------- .../editor/canvas/CanvasObserverRegistry.kt | 12 ++ .../notable/editor/canvas/OnyxInputHandler.kt | 55 ++++++++-- .../notable/editor/drawing/pageDrawing.kt | 12 +- .../ethran/notable/editor/ui/EditorSidebar.kt | 41 ++++++- .../com/ethran/notable/editor/utils/eraser.kt | 58 +++++++++- .../notable/gestures/EditorGestureReceiver.kt | 1 + .../com/ethran/notable/io/InboxSyncEngine.kt | 92 +++++++++++----- 8 files changed, 276 insertions(+), 98 deletions(-) diff --git a/app/src/main/java/com/ethran/notable/editor/EditorView.kt b/app/src/main/java/com/ethran/notable/editor/EditorView.kt index 80a47f99..f6f1533d 100644 --- a/app/src/main/java/com/ethran/notable/editor/EditorView.kt +++ b/app/src/main/java/com/ethran/notable/editor/EditorView.kt @@ -1,10 +1,14 @@ package com.ethran.notable.editor +import androidx.compose.foundation.layout.Box import androidx.compose.foundation.layout.BoxWithConstraints import androidx.compose.foundation.layout.Row import androidx.compose.foundation.layout.Spacer import androidx.compose.foundation.layout.fillMaxHeight +import androidx.compose.foundation.layout.fillMaxSize import androidx.compose.foundation.layout.fillMaxWidth +import androidx.compose.foundation.layout.width +import androidx.compose.ui.unit.dp import androidx.compose.runtime.Composable import androidx.compose.runtime.DisposableEffect import androidx.compose.runtime.LaunchedEffect @@ -23,6 +27,7 @@ import com.ethran.notable.editor.state.EditorState import com.ethran.notable.editor.state.History import com.ethran.notable.editor.ui.EditorSidebar import com.ethran.notable.editor.ui.EditorSurface +import com.ethran.notable.editor.ui.SIDEBAR_WIDTH import com.ethran.notable.editor.ui.HorizontalScrollIndicator import com.ethran.notable.editor.ui.InboxToolbar import com.ethran.notable.editor.ui.ScrollIndicator @@ -194,55 +199,59 @@ fun EditorView( InkaTheme { - EditorGestureReceiver(controlTower = editorControlTower) - EditorSurface( - appRepository = appRepository, - state = editorState, - page = page, - history = history - ) - SelectedBitmap( - context = context, - controlTower = editorControlTower - ) - Row( - modifier = Modifier - .fillMaxWidth() - .fillMaxHeight() - ) { - Spacer(modifier = Modifier.weight(1f)) - ScrollIndicator(state = editorState) - } - if (isInboxPage) { - // Inbox toolbar at top (Back, Capture, Save & Exit) - InboxToolbar( - selectedTags = selectedTags, - suggestedTags = suggestedTags, - isExpanded = editorState.isInboxTagsExpanded, - isToolbarOpen = editorState.isToolbarOpen, - onToggleExpanded = { - editorState.isInboxTagsExpanded = !editorState.isInboxTagsExpanded - }, - onToggleToolbar = { - editorState.isToolbarOpen = !editorState.isToolbarOpen - }, - onTagAdd = { tag -> - if (tag !in selectedTags) selectedTags.add(tag) - }, - onTagRemove = { tag -> selectedTags.remove(tag) }, - onSave = { - SyncState.launchSync( - appRepository, pageId, selectedTags.toList(), context + Row(modifier = Modifier.fillMaxSize()) { + // Left-edge sidebar — physically outside the canvas SurfaceView + // so finger taps always work even when Onyx SDK raw drawing is active + EditorSidebar(exportEngine, navController, appRepository, editorState, editorControlTower) + // Canvas area takes remaining space + Box(modifier = Modifier.weight(1f).fillMaxHeight()) { + EditorGestureReceiver(controlTower = editorControlTower) + EditorSurface( + appRepository = appRepository, + state = editorState, + page = page, + history = history + ) + SelectedBitmap( + context = context, + controlTower = editorControlTower + ) + Row( + modifier = Modifier + .fillMaxWidth() + .fillMaxHeight() + ) { + Spacer(modifier = Modifier.weight(1f)) + ScrollIndicator(state = editorState) + } + if (isInboxPage) { + InboxToolbar( + selectedTags = selectedTags, + suggestedTags = suggestedTags, + isExpanded = editorState.isInboxTagsExpanded, + isToolbarOpen = editorState.isToolbarOpen, + onToggleExpanded = { + editorState.isInboxTagsExpanded = !editorState.isInboxTagsExpanded + }, + onToggleToolbar = { + editorState.isToolbarOpen = !editorState.isToolbarOpen + }, + onTagAdd = { tag -> + if (tag !in selectedTags) selectedTags.add(tag) + }, + onTagRemove = { tag -> selectedTags.remove(tag) }, + onSave = { + SyncState.launchSync( + appRepository, pageId, selectedTags.toList(), context + ) + navController.popBackStack() + }, + onDiscard = { navController.popBackStack() } ) - navController.popBackStack() - }, - onDiscard = { navController.popBackStack() } - ) + } + HorizontalScrollIndicator(state = editorState) + } } - // Left-edge sidebar with all tools - EditorSidebar(exportEngine, navController, appRepository, editorState, editorControlTower) - HorizontalScrollIndicator(state = editorState) - } } } diff --git a/app/src/main/java/com/ethran/notable/editor/canvas/CanvasObserverRegistry.kt b/app/src/main/java/com/ethran/notable/editor/canvas/CanvasObserverRegistry.kt index 3659a60b..110faa84 100644 --- a/app/src/main/java/com/ethran/notable/editor/canvas/CanvasObserverRegistry.kt +++ b/app/src/main/java/com/ethran/notable/editor/canvas/CanvasObserverRegistry.kt @@ -55,6 +55,7 @@ class CanvasObserverRegistry( observeIsDrawingSnapshot() observeToolbar() observeMode() + observeAnnotationMode() observeHistory() observeSaveCurrent() observeQuickNav() @@ -233,6 +234,17 @@ class CanvasObserverRegistry( } } + private fun observeAnnotationMode() { + coroutineScope.launch { + snapshotFlow { drawCanvas.getActualState().annotationMode }.drop(1).collect { + logCanvasObserver.v("annotation mode change: ${drawCanvas.getActualState().annotationMode}") + inputHandler.updatePenAndStroke() + // Briefly unfreeze e-ink display so sidebar button state becomes visible + refreshManager.refreshUi(null) + } + } + } + @OptIn(FlowPreview::class) private fun observeHistory() { coroutineScope.launch { diff --git a/app/src/main/java/com/ethran/notable/editor/canvas/OnyxInputHandler.kt b/app/src/main/java/com/ethran/notable/editor/canvas/OnyxInputHandler.kt index d72f74d0..16933d39 100644 --- a/app/src/main/java/com/ethran/notable/editor/canvas/OnyxInputHandler.kt +++ b/app/src/main/java/com/ethran/notable/editor/canvas/OnyxInputHandler.kt @@ -30,6 +30,8 @@ import com.ethran.notable.editor.utils.handleSelect import com.ethran.notable.editor.utils.onSurfaceInit import com.ethran.notable.editor.utils.partialRefreshRegionOnce import com.ethran.notable.editor.utils.prepareForPartialUpdate +import com.ethran.notable.editor.utils.refreshScreenRegion +import com.ethran.notable.editor.utils.resetScreenFreeze import com.ethran.notable.editor.utils.restoreDefaults import com.ethran.notable.editor.utils.setupSurface import com.ethran.notable.editor.utils.transformToLine @@ -132,6 +134,20 @@ class OnyxInputHandler( fun updatePenAndStroke() { if(touchHelper == null) return log.i("Update pen and stroke: pen=${state.pen}, mode=${state.mode}") + // When annotation mode is active, show a dashed outline instead of the normal pen stroke + if (state.annotationMode != AnnotationMode.None) { + val annotColor = if (state.annotationMode == AnnotationMode.WikiLink) + Color.rgb(0, 100, 255) else Color.rgb(0, 180, 0) + touchHelper!!.setStrokeStyle(Pen.DASHED.strokeStyle) + ?.setStrokeWidth(3f) + ?.setStrokeColor(annotColor) + Device.currentDevice().setStrokeParameters( + Pen.DASHED.strokeStyle, + floatArrayOf(5f, 9f, 9f, 0f) + ) + return + } + when (state.mode) { Mode.Draw, Mode.Line -> { val setting = state.penSettings[state.pen.penName] ?: return @@ -293,21 +309,25 @@ class OnyxInputHandler( val lock = System.currentTimeMillis() log.d("lock obtained in ${lock - startTime} ms") - // Thread.sleep(1000) // transform points to page space val scaledPoints = copyInput(plist.points, page.scroll, page.zoomLevel.value) - val firstPointTime = plist.points.first().timestamp - val erasedByScribbleDirtyRect = handleScribbleToErase( - page, - scaledPoints, - history, - drawCanvas.getActualState().pen, - currentLastStrokeEndTime, - firstPointTime - ) + val annotMode = drawCanvas.getActualState().annotationMode + // Skip scribble-to-erase when in annotation mode + val erasedByScribbleDirtyRect = if (annotMode != AnnotationMode.None) { + null + } else { + val firstPointTime = plist.points.first().timestamp + handleScribbleToErase( + page, + scaledPoints, + history, + drawCanvas.getActualState().pen, + currentLastStrokeEndTime, + firstPointTime + ) + } if (erasedByScribbleDirtyRect.isNullOrEmpty()) { - val annotMode = drawCanvas.getActualState().annotationMode if (annotMode != AnnotationMode.None) { log.d("Creating annotation...") handleAnnotation( @@ -317,8 +337,19 @@ class OnyxInputHandler( ) // Reset to one-shot: annotation mode turns off after one box drawCanvas.getActualState().annotationMode = AnnotationMode.None - // Refresh canvas to show the annotation overlay + // Redraw canvas, then do a full GC refresh to clear + // the Onyx SDK's dashed line preview artifacts from e-ink drawCanvas.drawCanvasToView(null) + resetScreenFreeze(touchHelper!!) + val bounds = calculateBoundingBox(scaledPoints) { Pair(it.x, it.y) } + val padding = 20 + val dirtyRect = Rect( + (bounds.left - page.scroll.x - padding).toInt(), + (bounds.top - page.scroll.y - padding).toInt(), + (bounds.right - page.scroll.x + padding).toInt(), + (bounds.bottom - page.scroll.y + padding).toInt() + ) + refreshScreenRegion(drawCanvas, dirtyRect) } else { log.d("Drawing...") // draw the stroke diff --git a/app/src/main/java/com/ethran/notable/editor/drawing/pageDrawing.kt b/app/src/main/java/com/ethran/notable/editor/drawing/pageDrawing.kt index 7da56a4e..6c97eb36 100644 --- a/app/src/main/java/com/ethran/notable/editor/drawing/pageDrawing.kt +++ b/app/src/main/java/com/ethran/notable/editor/drawing/pageDrawing.kt @@ -32,23 +32,23 @@ private val pageDrawingLog = ShipBook.getLogger("PageDrawingLog") // Annotation overlay paints private val wikiLinkPaint = Paint().apply { - color = Color.argb(50, 0, 100, 255) // semi-transparent blue fill + color = Color.argb(130, 0, 100, 255) // semi-transparent blue fill style = Paint.Style.FILL } private val wikiLinkBorderPaint = Paint().apply { - color = Color.argb(180, 0, 100, 255) // blue border + color = Color.argb(255, 0, 80, 220) // solid blue border style = Paint.Style.STROKE - strokeWidth = 2f + strokeWidth = 4f isAntiAlias = true } private val tagPaint = Paint().apply { - color = Color.argb(50, 0, 180, 0) // semi-transparent green fill + color = Color.argb(130, 0, 180, 0) // semi-transparent green fill style = Paint.Style.FILL } private val tagBorderPaint = Paint().apply { - color = Color.argb(180, 0, 180, 0) // green border + color = Color.argb(255, 0, 150, 0) // solid green border style = Paint.Style.STROKE - strokeWidth = 2f + strokeWidth = 4f isAntiAlias = true } diff --git a/app/src/main/java/com/ethran/notable/editor/ui/EditorSidebar.kt b/app/src/main/java/com/ethran/notable/editor/ui/EditorSidebar.kt index 08036c3d..a4bd5988 100644 --- a/app/src/main/java/com/ethran/notable/editor/ui/EditorSidebar.kt +++ b/app/src/main/java/com/ethran/notable/editor/ui/EditorSidebar.kt @@ -34,6 +34,7 @@ import androidx.compose.ui.Modifier import androidx.compose.ui.graphics.Color import androidx.compose.ui.graphics.vector.ImageVector import androidx.compose.ui.platform.LocalContext +import androidx.compose.ui.platform.LocalView import androidx.compose.ui.res.painterResource import androidx.compose.ui.res.stringResource import androidx.compose.ui.text.TextStyle @@ -70,6 +71,8 @@ import compose.icons.FeatherIcons import compose.icons.feathericons.EyeOff import compose.icons.feathericons.RefreshCcw import compose.icons.feathericons.Clipboard +import com.onyx.android.sdk.api.device.epd.EpdController +import com.onyx.android.sdk.api.device.epd.UpdateMode import io.shipbook.shipbooksdk.ShipBook import kotlinx.coroutines.Dispatchers import kotlinx.coroutines.launch @@ -80,7 +83,7 @@ private val log = ShipBook.getLogger("EditorSidebar") private val SIZES_STROKES_DEFAULT = listOf("S" to 3f, "M" to 5f, "L" to 10f, "XL" to 20f) private val SIZES_MARKER_DEFAULT = listOf("M" to 25f, "L" to 40f, "XL" to 60f, "XXL" to 80f) -private const val SIDEBAR_WIDTH = 56 +const val SIDEBAR_WIDTH = 56 private const val BUTTON_SIZE = 46 private const val ICON_SIZE = 26 @@ -95,8 +98,28 @@ fun EditorSidebar( ) { val scope = rememberCoroutineScope() val context = LocalContext.current + val view = LocalView.current val zoomLevel by state.pageView.zoomLevel.collectAsState() + // Force e-ink refresh when sidebar state changes. + // The Onyx SDK sets a global display scheme that suppresses normal view updates, + // so we need to explicitly tell the e-ink controller to refresh. + fun refreshSidebar() { + view.postInvalidate() + try { + val sidebarPx = convertDpToPixel(SIDEBAR_WIDTH.dp, context).toInt() + // Use the root view's location to get absolute screen coordinates + val loc = IntArray(2) + view.getLocationOnScreen(loc) + EpdController.refreshScreenRegion( + view, loc[0], loc[1], sidebarPx, view.height, UpdateMode.GC + ) + log.i("Sidebar e-ink refresh triggered") + } catch (e: Exception) { + log.w("E-ink sidebar refresh failed: ${e.message}") + } + } + val pickMedia = rememberLauncherForActivityResult(contract = PickVisualMedia()) { uri -> if (uri == null) { log.w("PickVisualMedia: uri is null") @@ -182,6 +205,8 @@ fun EditorSidebar( modifier = Modifier .width(SIDEBAR_WIDTH.dp) .fillMaxHeight() + .background(Color.White) + .noRippleClickable { log.i("Sidebar background tapped (event consumed)") } .padding(horizontal = 4.dp), verticalArrangement = Arrangement.Center, horizontalAlignment = Alignment.CenterHorizontally @@ -190,7 +215,10 @@ fun EditorSidebar( SidebarIconButton( vectorIcon = FeatherIcons.EyeOff, contentDescription = "close toolbar", - onClick = { state.isToolbarOpen = false } + onClick = { + log.i("Close toolbar tapped") + state.isToolbarOpen = false + } ) SidebarDivider() @@ -204,6 +232,7 @@ fun EditorSidebar( contentDescription = "pen", isSelected = state.mode == Mode.Draw, onClick = { + log.i("Pen button tapped, current mode=${state.mode}") if (state.mode == Mode.Draw) { isPenPickerOpen = !isPenPickerOpen isEraserMenuOpen = false @@ -211,6 +240,7 @@ fun EditorSidebar( state.mode = Mode.Draw isPenPickerOpen = false } + refreshSidebar() } ) if (isPenPickerOpen) { @@ -237,6 +267,7 @@ fun EditorSidebar( state.mode = Mode.Erase isEraserMenuOpen = false } + refreshSidebar() } ) if (isEraserMenuOpen) { @@ -264,6 +295,7 @@ fun EditorSidebar( state.mode = Mode.Select isPenPickerOpen = false isEraserMenuOpen = false + refreshSidebar() } ) @@ -277,6 +309,7 @@ fun EditorSidebar( else state.mode = Mode.Line isPenPickerOpen = false isEraserMenuOpen = false + refreshSidebar() } ) @@ -308,12 +341,15 @@ fun EditorSidebar( contentDescription = "Wiki link", isSelected = state.annotationMode == AnnotationMode.WikiLink, onClick = { + log.i("[[ button tapped, annotationMode=${state.annotationMode}") state.annotationMode = if (state.annotationMode == AnnotationMode.WikiLink) AnnotationMode.None else AnnotationMode.WikiLink + log.i("[[ button: annotationMode now=${state.annotationMode}, setting mode=Draw") // Ensure we're in draw mode so the stylus gesture gets captured if (state.annotationMode != AnnotationMode.None) state.mode = Mode.Draw isPenPickerOpen = false isEraserMenuOpen = false + refreshSidebar() } ) @@ -328,6 +364,7 @@ fun EditorSidebar( if (state.annotationMode != AnnotationMode.None) state.mode = Mode.Draw isPenPickerOpen = false isEraserMenuOpen = false + refreshSidebar() } ) diff --git a/app/src/main/java/com/ethran/notable/editor/utils/eraser.kt b/app/src/main/java/com/ethran/notable/editor/utils/eraser.kt index 6b02b341..012b9f55 100644 --- a/app/src/main/java/com/ethran/notable/editor/utils/eraser.kt +++ b/app/src/main/java/com/ethran/notable/editor/utils/eraser.kt @@ -4,7 +4,9 @@ import android.graphics.Paint import android.graphics.Path import android.graphics.Rect import android.graphics.RectF +import android.graphics.Region import com.ethran.notable.data.datastore.GlobalAppSettings +import com.ethran.notable.data.db.Annotation import com.ethran.notable.data.db.Stroke import com.ethran.notable.data.db.StrokePoint import com.ethran.notable.data.model.SimplePointF @@ -164,20 +166,68 @@ fun handleErase( } val deletedStrokes = selectStrokesFromPath(page.strokes, outPath) + val deletedAnnotations = selectAnnotationsFromPath(page.annotations, outPath) val deletedStrokeIds = deletedStrokes.map { it.id } + val deletedAnnotationIds = deletedAnnotations.map { it.id } - if (deletedStrokes.isEmpty()) return null - page.removeStrokes(deletedStrokeIds) + if (deletedStrokes.isEmpty() && deletedAnnotations.isEmpty()) return null - history.addOperationsToHistory(listOf(Operation.AddStroke(deletedStrokes))) + if (deletedStrokes.isNotEmpty()) { + page.removeStrokes(deletedStrokeIds) + history.addOperationsToHistory(listOf(Operation.AddStroke(deletedStrokes))) + } + if (deletedAnnotations.isNotEmpty()) { + page.removeAnnotations(deletedAnnotationIds) + } - val effectedArea = page.toScreenCoordinates(strokeBounds(deletedStrokes)) + val strokeArea = if (deletedStrokes.isNotEmpty()) strokeBounds(deletedStrokes) else null + val annotArea = if (deletedAnnotations.isNotEmpty()) annotationBounds(deletedAnnotations) else null + val combinedArea = when { + strokeArea != null && annotArea != null -> { + strokeArea.union(annotArea); strokeArea + } + strokeArea != null -> strokeArea + annotArea != null -> annotArea + else -> return null + } + + val effectedArea = page.toScreenCoordinates(combinedArea) page.drawAreaScreenCoordinates(screenArea = effectedArea) return effectedArea } +private fun selectAnnotationsFromPath(annotations: List, eraserPath: Path): List { + val eraserRegion = Region() + val clipBounds = Region(-10000, -10000, 10000, 10000) + eraserRegion.setPath(eraserPath, clipBounds) + + return annotations.filter { annotation -> + val annotRect = Rect( + annotation.x.toInt(), annotation.y.toInt(), + (annotation.x + annotation.width).toInt(), + (annotation.y + annotation.height).toInt() + ) + val annotRegion = Region(annotRect) + !annotRegion.op(eraserRegion, Region.Op.INTERSECT).not() + } +} + +private fun annotationBounds(annotations: List): Rect { + var left = Float.MAX_VALUE + var top = Float.MAX_VALUE + var right = Float.MIN_VALUE + var bottom = Float.MIN_VALUE + for (a in annotations) { + if (a.x < left) left = a.x + if (a.y < top) top = a.y + if (a.x + a.width > right) right = a.x + a.width + if (a.y + a.height > bottom) bottom = a.y + a.height + } + return Rect(left.toInt(), top.toInt(), right.toInt(), bottom.toInt()) +} + // points is in page coordinates, returns effected area. fun cleanAllStrokes( page: PageView, history: History diff --git a/app/src/main/java/com/ethran/notable/gestures/EditorGestureReceiver.kt b/app/src/main/java/com/ethran/notable/gestures/EditorGestureReceiver.kt index 87097391..ecc5c7b8 100644 --- a/app/src/main/java/com/ethran/notable/gestures/EditorGestureReceiver.kt +++ b/app/src/main/java/com/ethran/notable/gestures/EditorGestureReceiver.kt @@ -50,6 +50,7 @@ fun EditorGestureReceiver( try { // Detect initial touch val down = awaitFirstDown() + log.i("GestureReceiver got touch at (${down.position.x}, ${down.position.y}), type=${down.type}, consumed=${down.isConsumed}") // We should not get any stylus events require( diff --git a/app/src/main/java/com/ethran/notable/io/InboxSyncEngine.kt b/app/src/main/java/com/ethran/notable/io/InboxSyncEngine.kt index f7b2a76f..08522ecc 100644 --- a/app/src/main/java/com/ethran/notable/io/InboxSyncEngine.kt +++ b/app/src/main/java/com/ethran/notable/io/InboxSyncEngine.kt @@ -1,8 +1,12 @@ package com.ethran.notable.io import android.content.Context +import android.graphics.RectF import android.os.Environment import com.ethran.notable.data.AppRepository +import com.ethran.notable.data.db.Annotation +import com.ethran.notable.data.db.AnnotationType +import com.ethran.notable.data.db.Stroke import com.ethran.notable.data.datastore.GlobalAppSettings import io.shipbook.shipbooksdk.ShipBook import java.io.File @@ -17,6 +21,7 @@ object InboxSyncEngine { /** * Sync an inbox page to Obsidian. Tags come from the UI (pill selection), * content is recognized from all strokes on the page via Onyx HWR (MyScript). + * Annotation boxes mark regions to wrap in [[wiki links]] or #tags. */ suspend fun syncInboxPage( appRepository: AppRepository, @@ -28,44 +33,60 @@ object InboxSyncEngine { val pageWithStrokes = appRepository.pageRepository.getWithStrokeById(pageId) val page = pageWithStrokes.page - val strokes = pageWithStrokes.strokes + val allStrokes = pageWithStrokes.strokes + val annotations = appRepository.annotationRepository.getByPageId(pageId) - if (strokes.isEmpty() && tags.isEmpty()) { + if (allStrokes.isEmpty() && tags.isEmpty()) { log.i("No strokes and no tags on inbox page, skipping sync") return } - val contentText = if (strokes.isNotEmpty()) { - log.i("Recognizing ${strokes.size} content strokes") - try { - val serviceReady = OnyxHWREngine.bindAndAwait(context) - if (serviceReady) { - val result = OnyxHWREngine.recognizeStrokes( - strokes, - viewWidth = 1404f, - viewHeight = 1872f - ) - if (result != null) { - log.i("OnyxHWR succeeded, ${result.length} chars") - postProcessRecognition(result) - } else { - log.w("OnyxHWR returned null") - "" + val serviceReady = try { + OnyxHWREngine.bindAndAwait(context) + } catch (e: Exception) { + log.e("OnyxHWR bind failed: ${e.message}") + false + } + + // 1. Recognize ALL strokes together to preserve natural text flow + var fullText = if (serviceReady && allStrokes.isNotEmpty()) { + log.i("Recognizing all ${allStrokes.size} strokes") + val result = recognizeStrokesSafe(allStrokes) + postProcessRecognition(result) + } else "" + + log.i("Full recognized text: '${fullText.take(200)}'") + + // 2. For each annotation, recognize its strokes separately to get the annotation text, + // then find and replace that text inline with the wrapped version + if (serviceReady && annotations.isNotEmpty()) { + for (annotation in annotations) { + val annotRect = RectF( + annotation.x, annotation.y, + annotation.x + annotation.width, + annotation.y + annotation.height + ) + val overlapping = findStrokesInRect(allStrokes, annotRect) + if (overlapping.isNotEmpty()) { + val annotText = recognizeStrokesSafe(overlapping).trim() + if (annotText.isNotBlank()) { + log.i("Annotation ${annotation.type}: '$annotText'") + val wrapped = when (annotation.type) { + AnnotationType.WIKILINK.name -> "[[${annotText}]]" + AnnotationType.TAG.name -> "#${annotText.replace(" ", "-")}" + else -> annotText + } + // Replace the annotation text inline in the full recognized text + fullText = fullText.replaceFirst(annotText, wrapped) } - } else { - log.w("OnyxHWR service not available") - "" } - } catch (e: Exception) { - log.e("OnyxHWR failed: ${e.message}") - "" } - } else "" + } - log.i("Recognized content: '${contentText.take(100)}'") + val finalContent = fullText val createdDate = SimpleDateFormat("yyyy-MM-dd", Locale.US).format(page.createdAt) - val markdown = generateMarkdown(createdDate, tags, contentText) + val markdown = generateMarkdown(createdDate, tags, finalContent) val inboxPath = GlobalAppSettings.current.obsidianInboxPath writeMarkdownFile(markdown, page.createdAt, inboxPath) @@ -73,6 +94,23 @@ object InboxSyncEngine { log.i("Inbox sync complete for page $pageId") } + private suspend fun recognizeStrokesSafe(strokes: List): String { + return try { + OnyxHWREngine.recognizeStrokes(strokes, viewWidth = 1404f, viewHeight = 1872f) ?: "" + } catch (e: Exception) { + log.e("OnyxHWR failed: ${e.message}") + "" + } + } + + private fun findStrokesInRect(strokes: List, rect: RectF): List { + return strokes.filter { stroke -> + val strokeRect = RectF(stroke.left, stroke.top, stroke.right, stroke.bottom) + RectF.intersects(strokeRect, rect) + } + } + + /** * Post-process recognition output: * - Normalize any bracket/paren wrapping to [[wiki links]] From 2f1e3674c6d4675daa6193f694ca868fe93d1d41 Mon Sep 17 00:00:00 2001 From: Joshua Pham Date: Mon, 9 Mar 2026 20:16:55 -0400 Subject: [PATCH 17/23] Render annotation overlays as [[ ]] brackets and # glyphs, fix HWR sync accuracy - Replace solid rectangle overlays with typographic indicators: wiki links show [[ ]] brackets flanking content, tags show # prefix, both with light color wash, rounded corners, and subtle underline - Expand canvas clip and e-ink refresh regions to include bracket/hash glyphs that extend beyond the annotation bounding box - Fix HWR annotation wrapping by diffing full-page vs non-annotation stroke recognition instead of recognizing annotation strokes in isolation (which had worse accuracy due to less context, e.g. "pkm" recognized as "plan") - Add 10s timeout to OnyxHWR calls to prevent infinite hang on service disconnect Co-Authored-By: Claude Opus 4.6 --- .../notable/editor/canvas/OnyxInputHandler.kt | 10 +- .../notable/editor/drawing/pageDrawing.kt | 91 ++++++++++++--- .../com/ethran/notable/editor/utils/draw.kt | 14 ++- .../com/ethran/notable/io/InboxSyncEngine.kt | 105 +++++++++++++++--- .../com/ethran/notable/io/OnyxHWREngine.kt | 65 ++++++----- 5 files changed, 223 insertions(+), 62 deletions(-) diff --git a/app/src/main/java/com/ethran/notable/editor/canvas/OnyxInputHandler.kt b/app/src/main/java/com/ethran/notable/editor/canvas/OnyxInputHandler.kt index 16933d39..55acbc43 100644 --- a/app/src/main/java/com/ethran/notable/editor/canvas/OnyxInputHandler.kt +++ b/app/src/main/java/com/ethran/notable/editor/canvas/OnyxInputHandler.kt @@ -342,11 +342,17 @@ class OnyxInputHandler( drawCanvas.drawCanvasToView(null) resetScreenFreeze(touchHelper!!) val bounds = calculateBoundingBox(scaledPoints) { Pair(it.x, it.y) } + // Extra padding accounts for rendered bracket/hash glyphs + // that extend beyond the annotation bounding box. + // Brackets use fontSize = height*0.55, "[[" is ~1.2x fontSize wide, + // plus gap, so we need ~height on each side to be safe. + val boxHeight = bounds.bottom - bounds.top + val extraHorizontal = (boxHeight * 1.2f).toInt().coerceAtLeast(60) val padding = 20 val dirtyRect = Rect( - (bounds.left - page.scroll.x - padding).toInt(), + (bounds.left - page.scroll.x - padding - extraHorizontal).toInt(), (bounds.top - page.scroll.y - padding).toInt(), - (bounds.right - page.scroll.x + padding).toInt(), + (bounds.right - page.scroll.x + padding + extraHorizontal).toInt(), (bounds.bottom - page.scroll.y + padding).toInt() ) refreshScreenRegion(drawCanvas, dirtyRect) diff --git a/app/src/main/java/com/ethran/notable/editor/drawing/pageDrawing.kt b/app/src/main/java/com/ethran/notable/editor/drawing/pageDrawing.kt index 6c97eb36..38e217de 100644 --- a/app/src/main/java/com/ethran/notable/editor/drawing/pageDrawing.kt +++ b/app/src/main/java/com/ethran/notable/editor/drawing/pageDrawing.kt @@ -31,24 +31,29 @@ import io.shipbook.shipbooksdk.ShipBook private val pageDrawingLog = ShipBook.getLogger("PageDrawingLog") // Annotation overlay paints -private val wikiLinkPaint = Paint().apply { - color = Color.argb(130, 0, 100, 255) // semi-transparent blue fill +private val wikiLinkFillPaint = Paint().apply { + color = Color.argb(50, 0, 100, 255) // very light blue wash style = Paint.Style.FILL } -private val wikiLinkBorderPaint = Paint().apply { - color = Color.argb(255, 0, 80, 220) // solid blue border - style = Paint.Style.STROKE - strokeWidth = 4f +private val wikiLinkBracketPaint = Paint().apply { + color = Color.argb(200, 0, 80, 220) // blue brackets + style = Paint.Style.FILL isAntiAlias = true + typeface = android.graphics.Typeface.create(android.graphics.Typeface.MONOSPACE, android.graphics.Typeface.BOLD) } -private val tagPaint = Paint().apply { - color = Color.argb(130, 0, 180, 0) // semi-transparent green fill +private val tagFillPaint = Paint().apply { + color = Color.argb(50, 0, 180, 0) // very light green wash style = Paint.Style.FILL } -private val tagBorderPaint = Paint().apply { - color = Color.argb(255, 0, 150, 0) // solid green border +private val tagHashPaint = Paint().apply { + color = Color.argb(200, 0, 150, 0) // green hash symbol + style = Paint.Style.FILL + isAntiAlias = true + typeface = android.graphics.Typeface.create(android.graphics.Typeface.MONOSPACE, android.graphics.Typeface.BOLD) +} +private val annotationUnderlinePaint = Paint().apply { style = Paint.Style.STROKE - strokeWidth = 4f + strokeWidth = 3f isAntiAlias = true } @@ -59,13 +64,67 @@ fun drawAnnotation(canvas: Canvas, annotation: Annotation, offset: Offset) { annotation.x + annotation.width + offset.x, annotation.y + annotation.height + offset.y ) - val (fillPaint, borderPaint) = if (annotation.type == AnnotationType.WIKILINK.name) { - wikiLinkPaint to wikiLinkBorderPaint + val boxHeight = rect.height() + val padding = boxHeight * 0.15f + + if (annotation.type == AnnotationType.WIKILINK.name) { + // Size brackets proportional to annotation height + val bracketSize = boxHeight * 0.55f + wikiLinkBracketPaint.textSize = bracketSize + + val bracketWidth = wikiLinkBracketPaint.measureText("[[") + val bracketGap = padding * 0.5f + + // Expanded rect includes brackets + val expandedRect = RectF( + rect.left - bracketWidth - bracketGap, + rect.top - padding, + rect.right + bracketWidth + bracketGap, + rect.bottom + padding + ) + + // Light fill over the whole area + val cornerRadius = boxHeight * 0.15f + canvas.drawRoundRect(expandedRect, cornerRadius, cornerRadius, wikiLinkFillPaint) + + // Draw [[ on the left + val textY = rect.centerY() + bracketSize * 0.35f + canvas.drawText("[[", expandedRect.left + bracketGap * 0.5f, textY, wikiLinkBracketPaint) + + // Draw ]] on the right + canvas.drawText("]]", rect.right + bracketGap * 0.5f, textY, wikiLinkBracketPaint) + + // Subtle underline under the handwritten content + annotationUnderlinePaint.color = Color.argb(120, 0, 80, 220) + canvas.drawLine(rect.left, rect.bottom + padding * 0.3f, rect.right, rect.bottom + padding * 0.3f, annotationUnderlinePaint) } else { - tagPaint to tagBorderPaint + // TAG: draw # prefix + val hashSize = boxHeight * 0.65f + tagHashPaint.textSize = hashSize + + val hashWidth = tagHashPaint.measureText("#") + val hashGap = padding * 0.6f + + // Expanded rect includes # prefix + val expandedRect = RectF( + rect.left - hashWidth - hashGap, + rect.top - padding, + rect.right + padding, + rect.bottom + padding + ) + + // Light fill over the whole area + val cornerRadius = boxHeight * 0.15f + canvas.drawRoundRect(expandedRect, cornerRadius, cornerRadius, tagFillPaint) + + // Draw # to the left + val textY = rect.centerY() + hashSize * 0.35f + canvas.drawText("#", expandedRect.left + hashGap * 0.3f, textY, tagHashPaint) + + // Subtle underline under the handwritten content + annotationUnderlinePaint.color = Color.argb(120, 0, 150, 0) + canvas.drawLine(rect.left, rect.bottom + padding * 0.3f, rect.right, rect.bottom + padding * 0.3f, annotationUnderlinePaint) } - canvas.drawRect(rect, fillPaint) - canvas.drawRect(rect, borderPaint) } diff --git a/app/src/main/java/com/ethran/notable/editor/utils/draw.kt b/app/src/main/java/com/ethran/notable/editor/utils/draw.kt index 80790471..c5fda8f2 100644 --- a/app/src/main/java/com/ethran/notable/editor/utils/draw.kt +++ b/app/src/main/java/com/ethran/notable/editor/utils/draw.kt @@ -1,5 +1,6 @@ package com.ethran.notable.editor.utils +import android.graphics.Rect import androidx.core.graphics.toRect import com.ethran.notable.data.db.Annotation import com.ethran.notable.data.db.AnnotationType @@ -75,7 +76,18 @@ fun handleAnnotation( ) page.addAnnotations(listOf(annotation)) - page.drawAreaPageCoordinates(boundingBox.toRect()) + // Expand redraw area to include rendered bracket/hash glyphs + // that extend beyond the annotation bounding box + val boxHeight = (boundingBox.bottom - boundingBox.top) + val extraH = (boxHeight * 1.2f).toInt().coerceAtLeast(60) + val extraV = (boxHeight * 0.2f).toInt().coerceAtLeast(10) + val expandedBounds = Rect( + (boundingBox.left - extraH).toInt(), + (boundingBox.top - extraV).toInt(), + (boundingBox.right + extraH).toInt(), + (boundingBox.bottom + extraV).toInt() + ) + page.drawAreaPageCoordinates(expandedBounds) return annotation.id } catch (e: Exception) { log.e("Handle Annotation: An error occurred: ${e.message}") diff --git a/app/src/main/java/com/ethran/notable/io/InboxSyncEngine.kt b/app/src/main/java/com/ethran/notable/io/InboxSyncEngine.kt index 08522ecc..e8c7b564 100644 --- a/app/src/main/java/com/ethran/notable/io/InboxSyncEngine.kt +++ b/app/src/main/java/com/ethran/notable/io/InboxSyncEngine.kt @@ -57,28 +57,49 @@ object InboxSyncEngine { log.i("Full recognized text: '${fullText.take(200)}'") - // 2. For each annotation, recognize its strokes separately to get the annotation text, - // then find and replace that text inline with the wrapped version + // 2. Find annotation text by diffing full recognition vs non-annotation recognition. + // This is more reliable than recognizing annotation strokes alone (fewer strokes = + // less context = worse HWR accuracy). if (serviceReady && annotations.isNotEmpty()) { + // Collect stroke IDs that fall inside any annotation box + val annotationStrokeIds = mutableSetOf() for (annotation in annotations) { val annotRect = RectF( annotation.x, annotation.y, annotation.x + annotation.width, annotation.y + annotation.height ) - val overlapping = findStrokesInRect(allStrokes, annotRect) - if (overlapping.isNotEmpty()) { - val annotText = recognizeStrokesSafe(overlapping).trim() - if (annotText.isNotBlank()) { - log.i("Annotation ${annotation.type}: '$annotText'") - val wrapped = when (annotation.type) { - AnnotationType.WIKILINK.name -> "[[${annotText}]]" - AnnotationType.TAG.name -> "#${annotText.replace(" ", "-")}" - else -> annotText - } - // Replace the annotation text inline in the full recognized text - fullText = fullText.replaceFirst(annotText, wrapped) + findStrokesInRect(allStrokes, annotRect).forEach { annotationStrokeIds.add(it.id) } + } + + // Recognize non-annotation strokes to get "base text" + val nonAnnotStrokes = allStrokes.filter { it.id !in annotationStrokeIds } + val baseText = if (nonAnnotStrokes.isNotEmpty()) { + postProcessRecognition(recognizeStrokesSafe(nonAnnotStrokes)) + } else "" + log.i("Base text (without annotations): '${baseText.take(200)}'") + + // Diff to find word segments in full text that are missing from base text. + // These are the annotation-contributed words, in reading order. + val annotationTexts = diffWords(baseText, fullText) + log.i("Diffed annotation texts: $annotationTexts") + + // Sort annotations by position (top-to-bottom, left-to-right) to match diff order + val sortedAnnotations = annotations.sortedWith(compareBy({ it.y }, { it.x })) + + if (annotationTexts.size != sortedAnnotations.size) { + log.w("Diff found ${annotationTexts.size} segments but have ${sortedAnnotations.size} annotations — skipping wrapping") + } else { + for ((i, annotation) in sortedAnnotations.withIndex()) { + val annotText = annotationTexts[i] + if (annotText.isBlank()) continue + log.i("Annotation ${annotation.type}: '$annotText'") + val wrapped = when (annotation.type) { + AnnotationType.WIKILINK.name -> "[[${annotText}]]" + AnnotationType.TAG.name -> "#${annotText.replace(" ", "-")}" + else -> annotText } + fullText = fullText.replaceFirst(annotText, wrapped) } } } @@ -103,6 +124,62 @@ object InboxSyncEngine { } } + /** + * Find contiguous word segments in [fullText] that are absent from [baseText]. + * Uses a simple word-level LCS diff. Returns segments in the order they appear + * in fullText, with consecutive inserted words joined by spaces. + * + * Example: baseText="This is a document", fullText="This is a new document pkm" + * → ["new", "pkm"] + */ + private fun diffWords(baseText: String, fullText: String): List { + val baseWords = baseText.split(Regex("\\s+")).filter { it.isNotBlank() } + val fullWords = fullText.split(Regex("\\s+")).filter { it.isNotBlank() } + + // LCS to find which words in fullText are "matched" to baseText + val m = baseWords.size + val n = fullWords.size + val dp = Array(m + 1) { IntArray(n + 1) } + for (i in 1..m) { + for (j in 1..n) { + dp[i][j] = if (baseWords[i - 1].equals(fullWords[j - 1], ignoreCase = true)) { + dp[i - 1][j - 1] + 1 + } else { + maxOf(dp[i - 1][j], dp[i][j - 1]) + } + } + } + + // Backtrack to find which fullText words are NOT in the LCS (= annotation words) + val matched = BooleanArray(n) + var i = m; var j = n + while (i > 0 && j > 0) { + if (baseWords[i - 1].equals(fullWords[j - 1], ignoreCase = true)) { + matched[j - 1] = true + i--; j-- + } else if (dp[i - 1][j] > dp[i][j - 1]) { + i-- + } else { + j-- + } + } + + // Group consecutive unmatched words into segments + val segments = mutableListOf() + var current = mutableListOf() + for (k in fullWords.indices) { + if (!matched[k]) { + current.add(fullWords[k]) + } else if (current.isNotEmpty()) { + segments.add(current.joinToString(" ")) + current = mutableListOf() + } + } + if (current.isNotEmpty()) segments.add(current.joinToString(" ")) + + return segments + } + private fun findStrokesInRect(strokes: List, rect: RectF): List { return strokes.filter { stroke -> val strokeRect = RectF(stroke.left, stroke.top, stroke.right, stroke.bottom) diff --git a/app/src/main/java/com/ethran/notable/io/OnyxHWREngine.kt b/app/src/main/java/com/ethran/notable/io/OnyxHWREngine.kt index 28d9db06..1530f8d5 100644 --- a/app/src/main/java/com/ethran/notable/io/OnyxHWREngine.kt +++ b/app/src/main/java/com/ethran/notable/io/OnyxHWREngine.kt @@ -14,6 +14,7 @@ import com.onyx.android.sdk.hwr.service.HWROutputCallback import com.onyx.android.sdk.hwr.service.IHWRService import io.shipbook.shipbooksdk.ShipBook import kotlinx.coroutines.suspendCancellableCoroutine +import kotlinx.coroutines.withTimeoutOrNull import org.json.JSONObject import java.io.ByteArrayOutputStream import java.io.FileDescriptor @@ -134,38 +135,44 @@ object OnyxHWREngine { ?: throw IllegalStateException("Failed to create MemoryFile PFD") return try { - suspendCancellableCoroutine { cont -> - svc.batchRecognize(pfd, object : HWROutputCallback.Stub() { - override fun read(args: HWROutputArgs?) { - try { - // Check for error in hwrResult first - val errorJson = args?.hwrResult - if (!errorJson.isNullOrBlank()) { - log.e("OnyxHWR error: ${errorJson.take(300)}") - cont.resume("") - return + val result = withTimeoutOrNull(10_000) { + suspendCancellableCoroutine { cont -> + svc.batchRecognize(pfd, object : HWROutputCallback.Stub() { + override fun read(args: HWROutputArgs?) { + try { + // Check for error in hwrResult first + val errorJson = args?.hwrResult + if (!errorJson.isNullOrBlank()) { + log.e("OnyxHWR error: ${errorJson.take(300)}") + cont.resume("") + return + } + + // Success: result is in PFD as JSON + val resultPfd = args?.pfd + if (resultPfd == null) { + log.w("OnyxHWR returned no PFD and no hwrResult") + cont.resume("") + return + } + + val json = readPfdAsString(resultPfd) + resultPfd.close() + val text = parseHwrResult(json) + log.i("OnyxHWR recognized ${text.length} chars") + cont.resume(text) + } catch (e: Exception) { + log.e("Error parsing OnyxHWR result: ${e.message}") + cont.resumeWithException(e) } - - // Success: result is in PFD as JSON - val resultPfd = args?.pfd - if (resultPfd == null) { - log.w("OnyxHWR returned no PFD and no hwrResult") - cont.resume("") - return - } - - val json = readPfdAsString(resultPfd) - resultPfd.close() - val text = parseHwrResult(json) - log.i("OnyxHWR recognized ${text.length} chars") - cont.resume(text) - } catch (e: Exception) { - log.e("Error parsing OnyxHWR result: ${e.message}") - cont.resumeWithException(e) } - } - }) + }) + } + } + if (result == null) { + log.e("OnyxHWR timed out after 10s") } + result } finally { pfd.close() } From df58035d3c9d57b86bf017d69209e8e0e1995daa Mon Sep 17 00:00:00 2001 From: Joshua Pham Date: Mon, 9 Mar 2026 20:20:43 -0400 Subject: [PATCH 18/23] Rewrite README for Obsidian-specific fork Replace upstream Ethran README with one that reflects what this fork actually is: a handwriting capture surface for Obsidian on Boox tablets. Covers the capture workflow, HWR pipeline, vault sync, editor changes, and the UX principles visible across the commit history. Co-Authored-By: Claude Opus 4.6 --- readme.md | 320 ++++++++++++++---------------------------------------- 1 file changed, 81 insertions(+), 239 deletions(-) diff --git a/readme.md b/readme.md index e909552d..78dd18f0 100644 --- a/readme.md +++ b/readme.md @@ -1,289 +1,131 @@ - +# Notable for Obsidian -
+A handwriting capture surface for [Obsidian](https://obsidian.md), built for [Onyx Boox](https://www.boox.com/) e-ink tablets. -[![License][license-shield]][license-url] -[![Total Downloads][downloads-shield]][downloads-url] -[![Discord][discord-shield]][discord-url] - -![Notable App][logo] - -# Notable (Fork) - -A maintained and customized fork of the archived [olup/notable](https://github.com/olup/notable) project. - -[![🐛 Report Bug][bug-shield]][bug-url] -[![Download Latest][download-shield]][download-url] -[![💡 Request Feature][feature-shield]][feature-url] - - - Sponsor on GitHub - - - - Support me on Ko-fi - - -
+Fork of [Ethran/notable](https://github.com/Ethran/notable). The upstream project is a general-purpose note-taking app for Boox devices. This fork strips it down to a single purpose: **capture handwritten thoughts and sync them into your Obsidian vault as markdown**. --- -
- Table of Contents - -- [About This Fork](#about-this-fork) -- [Features](#features) -- [Download](#download) -- [Gestures](#gestures) -- [System Requirements and Permissions](#system-requirements-and-permissions) -- [Export and Import](#export-and-import) -- [Roadmap](#roadmap) -- [Troubleshooting and FAQ](#troubleshooting-and-faq) -- [Bug Reporting](#bug-reporting) -- [Screenshots](#screenshots) -- [Working with LaTeX](#working-with-latex) -- [App Distribution](#app-distribution) -- [For Developers & Contributing](#for-developers--contributing) - -
+## Why this exists ---- +Obsidian is great for organizing knowledge, but it assumes a keyboard. If you think with a pen — sketching, scrawling, drawing connections — there's a gap between the page and the graph. The Boox tablet is the best e-ink hardware for writing, and Notable is the best open-source app for that hardware. This fork bridges the two. -## About This Fork -This project began as a fork of the original Notable app and has since evolved into a continuation of it. The architecture is largely the same, but many of the functions have been rewritten and expanded with a focus on practical, everyday use. Development is active when possible, guided by the principle that the app must be fast and dependable — performance comes first, and the basics need to feel right before new features are introduced. Waiting for things to load is seen as unacceptable, so responsiveness is a core priority. +The core idea is **atomic capture**: each page is one thought. You write it, tag it with tags from your vault, and hit save. Handwriting recognition converts your strokes to text. The result lands in your Obsidian inbox as a markdown file with frontmatter, ready to be processed in your normal workflow. -Future plans include exploring how AI can enhance the app, with a focus on solutions that run directly on the device. A long-term goal is local handwriting conversion to LaTeX or plain text, making advanced features available without relying on external services. +The tablet becomes a dedicated input device for your second brain. No file management, no notebooks, no folders. Just capture and sync. --- -## Features -* ⚡ **Fast page turns with caching:** smooth, swift page transitions, including quick navigation to the next and previous pages. -* ↕️ **Infinite vertical scroll:** a virtually endless canvas for notes with smooth vertical scrolling. -* 📝 **Quick Pages:** instantly create a new page. -* 📒 **Notebooks:** group related notes and switch easily between notebooks. -* 📁 **Folders:** organize notes with folders. -* 🤏 **Editor mode gestures:** [intuitive gesture controls](#gestures) to enhance editing. -* 🌅 **Images:** add, move, scale, and remove images. -* ➤ **Selection export:** export or share selected handwriting as PNG. -* ✏️ **Scribble to erase:** erase content by scribbling over it (disabled by default) — contributed by [@niknal357](https://github.com/niknal357). -* 🔄 **Auto-refresh on background change:** useful when using a tablet as a second display — see [Working with LaTeX](#working-with-latex). - ---- +## What changed from upstream -## Download -**Download the latest stable version of the [Notable app here.](https://github.com/Ethran/notable/releases/latest)** +Everything here serves the Obsidian capture loop. Changes fall into three categories: -Alternatively, get the latest build from the main branch via the ["next" release](https://github.com/Ethran/notable/releases/next). +### Capture workflow +- **Atomic capture model** — every new page is a capture. No notebooks, folders, or file organization on the device. The library is a flat grid of captures. +- **Tag UI** — collapsible toolbar at the top with tag pills pulled from your Obsidian vault (ranked by frequency and recency), text search with autocomplete. Tags flow into the markdown frontmatter on sync. +- **Annotation boxes** — draw a box over handwritten text to mark it as a `[[wiki link]]` or `#tag`. These render as bracket/hash overlays on the canvas and get recognized inline during sync. +- **Save & exit** — one button. Returns to the library instantly; sync happens in background. -Open the **Assets** section of the release and select the `.apk` file. +### Handwriting recognition +- **Onyx HWR (MyScript)** — uses the Boox firmware's built-in MyScript engine via AIDL IPC. Replaced Google ML Kit, which had poor accuracy. MyScript is significantly better, especially for short phrases and mixed content. +- **Annotation-aware recognition** — strokes inside `[[]]` boxes become wiki links, strokes inside `#` boxes become tags. Recognition diffs full-page vs. non-annotated strokes for better accuracy (recognizing isolated short words like "pkm" in a box is harder than letting full-page context disambiguate). +- **Line segmentation** — multi-line captures are clustered by vertical position and recognized line-by-line with sequential context feeding forward. -
❓ Where can I see alternative/older releases?
-You can go to the original olup Releases and download alternative versions of the Notable app. -
+### Obsidian sync +- **Markdown output** — captures sync as `.md` files to a configurable folder in your vault with YAML frontmatter (`created`, `tags`). +- **Vault tag scanning** — parses tags from existing markdown files in your inbox folder so you can reuse your vocabulary. +- **Background sync** — recognition and file writing happen in a coroutine after you've already navigated away. A "Syncing..." overlay shows on pages still processing. -
❓ What is a 'next' release?
-The "next" release is a pre-release and may contain features implemented but not yet released as part of a stable version — and sometimes experiments that may not make it into a release. -
+### Editor changes +- **Left-edge sidebar** — all tools moved from a bottom toolbar to a vertical sidebar on the left. Pen picker and eraser flyouts appear to the right. Keeps the writing area unobstructed. +- **Jetpack Ink API** — replaced ~200 lines of custom stroke rendering with `androidx.ink`'s `CanvasStrokeRenderer`. +- **Fountain pen default** — with a sqrt pressure curve so light strokes are visible and heavy strokes feel natural on e-ink. --- -## Gestures -Notable features intuitive gesture controls within Editor mode to optimize the editing experience: - -#### ☝️ 1 Finger -* **Swipe up or down:** scroll the page. -* **Swipe left or right:** change to the previous/next page (only available in notebooks). -* **Double tap:** undo. -* **Hold and drag:** select text and images. +## UX principles -#### ✌️ 2 Fingers -* **Swipe left or right:** show or hide the toolbar. -* **Single tap:** switch between writing and eraser modes. -* **Pinch:** zoom in and out. -* **Hold and drag:** move the canvas. +These aren't stated goals — they're patterns visible in every commit: -#### 🔲 Selection -* **Drag:** move the selection. -* **Double tap:** copy the selected writing. +- **Speed is non-negotiable.** Sync is background. Navigation is instant. Tag suggestions are cached. Nothing blocks the pen. +- **Minimal chrome.** If it's not capture, tagging, or saving, it's removed. No folders, no notebooks, no import/export UI. The library is a grid. The editor is a page. +- **Pen-first interaction.** Annotations are drawn, not typed. The sidebar stays out of the way. The default pen feels good at first touch. +- **Obsidian is the system of record.** The tablet captures; Obsidian organizes. No duplicate taxonomy, no competing folder structures. Tags come from the vault and go back to the vault. --- -## System Requirements and Permissions -The app targets Onyx BOOX devices and requires Android 10 (SDK 29) or higher. Limited support for Android 9 (SDK 28) may be possible if [issue #93](https://github.com/Ethran/notable/issues/93) is resolved. Handwriting functionality is currently not available on non-Onyx devices. Enabling handwriting on other devices may be possible in the future but is not supported at the moment. - -Storage access is required to manage notes, assets, and to observe PDF backgrounds, which need “all files access”. The database is stored at `Documents/notabledb` to simplify backups and reduce the risk of accidental deletion. Exports are written to `Documents/notable`. - ---- +## Setup -## Export and Import +### Prerequisites +- An Onyx Boox tablet (pen input requires Onyx hardware) +- An Obsidian vault accessible on the device (e.g. via Syncthing, Dropsync, or USB) -The app supports the following formats: +### Install +Build from source (see [CLAUDE.md](./CLAUDE.md) for full build instructions): -- **PDF** — export and import supported. You can also link a page to an external PDF so that changes on your computer are reflected live on the tablet (see [Working with LaTeX](#working-with-latex)). -- **PNG** — export supported for handwriting selections, individual pages, and entire books. -- **JPEG** — export supported for individual pages. -- **XOPP** — export and import partially supported. Only stroke and image data are preserved; tool information for strokes may be lost when files are opened and saved with [Xournal++](https://xournalpp.github.io/). Backgrounds are not exported. +```bash +./gradlew assembleDebug +adb install -r app/build/outputs/apk/debug/app-debug.apk +``` +### Configure +1. Open Settings in the app +2. Set your **vault path** to the Obsidian vault directory on the device +3. Set the **inbox folder** where captures should land (e.g. `inbox/`) +4. Start capturing --- -## Roadmap - -### Near-term -- Better selection tools: - - Stroke editing (color, size, etc.) - - Rotate and flip selection - - Auto‑scroll when dragging a selection near screen edges - - Easier selection movement, including dragging while scrolling -- PDF improvements: - - Migration to a dedicated PDF library to replace the default Android renderer - - Allow saving annotations back to the original PDF - - Improved rendering and stability across devices - -### Planned -- PDF annotation enhancements: - - Display annotations from other programs - - Additional quality‑of‑life tools for annotating imported PDFs - -### Long-term -- Bookmarks, tags, and internal links — see [issue #52](https://github.com/Ethran/notable/issues/52), including link export to PDF. -- Figure and text recognition — see [issue #44](https://github.com/Ethran/notable/issues/44): - - Searchable notes - - Automatic creation of tag descriptions - - Shape recognition - - Handwriting to Latex +## How it works ---- - -## Troubleshooting and FAQ -**What are “NeoTools,” and why are some disabled?** -NeoTools are components of the Onyx E-Ink toolset, made available through Onyx’s libraries. However, certain tools are unstable and can cause crashes, so they are disabled by default to ensure better app stability. Examples include: +1. Tap **New Capture** in the library +2. Write with the pen. Tag by selecting existing vault tags from the toolbar, or typing new ones. +3. Optionally draw annotation boxes: tap `[[` or `#` in the sidebar, then draw a rectangle over text to mark it as a wiki link or tag +4. Tap **Save & Exit** +5. The app navigates back immediately. In the background, strokes are recognized, annotations are resolved, and a markdown file is written to your vault inbox. -* `com.onyx.android.sdk.pen.NeoCharcoalPenV2` -* `com.onyx.android.sdk.pen.NeoMarkerPen` -* `com.onyx.android.sdk.pen.NeoBrushPen` +The resulting file looks like: +```markdown --- - -## Bug Reporting - -If you encounter unexpected behavior, please include an app log with your report. To do this: -1. Navigate to the page where the issue occurs. -2. Reproduce the problem. -3. Open the page menu. -4. Select **“Bug Report”** and either copy the log or submit it directly. - -This will open a new GitHub issue in your browser with useful device information attached, which greatly helps in diagnosing and resolving the problem. - -Bug reporting with logs is currently supported only in notebooks/pages. Issues outside of writing are unlikely to require this level of detail. - +created: "[[2025-01-15]]" +tags: + - meeting-notes + - project-alpha --- - -## Screenshots - -
- Writing on a page - Notebook overview - Gestures and selection - Image handling - Toolbar and tools - Page management - PDF viewing - Customization - Settings -
+discussed the [[API redesign]] with the team +need to revisit #authentication flow before launch +``` --- -## Working with LaTeX - -The app can be used as a **primitive second monitor** for LaTeX editing — previewing compiled PDFs -in real time on your tablet. - -### Steps: - -- Connect your device to your computer via USB (MTP). -- Set up automatic copying of the compiled PDF to the tablet: -
- Example using a custom latexmkrc: - - ```perl - $pdf_mode = 1; - $out_dir = 'build'; +## Upstream features retained - sub postprocess { - system("cp build/main.pdf '/run/user/1000/gvfs/mtp:host=DEVICE/Internal shared storage/Documents/Filename.pdf'"); - } +This fork keeps the core Notable functionality that matters for capture: - END { - postprocess(); - } - ``` - - It was also tested with `adb push`, instead of `cp`. +- Low-latency Onyx Pen SDK input (`TouchHelper` + `RawInputCallback`) +- E-ink optimized rendering (no animations, batched refreshes, `EpdController` refresh modes) +- Undo/redo history +- Selection, copy, and lasso tools +- Multiple pen types (fountain, ballpoint, marker, pencil) and sizes +- Eraser (stroke and area modes, scribble-to-erase) +- Image insertion +- Zoom and pan -
-- Compile, and test if it copies the file to the tablet. -- Import your compiled PDF document into Notable, and choose to observe the PDF file. - -> After each recompilation, Notable will detect the updated PDF and automatically refresh the view. - ---- - -## App Distribution -Notable is not distributed on Google Play or F-Droid. Official builds are provided exclusively via [GitHub Releases](https://github.com/Ethran/notable/releases). +Features removed from the UI: notebooks, folders, import/export, background templates, PDF annotation. --- -## For Developers & Contributing +## Credits -- Project file layout: see [docs/file-structure.md](./docs/file-structure.md) -- Data model and stroke encoding: see [docs/database-structure.md](./docs/database-structure.md) -- Additional documentation will be added as needed - Note: These documents were AI-generated and lightly verified; refer to the code for the authoritative source. - -### Development Notes - -- Edit the `DEBUG_STORE_FILE` in `/app/gradle.properties` to point to your local keystore file. This is typically located in the `.android` directory. -- To debug on a BOOX device, enable developer mode. You can follow [this guide](https://imgur.com/a/i1kb2UQ). - -Feel free to open issues or submit pull requests. I appreciate your help! +- [Ethran/notable](https://github.com/Ethran/notable) — the actively maintained fork this builds on +- [olup/notable](https://github.com/olup/notable) — the original project +- Onyx Pen SDK and MyScript HWR engine (via Boox firmware) +- [Jetpack Ink](https://developer.android.com/jetpack/androidx/releases/ink) for stroke rendering --- - -[logo]: https://github.com/Ethran/notable/blob/main/app/src/main/res/mipmap-xxxhdpi/ic_launcher.png?raw=true "Notable Logo" -[contributors-shield]: https://img.shields.io/github/contributors/Ethran/notable.svg?style=for-the-badge -[contributors-url]: https://github.com/Ethran/notable/graphs/contributors -[forks-shield]: https://img.shields.io/github/forks/Ethran/notable.svg?style=for-the-badge -[forks-url]: https://github.com/Ethran/notable/network/members -[stars-shield]: https://img.shields.io/github/stars/Ethran/notable.svg?style=for-the-badge -[stars-url]: https://github.com/Ethran/notable/stargazers -[issues-shield]: https://img.shields.io/github/issues/Ethran/notable.svg?style=for-the-badge -[issues-url]: https://github.com/Ethran/notable/issues -[license-shield]: https://img.shields.io/github/license/Ethran/notable.svg?style=for-the-badge - -[license-url]: https://github.com/Ethran/notable/blob/main/LICENSE -[download-shield]: https://img.shields.io/github/v/release/Ethran/notable?style=for-the-badge&label=⬇️%20Download -[download-url]: https://github.com/Ethran/notable/releases/latest -[downloads-shield]: https://img.shields.io/github/downloads/Ethran/notable/total?style=for-the-badge&color=47c219&logo=cloud-download -[downloads-url]: https://github.com/Ethran/notable/releases/latest - -[discord-shield]: https://img.shields.io/badge/Discord-Join%20Chat-7289DA?style=for-the-badge&logo=discord -[discord-url]: https://discord.gg/rvNHgaDmN2 -[kofi-shield]: https://img.shields.io/badge/Buy%20Me%20a%20Coffee-ko--fi-ff5f5f?style=for-the-badge&logo=ko-fi&logoColor=white -[kofi-url]: https://ko-fi.com/rethran - -[sponsor-shield]: https://img.shields.io/badge/Sponsor-GitHub-%23ea4aaa?style=for-the-badge&logo=githubsponsors&logoColor=white -[sponsor-url]: https://github.com/sponsors/rethran - -[docs-url]: https://github.com/Ethran/notable -[bug-url]: https://github.com/Ethran/notable/issues/new?template=bug_report.md -[feature-url]: https://github.com/Ethran/notable/issues/new?labels=enhancement&template=feature-request---.md -[bug-shield]: https://img.shields.io/badge/🐛%20Report%20Bug-red?style=for-the-badge -[feature-shield]: https://img.shields.io/badge/💡%20Request%20Feature-blueviolet?style=for-the-badge +## License + +[MIT](./LICENSE) From 5d6956d01e63743cc136b6e0c67ee01c55643bc6 Mon Sep 17 00:00:00 2001 From: Joshua Pham Date: Tue, 10 Mar 2026 00:05:54 -0400 Subject: [PATCH 19/23] Add decorative header and knowledge curation shoutouts to README Co-Authored-By: Claude Opus 4.6 --- readme.md | 29 ++++++++++++++++++++++++++++- 1 file changed, 28 insertions(+), 1 deletion(-) diff --git a/readme.md b/readme.md index 78dd18f0..cbe13b72 100644 --- a/readme.md +++ b/readme.md @@ -1,6 +1,19 @@ +
+ # Notable for Obsidian -A handwriting capture surface for [Obsidian](https://obsidian.md), built for [Onyx Boox](https://www.boox.com/) e-ink tablets. +**Handwriting → Markdown → Knowledge** + +A capture surface for [Obsidian](https://obsidian.md) on [Onyx Boox](https://www.boox.com/) e-ink tablets. +Write with a pen. Tag with your vault's vocabulary. Sync as markdown. + +[![Kotlin](https://img.shields.io/badge/Kotlin-2.3-7F52FF?logo=kotlin&logoColor=white)](https://kotlinlang.org) +[![Jetpack Compose](https://img.shields.io/badge/Jetpack_Compose-Material-4285F4?logo=jetpackcompose&logoColor=white)](https://developer.android.com/jetpack/compose) +[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](./LICENSE) + +
+ +--- Fork of [Ethran/notable](https://github.com/Ethran/notable). The upstream project is a general-purpose note-taking app for Boox devices. This fork strips it down to a single purpose: **capture handwritten thoughts and sync them into your Obsidian vault as markdown**. @@ -117,6 +130,20 @@ Features removed from the UI: notebooks, folders, import/export, background temp --- +## If you're into knowledge curation + +This project is about getting handwritten thoughts into Obsidian. If you're looking for tools that help you *rediscover* those thoughts once they're there: + +### [Enzyme](https://enzyme.garden) · [GitHub](https://github.com/jshph/enzyme) + +An AI layer for Obsidian that surfaces connections between your notes — the ones you forgot you made. Semantic search over your vault, concept-level linking, and pattern discovery across everything you've written. < 40 MB bundle, 10x faster initialization than QMD. Lightweight enough to run alongside your vault without friction. + +### [Aside](https://github.com/jshph/aside) + +Local-first meeting memos with timestamp-synced transcription. Record a conversation, get a structured note aligned to your audio timeline. Everything stays on your machine. + +--- + ## Credits - [Ethran/notable](https://github.com/Ethran/notable) — the actively maintained fork this builds on From b3ffefdf39c360d6171a877a4de218f3023f6c57 Mon Sep 17 00:00:00 2001 From: Joshua Pham Date: Tue, 10 Mar 2026 00:18:22 -0400 Subject: [PATCH 20/23] Add annotation undo/redo, fix glyph clipping, improve HWR sync fallback, simplify CI workflows - Annotation create/delete now tracked in undo/redo history - Expand redraw area to include bracket/hash glyph visual bounds, fixing partial clipping - InboxSyncEngine: fall back to per-annotation HWR when diff-based approach has count mismatch - Simplify CI workflows: update action versions, use GITHUB_TOKEN, trigger release on tag push Co-Authored-By: Claude Opus 4.6 --- .github/workflows/preview.yml | 77 ++++--------- .github/workflows/release.yml | 102 ++++++---------- .../com/ethran/notable/editor/PageView.kt | 19 ++- .../notable/editor/canvas/OnyxInputHandler.kt | 8 +- .../notable/editor/drawing/pageDrawing.kt | 42 ++++++- .../ethran/notable/editor/state/history.kt | 19 +++ .../com/ethran/notable/editor/utils/eraser.kt | 14 --- .../ethran/notable/editor/utils/operations.kt | 8 ++ .../com/ethran/notable/io/InboxSyncEngine.kt | 109 +++++++++++++----- 9 files changed, 225 insertions(+), 173 deletions(-) diff --git a/.github/workflows/preview.yml b/.github/workflows/preview.yml index 9fc952ed..6c466739 100644 --- a/.github/workflows/preview.yml +++ b/.github/workflows/preview.yml @@ -1,78 +1,43 @@ -name: Build and Release Preview +name: Build Preview + on: push: - branches: ["dev"] + branches: ['main'] paths: - 'app/**' workflow_dispatch: -env: - MAVEN_OPTS: >- - -Dmaven.wagon.httpconnectionManager.ttlSeconds=120 - jobs: build: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v3 - - name: set up JDK 18 - uses: actions/setup-java@v3 - with: - java-version: "18" - distribution: "temurin" - # cache: gradle + - uses: actions/checkout@v4 - - uses: gradle/gradle-build-action@v2 + - name: Set up JDK 17 + uses: actions/setup-java@v4 with: - gradle-version: 8.5 + java-version: '17' + distribution: 'temurin' - - name: Decode Keystore - id: decode_keystore - uses: timheuer/base64-to-file@v1.1 - with: - fileDir: "./secrets" - fileName: "my.keystore" - encodedString: ${{ secrets.KEYSTORE_FILE }} + - name: Setup Gradle + uses: gradle/actions/setup-gradle@v4 - - name: Remove `google-services.json` - run: rm ${{ github.workspace }}/app/google-services.json + - name: Build debug APK + run: ./gradlew -PIS_NEXT=true assembleDebug - - name: Decode and Replace `google-services.json` - env: - FIREBASE_CONFIG: ${{ secrets.FIREBASE_CONFIG }} + - name: Rename APK run: | - echo $FIREBASE_CONFIG | base64 --decode > ${{ github.workspace }}/app/google-services.json - wc -l ${{ github.workspace }}/app/google-services.json + mv app/build/outputs/apk/debug/app-debug.apk \ + app/build/outputs/apk/debug/notable-next.apk - - name: Execute Gradle build - run: | - export STORE_FILE="../${{ steps.decode_keystore.outputs.filePath }}" - export STORE_PASSWORD="${{ secrets.KEYSTORE_PASSWORD }}" - export KEY_ALIAS="${{ secrets.KEY_ALIAS }}" - export KEY_PASSWORD="${{ secrets.KEY_PASSWORD }}" - export SHIPBOOK_APP_ID="${{ secrets.SHIPBOOK_APP_ID }}" - export SHIPBOOK_APP_KEY="${{ secrets.SHIPBOOK_APP_KEY }}" - - ./gradlew \ - -PIS_NEXT=true \ - assembleDebug - - # - name: Cache Gradle packages - # uses: actions/cache@v1 - # with: - # path: ~/.gradle/caches - # key: ${{ runner.os }}-gradle-${{ hashFiles('**/*.gradle') }} - # restore-keys: ${{ runner.os }}-gradle - - - run: mv ${{ github.workspace }}/app/build/outputs/apk/debug/app-debug.apk ${{ github.workspace }}/app/build/outputs/apk/debug/notable-next.apk - - - name: Release - uses: softprops/action-gh-release@v1 + - name: Update preview release + uses: softprops/action-gh-release@v2 with: - files: ${{ github.workspace }}/app/build/outputs/apk/debug/notable-next.apk + files: app/build/outputs/apk/debug/notable-next.apk tag_name: next name: next prerelease: true - body: "Preview version built from branch: ${{ github.ref_name }}" - token: ${{ secrets.TOKEN }} + body: "Preview build from ${{ github.sha }}" + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index e0b5f539..28e9e216 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -1,91 +1,59 @@ -name: Build and Release Version +name: Build and Release on: + push: + tags: + - 'v*' workflow_dispatch: -env: - MAVEN_OPTS: >- - -Dmaven.wagon.httpconnectionManager.ttlSeconds=120 - jobs: build: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v3 - - name: set up JDK 18 - uses: actions/setup-java@v3 - with: - java-version: "18" - distribution: "temurin" - # cache: gradle + - uses: actions/checkout@v4 - - uses: gradle/gradle-build-action@v2 + - name: Set up JDK 17 + uses: actions/setup-java@v4 with: - gradle-version: 8.5 + java-version: '17' + distribution: 'temurin' + + - name: Setup Gradle + uses: gradle/actions/setup-gradle@v4 - name: Decode Keystore id: decode_keystore - uses: timheuer/base64-to-file@v1.1 + uses: timheuer/base64-to-file@v1.2 with: - fileDir: "./secrets" - fileName: "my.keystore" + fileDir: './secrets' + fileName: 'notable-release.jks' encodedString: ${{ secrets.KEYSTORE_FILE }} - - name: Decode and Replace `google-services.json` - env: - FIREBASE_CONFIG: ${{ secrets.FIREBASE_CONFIG }} - run: | - echo $FIREBASE_CONFIG | base64 --decode > ${{ github.workspace }}/app/google-services.json - - - name: Execute Gradle build - env: - STORE_FILE: ${{ github.workspace }}/secrets/my.keystore - STORE_PASSWORD: ${{ secrets.KEYSTORE_PASSWORD }} - KEY_ALIAS: ${{ secrets.RELEASE_KEY_ALIAS }} - KEY_PASSWORD: ${{ secrets.RELEASE_KEY_PASSWORD }} - SHIPBOOK_APP_ID: ${{ secrets.SHIPBOOK_APP_ID }} - SHIPBOOK_APP_KEY: ${{ secrets.SHIPBOOK_APP_KEY }} - run: | - ./gradlew assembleRelease \ - -PSTORE_FILE="$STORE_FILE" \ - -PSTORE_PASSWORD="$STORE_PASSWORD" \ - -PKEY_ALIAS="$KEY_ALIAS" \ - -PKEY_PASSWORD="$KEY_PASSWORD" - - - - # - name: Cache Gradle packages - # uses: actions/cache@v1 - # with: - # path: ~/.gradle/caches - # key: ${{ runner.os }}-gradle-${{ hashFiles('**/*.gradle') }} - # restore-keys: ${{ runner.os }}-gradle - - - name: Verify APK Files - run: find ${{ github.workspace }}/app/build/outputs/apk/ -name "*.apk" - - - name: Retrieve Version + - name: Build signed release APK env: - STORE_FILE: ${{ github.workspace }}/secrets/my.keystore - STORE_PASSWORD: ${{ secrets.KEYSTORE_PASSWORD }} - KEY_ALIAS: ${{ secrets.RELEASE_KEY_ALIAS }} - KEY_PASSWORD: ${{ secrets.RELEASE_KEY_PASSWORD }} - SHIPBOOK_APP_ID: ${{ secrets.SHIPBOOK_APP_ID }} - SHIPBOOK_APP_KEY: ${{ secrets.SHIPBOOK_APP_KEY }} + STORE_FILE: ${{ github.workspace }}/secrets/notable-release.jks + STORE_PASSWORD: ${{ secrets.STORE_PASSWORD }} + KEY_ALIAS: ${{ secrets.KEY_ALIAS }} + KEY_PASSWORD: ${{ secrets.KEY_PASSWORD }} + run: ./gradlew assembleRelease + + - name: Get version name + id: version run: | - ./gradlew -q printVersionName VERSION_NAME=$(./gradlew -q printVersionName) - echo "VERSION_NAME=$VERSION_NAME" >> $GITHUB_ENV - id: android_version - + echo "VERSION_NAME=$VERSION_NAME" >> "$GITHUB_OUTPUT" - name: Rename APK - run: mv ${{ github.workspace }}/app/build/outputs/apk/release/app-release.apk ${{ github.workspace }}/app/build/outputs/apk/release/notable-${{ env.VERSION_NAME }}.apk + run: | + mv app/build/outputs/apk/release/app-release.apk \ + app/build/outputs/apk/release/notable-${{ steps.version.outputs.VERSION_NAME }}.apk - - name: Release - uses: softprops/action-gh-release@v1 + - name: Create GitHub Release + uses: softprops/action-gh-release@v2 with: - files: ${{ github.workspace }}/app/build/outputs/apk/release/notable-${{ env.VERSION_NAME }}.apk - tag_name: v${{env.VERSION_NAME}} - token: ${{ secrets.TOKEN }} + files: app/build/outputs/apk/release/notable-${{ steps.version.outputs.VERSION_NAME }}.apk + tag_name: ${{ github.ref_name }} + generate_release_notes: true + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} diff --git a/app/src/main/java/com/ethran/notable/editor/PageView.kt b/app/src/main/java/com/ethran/notable/editor/PageView.kt index 121d8d03..5088f7a8 100644 --- a/app/src/main/java/com/ethran/notable/editor/PageView.kt +++ b/app/src/main/java/com/ethran/notable/editor/PageView.kt @@ -27,6 +27,7 @@ import com.ethran.notable.data.db.Stroke import com.ethran.notable.data.db.getBackgroundType import com.ethran.notable.data.model.BackgroundType import com.ethran.notable.editor.canvas.CanvasEventBus +import com.ethran.notable.editor.drawing.annotationVisualBounds import com.ethran.notable.editor.drawing.drawBg import com.ethran.notable.editor.drawing.drawOnCanvasFromPage import com.ethran.notable.editor.utils.div @@ -491,7 +492,23 @@ class PageView( ignoredImageIds: List = listOf(), canvas: Canvas? = null ) { - val areaInScreen = toScreenCoordinates(pageArea) + // Expand the redraw area to include the full visual extent of any + // annotations whose glyphs (brackets, hash) overlap with pageArea. + // Without this, partial redraws clip away glyph parts that extend + // beyond the annotation's data bounds. + var expanded = pageArea + annotations.forEach { annotation -> + val visualBounds = annotationVisualBounds(annotation) + if (Rect.intersects(visualBounds, pageArea)) { + expanded = Rect( + min(expanded.left, visualBounds.left), + min(expanded.top, visualBounds.top), + max(expanded.right, visualBounds.right), + max(expanded.bottom, visualBounds.bottom) + ) + } + } + val areaInScreen = toScreenCoordinates(expanded) drawAreaScreenCoordinates(areaInScreen, ignoredStrokeIds, ignoredImageIds, canvas) } diff --git a/app/src/main/java/com/ethran/notable/editor/canvas/OnyxInputHandler.kt b/app/src/main/java/com/ethran/notable/editor/canvas/OnyxInputHandler.kt index 55acbc43..6ad90991 100644 --- a/app/src/main/java/com/ethran/notable/editor/canvas/OnyxInputHandler.kt +++ b/app/src/main/java/com/ethran/notable/editor/canvas/OnyxInputHandler.kt @@ -14,6 +14,7 @@ import com.ethran.notable.editor.ui.INBOX_TOOLBAR_EXPANDED_HEIGHT import com.ethran.notable.editor.state.EditorState import com.ethran.notable.editor.state.History import com.ethran.notable.editor.state.Mode +import com.ethran.notable.editor.state.Operation import com.ethran.notable.editor.utils.DeviceCompat import com.ethran.notable.editor.utils.Eraser import com.ethran.notable.editor.utils.Pen @@ -330,11 +331,16 @@ class OnyxInputHandler( if (erasedByScribbleDirtyRect.isNullOrEmpty()) { if (annotMode != AnnotationMode.None) { log.d("Creating annotation...") - handleAnnotation( + val annotationId = handleAnnotation( drawCanvas.page, annotMode, scaledPoints ) + if (annotationId != null) { + history.addOperationsToHistory( + listOf(Operation.DeleteAnnotation(listOf(annotationId))) + ) + } // Reset to one-shot: annotation mode turns off after one box drawCanvas.getActualState().annotationMode = AnnotationMode.None // Redraw canvas, then do a full GC refresh to clear diff --git a/app/src/main/java/com/ethran/notable/editor/drawing/pageDrawing.kt b/app/src/main/java/com/ethran/notable/editor/drawing/pageDrawing.kt index 38e217de..3e203fa0 100644 --- a/app/src/main/java/com/ethran/notable/editor/drawing/pageDrawing.kt +++ b/app/src/main/java/com/ethran/notable/editor/drawing/pageDrawing.kt @@ -57,6 +57,41 @@ private val annotationUnderlinePaint = Paint().apply { isAntiAlias = true } +/** + * Returns the visual bounding rect of an annotation in page coordinates, + * including the bracket/hash glyphs that extend beyond the annotation's data bounds. + */ +fun annotationVisualBounds(annotation: Annotation): Rect { + val boxHeight = annotation.height + val padding = boxHeight * 0.15f + + val expandLeft: Float + val expandRight: Float + + if (annotation.type == AnnotationType.WIKILINK.name) { + val bracketSize = boxHeight * 0.55f + wikiLinkBracketPaint.textSize = bracketSize + val bracketWidth = wikiLinkBracketPaint.measureText("[[") + val bracketGap = padding * 0.5f + expandLeft = bracketWidth + bracketGap + expandRight = bracketWidth + bracketGap + } else { + val hashSize = boxHeight * 0.65f + tagHashPaint.textSize = hashSize + val hashWidth = tagHashPaint.measureText("#") + val hashGap = padding * 0.6f + expandLeft = hashWidth + hashGap + expandRight = padding + } + + return Rect( + (annotation.x - expandLeft).toInt(), + (annotation.y - padding).toInt(), + (annotation.x + annotation.width + expandRight).toInt(), + (annotation.y + annotation.height + padding).toInt() + ) +} + fun drawAnnotation(canvas: Canvas, annotation: Annotation, offset: Offset) { val rect = RectF( annotation.x + offset.x, @@ -278,11 +313,8 @@ fun drawOnCanvasFromPage( // Draw annotation overlays on top of strokes try { page.annotations.forEach { annotation -> - val annotRect = RectF( - annotation.x, annotation.y, - annotation.x + annotation.width, annotation.y + annotation.height - ) - if (!annotRect.toRect().intersect(pageArea)) return@forEach + val visualBounds = annotationVisualBounds(annotation) + if (!visualBounds.intersect(pageArea)) return@forEach drawAnnotation(this, annotation, -page.scroll) } } catch (e: Exception) { diff --git a/app/src/main/java/com/ethran/notable/editor/state/history.kt b/app/src/main/java/com/ethran/notable/editor/state/history.kt index 639f044e..6266eb28 100644 --- a/app/src/main/java/com/ethran/notable/editor/state/history.kt +++ b/app/src/main/java/com/ethran/notable/editor/state/history.kt @@ -1,10 +1,12 @@ package com.ethran.notable.editor.state import android.graphics.Rect +import com.ethran.notable.data.db.Annotation import com.ethran.notable.data.db.Image import com.ethran.notable.data.db.Stroke import com.ethran.notable.editor.canvas.CanvasEventBus import com.ethran.notable.editor.PageView +import com.ethran.notable.editor.utils.annotationBounds import com.ethran.notable.editor.utils.imageBoundsInt import com.ethran.notable.editor.utils.strokeBounds import com.ethran.notable.ui.SnackConf @@ -18,6 +20,8 @@ sealed class Operation { data class AddStroke(val strokes: List) : Operation() data class AddImage(val images: List) : Operation() data class DeleteImage(val imageIds: List) : Operation() + data class AddAnnotation(val annotations: List) : Operation() + data class DeleteAnnotation(val annotationIds: List) : Operation() } typealias OperationBlock = List @@ -104,6 +108,21 @@ class History(pageView: PageView) { pageModel.removeImages(operation.imageIds) return Operation.AddImage(images = images) to imageBoundsInt(images) } + + is Operation.AddAnnotation -> { + pageModel.addAnnotations(operation.annotations) + return Operation.DeleteAnnotation(annotationIds = operation.annotations.map { it.id }) to annotationBounds( + operation.annotations + ) + } + + is Operation.DeleteAnnotation -> { + val annotations = operation.annotationIds.mapNotNull { id -> + pageModel.annotations.find { it.id == id } + } + pageModel.removeAnnotations(operation.annotationIds) + return Operation.AddAnnotation(annotations = annotations) to annotationBounds(annotations) + } } } diff --git a/app/src/main/java/com/ethran/notable/editor/utils/eraser.kt b/app/src/main/java/com/ethran/notable/editor/utils/eraser.kt index 012b9f55..1ec451c1 100644 --- a/app/src/main/java/com/ethran/notable/editor/utils/eraser.kt +++ b/app/src/main/java/com/ethran/notable/editor/utils/eraser.kt @@ -214,20 +214,6 @@ private fun selectAnnotationsFromPath(annotations: List, eraserPath: } } -private fun annotationBounds(annotations: List): Rect { - var left = Float.MAX_VALUE - var top = Float.MAX_VALUE - var right = Float.MIN_VALUE - var bottom = Float.MIN_VALUE - for (a in annotations) { - if (a.x < left) left = a.x - if (a.y < top) top = a.y - if (a.x + a.width > right) right = a.x + a.width - if (a.y + a.height > bottom) bottom = a.y + a.height - } - return Rect(left.toInt(), top.toInt(), right.toInt(), bottom.toInt()) -} - // points is in page coordinates, returns effected area. fun cleanAllStrokes( page: PageView, history: History diff --git a/app/src/main/java/com/ethran/notable/editor/utils/operations.kt b/app/src/main/java/com/ethran/notable/editor/utils/operations.kt index e486e487..06fb25f3 100644 --- a/app/src/main/java/com/ethran/notable/editor/utils/operations.kt +++ b/app/src/main/java/com/ethran/notable/editor/utils/operations.kt @@ -8,8 +8,10 @@ import android.graphics.RectF import android.graphics.Region import androidx.compose.ui.geometry.Offset import androidx.core.graphics.toRegion +import com.ethran.notable.data.db.Annotation import com.ethran.notable.data.db.Image import com.ethran.notable.data.db.Stroke +import com.ethran.notable.editor.drawing.annotationVisualBounds import com.ethran.notable.data.db.StrokePoint import com.ethran.notable.data.model.SimplePointF import com.onyx.android.sdk.data.note.TouchPoint @@ -118,6 +120,12 @@ fun imageBoundsInt(images: List): Rect { return rect } +fun annotationBounds(annotations: List): Rect { + if (annotations.isEmpty()) return Rect() + val rect = annotationVisualBounds(annotations[0]) + annotations.drop(1).forEach { rect.union(annotationVisualBounds(it)) } + return rect +} fun pathToRegion(path: Path): Region { val bounds = RectF() diff --git a/app/src/main/java/com/ethran/notable/io/InboxSyncEngine.kt b/app/src/main/java/com/ethran/notable/io/InboxSyncEngine.kt index e8c7b564..cf6d6a5f 100644 --- a/app/src/main/java/com/ethran/notable/io/InboxSyncEngine.kt +++ b/app/src/main/java/com/ethran/notable/io/InboxSyncEngine.kt @@ -58,48 +58,74 @@ object InboxSyncEngine { log.i("Full recognized text: '${fullText.take(200)}'") // 2. Find annotation text by diffing full recognition vs non-annotation recognition. - // This is more reliable than recognizing annotation strokes alone (fewer strokes = - // less context = worse HWR accuracy). + // Falls back to per-annotation recognition if the diff produces a count mismatch + // (which happens when removing strokes changes HWR context enough to alter other words). if (serviceReady && annotations.isNotEmpty()) { + val sortedAnnotations = annotations.sortedWith(compareBy({ it.y }, { it.x })) + // Collect stroke IDs that fall inside any annotation box val annotationStrokeIds = mutableSetOf() - for (annotation in annotations) { + val annotationStrokeMap = mutableMapOf>() + for (annotation in sortedAnnotations) { val annotRect = RectF( annotation.x, annotation.y, annotation.x + annotation.width, annotation.y + annotation.height ) - findStrokesInRect(allStrokes, annotRect).forEach { annotationStrokeIds.add(it.id) } + val overlapping = findStrokesInRect(allStrokes, annotRect) + annotationStrokeMap[annotation.id] = overlapping + overlapping.forEach { annotationStrokeIds.add(it.id) } } - // Recognize non-annotation strokes to get "base text" + // Try diff-based approach first (better accuracy when it works) + var diffSucceeded = false val nonAnnotStrokes = allStrokes.filter { it.id !in annotationStrokeIds } - val baseText = if (nonAnnotStrokes.isNotEmpty()) { - postProcessRecognition(recognizeStrokesSafe(nonAnnotStrokes)) - } else "" - log.i("Base text (without annotations): '${baseText.take(200)}'") - - // Diff to find word segments in full text that are missing from base text. - // These are the annotation-contributed words, in reading order. - val annotationTexts = diffWords(baseText, fullText) - log.i("Diffed annotation texts: $annotationTexts") - - // Sort annotations by position (top-to-bottom, left-to-right) to match diff order - val sortedAnnotations = annotations.sortedWith(compareBy({ it.y }, { it.x })) + if (nonAnnotStrokes.isNotEmpty()) { + val baseText = postProcessRecognition(recognizeStrokesSafe(nonAnnotStrokes)) + log.i("Base text (without annotations): '${baseText.take(200)}'") + + val annotationTexts = diffWords(baseText, fullText) + log.i("Diffed annotation texts: $annotationTexts") + + if (annotationTexts.size == sortedAnnotations.size) { + diffSucceeded = true + for ((i, annotation) in sortedAnnotations.withIndex()) { + val annotText = annotationTexts[i] + if (annotText.isBlank()) continue + log.i("Annotation ${annotation.type} (diff): '$annotText'") + // Strip trailing punctuation so it stays outside the markup + val cleaned = annotText.trimEnd('.', ',', ';', ':', '!', '?') + val trailing = annotText.removePrefix(cleaned) + val wrapped = wrapAnnotationText(annotation, cleaned) + trailing + fullText = fullText.replaceFirst(annotText, wrapped) + } + } else { + log.w("Diff found ${annotationTexts.size} segments but have ${sortedAnnotations.size} annotations — falling back to per-annotation recognition") + } + } - if (annotationTexts.size != sortedAnnotations.size) { - log.w("Diff found ${annotationTexts.size} segments but have ${sortedAnnotations.size} annotations — skipping wrapping") - } else { - for ((i, annotation) in sortedAnnotations.withIndex()) { - val annotText = annotationTexts[i] - if (annotText.isBlank()) continue - log.i("Annotation ${annotation.type}: '$annotText'") - val wrapped = when (annotation.type) { - AnnotationType.WIKILINK.name -> "[[${annotText}]]" - AnnotationType.TAG.name -> "#${annotText.replace(" ", "-")}" - else -> annotText + // Fallback: recognize each annotation's strokes individually + if (!diffSucceeded) { + for (annotation in sortedAnnotations) { + val overlapping = annotationStrokeMap[annotation.id] ?: continue + if (overlapping.isEmpty()) continue + val rawAnnotText = recognizeStrokesSafe(overlapping).trim() + if (rawAnnotText.isBlank()) continue + log.i("Annotation ${annotation.type} (fallback raw): '$rawAnnotText'") + + // Per-annotation HWR can be noisy — find the best matching + // substring in fullText, trying the full text first, then + // individual words from longest to shortest + val matchText = findBestMatch(rawAnnotText, fullText) + if (matchText != null) { + log.i("Annotation ${annotation.type} (fallback matched): '$matchText'") + val cleaned = matchText.trimEnd('.', ',', ';', ':', '!', '?') + val trailing = matchText.removePrefix(cleaned) + val wrapped = wrapAnnotationText(annotation, cleaned) + trailing + fullText = fullText.replaceFirst(matchText, wrapped) + } else { + log.w("Annotation ${annotation.type}: could not find '$rawAnnotText' in full text") } - fullText = fullText.replaceFirst(annotText, wrapped) } } } @@ -115,6 +141,31 @@ object InboxSyncEngine { log.i("Inbox sync complete for page $pageId") } + /** + * Find the best matching substring of [annotText] within [fullText]. + * Tries the full recognized text first, then individual words (longest first). + * Returns the matching substring as it appears in fullText, or null. + */ + private fun findBestMatch(annotText: String, fullText: String): String? { + if (fullText.contains(annotText)) return annotText + + // Try individual words, longest first (longer words are more specific) + val words = annotText.split(Regex("\\s+")).filter { it.isNotBlank() } + val sorted = words.sortedByDescending { it.length } + for (word in sorted) { + if (word.length >= 2 && fullText.contains(word)) return word + } + return null + } + + private fun wrapAnnotationText(annotation: Annotation, text: String): String { + return when (annotation.type) { + AnnotationType.WIKILINK.name -> "[[${text}]]" + AnnotationType.TAG.name -> "#${text.replace(" ", "-")}" + else -> text + } + } + private suspend fun recognizeStrokesSafe(strokes: List): String { return try { OnyxHWREngine.recognizeStrokes(strokes, viewWidth = 1404f, viewHeight = 1872f) ?: "" From 51010bd37a8bd1b4b0f5a0c8b0000e0367ba2533 Mon Sep 17 00:00:00 2001 From: Joshua Pham Date: Tue, 10 Mar 2026 00:19:40 -0400 Subject: [PATCH 21/23] Release v0.1.12 Co-Authored-By: Claude Opus 4.6 --- app/build.gradle | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/app/build.gradle b/app/build.gradle index 75b90ae5..54da75f4 100644 --- a/app/build.gradle +++ b/app/build.gradle @@ -17,8 +17,8 @@ android { minSdk 29 targetSdk 35 - versionCode 33 - versionName '0.1.11' + versionCode 34 + versionName '0.1.12' if (project.hasProperty('IS_NEXT') && project.IS_NEXT.toBoolean()) { def timestamp = new Date().format('dd.MM.YYYY-HH:mm') versionName = "${versionName}-next-${timestamp}" From 8bdbb36b5d22f3f8896c13609ddb0d1cb6adc698 Mon Sep 17 00:00:00 2001 From: Joshua Pham Date: Tue, 10 Mar 2026 00:52:24 -0400 Subject: [PATCH 22/23] Fix printVersionName CI task polluted by signing config output Co-Authored-By: Claude Opus 4.6 --- app/build.gradle | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/app/build.gradle b/app/build.gradle index 54da75f4..02fc0712 100644 --- a/app/build.gradle +++ b/app/build.gradle @@ -50,7 +50,7 @@ android { v1SigningEnabled true v2SigningEnabled true } else { - println "Running locally, skipping release signing..." + logger.info "Running locally, skipping release signing..." } } release { @@ -63,7 +63,7 @@ android { v1SigningEnabled true v2SigningEnabled true } else { - println "Running locally, skipping release signing..." + logger.info "Running locally, skipping release signing..." } } @@ -227,5 +227,7 @@ dependencies { } tasks.register('printVersionName') { - println android.defaultConfig.versionName + doLast { + println android.defaultConfig.versionName + } } From 79c13ed76b445456f2c8add16d5c10a37cd6624e Mon Sep 17 00:00:00 2001 From: jdkruzr Date: Sun, 15 Mar 2026 12:37:05 -0500 Subject: [PATCH 23/23] Fix invisible strokes caused by Ink API rejecting duplicate point timestamps toInkStroke() was passing per-point dt deltas directly as elapsed_time instead of accumulating them. This caused the Jetpack Ink API to reject strokes with INVALID_ARGUMENT ("duplicate position and elapsed_time"), making them invisible despite being saved to the database. Particularly severe on the Palma 2 Pro where the slower SoC produces more duplicate sample points. Co-Authored-By: Claude Opus 4.6 --- .../notable/editor/drawing/InkBrushes.kt | 18 ++++++++++++++++-- 1 file changed, 16 insertions(+), 2 deletions(-) diff --git a/app/src/main/java/com/ethran/notable/editor/drawing/InkBrushes.kt b/app/src/main/java/com/ethran/notable/editor/drawing/InkBrushes.kt index d7a52890..d019860e 100644 --- a/app/src/main/java/com/ethran/notable/editor/drawing/InkBrushes.kt +++ b/app/src/main/java/com/ethran/notable/editor/drawing/InkBrushes.kt @@ -63,16 +63,30 @@ fun Stroke.toInkStroke(offset: Offset = Offset.Zero): InkStroke { return InkStroke(brushForStroke(this), batch.toImmutable()) } + // Track cumulative elapsed time (dt stores per-point deltas, Ink API wants cumulative) + var cumulativeMs = 0L + var prevX = Float.NaN + var prevY = Float.NaN + var prevElapsed = -1L + for (i in points.indices) { val pt = points[i] - // Use real dt if available, otherwise synthesize realistic timing - val elapsedMs = pt.dt?.toLong() ?: (i * SYNTHETIC_DT_MS) + // Accumulate dt into cumulative elapsed time + cumulativeMs += pt.dt?.toLong() ?: SYNTHETIC_DT_MS + val elapsedMs = cumulativeMs val pressure = pt.pressure?.let { it / maxPressure.toFloat() } ?: 0.5f // tiltRadians must be in [0, π/2] or -1 (unset). Our tiltX is degrees [-90, 90]. val tiltRad = pt.tiltX?.let { Math.toRadians(it.toDouble()).toFloat().coerceIn(0f, Math.PI.toFloat() / 2f) } ?: -1f + // Ink API rejects duplicate (position, elapsed_time) pairs — skip them + if (pt.x == prevX && pt.y == prevY && elapsedMs == prevElapsed) continue + + prevX = pt.x + prevY = pt.y + prevElapsed = elapsedMs + batch.add( type = InputToolType.STYLUS, x = pt.x,