-
-
Notifications
You must be signed in to change notification settings - Fork 2.8k
feat: setup Baseline profiles #20702
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
19542e0
9640964
a769132
2ff7e88
97e925d
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,21 @@ | ||
| <?xml version="1.0" encoding="utf-8"?> | ||
| <!-- | ||
| Benchmark-only manifest overlay. Merged into the benchmark buildType only. | ||
| Keeps <profileable> out of production release APKs. | ||
| --> | ||
| <manifest xmlns:android="http://schemas.android.com/apk/res/android" | ||
| xmlns:tools="http://schemas.android.com/tools"> | ||
|
|
||
| <application> | ||
| <!-- | ||
| Allows Macrobenchmark / Perfetto to attach via adb shell. | ||
| Required by the :baselineprofile module to collect baseline | ||
| profiles and run StartupBenchmark. `android:shell="true"` | ||
| restricts attachment to shell-initiated profiling only. | ||
| --> | ||
| <profileable | ||
| android:shell="true" | ||
| tools:targetApi="29" /> | ||
| </application> | ||
|
|
||
| </manifest> |
Large diffs are not rendered by default.
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,54 @@ | ||
| # `:baselineprofile` | ||
|
|
||
| This module generates AnkiDroid's **baseline profile** - a list of classes | ||
| and methods ART pre-compiles at install time so cold start is faster. | ||
|
|
||
| It has two classes: | ||
|
|
||
| - **`BaselineProfileGenerator`** captures a cold-start trace and writes | ||
| `baseline-prof.txt` into `:AnkiDroid`. | ||
| - **`StartupBenchmark`** measures cold-start time with vs without the | ||
| profile so you can see the impact. | ||
|
|
||
| For results to be valid for comparisons and real-world impact, they must | ||
| be run on real hardware devices ideally the same device over time and | ||
| representative of typical user hardware. So this is currently not | ||
| intended to be used in CI unless CI is connected to real devices (e.g. | ||
| Firebase Test Lab or similar). | ||
|
|
||
| ## Requirements | ||
|
|
||
| - Physical Android device, API 28+ (emulators aren't trustworthy) | ||
| - USB debugging on, device plugged in | ||
|
|
||
| ## Regenerate the profile | ||
|
|
||
| Run with `-PcustomSuffix` and `-PcustomName` so the benchmarked APK installs | ||
| alongside your real AnkiDroid (`com.ichi2.anki.bp`) instead of replacing it. | ||
| Without these flags the build targets `com.ichi2.anki`, the production app | ||
| ID — installing it would clobber your real collection or fail with a | ||
| signing-key mismatch. | ||
|
|
||
| ```sh | ||
| ./gradlew :AnkiDroid:generateBaselineProfile \ | ||
| -PcustomSuffix=".bp" \ | ||
| -PcustomName="AnkiDroid Baseline" | ||
| ``` | ||
|
|
||
| Takes a few minutes. The plugin installs a non-minified build, runs the | ||
| cold-start journey (launch app -> dismiss intro screen if present -> wait | ||
| for DeckPicker), and writes the result to `baseline-prof.txt`. | ||
|
|
||
| ## When to regenerate | ||
|
|
||
| When the startup path changes meaningfully — new init work in | ||
| `AnkiDroidApp.onCreate`, a refactor to `IntentHandler` or `DeckPicker`, | ||
| a new dependency wired into `Application`. Otherwise, leave it alone; | ||
| the profile is device- and OS-agnostic, one commit covers everyone. | ||
|
Comment on lines
+42
to
+47
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Is it worth adding an annotation for this case, to flag to developers that changes may be necessary Is it worth enforcing that this is run before a public release?
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Not worth it IMO since developer will be concerned with their change, so a change to DeckPicker rarely touches the cold-start path meaningfully.
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 👍
I suspect this is a more interesting question
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. very interesting -
Anyway, will research this a bit, and as a dev / CI feature will likely at least get it merged in some form even if imperfect now so we can start "feeling" it and see how it works, what we really want out of it
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Having read the documentation and this PR and these comments I think the ideal path forward is to connect to a test lab that is CI accessible but has real hardware devices and a free tier or is free for open source. I have not looked extensively but Firebase Test Lab has some free runs per day on their free tier and I could create an AnkiDroid project there to take advantage of that, in the absence of some better idea. If it goes away, no big deal. But assuming it stays we could select our idea of an "old phone" and a "new phone* (over time new phone becomes old phone and you pick new phone but you'll maintain comparables between each pair for a while to see if anything goes seriously wrong), and start collecting metrics over time (TBD: somehow? that was CI available in order to graph over time?) and we could run it on a schedule or on push to main but only if versionCode changed to capture change-over-time-for-releases ? Not sure how to hook Firebase Test Lab up to CI but they sell that service so docs should be commercial-grade |
||
|
|
||
| ## Config notes | ||
|
|
||
| See comments in [`build.gradle.kts`](build.gradle.kts) for the rationale | ||
| behind `minSdk`, `missingDimensionStrategy`, the `benchmark` buildType, | ||
| and `self-instrumenting`. The KDoc on each class explains the journey | ||
| and the measurement modes. | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,93 @@ | ||
| import com.android.build.api.dsl.TestExtension | ||
|
|
||
| plugins { | ||
| alias(libs.plugins.android.test) | ||
| alias(libs.plugins.androidx.baselineprofile) | ||
| } | ||
|
|
||
| configure<TestExtension> { | ||
| namespace = "com.ichi2.anki.baselineprofile" | ||
|
|
||
| compileSdk = | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. cross-linking with related PR for boilerplate-ness:
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'll rebase and see what boilerplate in baselineprofile/build.gradle.kts can be consolidated. Not blocking this PR. Or David can if this gets in first, all good for now |
||
| libs.versions.compileSdk | ||
| .get() | ||
| .toInt() | ||
|
|
||
| compileOptions { | ||
| sourceCompatibility = JavaVersion.VERSION_17 | ||
| targetCompatibility = JavaVersion.VERSION_17 | ||
| } | ||
|
|
||
| defaultConfig { | ||
| // Macrobenchmark + baseline profile generation require API 28+ | ||
| // even though the app may support a lower version. If the app's | ||
| // minSdk ever rises above 28, follow it automatically. | ||
| minSdk = | ||
| maxOf( | ||
| 28, | ||
| libs.versions.minSdk | ||
| .get() | ||
| .toInt(), | ||
| ) | ||
| targetSdk = | ||
| libs.versions.targetSdk | ||
| .get() | ||
| .toInt() | ||
|
|
||
| testInstrumentationRunner = "androidx.test.runner.AndroidJUnitRunner" | ||
|
|
||
| // AnkiDroid defines three product flavors (play, amazon, full) in the | ||
| // `appStore` dimension. Rather than duplicate them here we target the | ||
| // `play` flavor via `missingDimensionStrategy`, which keeps this module | ||
| // flavor-free but still able to resolve `:AnkiDroid` variants. | ||
|
Comment on lines
+41
to
+42
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. what a coincidence, the documentation for macrobenchmark was also important, but flavorless |
||
| missingDimensionStrategy("appStore", "play") | ||
|
david-allison marked this conversation as resolved.
|
||
| } | ||
|
|
||
| buildTypes { | ||
| create("benchmark") { | ||
| isDebuggable = true | ||
| signingConfig = signingConfigs.getByName("debug") | ||
| matchingFallbacks += listOf("release") | ||
| } | ||
| } | ||
|
|
||
| targetProjectPath = ":AnkiDroid" | ||
|
|
||
| @Suppress("UnstableApiUsage") | ||
| experimentalProperties["android.experimental.self-instrumenting"] = true | ||
| } | ||
|
|
||
| baselineProfile { | ||
| useConnectedDevices = true | ||
| } | ||
|
|
||
| apply(from = "../lint.gradle") | ||
|
|
||
| dependencies { | ||
| implementation(libs.androidx.test.junit) | ||
| implementation(libs.androidx.espresso.core) | ||
| implementation(libs.androidx.uiautomator) | ||
| implementation(libs.androidx.benchmark.macro.junit4) | ||
| } | ||
|
|
||
| androidComponents { | ||
| onVariants { v -> | ||
| val artifactsLoader = v.artifacts.getBuiltArtifactsLoader() | ||
| // Pull the applicationId from the actual built APK instead of | ||
| // hardcoding `com.ichi2.anki`. Most of us build with something like | ||
| // `-PcustomSuffix=".bp"` so the benchmark APK installs next to our | ||
| // real AnkiDroid (`com.ichi2.anki.bp`) rather than replacing it. | ||
| // If we hardcoded the fallback, the benchmark would happily target | ||
| // the production app and nuke your actual collection. | ||
| v.instrumentationRunnerArguments.put( | ||
| "targetAppId", | ||
| v.testedApks.map { | ||
| artifactsLoader.load(it)?.applicationId | ||
| ?: error( | ||
| "Could not resolve targetAppId from testedApks. " + | ||
| "Refusing to fall back to a default to avoid clobbering the production install.", | ||
| ) | ||
| }, | ||
| ) | ||
| } | ||
| } | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,4 @@ | ||
| <?xml version="1.0" encoding="utf-8"?> | ||
| <manifest> | ||
|
|
||
| </manifest> |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,82 @@ | ||
| /* | ||
| * Copyright (c) 2026 Ashish Yadav <mailtoashish693@gmail.com> | ||
| * | ||
| * This program is free software; you can redistribute it and/or modify it under | ||
| * the terms of the GNU General Public License as published by the Free Software | ||
| * Foundation; either version 3 of the License, or (at your option) any later | ||
| * version. | ||
| * | ||
| * This program is distributed in the hope that it will be useful, but WITHOUT ANY | ||
| * WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A | ||
| * PARTICULAR PURPOSE. See the GNU General Public License for more details. | ||
| * | ||
| * You should have received a copy of the GNU General Public License along with | ||
| * this program. If not, see <http://www.gnu.org/licenses/>. | ||
| */ | ||
|
|
||
| package com.ichi2.anki.baselineprofile | ||
|
|
||
| import androidx.benchmark.macro.junit4.BaselineProfileRule | ||
| import androidx.test.ext.junit.runners.AndroidJUnit4 | ||
| import androidx.test.filters.LargeTest | ||
| import androidx.test.platform.app.InstrumentationRegistry | ||
| import androidx.test.uiautomator.By | ||
| import androidx.test.uiautomator.Until | ||
| import org.junit.Rule | ||
| import org.junit.Test | ||
| import org.junit.runner.RunWith | ||
|
|
||
| /** | ||
| * Generates a baseline profile that captures the critical user journey for AnkiDroid's | ||
| * cold start: launching the app and reaching the main [com.ichi2.anki.DeckPicker] screen. | ||
| * | ||
| * The captured launch path is: | ||
| * 1. [com.ichi2.anki.IntentHandler] — the MAIN/LAUNCHER activity declared in | ||
| * the manifest. `startActivityAndWait()` launches this. | ||
| * 2. [com.ichi2.anki.DeckPicker] — IntentHandler.onCreate() immediately | ||
| * creates an intent for DeckPicker and calls `launchDeckPickerIfNoOtherTasks()`. | ||
| * 3. The `device.wait` + `device.waitForIdle()` calls wait for DeckPicker to | ||
| * fully render before the trace window closes. | ||
| * | ||
| * On a fresh install, [com.ichi2.anki.IntroductionActivity] appears before | ||
| * DeckPicker. The journey handles this by tapping the "Get Started" button | ||
| * (resource id `get_started`) if it is present, so the profile captures the | ||
| * full path to DeckPicker regardless of device state. | ||
| * | ||
| * Run on a connected API 28+ physical device with: | ||
| * ``` | ||
| * ./gradlew :AnkiDroid:generateBaselineProfile | ||
| * ``` | ||
| * The generated profile is committed to | ||
| * `AnkiDroid/src/main/baselineProfiles/baseline-prof.txt` by the | ||
| * [androidx.baselineprofile] plugin and is automatically embedded in subsequent | ||
| * app builds. | ||
| */ | ||
| @LargeTest | ||
| @RunWith(AndroidJUnit4::class) | ||
| class BaselineProfileGenerator { | ||
| @get:Rule | ||
| val rule = BaselineProfileRule() | ||
|
|
||
| @Test | ||
| fun generate() { | ||
| val targetPackage = | ||
| InstrumentationRegistry.getArguments().getString("targetAppId") | ||
| ?: throw IllegalStateException("targetAppId not passed as instrumentation runner arg") | ||
|
|
||
| rule.collect(packageName = targetPackage) { | ||
| pressHome() | ||
| // Launches IntentHandler (MAIN/LAUNCHER), which routes to DeckPicker. | ||
| startActivityAndWait() | ||
|
david-allison marked this conversation as resolved.
|
||
|
|
||
| // On a fresh install, IntroductionActivity appears before DeckPicker. | ||
| // Dismiss it by tapping "Get Started" so the profile always captures | ||
| // the full path through to DeckPicker, regardless of device state. | ||
| device.findObject(By.res(targetPackage, "get_started"))?.click() | ||
|
|
||
| // Wait for DeckPicker to fully render so all startup classes are captured. | ||
| device.wait(Until.hasObject(By.pkg(targetPackage).depth(0)), 5_000) | ||
| device.waitForIdle() | ||
| } | ||
| } | ||
| } | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,80 @@ | ||
| /* | ||
| * Copyright (c) 2026 Ashish Yadav <mailtoashish693@gmail.com> | ||
| * | ||
| * This program is free software; you can redistribute it and/or modify it under | ||
| * the terms of the GNU General Public License as published by the Free Software | ||
| * Foundation; either version 3 of the License, or (at your option) any later | ||
| * version. | ||
| * | ||
| * This program is distributed in the hope that it will be useful, but WITHOUT ANY | ||
| * WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A | ||
| * PARTICULAR PURPOSE. See the GNU General Public License for more details. | ||
| * | ||
| * You should have received a copy of the GNU General Public License along with | ||
| * this program. If not, see <http://www.gnu.org/licenses/>. | ||
| */ | ||
|
|
||
| package com.ichi2.anki.baselineprofile | ||
|
|
||
| import androidx.benchmark.macro.BaselineProfileMode | ||
| import androidx.benchmark.macro.CompilationMode | ||
| import androidx.benchmark.macro.StartupMode | ||
| import androidx.benchmark.macro.StartupTimingMetric | ||
| import androidx.benchmark.macro.junit4.MacrobenchmarkRule | ||
| import androidx.test.ext.junit.runners.AndroidJUnit4 | ||
| import androidx.test.filters.LargeTest | ||
| import androidx.test.platform.app.InstrumentationRegistry | ||
| import org.junit.Rule | ||
| import org.junit.Test | ||
| import org.junit.runner.RunWith | ||
|
|
||
| /** | ||
| * Measures AnkiDroid cold-startup time in two configurations so the | ||
| * before/after impact of the baseline profile is visible in a single run: | ||
| * | ||
| * - [startupCompilationNone] — [CompilationMode.None]: no AOT compilation, | ||
| * all code is JIT'd on the fly. This is the "before" baseline. | ||
| * - [startupCompilationBaselineProfile] — [CompilationMode.Partial] with | ||
| * [BaselineProfileMode.Require]: AOT-compiles exactly the methods listed | ||
| * in `baseline-prof.txt`. `Require` fails the test if the profile isn't | ||
| * installed, so any improvement is guaranteed to come from the profile. | ||
| * | ||
| * Both methods run [StartupMode.COLD] with 10 iterations. Compare the | ||
| * median of `timeToInitialDisplayMs` across the two methods. | ||
| * | ||
| * **This benchmark is for local, manual use only.** It requires a connected | ||
| * physical device, has no pass/fail assertions, and should not be included | ||
| * in CI test suites. Run with: | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. |
||
| * ``` | ||
| * ./gradlew :baselineprofile:connectedBenchmarkBenchmarkAndroidTest | ||
| * ``` | ||
| */ | ||
| @LargeTest | ||
| @RunWith(AndroidJUnit4::class) | ||
|
david-allison marked this conversation as resolved.
|
||
| class StartupBenchmark { | ||
| @get:Rule | ||
| val rule = MacrobenchmarkRule() | ||
|
|
||
| @Test | ||
| fun startupCompilationNone() = startup(CompilationMode.None()) | ||
|
|
||
| @Test | ||
| fun startupCompilationBaselineProfile() = startup(CompilationMode.Partial(baselineProfileMode = BaselineProfileMode.Require)) | ||
|
|
||
| private fun startup(mode: CompilationMode) { | ||
| val targetPackage = | ||
| InstrumentationRegistry.getArguments().getString("targetAppId") | ||
| ?: throw IllegalStateException("targetAppId not passed as instrumentation runner arg") | ||
|
|
||
| rule.measureRepeated( | ||
| packageName = targetPackage, | ||
| metrics = listOf(StartupTimingMetric()), | ||
| iterations = 10, | ||
| startupMode = StartupMode.COLD, | ||
| compilationMode = mode, | ||
| ) { | ||
| pressHome() | ||
| startActivityAndWait() | ||
| } | ||
| } | ||
| } | ||

Uh oh!
There was an error while loading. Please reload this page.