feat: setup Baseline profiles by criticalAY · Pull Request #20702 · ankidroid/Anki-Android

criticalAY · 2026-04-09T14:31:33Z

Note

Assisted-by: Claude Opus 4.6 - writing readme

Purpose / Description

This PR setsups Baseline Profiles for the app i.e. macrobenchmark producer module that generates AnkiDroid's baseline profile and measures cold-start time.

Fixes

Fixes part of [Inspect] Baseline profiles and startup profiles #20690

Approach

See commits

How Has This Been Tested?

./gradlew :baselineprofile:connectedBenchmarkBenchmarkAndroidTest -Pandroid.testInstrumentationRunnerArguments.class=com.ichi2.anki.baselineprofile.StartupBenchmark

Learning (optional, can help others)

Note: I think these are the casis where we want to refenerate the profiles

AnkiDroidApp.onCreate / IntentHandler / DeckPicker.onCreate changed.
New dependency initialized during startup.
StartupBenchmark shows baseline ~ none (profile has drifted).
Per release cycle as a sanity check.

Checklist

Please, go through these checks before submitting the PR.

You have a descriptive commit message with a short title (first line, max 50 chars).
You have commented your code, particularly in hard-to-understand areas
You have performed a self-review of your own code
UI changes: include screenshots of all affected screens (in particular showing any new or changed strings)
UI Changes: You have tested your change using the Google Accessibility Scanner

criticalAY · 2026-04-09T14:42:37Z

As I mentioned in the issue on my device (Pixel 10) this is around 5% but if there would be a low spec device then it should have a better result so we can expect around 5-10% improvement imo

david-allison

I need to do more research, just my initial 'high level' thoughts here

I love the concept!

david-allison · 2026-04-10T09:12:21Z

Please add more comments to the docs in the commit messages

david-allison

LGTM! Confirmed DeckPicker:onResume is called.

Maybe add a TODO to add a test for this?

SPLcom/ichi2/anki/DeckPicker;->onResume()V

david-allison · 2026-04-18T06:04:48Z

@criticalAY

This is likely going to have major conflicts with:

Please feel free to assign this to me, there was a lot of nastiness in both PRs

criticalAY · 2026-04-18T06:06:30Z

I tried to rebase but it seems it would need more resolving, thankyou @david-allison

david-allison

Please add a readme to the module

david-allison · 2026-04-18T09:38:44Z

+
+- Physical Android device, API 28+ (emulators aren't trustworthy)
+- USB debugging on, device plugged in
+- `adb uninstall com.ichi2.anki` first if you hit a signing-key error


Do we /need/ to use com.ichi2.anki, or could we get away with .debug so the maintainer doesn't need to uninstall the app

Release only. Baseline profiles have no effect on debug builds.

Debug APKs aren't minified, aren't processed by R8, and the runtime doesn't apply AOT compilation to debuggable processes. The profile would just sit in the APK as dead weight.

This isn't about a debug app (.A is just as relevant, sorry for the confusion).

Using com.ichi2.anki is the production app identifier, and would mean that I'm having to uninstall the app which runs on my live collection on my main phone for a development task.

This should be changed if possible to have the interpolated app name - Brayan set up infra to make that possible in the past, I'll try to drop a comment on the specific affected line in the gradle files

Ok, tthe hardcoded fallback in targetAppId is gone (it now errors instead of silently defaulting), and the README documents the parallel-install workflow

david-allison · 2026-04-18T09:39:14Z

+## When to regenerate
+
+When the startup path changes meaningfully — new init work in
+`AnkiDroidApp.onCreate`, a refactor to `IntentHandler` or `DeckPicker`,
+a new dependency wired into `Application`. Otherwise, leave it alone;
+the profile is device- and OS-agnostic, one commit covers everyone.


Is it worth adding an annotation for this case, to flag to developers that changes may be necessary

Is it worth enforcing that this is run before a public release?

Not worth it IMO since developer will be concerned with their change, so a change to DeckPicker rarely touches the cold-start path meaningfully.

👍

Is it worth enforcing that this is run before a public release?

I suspect this is a more interesting question

very interesting -

the onCreate "you should regenerate baselines" thing is interesting, note that we just touched onCreate in order to set up widget reminders on boot, and I'd be shocked if in PRs similar to that one the person remembered. Bears some thought 🤔

when to run / run before release builds - would be interesting to perhaps check for a change in the version code of format "switching from alpha to beta" or "beta to public release" - if that digit in the versionCode changed, the android test release build run could re-run baselines and...do I don't know what with the information - similar to APK size check I suppose it could post a comment on the commit line with the information and a link to git blame where you could click and see the last commit with the last baseline commit message from it with baseline timing ? Just a brainstorm

Anyway, will research this a bit, and as a dev / CI feature will likely at least get it merged in some form even if imperfect now so we can start "feeling" it and see how it works, what we really want out of it

Having read the documentation and this PR and these comments I think the ideal path forward is to connect to a test lab that is CI accessible but has real hardware devices and a free tier or is free for open source.

I have not looked extensively but Firebase Test Lab has some free runs per day on their free tier and I could create an AnkiDroid project there to take advantage of that, in the absence of some better idea. If it goes away, no big deal.

But assuming it stays we could select our idea of an "old phone" and a "new phone* (over time new phone becomes old phone and you pick new phone but you'll maintain comparables between each pair for a while to see if anything goes seriously wrong), and start collecting metrics over time (TBD: somehow? that was CI available in order to graph over time?) and we could run it on a schedule or on push to main but only if versionCode changed to capture change-over-time-for-releases ?

Not sure how to hook Firebase Test Lab up to CI but they sell that service so docs should be commercial-grade
Not sure how to persist state (performance over time each versionCode change) in a CI-accessible way but there must be some way ? Maybe even a wiki page with entries and a script-generated graph of performance over time, and the CI script edits it or something

mikehardy

I think the app name thing is a worthy change in this same PR, I think it's table stakes for this to work

I think the thoughts about generating it over time in CI with a test lab is a splittable item for later

mikehardy · 2026-04-19T21:18:45Z

+ *
+ * **This benchmark is for local, manual use only.** It requires a connected
+ * physical device, has no pass/fail assertions, and should not be included
+ * in CI test suites. Run with:


I know these need to run on device but I work with Firebase a lot, and...they have a free plan which allows for a sufficient (IMHO) number of runs a day for this to run as a scheduled thing for graphs-over-time and/or the occasional manual_dispatch...first column is the "free" plan:

mikehardy · 2026-04-19T21:20:41Z

+configure<TestExtension> {
+    namespace = "com.ichi2.anki.baselineprofile"
+
+    compileSdk =


cross-linking with related PR for boilerplate-ness:

build: Introduce Gradle Convention Plugins #20778

I'll rebase and see what boilerplate in baselineprofile/build.gradle.kts can be consolidated. Not blocking this PR. Or David can if this gets in first, all good for now

mikehardy · 2026-04-19T21:22:25Z

+        // `play` flavor via `missingDimensionStrategy`, which keeps this module
+        // flavor-free but still able to resolve `:AnkiDroid` variants.


what a coincidence, the documentation for macrobenchmark was also important, but flavorless

mikehardy · 2026-04-19T21:28:39Z

+- **`StartupBenchmark`** measures cold-start time with vs without the
+  profile so you can see the impact.
+
+Neither runs in CI. Both are manual tools for maintainers.


related to my firebase test labs comment I think this is an assertion based on actual information and exposing the actual information is useful

specifically I would alter this perhaps like

Suggested change

Neither runs in CI. Both are manual tools for maintainers.

For results to be valid for comparisons and real-world impact, they must be run on real hardware devices - ideally the same device over time and representative of typical user hardware. So this is currently not intended to be used in CI unless CI is connected to real devices (e.g. Firebase Test Lab or similar)

mikehardy · 2026-04-19T21:29:31Z

+
+- Physical Android device, API 28+ (emulators aren't trustworthy)
+- USB debugging on, device plugged in
+- `adb uninstall com.ichi2.anki` first if you hit a signing-key error


This should be changed if possible to have the interpolated app name - Brayan set up infra to make that possible in the past, I'll try to drop a comment on the specific affected line in the gradle files

mikehardy · 2026-04-19T21:34:12Z

+## When to regenerate
+
+When the startup path changes meaningfully — new init work in
+`AnkiDroidApp.onCreate`, a refactor to `IntentHandler` or `DeckPicker`,
+a new dependency wired into `Application`. Otherwise, leave it alone;
+the profile is device- and OS-agnostic, one commit covers everyone.


Having read the documentation and this PR and these comments I think the ideal path forward is to connect to a test lab that is CI accessible but has real hardware devices and a free tier or is free for open source.

I have not looked extensively but Firebase Test Lab has some free runs per day on their free tier and I could create an AnkiDroid project there to take advantage of that, in the absence of some better idea. If it goes away, no big deal.

But assuming it stays we could select our idea of an "old phone" and a "new phone* (over time new phone becomes old phone and you pick new phone but you'll maintain comparables between each pair for a while to see if anything goes seriously wrong), and start collecting metrics over time (TBD: somehow? that was CI available in order to graph over time?) and we could run it on a schedule or on push to main but only if versionCode changed to capture change-over-time-for-releases ?

Not sure how to hook Firebase Test Lab up to CI but they sell that service so docs should be commercial-grade
Not sure how to persist state (performance over time each versionCode change) in a CI-accessible way but there must be some way ? Maybe even a wiki page with entries and a script-generated graph of performance over time, and the CI script edits it or something

mikehardy · 2026-04-19T21:49:58Z

+        val artifactsLoader = v.artifacts.getBuiltArtifactsLoader()
+        v.instrumentationRunnerArguments.put(
+            "targetAppId",
+            v.testedApks.map { artifactsLoader.load(it)?.applicationId ?: "com.ichi2.anki" },


this...may work but should really be tested with -PcustomSuffix=".A" -PcustomName="AnkiDroid ParallelA" (or similar) to make sure, and if not it should use the style here to make sure it does work

Otherwise developers will never want to do this as it'll nuke their actual app install

https://github.com/criticalAY/Anki-Android/blob/570c05accde169cfadff368290f52d576d3f4f5b/AnkiDroid/build.gradle#L169-L173

mikehardy · 2026-04-19T21:51:47Z

For the avoidance of doubt I want to say I love this though - I read about baseline profiles when they came out, and the macrobenchmark (and microbenchmark...) libraries when they came out and thought "man, if I had time I'd love to do this"

you did it!

david-allison · 2026-04-20T10:44:48Z

I'll push back with a question: should we split out the CI runner into a separate issue/PR? It feels like scope creep

As far as I understand it (once we get the version name/docs sorted out), this is a win on its own, a somewhat stale profile is better than no profile, and setting up CI to run this properly on Firebase might take a while to get right.

mikehardy · 2026-04-20T11:14:29Z

@david-allison

I'll push back with a question: should we split out the CI runner into a separate issue/PR? It feels like scope creep

I agree and indeed said so explicitly:

I think the thoughts about generating it over time in CI with a test lab is a splittable item for later

...trying to do that here would make this PR kind of ridiculous IMHO

- BaselineProfileGenerator: Automates the critical user journey from home screen to DeckPicker to capture startup traces. - StartupBenchmark: Uses Macrobenchmark to compare "None" vs. "Baseline Profile" compilation modes. This lets us isolate and measure the speed boost we get from install-time AOT.

…ofile

Assisted-by: Claude Opus 4.6

criticalAY · 2026-05-02T05:24:37Z

Verified parallel build it works fine

re-requested

github-actions Bot added the Needs Review label Apr 9, 2026

criticalAY force-pushed the baseline-profiles branch 2 times, most recently from 57a4fee to bc00bd3 Compare April 9, 2026 14:34

criticalAY removed the Needs Review label Apr 9, 2026

criticalAY marked this pull request as draft April 9, 2026 14:58

criticalAY marked this pull request as ready for review April 9, 2026 16:06

criticalAY added Needs Review labels Apr 9, 2026

david-allison reviewed Apr 9, 2026

View reviewed changes

david-allison reviewed Apr 10, 2026

View reviewed changes

Comment thread baselineprofile/build.gradle.kts

david-allison reviewed Apr 10, 2026

View reviewed changes

Comment thread settings.gradle.kts

david-allison added the Needs Author Reply Waiting for a reply from the original author label Apr 12, 2026

criticalAY force-pushed the baseline-profiles branch from bc00bd3 to bc0149c Compare April 13, 2026 08:59

david-allison requested changes Apr 13, 2026

View reviewed changes

Comment thread AnkiDroid/build.gradle

Comment thread baselineprofile/src/main/kotlin/com/ichi2/anki/baselineprofile/BaselineProfileGenerator.kt Outdated

criticalAY force-pushed the baseline-profiles branch 3 times, most recently from 5fcd050 to c915ac5 Compare April 13, 2026 09:31

criticalAY removed the Needs Author Reply Waiting for a reply from the original author label Apr 13, 2026

david-allison approved these changes Apr 13, 2026

View reviewed changes

criticalAY added Needs Second Approval Has one approval, one more approval to merge and removed Needs Review labels Apr 13, 2026

criticalAY mentioned this pull request Apr 13, 2026

[Baseline Profile]: Add a test to confirm DeckPicker:onResume is called #20734

Open

github-actions Bot added the Has Conflicts label Apr 16, 2026

criticalAY force-pushed the baseline-profiles branch from c915ac5 to 2e2ce73 Compare April 18, 2026 05:57

criticalAY removed the Has Conflicts label Apr 18, 2026

criticalAY assigned david-allison Apr 18, 2026

david-allison force-pushed the baseline-profiles branch from 2e2ce73 to 303b044 Compare April 18, 2026 06:46

david-allison reviewed Apr 18, 2026

View reviewed changes

criticalAY force-pushed the baseline-profiles branch from 303b044 to b9c87cf Compare April 18, 2026 07:16

david-allison reviewed Apr 18, 2026

View reviewed changes

mikehardy previously requested changes Apr 19, 2026

View reviewed changes

mikehardy added the Needs Author Reply Waiting for a reply from the original author label Apr 19, 2026

david-allison removed their assignment Apr 22, 2026

criticalAY and others added 5 commits May 2, 2026 09:51

feat: add baseline profile plugin and macrobenchmark library

19542e0

build(AnkiDroid): consume generated baseline profile from :baselinepr…

a769132

…ofile

chore(baselineprofile): add generated baseline profile

2ff7e88

chore: add readme for baselineprofiles module

97e925d

Assisted-by: Claude Opus 4.6

criticalAY force-pushed the baseline-profiles branch from b9c87cf to 97e925d Compare May 2, 2026 04:55

criticalAY mentioned this pull request May 2, 2026

Run baseline profile generation + StartupBenchmark in CI via Firebase Test Lab #20924

Open

criticalAY removed the Needs Author Reply Waiting for a reply from the original author label May 2, 2026

criticalAY requested a review from mikehardy May 2, 2026 05:07

github-actions Bot added the Has Conflicts label May 6, 2026

		// `play` flavor via `missingDimensionStrategy`, which keeps this module
		// flavor-free but still able to resolve `:AnkiDroid` variants.

	Neither runs in CI. Both are manual tools for maintainers.
	For results to be valid for comparisons and real-world impact, they must be run on real hardware devices - ideally the same device over time and representative of typical user hardware. So this is currently not intended to be used in CI unless CI is connected to real devices (e.g. Firebase Test Lab or similar)

Uh oh!

Conversation

criticalAY commented Apr 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose / Description

Fixes

Approach

How Has This Been Tested?

Learning (optional, can help others)

Checklist

Uh oh!

criticalAY commented Apr 9, 2026

Uh oh!

david-allison left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

david-allison commented Apr 10, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

david-allison left a comment

Choose a reason for hiding this comment

Uh oh!

david-allison commented Apr 18, 2026

Uh oh!

criticalAY commented Apr 18, 2026

Uh oh!

david-allison left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

david-allison Apr 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mikehardy Apr 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mikehardy left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mikehardy Apr 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

criticalAY commented Apr 9, 2026 •

edited

Loading

david-allison left a comment •

edited

Loading

david-allison Apr 18, 2026 •

edited

Loading

mikehardy Apr 19, 2026 •

edited

Loading

mikehardy Apr 19, 2026 •

edited

Loading

david-allison commented Apr 20, 2026 •

edited

Loading