perf: spanner mutation limit increase to 80000 max_total_records by taddes · Pull Request #2327 · mozilla-services/syncstorage-rs

taddes · 2026-05-21T19:45:41Z

Description

Much of this was understanding the underlying mechanics of mutations and updating the proper math to reflect the mutation limit capacity ok 80,000 in relation to max_total_records

Some details on ths: Spanner caps mutations at 80,000 per commit, and each BSO row in the batch commit costs up to 12 mutations in the worst case (4 primary-key columns + 4 non-PK columns + a delete-and-insert on each of the two secondary indexes). On top of that, the commit pays a fixed 13 mutations of overhead: 6 for the parent user_collections upsert, 1 for the batch delete, and 6 to refresh quota counts. Solving 12N + 13 <= 80,000 gives a ceiling of 6,665 records, so we pick 6,656 (= 1,664 × 4) to preserve the same propotion the legacy 1,664/20,000 cap had, leaving 115 mutations of margin for breathing room.

Add docs/src/syncstorage/syncstorage-spanner-db.md as the canonical reference for the Spanner backend with schema, full configuration option, and per-commit mutation budget.
Bumped dev max_total_records in config/local.example.toml from 1666 to 6656 to match the value to Spanner's 80,000-mutation-per-commit ceiling.

To Check

make doc-prev renders the new page cleanly (mermaid diagram + internal links)
/info/configuration returns max_total_records: 6656 against the bumped dev config
Ensure things look good in stage when we roll out update

Issue(s)

Helm Chart update here
Closes STOR-190.

chenba · 2026-05-22T16:29:49Z

+| Table              | Description                                                                                                  |
+| ------------------ | ------------------------------------------------------------------------------------------------------------ |
+| `user_collections` | Per-user metadata about each collection (modified time, record count, total bytes). Parent of `bsos`/`batches` via `INTERLEAVE IN PARENT`. |
+| `bsos`             | Stores Basic Storage Objects (BSOs) the synced records. Interleaved in `user_collections`.                |


Suggested change

| `bsos` | Stores Basic Storage Objects (BSOs) the synced records. Interleaved in `user_collections`. |

| `bsos` | Stores Basic Storage Objects (BSOs), the synced records. Interleaved in `user_collections`. |

chenba · 2026-05-22T16:40:28Z

+| `batches`          | Temporary staging row per in-progress batch upload. Interleaved in `user_collections`.                       |
+| `batch_bsos`       | BSOs belonging to a batch, pending commit. Interleaved in `batches`.                                         |
+
+All `bsos` and `batches` rows are physically co-located with their `user_collections` parent Spanner's interleaving puts a user's collection metadata, BSOs, and pending batches on the same split. `ON DELETE CASCADE` from `batches` to `batch_bsos` and `user_collections` `bsos`/`batches` means parent deletes wipe descendants atomically.


Suggested change

All `bsos` and `batches` rows are physically co-located with their `user_collections` parent Spanner's interleaving puts a user's collection metadata, BSOs, and pending batches on the same split. `ON DELETE CASCADE` from `batches` to `batch_bsos` and `user_collections` `bsos`/`batches` means parent deletes wipe descendants atomically.

All `bsos` and `batches` rows are physically co-located with their `user_collections` parent. Spanner's interleaving puts a user's collection metadata, BSOs, and pending batches on the same split. `ON DELETE CASCADE` from `batches` to `batch_bsos` and `user_collections` `bsos`/`batches` means parent deletes wipe descendants atomically.

chenba · 2026-05-22T17:49:53Z

+
+### Env var
+
+`max_total_records` is set in production via the `SYNC_SYNCSTORAGE__LIMITS__MAX_TOTAL_RECORDS` environment variable with no redeploy required to change it. The standalone-server default in `syncstorage-settings/src/lib.rs` is 10,000 and applies to non-Spanner backends; the Spanner production deployment always overrides it.


no redeploy required to change it

I'm having trouble picturing how this works. How does the running process get the new value?

Sry, this was in the local context when working with the config.local.toml which I altered to reflect the new value. Good catch

When working locally, one needs to restart the service when the settings changed no?

Yup, and often just a fresh cargo build (or more scoped command depending on which db)

I'm more confused. Assuming that "Yup" is "yup the service needs to be restarted when changing settings", then I don't understand how that led to "no redeploy required".

I don't get the relevance of cargo build. That compounds my confusion.

Are we talking about different things?

Haha, yeah maybe so. It's really not relevant anymore since its gone now. I just meant for local development: cargo build just means having to recompile the project after changing an env variable

having to recompile the project after changing an env variable

I don't see why that'd be necessary since SYNC_SYNCSTORAGE__LIMITS__MAX_TOTAL_RECORDS is an existing setting.

taddes · 2026-05-26T15:09:28Z

@pjenvey @chenba Changes made in Webservices infra. Lmk if anything remains or needs to be added to the context here

chenba · 2026-05-27T17:44:35Z

Since we are about to drop the secondary indexes, can the patch be updated accordingly?

taddes · 2026-06-01T20:37:11Z

We can merge this in,but will land final update for the higher limit once we finally get rid of last secondary index.

pjenvey · 2026-06-04T21:09:42Z

@taddes We'll go ahead and drop BsoExpiry early next week, so we can apply that to this doc now. LGTM otherwise

taddes · 2026-06-05T02:22:22Z

@taddes We'll go ahead and drop BsoExpiry early next week, so we can apply that to this doc now. LGTM otherwise

Sounds good, just drop the approve next wk and I'll land it 👍

pjenvey

can land this as is now but I figured you'd drop the secondary indices paragraph/numbers first

chenba · 2026-06-16T14:51:29Z

 # max_quota_limit = 200000000
 syncstorage.enabled = true
-syncstorage.limits.max_total_records = 1666 # See issues #298/#333
+syncstorage.limits.max_total_records = 9984 # Post-#2382 (no `bsos` secondary indexes); see docs/src/syncstorage/syncstorage-spanner-db.md


Either remove it or mention what #2382 is. Or make it an url?

taddes self-assigned this May 21, 2026

taddes force-pushed the perf/spanner-mutation-limit-STOR-190 branch 2 times, most recently from 13101bf to 30a52ea Compare May 21, 2026 19:59

taddes changed the title ~~perf: spanner mutation limit stor 190~~ perf: spanner mutation limit increase to 80000 May 21, 2026

taddes changed the title ~~perf: spanner mutation limit increase to 80000~~ perf: spanner mutation limit increase to 80000 max_total_records May 21, 2026

taddes marked this pull request as ready for review May 21, 2026 20:00

taddes force-pushed the perf/spanner-mutation-limit-STOR-190 branch from 30a52ea to 3693933 Compare May 21, 2026 20:09

chenba mentioned this pull request May 21, 2026

refactor(db): use Spanner INSERT OR UPDATE DML #2314

Merged

taddes requested review from chenba and pjenvey May 22, 2026 00:31

chenba reviewed May 22, 2026

View reviewed changes

taddes requested a review from chenba May 22, 2026 20:26

taddes force-pushed the perf/spanner-mutation-limit-STOR-190 branch 2 times, most recently from 508d769 to fbfa8bd Compare May 26, 2026 22:18

taddes force-pushed the perf/spanner-mutation-limit-STOR-190 branch from fbfa8bd to 7f2dfd8 Compare June 1, 2026 14:01

taddes force-pushed the perf/spanner-mutation-limit-STOR-190 branch from 7f2dfd8 to fe65f21 Compare June 5, 2026 02:25

pjenvey previously approved these changes Jun 12, 2026

View reviewed changes

taddes dismissed pjenvey’s stale review via e95e9c4 June 15, 2026 14:57

taddes force-pushed the perf/spanner-mutation-limit-STOR-190 branch from fe65f21 to e95e9c4 Compare June 15, 2026 14:57

chenba reviewed Jun 16, 2026

View reviewed changes

taddes force-pushed the perf/spanner-mutation-limit-STOR-190 branch 2 times, most recently from c7bcaf3 to ca87d88 Compare June 16, 2026 16:50

taddes added 3 commits June 16, 2026 12:51

docs(spanner): add backend reference page

1574c1f

chore(config): bump dev max_total_records to 6656

e191a0e

review

3a6022b

taddes added 3 commits June 16, 2026 12:51

docs(spanner): refresh mutation budget for post-index-drop math

4d8fe4a

chore(config): bump dev max_total_records to 9984

f89e691

chore(config): drop max_total_records inline comment

ca87d88

chenba approved these changes Jun 16, 2026

View reviewed changes

Merge branch 'master' into perf/spanner-mutation-limit-STOR-190

aad8e5c

taddes merged commit a70ca64 into master Jun 16, 2026
24 checks passed

taddes deleted the perf/spanner-mutation-limit-STOR-190 branch June 16, 2026 23:54

	\| `bsos` \| Stores Basic Storage Objects (BSOs) the synced records. Interleaved in `user_collections`. \|
	\| `bsos` \| Stores Basic Storage Objects (BSOs), the synced records. Interleaved in `user_collections`. \|

	All `bsos` and `batches` rows are physically co-located with their `user_collections` parent Spanner's interleaving puts a user's collection metadata, BSOs, and pending batches on the same split. `ON DELETE CASCADE` from `batches` to `batch_bsos` and `user_collections` `bsos`/`batches` means parent deletes wipe descendants atomically.
	All `bsos` and `batches` rows are physically co-located with their `user_collections` parent. Spanner's interleaving puts a user's collection metadata, BSOs, and pending batches on the same split. `ON DELETE CASCADE` from `batches` to `batch_bsos` and `user_collections` `bsos`/`batches` means parent deletes wipe descendants atomically.


		### Env var

		`max_total_records` is set in production via the `SYNC_SYNCSTORAGE__LIMITS__MAX_TOTAL_RECORDS` environment variable with no redeploy required to change it. The standalone-server default in `syncstorage-settings/src/lib.rs` is 10,000 and applies to non-Spanner backends; the Spanner production deployment always overrides it.

Conversation

taddes commented May 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

To Check

Issue(s)

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chenba May 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

taddes commented May 26, 2026

Uh oh!

chenba commented May 27, 2026

Uh oh!

taddes commented Jun 1, 2026

Uh oh!

pjenvey commented Jun 4, 2026

Uh oh!

taddes commented Jun 5, 2026

Uh oh!

pjenvey left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

taddes commented May 21, 2026 •

edited

Loading

chenba May 26, 2026 •

edited

Loading