Translate smoketests from Python to Rust #4102

cloutiertyler · 2026-01-23T03:37:05Z

Description of Changes

This PR translates all of our Python smoketests into Rust tests which can be run from cargo run

Motivation

The purpose of this fivefold:

All developers on the team are familiar with Rust
It simplifies our devops because we can drop Python as a dependency to run the tests
You can now run all tests in the repo through the single cargo test interface
Because we use the SpacetimeDbGuard and cargo test/cargo nextest we can easily parallelize the smoke tests
The smoketests can now use machinery imported from SpacetimeDB crates (e.g. bsatn etc.)

IMPORTANT NOTE!

There are several ways to implement the smoke tests in Rust (none are great):

A separate xtask specifically for the smoke tests
- This doesn't solve the problem of the CLI tests which also use the guard crate
- Idiosyncratic way to run the smoke tests as opposed to cargo test
- Does NOT resolve the cargo within cargo problem because we still have to build the test modules with cargo
A build.rs script in guard which first builds the executables as a compile step for compiling guard
- Deadlocks on a cargo lock file conflict (Outer cargo compiles guard → runs build.rs, inner cargo tries to acquire the build directory lock, outer cargo holds the directory lock, deadlock)
- If you fix the deadlock by using different target dirs, it still looks stuck on building guard because it's actually compiling all of spacetimedb-standalone and spacetimedb-cli.
- Still technically runs cargo inside of cargo.
Add spacetimedb-cli and spacetimedb-standalone as an artifact dependency of the guard crate
- Has good and clear output but requires +nightly when running the smoketests and CLI tests, otherwise won't do the right thing. See Tracking Issue for RFC 3028: Allow "artifact dependencies" on bin, cdylib, and staticlib crates rust-lang/cargo#9096
Compile the executables at runtime during the tests themselves where the first test takes a lock while the executables are building using cargo within cargo
- Makes the tests look like they're taking a long time when they're just waiting for the build to complete
- Requires relatively complex locking machinery across binaries/tests/processes
A two step solution where the developer has to build the binaries before calling the smoke tests
- Very error prone

None of these are good. xtask is not bad, but doesn't enable us to run other integration tests in other crates (e.g. the CLI)

(3) is the correct solution and has the best user experience, but it requires nightly and I don't want to introduce that for all of our tests.

I have chosen to do a combination of (1) and (4). You will now run the smoketests with cargo smoketest. If you run cargo test --all (or use guard) without doing cargo smoketest it will fall back to (4) which compiles the executables at runtime. Running cargo build is the only way to ensure that the executables are not stale because of the internal fingerprint checking. Everything else is fragile not robust.

NOTE! There is no way to avoid cargo within cargo and have the smoke tests be run as cargo tests because the modules under test must be compiled with cargo.

API and ABI breaking changes

Note that this is a BREAKING CHANGE to cargo test --all. The smoketests are now part of cargo test --all unless you specifically exclude them.

Expected complexity level and risk

3, this is partially AI translated. We need to carefully review to ensure the semantics have not regressed.

Testing

Use OnceLock to build spacetimedb-cli and spacetimedb-standalone once per test process, then run the pre-built binary directly instead of using `cargo run`. This avoids repeated cargo overhead and ensures consistent binary reuse across parallel tests.

Create `crates/smoketests/` to translate Python smoketests to Rust: - Add `Smoketest` struct with builder pattern for test setup - Implement CLI helpers: `spacetime_cmd()`, `call()`, `sql()`, `logs()`, etc. - Translate `smoketests/tests/sql.py` → `tests/sql.rs` - Translate `smoketests/tests/call.py` → `tests/call.rs` - Reuse `ensure_binaries_built()` from guard crate (now public) Also fix Windows process cleanup in `SpacetimeDbGuard`: - Use `taskkill /F /T /PID` to kill entire process tree - Prevents orphaned `spacetimedb-standalone.exe` processes

- Translate 4 more Python smoketests to Rust: auto_inc, describe, module_nested_op, panic - Simplify the call API by removing the generic call<T: Serialize> method and renaming call_raw to call, since CLI args are strings - Remove unused serde dependency

Translate additional Python smoketests to Rust: - dml.rs: DML subscription tests - filtering.rs: Unique/non-unique index filtering tests - namespaces.rs: C# code generation namespace tests - add_remove_index.rs: Index add/remove with subscription tests - schedule_reducer.rs: Scheduled reducer tests Infrastructure improvements: - Add subscribe_background() and SubscriptionHandle for proper background subscription semantics matching Python tests - Add spacetime_local() for commands that don't need --server flag - Add timing instrumentation for debugging test performance

- Separate build and publish timing in lib.rs to identify bottlenecks - Use --bin-path to skip redundant rebuild during publish - Add DEVELOP.md explaining cargo-nextest for faster test runs Timing breakdown per test: - WASM build: ~12s (75%) - Server publish: ~2s (12%) - Server spawn: ~2s (12%) cargo-nextest runs all test binaries in parallel, reducing total runtime from ~265s to ~160s (40% faster).

Translate from Python smoketests: - detect_wasm_bindgen.rs: Tests build rejects wasm_bindgen and getrandom (2 tests) - default_module_clippy.rs: Tests default module passes clippy - delete_database.rs: Tests deleting database stops scheduled reducers - fail_initial_publish.rs: Tests failed publish doesn't corrupt control DB - modules.rs: Tests module update lifecycle and breaking changes (2 tests) Also adds spacetime_build() method to Smoketest for testing build failures. Total: 16 test files translated, 32 tests

Clear CARGO* environment variables (except CARGO_HOME) when spawning child cargo build processes. When running under `cargo test`, cargo sets env vars like CARGO_ENCODED_RUSTFLAGS that differ from a normal build, causing child cargo processes to think they need to recompile. This reduces single-test runtime from ~45s to ~18s by avoiding redundant rebuilds of spacetimedb-standalone and spacetimedb-cli.

Add test translations for: - connect_disconnect_from_cli.rs - client connection callbacks - domains.rs - database rename functionality - client_connection_errors.rs - client_connected error handling - confirmed_reads.rs - --confirmed flag for subscriptions/SQL - create_project.rs - spacetime init command Also fix subscription race condition by waiting for initial update before returning from subscribe_background_*, matching Python behavior.

Translate tests for: - views.rs: st_view_* system tables, namespace collisions, SQL views - auto_migration.rs: schema changes, add table migration

Add new_identity() method to support multi-identity tests. Translate tests for: - rls.rs: Row-level security filter tests - energy.rs: Energy balance endpoint test - permissions.rs: Private tables, lifecycle reducers, delete protection

Translate tests for: - new_user_flow.rs: Basic publish/call/SQL workflow - servers.rs: Server add/list/edit commands

.github/workflows/ci.yml

jdetter

I really like these changes so far, I'm excited to talk about this more later today 👍

jdetter · 2026-01-23T09:32:06Z

crates/smoketests/tests/smoketests/add_remove_index.rs

+fn test_add_then_remove_index() {
+    let mut test = Smoketest::builder().module_code(MODULE_CODE).autopublish(false).build();
+
+    let name = format!("test-db-{}", std::process::id());


This would have been an issue if we were to parallelize the smoketests but now we're running 1 spacetime server per test right?

This is actually kind of weird now that we're appending the process ID to the name - why not just use a static name like "test-db"?

I have the same questions. I do feel like the old random-string method was the most trivially "this should be fine" approach.

@jdetter one reason to not use a static name is running the tests against a remote server. Now you can run the tests multiple times without conflicting database names.

crates/smoketests/tests/auto_inc.rs

crates/smoketests/tests/describe.rs

crates/smoketests/tests/smoketests/new_user_flow.rs

crates/smoketests/src/lib.rs

crates/smoketests/tests/smoketests/add_remove_index.rs

crates/guard/src/lib.rs

- Add --config-path to spacetime_local() for test isolation - Fix new_identity() to not pass server arg to logout (matches Python) - Insert --server flag before -- separator in spacetime_cmd() - Update servers.rs to use spacetime_local() for local-only commands - Simplify test files by removing redundant publish_module() calls All 56 smoketests now pass.

Translate smoketests/tests/quickstart.py to Rust. This test validates that the quickstart documentation is correct by extracting code from markdown docs and running it. - Add parse_quickstart() to parse code blocks from markdown with CRLF handling - Add have_pnpm() to check for pnpm availability - Implement QuickstartTest with support for Rust, C#, and TypeScript servers - Rust test passes; C#/TypeScript skip gracefully if dependencies unavailable

bfops · 2026-01-27T21:43:05Z

crates/smoketests/tests/smoketests/modules.rs

+        inserts.as_array().unwrap().iter().any(|r| r["name"] == "Cindy"),
+        "Expected Cindy in second update: {:?}",
+        second
+    );


technically, the original tests also check things like the id of the entity, and that deletes is empty. unsure if there's an easy way to do a full structural comparison.

bfops · 2026-01-27T21:43:45Z

crates/smoketests/tests/smoketests/namespaces.rs

 #[test]
 fn test_spacetimedb_ns_csharp() {
-    let test = Smoketest::builder().module_code(MODULE_CODE).autopublish(false).build();
+    let _test = Smoketest::builder()


out of curiosity, why _test?

Signed-off-by: Tyler Cloutier <cloutiertyler@users.noreply.github.com>

bfops · 2026-01-27T22:32:43Z

crates/smoketests/tests/smoketests/views.rs

+pub fn person(ctx: &ViewContext) -> Option<Person> {
+    None
+}
+"#;


there's a precompiled module for this

bfops · 2026-01-27T22:33:41Z

crates/smoketests/tests/smoketests/views.rs

+pub fn person(ctx: &ViewContext) -> Option<ABC> {
+    None
+}
+"#;


there's a precompiled module for this

bfops · 2026-01-27T22:36:28Z

crates/smoketests/tests/smoketests/views.rs

+----+-------
+ 2  | 2"#,
+    );
+}


we're missing several tests from the python smoketests/tests/views.py

bfops · 2026-01-27T22:37:14Z

crates/smoketests/tests/smoketests/servers.rs

+        output.contains("testnet.spacetimedb.com"),
+        "Expected host in output: {}",
+        output
+    );


nit: the python version checks the host and protocol

bfops · 2026-01-27T22:37:39Z

crates/smoketests/tests/smoketests/servers.rs

+        testnet_re.is_match(&servers),
+        "Expected testnet in server list: {}",
+        servers
+    );


nit: the python version also tests for localhost

bfops · 2026-01-27T22:39:10Z

crates/smoketests/tests/smoketests/servers.rs

+        output.contains("fingerprint") || output.contains("Fingerprint"),
+        "Expected fingerprint message: {}",
+        output
+    );


nit: the python version tests fingerprinting of ip address and the presence/absence of -y as well. that could maybe be a separate test.

…files

…ler/translate-smoketests' into tyler/translate-smoketests

crates/smoketests/tests/smoketests/new_user_flow.rs

Signed-off-by: Zeke Foppa <196249+bfops@users.noreply.github.com>

bfops · 2026-01-27T22:57:11Z

crates/smoketests/tests/smoketests/schedule_reducer.rs

+        invoked_count,
+        logs
+    );
+}


I think we're missing the procedure tests from The Original Python

bfops · 2026-01-27T22:58:07Z

crates/smoketests/tests/smoketests/schedule_reducer.rs

+        invoked_count, 1,
+        "Expected scheduled reducer to run exactly once, but it ran {} times. Logs: {:?}",
+        invoked_count, logs
+    );


The Original Python also has:

row_entry = { "prev": TIMESTAMP_ZERO, "scheduled_id": 2, "sched_at": {"Time": TIMESTAMP_ZERO}, } # subscription should have 2 updates, first for row insert in scheduled table and second for row deletion. self.assertEqual( sub(), [ {"scheduled_table": {"deletes": [], "inserts": [row_entry]}}, {"scheduled_table": {"deletes": [row_entry], "inserts": []}}, ], )

bfops · 2026-01-27T22:59:21Z

crates/smoketests/tests/smoketests/schedule_reducer.rs

+        invoked_count,
+        logs
+    );
+}


The Original Python also has this at the end:

# scheduling repeated reducer again just to get 2nd subscription update. self.call("schedule_reducer") repeated_row_entry = { "prev": TIMESTAMP_ZERO, "scheduled_id": 1, "sched_at": {"Interval": {"__time_duration_micros__": 100000}}, } row_entry = { "prev": TIMESTAMP_ZERO, "scheduled_id": 2, "sched_at": {"Time": TIMESTAMP_ZERO}, } # subscription should have 2 updates and should not have any deletes self.assertEqual( sub(), [ {"scheduled_table": {"deletes": [], "inserts": [repeated_row_entry]}}, {"scheduled_table": {"deletes": [], "inserts": [row_entry]}}, ], )

bfops · 2026-01-27T23:01:09Z

crates/smoketests/tests/smoketests/schedule_reducer.rs

+        result.contains("yay!") && result.contains("hello"),
+        "Expected both 'yay!' and 'hello' in table, got: {}",
+        result
+    );


the original code did a background subscription and then checked it after the calls. I'm not sure whether that was important to what was being tested (i.e. if there was a specific worry about volatile_nonatomic_schedule_immediate wasn't updating subscriptions properly)

bfops · 2026-01-27T23:07:37Z

crates/smoketests/tests/smoketests/auto_migration.rs

+        "Expected book ISBN in AFTER logs: {:?}",
+        logs
+    );
+}


I believe we're missing the AddTableColumns class equivalents from The Original Python

bfops · 2026-01-27T23:10:17Z

crates/smoketests/tests/smoketests/auto_migration.rs

+        "Expected Robert in logs: {:?}",
+        logs
+    );
+


we don't print the log messages that The Original Python prints, unsure if intentional

bfops · 2026-01-27T23:11:27Z

crates/smoketests/tests/smoketests/auto_migration.rs

+    test.publish_module_clear(false).unwrap();
+
+    // Add new data with updated schema
+    test.call("add_person", &["Husserl", "Student"]).unwrap();


The Original Python does this after this line:

# If subscription, we should get 4 rows corresponding to 4 reducer calls (including before and after update) sub = sub(); self.assertEqual(len(sub), 4)

bfops · 2026-01-27T23:19:25Z

crates/smoketests/tests/smoketests/pg_wire.rs

+#[test]
+fn test_failures() {
+    if !have_psql() {
+        eprintln!("Skipping test_failures: psql not available");


should this fail rather than quietly-ish skipping?

bfops · 2026-01-27T23:22:47Z

crates/smoketests/tests/smoketests/pg_wire.rs

+    );
+
+    // Connection fails with invalid token - we can't easily test this without
+    // modifying the token, so skip this part


this case seems worth testing

bfops · 2026-01-27T23:23:36Z

crates/smoketests/tests/smoketests/pg_wire.rs

+(1 row)"#,
+    );
+}
+


I think we're missing a test_sql_conn equivalent to The Original Python

bfops · 2026-01-27T23:25:23Z

crates/smoketests/tests/smoketests/csharp_module.rs

+#[test]
+fn test_build_csharp_module() {
+    if !have_dotnet() {
+        eprintln!("Skipping test_build_csharp_module: dotnet 8.0+ not available");


I would prefer if this failed instead of skipping, forcing the user to pass a --skip-dotnet or similar like The Original Python did. That way, we can tell the difference between the "tests not failing because not run" and "tests not failing because behavior is correct" cases.

bfops · 2026-01-27T23:26:33Z

crates/smoketests/tests/smoketests/csharp_module.rs

+    // CLI is pre-built by artifact dependencies during compilation
+    let cli_path = ensure_binaries_built();
+
+    // Install wasi-experimental workload


python did this before this step:

run_cmd("dotnet", "nuget", "locals", "all", "--clear", cwd=bindings, capture_stderr=True)

unsure if the change is intentional

bfops · 2026-01-27T23:33:31Z

crates/smoketests/src/modules.rs

+            .unwrap()
+            .strip_suffix(".wasm")
+            .unwrap()
+            .replace('_', "-");


nit: perhaps we filter out lines 87-98 into a function, and then call that function in the test?

bfops · 2026-01-27T23:33:54Z

crates/smoketests/src/modules.rs

+///
+/// This checks if the modules workspace target directory exists and contains
+/// at least one WASM file.
+pub fn precompiled_modules_available() -> bool {


is this used?

bfops · 2026-01-27T23:39:27Z

.github/workflows/ci.yml

+
+      - name: Fail if Python smoketests were modified
+        run: |
+          PYTHON_SMOKETEST_CHANGES=$(git diff --name-only origin/${{ github.base_ref }} HEAD -- 'smoketests/**.py')


heads up, I realized that this will start failing in this PR if anyone merges changes into master that touch the smoketests, even if you don't merge master into this PR. I don't know that that's a bad thing, just wanted to mention it before it's confusing.

if we wanted only-changed-in-this-PR logic, it would have to be:

MERGE_BASE=$(git merge-base origin/${{ github.base_ref }} HEAD) PYTHON_SMOKETEST_CHANGES="$(git diff --name-only $MERGE_BASE HEAD -- 'smoketests/**.py')"

also we probably want to make this check required just before merging - I suggest leaving this comment unresolved until then.

bfops · 2026-01-27T23:41:54Z

tools/xtask-smoketest/src/main.rs

+    for (key, _) in env::vars() {
+        let should_remove = (key.starts_with("CARGO") && key != "CARGO_HOME" && key != "CARGO_TARGET_DIR")
+            || key.starts_with("RUST")
+            || key == "__CARGO_FIX_YOLO";


bfops · 2026-01-27T23:43:41Z

tools/xtask-smoketest/src/main.rs

+        // Set remote server environment variable if specified
+        if let Some(ref server_url) = server {
+            cmd.env("SPACETIME_REMOTE_SERVER", server_url);
+        }


nit: this could be moved out of the if entirely

cloutiertyler added 5 commits January 22, 2026 21:25

cloutiertyler changed the title ~~Tyler/translate smoketests~~ Translate smoketests from Python to Rust Jan 23, 2026

cloutiertyler added 4 commits January 23, 2026 01:10

Add views and auto_migration smoketest translations

bea9069

Translate tests for: - views.rs: st_view_* system tables, namespace collisions, SQL views - auto_migration.rs: schema changes, add table migration

cloutiertyler force-pushed the tyler/translate-smoketests branch from bda7652 to bea9069 Compare January 23, 2026 07:06

cloutiertyler added 2 commits January 23, 2026 02:11

Add new_user_flow and servers smoketest translations

60cba8d

Translate tests for: - new_user_flow.rs: Basic publish/call/SQL workflow - servers.rs: Server add/list/edit commands

jdetter assigned cloutiertyler Jan 23, 2026

cargo fmt + don't block on lints for now

afb0a71

jdetter reviewed Jan 23, 2026

View reviewed changes

.github/workflows/ci.yml Outdated Show resolved Hide resolved

jdetter reviewed Jan 23, 2026

View reviewed changes

kim reviewed Jan 23, 2026

View reviewed changes

crates/smoketests/src/lib.rs Outdated Show resolved Hide resolved

crates/smoketests/tests/smoketests/add_remove_index.rs Outdated Show resolved Hide resolved

crates/guard/src/lib.rs Show resolved Hide resolved

cloutiertyler and others added 12 commits January 23, 2026 11:42

[tyler/translate-smoketests]: lints

dafa004

[tyler/translate-smoketests]: more lints

8b6506b

[tyler/translate-smoketests]: more lints

447413b

[tyler/translate-smoketests]: more lints

c18ff5c

[tyler/translate-smoketests]: update ci stuff

95308f2

[tyler/translate-smoketests]: fix build

ed2735e

[tyler/translate-smoketests]: CI fixes?

fdba9e5

[tyler/translate-smoketests]: ci

09b53de

[tyler/translate-smoketests]: windows CI

017a744

[tyler/translate-smoketests]: fix windows ci

8ebf8e6

bfops reviewed Jan 27, 2026

View reviewed changes

Merge branch 'master' into tyler/translate-smoketests

45498b6

Signed-off-by: Tyler Cloutier <cloutiertyler@users.noreply.github.com>

bfops reviewed Jan 27, 2026

View reviewed changes

bfops added 2 commits January 27, 2026 14:41

[tyler/translate-smoketests]: remove comments referencing deprecated …

7790028

…files

[tyler/translate-smoketests]: Merge remote-tracking branch 'origin/ty…

fe717bf

…ler/translate-smoketests' into tyler/translate-smoketests

bfops reviewed Jan 27, 2026

View reviewed changes

crates/smoketests/tests/smoketests/new_user_flow.rs Show resolved Hide resolved

add todo from @jdetter

1a4ebcc

Signed-off-by: Zeke Foppa <196249+bfops@users.noreply.github.com>

bfops reviewed Jan 27, 2026

View reviewed changes

+              ----+-------
+| 2"#,
+                  );
+              }

Translate smoketests from Python to Rust #4102

Are you sure you want to change the base?

Translate smoketests from Python to Rust #4102

Uh oh!

Conversation

cloutiertyler commented Jan 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of Changes

Motivation

API and ABI breaking changes

Expected complexity level and risk

Testing

Uh oh!

Uh oh!

jdetter left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bfops Jan 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bfops Jan 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

cloutiertyler commented Jan 23, 2026 •

edited

Loading

bfops Jan 27, 2026 •

edited

Loading

bfops Jan 27, 2026 •

edited

Loading