Skip to content

Conversation

@clockwork-tien
Copy link
Contributor

@clockwork-tien clockwork-tien commented Jan 23, 2026

Description of Changes

  • Quickstart template tanstack-ts and client for TanStack Start
  • Add Vue Quickstart Docs

Screenshots

  • tanstack-ts template
image
  • TanStack Start Quickstart Docs
image

API and ABI breaking changes

Expected complexity level and risk

Testing

@clockwork-tien clockwork-tien force-pushed the tien/feat/tanstack-start branch 4 times, most recently from eadeaa6 to 0c1ca41 Compare January 26, 2026 17:20
@clockwork-tien
Copy link
Contributor Author

/update-llm-benchmark

@clockwork-labs-bot
Copy link
Collaborator

LLM Benchmark Results (ci-quickfix)

Language Mode Category Tests Passed Task Pass %
Rust rustdoc_json basics 20/27 68.8% ⬇️ -7.6%
Rust rustdoc_json schema 26/34 75.3%
Rust rustdoc_json total 46/61 71.7% ⬇️ -4.2%
Rust docs basics 4/21 9.0% ⬇️ -2.1%
Rust docs schema 8/34 20.5%
Rust docs total 12/55 14.2% ⬇️ -1.1%
C# docs basics 27/27 100.0%
C# docs schema 24/34 70.3% ⬆️ +6.7%
C# docs total 51/61 86.5% ⬆️ +3.0%

Compared against master branch baseline

Generated at: 2026-01-26T17:39:25.261Z

Failure Analysis (click to expand)

Benchmark Failure Analysis

Generated from: /__w/SpacetimeDB/SpacetimeDB/tools/xtask-llm-benchmark/../../docs/llms/docs-benchmark-details.json

Summary

  • Total failures analyzed: 35

Analysis of SpacetimeDB Benchmark Failures

Rust / rustdoc_json Failures

Compile/Publish Errors (2 failures)

t_002_scheduled_table

  • Generated Code:

    use spacetimedb::{table, reducer, ReducerContext, Table, UniqueColumn, ScheduleAt};
    
    #[table(name = tick_timer, schedule(reducer = tick, column = scheduled_at))]
    pub struct TickTimer {
        #[primary_key]
        #[auto_inc]
        scheduled_id: u64,
        scheduled_at: ScheduleAt,
    }
    
    #[reducer(init)]
    pub fn init(ctx: &ReducerContext) {
        if ctx.db.tick_timer().count() == 0 {
            ctx.db.tick_timer().insert(TickTimer {
                scheduled_id: 0,
                scheduled_at: ScheduleAt::in_micros(50_000),
            });
        }
    }
    
    #[reducer(scheduled)]
    pub fn tick(ctx: &ReducerContext, mut timer: TickTimer) {
        timer.scheduled_at = ScheduleAt::in_micros(50_000);
        ctx.db.tick_timer().scheduled_id().update(timer);
    }
  • Expected (Golden):

    use spacetimedb::{reducer, table, ReducerContext, ScheduleAt, Table};
    use std::time::Duration;
    
    #[table(name = tick_timer, scheduled(tick))]
    pub struct TickTimer {
        #[primary_key]
        #[auto_inc]
        scheduled_id: u64,
        scheduled_at: ScheduleAt,
    }
    
    #[reducer]
    pub fn tick(_ctx: &ReducerContext, _row: TickTimer) -> Result<(), String> {
        Ok(())
    }
    
    #[reducer(init)]
    pub fn init(ctx: &ReducerContext) -> Result<(), String> {
        ctx.db.tick_timer().insert(TickTimer {
            scheduled_id: 0,
            scheduled_at: ScheduleAt::Interval(Duration::from_millis(50).into()),
        });
        Ok(())
    }
  • Error: publish_error: spacetime publish failed (exit=1)

  • Explanation: The generated code incorrectly uses ScheduleAt::in_micros instead of the correct ScheduleAt::Interval(Duration::from_millis(...)). Additionally, the schedule parameter should not specify column.

  • Root Cause: The LLM misunderstood the function of ScheduleAt and how to specify the schedule.

  • Recommendation: Update documentation examples to clarify the expected format for using ScheduleAt and the table(schedule) syntax.


t_017_scheduled_columns

  • Generated Code:

    use spacetimedb::{table, reducer, ReducerContext, Table, ScheduleAt};
    
    #[table(name = tick_timer, schedule(reducer = tick, column = scheduled_at))]
    pub struct TickTimer {
        #[primary_key]
        #[auto_inc]
        scheduled_id: u64,
        scheduled_at: ScheduleAt,
    }
    
    #[reducer(init)]
    pub fn init(ctx: &ReducerContext) {
        ctx.db.tick_timer().insert(TickTimer {
            scheduled_id: 0,
            scheduled_at: ScheduleAt::repeat_micros(50_000),
        });
    }
    
    #[reducer]
    pub fn tick(_ctx: &ReducerContext, _row: TickTimer) {
    }
  • Expected (Golden):

    use spacetimedb::{reducer, table, ReducerContext, ScheduleAt, Table};
    use std::time::Duration;
    
    #[table(name = tick_timer, scheduled(tick))]
    pub struct TickTimer {
        #[primary_key]
        #[auto_inc]
        pub scheduled_id: u64,
        pub scheduled_at: ScheduleAt,
    }
    
    #[reducer]
    pub fn tick(_ctx: &ReducerContext, _schedule: TickTimer) {
    }
    
    #[reducer(init)]
    pub fn init(ctx: &ReducerContext) {
        let every_50ms: ScheduleAt = Duration::from_millis(50).into();
        ctx.db.tick_timer().insert(TickTimer {
            scheduled_id: 0,
            scheduled_at: every_50ms,
        });
    }
  • Error: publish_error: spacetime publish failed (exit=1)

  • Explanation: The LLM incorrectly uses ScheduleAt::repeat_micros(...) instead of a duration-based representation.

  • Root Cause: Insufficient understanding regarding the ScheduleAt types and misconfiguration of the table's schedule.

  • Recommendation: Provide documentation that explicitly demonstrates the correct use of duration in setting up scheduled tasks.


Other Failures (7 failures)

t_003_struct_in_table

  • Generated Code:

    use spacetimedb::{ReducerContext, SpacetimeType, Table};
    
    #[derive(SpacetimeType, Clone)]
    pub struct Position {
        x: i32,
        y: i32,
    }
    
    #[spacetimedb::table(name = entity)]
    pub struct Entity {
        #[primary_key]
        id: i32,
        pos: Position,
    }
    
    #[spacetimedb::reducer(init)]
    pub fn init(_ctx: &ReducerContext) {}
    
    #[spacetimedb::reducer]
    pub fn add_entity(ctx: &ReducerContext, id: i32, x: i32, y: i32) {
        let pos = Position { x, y };
        ctx.db.entity().insert(Entity { id, pos });
    }
    
    #[spacetimedb::reducer]
    pub fn move_entity(ctx: &ReducerContext, id: i32, x: i32, y: i32) {
        if let Some(mut e) = ctx.db.entity().id().find(id) {
            e.pos = Position { x, y };
            let _ = ctx.db.entity().id().update(e);
        }
    }
  • Expected (Golden):

    use spacetimedb::{table, SpacetimeType};
    
    #[derive(SpacetimeType, Clone, Debug)]
    pub struct Position {
        pub x: i32,
        pub y: i32,
    }
    
    #[table(name = entity)]
    pub struct Entity {
        #[primary_key]
        pub id: i32,
        pub pos: Position,
    }
  • Error: schema_parity: reducers differ - expected [], got ["add_entity()", "init()", "move_entity()"]

  • Explanation: The generated code included unnecessary reducers which contradicted the expected schema requiring only the table structure.

  • Root Cause: The LLM generated additional reducer functions when only a definition for the table was expected.

  • Recommendation: Add documentation examples highlighting simple table definitions without extraneous reducers.


t_004_insert

  • Generated Code:

    use spacetimedb::{table, reducer, ReducerContext, Table};
    
    #[table(name = user)]
    pub struct User {
        #[primary_key]
        id: i32,
        name: String,
        age: i32,
        active: bool,
    }
    
    #[reducer]
    pub fn insert_user(ctx: &ReducerContext, id: i32, name: String, age: i32, active: bool) {
        ctx.db.user().insert(User { id, name, age, active });
    }
  • Expected (Golden):

    use spacetimedb::{reducer, table, ReducerContext, Table};
    
    #[table(name = user)]
    pub struct User {
        #[primary_key]
        pub id: i32,
        pub name: String,
        pub age: i32,
        pub active: bool,
    }
    
    #[reducer]
    pub fn insert_user(ctx: &ReducerContext, id: i32, name: String, age: i32, active: bool) -> Result<(), String> {
        ctx.db.user().insert(User { id, name, age, active });
        Ok(())
    }
  • Error: data_parity_insert_user: spacetime sql failed ... no such table: user

  • Explanation: The generated code had a missing pub keyword for the id field, which may lead to visibility issues in a context where it is accessed.

  • Root Cause: The LLM did not apply visibility specifiers correctly.

  • Recommendation: Update the documentation to clarify the need for public access in struct fields when dealing with database entities.


t_006_delete

  • Generated Code:

    use spacetimedb::{table, reducer, ReducerContext, Table};
    
    #[table(name = user)]
    pub struct User {
        #[primary_key]
        id: i32,
        name: String,
        age: i32,
        active: bool,
    }
    
    #[reducer]
    pub fn delete_user(ctx: &ReducerContext, id: i32) {
        ctx.db.user().id().delete(&id);
    }
  • Expected (Golden):

    use spacetimedb::{reducer, table, ReducerContext};
    
    #[table(name = user)]
    pub struct User {
        #[primary_key]
        pub id: i32,
        pub name: String,
        pub age: i32,
        pub active: bool,
    }
    
    #[reducer]
    pub fn delete_user(ctx: &ReducerContext, id: i32) {
        ctx.db.user().id().delete(id);
    }
  • Error: seed_users_row: spacetime sql failed ... no such table: user

  • Explanation: The missing pub keyword in the User struct fields is likely the issue causing the visibility problem.

  • Root Cause: Similar visibility issue as with t_004_insert.

  • Recommendation: Reinforce documentation on the importance of ensuring all struct fields are properly public when needed.


t_007_crud

  • Generated Code:

    use spacetimedb::{ReducerContext, Table};
    
    #[spacetimedb::table(name = user)]
    pub struct User {
        #[primary_key]
        id: i32,
        name: String,
        age: i32,
        active: bool,
    }
    
    #[spacetimedb::reducer]
    pub fn crud(ctx: &ReducerContext) {
        let users = ctx.db.user();
    
        users.insert(User { id: 1, name: "Alice".to_string(), age: 30, active: true });
        users.insert(User { id: 2, name: "Bob".to_string(), age: 22, active: false });
    
        if let Some(mut u1) = users.id().find(1) {
            u1.name = "Alice2".to_string();
            u1.age = 31;
            u1.active = false;
            let _ = users.id().update(u1);
        }
    
        users.id().delete(&2);
    }
  • Expected (Golden):

    use spacetimedb::{reducer, table, ReducerContext, Table};
    
    #[table(name = user)]
    pub struct User {
        #[primary_key]
        pub id: i32,
        pub name: String,
        pub age: i32,
        pub active: bool,
    }
    
    #[reducer]
    pub fn crud(ctx: &ReducerContext) {
        ctx.db.user().insert(User { id: 1, name: "Alice".into(), age: 30, active: true });
        ctx.db.user().insert(User { id: 2, name: "Bob".into(), age: 22, active: false });
        ctx.db.user().id().update(User { id: 1, name: "Alice2".into(), age: 31, active: false });
        ctx.db.user().id().delete(2);
    }
  • Error: crud_row_id1_parity: spacetime sql failed ... no such table: user

  • Explanation: Similar visibility issues as previous failures continued to propagate.

  • Root Cause: Persisting issue with struct fields missing the pub keyword.

  • Recommendation: Address documentation on struct visibility and its implications for database access.


Summary of Recommendations:

  1. Clarify visibility in struct definitions - Provide clear examples demonstrating the need for pub keywords in struct fields meant to interact with the database.
  2. Strengthen examples for scheduled tasks - Emphasize the proper usage of types and syntax for ScheduleAt in scheduled table definitions.
  3. Improve clarity on reducer expectations - Offer guidance on when and how reducers should be used in relation to table definitions to avoid extraneous implementations.
  4. Update syntax rules and examples - Regularly review and revise documentation to ensure that API syntax and usage examples reflect current best practices and avoid common pitfalls.

@clockwork-tien clockwork-tien force-pushed the tien/feat/tanstack-start branch from bf75a8c to 7e7f578 Compare January 26, 2026 18:26
@clockwork-tien
Copy link
Contributor Author

/update-llm-benchmark

@clockwork-labs-bot
Copy link
Collaborator

LLM Benchmark Results (ci-quickfix)

Language Mode Category Tests Passed Task Pass %
Rust rustdoc_json basics 25/27 83.3% ⬆️ +6.9%
Rust rustdoc_json schema 23/34 65.3% ⬇️ -10.0%
Rust rustdoc_json total 48/61 75.2% ⬇️ -0.8%
Rust docs basics 5/27 11.1%
Rust docs schema 8/34 20.5%
Rust docs total 13/61 15.4%
C# docs basics 23/27 91.7% ⬇️ -8.3%
C# docs schema 25/34 73.7% ⬆️ +10.0%
C# docs total 48/61 83.5%

Compared against master branch baseline

Generated at: 2026-01-26T18:42:36.258Z

Failure Analysis (click to expand)

Benchmark Failure Analysis

Generated from: /__w/SpacetimeDB/SpacetimeDB/tools/xtask-llm-benchmark/../../docs/llms/docs-benchmark-details.json

Summary

  • Total failures analyzed: 35

Analysis of SpacetimeDB Benchmark Test Failures

Rust / rustdoc_json Failures

Compile/Publish Errors

t_002_scheduled_table & t_017_scheduled_columns

  1. LLM Output:

    #[table(name = tick_timer, schedule(reducer = tick, column = scheduled_at))]
    pub struct TickTimer {
        #[primary_key]
        #[auto_inc]
        scheduled_id: u64,
        scheduled_at: ScheduleAt,
    }
    
    #[reducer(init)]
    pub fn init(ctx: &ReducerContext) {
        ctx.db.tick_timer().insert(TickTimer {
            scheduled_id: 0,
            scheduled_at: ScheduleAt::RepeatMicros(50_000),
        });
    }
  2. Expected (golden):

    #[table(name = tick_timer, scheduled(tick))]
    pub struct TickTimer {
        #[primary_key]
        #[auto_inc]
        pub scheduled_id: u64,
        pub scheduled_at: ScheduleAt,
    }
    
    #[reducer(init)]
    pub fn init(ctx: &ReducerContext) {
        let every_50ms: ScheduleAt = Duration::from_millis(50).into();
        ctx.db.tick_timer().insert(TickTimer {
            scheduled_id: 0,
            scheduled_at: every_50ms,
        });
    }
  3. Error: publish_error: spacetime publish failed (exit=1)

  4. Explain the difference:

    • The struct attributes from the LLM code did not match the expected schedule(tick) syntax.
    • Missing pub on struct fields prevents the struct from being accessible outside its module.
  5. Root cause:

    • Documentation needs clarity on the correct syntax for scheduling and access modifiers for struct fields.
  6. Recommendation: Ensure code generation uses correct API syntax and includes access modifiers (e.g., pub) for external access.


Other Failures

t_003_struct_in_table - 0/1 tests passed

  1. LLM Output:

    #[derive(SpacetimeType, Clone)]
    pub struct Position {
        x: i32,
        y: i32,
    }
    
    #[table(name = entity)]
    pub struct Entity {
        #[primary_key]
        id: i32,
        pos: Position,
    }
  2. Expected (golden):

    #[derive(SpacetimeType, Clone, Debug)]
    pub struct Position {
        pub x: i32,
        pub y: i32,
    }
    
    #[table(name = entity)]
    pub struct Entity {
        #[primary_key]
        pub id: i32,
        pub pos: Position,
    }
  3. Error: schema_parity: reducers differ - expected [], got ["add_entity()", "init()", "move_entity()"]

  4. Explain the difference:

    • Fields in the Position struct are missing pub, which makes them inaccessible outside of the current module.
  5. Root cause:

    • Lack of instruction in the documentation for making struct fields public within types used in tables.
  6. Recommendation: Update documentation to emphasize the importance of public access on struct fields within SpacetimeType structs.


t_013_spacetime_sum_type - 2/3 tests passed

  1. LLM Output:

    #[table(name = result)]
    pub struct ResultRow {
        #[primary_key]
        id: i32,
        value: Shape,
    }
  2. Expected (golden):

    #[table(name = result)]
    pub struct ResultRow {
        #[primary_key]
        pub id: i32,
        pub value: Shape,
    }
  3. Error: sum_type_row_parity: spacetime sql failed: no such table: result.

  4. Explain the difference:

    • Missing pub attribute on the id field leading to its inaccessibility.
  5. Root cause:

    • Documentation does not clearly state the need for public access on all fields involved in database schema definitions.
  6. Recommendation: Amend documentation to specify that all fields in structs representing database rows must be public.


C# / Docs Failures

t_009_init - 0/4 tests passed

  1. LLM Output:

    [SpacetimeDB.Table(Name = "User", Public = true)]
    public partial struct User
    {
        [SpacetimeDB.PrimaryKey]
        public int Id;
        public string Name;
        public int Age;
        public bool Active;
    }
  2. Expected (golden):

    [Table(Name = "User")]
    public partial struct User
    {
        [PrimaryKey] public int Id;
        public string Name;
        public int Age;
        public bool Active;
    }
  3. Error: publish_error: spacetime build (csharp) failed (exit=1)

  4. Explain the difference:

    • The generated code uses Public = true unnecessarily, leading to syntax errors in C#.
  5. Root cause:

    • Documentation lacks clarity on attributes needed for C# struct declarations.
  6. Recommendation: Improve documentation to focus on attributes suitable for C# and rectify any possible misconceptions regarding their usage.


Conclusion

The key mistakes causing test failures often relate to incorrect accessibility of struct fields and improper syntax. Clearer guidelines in the SpacetimeDB documentation regarding the use of pub for Rust and attribute specifications for C# structs can significantly reduce these errors. All recommendations revolve around emphasizing the importance of public access for fields in structs that interact with the database schema.

@clockwork-tien clockwork-tien marked this pull request as ready for review January 26, 2026 18:51
@clockwork-tien clockwork-tien force-pushed the tien/feat/tanstack-start branch from 2219c54 to add45fb Compare January 26, 2026 19:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants