Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 8 additions & 4 deletions agent/skills/concurrency/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -89,15 +89,19 @@ public async Task ConcurrentTests_FindsRaceConditions()
spec.GetOperation<(string, string), ApiResult<Slot>>("BookSlot").With(("9am", "Bob"), "Bob books"),
};

var testCases = spec.GenerateConcurrentTests(
new BookingState(),
inputs,
new TestGenerationOptions { MaxDepth = 4 });

var context = spec.CreateTestingContext();
context.Register(new BookingApiClient(CreateHttpClient()));

var results = await spec.RunConcurrentTests(
var results = await spec.RunTests(
context,
new BookingState(),
inputs,
generationOptions: new TestGenerationOptions { MaxDepth = 4 },
executionOptions: new TestExecutionOptions
testCases,
new TestExecutionOptions
{
BeforeEachAsync = async _ => await ResetDatabase()
});
Expand Down
2 changes: 1 addition & 1 deletion agent/skills/operations/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ public class WithdrawOperation : Operation<WithdrawRequest, ApiResult<decimal>,

public override async Task<ApiResult<decimal>> ExecuteAsync(
WithdrawRequest request,
ITestContext context)
TestingContext context)
{
var client = context.Get<BankApiClient>();
return await client.WithdrawAsync(request.AccountId, request.Amount);
Expand Down
13 changes: 6 additions & 7 deletions agent/skills/quickref/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -186,12 +186,11 @@ Assert.That(results.All(r => r.Success));
## Concurrent Tests

```csharp
var results = await spec.RunConcurrentTests(
context,
initialState,
inputs,
generationOptions: new TestGenerationOptions { MaxDepth = 4 },
executionOptions: new TestExecutionOptions { ... });
var concurrentTestCases = spec.GenerateConcurrentTests(initialState, inputs,
new TestGenerationOptions { MaxDepth = 4 });

var results = await spec.RunTests(context, initialState, concurrentTestCases,
new TestExecutionOptions { ... });
```

## Manual Conformance Testing
Expand Down Expand Up @@ -220,7 +219,7 @@ File.WriteAllText("graph.dot", dot);
|--------|------|---------|-------------|
| `MaxDepth` | int | 5 | Max sequence length |
| `StateConstraint` | Func<State, bool> | null | Prune states |
| `MaxConcurrencyLevel` | int | 2 | Concurrent ops |
| `MaxConcurrencyLevel` | int | 3 | Concurrent ops |
| `SequentialTestCaseAlgorithm` | delegate | StateCoverage | Walk algorithm |

## Test Case Algorithms
Expand Down
9 changes: 4 additions & 5 deletions agent/skills/test-generation/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ var testCases = spec.GenerateTests(initialState, inputs, new TestGenerationOptio
|--------|-------------|---------|
| `MaxDepth` | Maximum sequence length | 5 |
| `StateConstraint` | Prune states during exploration | None |
| `MaxConcurrencyLevel` | For concurrent test generation | 2 |
| `MaxConcurrencyLevel` | For concurrent test generation | 3 |
| `SequentialTestCaseAlgorithm` | How paths are extracted | StateCoverage |

### Constraining the State Space
Expand Down Expand Up @@ -135,7 +135,7 @@ Assert.IsEmpty(failures, $"Failed: {failures.FirstOrDefault()?.LastFailureMessag
|--------|-------------|
| `BeforeEachAsync` | Reset system before each test case |
| `AfterEachAsync` | Cleanup after each test case |
| `LogLevel` | Control verbosity |
| `OnStepExecuted` | Log each operation's request/response for debugging |

## Resetting State

Expand Down Expand Up @@ -170,8 +170,7 @@ File.WriteAllText("state-graph.dot", dot);
### Custom Node Labels

```csharp
var dot = TestCaseGenerator.VisualizeStateSpace(
new TestingContext(spec),
var dot = spec.VisualizeStateSpace(
initialState,
inputs,
new TestGenerationOptions { MaxDepth = 3 },
Expand Down Expand Up @@ -259,5 +258,5 @@ public async Task AutoGeneratedTests()

## Next Steps

- **Concurrency Testing**: Test race conditions with `RunConcurrentTests`
- **Concurrency Testing**: Test race conditions with `GenerateConcurrentTests` and `RunTests`
- **Async Operations**: Handle background work with polling
12 changes: 9 additions & 3 deletions agent/skills/troubleshooting/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -264,7 +264,8 @@ var seqResults = await spec.RunTests(context, initialState, testCases);
Assert.That(seqResults.All(r => r.Success), "Fix sequential tests first!");

// Then run concurrent
var concResults = await spec.RunConcurrentTests(context, initialState, inputs, options);
var concTestCases = spec.GenerateConcurrentTests(initialState, inputs, options);
var concResults = await spec.RunTests(context, initialState, concTestCases);
```

### Non-Deterministic Failures
Expand All @@ -282,7 +283,8 @@ var concResults = await spec.RunConcurrentTests(context, initialState, inputs, o
for (int i = 0; i < 10; i++)
{
await ResetState();
var result = await spec.RunConcurrentTests(...);
var concTestCases = spec.GenerateConcurrentTests(initialState, inputs, options);
var result = await spec.RunTests(context, initialState, concTestCases);
Console.WriteLine($"Run {i}: {(result.All(r => r.Success) ? "PASS" : "FAIL")}");
}
```
Expand Down Expand Up @@ -342,7 +344,11 @@ Console.WriteLine($"Remaining possible states: {stateProfile.StatesAndStepFuncti
```csharp
var results = await spec.RunTests(context, initialState, testCases, new TestExecutionOptions
{
LogLevel = LogLevel.Verbose
OnStepExecuted = info =>
{
foreach (var (op, req, resp) in info.Operations)
Console.WriteLine($"{op.Name}({req}) → {resp}");
}
});
```

Expand Down
30 changes: 16 additions & 14 deletions agent/starter/ExampleTests.cs
Original file line number Diff line number Diff line change
Expand Up @@ -25,13 +25,7 @@ public async Task GeneratedSequentialTests()
.BindAsync<string, ApiResult>("Delete",
(client, key) => client.DeleteAsync(key));

// 3. Provide a way to get a fresh client and initial state
spec.ProvideTargetAndInitialState(() => (
new KvClient(/* your HttpClient or service reference */),
new KvState() // Empty — no items exist initially
));

// 4. Define inputs — Accordant explores all sequences of these
// 3. Define inputs — Accordant explores all sequences of these
var put = spec.GetOperation<PutRequest, ApiResult>("Put");
var get = spec.GetOperation<string, ApiResult<string>>("Get");
var delete = spec.GetOperation<string, ApiResult>("Delete");
Expand All @@ -45,14 +39,22 @@ public async Task GeneratedSequentialTests()
delete.With("key1", "Delete key1"),
};

// 5. Run tests
// 4. Generate test cases
var initialState = new KvState();
var testCases = spec.GenerateTests(initialState, inputs, new TestGenerationOptions
{
MaxDepth = 4 // Sequences up to 4 operations long
});

// 5. Create context and run tests
var context = spec.CreateTestingContext();
context.Register(new KvClient(/* your HttpClient or service reference */));

var results = await spec.RunTests(
inputs,
generationOptions: new TestGenerationOptions
{
MaxDepth = 4 // Sequences up to 4 operations long
},
executionOptions: new TestExecutionOptions
context,
initialState,
testCases,
new TestExecutionOptions
{
BeforeEachAsync = async ctx =>
{
Expand Down
4 changes: 2 additions & 2 deletions agent/starter/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ Use this as a reference when helping a user build their first spec. Adapt the pa
|------|---------|
| `ExampleState.cs` | The `[State]` class — minimal, just what operations need |
| `ExampleSpec.cs` | A few related operations with error cases and success cases |
| `ExampleTests.cs` | Wiring up execution, state reset, inputs, and running tests |
| `ExampleTests.cs` | Wiring up execution, generating tests, state reset, and running tests |

## Key Patterns Shown

Expand All @@ -26,4 +26,4 @@ Use this as a reference when helping a user build their first spec. Adapt the pa
- Execution bindings with `spec.ExecuteWith<T>().BindAsync(...)`
- `InputSet` with labeled inputs
- `BeforeEachAsync` for state reset
- `spec.RunTests(...)` to generate and execute
- `spec.GenerateTests(...)` and `spec.RunTests(...)` to generate and execute
4 changes: 2 additions & 2 deletions docs/concepts/conformance-testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -261,7 +261,7 @@ public async Task SequentialConformanceTests()
var testCases = spec.GenerateTests(initialState, inputs);
var results = await spec.RunTests(context, initialState, testCases);

Assert.IsTrue(results.All(r => r.Passed));
Assert.IsTrue(results.All(r => r.Success));
}

[TestMethod]
Expand All @@ -275,7 +275,7 @@ public async Task ConcurrentConformanceTests()
var testCases = spec.GenerateConcurrentTests(initialState, inputs);
var results = await spec.RunTests(context, initialState, testCases);

Assert.IsTrue(results.All(r => r.Passed));
Assert.IsTrue(results.All(r => r.Success));
}
```

Expand Down
3 changes: 2 additions & 1 deletion docs/concepts/models-vs-fakes.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,8 @@ public class KeyValueFake
### As a Model

```csharp
public class KeyValueState : JsonState
[State]
public partial class KeyValueState
{
public Dictionary<string, string> Store { get; set; } = new();
}
Expand Down
25 changes: 17 additions & 8 deletions docs/concepts/operations-and-expect.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,9 +11,14 @@ An **operation** represents a single, atomic action your system can perform —
Every operation has a name, a typed request, and a typed response. The key design decision: *specification* and *execution* are separate.

```csharp
// Define the spec (Apply): what SHOULD happen
spec.Operation<WithdrawRequest, WithdrawResponse>("Withdraw",
(request, state) => { ... }) // Apply: what SHOULD happen
.WithExecution(async (request, ctx) => { ... }); // Execute: what ACTUALLY happens
(request, state) => { ... });

// Bind execution: what ACTUALLY happens
spec.ExecuteWith<BankApiClient>()
.BindAsync<WithdrawRequest, WithdrawResponse>("Withdraw",
(client, request) => client.WithdrawAsync(request));
```

The **Apply** method describes valid behavior. Given this request and this state, what should the response look like? How should the state change? Apply doesn't touch the real system — it's pure logic, a specification of correctness.
Expand Down Expand Up @@ -187,16 +192,18 @@ So far we've talked about Apply — the specification side. But at some point, y
When you define operations as classes, you override both Apply and ExecuteAsync:

```csharp
public class WithdrawOperation : Operation<WithdrawRequest, decimal>
public class WithdrawOperation : Operation<WithdrawRequest, decimal, BankState>
{
public WithdrawOperation() : base("Withdraw") { }

public override ExpectedOutcomes Apply(WithdrawRequest request, BankState state)
{
// ... spec logic ...
}

public override async Task<decimal> ExecuteAsync(
WithdrawRequest request,
ITestContext context)
TestingContext context,
WithdrawRequest request)
{
var client = context.Get<HttpClient>();
var response = await client.PostAsync($"/accounts/{request.Id}/withdraw", ...);
Expand All @@ -212,9 +219,11 @@ Apply says what *should* happen. ExecuteAsync makes it *actually* happen.
For simpler specs, you can define everything inline:

```csharp
spec.Operation<GetBalanceRequest, GetBalanceResponse>("GetBalance", (request, state) => { ... })
.WithExecution(async (request, ctx) =>
await ctx.Get<BankClient>().GetBalanceAsync(request.AccountId));
spec.Operation<GetBalanceRequest, GetBalanceResponse>("GetBalance", (request, state) => { ... });

spec.ExecuteWith<BankClient>()
.BindAsync<GetBalanceRequest, GetBalanceResponse>("GetBalance",
(client, request) => client.GetBalanceAsync(request.AccountId));
```

Same separation, more compact syntax.
Expand Down
4 changes: 2 additions & 2 deletions docs/how-to/indefinite-failures.md
Original file line number Diff line number Diff line change
Expand Up @@ -133,8 +133,8 @@ spec.Operation<string, ApiResult<Job>>("CreateJob", (jobName, state) =>
return Expect.OneOf(
// Success - server created the job and assigned an ID
Expect.That<ApiResult<Job>>(r => r.IsSuccess && !string.IsNullOrEmpty(r.Data.JobId))
.ThenState<AppState, ApiResult<Job>>(
(response, next) =>
.ThenState<AppState>(
(ApiResult<Job> response, AppState next) =>
next.Jobs[response.Data.JobId] = new JobState
{
Name = jobName,
Expand Down
39 changes: 21 additions & 18 deletions docs/tutorials/01-your-first-spec.md
Original file line number Diff line number Diff line change
Expand Up @@ -277,12 +277,7 @@ public async Task SequentialTests_UsersAndTodos()
{
using var factory = new TodoServiceFactory(); // Starts the API
var spec = CreateSpec();

// Tell the spec how to get a client and initial state
spec.ProvideTargetAndInitialState(() => (
new TodoApiClient(factory.CreateTestClient()),
new AppState() // Empty state
));
var client = new TodoApiClient(factory.CreateTestClient());

// Get operation references for building inputs
var createUser = spec.GetOperation<User, ApiResult<User>>("CreateUser");
Expand All @@ -300,21 +295,27 @@ public async Task SequentialTests_UsersAndTodos()
getTodo.With(("alice", "todo-1"), "Get todo"),
};

// Generate test sequences from inputs
var initialState = new AppState();
var testCases = spec.GenerateTests(initialState, inputs, new TestGenerationOptions
{
MaxDepth = 4 // Sequences up to length 4
});

// Create a testing context and register the client
var context = spec.CreateTestingContext();
context.Register(client);

// Run the tests
var results = await spec.RunTests(
inputs,
generationOptions: new TestGenerationOptions
{
MaxDepth = 4 // Sequences up to length 4
},
executionOptions: new TestExecutionOptions
var results = await spec.RunTests(context, initialState, testCases,
new TestExecutionOptions
{
BeforeEachAsync = async ctx =>
BeforeEachAsync = async info =>
{
// Reset database before each test
var client = ctx.Context.Get<TodoApiClient>();
await client.DeleteUserAsync("alice");
await client.DeleteUserAsync("unknown");
var c = info.Context.Get<TodoApiClient>();
await c.DeleteUserAsync("alice");
await c.DeleteUserAsync("unknown");
}
});

Expand Down Expand Up @@ -375,7 +376,7 @@ You've learned the core Accordant workflow:
1. **Define State** - `[State]` partial class tracking what matters
2. **Define Operations** - `spec.Operation<TReq, TResp>(...)` with `Expect.That(...)`
3. **Bind Execution** - `spec.ExecuteWith<T>().BindAsync(...)`
4. **Configure & Run** - `ProvideTargetAndInitialState`, `InputSet`, `RunTests`
4. **Configure & Run** - `CreateTestingContext`, `Register`, `InputSet`, `GenerateTests`, `RunTests`

### Key Concepts

Expand All @@ -386,6 +387,8 @@ You've learned the core Accordant workflow:
| `.SameState()` | Operation doesn't change state |
| `.ThenState(newState)` | Operation transitions to new state |
| `InputSet` | Values to try—Accordant explores sequences |
| `CreateTestingContext` | Create context for DI and test execution |
| `context.Register(client)` | Register your API client for use during tests |
| `MaxDepth` | Limit sequence length |
| `BeforeEachAsync` | Reset state before each test |

Expand Down
Loading
Loading