Add webhook callbacks triggered by suite execution events

[snytkine]

Summary

Allow a test suite YAML to declare a list of callbacks — outbound HTTP POST requests that are fired automatically when specific execution events occur (e.g. suite started, suite completed). Each callback carries a structured payload defined by the event type. This is a high-level feature request; payload schemas, the exact event set, and some implementation details are still to be determined.

Dependencies: Issue #7 (Basic Auth) should be merged first; callback definitions will share the auth block introduced there.

---

YAML definition

Callbacks are declared at the top level of the test-suite YAML, alongside rest_client, variables, and tests:

callbacks:
  - id: "on-suite-start"
    url: "https://hooks.example.com/api-tester/started"
    events:
      - suite_started
    headers:
      X-Api-Key: "[[${env.HOOKS_API_KEY}]]"
    auth:
      type: basic
      username: "[[${env.CB_USER}]]"
      password: "[[${env.CB_PASS}]]"

  - id: "on-suite-end"
    url: "https://hooks.example.com/api-tester/completed"
    events:
      - suite_completed
    headers:
      Content-Type: "application/json"

Callback definition fields

Field	Required	Notes
id	yes	Unique string identifier within the suite; used in logs and error messages
url	yes	POST target; supports Thymeleaf expressions
events	yes	Non-empty list of event names that trigger this callback
headers	no	Per-callback request headers; values support Thymeleaf expressions
auth	no	Same RequestAuth structure introduced in issue #7

All callbacks are HTTP POST. The request body is always a JSON payload determined by the event type (see Payloads below), except for REPORT_GENERATED which posts a multipart/form-data request with the HTML report as a file attachment (see below).

Multiple callbacks may listen to the same event. Each is invoked independently.

---

New model: CallbackDefinition

Add a record in the model package:

/**
 * A single webhook callback declared in a test-suite YAML.
 * Each callback fires as an HTTP POST to {@code url} when any of the
 * listed {@code events} is raised during suite execution.
 *
 * @param id      unique identifier within the suite
 * @param url     POST target URL; supports Thymeleaf template expressions
 * @param events  non-empty list of event names that trigger this callback
 * @param headers optional per-callback HTTP headers; values support Thymeleaf expressions
 * @param auth    optional authentication (shares {@link RequestAuth} from issue #7)
 */
public record CallbackDefinition(
    String id,
    String url,
    List<CallbackEvent> events,
    @Nullable Map<String, String> headers,
    @Nullable RequestAuth auth
) {}

CallbackEvent enum

public enum CallbackEvent {
    SUITE_STARTED,     // fired once before any tests run
    SUITE_COMPLETED,   // fired once after all tests finish
    TEST_STARTED,      // fired before each test's HTTP request
    TEST_COMPLETED,    // fired after each test's assertions
    VALIDATION_FAILED, // fired when pre-execution validation fails
    REPORT_GENERATED   // fired after the HTML report file has been written to disk
}

These align directly with the existing TestProgressEvent sealed-interface variants (SuiteStarted, TestCompleted, etc.) already fired by PureJavaTestEngine. The final enum values and whether all variants are supported in v1 is TBD.

---

Payloads (to be determined)

Each event type delivers a different payload. Exact schemas are TBD; the guiding principle is:

Event	Payload type	Likely content
SUITE_STARTED	JSON	Suite name, total test count, start timestamp, resolved cli and env variables (sensitive values masked or excluded)
SUITE_COMPLETED	JSON	Suite name, passed/failed/skipped/error counts, total duration, end timestamp; possibly a List<TestCaseResult> summary
TEST_STARTED	JSON	Suite name, test name, test index
TEST_COMPLETED	JSON	Suite name, test name, result status, assertion counts, failure details
REPORT_GENERATED	multipart/form-data	The generated HTML report posted as a file attachment (see below)

Payload records should be defined in a model.callback sub-package and serialised to JSON with the existing Jackson ObjectMapper.

REPORT_GENERATED — HTML report as a form attachment

When a callback is subscribed to REPORT_GENERATED, the dispatcher fires after the HTML report has been successfully written to disk. Instead of a JSON body, the POST is sent as multipart/form-data with the report file attached:

• Content-Type: multipart/form-data
• Form field name: report (TBD — can be made configurable)
• File name: the report file name (e.g. test-suite_My_Suite_20260608.html)
• Part Content-Type: text/html; charset=UTF-8

This allows the receiving endpoint (e.g. a Slack incoming webhook, a CI notification service, or an internal portal) to store or display the report without needing filesystem access.

Note: REPORT_GENERATED is only fired when --report is passed on the command line and the report is successfully generated. If no report is produced, the event is never raised.

---

Dispatch mechanism

Existing event infrastructure

PureJavaTestEngine already fires a TestProgressEvent for every milestone (see event/TestProgressEvent.java). The current TestProgressListener functional interface receives all events on the execution thread.

The current engine accepts a single TestProgressListener. The recommended approach to add callback dispatch without changing the engine signature is a composite listener:

/**
 * Multiplexes a {@link TestProgressEvent} to an ordered list of delegates.
 * Thread-safe: delegates are called in order on the invoking thread.
 */
public final class CompositeProgressListener implements TestProgressListener {
    public CompositeProgressListener(List<TestProgressListener> delegates) { ... }

    @Override
    public void onProgress(TestProgressEvent event) {
        delegates.forEach(d -> d.onProgress(event));
    }
}

RunSuiteCommand then wraps the existing UI or no-op listener together with the new CallbackDispatcher in a CompositeProgressListener when the suite declares callbacks.

CallbackDispatcher

A new TestProgressListener implementation in the service package:

/**
 * Listens for {@link TestProgressEvent} and fires any {@link CallbackDefinition}
 * entries whose {@code events} list includes the received event type.
 *
 * Callback HTTP calls are made synchronously on the listener thread (or
 * optionally dispatched to a dedicated executor — TBD).
 */
@Service
public class CallbackDispatcher implements TestProgressListener {

    public CallbackDispatcher(/* RestClient or RequestFactory, ObjectMapper */) { ... }

    @Override
    public void onProgress(TestProgressEvent event) { ... }
}

Threading and error handling (TBD):

• Whether callback HTTP calls block the execution thread or are dispatched asynchronously to an executor is to be decided.
• Whether a failed callback POST (network error, non-2xx response) is silently logged, causes a warning, or fails the suite is to be decided.
• Retry behaviour is out of scope for v1.

---

TestSuite model change

Add an optional callbacks field to TestSuite:

@Nullable List<CallbackDefinition> callbacks();

JSON schema (test-suite-configuration-schema.json) gains a top-level callbacks array property whose items follow the CallbackDefinition shape.

---

Open questions

1. Final event set for v1 — Should TEST_STARTED and TEST_COMPLETED be supported immediately, or only suite-level events (SUITE_STARTED, SUITE_COMPLETED, REPORT_GENERATED) in the first version?
2. Payload schemas — Exact fields and whether sensitive variable values are included/masked.
3. Synchronous vs. asynchronous dispatch — Blocking the test thread for each callback adds latency; async introduces lifecycle complexity in a native binary.
4. Failure policy — Log-and-continue vs. abort-suite on callback HTTP errors.
5. Thymeleaf scope for callback URLs/headers — Which variable namespaces (cli, env, suite) are available when resolving callback expressions? Suite-level variables seem appropriate; per-test variables do not apply to suite-level events.
6. CompositeProgressListener — Should this live in event package as a general utility, or only be instantiated inline in RunSuiteCommand?
7. REPORT_GENERATED form field name — Should the multipart field name (report) be hardcoded or configurable per callback definition?

---

Checklist

This list will be refined once open questions are resolved.

• CallbackEvent enum added to event (or model) package with values: SUITE_STARTED, SUITE_COMPLETED, TEST_STARTED, TEST_COMPLETED, VALIDATION_FAILED, REPORT_GENERATED
• CallbackDefinition record added to model package
• TestSuite gains nullable callbacks() component
• Payload records defined in model.callback sub-package
• CompositeProgressListener added to event package
• CallbackDispatcher service added to service package
• RunSuiteCommand wires CallbackDispatcher into a CompositeProgressListener when suite has callbacks
• REPORT_GENERATED event fired from RunSuiteCommand after the HTML report is written; dispatcher posts report as multipart/form-data attachment
• Callback URL and header values processed through Thymeleaf before dispatch
• JSON schema updated for callbacks array
• GraalVM: any new payload types registered for Jackson serialisation
• Unit tests: callback fires on correct events; non-matching events are ignored; failed HTTP call is handled per policy; REPORT_GENERATED dispatches multipart POST with correct file attachment
• spotless:apply run; all tests pass