DeltaZulu.LocalStream

A local, durable, append-only stream log for .NET — Kafka/Pulsar/Flink-inspired, not Kafka-compatible. Topics, partitions, monotonic offsets, named subscriptions with durable checkpoints, replay, and policy-based retention, all without an external broker.

See docs/LOCAL_STREAM_ARCHITECTURE.md for the full architecture. DeltaZulu.DurableBuffer is a sibling durable queue primitive that can be used at service edges, but it is not a required dependency of this package.

Getting started

Clone the repository:

git clone https://github.com/DeltaZulu-OU/DeltaZulu.LocalStream.git

Build and test (requires the .NET 10 SDK):

dotnet build DeltaZulu.LocalStream.slnx
dotnet test DeltaZulu.LocalStream.slnx

Usage

var host = new LocalStreamHost(new LocalStreamOptions
{
    StoragePath = "./data/localstream",
    Topics =
    {
        new TopicOptions
        {
            Name = "agent.output",
            Partitions = 4,
            Retention = new RetentionOptions { MaxBytes = 20L * 1024 * 1024 * 1024, MaxAge = TimeSpan.FromDays(7) },
        },
    },
});
await host.StartAsync();

// Append once...
var producer = host.CreateProducer<AgentEvent>();
await producer.AppendAsync("agent.output", theEvent, new AppendOptions { PartitionKey = theEvent.Source });

// ...read many times, each subscription with its own durable offset.
var archive = host.CreateConsumer<AgentEvent>("archive");
await foreach (var record in archive.ReadAsync("agent.output"))
{
    await WriteToDataLakeAsync(record);
    await archive.CommitAsync(record.Position);
}

Subscriptions can also be declared in LocalStreamOptions.Subscriptions (with Required and StartPosition), observed via GetTopicMetrics/GetSubscriptionMetrics, and driven by processors:

// Runs under the durable subscription "processor.logcluster-sampler";
// output is appended before the input offset is committed.
await host.RunProcessorOnceAsync("logcluster-sampler", "agent.output", new LogClusterSampler());

Delivery is at-least-once; ordering is preserved within a partition; retention is independent of subscription progress (a lagging subscription enters OffsetExpired and must be reset). Sinks should use StreamRecord.EventId or stream coordinates for idempotency.