Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
e857340
feat(ee): add Enterprise client scaffold and extensible source/format…
PabloPardoGarcia Jun 11, 2026
c567511
docs: update coverage badge
github-actions[bot] Jun 11, 2026
eb395b4
feat(ee): add DLQ list / reprocess / discard to the Enterprise client
PabloPardoGarcia Jun 11, 2026
9f18cae
feat(ee): surface 409 on reprocess as PipelineNotRunningError
PabloPardoGarcia Jun 12, 2026
b3c5f95
docs(readme): document the Enterprise client and DLQ message processing
PabloPardoGarcia Jun 12, 2026
f04afa8
feat(ee): add DLQ component filter, list pagination, and list_iter
PabloPardoGarcia Jun 15, 2026
2e7ed6d
feat(ee): validate DLQ list component filter client-side
PabloPardoGarcia Jun 15, 2026
4453cfd
Merge pull request #69 from glassflow/pablo/etl-1187-sdk-add-dlq-repr…
PabloPardoGarcia Jun 15, 2026
75f0908
docs: update coverage badge
github-actions[bot] Jun 15, 2026
5ee4afd
feat(models): add Avro and Protobuf format support to Kafka source (E…
PabloPardoGarcia Jun 16, 2026
5c69272
Merge pull request #70 from glassflow/pablo/etl-1197-sdk-add-support-…
PabloPardoGarcia Jun 16, 2026
bd99889
docs: update coverage badge
github-actions[bot] Jun 16, 2026
192b47e
Merge bd998894de6f4f3bd5bbb93143040c0d2954e4cc into 34f99448211151493…
PabloPardoGarcia Jun 16, 2026
f96cd39
chore: bump version to 4.1.0
github-actions[bot] Jun 16, 2026
74571aa
docs: update coverage badge
github-actions[bot] Jun 16, 2026
69359ab
feat(ee): add Pipeline.get_streams for NATS stream names (ETL-1188)
PabloPardoGarcia Jun 16, 2026
0ecd447
Merge pull request #71 from glassflow/pablo/etl-1188-sdk-add-method-t…
PabloPardoGarcia Jun 16, 2026
9c25d52
docs: update coverage badge
github-actions[bot] Jun 16, 2026
4521d54
refactor(models): unify Kafka avro/protobuf schema under schema.file
PabloPardoGarcia Jun 16, 2026
0da244d
docs: update coverage badge
github-actions[bot] Jun 16, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 40 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
<img src="https://github.com/glassflow/glassflow-python-sdk/workflows/Test/badge.svg?labelColor=&color=e69e3a">
</a>
<!-- Pytest Coverage Comment:Begin -->
<img src="https://img.shields.io/badge/coverage-94%25-brightgreen">
<img src="https://img.shields.io/badge/coverage-95%25-brightgreen">
<!-- Pytest Coverage Comment:End -->
</p>

Expand All @@ -31,6 +31,7 @@ A Python SDK for creating and managing data pipelines between Kafka and ClickHou
- Pipeline configuration via YAML or JSON
- Schema validation and configuration management
- Fine-grained resource control per pipeline component
- Enterprise Edition client (`glassflow.ee`) with DLQ reprocessing and discard

## Installation

Expand Down Expand Up @@ -157,6 +158,44 @@ client.delete_pipeline("my-pipeline-id")
pipeline.delete()
```

## Enterprise Edition

The GlassFlow Enterprise Edition adds capabilities on top of the Open Source engine. The SDK exposes them through a drop-in client that extends the Open Source one. Import `Client` from `glassflow.ee` instead of `glassflow.etl`:

```python
from glassflow.ee import Client

client = Client(host="your-glassflow-etl-url")
```

The Enterprise client does everything the Open Source client does, plus the Enterprise-only features below. Entitlement is enforced by the backend: calling an Enterprise-only operation against a backend that is not licensed for it raises `FeatureNotLicensedError`.

### DLQ message processing

When a pipeline component fails to process a message, that message lands in the pipeline's dead-letter queue (DLQ). On the Enterprise client, `pipeline.dlq` adds message management on top of the Open Source `state`, `consume`, and `purge`:

- `list(batch_size, cursor, component)`: non-destructive paginated read. Returns a page dict with `messages` (each carrying a stable `message_id`, `component`, `error`, `original_message`, and `received_at`), `has_more`, and `next_cursor`. Pass `component` to filter to a single component (`ingestor`, `join`, `sink`, `dedup`, `oltp-receiver`), and pass `next_cursor` back as `cursor` to page.
- `list_iter(batch_size, component)`: lazily iterate over every message, paging via the cursor for you. Yields individual messages, so you do not manage the cursor by hand.
- `reprocess(message_ids)` / `reprocess_all()`: move messages back into the pipeline input to be processed again.
- `discard(message_ids)` / `discard_all()`: permanently remove messages.

```python
pipeline = client.get_pipeline("my-pipeline-id")

# Inspect failed messages from the sink only (paged automatically)
ids = [m["message_id"] for m in pipeline.dlq.list_iter(component="sink")]

# Retry them after fixing the underlying issue
pipeline.dlq.reprocess(ids) # or pipeline.dlq.reprocess_all()

# Or drop the ones you do not want
pipeline.dlq.discard(["seq_200"]) # or pipeline.dlq.discard_all()
```

Reprocessing replays messages through the running pipeline, so the pipeline must be in the `Running` state. Calling `reprocess` on a stopped, terminated, or failed pipeline raises `PipelineNotRunningError`. Discard acts on the queue directly and works in any state.

`reprocess` and `discard` accept at most 1000 `message_id` values per call. For larger sets, use the `*_all` variants. See the [DLQ documentation](https://docs.glassflow.dev/configuration/dlq) for the full reference.

## Migrating from V2 to V3

Pipeline version `v2` has been removed. Use `Client.migrate_pipeline_v2_to_v3()` to convert an existing configuration automatically:
Expand Down
2 changes: 1 addition & 1 deletion VERSION
Original file line number Diff line number Diff line change
@@ -1 +1 @@
4.0.0
4.1.0
Loading