|
| 1 | +# Signet Forge {#mainpage} |
| 2 | + |
| 3 | +**C++20 Parquet library with AI-native extensions — zero mandatory dependencies.** |
| 4 | + |
| 5 | +[GitHub Repository](https://github.com/SIGNETSTACK/signet-forge) |
| 6 | + |
| 7 | +--- |
| 8 | + |
| 9 | +## Features |
| 10 | + |
| 11 | +- **Standalone C++20 Parquet** — no Arrow dependency, header-mostly, pure stdlib core |
| 12 | +- **All standard encodings** — PLAIN, RLE, DELTA_BINARY_PACKED, BYTE_STREAM_SPLIT, RLE_DICTIONARY |
| 13 | +- **Compression** — bundled Snappy; optional ZSTD, LZ4, Gzip |
| 14 | +- **Parquet Modular Encryption** — AES-256-GCM/CTR column-level encryption (PME) |
| 15 | +- **Post-quantum cryptography** — Kyber-768 KEM + Dilithium-3 signatures + X25519 hybrid |
| 16 | +- **AI-native vector columns** — `FLOAT32_VECTOR(dim)` with INT8/INT4 quantization |
| 17 | +- **Zero-copy tensor bridge** — Parquet → ONNX OrtValue / Arrow / NumPy / DLPack |
| 18 | +- **Streaming WAL** — sub-microsecond append with mmap ring buffer |
| 19 | +- **Feature store** — point-in-time joins, versioned feature vectors, Parquet-backed |
| 20 | +- **MPMC event bus** — lock-free Vyukov ring + columnar batches |
| 21 | +- **Cryptographic audit chain** — SHA-256 hash chain across row groups |
| 22 | +- **Compliance reporters** — MiFID II RTS 24, EU AI Act Art. 12/13/19 |
| 23 | + |
| 24 | +--- |
| 25 | + |
| 26 | +## Modules |
| 27 | + |
| 28 | +| Module | Namespace | Headers | |
| 29 | +|--------|-----------|---------| |
| 30 | +| **Core** | `signet::forge` | schema.hpp, types.hpp, error.hpp, writer.hpp, reader.hpp | |
| 31 | +| **Encodings** | `signet::forge` | encoding/rle.hpp, encoding/delta.hpp, encoding/dictionary.hpp, encoding/byte_stream_split.hpp | |
| 32 | +| **Compression** | `signet::forge` | compression/codec.hpp, compression/snappy.hpp, compression/zstd.hpp, compression/lz4.hpp, compression/gzip.hpp | |
| 33 | +| **Crypto** | `signet::forge::crypto` | crypto/aes_core.hpp, crypto/aes_gcm.hpp, crypto/aes_ctr.hpp, crypto/pme.hpp, crypto/key_metadata.hpp, crypto/post_quantum.hpp | |
| 34 | +| **Bloom Filters** | `signet::forge` | bloom/split_block.hpp, bloom/xxhash.hpp | |
| 35 | +| **Indexes** | `signet::forge` | column_index.hpp, mmap_reader.hpp | |
| 36 | +| **AI Vectors** | `signet::forge` | ai/vector_type.hpp, ai/quantized_vector.hpp, ai/tensor_bridge.hpp | |
| 37 | +| **Streaming** | `signet::forge` | ai/wal.hpp, ai/wal_mapped_segment.hpp, ai/streaming_sink.hpp | |
| 38 | +| **Feature Store** | `signet::forge` | ai/feature_writer.hpp, ai/feature_reader.hpp | |
| 39 | +| **Event Bus** | `signet::forge` | ai/mpmc_ring.hpp, ai/column_batch.hpp, ai/event_bus.hpp | |
| 40 | +| **Audit & Compliance** | `signet::forge` | ai/audit_chain.hpp, ai/decision_log.hpp, ai/inference_log.hpp, ai/compliance/* | |
| 41 | +| **Interop** | `signet::forge` | interop/arrow_bridge.hpp, interop/onnx_bridge.hpp, interop/numpy_bridge.hpp | |
| 42 | + |
| 43 | +--- |
| 44 | + |
| 45 | +## Quick Start |
| 46 | + |
| 47 | +```cpp |
| 48 | +#include <signet/forge.hpp> |
| 49 | + |
| 50 | +using namespace signet::forge; |
| 51 | + |
| 52 | +// Define schema |
| 53 | +auto schema = Schema::build("example", |
| 54 | + Column<int64_t>{"id"}, |
| 55 | + Column<double>{"price"}, |
| 56 | + Column<std::string>{"symbol"}); |
| 57 | + |
| 58 | +// Write |
| 59 | +auto writer = ParquetWriter::open("data.parquet", schema).value(); |
| 60 | +int64_t ids[] = {1, 2, 3}; |
| 61 | +double prices[] = {100.5, 200.75, 300.0}; |
| 62 | +std::string symbols[] = {"AAPL", "GOOGL", "MSFT"}; |
| 63 | +(void)writer.write_column(0, ids, 3); |
| 64 | +(void)writer.write_column(1, prices, 3); |
| 65 | +(void)writer.write_column(2, symbols, 3); |
| 66 | +(void)writer.close(); |
| 67 | + |
| 68 | +// Read |
| 69 | +auto reader = ParquetReader::open("data.parquet").value(); |
| 70 | +auto col = reader.read_column<double>(0, 1).value(); // row group 0, column 1 |
| 71 | +``` |
| 72 | + |
| 73 | +--- |
| 74 | + |
| 75 | +## Build |
| 76 | + |
| 77 | +```cmake |
| 78 | +include(FetchContent) |
| 79 | +FetchContent_Declare(signet_forge |
| 80 | + GIT_REPOSITORY https://github.com/SIGNETSTACK/signet-forge.git |
| 81 | + GIT_TAG main) |
| 82 | +FetchContent_MakeAvailable(signet_forge) |
| 83 | +target_link_libraries(your_target PRIVATE signet::forge) |
| 84 | +``` |
| 85 | + |
| 86 | +--- |
| 87 | + |
| 88 | +## License |
| 89 | + |
| 90 | +- **Core library** (Phases 1–9): Apache License 2.0 |
| 91 | +- **AI Audit & Compliance tier**: Business Source License 1.1 |
0 commit comments