3. Core Component Requirements

A. C++ Ingestion & Execution Engine (Latency-Critical Path)

OS-Level Primitives

• Map input packet capture files (.pcap) or simulated exchange packet feeds into RAM via mmap() using 2MB or 1GB Linux Huge Pages (MAP_HUGE_2MB, MAP_POPULATE).
• Enforce hardware isolation by pinning execution threads to designated physical CPU cores via pthread_setaffinity_np().
• Elevate thread execution priority to real-time using SCHED_FIFO scheduler attributes.

Data Flow Mechanics

• Process raw binary headers via pointer offsets. Avoid deep object conversion.
• Reinterpret-cast raw buffer pointers directly into explicit, alignas(64) memory-aligned Plain Old Data (POD) structures matching the exact exchange byte layout.
• Pass parsed messages directly into the SIMD FIX parser and matching Limit Order Book (LOB).

B. Inter-Process Communication Layer (Shared Memory Ring Buffer)

Storage Allocation

• Provision a fast virtual file path inside /dev/shm using POSIX shm_open() and ftruncate().

Concurrency Model

• Implement an asymmetric SPSC (or MPSC) circular lock-free ring buffer directly within the mapped memory footprint.
• Use strict atomic fence ordering (std::memory_order_acquire / std::memory_order_release) to prevent CPU pipeline instruction reordering.
• Apply alignas(64) boundaries to read/write index variables to prevent L1/L2 cache line bouncing (false sharing).

C. Rust Logging Daemon & Data Engineering Pipeline (Throughput Path)

Shared Memory Access

• Initialize an asynchronous consumer thread mapped directly into the persistent C++ shared memory block via the nix crate or unsafe raw system mappings.

Serialization

• Read binary structures directly out of the memory segment and pack payloads into FlatBuffers schemas.

Event Pipeline

• Stream the FlatBuffers payloads concurrently into a designated Apache Kafka topic partition.

D. Persistence & Analytics Layer (Analytics Path)

Storage Matrix

• Connect a database writer client to ingest the streaming Kafka topics, chunking payloads into batches of 10,000+ items into a partitioned TimescaleDB instance.

Post-Trade Feedback Loop

• Implement a Python framework using pandas to extract logged execution states from TimescaleDB, performing automated transaction cost analysis (TCA) and tracking structural tracking error benchmarks.

PROJECT SPECIFICATION: HIGH-PERFORMANCE SYSTEM ARCHITECTURE SANDBOX## 1. Core Architectural Goals

• Objective: Design an end-to-end, ultra-low latency HFT ingestion and platform engineering system.
• Latency Strategy: Isolate the latency-critical execution path from throughput-heavy storage mechanics.
• Zero-Overhead Constraints: Eliminate kernel context switches, heap allocations during execution, thread scheduling migration, and data serialization copying.

---

2. Target Component Architecture

                           [ DATA STREAMING & STORAGE PATH ]
                                          │
                                          ▼

[ C++ INGESTION ENGINE ] ───────► [ LINUX SHARED MEMORY ] ───────► [ RUST DAEMON ] ├─ Thread Pinning ├─ /dev/shm Virtual File ├─ Lock-Free Reader ├─ Huge Pages mmap() ├─ Custom Cache-Aligned SPSC ├─ FlatBuffers Encode └─ SIMD FIX / Pointer-Cast └─ std::memory_order_release └─ Kafka Producer Sync

---

3. Core Component Requirements## A. The C++ Ingestion & Execution Engine (Latency-Critical Path)

• OS-Level Primitives:
• Map input packet capture files (.pcap) or simulated exchange packet feeds into RAM via mmap() utilizing 2MB or 1GB Linux Huge Pages (MAP_HUGE_2MB, MAP_POPULATE).
  • Enforce hardware isolation by pinning execution threads to designated physical CPU cores via pthread_setaffinity_np().
  • Elevate thread execution priority to real-time status using SCHED_FIFO scheduler attributes.
• Data Flow Mechanics:
• Process raw binary headers via pointer offsets. Avoid deep object conversion.
  • Reinterpret-cast the raw buffer pointers directly into explicit, alignas(64) memory-aligned Plain Old Data (POD) C++ structures matching the exact exchange specification byte layout.
  • Pass parsed messages directly into the existing SIMD FIX parser and matching Limit Order Book (LOB).

B. The Inter-Process Communication Layer (The Shared Memory Ring Buffer)

• Storage Allocation: Provision a fast virtual file path located inside /dev/shm utilizing standard Unix POSIX shm_open() and ftruncate() constraints.
• Concurrency Model:
• Code an asymmetric Single-Producer Single-Consumer (SPSC) or Multi-Producer Single-Consumer (MPSC) circular lock-free ring buffer directly within the mapped memory footprint.
  • Utilize strict atomic fence ordering operations (std::memory_order_acquire and std::memory_order_release) to prevent runtime CPU pipeline instruction reordering.
  • Explicitly apply alignas(64) boundaries to read and write index variables to actively prevent hardware L1/L2 cache line bouncing (false sharing).

C. The Rust Logging Daemon & Data Engineering Pipeline (Throughput Path)

• Shared Memory Access: Initialize an asynchronous consumer thread mapped directly into the persistent C++ shared memory block via the nix crate or unsafe raw system mappings.
• Zero-Copy Serialization: Read binary structures directly out of the memory segment and pack the payloads into highly optimized binary data objects using FlatBuffers schemas to bypass runtime field-by-field memory allocations.
• Event Pipeline: Stream the FlatBuffers payloads concurrently into a designated Apache Kafka topic partition.

D. The Persistence & Analytics Layer (Analytics Path)

• Storage Matrix: Connect an optimized database writer client to ingest the streaming Kafka event topics, chunking payloads into batches of 10,000+ items for raw transaction logging inside a partitioned TimescaleDB instance.
• Post-Trade Feedback Loop: Implement a Python framework leveraging pandas to extract the logged execution states from TimescaleDB, performing automated transaction cost analysis (TCA) and tracking structural tracking error benchmarks.

---

4. Resume-Ready Target Metrics to Validate

The completed implementation must be benchmarked using custom high-resolution microsecond/nanosecond timer utilities (std::chrono::high_resolution_clock) to report the following target metrics:

• 99th percentile processing latency across the C++ ingestion block (< 50 nanoseconds under peak simulation loads).
• Eliminating TLB cache misses during tight loops via verified Huge Page virtual memory mappings.
• Zero data copies between raw network byte representation and inside the lock-free shared memory tracking loop.

---

5. Instructions for Next LLM Session

1. Context Check: Review the programmer's existing implementation of the C++ Limit Order Book (LOB) and SIMD FIX Parser.
2. Phase 1 Focus: Generate the precise C++ Linux systems programming templates required to setup shm_open, enforce pthread_setaffinity_np, and build out the fixed-size atomic ring buffer struct configuration.
3. Phase 2 Focus: Generate the corresponding unsafe Rust architecture to bind directly to the specified C++ memory layout layout cleanly.

HFT-Platform

HFT-Platform

COOL SITE hftuniversity.com