Multi-threaded Transaction Processor

A simple Python-based multi-threaded transaction processing layer built on top of an embedded key/value store. The project supports two concurrency control schemes:

• Optimistic Concurrency Control (OCC)
• Conservative Two-Phase Locking (2PL)

Transactions are defined via workload templates and executed in parallel worker threads. The system collects basic metrics (throughput, response time, aborts) and is intended as an educational prototype.

Prerequisites

Before getting started, ensure you have:

• Python 3.8+ (tested on 3.9)
• RocksDB installed on your system
• C compiler (for macOS: run xcode-select --install)
• pip package manager

Installation & Setup

1. Clone or Navigate to the Project

cd /path/to/Multi-threaded_Transaction_Processor

2. Install RocksDB

Install RocksDB using official Installation Guide: https://github.com/facebook/rocksdb/blob/main/INSTALL.md

3. Create a Virtual Environment (Recommended)

python3 -m venv venv
source venv/bin/activate

4. Install Dependencies

pip install --upgrade pip
pip install -r requirements.txt

This installs:

• rockstore==0.2.0 – embedded key/value store
• cffi, pycparser – transitive dependencies

Configuration

Edit config.py to customize your experiment:

NUM_THREADS = 4                             # Number of worker threads
CONTENTION_PROBABILITY = 0.3                # Probability of accessing hotset (0.0-1.0)
MODE = "CCC"                                # Concurrency control: "OCC" or "2PL" (or "CCC")
WORKLOAD_FILE = "workload1/workload1.txt"   # Path to workload template
INPUT_FILE = "workload1/input1.txt"         # Path to initial data

Configuration Parameters Explained

Parameter	Description	Example Values
NUM_THREADS	Number of parallel worker threads	2, 4, 8, 16
CONTENTION_PROBABILITY	Fraction of operations on hotset (high contention data)	0.1 (low), 0.5 (medium), 0.9 (high)
MODE	Concurrency control scheme	"OCC" (optimistic), "2PL" or "CCC" (pessimistic locking)
WORKLOAD_FILE	Template defining transaction operations	"workload1/workload1.txt", "workload2/workload2.txt"
INPUT_FILE	Initial keyspace population data	"workload1/input1.txt", "workload2/input2.txt"

Quick Configuration Examples

Low Contention Experiment:

NUM_THREADS = 8
CONTENTION_PROBABILITY = 0.1
MODE = "OCC"

High Contention 2PL Test:

NUM_THREADS = 4
CONTENTION_PROBABILITY = 0.8
MODE = "2PL"

Running the Project

Start the processor with:

python3 main.py

Expected Output

The program will:

1. Load initial data from INPUT_FILE
2. Parse the workload template from WORKLOAD_FILE
3. Execute transactions across all worker threads
4. Display per-transaction logs during execution
5. Print a summary at the end:

Throughput: 123.4
Average Response Time: 0.056
Aborts: 10

Architecture & Components

• main.py – bootstrapper: initializes DB, loads data, parses workload, picks concurrency manager, and starts the executor.
• storage/database.py – thread-safe wrapper around rockstore with JSON serialization.
• loader/data_loader.py – reads INPUT_FILE and populates the database.
• transaction/ – defines runtime transaction objects with fields used by OCC or 2PL.
• occ/occ_manager.py – optimistic concurrency controller with timestamp validation.
• ccc/twopl_manager.py – conservative two-phase locking manager with wait‑die policy.
• workload/workload_parser.py – converts text templates into lightweight objects.
• workload/workload_executor.py – worker pool and execution logic (OCC and 2PL variants).
• workload/exec_helpers.py – utility functions for building READ/WRITE closures.
• metrics/metrics.py – collects commit/abort counts and response times.

The executor uses a queue of templates and spawns a fixed number of worker threads; each worker creates a new runtime transaction object from the template, assigns keys according to the configured contention level, and then executes operations sequentially while interacting with the chosen concurrency manager. 2PL transactions acquire locks before accessing records and release them on commit/abort; OCC transactions buffer reads/writes and validate at commit.

Workloads

Workload files live under workload/ and use a simple format:

WORKLOAD
TRANSACTION 1 INPUTS: A_KEY, B_KEY
READ(A_KEY)
READ(B_KEY)
WRITE(A_KEY, {...})
WRITE(B_KEY, {...})
COMMIT
TRANSACTION 2 INPUTS: ...
...
END

Input files contain initial key insertions used to populate the database before execution.

Two example workloads are included:

• workload/workload1/workload1.txt with workload1/input1.txt
• workload/workload2/workload2.txt with workload2/input2.txt

Run either by pointing config.WORKLOAD_FILE/INPUT_FILE accordingly.

---

Happy experimenting! 🚀