TinyWebServer（Personal Learning Program） 🚀

English | 中文 (简体)

---

🌟 Overview

TinyWebServer is a C++17 HTTP/1.1 learning project focused on Linux network programming.
It demonstrates:

• epoll ET (Edge-Triggered) non-blocking I/O
• single event loop + worker thread pool
• basic HTTP request parsing and response generation
• asynchronous logging via a bounded queue + background flush thread
• connection timeout cleanup via min-heap with active-connection refresh
• RAII-style resource management with modern C++

This project is useful for learning socket programming, event-driven server design, concurrency control, and HTTP basics, but the current implementation is not production-ready.

📊 Actual Performance (wrk Benchmark)

Tested on AMD Ryzen 7 6800H (8 cores / 16 threads), Linux 6.6 / WSL, Release build, serving index.html.

Connections	Threads	Requests/sec	Avg Latency	Throughput	Notes
100	8	~8,000	~3ms	~12 MB/s	Light load
512	8	~9,000	~12ms	~14 MB/s	Moderate
1024	8	~24,800	~20ms	~38 MB/s	Current peak observed
2048	8	~24,400	~55ms	~37 MB/s	Saturated
4096	12	~17,500	~38ms	~27 MB/s	Oversaturated

Important Notes About These Numbers

These are current implementation results, not the theoretical limit of epoll or C++.

Performance is strongly affected by several implementation choices in the current codebase:

• all socket read/write events are dispatched through the thread pool
• static files are read into userspace buffers and copied again into response buffers
• write path performs an extra full response copy before sending
• INFO logging still has non-trivial formatting cost under load
• a single main event loop serializes accept / epoll wait / timeout scanning / epoll mod operations

So ~25k QPS should be understood as:

the current measured ceiling of this implementation on this machine, not a generic ceiling of the architecture.

# Recommended wrk command for this server
wrk -t8 -c1024 -d20s --latency http://localhost:8080/

🚀 Quick Start

# Navigate to project directory
cd TinyWebServer

# Build with CMake (Release)
mkdir -p build && cd build
cmake .. -DCMAKE_BUILD_TYPE=Release
make -j$(nproc)

# Run server
./tinywebserver -p 8080 -t 4

# Test with curl
curl http://localhost:8080/
curl http://localhost:8080/index.html
curl http://localhost:8080/api.json

# Stress test (requires wrk)
wrk -t8 -c1024 -d20s http://localhost:8080/

🏗️ Architecture

Technology Stack

Layer	Technology	Description
Language	C++17	RAII, smart pointers, move semantics, templates
I/O Multiplexing	epoll (ET)	Edge-triggered, non-blocking socket I/O
Concurrency Model	Single event loop + thread pool	Main thread polls epoll, workers process connection tasks
Logging	AsyncLogger v2	Bounded queue + single backend flush thread + log rotation
Timeout	Min-Heap (std::push_heap)	Idle timeout tracking with active-connection timer refresh
Memory Safety	RAII, unique_ptr, shared_ptr	Automatic cleanup for major owned resources
Security	Path traversal prevention	weakly_canonical() + root prefix check
Build System	CMake 3.10+	Release/Debug/ASAN configurable
TLS (optional)	OpenSSL	#ifdef USE_TLS, compile-time opt-in

Current Event Flow

Main Thread
  epoll_wait()
      |
      +--> accept new connections
      |
      +--> EPOLLIN / EPOLLOUT
              |
              +--> submit task to ThreadPool
                        |
                        +--> Connection::handle_read()
                        |       -> read socket
                        |       -> parse request
                        |       -> build response
                        |       -> maybe write immediately
                        |
                        +--> Connection::handle_write()
                                -> send response

Architectural Characteristics

• Single reactor thread handles accept, epoll wait, timeout checks, and epoll interest updates.
• Per-connection CAS flag avoids duplicate worker execution on the same fd.
• GET static file path currently uses ifstream + std::string instead of sendfile.
• Keep-alive is supported, and read/write activity now refreshes timeout heap entries.

🔧 Core Components

1. Socket 🔌

• RAII wrapper around TCP socket fd
• Supports bind, listen, accept4, SO_REUSEADDR, SO_REUSEPORT
• Move-only ownership model

2. Epoller 📡

• Small wrapper around epoll_create1, epoll_ctl, epoll_wait
• Uses ET mode for client sockets
• Returns std::vector<epoll_event>

3. ThreadPool ⚙️

• Fixed worker threads
• std::queue<std::function<void()>> task queue
• submit() returns std::future
• Current implementation is unbounded, so bursts can grow memory usage

4. Connection 🔗

• Holds per-socket state:
  • read_buf_
  • write_buf_
  • parsed HttpRequest
  • keep-alive state
  • TLS state (optional)
• Uses task_pending_ to avoid concurrent duplicate processing of the same connection
• Current write path copies write_buf_ into a local buf_copy before sending, which adds overhead

5. HttpRequest 📨

• Parses request line, headers, and POST body
• Lowercases header keys
• Supports GET and POST handling in current server flow
• Rejects unsupported Transfer-Encoding
• Enforces:
  • max request line = 8192 bytes
  • max header region ≈ 64 KB
  • POST body limit = 1 MB

6. HttpResponse 📤

• Builds HTTP/1.1 status line and headers
• MIME type lookup by extension
• Supports keep-alive header
• Generates HTML error pages

7. Server 🖥️

• Owns listen socket, epoller, thread pool, connection map
• Main loop:
  • epoll_wait(10)
  • check_timeouts()
  • dispatch accept/read/write work
• Uses a min-heap for timeout entries, but active connections are not fully reinserted/refreshed in the heap

8. AsyncLogger 📝

• Reworked as AsyncLogger v2
• Frontend formats each log line into a fixed-size LogMessage
• Logs are pushed into a bounded in-memory queue
• A single backend thread batches writes to file
• Supports log rotation and dropped-log counters when the queue is full
• Keeps the original LOG_INFO / LOG_WARN / LOG_ERROR call sites compatible

💡 Design Highlights

CAS-Based Task Deduplication

The server may observe both EPOLLIN and EPOLLOUT around the same connection.
To avoid multiple workers running on the same Connection, it uses a per-connection atomic guard:

bool expected = false;
if (!task_pending_.compare_exchange_strong(expected, true)) {
    return;
}

This solves one class of race, but it also means some events are skipped and must wait for later progress, which can hurt tail latency in ET mode.

Edge-Triggered I/O

Client sockets are used in ET mode, so read/write handlers attempt to drain until EAGAIN.

while (true) {
    ssize_t n = ::read(fd, buf, sizeof(buf));
    if (n > 0) {
        read_buf_.append(buf, n);
    } else if (n == 0) {
        return false;
    } else if (errno == EAGAIN) {
        break;
    } else {
        return false;
    }
}

Static File Serving Path

Current GET serving path is roughly:

URI
 -> resolve_safe_path()
 -> open file with ifstream
 -> read full file into std::string
 -> append body to response buffer
 -> copy response buffer again before write()

This is simple and easy to understand, but it is one of the major reasons the server stops scaling near ~25k QPS.

🛠️ Building

CMake Build Options

# Release build
mkdir -p build && cd build
cmake .. -DCMAKE_BUILD_TYPE=Release
make -j$(nproc)

# Debug build
cmake .. -DCMAKE_BUILD_TYPE=Debug
make -j$(nproc)

# With AddressSanitizer
cmake .. -DCMAKE_BUILD_TYPE=Debug -DENABLE_ASAN=ON
make -j$(nproc)

# With TLS/HTTPS support
cmake .. -DCMAKE_BUILD_TYPE=Release -DUSE_TLS=ON
make -j$(nproc)

Run with Custom Options

# See all available options
./build/tinywebserver --help

# Custom port
./build/tinywebserver -p 3000

# Custom thread count
./build/tinywebserver -p 8080 -t 8

# Custom timeout (seconds)
./build/tinywebserver -p 8080 -T 60

# Custom www root
./build/tinywebserver -p 8080 -r /var/www/html

# Enable TLS (requires OpenSSL + build with -DUSE_TLS=ON)
./build/tinywebserver -p 443 -s -c /path/to/cert.pem -k /path/to/key.pem

📁 Project Structure

TinyWebServer/
├── CMakeLists.txt
├── README.md
├── README.zh-CN.md
├── build/
├── src/
│   ├── main.cpp
│   ├── Socket.h / Socket.cpp
│   ├── Epoller.h / Epoller.cpp
│   ├── ThreadPool.h / ThreadPool.cpp
│   ├── Connection.h / Connection.cpp
│   ├── Server.h / Server.cpp
│   ├── HttpRequest.h / HttpRequest.cpp
│   ├── HttpResponse.h / HttpResponse.cpp
│   └── AsyncLogger.h / AsyncLogger.cpp
├── www/
│   ├── index.html
│   ├── style.css
│   └── api.json
└── test/
    ├── run.sh
    └── test_httprequest.cpp

🎯 API Endpoints

Endpoint	Method	Result	Description
/	GET	200	Serves index.html
/index.html	GET	200	Homepage
/style.css	GET	200	Stylesheet
/api.json	GET	200	JSON file
/any_file.ext	GET	200/404	Any file under www/
/nonexistent	GET	404	HTML error page
POST /api	POST	200	Echo body as plain text
POST /	POST	200	Returns default JSON if body is empty

✨ Modern C++ Features

Feature	Location	Purpose
RAII	Socket, Epoller, AsyncLogger	Automatic cleanup
unique_ptr	socket / epoller ownership	Exclusive ownership
shared_ptr	connection lifetime across workers	Shared lifetime
Move semantics	Socket, buffers	Cheap ownership transfer
Lambda + Thread	ThreadPool, AsyncLogger	Async execution
std::mutex / CV	pool, connection, logger	Synchronization
std::lock_guard	multiple classes	Scoped locking
std::optional	Socket::accept_fd()	Optional return
std::string_view	HttpRequest::parse()	Avoids one copy during parse entry
Perfect forwarding	ThreadPool::submit()	Generic task submission
Variadic args	AsyncLogger::log()	printf-style logging
std::atomic	running flag / task guard	Lock-free flags

🧪 Testing

# Automated test suite
cd TinyWebServer && bash test/run.sh

# Manual tests
curl -i http://localhost:8080/
curl -i http://localhost:8080/nonexist

# Stress test
wrk -t8 -c1024 -d20s --latency http://localhost:8080/

🔐 Security

Implemented defensive checks include:

• path traversal prevention via canonical path validation
• null-byte (%00) rejection during URL decode
• request size limits for request line and header area
• POST body size limit
• unsupported Transfer-Encoding rejection
• X-Content-Type-Options: nosniff
• SIGPIPE ignored to avoid process crash on broken pipe
• EPOLLRDHUP used to detect peer disconnect

These protections improve safety for learning experiments, but they do not make the server production hardened.

📈 Design Patterns

Pattern	Implementation	Benefit
Reactor	epoll + event loop	Non-blocking event dispatch
Thread Pool	producer-consumer task queue	Reusable worker threads
RAII	resource wrappers	Automatic cleanup
Singleton	AsyncLogger::instance()	One global logger
Bounded Queue	LogQueue + backend thread	Decouple producers from file I/O with bounded memory
CAS Guard	Connection::task_pending_	Prevent duplicate concurrent worker handling

🚨 Known Issues

These issues exist in the current codebase and directly affect performance and/or correctness.

#	Severity	Issue	Root Cause	Impact
1	🟠 High	Every socket event is dispatched to the thread pool	Even lightweight read/write work pays queue, lock, wakeup, and scheduling cost	QPS ceiling is lower and p99 latency is higher
2	🟠 High	Static file path performs multiple memory copies	ifstream reads file into std::string, then appends into write_buf_, then write path copies again into buf_copy	Throughput waste, extra CPU, worse cache locality
3	🟠 High	Thread pool queue is unbounded	std::queue has no capacity limit or backpressure	Burst traffic can grow memory without bound
4	🟡 Medium	INFO logging still has noticeable frontend formatting cost	Each log call still performs timestamp formatting and vsnprintf before queueing	Benchmark performance drops if verbose INFO logs stay enabled
5	🟡 Medium	check_timeouts() runs every loop with fixed 10ms wakeup	Main loop wakes frequently even when idle	Unnecessary CPU overhead
6	🟡 Medium	Connection map is protected by one global mutex	accept/read/write/timeout paths all contend on connections_mutex_	Adds contention under load

🚨 Limitations & Future Work

Current limitations

• single process
• single main event loop
• static file serving only
• no HTTP pipelining
• no chunked transfer support
• no sendfile / writev static file fast path
• TLS is compile-time optional and not performance-tuned

Good next improvements

• replace full-buffer static file path with sendfile
• remove extra response copy in write path
• reduce or disable verbose INFO logs during benchmark
• redesign event model (e.g. main-reactor/sub-reactor or one-loop-per-thread)
• add bounded queue / overload protection for thread pool
• further optimize AsyncLogger v2 (e.g. cached timestamp / lighter frontend formatting)
• add file cache for hot static assets

🎓 Learning Outcomes

By studying this project, you can practice:

1. Linux socket programming with non-blocking I/O
2. epoll ET event handling
3. HTTP/1.1 request parsing and response formatting
4. thread pool design and synchronization
5. RAII and modern C++ ownership models
6. debugging and benchmarking with curl, wrk, ASAN, and gdb
7. performance analysis: identifying architectural bottlenecks beyond “just faster code”

🙏 References

• Muduo — Reactor-based C++ network library
• libevent — Event notification library
• Boost.Asio — Cross-platform async I/O
• epoll(7) man page
• wrk

📄 License

MIT License — see LICENSE

---

Author: lildengzi
Version: 1.0.0
Language: C++17
Platform: Linux (x86_64)
Note: This repository is primarily a learning-oriented web server implementation.