A scalable real-time log monitoring system designed to explore different backend architectures for streaming live log updates efficiently to multiple clients.
The project focuses on backend scalability, real-time communication, efficient file processing, event-driven systems, and low-latency streaming techniques commonly used in production monitoring systems.
Build a system to monitor a server log file in real time with the following requirements:
- Show the latest 10 log lines when a client connects
- Stream new log updates without page refresh
- Efficiently handle very large log files
- Support multiple concurrent clients
- Minimize CPU and disk usage
- Handle log rotation gracefully
Reads the entire file repeatedly for every request.
- Simple implementation
- High CPU and disk usage
- Extremely inefficient for large files
- Poor scalability
Example: A 2GB log file causes heavy repeated reads and high resource consumption.
Each client tracks file position and periodically polls for new updates.
- Reads only newly appended lines
- Better than brute force
- CPU usage increases linearly with number of clients
- Polling introduces latency
- Redundant reads for multiple clients
Example: 5 clients polling every 0.5 seconds results in 5 repeated reads.
Server polls the file once and pushes updates to all connected clients using Server-Sent Events (SSE).
- Single read shared across all clients
- Lower CPU usage
- Better scalability
- Still depends on polling interval
- Small latency remains
Example: All clients receive updates simultaneously within polling interval.
Uses OS-level file watchers to trigger updates only when the log file changes.
- Real-time streaming
- Minimal CPU usage
- Efficient for large files
- Highly scalable
- Supports multiple clients efficiently
- More complex implementation
Example: 100 clients receive instant updates from a 2GB log file with minimal resource usage.
The project progressively evolves through increasingly scalable approaches:
Brute Force → Polling → SSE → Event-Driven Streaming
Each iteration improves:
- scalability
- latency
- concurrent client handling
- resource utilization
Streams real-time updates from server → client over HTTP.
Tracks last read byte position to avoid re-reading entire files.
Periodically checks file for updates.
Uses event-driven mechanisms (e.g. inotify / watchdog) to detect file changes.
Maintains independent queues for connected clients.
Reduces redundant reads and minimizes CPU usage for concurrent clients.
The project explores several real-world backend engineering concerns:
- concurrent client management
- efficient file reading
- event-driven architectures
- low-latency streaming
- scalability under high load
- log rotation handling
- resource optimization
repo/
├─ README.md
├─ log_viewer_all_approaches.md
└─ log_viewer_implementations.py