Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 16 additions & 16 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

12 changes: 6 additions & 6 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ fix: setup
cargo clippy $(CORE_WORKSPACE) --fix --all-targets --all-features --allow-dirty --allow-staged -- -D warnings

test: setup
cargo build -p logjetd -p ljx
cargo build -p ljd -p ljx
cargo build -p otlp-demo --bin otlp-bofh-emitter
@if command -v cargo-nextest >/dev/null 2>&1; then \
cargo nextest run $(CORE_WORKSPACE); \
Expand All @@ -33,23 +33,23 @@ test: setup

test-unit: setup
@if command -v cargo-nextest >/dev/null 2>&1; then \
cargo nextest run -p logjet --lib -p logjetd --bins -p ljx --bin ljx; \
cargo nextest run -p logjet --lib -p ljd --bins -p ljx --bin ljx; \
else \
echo "cargo-nextest not available, falling back to cargo test unit-only targets"; \
cargo test -p logjet --lib; \
cargo test -p logjetd --bin logjetd; \
cargo test -p ljd --bin ljd; \
cargo test -p ljx --bin ljx; \
fi

test-integration: setup
cargo build -p logjetd -p ljx
cargo build -p ljd -p ljx
cargo build -p otlp-demo --bin otlp-bofh-emitter
@if command -v cargo-nextest >/dev/null 2>&1; then \
cargo nextest run -p logjetd --test bridge_flows; \
cargo nextest run -p ljd --test bridge_flows; \
cargo nextest run -p logjet --test ljx_cli; \
else \
echo "cargo-nextest not available, falling back to cargo test integration targets"; \
cargo test -p logjetd --test bridge_flows; \
cargo test -p ljd --test bridge_flows; \
cargo test -p logjet --test ljx_cli; \
fi

Expand Down
12 changes: 6 additions & 6 deletions demo/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,19 +18,19 @@ This directory contains two binaries for a two-terminal OTLP demo over TCP.
It also contains scenario demos under subdirectories:

- [`logjet-file`](./logjet-file)
- OTLP/HTTP emitter into file-backed `logjetd`
- OTLP/HTTP emitter into file-backed `ljd`
- [`logjet-grpc-file`](./logjet-grpc-file)
- OTLP/gRPC emitter into file-backed `logjetd`
- OTLP/gRPC emitter into file-backed `ljd`
- [`kill-bill`](./kill-bill)
- cut a `.logjet` file down to its middle third and recover later good blocks
- [`memory-buffer`](./memory-buffer)
- kept-front-jar plus rotating-tail memory retention
- [`drain-once`](./drain-once)
- preserved startup messages are consumed on the first drain and do not appear on the second
- [`multi-emitter`](./multi-emitter)
- five emitters into one `logjetd`, then late replay into one collector
- five emitters into one `ljd`, then late replay into one collector
- [`multi-emitter-continuous`](./multi-emitter-continuous)
- five emitters running continuously into one `logjetd` and one live collector
- five emitters running continuously into one `ljd` and one live collector
- [`multi-client-behaviour`](./multi-client-behaviour)
- one replay client stalls while another keeps flowing
- [`replay-handoff`](./replay-handoff)
Expand All @@ -52,11 +52,11 @@ It also contains scenario demos under subdirectories:
- [`ingest-overload`](./ingest-overload)
- rate-limited ingest with operator-visible counters and severity-aware shedding
- [`remote-drain`](./remote-drain)
- appliance-side `logjetd` drained by a remote-side `logjetd bridge`
- appliance-side `ljd` drained by a remote-side `ljd bridge`
- [`remote-drain-tls`](./remote-drain-tls)
- same remote-drain topology, but with TLS and mutual TLS on the replay link
- [`secure-pipeline`](./secure-pipeline)
- HTTPS OTLP ingest into `logjetd`, then HTTPS collector export on replay
- HTTPS OTLP ingest into `ljd`, then HTTPS collector export on replay

## Enjoy It

Expand Down
4 changes: 2 additions & 2 deletions demo/backpressure/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ From this directory:

This starts:

1. appliance-side `logjetd`
1. appliance-side `ljd`
2. a fast continuous emitter

The emitter sends every 200 ms.
Expand Down Expand Up @@ -118,7 +118,7 @@ Expected result:
- the collector still prints slowly
- the bridge keeps forwarding whatever fits into the three-record queue
- newer records are dropped once the queue is full
- terminal output from `logjetd` shows dropped sequence numbers
- terminal output from `ljd` shows dropped sequence numbers

## Point of the Demo

Expand Down
14 changes: 7 additions & 7 deletions demo/backpressure/run-appliance.sh
Original file line number Diff line number Diff line change
Expand Up @@ -3,11 +3,11 @@ set -eu

SCRIPT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
TARGET_DIR="$SCRIPT_DIR/../../target/debug"
LOGJETD="$TARGET_DIR/logjetd"
LJD="$TARGET_DIR/ljd"
EMITTER="$TARGET_DIR/otlp-bofh-emitter"
CONFIG="$SCRIPT_DIR/appliance-logjetd.conf"

for bin in "$LOGJETD" "$EMITTER"; do
for bin in "$LJD" "$EMITTER"; do
if [ ! -x "$bin" ]; then
echo "missing $bin"
echo "build everything first with: make demo"
Expand All @@ -17,17 +17,17 @@ done

cd "$SCRIPT_DIR"

echo "starting appliance-side logjetd with config $CONFIG"
"$LOGJETD" --config "$CONFIG" &
LOGJETD_PID=$!
echo "starting appliance-side ljd with config $CONFIG"
"$LJD" --config "$CONFIG" &
LJD_PID=$!

cleanup() {
kill "${LOGJETD_PID:-}" 2>/dev/null || true
kill "${LJD_PID:-}" 2>/dev/null || true
}

trap cleanup EXIT INT TERM

sleep 1

echo "starting fast BOFH traffic toward appliance-side logjetd"
echo "starting fast BOFH traffic toward appliance-side ljd"
"$EMITTER" 127.0.0.1:4318 --service-name "backpressure-emitter" --interval-ms 200
6 changes: 3 additions & 3 deletions demo/backpressure/run-consumer-block.sh
Original file line number Diff line number Diff line change
Expand Up @@ -3,11 +3,11 @@ set -eu

SCRIPT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
TARGET_DIR="$SCRIPT_DIR/../../target/debug"
LOGJETD="$TARGET_DIR/logjetd"
LJD="$TARGET_DIR/ljd"
COLLECTOR="$TARGET_DIR/otlp-demo-collector"
CONFIG="$SCRIPT_DIR/consumer-block.conf"

for bin in "$LOGJETD" "$COLLECTOR"; do
for bin in "$LJD" "$COLLECTOR"; do
if [ ! -x "$bin" ]; then
echo "missing $bin"
echo "build everything first with: make demo"
Expand All @@ -30,4 +30,4 @@ trap cleanup EXIT INT TERM
sleep 1

echo "starting consumer-side bridge in block mode using $CONFIG"
"$LOGJETD" --config "$CONFIG" bridge
"$LJD" --config "$CONFIG" bridge
6 changes: 3 additions & 3 deletions demo/backpressure/run-consumer-disconnect.sh
Original file line number Diff line number Diff line change
Expand Up @@ -3,11 +3,11 @@ set -eu

SCRIPT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
TARGET_DIR="$SCRIPT_DIR/../../target/debug"
LOGJETD="$TARGET_DIR/logjetd"
LJD="$TARGET_DIR/ljd"
COLLECTOR="$TARGET_DIR/otlp-demo-collector"
CONFIG="$SCRIPT_DIR/consumer-disconnect.conf"

for bin in "$LOGJETD" "$COLLECTOR"; do
for bin in "$LJD" "$COLLECTOR"; do
if [ ! -x "$bin" ]; then
echo "missing $bin"
echo "build everything first with: make demo"
Expand All @@ -30,4 +30,4 @@ trap cleanup EXIT INT TERM
sleep 1

echo "starting consumer-side bridge in disconnect mode using $CONFIG"
"$LOGJETD" --config "$CONFIG" bridge
"$LJD" --config "$CONFIG" bridge
6 changes: 3 additions & 3 deletions demo/backpressure/run-consumer-drop-newest.sh
Original file line number Diff line number Diff line change
Expand Up @@ -3,11 +3,11 @@ set -eu

SCRIPT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
TARGET_DIR="$SCRIPT_DIR/../../target/debug"
LOGJETD="$TARGET_DIR/logjetd"
LJD="$TARGET_DIR/ljd"
COLLECTOR="$TARGET_DIR/otlp-demo-collector"
CONFIG="$SCRIPT_DIR/consumer-drop-newest.conf"

for bin in "$LOGJETD" "$COLLECTOR"; do
for bin in "$LJD" "$COLLECTOR"; do
if [ ! -x "$bin" ]; then
echo "missing $bin"
echo "build everything first with: make demo"
Expand All @@ -30,4 +30,4 @@ trap cleanup EXIT INT TERM
sleep 1

echo "starting consumer-side bridge in drop-newest mode using $CONFIG"
"$LOGJETD" --config "$CONFIG" bridge
"$LJD" --config "$CONFIG" bridge
10 changes: 5 additions & 5 deletions demo/bridge-resume/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
This demo shows the point of `upstream.state-file`.

The appliance side keeps producing logs. The consumer side can die and come
back. When it returns, `logjetd bridge` resumes from the last successfully
back. When it returns, `ljd bridge` resumes from the last successfully
forwarded sequence instead of replaying from zero.

That means:
Expand All @@ -21,13 +21,13 @@ appliance side. It is written on the downstream consumer side.

The split is:

- appliance-side `logjetd`
- appliance-side `ljd`
- receives OTLP
- keeps backlog in memory
- exposes a replay listener
- does not need to write a bridge checkpoint file

- consumer-side `logjetd bridge`
- consumer-side `ljd bridge`
- connects to the appliance replay listener
- forwards logs to the collector
- stores the last forwarded sequence in `upstream.state-file`
Expand All @@ -53,7 +53,7 @@ From this directory:

This starts:

1. appliance-side `logjetd`
1. appliance-side `ljd`
2. a dialogue-style message emitter

The appliance side uses in-memory retention only.
Expand All @@ -72,7 +72,7 @@ From this directory:
This starts:

1. the collector
2. consumer-side `logjetd bridge`
2. consumer-side `ljd bridge`

The consumer config contains:

Expand Down
14 changes: 7 additions & 7 deletions demo/bridge-resume/run-appliance.sh
Original file line number Diff line number Diff line change
Expand Up @@ -3,12 +3,12 @@ set -eu

SCRIPT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
TARGET_DIR="$SCRIPT_DIR/../../target/debug"
LOGJETD="$TARGET_DIR/logjetd"
LJD="$TARGET_DIR/ljd"
EMITTER="$TARGET_DIR/otlp-bofh-emitter"
CONFIG="$SCRIPT_DIR/appliance-logjetd.conf"
STATE_FILE="$SCRIPT_DIR/bridge.state"

for bin in "$LOGJETD" "$EMITTER"; do
for bin in "$LJD" "$EMITTER"; do
if [ ! -x "$bin" ]; then
echo "missing $bin"
echo "build everything first with: make demo"
Expand All @@ -20,12 +20,12 @@ cd "$SCRIPT_DIR"

rm -f "$STATE_FILE"

echo "starting appliance-side logjetd with config $CONFIG"
"$LOGJETD" --config "$CONFIG" &
LOGJETD_PID=$!
echo "starting appliance-side ljd with config $CONFIG"
"$LJD" --config "$CONFIG" &
LJD_PID=$!

cleanup() {
kill "${LOGJETD_PID:-}" 2>/dev/null || true
kill "${LJD_PID:-}" 2>/dev/null || true
}

trap cleanup EXIT INT TERM
Expand All @@ -45,7 +45,7 @@ dialogue_line() {
esac
}

echo "starting dialogue traffic toward appliance-side logjetd"
echo "starting dialogue traffic toward appliance-side ljd"

seq_no=1
while :; do
Expand Down
6 changes: 3 additions & 3 deletions demo/bridge-resume/run-consumer.sh
Original file line number Diff line number Diff line change
Expand Up @@ -3,11 +3,11 @@ set -eu

SCRIPT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
TARGET_DIR="$SCRIPT_DIR/../../target/debug"
LOGJETD="$TARGET_DIR/logjetd"
LJD="$TARGET_DIR/ljd"
COLLECTOR="$TARGET_DIR/otlp-demo-collector"
CONFIG="$SCRIPT_DIR/consumer-logjetd.conf"

for bin in "$LOGJETD" "$COLLECTOR"; do
for bin in "$LJD" "$COLLECTOR"; do
if [ ! -x "$bin" ]; then
echo "missing $bin"
echo "build everything first with: make demo"
Expand All @@ -30,4 +30,4 @@ trap cleanup EXIT INT TERM
sleep 1

echo "starting consumer-side bridge using $CONFIG"
"$LOGJETD" --config "$CONFIG" bridge
"$LJD" --config "$CONFIG" bridge
2 changes: 1 addition & 1 deletion demo/drain-once/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ From this directory:

This starts:

1. appliance-side `logjetd`
1. appliance-side `ljd`
2. three preserved startup messages:
- `BOOT MESSAGE #1`
- `BOOT MESSAGE #2`
Expand Down
14 changes: 7 additions & 7 deletions demo/drain-once/run-appliance.sh
Original file line number Diff line number Diff line change
Expand Up @@ -3,11 +3,11 @@ set -eu

SCRIPT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
TARGET_DIR="$SCRIPT_DIR/../../target/debug"
LOGJETD="$TARGET_DIR/logjetd"
LJD="$TARGET_DIR/ljd"
EMITTER="$TARGET_DIR/otlp-bofh-emitter"
CONFIG="$SCRIPT_DIR/appliance-logjetd.conf"

for bin in "$LOGJETD" "$EMITTER"; do
for bin in "$LJD" "$EMITTER"; do
if [ ! -x "$bin" ]; then
echo "missing $bin"
echo "build everything first with: make demo"
Expand All @@ -17,12 +17,12 @@ done

cd "$SCRIPT_DIR"

echo "starting appliance-side logjetd with config $CONFIG"
"$LOGJETD" --config "$CONFIG" &
LOGJETD_PID=$!
echo "starting appliance-side ljd with config $CONFIG"
"$LJD" --config "$CONFIG" &
LJD_PID=$!

cleanup() {
kill "${LOGJETD_PID:-}" 2>/dev/null || true
kill "${LJD_PID:-}" 2>/dev/null || true
}

trap cleanup EXIT INT TERM
Expand All @@ -39,5 +39,5 @@ for i in 4 5 6; do
"$EMITTER" 127.0.0.1:4318 --once --message "BOOT MESSAGE #$i"
done

echo "starting continuous BOFH traffic toward appliance-side logjetd"
echo "starting continuous BOFH traffic toward appliance-side ljd"
"$EMITTER" 127.0.0.1:4318 --interval-ms 700
Loading