Implement ljx tool

Cargo.toml

@@ -5,8 +5,8 @@ edition = "2024"
 license = "Apache-2.0"
 
 [workspace]
-members = ["logjetd", "demo"]
-default-members = [".", "logjetd", "demo"]
+members = [".", "logjetd", "demo", "ljx"]
+default-members = [".", "logjetd", "demo", "ljx"]
 
 [profile.release]
 lto = true
@@ -19,3 +19,7 @@ incremental = false
 [dependencies]
 crc32c = "0.6"
 lz4_flex = { version = "0.11", default-features = false, features = ["std"] }
+
+[dev-dependencies]
+opentelemetry-proto = { version = "0.28", features = ["gen-tonic", "logs"] }
+prost = "0.13"

Makefile

@@ -1,9 +1,11 @@
-.PHONY: build dev devel check fix test test-unit test-all test-integration setup clean stats arm-devel arm x86-devel x86 setup-arm setup-x86 demo man
+.PHONY: build dev devel check fix test test-unit test-integration setup clean stats arm-devel arm x86-devel x86 setup-arm setup-x86 demo man
 
 DEFAULT_TARGET := build
 ARM_TARGET ?= aarch64-unknown-linux-musl
 X86_TARGET ?= x86_64-unknown-linux-musl
 CORE_WORKSPACE := --workspace --exclude otlp-demo
+MANPAGE_MD := $(wildcard doc/manpage/*.1.md)
+MANPAGE_OUT := $(MANPAGE_MD:.md=)
 
 build: setup
 	cargo build $(CORE_WORKSPACE) --release
@@ -29,23 +31,26 @@ test: setup
 
 test-unit: setup
 	@if command -v cargo-nextest >/dev/null 2>&1; then \
-		cargo nextest run -p logjet --lib -p logjetd --bins; \
+		cargo nextest run -p logjet --lib -p logjetd --bins -p ljx --bin ljx; \
 	else \
 		echo "cargo-nextest not available, falling back to cargo test unit-only targets"; \
 		cargo test -p logjet --lib; \
 		cargo test -p logjetd --bin logjetd; \
+		cargo test -p ljx --bin ljx; \
 	fi
 
 test-integration: setup
+	cargo build -p logjetd -p ljx
+	cargo build -p otlp-demo --bin otlp-bofh-emitter
 	@if command -v cargo-nextest >/dev/null 2>&1; then \
 		cargo nextest run -p logjetd --test bridge_flows; \
+		cargo nextest run -p logjet --test ljx_cli; \
 	else \
-		echo "cargo-nextest not available, falling back to cargo test -p logjetd --test bridge_flows"; \
+		echo "cargo-nextest not available, falling back to cargo test integration targets"; \
 		cargo test -p logjetd --test bridge_flows; \
+		cargo test -p logjet --test ljx_cli; \
 	fi
 
-test-all: test
-
 arm-devel: setup setup-arm
 	cargo build $(CORE_WORKSPACE) --target $(ARM_TARGET)
 
@@ -61,10 +66,12 @@ x86: setup setup-x86
 demo: devel
 	cargo build -p otlp-demo
 
-man:
+man: $(MANPAGE_OUT)
+
+$(MANPAGE_OUT): doc/manpage/%.1: doc/manpage/%.1.md
 	@command -v pandoc >/dev/null 2>&1 || { echo "pandoc not found. Install pandoc to build manpages."; exit 1; }
 	@mkdir -p doc/manpage
-	pandoc --standalone --to man doc/manpage/logjetd.1.md -o doc/manpage/logjetd.1
+	pandoc --standalone --to man $< -o $@
 
 clean:
 	cargo clean

doc/README.md

@@ -1,9 +1,11 @@
 # Documentation
 
 - [overview.md](./overview.md): project overview
+- [ljx.md](./ljx.md): `ljx` offline CLI scope and command plan
 - [daemon.md](./daemon.md): `logjetd` behaviour and current limits
 - [configuration.md](./configuration.md): YAML keys and defaults
 - [features.md](./features.md): current implemented daemon features and use cases
 - [config-samples.md](./config-samples.md): ready-to-use config examples
 - [protocol.md](./protocol.md): current TCP wire framing used by `logjetd`
+- [manpage/ljx.1.md](./manpage/ljx.1.md): Markdown manpage source
 - [manpage/logjetd.1.md](./manpage/logjetd.1.md): Markdown manpage source

doc/ljx.md

@@ -0,0 +1,165 @@
+# `ljx`
+
+`ljx` is the offline command-line toolbox for `.logjet` files.
+
+It is separate from `logjetd` and must stay separate in purpose:
+
+- `logjet` is the Rust library and file format
+- `logjetd` is the daemon for ingest, transport, replay, and spool management
+- `ljx` is the standalone file tool for inspection and transformation
+
+`ljx` does not control the daemon and does not depend on daemon runtime state.
+It operates on `.logjet` files directly.
+
+## Design Goals
+
+The tool is intended to feel closer to `jq`, `parquet-tools`, or plumbing-style
+UNIX commands than to a daemon control plane.
+
+Core goals:
+
+- stream records instead of loading entire files into memory
+- preserve record ordering
+- operate on structured records, not raw bytes
+- support stdout where it makes sense
+- work cleanly in pipelines
+- keep errors direct and actionable
+
+## Current Status
+
+`ljx` is being introduced incrementally.
+
+Documented command set:
+
+- `count`
+- `filter`
+- `stats`
+- `cat`
+- `split`
+- `join`
+
+Current implementation status for release `0.1`:
+
+- implemented first: `count`
+- implemented first: `filter`
+- planned after that: `stats`, `cat`, `split`, `join`
+
+The CLI may already expose planned command names, but release `0.1` should only
+promise the commands that are actually complete and tested.
+
+## Input and Output Model
+
+`.logjet` files are read using the streaming reader from the `logjet` crate.
+That reader is sequential and corruption-tolerant, but the current API expects a
+`Read + Seek` source.
+
+That means:
+
+- normal file paths are the primary input mode
+- stdout output is straightforward for stream-producing commands
+- stdin support needs explicit policy because generic pipes are not seekable
+
+If stdin is unsupported for a given release, `ljx` should fail loudly and say
+why. If stdin is later supported by spooling to a temporary file, that behaviour
+should be documented as an explicit implementation choice.
+
+## Command Intent
+
+## `ljx count`
+
+Count records in one `.logjet` file, optionally subject to a record-aware
+predicate.
+
+Intended examples:
+
+```text
+ljx count telemetry.logjet
+ljx count telemetry.logjet --type logs
+ljx count telemetry.logjet --seq-min 1000 --seq-max 2000
+ljx count telemetry.logjet -F error -i
+ljx count telemetry.logjet -e 'java\..*\.bs'
+```
+
+Expected properties:
+
+- reads sequentially
+- preserves file order even though output is only a number
+- does not decode or inspect payload schema
+
+## `ljx filter`
+
+Write only matching records to another `.logjet` stream.
+
+Intended examples:
+
+```text
+ljx filter telemetry.logjet -o errors.logjet --type logs
+ljx filter telemetry.logjet -o - --ts-min 1700000000000000000 > tail.logjet
+ljx filter telemetry.logjet -o only-errors.logjet -F error -i
+ljx filter telemetry.logjet -o suspect.logjet -e 'java\..*\.bs'
+```
+
+Expected properties:
+
+- input order is preserved
+- output stays valid `.logjet`
+- matching is done per record, not by byte scanning the file
+
+Supported payload matching modes:
+
+- `-F`, `--fixed-string` for literal payload substring matching
+- `-e`, `--grep` for grep-style regex matching
+- `-i`, `--ignore-case` to make either payload matcher case-insensitive
+
+`ljx` uses one payload matcher at a time. `-F` and `-e` are mutually exclusive.
+
+## `ljx stats`
+
+Compute summary information for one file.
+
+Intended summary fields:
+
+- record count
+- byte size
+- timestamp range
+- optional per-type or per-field summaries
+
+## `ljx cat`
+
+Render records in a human-readable form suitable for terminal inspection.
+
+Open questions:
+
+- whether payload bytes should default to hex, escaped text, or a compact mixed format
+- how much payload to print before truncating
+
+## `ljx split`
+
+Split one `.logjet` input into multiple `.logjet` outputs.
+
+Target split modes:
+
+- by record count
+- by byte budget
+- by timestamp window, when the semantics are nailed down
+
+## `ljx join`
+
+Join multiple `.logjet` segments into one ordered output stream.
+
+Potential validation:
+
+- sequence continuity checks
+- timestamp monotonicity checks
+
+## Implementation Notes
+
+The simplest useful internal shape for `ljx` is:
+
+- a thin `clap` CLI layer
+- one module per subcommand
+- small shared helpers for input/output handling
+- small shared predicate parsing for record-aware matching
+
+This keeps the code close to the `logjet` reader and writer APIs and avoids
+inventing a second abstraction stack before the command surface is proven.