Skip to content

Add DHT benchmark simulation and finger-table override#656

Draft
RyanKung wants to merge 14 commits into
masterfrom
codex/dht-benchmark-653
Draft

Add DHT benchmark simulation and finger-table override#656
RyanKung wants to merge 14 commits into
masterfrom
codex/dht-benchmark-653

Conversation

@RyanKung

@RyanKung RyanKung commented Jul 4, 2026

Copy link
Copy Markdown
Member

Summary

  • Add dht_network_sim for small-scale DHT evaluation using full native WebRTC nodes rather than the dummy transport.
  • Emit JSON-line webrtc_dht reports for stable, failed-node, and churn scenarios, including convergence, connected WebRTC edges, topology/full finger matches, lookup success/correctness, average hops, lookup timeouts, and lookup failures per 10k.
  • Emit JSON-line webrtc_transport reports for WebRTC data-channel throughput through Rings messages, including payload size, message count, send/receive elapsed time, and Mbps.
  • Thread dht_finger_table_size through native config, serialized processor config, and ProcessorBuilder, with --dht-finger-table-size and RINGS_DHT_FINGER_TABLE_SIZE runtime overrides.

Validation

  • CARGO_TARGET_DIR=/private/tmp/rings-dht-benchmark-653-target cargo check -p rings-node --no-default-features --features node --lib
  • CARGO_TARGET_DIR=/private/tmp/rings-dht-benchmark-653-target cargo check -p rings-core --bench dht_network_sim
  • CARGO_TARGET_DIR=/private/tmp/rings-dht-benchmark-653-target RINGS_DHT_BENCH_NODES=3 RINGS_DHT_BENCH_FINGER_TABLE_SIZES=4 RINGS_DHT_BENCH_FAILED_NODE_PCTS=0 RINGS_DHT_BENCH_CHURN_RATE_PCTS=0 RINGS_DHT_BENCH_MAX_ROUNDS=5 RINGS_DHT_BENCH_THROUGHPUT_MESSAGES=2 RINGS_DHT_BENCH_THROUGHPUT_PAYLOAD_BYTES=1024 RINGS_DHT_BENCH_WAIT_TIMEOUT_MS=30000 cargo bench -p rings-core --bench dht_network_sim
  • CARGO_TARGET_DIR=/private/tmp/rings-dht-benchmark-653-target cargo test -p rings-node --lib dht_finger_table_size
  • CARGO_TARGET_DIR=/private/tmp/rings-dht-benchmark-653-target cargo test -p rings-node --lib test_deserialization_with_missed_field
  • CARGO_TARGET_DIR=/private/tmp/rings-dht-benchmark-653-target cargo clippy -p rings-core --bench dht_network_sim -- -D warnings -D clippy::unwrap_used -D clippy::expect_used -D clippy::panic -D clippy::todo -D clippy::unimplemented -D clippy::indexing_slicing
  • CARGO_TARGET_DIR=/private/tmp/rings-dht-benchmark-653-target cargo clippy -p rings-node --no-default-features --features node --lib -- -D warnings -D clippy::unwrap_used -D clippy::expect_used -D clippy::panic -D clippy::todo -D clippy::unimplemented -D clippy::indexing_slicing
  • git diff --check

Fixes #653

@RyanKung

RyanKung commented Jul 4, 2026

Copy link
Copy Markdown
Member Author

Docker cluster benchmark update from the 16-node runs:

  • This PR now adds scripts/dht_docker_cluster_bench.py and passes RINGS_DHT_FINGER_TABLE_SIZE through the Docker cluster node configs.
  • Fresh 16-node ring cluster with RINGS_STABILIZE_INTERVAL=1 still does not match the paper's stable Chord assumptions after the initial ready window: successor_matches=0/16, predecessor_matches=4/16, correctness_rate=31.17%, avg_hops=1.018, lookup_failures_per_10k=6882.8.
  • Existing long-running 16-node Docker cluster is close to steady state: successor_matches=16/16, predecessor_matches=15/16, full_matches=14/16, correctness_rate=99.69%, avg_hops=1.036, lookup_failures_per_10k=31.25.
  • Interpretation: the Docker cluster is the right integration benchmark because it exercises real native nodes and WebRTC, but we need to report convergence/time-to-steady-state separately from steady-state lookup results. The long-running cluster is also highly connected, so it validates reachability more than it reproduces the Chord paper's multi-hop path length.
  • Transport smoke on the fresh cluster sent 4 x 1 KiB via sendBackendMessage in 163 ms send-side (0.20 Mbps). The old image used for that run did not expose listPeerMeasurements, so receive-side counters should be collected from a rebuilt PR image.

RyanKung commented Jul 4, 2026

Copy link
Copy Markdown
Member Author

Paper-facing review, scoped to whether this PR can support PR #655's evaluation claims.

This PR is useful for the paper, but it supports two different objects and the data should stay separated:

dummy_dht_sim          -> B_C(eta), the Chord-style lookup/churn baseline
Docker/WebRTC cluster  -> Report_Rings, implementation evidence for native nodes + WebRTC

The dummy simulator can back the formal B_C(eta) object because it reports stable / failed-node / churn scenarios with avg_hops, mean_lookup_timeouts, and lookup_failures_per_10k over deterministic Chord routing state. The Docker collector is more valuable for Rings evidence, but it should not be used as the Chord baseline: the current 16-node note says the fresh cluster is not at the stable Chord assumption yet, while the long-running cluster is highly connected and has avg_hops ~= 1, so it validates reachability/steady-state behavior more than it reproduces the Chord paper's multi-hop lookup regime.

For the paper, I need the PR to attach or paste reproducible JSONL result sets, not only summary bullets:

  1. dummy_dht_sim runs for stable, failed-node, and churn scenarios, ideally with the default paper-scale shape (N=1600, lookups_per_node=64, finger_table_size=160, failed-node pct {0,10,20}, churn pct {5,10,15,20,25,30,35,40}). Include exact command/env, commit SHA, machine class, and elapsed time.
  2. A Docker/WebRTC convergence series from a freshly started cluster: repeated samples from cluster-ready until steady state, with sample_time_ms, successor_matches, predecessor_matches, full_matches, connected_directed_edges, lookup correctness, avg hops, timeouts, and failures per 10k. This is what the paper should cite as T_converge / evidence scope.
  3. A steady-state Docker/WebRTC window after convergence, not a single sample. Please provide at least mean/min/max or all JSONL samples for lookup correctness, avg hops, timeout rate, failures per 10k, topology/full matches, and connected edges.
  4. Transport data from a rebuilt PR image where listPeerMeasurements is available, so source_sent_delta, destination_received_delta, and failure counters are populated. The current 4 x 1 KiB / 0.20 Mbps smoke is useful as a smoke test, but it is not paper-grade without receive-side counters and repeated payload/message sizes.
  5. Keep labels explicit in the emitted artifacts: backend=dummy_dht_sim vs backend=docker_cluster / lookup_model=status_snapshot_route, plus RINGS_DHT_FINGER_TABLE_SIZE, stabilize interval, topology mode, node count, payload bytes, and message count.

With those artifacts, PR #655 can safely say: Chord baseline is measured by B_C(eta) from the simulator; Rings implementation evidence is measured separately by Docker/WebRTC reports; fresh-cluster convergence and steady-state lookup must not be collapsed into one number.

@RyanKung

RyanKung commented Jul 4, 2026

Copy link
Copy Markdown
Member Author

Addressing the paper-facing benchmark artifact request from the earlier review comment.

I added reproducible result artifacts in benchmark-results/dht/ at head 3b1d0191bf13533bc4d667f0b909b61cacb5d26f:

  • benchmark-results/dht/dummy-paper-scale-2026-07-05.jsonl

    • backend=dummy_dht_sim / Chord baseline data for B_C(eta).
    • Run shape: N=1600, lookups_per_node=64, finger_table_size=160.
    • Covers stable, failed-node {10,20}%, and churn {5,10,15,20,25,30,35,40}%.
    • Stable result: correctness=100%, avg_hops=5.469, lookup_failures_per_10k=0.
  • benchmark-results/dht/docker-convergence-16node-2026-07-05.jsonl

    • backend=docker_cluster / lookup_model=status_snapshot_route implementation evidence.
    • Fresh 16-node Docker/WebRTC ring, RINGS_STABILIZE_INTERVAL=1, RINGS_DHT_FINGER_TABLE_SIZE=16.
    • 30 samples over ~309 seconds from cluster ready.
    • This run did not reach steady state: first and last samples remained successor_matches=0, predecessor_matches=3, full_matches=0, correctness=19.41%, lookup_failures_per_10k=8058.594.
    • So this is convergence evidence, not a valid steady-state Docker lookup window. The paper should not collapse this into the Chord baseline or a steady-state lookup number.
  • benchmark-results/dht/docker-transport-16node-2026-07-05.jsonl

    • Real node-internal WebRTC burst via transportBenchmark, fixed node0 -> node1, with listPeerMeasurements counters populated.
    • Payload sweep results:
      • 1 KiB * 4096: mean flush_mbps=10.462, send/receive failures 0.
      • 16 KiB * 1024: mean flush_mbps=107.353, send/receive failures 0.
      • 64 KiB * 1024: mean flush_mbps=90.358, send/receive failures 0.
    • This replaces the earlier 4 x 1 KiB / 0.20 Mbps smoke number, which was dominated by per-message docker exec + curl + JSON-RPC + base64/envelope control-plane overhead rather than WebRTC transport throughput.
  • benchmark-results/dht/environment-2026-07-05.json

    • Includes commit SHA, branch, machine class, Rust/Cargo/Docker versions, Docker image tag/ID, and exact commands.

I also updated scripts/dht_docker_cluster_bench.py so Docker samples carry explicit labels for cluster_topology, stabilize_interval_seconds, docker_image, and configured_finger_table_size, and transport sampling can pin --throughput-source-index / --throughput-destination-index for reproducible pair selection.

Net interpretation for PR #655 / the paper: use the dummy simulator JSONL for the Chord baseline; use Docker/WebRTC artifacts as Rings implementation evidence. The current Docker ring convergence artifact exposes a real limitation: the fresh 16-node ring did not converge within the sampled window, so a steady-state Docker lookup table still needs either a successfully converged run or a separate fix to cluster stabilization behavior.

RyanKung commented Jul 4, 2026

Copy link
Copy Markdown
Member Author

Follow-up after the benchmark artifact update.

This now gives PR #655 usable paper inputs, with one important boundary:

dummy-paper-scale-2026-07-05.jsonl      -> usable for B_C(eta)
docker-transport-16node-2026-07-05.jsonl -> usable for Report_Rings transport evidence
docker-convergence-16node-2026-07-05.jsonl -> usable only as non-convergence / convergence-scope evidence

The dummy baseline is now paper-grade for the formal Chord baseline object: it has the requested N=1600, lookups_per_node=64, finger_table_size=160, stable / failed-node / churn scenarios, exact environment metadata, and per-scenario JSONL. The stable baseline is especially clean: 102400 lookups, 100% correctness, avg_hops=5.469, timeouts=0, failures_per_10k=0.

The Docker transport artifact is also now usable as Rings implementation evidence. It has repeated payload sizes, source and destination counters, zero send/receive failures, and no flush timeouts. The paper can cite it as application-payload-through-WebRTC evidence, with the explicit caveat already documented for 64 KiB payloads: destination_received_delta counts lower-level accepted transport chunks/messages, so it is not a one-to-one application message counter at that size.

The Docker convergence artifact should not be used as a steady-state lookup result. Across 30 samples / about 309.5 seconds it stays flat at successor_matches=0/16, predecessor_matches=3/16, full_matches=0/16, correctness_rate=19.41%, avg_hops=1.026, lookup_failures_per_10k=8058.59. That is valuable negative/convergence evidence, but it means PR #655 should still say the current Docker/WebRTC lookup evidence is scoped to convergence failure under this fresh 16-node ring setup, not steady-state DHT performance.

For the paper side, the remaining missing artifact is a valid Docker/WebRTC steady-state lookup window if we want to make any steady-state Rings lookup claim. Without that, we can still write the evaluation section accurately as:

B_C(eta): measured by dummy_dht_sim paper-scale Chord baseline.
Report_Rings.transport: measured by Docker/WebRTC transport sweep.
Report_Rings.lookup/convergence: fresh 16-node Docker/WebRTC ring did not converge within the sampled 309.5s window; no steady-state lookup claim is made from that run.

@RyanKung

RyanKung commented Jul 4, 2026

Copy link
Copy Markdown
Member Author

Update after opening #658:

  • Changed the DHT finger table semantics from sparse/no-wrap entries to Chord wrapping entries: successor((n + 2^k) mod 2^160). This is now aligned across the pure topology model, FingerTable, the convergence comments/spec wrappers, and the dummy simulator.
  • Wrapping slots may legitimately point to the local node; lookup routing now skips self fingers and falls back to the successor head instead of forwarding to self.
  • Regenerated benchmark-results/dht/dummy-paper-scale-2026-07-05.jsonl with N=1600, finger_table_size=160, and lookups_per_node=64.

Result summary from the regenerated dummy baseline:

  • stable: correctness 100%, avg hops 5.469355, timeouts 0, known finger slots min/avg/max 160/160/160
  • failed_nodes 10% / 20%: correctness 99.02% / 96.85%, avg hops 5.758 / 6.186
  • churn 5%..40%: correctness drops from 94.54% to 48.39%, avg hops rises from 5.617 to 6.746

Important conclusion: the wrapping fix corrects a real Chord semantic mismatch, but it does not reduce the stable aggregate avg hops; the stable baseline remains 5.469355. So the remaining gap versus the original Chord paper is not explained by the previous no-wrap finger slots alone.

Validation run locally:

  • cargo test -p rings-core test_finger_table -- --nocapture
  • cargo test -p rings-core dht::topology -- --nocapture
  • cargo test -p rings-core test_chord_finger -- --nocapture
  • cargo test -p rings-core test_two_node_finger -- --nocapture
  • cargo test -p rings-core test_finger_when_disconnect -- --nocapture
  • cargo test -p rings-core dht_convergence -- --nocapture
  • cargo test -p rings-core dht_stateright -- --nocapture
  • git diff --check

Docker/WebRTC artifacts were not regenerated in this commit; the artifact metadata now explicitly scopes them to the existing rings-node-cluster:benchmark-artifacts-2efb302b image.

@RyanKung

RyanKung commented Jul 4, 2026

Copy link
Copy Markdown
Member Author

Formal hop-accounting pass:

Let d be the clockwise rank distance from the current node to the lookup target's responsible successor. For an ideal Chord ring with one terminal successor (K=1), the greedy recurrence is:

H_1(d) = 1                              if d <= 1
H_1(d) = 1 + H_1(d - 2^floor(log2(d-1))) otherwise

So for N = 2^m, H_1(d) = popcount(d - 1) + 1, and the average request-round count is 1 + m/2. If we count forwarding/path hops instead of request rounds, that is m/2, matching the classic Chord 0.5 * log2(N) expectation.

With our successor list terminal condition (K=3), the recurrence becomes:

H_K(d) = 1                              if d <= K
H_K(d) = 1 + H_K(d - 2^floor(log2(d-1))) otherwise

Independent rank-model calculation for N=1600 gives:

  • K=1: 6.160 request rounds, 5.160 forwarding hops
  • K=3: 5.410 request rounds, 4.410 forwarding hops

Independent reimplementation over the benchmark's splitmix DIDs and deterministic lookup targets gives:

  • K=1: 6.1572265625 request rounds, 5.1572265625 forwarding hops
  • K=3: 5.46935546875 request rounds, 4.46935546875 forwarding hops

This exactly reproduces the committed dummy result (avg_hops=5.469355). Therefore the remaining discrepancy is not a Chord finger implementation bug. It is a metric-definition mismatch: the benchmark field currently called avg_hops counts lookup rounds / visited nodes and includes the terminal resolution round, while the paper comparison should use forwarding/path hops. Under that comparable definition, the dummy baseline is below 0.5 * log2(1600) = 5.321928, mainly because the benchmark uses a successor list of capacity 3 for terminal resolution.

Recommended follow-up: rename or split this metric in the benchmark output, e.g. keep avg_lookup_rounds and add avg_forward_hops = avg_lookup_rounds - 1, then use avg_forward_hops for the Chord-theory comparison in the paper.

@RyanKung

RyanKung commented Jul 5, 2026

Copy link
Copy Markdown
Member Author

Additional note on the finger-table-size intuition:

finger_table_size=160 should not be interpreted as 160 independent useful shortcuts for an N=1600 network. The ring keyspace is 160 bits, but only 1600 live nodes occupy it, so many high-resolution Chord probe points collapse to the same successor node. The effective number of distinct routing choices is therefore bounded by the node distribution, not by the raw 160-bit slot count.

This is why the comparable theoretical scale is still O(log N) / roughly 0.5 * log2(N) forwarding hops, not something proportional to or dramatically improved by having 160 slots. In the dummy baseline, the observed avg_lookup_rounds=5.469 corresponds to about avg_forward_hops=4.469; that is consistent with the N=1600, K=3 recurrence and does not indicate that the 160-slot table is underperforming.

RyanKung commented Jul 5, 2026

Copy link
Copy Markdown
Member Author

Follow-up on the new hop-accounting notes.

The formal accounting resolves the main paper-comparison concern: the committed avg_hops=5.469355 is a lookup-round / visited-node count that includes the terminal resolution round. For a Chord-paper-style forwarding/path-hop comparison, the comparable value is approximately:

avg_forward_hops = avg_lookup_rounds - 1 = 4.469355

That changes the interpretation materially. Stable N=1600 is no longer 5.469 vs 3.84 on the same axis; it is closer to 4.469 forwarding hops vs 3.84 Chord routing hops, while also matching the independent K=3 recurrence scale. The remaining stable gap versus the original Chord table is therefore not evidence of a finger-table bug by itself.

For paper safety, I think this PR should make the metric split explicit in the emitted JSON and README before PR #655 consumes it:

avg_lookup_rounds := current avg_hops
avg_forward_hops := avg_lookup_rounds - 1

Then PR #655 can use avg_forward_hops for Chord-paper comparison and reserve avg_lookup_rounds for implementation/report accounting. Without that schema split, the paper is likely to repeat the exact metric mismatch this comment just clarified.

The churn reliability conclusion does not change: failures/10k and timeout growth are still much worse than the original Chord table under the current churn model, so the paper should present churn as a scoped baseline/obligation, not as a superiority claim. Docker/WebRTC lookup also remains convergence evidence only until a valid steady-state run is produced.

@RyanKung

RyanKung commented Jul 5, 2026

Copy link
Copy Markdown
Member Author

Added the maintained_chord dummy benchmark model.

What changed:

  • dht_network_sim now emits a maintenance_model field.
  • Failure/churn scenarios are emitted for both:
    • instant_stale_snapshot: previous behavior; abrupt failures/churn with stale routing state and no repair.
    • maintained_chord: post-maintenance steady-state; active nodes route over the current active ring after departed nodes are removed and joiners are learned.
  • Added avg_forward_hops beside the existing avg_hops. avg_hops remains lookup rounds / visited nodes; avg_forward_hops subtracts the terminal resolution round and is the paper-comparable Chord path-length metric.
  • Added RINGS_DHT_BENCH_MAINTENANCE_MODELS to limit output to instant_stale_snapshot, maintained_chord, or both.

Regenerated benchmark-results/dht/dummy-paper-scale-2026-07-05.jsonl; it now has 21 rows. Key maintained rows:

  • stable maintained: correctness 100%, avg lookup rounds 5.469, avg forward hops 4.469, timeouts 0, failures/10k 0
  • failed_nodes 10% maintained: correctness 100%, avg forward hops 4.394, timeouts 0, failures/10k 0
  • failed_nodes 20% maintained: correctness 100%, avg forward hops 4.316, timeouts 0, failures/10k 0
  • churn 5%..40% maintained: correctness stays 100%, avg forward hops stays about 4.45..4.47, timeouts 0, failures/10k 0

This confirms the earlier diagnosis: the large timeout/failure numbers belong to the no-maintenance stale-snapshot stress model, not to a maintained Chord baseline. The README now says to use maintained_chord rows for comparisons against the original Chord paper, and keep instant_stale_snapshot only for stale-state sensitivity discussion.

Validation:

  • full 1600-node dummy benchmark regenerated with default maintenance models
  • smoke run with RINGS_DHT_BENCH_MAINTENANCE_MODELS=maintained_chord
  • JSON/JSONL parse checks
  • cargo bench -p rings-core --bench dht_network_sim --no-run --no-default-features --features dummy
  • git diff --check

@RyanKung

RyanKung commented Jul 5, 2026

Copy link
Copy Markdown
Member Author

Follow-up on the remaining hop-count gap versus the original Chord table:

The maintained successor_capacity=3 rows are still slightly higher than the table values, but the gap is explained by successor-list capacity/configuration rather than finger semantics. I parameterized the dummy benchmark with RINGS_DHT_BENCH_SUCCESSOR_CAPACITY and ran the same maintained/stale paths with successor_capacity=8.

K=8 maintained-only probe:

  • stable: avg forward hops 3.807 vs Chord table 3.84
  • failed 10%: avg forward hops 3.732 vs Chord table 4.03
  • failed 20%: avg forward hops 3.647 vs Chord table 4.22

The failed rows are lower there because a fully maintained active ring has fewer live nodes and no stale pointers. The original Chord failure table appears closer to a stale-but-robust failure model: stale routing entries still exist, but a large enough successor list lets lookups route around them. Running successor_capacity=8 with instant_stale_snapshot gives:

  • stable: avg forward hops 3.807 vs 3.84 (-0.033)
  • failed 10%: avg forward hops 4.037 vs 4.03 (+0.007), failures/10k 0
  • failed 20%: avg forward hops 4.388 vs 4.22 (+0.168), failures/10k 0

So the previously reported K=3 maintained rows are a Rings-configuration baseline, not the closest reproduction of the Chord paper table. For paper-table reproduction, we should run/report a separate successor_capacity=8 series. For Rings implementation discussion, keep the K=3 rows because they match the current benchmark/default successor capacity.

I also added RINGS_DHT_BENCH_SUCCESSOR_CAPACITY=<n> to make this explicit and reproducible.

@RyanKung

RyanKung commented Jul 5, 2026

Copy link
Copy Markdown
Member Author

Correction on the Chord paper successor-list parameter:

The original Chord paper does not use K; it uses successor-list size r. For the Table II / Table III simulations, the paper uses N = 1000 and r = 20 = 2 log2 N. The theoretical statement is r = O(log N), but the concrete table values Ryan cited (3.84, 4.03, 4.22, and the churn rows) correspond to r=20.

So the direct parameter mapping to this benchmark is:

RINGS_DHT_BENCH_SUCCESSOR_CAPACITY=20

My previous K=8 probe is still useful as a sensitivity check, and it happens to put the stable row near 3.84 under our current metric/accounting, but it is not the paper's actual successor-list size. We should run/report a dedicated successor_capacity=20, N=1000 series if the goal is to reproduce Chord Table II/III as closely as possible.

@RyanKung

RyanKung commented Jul 5, 2026

Copy link
Copy Markdown
Member Author

Added a paper-aligned simulator artifact for the original Chord paper coverage.

New commit: 61aad7b5 Add Chord paper-aligned benchmark simulator

What changed:

  • Added scripts/chord_paper_sim.py.
  • Added benchmark-results/dht/chord-paper-sim-2026-07-05.jsonl with 63 JSONL rows.
  • Updated benchmark-results/dht/README.md and environment-2026-07-05.json with the paper coverage matrix.

Coverage now maps the original Chord paper's numbered tables and data figures:

  • Table I: semantic reference only, no benchmark row required.
  • Table II: paper_item=table_ii, N=1000, successor list r=20, failed nodes 0%-50%, 10,000 lookups.
  • Table III: paper_item=table_iii, paired join/leave rates 0.05-0.40/s, N~=1000, successor list r=20, event-driven stabilization model.
  • Table IV: paper_item=table_iv, N=2^16, s=1,2,4,8,16, iterative/recursive lookup, 3D and transit-stub latency models.
  • Fig. 8: paper_item=fig_8a and fig_8b, consistent-hashing load-balance data.
  • Fig. 9: paper_item=fig_9, virtual-node load-balance data, explicitly marked runtime_support=simulator_only.
  • Fig. 10: paper_item=fig_10a and fig_10b, path-length scaling and PDF at k=12.

Important boundary:

Rings does not currently implement Chord-style virtual nodes, i.e. one physical node owning multiple unrelated ring positions. The old VNode terminology is storage-entry terminology and is separate from this Chord feature. I opened #659 to track the design question: whether Rings needs runtime virtual-node support, and if so how it should interact with routing, storage ownership, replication/sync, authentication, and WebRTC transport mapping.

Validation:

  • python3 -m py_compile scripts/chord_paper_sim.py
  • Full generation: python3 scripts/chord_paper_sim.py --include all > benchmark-results/dht/chord-paper-sim-2026-07-05.jsonl
  • JSON/JSONL parse check, expected row counts: Table II 6, Table III 8, Table IV 20, Fig. 8a 10, Fig. 8b 1, Fig. 9 5, Fig. 10a 12, Fig. 10b 1.
  • git diff --check

@RyanKung

RyanKung commented Jul 5, 2026

Copy link
Copy Markdown
Member Author

Updated this PR to keep the benchmark implementation pure Rust.

Changes:

  • Removed the newly added Python scripts: scripts/chord_paper_sim.py and scripts/dht_docker_cluster_bench.py.
  • Added crates/core/benches/chord_paper_sim.rs plus crates/core/benches/chord_paper_sim/support.rs as the Rust cargo bench for Chord Table II/III/IV and Fig. 8/9/10 simulator rows.
  • Regenerated benchmark-results/dht/chord-paper-sim-2026-07-05.jsonl; rows now report simulator="crates/core/benches/chord_paper_sim.rs".
  • Updated benchmark docs and environment metadata to remove Python commands. Docker/WebRTC artifact docs now describe the /status, transportBenchmark, and listPeerMeasurements boundaries directly; any reusable collector should be implemented as Rust before being committed.

Validation:

  • cargo clippy -p rings-core --bench chord_paper_sim --no-default-features --features dummy -- -D warnings
  • cargo check -p rings-core --benches --no-default-features --features dummy
  • cargo bench -p rings-core --bench chord_paper_sim --no-default-features --features dummy -- --include all > benchmark-results/dht/chord-paper-sim-2026-07-05.jsonl
  • jq empty benchmark-results/dht/environment-2026-07-05.json
  • targeted scan for the removed Python benchmark paths returned no matches

@RyanKung
RyanKung force-pushed the codex/dht-benchmark-653 branch from 2b6f2ef to aa89fc4 Compare July 7, 2026 16:59
@RyanKung

RyanKung commented Jul 7, 2026

Copy link
Copy Markdown
Member Author

Rebased this PR onto origin/master after #660 was merged (2adbefb8, "Implement storage virtual nodes") and added a runtime storage-vnode supplement for the paper benchmark data.

What changed:

  • crates/core/benches/chord_paper_sim.rs now uses rings_core::dht::StorageVirtualNodes / VirtualNodeConfig for Fig. 9 instead of the old hand-rolled virtual-node simulator.
  • Added benchmark-results/dht/chord-paper-sim-2026-07-08.jsonl and environment-2026-07-08.json.
  • Updated the benchmark README to state the exact boundary: PR Implement storage virtual nodes #660 implements storage-owner virtual positions, not independent signing identities and not routing/finger-table virtual peers.

Fig. 9 runtime storage-vnode data (10^4 physical owners, 10^6 placement keys, 160-bit Rings DIDs):

vnodes / owner virtual positions mean keys p1 p99 max
1 10,000 100.0 0 457 1164
2 20,000 100.0 6 327 695
5 50,000 100.0 23 236 392
10 100,000 100.0 38 194 263
20 200,000 100.0 52 165 228

Validation run:

  • rustfmt --check crates/core/benches/chord_paper_sim.rs
  • cargo check -p rings-core --bench chord_paper_sim --no-default-features --features dummy
  • cargo clippy -p rings-core --bench chord_paper_sim --no-default-features --features dummy -- -D warnings
  • cargo test -p rings-core dht::virtual_node --no-default-features --features dummy
  • cargo test -p rings-node dht_virtual_nodes --no-default-features --features node
  • cargo bench -p rings-core --bench chord_paper_sim --no-default-features --features dummy -- --include all > benchmark-results/dht/chord-paper-sim-2026-07-08.jsonl

@RyanKung

RyanKung commented Jul 7, 2026

Copy link
Copy Markdown
Member Author

Follow-up cleanup: the branch no longer commits benchmark result data directly.

I removed the tracked benchmark-results/dht artifacts from the PR and added benchmark-results/ to .gitignore, so locally generated JSONL/metadata outputs won't be accidentally included in the branch. The PR now keeps the Rust benchmark/simulator code and generation paths, while the measured values remain in PR discussion/output rather than repository contents.

Current cleanup commit: 3f2bede4 (Remove committed DHT benchmark artifacts).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add a small-scale DHT benchmark for Chord behavior under network perturbations

1 participant