diff --git a/docs/p2p-metrics-capture.md b/docs/p2p-metrics-capture.md new file mode 100644 index 00000000..6cbafebf --- /dev/null +++ b/docs/p2p-metrics-capture.md @@ -0,0 +1,186 @@ +# P2P Metrics Capture — What Each Field Means and Where It’s Collected + +This guide explains every field we emit in Cascade events, how it is measured, and exactly where it is captured in the code. + +The design is minimal by intent: +- Metrics are collected only for the first pass of Register (store) and for the active Download operation. +- P2P APIs return errors only; per‑RPC details are captured via a small metrics package (`pkg/p2pmetrics`). +- No aggregation; we only group raw RPC attempts by IP. + +--- + +## Store (Register) Event + +Event payload shape + +```json +{ + "store": { + "duration_ms": 9876, + "symbols_first_pass": 220, + "symbols_total": 1200, + "id_files_count": 14, + "success_rate_pct": 82.5, + "calls_by_ip": { + "10.0.0.5": [ + {"ip": "10.0.0.5", "address": "A:4445", "keys": 100, "success": true, "duration_ms": 120}, + {"ip": "10.0.0.5", "address": "A:4445", "keys": 120, "success": false, "error": "timeout", "duration_ms": 300} + ] + } + } +} +``` + +### Fields + +- `store.duration_ms` + - Meaning: End‑to‑end elapsed time of the first‑pass store phase (Register’s storage section only). + - Where captured: `supernode/services/cascade/adaptors/p2p.go` + - A `time.Now()` timestamp is taken just before the first‑pass store function and measured on return. + +- `store.symbols_first_pass` + - Meaning: Number of symbols sent during the Register first pass (across the combined first batch and any immediate first‑pass symbol batches). + - Where captured: `supernode/services/cascade/adaptors/p2p.go` via `p2pmetrics.SetStoreSummary(...)` using the value returned by `storeCascadeSymbolsAndData`. + +- `store.symbols_total` + - Meaning: Total symbols available in the symbol directory (before sampling). Used to contextualize the first‑pass coverage. + - Where captured: Computed in `storeCascadeSymbolsAndData` and included in `SetStoreSummary`. + +- `store.id_files_count` + - Meaning: Number of redundant metadata files (ID files) sent in the first combined batch. + - Where captured: `len(req.IDFiles)` in `StoreArtefacts`, passed to `SetStoreSummary`. + +- `store.calls_by_ip` + - Meaning: All raw network store RPC attempts grouped by the node IP. + - Each array entry is a single RPC attempt with: + - `ip` — Node IP (fallback to `address` if missing). + - `address` — Node string `IP:port`. + - `keys` — Number of items in that RPC attempt (metadata + first symbols for the first combined batch, symbols for subsequent batches within the first pass). + - `success` — True if there was no transport error and no error message returned by the node response. Note: this flag does not explicitly check the `ResultOk` status; in rare cases, a non‑OK response with an empty error message may appear as `success` in metrics. (Internal success‑rate enforcement still uses explicit response status.) + - `error` — Any error string captured; omitted when success. + - `duration_ms` — RPC duration in milliseconds. + - `noop` — Present and `true` when no store payload was sent to the node (empty batch for that node). Such entries are recorded as `success=true`, `keys=0`, with no `error`. + - Where captured: + - Emission point (P2P): `p2p/kademlia/dht.go::IterateBatchStore(...)` + - After each node RPC returns, we call `p2pmetrics.RecordStore(taskID, Call{...})`. For nodes with no payload, a `noop: true` entry is emitted without sending a wire RPC. + - `taskID` is read from the context via `p2pmetrics.TaskIDFromContext(ctx)`. + - Grouping: `pkg/p2pmetrics/metrics.go` + - `StartStoreCapture(taskID)` enables capture; `StopStoreCapture(taskID)` disables it. + - Calls are grouped by `ip` (fallback to `address`) without further aggregation. + +- `store.success_rate_pct` + - Meaning: First‑pass store success rate computed from captured per‑RPC outcomes: successful responses divided by total recorded store RPC attempts, expressed as a percentage. + - Where captured: Computed in `pkg/p2pmetrics/metrics.go::BuildStoreEventPayloadFromCollector` from `calls_by_ip` data. + +### First‑Pass Success Threshold + +- Internal enforcement only: if DHT first‑pass success rate is below 75%, `IterateBatchStore` returns an error. +- We also emit `store.success_rate_pct` for analytics; the threshold only affects control flow (errors), not the emitted metric. +- Code: `p2p/kademlia/dht.go::IterateBatchStore`. + +### Scope Limits + +- Background worker (which continues storing remaining symbols) is NOT captured — we don’t set a metrics task ID on those paths. + +--- + +## Download Event + +Event payload shape + +```json +{ + "retrieve": { + "found_local": 42, + "retrieve_ms": 2000, + "decode_ms": 8000, + "calls_by_ip": { + "10.0.0.7": [ + {"ip": "10.0.0.7", "address": "B:4445", "keys": 13, "success": true, "duration_ms": 90} + ] + } + } +} +``` + +### Fields + +- `retrieve.found_local` + - Meaning: Number of items retrieved from local storage before any network calls. + - Where captured: `p2p/kademlia/dht.go::BatchRetrieve(...)` + - After `fetchAndAddLocalKeys`, we call `p2pmetrics.ReportFoundLocal(taskID, int(foundLocalCount))`. + - `taskID` is read from context with `p2pmetrics.TaskIDFromContext(ctx)`. + +- `retrieve.retrieve_ms` + - Meaning: Time spent in network batch‑retrieve. + - Where captured: `supernode/services/cascade/download.go` + - Timestamp before `BatchRetrieve`, measured after it returns. + +- `retrieve.decode_ms` + - Meaning: Time spent decoding symbols and reconstructing the file. + - Where captured: `supernode/services/cascade/download.go` + - Timestamp before decode, measured after it returns. + +- `retrieve.calls_by_ip` + - Meaning: All raw per‑RPC retrieve attempts grouped by node IP. + - Each array entry is a single RPC attempt with: + - `ip`, `address` — Identifiers as available. + - `keys` — Number of symbols returned by that node in that call. + - `success` — True if the RPC completed without error (even if `keys == 0`). Transport/status errors remain `success=false` with an `error` message. + - `error` — Error string when the RPC failed; omitted otherwise. + - `duration_ms` — RPC duration in milliseconds. + - `noop` — Present and `true` when no network request was actually sent to the node (e.g., all requested keys were already satisfied or deduped before issuing the call). Such entries are recorded as `success=true`, `keys=0`, with no `error`. + - Where captured: + - Emission point (P2P): `p2p/kademlia/dht.go::iterateBatchGetValues(...)` + - Each node attempt records a `p2pmetrics.RecordRetrieve(taskID, Call{...})`. For attempts where no network RPC is sent, a `noop: true` entry is emitted. + - `taskID` is extracted from context using `p2pmetrics.TaskIDFromContext(ctx)`. + - Grouping: `pkg/p2pmetrics/metrics.go` (same grouping/fallback as store). + +### Scope Limits + +- Metrics are captured only for the active Download call (context is tagged in `download.go`). + +--- + +## Context Tagging (Task ID) + +- We use an explicit, metrics‑only context key defined in `pkg/p2pmetrics` to tag P2P calls with a task ID. + - Setters: `p2pmetrics.WithTaskID(ctx, id)`. + - Getters: `p2pmetrics.TaskIDFromContext(ctx)`. +- Where it is set: + - Store (first pass): `supernode/services/cascade/adaptors/p2p.go` wraps `StoreBatch` calls. + - Download: `supernode/services/cascade/download.go` wraps `BatchRetrieve` call. + +--- + +## Building and Emitting Events + +- Store + - `supernode/services/cascade/helper.go::emitArtefactsStored(...)` + - Builds `store` payload via `p2pmetrics.BuildStoreEventPayloadFromCollector(taskID)`. + - Includes `success_rate_pct` (first‑pass store success rate computed from captured per‑RPC outcomes) in addition to the minimal fields. + - Emits the event. + +- Download + - `supernode/services/cascade/download.go` + - Builds `retrieve` payload via `p2pmetrics.BuildDownloadEventPayloadFromCollector(actionID)`. + - Emits the event. + +--- + +## Quick File Map + +- Capture + grouping: `supernode/pkg/p2pmetrics/metrics.go` +- Store adaptor: `supernode/supernode/services/cascade/adaptors/p2p.go` +- Store event: `supernode/supernode/services/cascade/helper.go` +- Download flow: `supernode/supernode/services/cascade/download.go` +- DHT store calls: `supernode/p2p/kademlia/dht.go::IterateBatchStore` +- DHT retrieve calls: `supernode/p2p/kademlia/dht.go::BatchRetrieve` and `iterateBatchGetValues` + +--- + +## Notes + +- No P2P stats/snapshots are used to build events. +- No aggregation is performed; we only group raw RPC attempts by IP. +- First‑pass success rate is enforced internally (75% threshold) but not emitted as a metric. diff --git a/gen/supernode/supernode.pb.go b/gen/supernode/supernode.pb.go index 5410f5c6..431bc8b5 100644 --- a/gen/supernode/supernode.pb.go +++ b/gen/supernode/supernode.pb.go @@ -1,6 +1,6 @@ // Code generated by protoc-gen-go. DO NOT EDIT. // versions: -// protoc-gen-go v1.35.2 +// protoc-gen-go v1.35.1 // protoc v3.21.12 // source: supernode/supernode.proto @@ -513,12 +513,16 @@ type StatusResponse_P2PMetrics struct { sizeCache protoimpl.SizeCache unknownFields protoimpl.UnknownFields - DhtMetrics *StatusResponse_P2PMetrics_DhtMetrics `protobuf:"bytes,1,opt,name=dht_metrics,json=dhtMetrics,proto3" json:"dht_metrics,omitempty"` - NetworkHandleMetrics map[string]*StatusResponse_P2PMetrics_HandleCounters `protobuf:"bytes,2,rep,name=network_handle_metrics,json=networkHandleMetrics,proto3" json:"network_handle_metrics,omitempty" protobuf_key:"bytes,1,opt,name=key,proto3" protobuf_val:"bytes,2,opt,name=value,proto3"` - ConnPoolMetrics map[string]int64 `protobuf:"bytes,3,rep,name=conn_pool_metrics,json=connPoolMetrics,proto3" json:"conn_pool_metrics,omitempty" protobuf_key:"bytes,1,opt,name=key,proto3" protobuf_val:"varint,2,opt,name=value,proto3"` - BanList []*StatusResponse_P2PMetrics_BanEntry `protobuf:"bytes,4,rep,name=ban_list,json=banList,proto3" json:"ban_list,omitempty"` - Database *StatusResponse_P2PMetrics_DatabaseStats `protobuf:"bytes,5,opt,name=database,proto3" json:"database,omitempty"` - Disk *StatusResponse_P2PMetrics_DiskStatus `protobuf:"bytes,6,opt,name=disk,proto3" json:"disk,omitempty"` + DhtMetrics *StatusResponse_P2PMetrics_DhtMetrics `protobuf:"bytes,1,opt,name=dht_metrics,json=dhtMetrics,proto3" json:"dht_metrics,omitempty"` + NetworkHandleMetrics map[string]*StatusResponse_P2PMetrics_HandleCounters `protobuf:"bytes,2,rep,name=network_handle_metrics,json=networkHandleMetrics,proto3" json:"network_handle_metrics,omitempty" protobuf_key:"bytes,1,opt,name=key,proto3" protobuf_val:"bytes,2,opt,name=value,proto3"` + ConnPoolMetrics map[string]int64 `protobuf:"bytes,3,rep,name=conn_pool_metrics,json=connPoolMetrics,proto3" json:"conn_pool_metrics,omitempty" protobuf_key:"bytes,1,opt,name=key,proto3" protobuf_val:"varint,2,opt,name=value,proto3"` + BanList []*StatusResponse_P2PMetrics_BanEntry `protobuf:"bytes,4,rep,name=ban_list,json=banList,proto3" json:"ban_list,omitempty"` + Database *StatusResponse_P2PMetrics_DatabaseStats `protobuf:"bytes,5,opt,name=database,proto3" json:"database,omitempty"` + Disk *StatusResponse_P2PMetrics_DiskStatus `protobuf:"bytes,6,opt,name=disk,proto3" json:"disk,omitempty"` + RecentBatchStore []*StatusResponse_P2PMetrics_RecentBatchStoreEntry `protobuf:"bytes,7,rep,name=recent_batch_store,json=recentBatchStore,proto3" json:"recent_batch_store,omitempty"` + RecentBatchRetrieve []*StatusResponse_P2PMetrics_RecentBatchRetrieveEntry `protobuf:"bytes,8,rep,name=recent_batch_retrieve,json=recentBatchRetrieve,proto3" json:"recent_batch_retrieve,omitempty"` + RecentBatchStoreByIp map[string]*StatusResponse_P2PMetrics_RecentBatchStoreList `protobuf:"bytes,9,rep,name=recent_batch_store_by_ip,json=recentBatchStoreByIp,proto3" json:"recent_batch_store_by_ip,omitempty" protobuf_key:"bytes,1,opt,name=key,proto3" protobuf_val:"bytes,2,opt,name=value,proto3"` + RecentBatchRetrieveByIp map[string]*StatusResponse_P2PMetrics_RecentBatchRetrieveList `protobuf:"bytes,10,rep,name=recent_batch_retrieve_by_ip,json=recentBatchRetrieveByIp,proto3" json:"recent_batch_retrieve_by_ip,omitempty" protobuf_key:"bytes,1,opt,name=key,proto3" protobuf_val:"bytes,2,opt,name=value,proto3"` } func (x *StatusResponse_P2PMetrics) Reset() { @@ -593,6 +597,34 @@ func (x *StatusResponse_P2PMetrics) GetDisk() *StatusResponse_P2PMetrics_DiskSta return nil } +func (x *StatusResponse_P2PMetrics) GetRecentBatchStore() []*StatusResponse_P2PMetrics_RecentBatchStoreEntry { + if x != nil { + return x.RecentBatchStore + } + return nil +} + +func (x *StatusResponse_P2PMetrics) GetRecentBatchRetrieve() []*StatusResponse_P2PMetrics_RecentBatchRetrieveEntry { + if x != nil { + return x.RecentBatchRetrieve + } + return nil +} + +func (x *StatusResponse_P2PMetrics) GetRecentBatchStoreByIp() map[string]*StatusResponse_P2PMetrics_RecentBatchStoreList { + if x != nil { + return x.RecentBatchStoreByIp + } + return nil +} + +func (x *StatusResponse_P2PMetrics) GetRecentBatchRetrieveByIp() map[string]*StatusResponse_P2PMetrics_RecentBatchRetrieveList { + if x != nil { + return x.RecentBatchRetrieveByIp + } + return nil +} + type StatusResponse_Resources_CPU struct { state protoimpl.MessageState sizeCache protoimpl.SizeCache @@ -1134,6 +1166,285 @@ func (x *StatusResponse_P2PMetrics_DiskStatus) GetFreeMb() float64 { return 0 } +// Last handled BatchStoreData requests (most recent first) +type StatusResponse_P2PMetrics_RecentBatchStoreEntry struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + TimeUnix int64 `protobuf:"varint,1,opt,name=time_unix,json=timeUnix,proto3" json:"time_unix,omitempty"` + SenderId string `protobuf:"bytes,2,opt,name=sender_id,json=senderId,proto3" json:"sender_id,omitempty"` + SenderIp string `protobuf:"bytes,3,opt,name=sender_ip,json=senderIp,proto3" json:"sender_ip,omitempty"` + Keys int32 `protobuf:"varint,4,opt,name=keys,proto3" json:"keys,omitempty"` + DurationMs int64 `protobuf:"varint,5,opt,name=duration_ms,json=durationMs,proto3" json:"duration_ms,omitempty"` + Ok bool `protobuf:"varint,6,opt,name=ok,proto3" json:"ok,omitempty"` + Error string `protobuf:"bytes,7,opt,name=error,proto3" json:"error,omitempty"` +} + +func (x *StatusResponse_P2PMetrics_RecentBatchStoreEntry) Reset() { + *x = StatusResponse_P2PMetrics_RecentBatchStoreEntry{} + mi := &file_supernode_supernode_proto_msgTypes[19] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) +} + +func (x *StatusResponse_P2PMetrics_RecentBatchStoreEntry) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*StatusResponse_P2PMetrics_RecentBatchStoreEntry) ProtoMessage() {} + +func (x *StatusResponse_P2PMetrics_RecentBatchStoreEntry) ProtoReflect() protoreflect.Message { + mi := &file_supernode_supernode_proto_msgTypes[19] + if x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use StatusResponse_P2PMetrics_RecentBatchStoreEntry.ProtoReflect.Descriptor instead. +func (*StatusResponse_P2PMetrics_RecentBatchStoreEntry) Descriptor() ([]byte, []int) { + return file_supernode_supernode_proto_rawDescGZIP(), []int{4, 3, 7} +} + +func (x *StatusResponse_P2PMetrics_RecentBatchStoreEntry) GetTimeUnix() int64 { + if x != nil { + return x.TimeUnix + } + return 0 +} + +func (x *StatusResponse_P2PMetrics_RecentBatchStoreEntry) GetSenderId() string { + if x != nil { + return x.SenderId + } + return "" +} + +func (x *StatusResponse_P2PMetrics_RecentBatchStoreEntry) GetSenderIp() string { + if x != nil { + return x.SenderIp + } + return "" +} + +func (x *StatusResponse_P2PMetrics_RecentBatchStoreEntry) GetKeys() int32 { + if x != nil { + return x.Keys + } + return 0 +} + +func (x *StatusResponse_P2PMetrics_RecentBatchStoreEntry) GetDurationMs() int64 { + if x != nil { + return x.DurationMs + } + return 0 +} + +func (x *StatusResponse_P2PMetrics_RecentBatchStoreEntry) GetOk() bool { + if x != nil { + return x.Ok + } + return false +} + +func (x *StatusResponse_P2PMetrics_RecentBatchStoreEntry) GetError() string { + if x != nil { + return x.Error + } + return "" +} + +// Last handled BatchGetValues requests (most recent first) +type StatusResponse_P2PMetrics_RecentBatchRetrieveEntry struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + TimeUnix int64 `protobuf:"varint,1,opt,name=time_unix,json=timeUnix,proto3" json:"time_unix,omitempty"` + SenderId string `protobuf:"bytes,2,opt,name=sender_id,json=senderId,proto3" json:"sender_id,omitempty"` + SenderIp string `protobuf:"bytes,3,opt,name=sender_ip,json=senderIp,proto3" json:"sender_ip,omitempty"` + Requested int32 `protobuf:"varint,4,opt,name=requested,proto3" json:"requested,omitempty"` + Found int32 `protobuf:"varint,5,opt,name=found,proto3" json:"found,omitempty"` + DurationMs int64 `protobuf:"varint,6,opt,name=duration_ms,json=durationMs,proto3" json:"duration_ms,omitempty"` + Error string `protobuf:"bytes,7,opt,name=error,proto3" json:"error,omitempty"` +} + +func (x *StatusResponse_P2PMetrics_RecentBatchRetrieveEntry) Reset() { + *x = StatusResponse_P2PMetrics_RecentBatchRetrieveEntry{} + mi := &file_supernode_supernode_proto_msgTypes[20] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) +} + +func (x *StatusResponse_P2PMetrics_RecentBatchRetrieveEntry) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*StatusResponse_P2PMetrics_RecentBatchRetrieveEntry) ProtoMessage() {} + +func (x *StatusResponse_P2PMetrics_RecentBatchRetrieveEntry) ProtoReflect() protoreflect.Message { + mi := &file_supernode_supernode_proto_msgTypes[20] + if x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use StatusResponse_P2PMetrics_RecentBatchRetrieveEntry.ProtoReflect.Descriptor instead. +func (*StatusResponse_P2PMetrics_RecentBatchRetrieveEntry) Descriptor() ([]byte, []int) { + return file_supernode_supernode_proto_rawDescGZIP(), []int{4, 3, 8} +} + +func (x *StatusResponse_P2PMetrics_RecentBatchRetrieveEntry) GetTimeUnix() int64 { + if x != nil { + return x.TimeUnix + } + return 0 +} + +func (x *StatusResponse_P2PMetrics_RecentBatchRetrieveEntry) GetSenderId() string { + if x != nil { + return x.SenderId + } + return "" +} + +func (x *StatusResponse_P2PMetrics_RecentBatchRetrieveEntry) GetSenderIp() string { + if x != nil { + return x.SenderIp + } + return "" +} + +func (x *StatusResponse_P2PMetrics_RecentBatchRetrieveEntry) GetRequested() int32 { + if x != nil { + return x.Requested + } + return 0 +} + +func (x *StatusResponse_P2PMetrics_RecentBatchRetrieveEntry) GetFound() int32 { + if x != nil { + return x.Found + } + return 0 +} + +func (x *StatusResponse_P2PMetrics_RecentBatchRetrieveEntry) GetDurationMs() int64 { + if x != nil { + return x.DurationMs + } + return 0 +} + +func (x *StatusResponse_P2PMetrics_RecentBatchRetrieveEntry) GetError() string { + if x != nil { + return x.Error + } + return "" +} + +// Per-IP buckets: last 10 per sender IP +type StatusResponse_P2PMetrics_RecentBatchStoreList struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + Entries []*StatusResponse_P2PMetrics_RecentBatchStoreEntry `protobuf:"bytes,1,rep,name=entries,proto3" json:"entries,omitempty"` +} + +func (x *StatusResponse_P2PMetrics_RecentBatchStoreList) Reset() { + *x = StatusResponse_P2PMetrics_RecentBatchStoreList{} + mi := &file_supernode_supernode_proto_msgTypes[21] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) +} + +func (x *StatusResponse_P2PMetrics_RecentBatchStoreList) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*StatusResponse_P2PMetrics_RecentBatchStoreList) ProtoMessage() {} + +func (x *StatusResponse_P2PMetrics_RecentBatchStoreList) ProtoReflect() protoreflect.Message { + mi := &file_supernode_supernode_proto_msgTypes[21] + if x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use StatusResponse_P2PMetrics_RecentBatchStoreList.ProtoReflect.Descriptor instead. +func (*StatusResponse_P2PMetrics_RecentBatchStoreList) Descriptor() ([]byte, []int) { + return file_supernode_supernode_proto_rawDescGZIP(), []int{4, 3, 9} +} + +func (x *StatusResponse_P2PMetrics_RecentBatchStoreList) GetEntries() []*StatusResponse_P2PMetrics_RecentBatchStoreEntry { + if x != nil { + return x.Entries + } + return nil +} + +type StatusResponse_P2PMetrics_RecentBatchRetrieveList struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + Entries []*StatusResponse_P2PMetrics_RecentBatchRetrieveEntry `protobuf:"bytes,1,rep,name=entries,proto3" json:"entries,omitempty"` +} + +func (x *StatusResponse_P2PMetrics_RecentBatchRetrieveList) Reset() { + *x = StatusResponse_P2PMetrics_RecentBatchRetrieveList{} + mi := &file_supernode_supernode_proto_msgTypes[22] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) +} + +func (x *StatusResponse_P2PMetrics_RecentBatchRetrieveList) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*StatusResponse_P2PMetrics_RecentBatchRetrieveList) ProtoMessage() {} + +func (x *StatusResponse_P2PMetrics_RecentBatchRetrieveList) ProtoReflect() protoreflect.Message { + mi := &file_supernode_supernode_proto_msgTypes[22] + if x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use StatusResponse_P2PMetrics_RecentBatchRetrieveList.ProtoReflect.Descriptor instead. +func (*StatusResponse_P2PMetrics_RecentBatchRetrieveList) Descriptor() ([]byte, []int) { + return file_supernode_supernode_proto_rawDescGZIP(), []int{4, 3, 10} +} + +func (x *StatusResponse_P2PMetrics_RecentBatchRetrieveList) GetEntries() []*StatusResponse_P2PMetrics_RecentBatchRetrieveEntry { + if x != nil { + return x.Entries + } + return nil +} + type StatusResponse_P2PMetrics_DhtMetrics_StoreSuccessPoint struct { state protoimpl.MessageState sizeCache protoimpl.SizeCache @@ -1147,7 +1458,7 @@ type StatusResponse_P2PMetrics_DhtMetrics_StoreSuccessPoint struct { func (x *StatusResponse_P2PMetrics_DhtMetrics_StoreSuccessPoint) Reset() { *x = StatusResponse_P2PMetrics_DhtMetrics_StoreSuccessPoint{} - mi := &file_supernode_supernode_proto_msgTypes[19] + mi := &file_supernode_supernode_proto_msgTypes[25] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -1159,7 +1470,7 @@ func (x *StatusResponse_P2PMetrics_DhtMetrics_StoreSuccessPoint) String() string func (*StatusResponse_P2PMetrics_DhtMetrics_StoreSuccessPoint) ProtoMessage() {} func (x *StatusResponse_P2PMetrics_DhtMetrics_StoreSuccessPoint) ProtoReflect() protoreflect.Message { - mi := &file_supernode_supernode_proto_msgTypes[19] + mi := &file_supernode_supernode_proto_msgTypes[25] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -1218,7 +1529,7 @@ type StatusResponse_P2PMetrics_DhtMetrics_BatchRetrievePoint struct { func (x *StatusResponse_P2PMetrics_DhtMetrics_BatchRetrievePoint) Reset() { *x = StatusResponse_P2PMetrics_DhtMetrics_BatchRetrievePoint{} - mi := &file_supernode_supernode_proto_msgTypes[20] + mi := &file_supernode_supernode_proto_msgTypes[26] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -1230,7 +1541,7 @@ func (x *StatusResponse_P2PMetrics_DhtMetrics_BatchRetrievePoint) String() strin func (*StatusResponse_P2PMetrics_DhtMetrics_BatchRetrievePoint) ProtoMessage() {} func (x *StatusResponse_P2PMetrics_DhtMetrics_BatchRetrievePoint) ProtoReflect() protoreflect.Message { - mi := &file_supernode_supernode_proto_msgTypes[20] + mi := &file_supernode_supernode_proto_msgTypes[26] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -1310,7 +1621,7 @@ var file_supernode_supernode_proto_rawDesc = []byte{ 0x0a, 0x0b, 0x53, 0x65, 0x72, 0x76, 0x69, 0x63, 0x65, 0x49, 0x6e, 0x66, 0x6f, 0x12, 0x12, 0x0a, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x12, 0x18, 0x0a, 0x07, 0x6d, 0x65, 0x74, 0x68, 0x6f, 0x64, 0x73, 0x18, 0x02, 0x20, 0x03, - 0x28, 0x09, 0x52, 0x07, 0x6d, 0x65, 0x74, 0x68, 0x6f, 0x64, 0x73, 0x22, 0x84, 0x19, 0x0a, 0x0e, + 0x28, 0x09, 0x52, 0x07, 0x6d, 0x65, 0x74, 0x68, 0x6f, 0x64, 0x73, 0x22, 0xf7, 0x23, 0x0a, 0x0e, 0x53, 0x74, 0x61, 0x74, 0x75, 0x73, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x18, 0x0a, 0x07, 0x76, 0x65, 0x72, 0x73, 0x69, 0x6f, 0x6e, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x07, 0x76, 0x65, 0x72, 0x73, 0x69, 0x6f, 0x6e, 0x12, 0x25, 0x0a, 0x0e, 0x75, 0x70, 0x74, 0x69, @@ -1391,7 +1702,7 @@ var file_supernode_supernode_proto_rawDesc = []byte{ 0x75, 0x6e, 0x74, 0x18, 0x01, 0x20, 0x01, 0x28, 0x05, 0x52, 0x0a, 0x70, 0x65, 0x65, 0x72, 0x73, 0x43, 0x6f, 0x75, 0x6e, 0x74, 0x12, 0x25, 0x0a, 0x0e, 0x70, 0x65, 0x65, 0x72, 0x5f, 0x61, 0x64, 0x64, 0x72, 0x65, 0x73, 0x73, 0x65, 0x73, 0x18, 0x02, 0x20, 0x03, 0x28, 0x09, 0x52, 0x0d, 0x70, - 0x65, 0x65, 0x72, 0x41, 0x64, 0x64, 0x72, 0x65, 0x73, 0x73, 0x65, 0x73, 0x1a, 0xf3, 0x0e, 0x0a, + 0x65, 0x65, 0x72, 0x41, 0x64, 0x64, 0x72, 0x65, 0x73, 0x73, 0x65, 0x73, 0x1a, 0xe6, 0x19, 0x0a, 0x0a, 0x50, 0x32, 0x50, 0x4d, 0x65, 0x74, 0x72, 0x69, 0x63, 0x73, 0x12, 0x50, 0x0a, 0x0b, 0x64, 0x68, 0x74, 0x5f, 0x6d, 0x65, 0x74, 0x72, 0x69, 0x63, 0x73, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x2f, 0x2e, 0x73, 0x75, 0x70, 0x65, 0x72, 0x6e, 0x6f, 0x64, 0x65, 0x2e, 0x53, 0x74, 0x61, @@ -1425,110 +1736,197 @@ var file_supernode_supernode_proto_rawDesc = []byte{ 0x28, 0x0b, 0x32, 0x2f, 0x2e, 0x73, 0x75, 0x70, 0x65, 0x72, 0x6e, 0x6f, 0x64, 0x65, 0x2e, 0x53, 0x74, 0x61, 0x74, 0x75, 0x73, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x2e, 0x50, 0x32, 0x50, 0x4d, 0x65, 0x74, 0x72, 0x69, 0x63, 0x73, 0x2e, 0x44, 0x69, 0x73, 0x6b, 0x53, 0x74, 0x61, - 0x74, 0x75, 0x73, 0x52, 0x04, 0x64, 0x69, 0x73, 0x6b, 0x1a, 0xc0, 0x05, 0x0a, 0x0a, 0x44, 0x68, - 0x74, 0x4d, 0x65, 0x74, 0x72, 0x69, 0x63, 0x73, 0x12, 0x73, 0x0a, 0x14, 0x73, 0x74, 0x6f, 0x72, - 0x65, 0x5f, 0x73, 0x75, 0x63, 0x63, 0x65, 0x73, 0x73, 0x5f, 0x72, 0x65, 0x63, 0x65, 0x6e, 0x74, - 0x18, 0x01, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x41, 0x2e, 0x73, 0x75, 0x70, 0x65, 0x72, 0x6e, 0x6f, - 0x64, 0x65, 0x2e, 0x53, 0x74, 0x61, 0x74, 0x75, 0x73, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, - 0x65, 0x2e, 0x50, 0x32, 0x50, 0x4d, 0x65, 0x74, 0x72, 0x69, 0x63, 0x73, 0x2e, 0x44, 0x68, 0x74, - 0x4d, 0x65, 0x74, 0x72, 0x69, 0x63, 0x73, 0x2e, 0x53, 0x74, 0x6f, 0x72, 0x65, 0x53, 0x75, 0x63, - 0x63, 0x65, 0x73, 0x73, 0x50, 0x6f, 0x69, 0x6e, 0x74, 0x52, 0x12, 0x73, 0x74, 0x6f, 0x72, 0x65, - 0x53, 0x75, 0x63, 0x63, 0x65, 0x73, 0x73, 0x52, 0x65, 0x63, 0x65, 0x6e, 0x74, 0x12, 0x76, 0x0a, - 0x15, 0x62, 0x61, 0x74, 0x63, 0x68, 0x5f, 0x72, 0x65, 0x74, 0x72, 0x69, 0x65, 0x76, 0x65, 0x5f, - 0x72, 0x65, 0x63, 0x65, 0x6e, 0x74, 0x18, 0x02, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x42, 0x2e, 0x73, + 0x74, 0x75, 0x73, 0x52, 0x04, 0x64, 0x69, 0x73, 0x6b, 0x12, 0x68, 0x0a, 0x12, 0x72, 0x65, 0x63, + 0x65, 0x6e, 0x74, 0x5f, 0x62, 0x61, 0x74, 0x63, 0x68, 0x5f, 0x73, 0x74, 0x6f, 0x72, 0x65, 0x18, + 0x07, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x3a, 0x2e, 0x73, 0x75, 0x70, 0x65, 0x72, 0x6e, 0x6f, 0x64, + 0x65, 0x2e, 0x53, 0x74, 0x61, 0x74, 0x75, 0x73, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, + 0x2e, 0x50, 0x32, 0x50, 0x4d, 0x65, 0x74, 0x72, 0x69, 0x63, 0x73, 0x2e, 0x52, 0x65, 0x63, 0x65, + 0x6e, 0x74, 0x42, 0x61, 0x74, 0x63, 0x68, 0x53, 0x74, 0x6f, 0x72, 0x65, 0x45, 0x6e, 0x74, 0x72, + 0x79, 0x52, 0x10, 0x72, 0x65, 0x63, 0x65, 0x6e, 0x74, 0x42, 0x61, 0x74, 0x63, 0x68, 0x53, 0x74, + 0x6f, 0x72, 0x65, 0x12, 0x71, 0x0a, 0x15, 0x72, 0x65, 0x63, 0x65, 0x6e, 0x74, 0x5f, 0x62, 0x61, + 0x74, 0x63, 0x68, 0x5f, 0x72, 0x65, 0x74, 0x72, 0x69, 0x65, 0x76, 0x65, 0x18, 0x08, 0x20, 0x03, + 0x28, 0x0b, 0x32, 0x3d, 0x2e, 0x73, 0x75, 0x70, 0x65, 0x72, 0x6e, 0x6f, 0x64, 0x65, 0x2e, 0x53, + 0x74, 0x61, 0x74, 0x75, 0x73, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x2e, 0x50, 0x32, + 0x50, 0x4d, 0x65, 0x74, 0x72, 0x69, 0x63, 0x73, 0x2e, 0x52, 0x65, 0x63, 0x65, 0x6e, 0x74, 0x42, + 0x61, 0x74, 0x63, 0x68, 0x52, 0x65, 0x74, 0x72, 0x69, 0x65, 0x76, 0x65, 0x45, 0x6e, 0x74, 0x72, + 0x79, 0x52, 0x13, 0x72, 0x65, 0x63, 0x65, 0x6e, 0x74, 0x42, 0x61, 0x74, 0x63, 0x68, 0x52, 0x65, + 0x74, 0x72, 0x69, 0x65, 0x76, 0x65, 0x12, 0x76, 0x0a, 0x18, 0x72, 0x65, 0x63, 0x65, 0x6e, 0x74, + 0x5f, 0x62, 0x61, 0x74, 0x63, 0x68, 0x5f, 0x73, 0x74, 0x6f, 0x72, 0x65, 0x5f, 0x62, 0x79, 0x5f, + 0x69, 0x70, 0x18, 0x09, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x3e, 0x2e, 0x73, 0x75, 0x70, 0x65, 0x72, + 0x6e, 0x6f, 0x64, 0x65, 0x2e, 0x53, 0x74, 0x61, 0x74, 0x75, 0x73, 0x52, 0x65, 0x73, 0x70, 0x6f, + 0x6e, 0x73, 0x65, 0x2e, 0x50, 0x32, 0x50, 0x4d, 0x65, 0x74, 0x72, 0x69, 0x63, 0x73, 0x2e, 0x52, + 0x65, 0x63, 0x65, 0x6e, 0x74, 0x42, 0x61, 0x74, 0x63, 0x68, 0x53, 0x74, 0x6f, 0x72, 0x65, 0x42, + 0x79, 0x49, 0x70, 0x45, 0x6e, 0x74, 0x72, 0x79, 0x52, 0x14, 0x72, 0x65, 0x63, 0x65, 0x6e, 0x74, + 0x42, 0x61, 0x74, 0x63, 0x68, 0x53, 0x74, 0x6f, 0x72, 0x65, 0x42, 0x79, 0x49, 0x70, 0x12, 0x7f, + 0x0a, 0x1b, 0x72, 0x65, 0x63, 0x65, 0x6e, 0x74, 0x5f, 0x62, 0x61, 0x74, 0x63, 0x68, 0x5f, 0x72, + 0x65, 0x74, 0x72, 0x69, 0x65, 0x76, 0x65, 0x5f, 0x62, 0x79, 0x5f, 0x69, 0x70, 0x18, 0x0a, 0x20, + 0x03, 0x28, 0x0b, 0x32, 0x41, 0x2e, 0x73, 0x75, 0x70, 0x65, 0x72, 0x6e, 0x6f, 0x64, 0x65, 0x2e, + 0x53, 0x74, 0x61, 0x74, 0x75, 0x73, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x2e, 0x50, + 0x32, 0x50, 0x4d, 0x65, 0x74, 0x72, 0x69, 0x63, 0x73, 0x2e, 0x52, 0x65, 0x63, 0x65, 0x6e, 0x74, + 0x42, 0x61, 0x74, 0x63, 0x68, 0x52, 0x65, 0x74, 0x72, 0x69, 0x65, 0x76, 0x65, 0x42, 0x79, 0x49, + 0x70, 0x45, 0x6e, 0x74, 0x72, 0x79, 0x52, 0x17, 0x72, 0x65, 0x63, 0x65, 0x6e, 0x74, 0x42, 0x61, + 0x74, 0x63, 0x68, 0x52, 0x65, 0x74, 0x72, 0x69, 0x65, 0x76, 0x65, 0x42, 0x79, 0x49, 0x70, 0x1a, + 0xc0, 0x05, 0x0a, 0x0a, 0x44, 0x68, 0x74, 0x4d, 0x65, 0x74, 0x72, 0x69, 0x63, 0x73, 0x12, 0x73, + 0x0a, 0x14, 0x73, 0x74, 0x6f, 0x72, 0x65, 0x5f, 0x73, 0x75, 0x63, 0x63, 0x65, 0x73, 0x73, 0x5f, + 0x72, 0x65, 0x63, 0x65, 0x6e, 0x74, 0x18, 0x01, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x41, 0x2e, 0x73, 0x75, 0x70, 0x65, 0x72, 0x6e, 0x6f, 0x64, 0x65, 0x2e, 0x53, 0x74, 0x61, 0x74, 0x75, 0x73, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x2e, 0x50, 0x32, 0x50, 0x4d, 0x65, 0x74, 0x72, 0x69, - 0x63, 0x73, 0x2e, 0x44, 0x68, 0x74, 0x4d, 0x65, 0x74, 0x72, 0x69, 0x63, 0x73, 0x2e, 0x42, 0x61, - 0x74, 0x63, 0x68, 0x52, 0x65, 0x74, 0x72, 0x69, 0x65, 0x76, 0x65, 0x50, 0x6f, 0x69, 0x6e, 0x74, - 0x52, 0x13, 0x62, 0x61, 0x74, 0x63, 0x68, 0x52, 0x65, 0x74, 0x72, 0x69, 0x65, 0x76, 0x65, 0x52, - 0x65, 0x63, 0x65, 0x6e, 0x74, 0x12, 0x31, 0x0a, 0x15, 0x68, 0x6f, 0x74, 0x5f, 0x70, 0x61, 0x74, - 0x68, 0x5f, 0x62, 0x61, 0x6e, 0x6e, 0x65, 0x64, 0x5f, 0x73, 0x6b, 0x69, 0x70, 0x73, 0x18, 0x03, - 0x20, 0x01, 0x28, 0x03, 0x52, 0x12, 0x68, 0x6f, 0x74, 0x50, 0x61, 0x74, 0x68, 0x42, 0x61, 0x6e, - 0x6e, 0x65, 0x64, 0x53, 0x6b, 0x69, 0x70, 0x73, 0x12, 0x35, 0x0a, 0x17, 0x68, 0x6f, 0x74, 0x5f, - 0x70, 0x61, 0x74, 0x68, 0x5f, 0x62, 0x61, 0x6e, 0x5f, 0x69, 0x6e, 0x63, 0x72, 0x65, 0x6d, 0x65, - 0x6e, 0x74, 0x73, 0x18, 0x04, 0x20, 0x01, 0x28, 0x03, 0x52, 0x14, 0x68, 0x6f, 0x74, 0x50, 0x61, - 0x74, 0x68, 0x42, 0x61, 0x6e, 0x49, 0x6e, 0x63, 0x72, 0x65, 0x6d, 0x65, 0x6e, 0x74, 0x73, 0x1a, - 0x8f, 0x01, 0x0a, 0x11, 0x53, 0x74, 0x6f, 0x72, 0x65, 0x53, 0x75, 0x63, 0x63, 0x65, 0x73, 0x73, - 0x50, 0x6f, 0x69, 0x6e, 0x74, 0x12, 0x1b, 0x0a, 0x09, 0x74, 0x69, 0x6d, 0x65, 0x5f, 0x75, 0x6e, - 0x69, 0x78, 0x18, 0x01, 0x20, 0x01, 0x28, 0x03, 0x52, 0x08, 0x74, 0x69, 0x6d, 0x65, 0x55, 0x6e, - 0x69, 0x78, 0x12, 0x1a, 0x0a, 0x08, 0x72, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x73, 0x18, 0x02, - 0x20, 0x01, 0x28, 0x05, 0x52, 0x08, 0x72, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x73, 0x12, 0x1e, - 0x0a, 0x0a, 0x73, 0x75, 0x63, 0x63, 0x65, 0x73, 0x73, 0x66, 0x75, 0x6c, 0x18, 0x03, 0x20, 0x01, - 0x28, 0x05, 0x52, 0x0a, 0x73, 0x75, 0x63, 0x63, 0x65, 0x73, 0x73, 0x66, 0x75, 0x6c, 0x12, 0x21, - 0x0a, 0x0c, 0x73, 0x75, 0x63, 0x63, 0x65, 0x73, 0x73, 0x5f, 0x72, 0x61, 0x74, 0x65, 0x18, 0x04, - 0x20, 0x01, 0x28, 0x01, 0x52, 0x0b, 0x73, 0x75, 0x63, 0x63, 0x65, 0x73, 0x73, 0x52, 0x61, 0x74, - 0x65, 0x1a, 0xc8, 0x01, 0x0a, 0x12, 0x42, 0x61, 0x74, 0x63, 0x68, 0x52, 0x65, 0x74, 0x72, 0x69, - 0x65, 0x76, 0x65, 0x50, 0x6f, 0x69, 0x6e, 0x74, 0x12, 0x1b, 0x0a, 0x09, 0x74, 0x69, 0x6d, 0x65, - 0x5f, 0x75, 0x6e, 0x69, 0x78, 0x18, 0x01, 0x20, 0x01, 0x28, 0x03, 0x52, 0x08, 0x74, 0x69, 0x6d, - 0x65, 0x55, 0x6e, 0x69, 0x78, 0x12, 0x12, 0x0a, 0x04, 0x6b, 0x65, 0x79, 0x73, 0x18, 0x02, 0x20, - 0x01, 0x28, 0x05, 0x52, 0x04, 0x6b, 0x65, 0x79, 0x73, 0x12, 0x1a, 0x0a, 0x08, 0x72, 0x65, 0x71, - 0x75, 0x69, 0x72, 0x65, 0x64, 0x18, 0x03, 0x20, 0x01, 0x28, 0x05, 0x52, 0x08, 0x72, 0x65, 0x71, - 0x75, 0x69, 0x72, 0x65, 0x64, 0x12, 0x1f, 0x0a, 0x0b, 0x66, 0x6f, 0x75, 0x6e, 0x64, 0x5f, 0x6c, - 0x6f, 0x63, 0x61, 0x6c, 0x18, 0x04, 0x20, 0x01, 0x28, 0x05, 0x52, 0x0a, 0x66, 0x6f, 0x75, 0x6e, - 0x64, 0x4c, 0x6f, 0x63, 0x61, 0x6c, 0x12, 0x23, 0x0a, 0x0d, 0x66, 0x6f, 0x75, 0x6e, 0x64, 0x5f, - 0x6e, 0x65, 0x74, 0x77, 0x6f, 0x72, 0x6b, 0x18, 0x05, 0x20, 0x01, 0x28, 0x05, 0x52, 0x0c, 0x66, - 0x6f, 0x75, 0x6e, 0x64, 0x4e, 0x65, 0x74, 0x77, 0x6f, 0x72, 0x6b, 0x12, 0x1f, 0x0a, 0x0b, 0x64, - 0x75, 0x72, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x5f, 0x6d, 0x73, 0x18, 0x06, 0x20, 0x01, 0x28, 0x03, - 0x52, 0x0a, 0x64, 0x75, 0x72, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x4d, 0x73, 0x1a, 0x74, 0x0a, 0x0e, - 0x48, 0x61, 0x6e, 0x64, 0x6c, 0x65, 0x43, 0x6f, 0x75, 0x6e, 0x74, 0x65, 0x72, 0x73, 0x12, 0x14, - 0x0a, 0x05, 0x74, 0x6f, 0x74, 0x61, 0x6c, 0x18, 0x01, 0x20, 0x01, 0x28, 0x03, 0x52, 0x05, 0x74, - 0x6f, 0x74, 0x61, 0x6c, 0x12, 0x18, 0x0a, 0x07, 0x73, 0x75, 0x63, 0x63, 0x65, 0x73, 0x73, 0x18, - 0x02, 0x20, 0x01, 0x28, 0x03, 0x52, 0x07, 0x73, 0x75, 0x63, 0x63, 0x65, 0x73, 0x73, 0x12, 0x18, - 0x0a, 0x07, 0x66, 0x61, 0x69, 0x6c, 0x75, 0x72, 0x65, 0x18, 0x03, 0x20, 0x01, 0x28, 0x03, 0x52, - 0x07, 0x66, 0x61, 0x69, 0x6c, 0x75, 0x72, 0x65, 0x12, 0x18, 0x0a, 0x07, 0x74, 0x69, 0x6d, 0x65, - 0x6f, 0x75, 0x74, 0x18, 0x04, 0x20, 0x01, 0x28, 0x03, 0x52, 0x07, 0x74, 0x69, 0x6d, 0x65, 0x6f, - 0x75, 0x74, 0x1a, 0x9d, 0x01, 0x0a, 0x08, 0x42, 0x61, 0x6e, 0x45, 0x6e, 0x74, 0x72, 0x79, 0x12, - 0x0e, 0x0a, 0x02, 0x69, 0x64, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x02, 0x69, 0x64, 0x12, - 0x0e, 0x0a, 0x02, 0x69, 0x70, 0x18, 0x02, 0x20, 0x01, 0x28, 0x09, 0x52, 0x02, 0x69, 0x70, 0x12, - 0x12, 0x0a, 0x04, 0x70, 0x6f, 0x72, 0x74, 0x18, 0x03, 0x20, 0x01, 0x28, 0x0d, 0x52, 0x04, 0x70, - 0x6f, 0x72, 0x74, 0x12, 0x14, 0x0a, 0x05, 0x63, 0x6f, 0x75, 0x6e, 0x74, 0x18, 0x04, 0x20, 0x01, - 0x28, 0x05, 0x52, 0x05, 0x63, 0x6f, 0x75, 0x6e, 0x74, 0x12, 0x26, 0x0a, 0x0f, 0x63, 0x72, 0x65, - 0x61, 0x74, 0x65, 0x64, 0x5f, 0x61, 0x74, 0x5f, 0x75, 0x6e, 0x69, 0x78, 0x18, 0x05, 0x20, 0x01, - 0x28, 0x03, 0x52, 0x0d, 0x63, 0x72, 0x65, 0x61, 0x74, 0x65, 0x64, 0x41, 0x74, 0x55, 0x6e, 0x69, - 0x78, 0x12, 0x1f, 0x0a, 0x0b, 0x61, 0x67, 0x65, 0x5f, 0x73, 0x65, 0x63, 0x6f, 0x6e, 0x64, 0x73, - 0x18, 0x06, 0x20, 0x01, 0x28, 0x03, 0x52, 0x0a, 0x61, 0x67, 0x65, 0x53, 0x65, 0x63, 0x6f, 0x6e, - 0x64, 0x73, 0x1a, 0x65, 0x0a, 0x0d, 0x44, 0x61, 0x74, 0x61, 0x62, 0x61, 0x73, 0x65, 0x53, 0x74, - 0x61, 0x74, 0x73, 0x12, 0x23, 0x0a, 0x0e, 0x70, 0x32, 0x70, 0x5f, 0x64, 0x62, 0x5f, 0x73, 0x69, - 0x7a, 0x65, 0x5f, 0x6d, 0x62, 0x18, 0x01, 0x20, 0x01, 0x28, 0x01, 0x52, 0x0b, 0x70, 0x32, 0x70, - 0x44, 0x62, 0x53, 0x69, 0x7a, 0x65, 0x4d, 0x62, 0x12, 0x2f, 0x0a, 0x14, 0x70, 0x32, 0x70, 0x5f, - 0x64, 0x62, 0x5f, 0x72, 0x65, 0x63, 0x6f, 0x72, 0x64, 0x73, 0x5f, 0x63, 0x6f, 0x75, 0x6e, 0x74, - 0x18, 0x02, 0x20, 0x01, 0x28, 0x03, 0x52, 0x11, 0x70, 0x32, 0x70, 0x44, 0x62, 0x52, 0x65, 0x63, - 0x6f, 0x72, 0x64, 0x73, 0x43, 0x6f, 0x75, 0x6e, 0x74, 0x1a, 0x55, 0x0a, 0x0a, 0x44, 0x69, 0x73, - 0x6b, 0x53, 0x74, 0x61, 0x74, 0x75, 0x73, 0x12, 0x15, 0x0a, 0x06, 0x61, 0x6c, 0x6c, 0x5f, 0x6d, - 0x62, 0x18, 0x01, 0x20, 0x01, 0x28, 0x01, 0x52, 0x05, 0x61, 0x6c, 0x6c, 0x4d, 0x62, 0x12, 0x17, - 0x0a, 0x07, 0x75, 0x73, 0x65, 0x64, 0x5f, 0x6d, 0x62, 0x18, 0x02, 0x20, 0x01, 0x28, 0x01, 0x52, - 0x06, 0x75, 0x73, 0x65, 0x64, 0x4d, 0x62, 0x12, 0x17, 0x0a, 0x07, 0x66, 0x72, 0x65, 0x65, 0x5f, - 0x6d, 0x62, 0x18, 0x03, 0x20, 0x01, 0x28, 0x01, 0x52, 0x06, 0x66, 0x72, 0x65, 0x65, 0x4d, 0x62, - 0x1a, 0x7c, 0x0a, 0x19, 0x4e, 0x65, 0x74, 0x77, 0x6f, 0x72, 0x6b, 0x48, 0x61, 0x6e, 0x64, 0x6c, - 0x65, 0x4d, 0x65, 0x74, 0x72, 0x69, 0x63, 0x73, 0x45, 0x6e, 0x74, 0x72, 0x79, 0x12, 0x10, 0x0a, - 0x03, 0x6b, 0x65, 0x79, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x03, 0x6b, 0x65, 0x79, 0x12, - 0x49, 0x0a, 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x18, 0x02, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x33, - 0x2e, 0x73, 0x75, 0x70, 0x65, 0x72, 0x6e, 0x6f, 0x64, 0x65, 0x2e, 0x53, 0x74, 0x61, 0x74, 0x75, - 0x73, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x2e, 0x50, 0x32, 0x50, 0x4d, 0x65, 0x74, - 0x72, 0x69, 0x63, 0x73, 0x2e, 0x48, 0x61, 0x6e, 0x64, 0x6c, 0x65, 0x43, 0x6f, 0x75, 0x6e, 0x74, - 0x65, 0x72, 0x73, 0x52, 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x3a, 0x02, 0x38, 0x01, 0x1a, 0x42, - 0x0a, 0x14, 0x43, 0x6f, 0x6e, 0x6e, 0x50, 0x6f, 0x6f, 0x6c, 0x4d, 0x65, 0x74, 0x72, 0x69, 0x63, - 0x73, 0x45, 0x6e, 0x74, 0x72, 0x79, 0x12, 0x10, 0x0a, 0x03, 0x6b, 0x65, 0x79, 0x18, 0x01, 0x20, - 0x01, 0x28, 0x09, 0x52, 0x03, 0x6b, 0x65, 0x79, 0x12, 0x14, 0x0a, 0x05, 0x76, 0x61, 0x6c, 0x75, - 0x65, 0x18, 0x02, 0x20, 0x01, 0x28, 0x03, 0x52, 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x3a, 0x02, - 0x38, 0x01, 0x32, 0xd7, 0x01, 0x0a, 0x10, 0x53, 0x75, 0x70, 0x65, 0x72, 0x6e, 0x6f, 0x64, 0x65, - 0x53, 0x65, 0x72, 0x76, 0x69, 0x63, 0x65, 0x12, 0x58, 0x0a, 0x09, 0x47, 0x65, 0x74, 0x53, 0x74, - 0x61, 0x74, 0x75, 0x73, 0x12, 0x18, 0x2e, 0x73, 0x75, 0x70, 0x65, 0x72, 0x6e, 0x6f, 0x64, 0x65, - 0x2e, 0x53, 0x74, 0x61, 0x74, 0x75, 0x73, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x1a, 0x19, - 0x2e, 0x73, 0x75, 0x70, 0x65, 0x72, 0x6e, 0x6f, 0x64, 0x65, 0x2e, 0x53, 0x74, 0x61, 0x74, 0x75, - 0x73, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x22, 0x16, 0x82, 0xd3, 0xe4, 0x93, 0x02, - 0x10, 0x12, 0x0e, 0x2f, 0x61, 0x70, 0x69, 0x2f, 0x76, 0x31, 0x2f, 0x73, 0x74, 0x61, 0x74, 0x75, - 0x73, 0x12, 0x69, 0x0a, 0x0c, 0x4c, 0x69, 0x73, 0x74, 0x53, 0x65, 0x72, 0x76, 0x69, 0x63, 0x65, - 0x73, 0x12, 0x1e, 0x2e, 0x73, 0x75, 0x70, 0x65, 0x72, 0x6e, 0x6f, 0x64, 0x65, 0x2e, 0x4c, 0x69, - 0x73, 0x74, 0x53, 0x65, 0x72, 0x76, 0x69, 0x63, 0x65, 0x73, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, - 0x74, 0x1a, 0x1f, 0x2e, 0x73, 0x75, 0x70, 0x65, 0x72, 0x6e, 0x6f, 0x64, 0x65, 0x2e, 0x4c, 0x69, - 0x73, 0x74, 0x53, 0x65, 0x72, 0x76, 0x69, 0x63, 0x65, 0x73, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, - 0x73, 0x65, 0x22, 0x18, 0x82, 0xd3, 0xe4, 0x93, 0x02, 0x12, 0x12, 0x10, 0x2f, 0x61, 0x70, 0x69, - 0x2f, 0x76, 0x31, 0x2f, 0x73, 0x65, 0x72, 0x76, 0x69, 0x63, 0x65, 0x73, 0x42, 0x36, 0x5a, 0x34, - 0x67, 0x69, 0x74, 0x68, 0x75, 0x62, 0x2e, 0x63, 0x6f, 0x6d, 0x2f, 0x4c, 0x75, 0x6d, 0x65, 0x72, - 0x61, 0x50, 0x72, 0x6f, 0x74, 0x6f, 0x63, 0x6f, 0x6c, 0x2f, 0x73, 0x75, 0x70, 0x65, 0x72, 0x6e, - 0x6f, 0x64, 0x65, 0x2f, 0x76, 0x32, 0x2f, 0x67, 0x65, 0x6e, 0x2f, 0x73, 0x75, 0x70, 0x65, 0x72, - 0x6e, 0x6f, 0x64, 0x65, 0x62, 0x06, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x33, + 0x63, 0x73, 0x2e, 0x44, 0x68, 0x74, 0x4d, 0x65, 0x74, 0x72, 0x69, 0x63, 0x73, 0x2e, 0x53, 0x74, + 0x6f, 0x72, 0x65, 0x53, 0x75, 0x63, 0x63, 0x65, 0x73, 0x73, 0x50, 0x6f, 0x69, 0x6e, 0x74, 0x52, + 0x12, 0x73, 0x74, 0x6f, 0x72, 0x65, 0x53, 0x75, 0x63, 0x63, 0x65, 0x73, 0x73, 0x52, 0x65, 0x63, + 0x65, 0x6e, 0x74, 0x12, 0x76, 0x0a, 0x15, 0x62, 0x61, 0x74, 0x63, 0x68, 0x5f, 0x72, 0x65, 0x74, + 0x72, 0x69, 0x65, 0x76, 0x65, 0x5f, 0x72, 0x65, 0x63, 0x65, 0x6e, 0x74, 0x18, 0x02, 0x20, 0x03, + 0x28, 0x0b, 0x32, 0x42, 0x2e, 0x73, 0x75, 0x70, 0x65, 0x72, 0x6e, 0x6f, 0x64, 0x65, 0x2e, 0x53, + 0x74, 0x61, 0x74, 0x75, 0x73, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x2e, 0x50, 0x32, + 0x50, 0x4d, 0x65, 0x74, 0x72, 0x69, 0x63, 0x73, 0x2e, 0x44, 0x68, 0x74, 0x4d, 0x65, 0x74, 0x72, + 0x69, 0x63, 0x73, 0x2e, 0x42, 0x61, 0x74, 0x63, 0x68, 0x52, 0x65, 0x74, 0x72, 0x69, 0x65, 0x76, + 0x65, 0x50, 0x6f, 0x69, 0x6e, 0x74, 0x52, 0x13, 0x62, 0x61, 0x74, 0x63, 0x68, 0x52, 0x65, 0x74, + 0x72, 0x69, 0x65, 0x76, 0x65, 0x52, 0x65, 0x63, 0x65, 0x6e, 0x74, 0x12, 0x31, 0x0a, 0x15, 0x68, + 0x6f, 0x74, 0x5f, 0x70, 0x61, 0x74, 0x68, 0x5f, 0x62, 0x61, 0x6e, 0x6e, 0x65, 0x64, 0x5f, 0x73, + 0x6b, 0x69, 0x70, 0x73, 0x18, 0x03, 0x20, 0x01, 0x28, 0x03, 0x52, 0x12, 0x68, 0x6f, 0x74, 0x50, + 0x61, 0x74, 0x68, 0x42, 0x61, 0x6e, 0x6e, 0x65, 0x64, 0x53, 0x6b, 0x69, 0x70, 0x73, 0x12, 0x35, + 0x0a, 0x17, 0x68, 0x6f, 0x74, 0x5f, 0x70, 0x61, 0x74, 0x68, 0x5f, 0x62, 0x61, 0x6e, 0x5f, 0x69, + 0x6e, 0x63, 0x72, 0x65, 0x6d, 0x65, 0x6e, 0x74, 0x73, 0x18, 0x04, 0x20, 0x01, 0x28, 0x03, 0x52, + 0x14, 0x68, 0x6f, 0x74, 0x50, 0x61, 0x74, 0x68, 0x42, 0x61, 0x6e, 0x49, 0x6e, 0x63, 0x72, 0x65, + 0x6d, 0x65, 0x6e, 0x74, 0x73, 0x1a, 0x8f, 0x01, 0x0a, 0x11, 0x53, 0x74, 0x6f, 0x72, 0x65, 0x53, + 0x75, 0x63, 0x63, 0x65, 0x73, 0x73, 0x50, 0x6f, 0x69, 0x6e, 0x74, 0x12, 0x1b, 0x0a, 0x09, 0x74, + 0x69, 0x6d, 0x65, 0x5f, 0x75, 0x6e, 0x69, 0x78, 0x18, 0x01, 0x20, 0x01, 0x28, 0x03, 0x52, 0x08, + 0x74, 0x69, 0x6d, 0x65, 0x55, 0x6e, 0x69, 0x78, 0x12, 0x1a, 0x0a, 0x08, 0x72, 0x65, 0x71, 0x75, + 0x65, 0x73, 0x74, 0x73, 0x18, 0x02, 0x20, 0x01, 0x28, 0x05, 0x52, 0x08, 0x72, 0x65, 0x71, 0x75, + 0x65, 0x73, 0x74, 0x73, 0x12, 0x1e, 0x0a, 0x0a, 0x73, 0x75, 0x63, 0x63, 0x65, 0x73, 0x73, 0x66, + 0x75, 0x6c, 0x18, 0x03, 0x20, 0x01, 0x28, 0x05, 0x52, 0x0a, 0x73, 0x75, 0x63, 0x63, 0x65, 0x73, + 0x73, 0x66, 0x75, 0x6c, 0x12, 0x21, 0x0a, 0x0c, 0x73, 0x75, 0x63, 0x63, 0x65, 0x73, 0x73, 0x5f, + 0x72, 0x61, 0x74, 0x65, 0x18, 0x04, 0x20, 0x01, 0x28, 0x01, 0x52, 0x0b, 0x73, 0x75, 0x63, 0x63, + 0x65, 0x73, 0x73, 0x52, 0x61, 0x74, 0x65, 0x1a, 0xc8, 0x01, 0x0a, 0x12, 0x42, 0x61, 0x74, 0x63, + 0x68, 0x52, 0x65, 0x74, 0x72, 0x69, 0x65, 0x76, 0x65, 0x50, 0x6f, 0x69, 0x6e, 0x74, 0x12, 0x1b, + 0x0a, 0x09, 0x74, 0x69, 0x6d, 0x65, 0x5f, 0x75, 0x6e, 0x69, 0x78, 0x18, 0x01, 0x20, 0x01, 0x28, + 0x03, 0x52, 0x08, 0x74, 0x69, 0x6d, 0x65, 0x55, 0x6e, 0x69, 0x78, 0x12, 0x12, 0x0a, 0x04, 0x6b, + 0x65, 0x79, 0x73, 0x18, 0x02, 0x20, 0x01, 0x28, 0x05, 0x52, 0x04, 0x6b, 0x65, 0x79, 0x73, 0x12, + 0x1a, 0x0a, 0x08, 0x72, 0x65, 0x71, 0x75, 0x69, 0x72, 0x65, 0x64, 0x18, 0x03, 0x20, 0x01, 0x28, + 0x05, 0x52, 0x08, 0x72, 0x65, 0x71, 0x75, 0x69, 0x72, 0x65, 0x64, 0x12, 0x1f, 0x0a, 0x0b, 0x66, + 0x6f, 0x75, 0x6e, 0x64, 0x5f, 0x6c, 0x6f, 0x63, 0x61, 0x6c, 0x18, 0x04, 0x20, 0x01, 0x28, 0x05, + 0x52, 0x0a, 0x66, 0x6f, 0x75, 0x6e, 0x64, 0x4c, 0x6f, 0x63, 0x61, 0x6c, 0x12, 0x23, 0x0a, 0x0d, + 0x66, 0x6f, 0x75, 0x6e, 0x64, 0x5f, 0x6e, 0x65, 0x74, 0x77, 0x6f, 0x72, 0x6b, 0x18, 0x05, 0x20, + 0x01, 0x28, 0x05, 0x52, 0x0c, 0x66, 0x6f, 0x75, 0x6e, 0x64, 0x4e, 0x65, 0x74, 0x77, 0x6f, 0x72, + 0x6b, 0x12, 0x1f, 0x0a, 0x0b, 0x64, 0x75, 0x72, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x5f, 0x6d, 0x73, + 0x18, 0x06, 0x20, 0x01, 0x28, 0x03, 0x52, 0x0a, 0x64, 0x75, 0x72, 0x61, 0x74, 0x69, 0x6f, 0x6e, + 0x4d, 0x73, 0x1a, 0x74, 0x0a, 0x0e, 0x48, 0x61, 0x6e, 0x64, 0x6c, 0x65, 0x43, 0x6f, 0x75, 0x6e, + 0x74, 0x65, 0x72, 0x73, 0x12, 0x14, 0x0a, 0x05, 0x74, 0x6f, 0x74, 0x61, 0x6c, 0x18, 0x01, 0x20, + 0x01, 0x28, 0x03, 0x52, 0x05, 0x74, 0x6f, 0x74, 0x61, 0x6c, 0x12, 0x18, 0x0a, 0x07, 0x73, 0x75, + 0x63, 0x63, 0x65, 0x73, 0x73, 0x18, 0x02, 0x20, 0x01, 0x28, 0x03, 0x52, 0x07, 0x73, 0x75, 0x63, + 0x63, 0x65, 0x73, 0x73, 0x12, 0x18, 0x0a, 0x07, 0x66, 0x61, 0x69, 0x6c, 0x75, 0x72, 0x65, 0x18, + 0x03, 0x20, 0x01, 0x28, 0x03, 0x52, 0x07, 0x66, 0x61, 0x69, 0x6c, 0x75, 0x72, 0x65, 0x12, 0x18, + 0x0a, 0x07, 0x74, 0x69, 0x6d, 0x65, 0x6f, 0x75, 0x74, 0x18, 0x04, 0x20, 0x01, 0x28, 0x03, 0x52, + 0x07, 0x74, 0x69, 0x6d, 0x65, 0x6f, 0x75, 0x74, 0x1a, 0x9d, 0x01, 0x0a, 0x08, 0x42, 0x61, 0x6e, + 0x45, 0x6e, 0x74, 0x72, 0x79, 0x12, 0x0e, 0x0a, 0x02, 0x69, 0x64, 0x18, 0x01, 0x20, 0x01, 0x28, + 0x09, 0x52, 0x02, 0x69, 0x64, 0x12, 0x0e, 0x0a, 0x02, 0x69, 0x70, 0x18, 0x02, 0x20, 0x01, 0x28, + 0x09, 0x52, 0x02, 0x69, 0x70, 0x12, 0x12, 0x0a, 0x04, 0x70, 0x6f, 0x72, 0x74, 0x18, 0x03, 0x20, + 0x01, 0x28, 0x0d, 0x52, 0x04, 0x70, 0x6f, 0x72, 0x74, 0x12, 0x14, 0x0a, 0x05, 0x63, 0x6f, 0x75, + 0x6e, 0x74, 0x18, 0x04, 0x20, 0x01, 0x28, 0x05, 0x52, 0x05, 0x63, 0x6f, 0x75, 0x6e, 0x74, 0x12, + 0x26, 0x0a, 0x0f, 0x63, 0x72, 0x65, 0x61, 0x74, 0x65, 0x64, 0x5f, 0x61, 0x74, 0x5f, 0x75, 0x6e, + 0x69, 0x78, 0x18, 0x05, 0x20, 0x01, 0x28, 0x03, 0x52, 0x0d, 0x63, 0x72, 0x65, 0x61, 0x74, 0x65, + 0x64, 0x41, 0x74, 0x55, 0x6e, 0x69, 0x78, 0x12, 0x1f, 0x0a, 0x0b, 0x61, 0x67, 0x65, 0x5f, 0x73, + 0x65, 0x63, 0x6f, 0x6e, 0x64, 0x73, 0x18, 0x06, 0x20, 0x01, 0x28, 0x03, 0x52, 0x0a, 0x61, 0x67, + 0x65, 0x53, 0x65, 0x63, 0x6f, 0x6e, 0x64, 0x73, 0x1a, 0x65, 0x0a, 0x0d, 0x44, 0x61, 0x74, 0x61, + 0x62, 0x61, 0x73, 0x65, 0x53, 0x74, 0x61, 0x74, 0x73, 0x12, 0x23, 0x0a, 0x0e, 0x70, 0x32, 0x70, + 0x5f, 0x64, 0x62, 0x5f, 0x73, 0x69, 0x7a, 0x65, 0x5f, 0x6d, 0x62, 0x18, 0x01, 0x20, 0x01, 0x28, + 0x01, 0x52, 0x0b, 0x70, 0x32, 0x70, 0x44, 0x62, 0x53, 0x69, 0x7a, 0x65, 0x4d, 0x62, 0x12, 0x2f, + 0x0a, 0x14, 0x70, 0x32, 0x70, 0x5f, 0x64, 0x62, 0x5f, 0x72, 0x65, 0x63, 0x6f, 0x72, 0x64, 0x73, + 0x5f, 0x63, 0x6f, 0x75, 0x6e, 0x74, 0x18, 0x02, 0x20, 0x01, 0x28, 0x03, 0x52, 0x11, 0x70, 0x32, + 0x70, 0x44, 0x62, 0x52, 0x65, 0x63, 0x6f, 0x72, 0x64, 0x73, 0x43, 0x6f, 0x75, 0x6e, 0x74, 0x1a, + 0x55, 0x0a, 0x0a, 0x44, 0x69, 0x73, 0x6b, 0x53, 0x74, 0x61, 0x74, 0x75, 0x73, 0x12, 0x15, 0x0a, + 0x06, 0x61, 0x6c, 0x6c, 0x5f, 0x6d, 0x62, 0x18, 0x01, 0x20, 0x01, 0x28, 0x01, 0x52, 0x05, 0x61, + 0x6c, 0x6c, 0x4d, 0x62, 0x12, 0x17, 0x0a, 0x07, 0x75, 0x73, 0x65, 0x64, 0x5f, 0x6d, 0x62, 0x18, + 0x02, 0x20, 0x01, 0x28, 0x01, 0x52, 0x06, 0x75, 0x73, 0x65, 0x64, 0x4d, 0x62, 0x12, 0x17, 0x0a, + 0x07, 0x66, 0x72, 0x65, 0x65, 0x5f, 0x6d, 0x62, 0x18, 0x03, 0x20, 0x01, 0x28, 0x01, 0x52, 0x06, + 0x66, 0x72, 0x65, 0x65, 0x4d, 0x62, 0x1a, 0x7c, 0x0a, 0x19, 0x4e, 0x65, 0x74, 0x77, 0x6f, 0x72, + 0x6b, 0x48, 0x61, 0x6e, 0x64, 0x6c, 0x65, 0x4d, 0x65, 0x74, 0x72, 0x69, 0x63, 0x73, 0x45, 0x6e, + 0x74, 0x72, 0x79, 0x12, 0x10, 0x0a, 0x03, 0x6b, 0x65, 0x79, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, + 0x52, 0x03, 0x6b, 0x65, 0x79, 0x12, 0x49, 0x0a, 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x18, 0x02, + 0x20, 0x01, 0x28, 0x0b, 0x32, 0x33, 0x2e, 0x73, 0x75, 0x70, 0x65, 0x72, 0x6e, 0x6f, 0x64, 0x65, + 0x2e, 0x53, 0x74, 0x61, 0x74, 0x75, 0x73, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x2e, + 0x50, 0x32, 0x50, 0x4d, 0x65, 0x74, 0x72, 0x69, 0x63, 0x73, 0x2e, 0x48, 0x61, 0x6e, 0x64, 0x6c, + 0x65, 0x43, 0x6f, 0x75, 0x6e, 0x74, 0x65, 0x72, 0x73, 0x52, 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, + 0x3a, 0x02, 0x38, 0x01, 0x1a, 0x42, 0x0a, 0x14, 0x43, 0x6f, 0x6e, 0x6e, 0x50, 0x6f, 0x6f, 0x6c, + 0x4d, 0x65, 0x74, 0x72, 0x69, 0x63, 0x73, 0x45, 0x6e, 0x74, 0x72, 0x79, 0x12, 0x10, 0x0a, 0x03, + 0x6b, 0x65, 0x79, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x03, 0x6b, 0x65, 0x79, 0x12, 0x14, + 0x0a, 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x18, 0x02, 0x20, 0x01, 0x28, 0x03, 0x52, 0x05, 0x76, + 0x61, 0x6c, 0x75, 0x65, 0x3a, 0x02, 0x38, 0x01, 0x1a, 0xc9, 0x01, 0x0a, 0x15, 0x52, 0x65, 0x63, + 0x65, 0x6e, 0x74, 0x42, 0x61, 0x74, 0x63, 0x68, 0x53, 0x74, 0x6f, 0x72, 0x65, 0x45, 0x6e, 0x74, + 0x72, 0x79, 0x12, 0x1b, 0x0a, 0x09, 0x74, 0x69, 0x6d, 0x65, 0x5f, 0x75, 0x6e, 0x69, 0x78, 0x18, + 0x01, 0x20, 0x01, 0x28, 0x03, 0x52, 0x08, 0x74, 0x69, 0x6d, 0x65, 0x55, 0x6e, 0x69, 0x78, 0x12, + 0x1b, 0x0a, 0x09, 0x73, 0x65, 0x6e, 0x64, 0x65, 0x72, 0x5f, 0x69, 0x64, 0x18, 0x02, 0x20, 0x01, + 0x28, 0x09, 0x52, 0x08, 0x73, 0x65, 0x6e, 0x64, 0x65, 0x72, 0x49, 0x64, 0x12, 0x1b, 0x0a, 0x09, + 0x73, 0x65, 0x6e, 0x64, 0x65, 0x72, 0x5f, 0x69, 0x70, 0x18, 0x03, 0x20, 0x01, 0x28, 0x09, 0x52, + 0x08, 0x73, 0x65, 0x6e, 0x64, 0x65, 0x72, 0x49, 0x70, 0x12, 0x12, 0x0a, 0x04, 0x6b, 0x65, 0x79, + 0x73, 0x18, 0x04, 0x20, 0x01, 0x28, 0x05, 0x52, 0x04, 0x6b, 0x65, 0x79, 0x73, 0x12, 0x1f, 0x0a, + 0x0b, 0x64, 0x75, 0x72, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x5f, 0x6d, 0x73, 0x18, 0x05, 0x20, 0x01, + 0x28, 0x03, 0x52, 0x0a, 0x64, 0x75, 0x72, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x4d, 0x73, 0x12, 0x0e, + 0x0a, 0x02, 0x6f, 0x6b, 0x18, 0x06, 0x20, 0x01, 0x28, 0x08, 0x52, 0x02, 0x6f, 0x6b, 0x12, 0x14, + 0x0a, 0x05, 0x65, 0x72, 0x72, 0x6f, 0x72, 0x18, 0x07, 0x20, 0x01, 0x28, 0x09, 0x52, 0x05, 0x65, + 0x72, 0x72, 0x6f, 0x72, 0x1a, 0xdc, 0x01, 0x0a, 0x18, 0x52, 0x65, 0x63, 0x65, 0x6e, 0x74, 0x42, + 0x61, 0x74, 0x63, 0x68, 0x52, 0x65, 0x74, 0x72, 0x69, 0x65, 0x76, 0x65, 0x45, 0x6e, 0x74, 0x72, + 0x79, 0x12, 0x1b, 0x0a, 0x09, 0x74, 0x69, 0x6d, 0x65, 0x5f, 0x75, 0x6e, 0x69, 0x78, 0x18, 0x01, + 0x20, 0x01, 0x28, 0x03, 0x52, 0x08, 0x74, 0x69, 0x6d, 0x65, 0x55, 0x6e, 0x69, 0x78, 0x12, 0x1b, + 0x0a, 0x09, 0x73, 0x65, 0x6e, 0x64, 0x65, 0x72, 0x5f, 0x69, 0x64, 0x18, 0x02, 0x20, 0x01, 0x28, + 0x09, 0x52, 0x08, 0x73, 0x65, 0x6e, 0x64, 0x65, 0x72, 0x49, 0x64, 0x12, 0x1b, 0x0a, 0x09, 0x73, + 0x65, 0x6e, 0x64, 0x65, 0x72, 0x5f, 0x69, 0x70, 0x18, 0x03, 0x20, 0x01, 0x28, 0x09, 0x52, 0x08, + 0x73, 0x65, 0x6e, 0x64, 0x65, 0x72, 0x49, 0x70, 0x12, 0x1c, 0x0a, 0x09, 0x72, 0x65, 0x71, 0x75, + 0x65, 0x73, 0x74, 0x65, 0x64, 0x18, 0x04, 0x20, 0x01, 0x28, 0x05, 0x52, 0x09, 0x72, 0x65, 0x71, + 0x75, 0x65, 0x73, 0x74, 0x65, 0x64, 0x12, 0x14, 0x0a, 0x05, 0x66, 0x6f, 0x75, 0x6e, 0x64, 0x18, + 0x05, 0x20, 0x01, 0x28, 0x05, 0x52, 0x05, 0x66, 0x6f, 0x75, 0x6e, 0x64, 0x12, 0x1f, 0x0a, 0x0b, + 0x64, 0x75, 0x72, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x5f, 0x6d, 0x73, 0x18, 0x06, 0x20, 0x01, 0x28, + 0x03, 0x52, 0x0a, 0x64, 0x75, 0x72, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x4d, 0x73, 0x12, 0x14, 0x0a, + 0x05, 0x65, 0x72, 0x72, 0x6f, 0x72, 0x18, 0x07, 0x20, 0x01, 0x28, 0x09, 0x52, 0x05, 0x65, 0x72, + 0x72, 0x6f, 0x72, 0x1a, 0x6c, 0x0a, 0x14, 0x52, 0x65, 0x63, 0x65, 0x6e, 0x74, 0x42, 0x61, 0x74, + 0x63, 0x68, 0x53, 0x74, 0x6f, 0x72, 0x65, 0x4c, 0x69, 0x73, 0x74, 0x12, 0x54, 0x0a, 0x07, 0x65, + 0x6e, 0x74, 0x72, 0x69, 0x65, 0x73, 0x18, 0x01, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x3a, 0x2e, 0x73, + 0x75, 0x70, 0x65, 0x72, 0x6e, 0x6f, 0x64, 0x65, 0x2e, 0x53, 0x74, 0x61, 0x74, 0x75, 0x73, 0x52, + 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x2e, 0x50, 0x32, 0x50, 0x4d, 0x65, 0x74, 0x72, 0x69, + 0x63, 0x73, 0x2e, 0x52, 0x65, 0x63, 0x65, 0x6e, 0x74, 0x42, 0x61, 0x74, 0x63, 0x68, 0x53, 0x74, + 0x6f, 0x72, 0x65, 0x45, 0x6e, 0x74, 0x72, 0x79, 0x52, 0x07, 0x65, 0x6e, 0x74, 0x72, 0x69, 0x65, + 0x73, 0x1a, 0x72, 0x0a, 0x17, 0x52, 0x65, 0x63, 0x65, 0x6e, 0x74, 0x42, 0x61, 0x74, 0x63, 0x68, + 0x52, 0x65, 0x74, 0x72, 0x69, 0x65, 0x76, 0x65, 0x4c, 0x69, 0x73, 0x74, 0x12, 0x57, 0x0a, 0x07, + 0x65, 0x6e, 0x74, 0x72, 0x69, 0x65, 0x73, 0x18, 0x01, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x3d, 0x2e, + 0x73, 0x75, 0x70, 0x65, 0x72, 0x6e, 0x6f, 0x64, 0x65, 0x2e, 0x53, 0x74, 0x61, 0x74, 0x75, 0x73, + 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x2e, 0x50, 0x32, 0x50, 0x4d, 0x65, 0x74, 0x72, + 0x69, 0x63, 0x73, 0x2e, 0x52, 0x65, 0x63, 0x65, 0x6e, 0x74, 0x42, 0x61, 0x74, 0x63, 0x68, 0x52, + 0x65, 0x74, 0x72, 0x69, 0x65, 0x76, 0x65, 0x45, 0x6e, 0x74, 0x72, 0x79, 0x52, 0x07, 0x65, 0x6e, + 0x74, 0x72, 0x69, 0x65, 0x73, 0x1a, 0x82, 0x01, 0x0a, 0x19, 0x52, 0x65, 0x63, 0x65, 0x6e, 0x74, + 0x42, 0x61, 0x74, 0x63, 0x68, 0x53, 0x74, 0x6f, 0x72, 0x65, 0x42, 0x79, 0x49, 0x70, 0x45, 0x6e, + 0x74, 0x72, 0x79, 0x12, 0x10, 0x0a, 0x03, 0x6b, 0x65, 0x79, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, + 0x52, 0x03, 0x6b, 0x65, 0x79, 0x12, 0x4f, 0x0a, 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x18, 0x02, + 0x20, 0x01, 0x28, 0x0b, 0x32, 0x39, 0x2e, 0x73, 0x75, 0x70, 0x65, 0x72, 0x6e, 0x6f, 0x64, 0x65, + 0x2e, 0x53, 0x74, 0x61, 0x74, 0x75, 0x73, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x2e, + 0x50, 0x32, 0x50, 0x4d, 0x65, 0x74, 0x72, 0x69, 0x63, 0x73, 0x2e, 0x52, 0x65, 0x63, 0x65, 0x6e, + 0x74, 0x42, 0x61, 0x74, 0x63, 0x68, 0x53, 0x74, 0x6f, 0x72, 0x65, 0x4c, 0x69, 0x73, 0x74, 0x52, + 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x3a, 0x02, 0x38, 0x01, 0x1a, 0x88, 0x01, 0x0a, 0x1c, 0x52, + 0x65, 0x63, 0x65, 0x6e, 0x74, 0x42, 0x61, 0x74, 0x63, 0x68, 0x52, 0x65, 0x74, 0x72, 0x69, 0x65, + 0x76, 0x65, 0x42, 0x79, 0x49, 0x70, 0x45, 0x6e, 0x74, 0x72, 0x79, 0x12, 0x10, 0x0a, 0x03, 0x6b, + 0x65, 0x79, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x03, 0x6b, 0x65, 0x79, 0x12, 0x52, 0x0a, + 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x18, 0x02, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x3c, 0x2e, 0x73, + 0x75, 0x70, 0x65, 0x72, 0x6e, 0x6f, 0x64, 0x65, 0x2e, 0x53, 0x74, 0x61, 0x74, 0x75, 0x73, 0x52, + 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x2e, 0x50, 0x32, 0x50, 0x4d, 0x65, 0x74, 0x72, 0x69, + 0x63, 0x73, 0x2e, 0x52, 0x65, 0x63, 0x65, 0x6e, 0x74, 0x42, 0x61, 0x74, 0x63, 0x68, 0x52, 0x65, + 0x74, 0x72, 0x69, 0x65, 0x76, 0x65, 0x4c, 0x69, 0x73, 0x74, 0x52, 0x05, 0x76, 0x61, 0x6c, 0x75, + 0x65, 0x3a, 0x02, 0x38, 0x01, 0x32, 0xd7, 0x01, 0x0a, 0x10, 0x53, 0x75, 0x70, 0x65, 0x72, 0x6e, + 0x6f, 0x64, 0x65, 0x53, 0x65, 0x72, 0x76, 0x69, 0x63, 0x65, 0x12, 0x58, 0x0a, 0x09, 0x47, 0x65, + 0x74, 0x53, 0x74, 0x61, 0x74, 0x75, 0x73, 0x12, 0x18, 0x2e, 0x73, 0x75, 0x70, 0x65, 0x72, 0x6e, + 0x6f, 0x64, 0x65, 0x2e, 0x53, 0x74, 0x61, 0x74, 0x75, 0x73, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, + 0x74, 0x1a, 0x19, 0x2e, 0x73, 0x75, 0x70, 0x65, 0x72, 0x6e, 0x6f, 0x64, 0x65, 0x2e, 0x53, 0x74, + 0x61, 0x74, 0x75, 0x73, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x22, 0x16, 0x82, 0xd3, + 0xe4, 0x93, 0x02, 0x10, 0x12, 0x0e, 0x2f, 0x61, 0x70, 0x69, 0x2f, 0x76, 0x31, 0x2f, 0x73, 0x74, + 0x61, 0x74, 0x75, 0x73, 0x12, 0x69, 0x0a, 0x0c, 0x4c, 0x69, 0x73, 0x74, 0x53, 0x65, 0x72, 0x76, + 0x69, 0x63, 0x65, 0x73, 0x12, 0x1e, 0x2e, 0x73, 0x75, 0x70, 0x65, 0x72, 0x6e, 0x6f, 0x64, 0x65, + 0x2e, 0x4c, 0x69, 0x73, 0x74, 0x53, 0x65, 0x72, 0x76, 0x69, 0x63, 0x65, 0x73, 0x52, 0x65, 0x71, + 0x75, 0x65, 0x73, 0x74, 0x1a, 0x1f, 0x2e, 0x73, 0x75, 0x70, 0x65, 0x72, 0x6e, 0x6f, 0x64, 0x65, + 0x2e, 0x4c, 0x69, 0x73, 0x74, 0x53, 0x65, 0x72, 0x76, 0x69, 0x63, 0x65, 0x73, 0x52, 0x65, 0x73, + 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x22, 0x18, 0x82, 0xd3, 0xe4, 0x93, 0x02, 0x12, 0x12, 0x10, 0x2f, + 0x61, 0x70, 0x69, 0x2f, 0x76, 0x31, 0x2f, 0x73, 0x65, 0x72, 0x76, 0x69, 0x63, 0x65, 0x73, 0x42, + 0x36, 0x5a, 0x34, 0x67, 0x69, 0x74, 0x68, 0x75, 0x62, 0x2e, 0x63, 0x6f, 0x6d, 0x2f, 0x4c, 0x75, + 0x6d, 0x65, 0x72, 0x61, 0x50, 0x72, 0x6f, 0x74, 0x6f, 0x63, 0x6f, 0x6c, 0x2f, 0x73, 0x75, 0x70, + 0x65, 0x72, 0x6e, 0x6f, 0x64, 0x65, 0x2f, 0x76, 0x32, 0x2f, 0x67, 0x65, 0x6e, 0x2f, 0x73, 0x75, + 0x70, 0x65, 0x72, 0x6e, 0x6f, 0x64, 0x65, 0x62, 0x06, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x33, } var ( @@ -1543,7 +1941,7 @@ func file_supernode_supernode_proto_rawDescGZIP() []byte { return file_supernode_supernode_proto_rawDescData } -var file_supernode_supernode_proto_msgTypes = make([]protoimpl.MessageInfo, 21) +var file_supernode_supernode_proto_msgTypes = make([]protoimpl.MessageInfo, 27) var file_supernode_supernode_proto_goTypes = []any{ (*StatusRequest)(nil), // 0: supernode.StatusRequest (*ListServicesRequest)(nil), // 1: supernode.ListServicesRequest @@ -1564,8 +1962,14 @@ var file_supernode_supernode_proto_goTypes = []any{ (*StatusResponse_P2PMetrics_DiskStatus)(nil), // 16: supernode.StatusResponse.P2PMetrics.DiskStatus nil, // 17: supernode.StatusResponse.P2PMetrics.NetworkHandleMetricsEntry nil, // 18: supernode.StatusResponse.P2PMetrics.ConnPoolMetricsEntry - (*StatusResponse_P2PMetrics_DhtMetrics_StoreSuccessPoint)(nil), // 19: supernode.StatusResponse.P2PMetrics.DhtMetrics.StoreSuccessPoint - (*StatusResponse_P2PMetrics_DhtMetrics_BatchRetrievePoint)(nil), // 20: supernode.StatusResponse.P2PMetrics.DhtMetrics.BatchRetrievePoint + (*StatusResponse_P2PMetrics_RecentBatchStoreEntry)(nil), // 19: supernode.StatusResponse.P2PMetrics.RecentBatchStoreEntry + (*StatusResponse_P2PMetrics_RecentBatchRetrieveEntry)(nil), // 20: supernode.StatusResponse.P2PMetrics.RecentBatchRetrieveEntry + (*StatusResponse_P2PMetrics_RecentBatchStoreList)(nil), // 21: supernode.StatusResponse.P2PMetrics.RecentBatchStoreList + (*StatusResponse_P2PMetrics_RecentBatchRetrieveList)(nil), // 22: supernode.StatusResponse.P2PMetrics.RecentBatchRetrieveList + nil, // 23: supernode.StatusResponse.P2PMetrics.RecentBatchStoreByIpEntry + nil, // 24: supernode.StatusResponse.P2PMetrics.RecentBatchRetrieveByIpEntry + (*StatusResponse_P2PMetrics_DhtMetrics_StoreSuccessPoint)(nil), // 25: supernode.StatusResponse.P2PMetrics.DhtMetrics.StoreSuccessPoint + (*StatusResponse_P2PMetrics_DhtMetrics_BatchRetrievePoint)(nil), // 26: supernode.StatusResponse.P2PMetrics.DhtMetrics.BatchRetrievePoint } var file_supernode_supernode_proto_depIdxs = []int32{ 3, // 0: supernode.ListServicesResponse.services:type_name -> supernode.ServiceInfo @@ -1582,18 +1986,26 @@ var file_supernode_supernode_proto_depIdxs = []int32{ 14, // 11: supernode.StatusResponse.P2PMetrics.ban_list:type_name -> supernode.StatusResponse.P2PMetrics.BanEntry 15, // 12: supernode.StatusResponse.P2PMetrics.database:type_name -> supernode.StatusResponse.P2PMetrics.DatabaseStats 16, // 13: supernode.StatusResponse.P2PMetrics.disk:type_name -> supernode.StatusResponse.P2PMetrics.DiskStatus - 19, // 14: supernode.StatusResponse.P2PMetrics.DhtMetrics.store_success_recent:type_name -> supernode.StatusResponse.P2PMetrics.DhtMetrics.StoreSuccessPoint - 20, // 15: supernode.StatusResponse.P2PMetrics.DhtMetrics.batch_retrieve_recent:type_name -> supernode.StatusResponse.P2PMetrics.DhtMetrics.BatchRetrievePoint - 13, // 16: supernode.StatusResponse.P2PMetrics.NetworkHandleMetricsEntry.value:type_name -> supernode.StatusResponse.P2PMetrics.HandleCounters - 0, // 17: supernode.SupernodeService.GetStatus:input_type -> supernode.StatusRequest - 1, // 18: supernode.SupernodeService.ListServices:input_type -> supernode.ListServicesRequest - 4, // 19: supernode.SupernodeService.GetStatus:output_type -> supernode.StatusResponse - 2, // 20: supernode.SupernodeService.ListServices:output_type -> supernode.ListServicesResponse - 19, // [19:21] is the sub-list for method output_type - 17, // [17:19] is the sub-list for method input_type - 17, // [17:17] is the sub-list for extension type_name - 17, // [17:17] is the sub-list for extension extendee - 0, // [0:17] is the sub-list for field type_name + 19, // 14: supernode.StatusResponse.P2PMetrics.recent_batch_store:type_name -> supernode.StatusResponse.P2PMetrics.RecentBatchStoreEntry + 20, // 15: supernode.StatusResponse.P2PMetrics.recent_batch_retrieve:type_name -> supernode.StatusResponse.P2PMetrics.RecentBatchRetrieveEntry + 23, // 16: supernode.StatusResponse.P2PMetrics.recent_batch_store_by_ip:type_name -> supernode.StatusResponse.P2PMetrics.RecentBatchStoreByIpEntry + 24, // 17: supernode.StatusResponse.P2PMetrics.recent_batch_retrieve_by_ip:type_name -> supernode.StatusResponse.P2PMetrics.RecentBatchRetrieveByIpEntry + 25, // 18: supernode.StatusResponse.P2PMetrics.DhtMetrics.store_success_recent:type_name -> supernode.StatusResponse.P2PMetrics.DhtMetrics.StoreSuccessPoint + 26, // 19: supernode.StatusResponse.P2PMetrics.DhtMetrics.batch_retrieve_recent:type_name -> supernode.StatusResponse.P2PMetrics.DhtMetrics.BatchRetrievePoint + 13, // 20: supernode.StatusResponse.P2PMetrics.NetworkHandleMetricsEntry.value:type_name -> supernode.StatusResponse.P2PMetrics.HandleCounters + 19, // 21: supernode.StatusResponse.P2PMetrics.RecentBatchStoreList.entries:type_name -> supernode.StatusResponse.P2PMetrics.RecentBatchStoreEntry + 20, // 22: supernode.StatusResponse.P2PMetrics.RecentBatchRetrieveList.entries:type_name -> supernode.StatusResponse.P2PMetrics.RecentBatchRetrieveEntry + 21, // 23: supernode.StatusResponse.P2PMetrics.RecentBatchStoreByIpEntry.value:type_name -> supernode.StatusResponse.P2PMetrics.RecentBatchStoreList + 22, // 24: supernode.StatusResponse.P2PMetrics.RecentBatchRetrieveByIpEntry.value:type_name -> supernode.StatusResponse.P2PMetrics.RecentBatchRetrieveList + 0, // 25: supernode.SupernodeService.GetStatus:input_type -> supernode.StatusRequest + 1, // 26: supernode.SupernodeService.ListServices:input_type -> supernode.ListServicesRequest + 4, // 27: supernode.SupernodeService.GetStatus:output_type -> supernode.StatusResponse + 2, // 28: supernode.SupernodeService.ListServices:output_type -> supernode.ListServicesResponse + 27, // [27:29] is the sub-list for method output_type + 25, // [25:27] is the sub-list for method input_type + 25, // [25:25] is the sub-list for extension type_name + 25, // [25:25] is the sub-list for extension extendee + 0, // [0:25] is the sub-list for field type_name } func init() { file_supernode_supernode_proto_init() } @@ -1607,7 +2019,7 @@ func file_supernode_supernode_proto_init() { GoPackagePath: reflect.TypeOf(x{}).PkgPath(), RawDescriptor: file_supernode_supernode_proto_rawDesc, NumEnums: 0, - NumMessages: 21, + NumMessages: 27, NumExtensions: 0, NumServices: 1, }, diff --git a/gen/supernode/supernode.swagger.json b/gen/supernode/supernode.swagger.json index e29dcbae..00a47bb8 100644 --- a/gen/supernode/supernode.swagger.json +++ b/gen/supernode/supernode.swagger.json @@ -249,6 +249,92 @@ }, "title": "Per-handler counters from network layer" }, + "P2PMetricsRecentBatchRetrieveEntry": { + "type": "object", + "properties": { + "timeUnix": { + "type": "string", + "format": "int64" + }, + "senderId": { + "type": "string" + }, + "senderIp": { + "type": "string" + }, + "requested": { + "type": "integer", + "format": "int32" + }, + "found": { + "type": "integer", + "format": "int32" + }, + "durationMs": { + "type": "string", + "format": "int64" + }, + "error": { + "type": "string" + } + }, + "title": "Last handled BatchGetValues requests (most recent first)" + }, + "P2PMetricsRecentBatchRetrieveList": { + "type": "object", + "properties": { + "entries": { + "type": "array", + "items": { + "type": "object", + "$ref": "#/definitions/P2PMetricsRecentBatchRetrieveEntry" + } + } + } + }, + "P2PMetricsRecentBatchStoreEntry": { + "type": "object", + "properties": { + "timeUnix": { + "type": "string", + "format": "int64" + }, + "senderId": { + "type": "string" + }, + "senderIp": { + "type": "string" + }, + "keys": { + "type": "integer", + "format": "int32" + }, + "durationMs": { + "type": "string", + "format": "int64" + }, + "ok": { + "type": "boolean" + }, + "error": { + "type": "string" + } + }, + "title": "Last handled BatchStoreData requests (most recent first)" + }, + "P2PMetricsRecentBatchStoreList": { + "type": "object", + "properties": { + "entries": { + "type": "array", + "items": { + "type": "object", + "$ref": "#/definitions/P2PMetricsRecentBatchStoreEntry" + } + } + }, + "title": "Per-IP buckets: last 10 per sender IP" + }, "ResourcesCPU": { "type": "object", "properties": { @@ -364,6 +450,32 @@ }, "disk": { "$ref": "#/definitions/P2PMetricsDiskStatus" + }, + "recentBatchStore": { + "type": "array", + "items": { + "type": "object", + "$ref": "#/definitions/P2PMetricsRecentBatchStoreEntry" + } + }, + "recentBatchRetrieve": { + "type": "array", + "items": { + "type": "object", + "$ref": "#/definitions/P2PMetricsRecentBatchRetrieveEntry" + } + }, + "recentBatchStoreByIp": { + "type": "object", + "additionalProperties": { + "$ref": "#/definitions/P2PMetricsRecentBatchStoreList" + } + }, + "recentBatchRetrieveByIp": { + "type": "object", + "additionalProperties": { + "$ref": "#/definitions/P2PMetricsRecentBatchRetrieveList" + } } }, "title": "P2P metrics and diagnostics (additive field)" diff --git a/p2p/client.go b/p2p/client.go index 6eb169b2..5d4a44be 100644 --- a/p2p/client.go +++ b/p2p/client.go @@ -22,17 +22,8 @@ type Client interface { // - the base58 encoded identifier will be returned Store(ctx context.Context, data []byte, typ int) (string, error) - // StoreBatch will store a batch of values with their Blake3 hash as the key. - // - // Semantics: - // - Returns `successRatePct` as a percentage (0–100) computed as - // successful node RPCs divided by total node RPCs attempted during the - // network store phase for this batch. - // - Returns `requests` as the total number of node RPCs attempted for this - // batch (not the number of items in `values`). - // - On error, `successRatePct` and `requests` may reflect partial progress; - // prefer using them only when err == nil, or treat as best‑effort metrics. - StoreBatch(ctx context.Context, values [][]byte, typ int, taskID string) (float64, int, error) + // StoreBatch stores a batch of values; returns error only. + StoreBatch(ctx context.Context, values [][]byte, typ int, taskID string) error // Delete a key, value Delete(ctx context.Context, key string) error diff --git a/p2p/kademlia/dht.go b/p2p/kademlia/dht.go index 00c86fb8..0209bd73 100644 --- a/p2p/kademlia/dht.go +++ b/p2p/kademlia/dht.go @@ -23,6 +23,7 @@ import ( "github.com/LumeraProtocol/supernode/v2/pkg/logtrace" "github.com/LumeraProtocol/supernode/v2/pkg/lumera" ltc "github.com/LumeraProtocol/supernode/v2/pkg/net/credentials" + "github.com/LumeraProtocol/supernode/v2/pkg/p2pmetrics" "github.com/LumeraProtocol/supernode/v2/pkg/storage" "github.com/LumeraProtocol/supernode/v2/pkg/storage/memory" "github.com/LumeraProtocol/supernode/v2/pkg/storage/rqstore" @@ -40,12 +41,13 @@ const ( defaultDeleteDataInterval = 11 * time.Hour delKeysCountThreshold = 10 lowSpaceThreshold = 50 // GB - batchRetrieveSize = 1000 + batchStoreSize = 2500 storeSameSymbolsBatchConcurrency = 3 - fetchSymbolsBatchConcurrency = 6 + storeSymbolsBatchConcurrency = 3.0 minimumDataStoreSuccessRate = 75.0 - maxIterations = 4 + maxIterations = 4 + macConcurrentNetworkStoreCalls = 16 ) // DHT represents the state of the queries node in the distributed hash table @@ -351,32 +353,27 @@ func (s *DHT) Store(ctx context.Context, data []byte, typ int) (string, error) { return retKey, nil } -// StoreBatch will store a batch of values with their Blake3 hash as the key. -// -// Returns: -// - successRatePct: percentage (0–100) of successful node RPCs during the -// network store phase for this batch. -// - requestCount: total number of node RPCs attempted (batch store calls) for -// this batch; this is not the number of values stored. -// - error: wrapped error if local DB store failed, or if the network store did -// not reach the configured minimum success rate. -func (s *DHT) StoreBatch(ctx context.Context, values [][]byte, typ int, taskID string) (float64, int, error) { +// StoreBatch stores a batch of values with their Blake3 hash as the key. +// It persists to the local store then performs the network store. If the +// measured success rate for node RPCs is below the configured minimum, an error +// is returned. Metrics are not returned through the API. +func (s *DHT) StoreBatch(ctx context.Context, values [][]byte, typ int, taskID string) error { logtrace.Info(ctx, "Store DB batch begin", logtrace.Fields{ logtrace.FieldModule: "dht", logtrace.FieldTaskID: taskID, "records": len(values), }) if err := s.store.StoreBatch(ctx, values, typ, true); err != nil { - return 0, 0, fmt.Errorf("store batch: %v", err) + return fmt.Errorf("store batch: %v", err) } logtrace.Info(ctx, "Store DB batch done, store network batch begin", logtrace.Fields{ logtrace.FieldModule: "dht", logtrace.FieldTaskID: taskID, }) - rate, requests, err := s.IterateBatchStore(ctx, values, typ, taskID) + err := s.IterateBatchStore(ctx, values, typ, taskID) if err != nil { - return rate, requests, fmt.Errorf("iterate batch store: %v", err) + return fmt.Errorf("iterate batch store: %v", err) } logtrace.Info(ctx, "Store network batch workers done", logtrace.Fields{ @@ -384,7 +381,7 @@ func (s *DHT) StoreBatch(ctx context.Context, values [][]byte, typ int, taskID s logtrace.FieldTaskID: taskID, }) - return rate, requests, nil + return nil } // Retrieve data from the networking using key. Key is the base58 encoded @@ -468,6 +465,17 @@ func (s *DHT) Stats(ctx context.Context) (map[string]interface{}, error) { dhtStats["peers_count"] = len(s.ht.nodes()) dhtStats["peers"] = s.ht.nodes() dhtStats["network"] = s.network.HandleMetricsSnapshot() + // Include recent request snapshots for observability + if s.network != nil { + if overall, byIP := s.network.RecentBatchStoreSnapshot(); len(overall) > 0 || len(byIP) > 0 { + dhtStats["recent_batch_store_overall"] = overall + dhtStats["recent_batch_store_by_ip"] = byIP + } + if overall, byIP := s.network.RecentBatchRetrieveSnapshot(); len(overall) > 0 || len(byIP) > 0 { + dhtStats["recent_batch_retrieve_overall"] = overall + dhtStats["recent_batch_retrieve_by_ip"] = byIP + } + } dhtStats["database"] = dbStats return dhtStats, nil @@ -730,14 +738,16 @@ func (s *DHT) BatchRetrieve(ctx context.Context, keys []string, required int32, if err != nil { return nil, fmt.Errorf("fetch and add local keys: %v", err) } + // Report how many were found locally, for event metrics + p2pmetrics.ReportFoundLocal(p2pmetrics.TaskIDFromContext(ctx), int(foundLocalCount)) if foundLocalCount >= required { return result, nil } - batchSize := batchRetrieveSize + batchSize := batchStoreSize var networkFound int32 totalBatches := int(math.Ceil(float64(required) / float64(batchSize))) - parallelBatches := int(math.Min(float64(totalBatches), fetchSymbolsBatchConcurrency)) + parallelBatches := int(math.Min(float64(totalBatches), storeSymbolsBatchConcurrency)) semaphore := make(chan struct{}, parallelBatches) var wg sync.WaitGroup @@ -775,13 +785,7 @@ func (s *DHT) BatchRetrieve(ctx context.Context, keys []string, required int32, wg.Wait() netFound := int(atomic.LoadInt32(&networkFound)) - totalFound := int(foundLocalCount) + netFound - - s.metrics.RecordBatchRetrieve(len(keys), int(required), int(foundLocalCount), netFound, time.Since(start)) - - if totalFound < int(required) { - return result, errors.Errorf("insufficient symbols: required=%d, found=%d", required, totalFound) - } + s.metrics.RecordBatchRetrieve(len(keys), int(required), int(foundLocalCount), netFound, time.Duration(time.Since(start).Milliseconds())) // NEW return result, nil } @@ -806,7 +810,7 @@ func (s *DHT) processBatch( defer wg.Done() defer func() { <-semaphore }() - for i := 0; i < 1; i++ { + for i := 0; i < maxIterations; i++ { select { case <-ctx.Done(): return @@ -828,18 +832,9 @@ func (s *DHT) processBatch( } } - knownMu.Lock() - nodesSnap := make(map[string]*Node, len(knownNodes)) - for id, n := range knownNodes { - nodesSnap[id] = n - } - knownMu.Unlock() - foundCount, newClosestContacts, batchErr := s.iterateBatchGetValues( - ctx, nodesSnap, batchKeys, batchHexKeys, fetchMap, resMap, - required, foundLocalCount+atomic.LoadInt32(networkFound), + ctx, knownNodes, batchKeys, batchHexKeys, fetchMap, resMap, required, foundLocalCount+atomic.LoadInt32(networkFound), ) - if batchErr != nil { logtrace.Error(ctx, "Iterate batch get values failed", logtrace.Fields{ logtrace.FieldModule: "dht", "txid": txID, logtrace.FieldError: batchErr.Error(), @@ -887,36 +882,19 @@ func (s *DHT) processBatch( } } -func (s *DHT) iterateBatchGetValues( - ctx context.Context, - nodes map[string]*Node, - keys []string, - hexKeys []string, - fetchMap map[string][]int, - resMap *sync.Map, - req, alreadyFound int32, -) (int, map[string]*NodeList, error) { - - semaphore := make(chan struct{}, storeSameSymbolsBatchConcurrency) +func (s *DHT) iterateBatchGetValues(ctx context.Context, nodes map[string]*Node, keys []string, hexKeys []string, fetchMap map[string][]int, + resMap *sync.Map, req, alreadyFound int32) (int, map[string]*NodeList, error) { + semaphore := make(chan struct{}, storeSameSymbolsBatchConcurrency) // Limit concurrency to 1 closestContacts := make(map[string]*NodeList) var wg sync.WaitGroup contactsMap := make(map[string]map[string][]*Node) var firstErr error - var mu sync.Mutex + var mu sync.Mutex // To protect the firstErr foundCount := int32(0) - gctx, cancel := context.WithCancel(ctx) + gctx, cancel := context.WithCancel(ctx) // Create a cancellable context defer cancel() - - // ✅ Iterate ONLY nodes that actually have work according to fetchMap - for nodeID, idxs := range fetchMap { - if len(idxs) == 0 { - continue - } - node, ok := nodes[nodeID] - if !ok { - continue - } + for nodeID, node := range nodes { if s.ignorelist.Banned(node) { logtrace.Info(ctx, "Ignore banned node in iterate batch get values", logtrace.Fields{ logtrace.FieldModule: "dht", @@ -926,9 +904,8 @@ func (s *DHT) iterateBatchGetValues( } contactsMap[nodeID] = make(map[string][]*Node) - wg.Add(1) - go func(node *Node, nodeID string, indices []int) { + go func(node *Node, nodeID string) { defer wg.Done() select { @@ -940,16 +917,30 @@ func (s *DHT) iterateBatchGetValues( defer func() { <-semaphore }() } - // Build requestKeys from the provided indices only - requestKeys := make(map[string]KeyValWithClosest, len(indices)) + callStart := time.Now() + indices := fetchMap[nodeID] + requestKeys := make(map[string]KeyValWithClosest) for _, idx := range indices { - if idx >= 0 && idx < len(hexKeys) { - if _, loaded := resMap.Load(hexKeys[idx]); !loaded { + if idx < len(hexKeys) { + _, loaded := resMap.Load(hexKeys[idx]) // check if key is already there in resMap + if !loaded { requestKeys[hexKeys[idx]] = KeyValWithClosest{} } } } + if len(requestKeys) == 0 { + // No keys to request from this node (e.g., all keys already satisfied elsewhere). + // Treat as a successful, no-op call for metrics when there is no error. + p2pmetrics.RecordRetrieve(p2pmetrics.TaskIDFromContext(ctx), p2pmetrics.Call{ + IP: node.IP, + Address: node.String(), + Keys: 0, + Success: true, + Error: "", + DurationMS: time.Since(callStart).Milliseconds(), + Noop: true, + }) return } @@ -960,23 +951,46 @@ func (s *DHT) iterateBatchGetValues( firstErr = err } mu.Unlock() + // record failed RPC per-node + p2pmetrics.RecordRetrieve(p2pmetrics.TaskIDFromContext(ctx), p2pmetrics.Call{ + IP: node.IP, + Address: node.String(), + Keys: 0, + Success: false, + Error: err.Error(), + DurationMS: time.Since(callStart).Milliseconds(), + }) return } - // Merge values or closest contacts + returned := 0 for k, v := range decompressedData { if len(v.Value) > 0 { - if _, loaded := resMap.LoadOrStore(k, v.Value); !loaded { - if atomic.AddInt32(&foundCount, 1) >= int32(req-alreadyFound) { - cancel() - return + _, loaded := resMap.LoadOrStore(k, v.Value) + if !loaded { + atomic.AddInt32(&foundCount, 1) + returned++ + if atomic.LoadInt32(&foundCount) >= int32(req-alreadyFound) { + cancel() // Cancel context to stop other goroutines + // don't early return; record metric and exit goroutine + break } } } else { contactsMap[nodeID][k] = v.Closest } } - }(node, nodeID, idxs) + + // record successful RPC per-node (returned may be 0). Success is true when no error. + p2pmetrics.RecordRetrieve(p2pmetrics.TaskIDFromContext(ctx), p2pmetrics.Call{ + IP: node.IP, + Address: node.String(), + Keys: returned, + Success: true, + Error: "", + DurationMS: time.Since(callStart).Milliseconds(), + }) + }(node, nodeID) } wg.Wait() @@ -994,19 +1008,19 @@ func (s *DHT) iterateBatchGetValues( }) } - // Build closestContacts from contactsMap (same as before) for _, closestNodes := range contactsMap { for key, nodes := range closestNodes { comparator, err := hex.DecodeString(key) if err != nil { - logtrace.Error(ctx, "Failed to decode hex key in closestNodes", logtrace.Fields{ + logtrace.Error(ctx, "Failed to decode hex key in closestNodes.Range", logtrace.Fields{ logtrace.FieldModule: "dht", "key": key, logtrace.FieldError: err.Error(), }) - return int(foundCount), nil, err + return 0, nil, err } bkey := base58.Encode(comparator) + if _, ok := closestContacts[bkey]; !ok { closestContacts[bkey] = &NodeList{Nodes: nodes, Comparator: comparator} } else { @@ -1014,12 +1028,12 @@ func (s *DHT) iterateBatchGetValues( } } } + for key, nodes := range closestContacts { nodes.Sort() nodes.TopN(Alpha) closestContacts[key] = nodes } - return int(foundCount), closestContacts, firstErr } @@ -1214,12 +1228,13 @@ func (s *DHT) iterate(ctx context.Context, iterativeType int, target []byte, dat func (s *DHT) handleResponses(ctx context.Context, responses <-chan *Message, nl *NodeList) (*NodeList, []byte) { for response := range responses { s.addNode(ctx, response.Sender) - if response.MessageType == FindNode || response.MessageType == StoreData { + switch response.MessageType { + case FindNode, StoreData: v, ok := response.Data.(*FindNodeResponse) if ok && v.Status.Result == ResultOk && len(v.Closest) > 0 { nl.AddNodes(v.Closest) } - } else if response.MessageType == FindValue { + case FindValue: v, ok := response.Data.(*FindValueResponse) if ok { if v.Status.Result == ResultOk && len(v.Value) > 0 { @@ -1630,7 +1645,7 @@ func (s *DHT) addKnownNodes(ctx context.Context, nodes []*Node, knownNodes map[s // during this run; success rate is successful responses divided by this count. // If the success rate is below `minimumDataStoreSuccessRate`, an error is // returned alongside the measured rate and request count. -func (s *DHT) IterateBatchStore(ctx context.Context, values [][]byte, typ int, id string) (float64, int, error) { +func (s *DHT) IterateBatchStore(ctx context.Context, values [][]byte, typ int, id string) error { globalClosestContacts := make(map[string]*NodeList) knownNodes := make(map[string]*Node) hashes := make([][]byte, len(values)) @@ -1665,38 +1680,54 @@ func (s *DHT) IterateBatchStore(ctx context.Context, values [][]byte, typ int, i storeResponses := s.batchStoreNetwork(ctx, values, knownNodes, storageMap, typ) for response := range storeResponses { requests++ + var nodeAddr string + var nodeIP string + if response.Receiver != nil { + nodeAddr = response.Receiver.String() + nodeIP = response.Receiver.IP + } else if response.Message != nil && response.Message.Sender != nil { + nodeAddr = response.Message.Sender.String() + nodeIP = response.Message.Sender.IP + } + + errMsg := "" if response.Error != nil { - sender := "" - if response.Message != nil && response.Message.Sender != nil { - sender = response.Message.Sender.String() - } + errMsg = response.Error.Error() logtrace.Error(ctx, "Batch store failed on a node", logtrace.Fields{ logtrace.FieldModule: "dht", - "node": sender, - logtrace.FieldError: response.Error.Error(), + "node": nodeAddr, + logtrace.FieldError: errMsg, }) } - if response.Message == nil { - continue + if response.Message != nil { + if v, ok := response.Message.Data.(*StoreDataResponse); ok { + if v.Status.Result == ResultOk { + successful++ + } else { + if v.Status.ErrMsg != "" { + errMsg = v.Status.ErrMsg + } + logtrace.Error(ctx, "Batch store to node failed", logtrace.Fields{ + logtrace.FieldModule: "dht", + "err": errMsg, + "task_id": id, + "node": nodeAddr, + }) + } + } } - v, ok := response.Message.Data.(*StoreDataResponse) - if ok && v.Status.Result == ResultOk { - successful++ - } else { - errMsg := "unknown error" - if v != nil { - errMsg = v.Status.ErrMsg - } + // Emit per-node store RPC call via metrics bridge (no P2P API coupling) + p2pmetrics.RecordStore(p2pmetrics.TaskIDFromContext(ctx), p2pmetrics.Call{ + IP: nodeIP, + Address: nodeAddr, + Keys: response.KeysCount, + Success: errMsg == "" && response.Error == nil, + Error: errMsg, + DurationMS: response.DurationMS, + }) - logtrace.Error(ctx, "Batch store to node failed", logtrace.Fields{ - logtrace.FieldModule: "dht", - "err": errMsg, - "task_id": id, - "node": response.Message.Sender.String(), - }) - } } if requests > 0 { @@ -1709,24 +1740,24 @@ func (s *DHT) IterateBatchStore(ctx context.Context, values [][]byte, typ int, i "task_id": id, "success_rate": fmt.Sprintf("%.2f%%", successRate), }) - return successRate, requests, nil + return nil } else { logtrace.Info(ctx, "Failed to achieve desired success rate", logtrace.Fields{ logtrace.FieldModule: "dht", "task_id": id, "success_rate": fmt.Sprintf("%.2f%%", successRate), }) - return successRate, requests, fmt.Errorf("failed to achieve desired success rate, only: %.2f%% successful", successRate) + return fmt.Errorf("failed to achieve desired success rate, only: %.2f%% successful", successRate) } } - return 0, 0, fmt.Errorf("no store operations were performed") + return fmt.Errorf("no store operations were performed") } func (s *DHT) batchStoreNetwork(ctx context.Context, values [][]byte, nodes map[string]*Node, storageMap map[string][]int, typ int) chan *MessageWithError { responses := make(chan *MessageWithError, len(nodes)) - maxStore := 16 + maxStore := macConcurrentNetworkStoreCalls if ln := len(nodes); ln < maxStore { maxStore = ln } @@ -1759,9 +1790,10 @@ func (s *DHT) batchStoreNetwork(ctx context.Context, values [][]byte, nodes map[ select { case <-ctx.Done(): - responses <- &MessageWithError{Error: ctx.Err()} + responses <- &MessageWithError{Error: ctx.Err(), Receiver: receiver} return default: + callStart := time.Now() keysToStore := storageMap[key] toStore := make([][]byte, len(keysToStore)) totalBytes := 0 @@ -1776,9 +1808,28 @@ func (s *DHT) batchStoreNetwork(ctx context.Context, values [][]byte, nodes map[ "size_before_compress": utils.BytesIntToMB(totalBytes), }) + // Skip empty payloads: avoid sending empty store RPCs, but record a noop metric for visibility. + if len(toStore) == 0 { + logtrace.Info(ctx, "Skipping store RPC with empty payload", logtrace.Fields{ + logtrace.FieldModule: "dht", + "node": receiver.String(), + }) + p2pmetrics.RecordStore(p2pmetrics.TaskIDFromContext(ctx), p2pmetrics.Call{ + IP: receiver.IP, + Address: receiver.String(), + Keys: 0, + Success: true, + Error: "", + DurationMS: time.Since(callStart).Milliseconds(), + Noop: true, + }) + return + } + data := &BatchStoreDataRequest{Data: toStore, Type: typ} request := s.newMessage(BatchStoreData, receiver, data) response, err := s.network.Call(ctx, request, false) + dur := time.Since(callStart).Milliseconds() if err != nil { if !isLocalCancel(err) { s.ignorelist.IncrementCount(receiver) @@ -1790,11 +1841,11 @@ func (s *DHT) batchStoreNetwork(ctx context.Context, values [][]byte, nodes map[ logtrace.FieldError: err.Error(), "request": request.String(), }) - responses <- &MessageWithError{Error: err, Message: response} + responses <- &MessageWithError{Error: err, Message: response, KeysCount: len(toStore), Receiver: receiver, DurationMS: dur} return } - responses <- &MessageWithError{Message: response} + responses <- &MessageWithError{Message: response, KeysCount: len(toStore), Receiver: receiver, DurationMS: dur} } }(node, key) } diff --git a/p2p/kademlia/message.go b/p2p/kademlia/message.go index 7ef3f206..0baef37c 100644 --- a/p2p/kademlia/message.go +++ b/p2p/kademlia/message.go @@ -54,6 +54,10 @@ func init() { type MessageWithError struct { Message *Message Error error + // Extended context for store RPCs + KeysCount int // number of items attempted in this RPC + Receiver *Node // receiver node info (target) + DurationMS int64 // duration of the RPC in milliseconds } // Message structure for kademlia network diff --git a/p2p/kademlia/network.go b/p2p/kademlia/network.go index 2c1479cb..a2322ff7 100644 --- a/p2p/kademlia/network.go +++ b/p2p/kademlia/network.go @@ -69,6 +69,13 @@ type Network struct { sem *semaphore.Weighted metrics sync.Map + + // recent request tracking (last 10 entries overall and per IP) + recentMu sync.Mutex + recentStoreOverall []RecentBatchStoreEntry + recentStoreByIP map[string][]RecentBatchStoreEntry + recentRetrieveOverall []RecentBatchRetrieveEntry + recentRetrieveByIP map[string][]RecentBatchRetrieveEntry } // NewNetwork returns a network service @@ -627,7 +634,7 @@ func (s *Network) Call(ctx context.Context, request *Message, isLong bool) (*Mes // ---- retryable RPC helpers ------------------------------------------------- func (s *Network) rpcOnceWrapper(ctx context.Context, cw *connWrapper, remoteAddr string, data []byte, timeout time.Duration, msgType int) (*Message, error) { - writeDL := calcWriteDeadline(timeout, len(data), 1.0) // target ~2 MB/s + writeDL := calcWriteDeadline(timeout, len(data), 1.0) // target ~1 MB/s retried := false for { @@ -895,15 +902,38 @@ func (s *Network) handleBatchFindValues(ctx context.Context, message *Message, r } func (s *Network) handleGetValuesRequest(ctx context.Context, message *Message, reqID string) (res []byte, err error) { + start := time.Now() + appended := false defer func() { if response, err := s.handlePanic(ctx, message.Sender, BatchGetValues); response != nil || err != nil { res = response + if !appended { + s.appendRetrieveEntry(message.Sender.IP, RecentBatchRetrieveEntry{ + TimeUnix: time.Now().UTC().Unix(), + SenderID: string(message.Sender.ID), + SenderIP: message.Sender.IP, + Requested: 0, + Found: 0, + DurationMS: time.Since(start).Milliseconds(), + Error: "panic/recovered", + }) + } } }() request, ok := message.Data.(*BatchGetValuesRequest) if !ok { err := errors.New("invalid BatchGetValuesRequest") + s.appendRetrieveEntry(message.Sender.IP, RecentBatchRetrieveEntry{ + TimeUnix: time.Now().UTC().Unix(), + SenderID: string(message.Sender.ID), + SenderIP: message.Sender.IP, + Requested: 0, + Found: 0, + DurationMS: time.Since(start).Milliseconds(), + Error: err.Error(), + }) + appended = true return s.generateResponseMessage(BatchGetValues, message.Sender, ResultFailed, err.Error()) } @@ -924,6 +954,16 @@ func (s *Network) handleGetValuesRequest(ctx context.Context, message *Message, values, count, err := s.dht.store.RetrieveBatchValues(ctx, keys, true) if err != nil { err = errors.Errorf("batch find values: %w", err) + s.appendRetrieveEntry(message.Sender.IP, RecentBatchRetrieveEntry{ + TimeUnix: time.Now().UTC().Unix(), + SenderID: string(message.Sender.ID), + SenderIP: message.Sender.IP, + Requested: len(keys), + Found: count, + DurationMS: time.Since(start).Milliseconds(), + Error: err.Error(), + }) + appended = true return s.generateResponseMessage(BatchGetValues, message.Sender, ResultFailed, err.Error()) } @@ -961,6 +1001,16 @@ func (s *Network) handleGetValuesRequest(ctx context.Context, message *Message, // new a response message resMsg := s.dht.newMessage(BatchGetValues, message.Sender, response) + s.appendRetrieveEntry(message.Sender.IP, RecentBatchRetrieveEntry{ + TimeUnix: time.Now().UTC().Unix(), + SenderID: string(message.Sender.ID), + SenderIP: message.Sender.IP, + Requested: len(keys), + Found: count, + DurationMS: time.Since(start).Milliseconds(), + Error: "", + }) + appended = true return s.encodeMesage(resMsg) } @@ -1132,15 +1182,38 @@ func findTopHeaviestKeys(dataMap map[string][]byte, size int) (int, []string) { } func (s *Network) handleBatchStoreData(ctx context.Context, message *Message) (res []byte, err error) { + start := time.Now() + appended := false defer func() { if response, err := s.handlePanic(ctx, message.Sender, BatchStoreData); response != nil || err != nil { res = response + if !appended { + s.appendStoreEntry(message.Sender.IP, RecentBatchStoreEntry{ + TimeUnix: time.Now().UTC().Unix(), + SenderID: string(message.Sender.ID), + SenderIP: message.Sender.IP, + Keys: 0, + DurationMS: time.Since(start).Milliseconds(), + OK: false, + Error: "panic/recovered", + }) + } } }() request, ok := message.Data.(*BatchStoreDataRequest) if !ok { err := errors.New("invalid BatchStoreDataRequest") + s.appendStoreEntry(message.Sender.IP, RecentBatchStoreEntry{ + TimeUnix: time.Now().UTC().Unix(), + SenderID: string(message.Sender.ID), + SenderIP: message.Sender.IP, + Keys: 0, + DurationMS: time.Since(start).Milliseconds(), + OK: false, + Error: err.Error(), + }) + appended = true return s.generateResponseMessage(BatchStoreData, message.Sender, ResultFailed, err.Error()) } @@ -1156,6 +1229,16 @@ func (s *Network) handleBatchStoreData(ctx context.Context, message *Message) (r if err := s.dht.store.StoreBatch(ctx, request.Data, 1, false); err != nil { err = errors.Errorf("batch store the data: %w", err) + s.appendStoreEntry(message.Sender.IP, RecentBatchStoreEntry{ + TimeUnix: time.Now().UTC().Unix(), + SenderID: string(message.Sender.ID), + SenderIP: message.Sender.IP, + Keys: len(request.Data), + DurationMS: time.Since(start).Milliseconds(), + OK: false, + Error: err.Error(), + }) + appended = true return s.generateResponseMessage(BatchStoreData, message.Sender, ResultFailed, err.Error()) } @@ -1173,6 +1256,16 @@ func (s *Network) handleBatchStoreData(ctx context.Context, message *Message) (r // new a response message resMsg := s.dht.newMessage(BatchStoreData, message.Sender, response) + s.appendStoreEntry(message.Sender.IP, RecentBatchStoreEntry{ + TimeUnix: time.Now().UTC().Unix(), + SenderID: string(message.Sender.ID), + SenderIP: message.Sender.IP, + Keys: len(request.Data), + DurationMS: time.Since(start).Milliseconds(), + OK: true, + Error: "", + }) + appended = true return s.encodeMesage(resMsg) } @@ -1439,9 +1532,9 @@ func calcWriteDeadline(timeout time.Duration, sizeBytes int, targetMBps float64) base := 2 * time.Second cushion := 5 * time.Second - // Softer floor: assume ~2 MB/s; increase if you like. + // Softer floor: assume ~1 MB/s by default; increase if you like. if targetMBps <= 0 { - targetMBps = 2.0 + targetMBps = 1.0 } est := time.Duration(sizeMB / targetMBps * float64(time.Second)) diff --git a/p2p/kademlia/recent.go b/p2p/kademlia/recent.go new file mode 100644 index 00000000..2467cf02 --- /dev/null +++ b/p2p/kademlia/recent.go @@ -0,0 +1,90 @@ +package kademlia + +import ( + "sync" + "time" +) + +// RecentBatchStoreEntry captures a handled BatchStoreData request outcome +type RecentBatchStoreEntry struct { + TimeUnix int64 `json:"time_unix"` + SenderID string `json:"sender_id"` + SenderIP string `json:"sender_ip"` + Keys int `json:"keys"` + DurationMS int64 `json:"duration_ms"` + OK bool `json:"ok"` + Error string `json:"error,omitempty"` +} + +// RecentBatchRetrieveEntry captures a handled BatchGetValues request outcome +type RecentBatchRetrieveEntry struct { + TimeUnix int64 `json:"time_unix"` + SenderID string `json:"sender_id"` + SenderIP string `json:"sender_ip"` + Requested int `json:"requested"` + Found int `json:"found"` + DurationMS int64 `json:"duration_ms"` + Error string `json:"error,omitempty"` +} + +func (s *Network) appendStoreEntry(ip string, e RecentBatchStoreEntry) { + s.recentMu.Lock() + defer s.recentMu.Unlock() + if s.recentStoreByIP == nil { + s.recentStoreByIP = make(map[string][]RecentBatchStoreEntry) + } + s.recentStoreOverall = append([]RecentBatchStoreEntry{e}, s.recentStoreOverall...) + if len(s.recentStoreOverall) > 10 { + s.recentStoreOverall = s.recentStoreOverall[:10] + } + lst := append([]RecentBatchStoreEntry{e}, s.recentStoreByIP[ip]...) + if len(lst) > 10 { + lst = lst[:10] + } + s.recentStoreByIP[ip] = lst +} + +func (s *Network) appendRetrieveEntry(ip string, e RecentBatchRetrieveEntry) { + s.recentMu.Lock() + defer s.recentMu.Unlock() + if s.recentRetrieveByIP == nil { + s.recentRetrieveByIP = make(map[string][]RecentBatchRetrieveEntry) + } + s.recentRetrieveOverall = append([]RecentBatchRetrieveEntry{e}, s.recentRetrieveOverall...) + if len(s.recentRetrieveOverall) > 10 { + s.recentRetrieveOverall = s.recentRetrieveOverall[:10] + } + lst := append([]RecentBatchRetrieveEntry{e}, s.recentRetrieveByIP[ip]...) + if len(lst) > 10 { + lst = lst[:10] + } + s.recentRetrieveByIP[ip] = lst +} + +// RecentBatchStoreSnapshot returns copies of recent store entries (overall and by IP) +func (s *Network) RecentBatchStoreSnapshot() (overall []RecentBatchStoreEntry, byIP map[string][]RecentBatchStoreEntry) { + s.recentMu.Lock() + defer s.recentMu.Unlock() + overall = append([]RecentBatchStoreEntry(nil), s.recentStoreOverall...) + byIP = make(map[string][]RecentBatchStoreEntry, len(s.recentStoreByIP)) + for k, v := range s.recentStoreByIP { + byIP[k] = append([]RecentBatchStoreEntry(nil), v...) + } + return +} + +// RecentBatchRetrieveSnapshot returns copies of recent retrieve entries (overall and by IP) +func (s *Network) RecentBatchRetrieveSnapshot() (overall []RecentBatchRetrieveEntry, byIP map[string][]RecentBatchRetrieveEntry) { + s.recentMu.Lock() + defer s.recentMu.Unlock() + overall = append([]RecentBatchRetrieveEntry(nil), s.recentRetrieveOverall...) + byIP = make(map[string][]RecentBatchRetrieveEntry, len(s.recentRetrieveByIP)) + for k, v := range s.recentRetrieveByIP { + byIP[k] = append([]RecentBatchRetrieveEntry(nil), v...) + } + return +} + +// helper to avoid unused import warning if needed +var _ = time.Now +var _ = sync.Mutex{} diff --git a/p2p/kademlia/rq_symbols.go b/p2p/kademlia/rq_symbols.go index 0b530f98..fbf6563d 100644 --- a/p2p/kademlia/rq_symbols.go +++ b/p2p/kademlia/rq_symbols.go @@ -96,9 +96,7 @@ func (s *DHT) storeSymbolsInP2P(ctx context.Context, dir string, keys []string) return fmt.Errorf("load symbols: %w", err) } - // Intentionally ignore (ratePct, requests) here; a non-nil error will already - // reflect whether the network store met the configured success threshold. - if _, _, err := s.StoreBatch(ctx, loaded, 1, dir); err != nil { + if err := s.StoreBatch(ctx, loaded, 1, dir); err != nil { return fmt.Errorf("p2p store batch: %w", err) } diff --git a/p2p/mocks/Client.go b/p2p/mocks/Client.go index 6d092c92..67991025 100644 --- a/p2p/mocks/Client.go +++ b/p2p/mocks/Client.go @@ -245,35 +245,15 @@ func (_m *Client) Store(ctx context.Context, data []byte, typ int) (string, erro } // StoreBatch provides a mock function with given fields: ctx, values, typ, taskID -func (_m *Client) StoreBatch(ctx context.Context, values [][]byte, typ int, taskID string) (float64, int, error) { +func (_m *Client) StoreBatch(ctx context.Context, values [][]byte, typ int, taskID string) error { ret := _m.Called(ctx, values, typ, taskID) - - var r0 float64 - if rf, ok := ret.Get(0).(func(context.Context, [][]byte, int, string) float64); ok { + var r0 error + if rf, ok := ret.Get(0).(func(context.Context, [][]byte, int, string) error); ok { r0 = rf(ctx, values, typ, taskID) } else { - if ret.Get(0) != nil { - r0 = ret.Get(0).(float64) - } - } - - var r1 int - if rf, ok := ret.Get(1).(func(context.Context, [][]byte, int, string) int); ok { - r1 = rf(ctx, values, typ, taskID) - } else { - if ret.Get(1) != nil { - r1 = ret.Get(1).(int) - } - } - - var r2 error - if rf, ok := ret.Get(2).(func(context.Context, [][]byte, int, string) error); ok { - r2 = rf(ctx, values, typ, taskID) - } else { - r2 = ret.Error(2) + r0 = ret.Error(0) } - - return r0, r1, r2 + return r0 } type mockConstructorTestingTNewClient interface { diff --git a/p2p/p2p.go b/p2p/p2p.go index a9ca294c..e3d6b40a 100644 --- a/p2p/p2p.go +++ b/p2p/p2p.go @@ -117,18 +117,12 @@ func (s *p2p) Store(ctx context.Context, data []byte, typ int) (string, error) { return s.dht.Store(ctx, data, typ) } -// StoreBatch will store a batch of values with their Blake3 hash as the key. -// -// It proxies to DHT.StoreBatch and returns: -// - successRatePct: percentage of successful node RPCs during the network store -// - requests: total node RPCs attempted for the batch -// - error: error if persistence or network store did not meet minimum success criteria -func (s *p2p) StoreBatch(ctx context.Context, data [][]byte, typ int, taskID string) (float64, int, error) { +// StoreBatch stores a batch of values by their Blake3 hash as the key. +func (s *p2p) StoreBatch(ctx context.Context, data [][]byte, typ int, taskID string) error { if !s.running { - return 0, 0, errors.New("p2p service is not running") + return errors.New("p2p service is not running") } - return s.dht.StoreBatch(ctx, data, typ, taskID) } @@ -182,7 +176,12 @@ func (s *p2p) Stats(ctx context.Context) (map[string]interface{}, error) { retStats["disk-info"] = &diskUse retStats["ban-list"] = s.dht.BanListSnapshot() retStats["conn-pool"] = s.dht.ConnPoolSnapshot() - dhtStats["dht_metrics"] = s.dht.MetricsSnapshot() + + // Expose DHT rolling metrics snapshot both under the top-level key (as expected by + // the status service) and also within the DHT map for backward compatibility. + snapshot := s.dht.MetricsSnapshot() + retStats["dht_metrics"] = snapshot + dhtStats["dht_metrics"] = snapshot return retStats, nil } diff --git a/pkg/cascade/constants.go b/pkg/cascade/constants.go deleted file mode 100644 index 1bee9824..00000000 --- a/pkg/cascade/constants.go +++ /dev/null @@ -1,16 +0,0 @@ -package cascade - -const ( - // SkipArtifactStorageHeader is propagated via gRPC metadata to indicate - // the supernode should bypass artifact persistence. - SkipArtifactStorageHeader = "x-lumera-skip-artifact-storage" - // SkipArtifactStorageHeaderValue marks the header value for opting out of storage. - SkipArtifactStorageHeaderValue = "true" - // LogFieldSkipStorage standardises the log key used when skip storage is triggered. - LogFieldSkipStorage = "skip_storage" -) - -const ( - // ArtifactStorageSkippedMessage is published when artifact persistence is skipped. - ArtifactStorageSkippedMessage = "Artifact storage skipped by client instruction" -) diff --git a/pkg/lumera/CONNECTION.md b/pkg/lumera/CONNECTION.md deleted file mode 100644 index 86162757..00000000 --- a/pkg/lumera/CONNECTION.md +++ /dev/null @@ -1,118 +0,0 @@ -# Lumera gRPC Connection Design - -This document explains how `pkg/lumera/connection.go` establishes and maintains a robust, long‑lived gRPC client connection. - -## Goals - -- Be resilient to varying deployment setups (LBs, custom ports, TLS/plain). -- Prefer a fast, successful connection without relying on URL scheme. -- Keep the connection alive for long periods (days) safely. -- Fail fast at startup if no candidate becomes READY. -- Exit the process if a previously established connection is irrecoverably lost. - -## High‑Level Flow - -1. Parse input into `host` and optional `port` (ignore scheme entirely). -2. Generate dial candidates across ports and security modes: - - If no port: use default ports `[9090, 443]`. - - Always generate both TLS and plaintext candidates (4 total when no port; 2 total when a port is specified). -3. Race all candidates concurrently. -4. Each candidate dials non‑blocking, then explicitly waits until the connection reaches `READY` (or times out). -5. The first candidate to hit `READY` is selected; all others are closed (including “late winners”). -6. A monitor goroutine observes the connection state and exits the process if the connection is lost. - -## Inputs and Parsing - -- `parseAddrMeta(raw string)` extracts `host` and `port`. -- URL schemes (e.g., `https`, `grpcs`, `http`, `grpc`) are ignored for policy decisions. Only host/port matter. -- If no port is provided, we consider it “unspecified” and will try defaults. - -## Candidate Generation - -- `generateCandidates(meta)` creates a de‑duplicated set of `(target, useTLS)` pairs: - - Ports: - - explicit port → `[port]` - - no port → `defaultDialPorts = [9090, 443]` - - Security: - - Always generates both TLS and plaintext for the chosen ports. -- This yields: - - No scheme + no port: 4 candidates (TLS/PLAIN × 9090/443) - - Any input with explicit port: 2 candidates (TLS/PLAIN on that port) - -Note: TLS creds use `credentials.NewClientTLSFromCert(nil, serverName)`; plaintext uses `insecure.NewCredentials()`. - -## Dialing and Readiness - -- `createGRPCConnection` uses `grpc.NewClient(target, opts...)` and then: - - `conn.Connect()` to begin dialing - - waits in a loop until: - - `Ready` → success; return the connection - - `Shutdown`/`TransientFailure` → close and return error - - idle/connecting → wait for state change with a per‑attempt timeout -- Timeouts: - - `dialReadyTimeout` (default 10s) applies if the provided context has no deadline. - -## Selecting the Winner - -- `newGRPCConnection` starts all candidate attempts and collects results on a buffered channel. -- The first `READY` result becomes the winner; we keep receiving remaining results to explicitly close any non‑winners (including late winners) to avoid leaks. -- All pending attempts are canceled via a parent context cancellation. - -## Blocking Behavior - -- The constructor returns only after a connection is in `READY` state. -- If all candidates fail, it returns an error (startup abort). - -## Keepalive for Long‑Lived Connections - -- Client keepalive parameters are tuned for Cosmos‑SDK environments: - - `keepaliveIdleTime = 10m` (send pings no more than every 10 minutes when idle) - - `keepaliveAckTimeout = 20s` - - `PermitWithoutStream = true` -- Rationale: - - Conservative ping interval avoids server `GOAWAY` for “too_many_pings”. - - Keeps NAT/firewalls from silently expiring idle connections. - -## Connection Monitor and Process Exit - -- `monitorConnection` watches the connection state: - - `Shutdown` → log and `os.Exit(1)` - - `TransientFailure` → allow up to `reconnectionGracePeriod = 30s` for recovery; if not recovered, log and exit. -- Normal reconnection behavior is handled by gRPC; the monitor only exits if the connection does not recover within the grace period or is shut down definitively. - -## Logging - -- On success, we log: target (`host:port`) and scheme (`tls` or `plaintext`). -- Errors during attempts surface as aggregated failure if no candidate succeeds. - -## Thread‑Safety - -- The only map (`seen`) is local to candidate generation and used single‑threaded. -- Concurrency is limited to goroutines dialing candidates and sending results through a channel; no shared maps are mutated concurrently. -- Late winner cleanup explicitly closes extra connections to avoid resource leaks. - -## Configuration Knobs (Constants) - -- `defaultDialPorts = [9090, 443]` -- `dialReadyTimeout = 10s` (per attempt, if no deadline present) -- `keepaliveIdleTime = 10m` -- `keepaliveAckTimeout = 20s` -- `reconnectionGracePeriod = 30s` - -## Extensibility - -- Ports: Adjust `defaultDialPorts` if your environment prefers a different port set. -- TLS: To support custom roots or mTLS, add an option to inject `TransportCredentials` instead of the defaults. -- Policies: If future schemes or resolvers are introduced, they can be layered in before candidate generation. - -## Error Cases and Behavior - -- If no candidate reaches `READY` within `dialReadyTimeout` per attempt, `newGRPCConnection` returns an error. -- If the connection later enters a prolonged `TransientFailure` or `Shutdown`, the monitor exits the process. - -## FAQ - -- Why try both TLS and plaintext? We avoid making assumptions based on scheme and instead race practical permutations to maximize robustness across deployments. -- Why include both 9090 and 443? These are common in gRPC deployments (custom service ports and TLS‑terminating LBs). Adjust as needed for your infra. -- Does this support Unix sockets? Not currently; could be added by extending candidate generation. - diff --git a/pkg/p2pmetrics/metrics.go b/pkg/p2pmetrics/metrics.go new file mode 100644 index 00000000..b483bb1d --- /dev/null +++ b/pkg/p2pmetrics/metrics.go @@ -0,0 +1,299 @@ +package p2pmetrics + +import ( + "context" + "sync" +) + +// Call represents a single per-node RPC outcome (store or retrieve). +type Call struct { + IP string `json:"ip"` + Address string `json:"address"` + Keys int `json:"keys"` + Success bool `json:"success"` + Error string `json:"error,omitempty"` + DurationMS int64 `json:"duration_ms"` + Noop bool `json:"noop,omitempty"` +} + +// -------- Lightweight hooks ------------------------- + +var ( + storeMu sync.RWMutex + storeHook = make(map[string]func(Call)) + + retrieveMu sync.RWMutex + retrieveHook = make(map[string]func(Call)) + + foundLocalMu sync.RWMutex + foundLocalCb = make(map[string]func(int)) +) + +// RegisterStoreHook registers a callback to receive store RPC calls for a task. +func RegisterStoreHook(taskID string, fn func(Call)) { + storeMu.Lock() + defer storeMu.Unlock() + if fn == nil { + delete(storeHook, taskID) + return + } + storeHook[taskID] = fn +} + +// UnregisterStoreHook removes the registered store callback for a task. +func UnregisterStoreHook(taskID string) { RegisterStoreHook(taskID, nil) } + +// RecordStore invokes the registered store callback for the given task, if any. +func RecordStore(taskID string, c Call) { + storeMu.RLock() + fn := storeHook[taskID] + storeMu.RUnlock() + if fn != nil { + fn(c) + } +} + +// RegisterRetrieveHook registers a callback to receive retrieve RPC calls. +func RegisterRetrieveHook(taskID string, fn func(Call)) { + retrieveMu.Lock() + defer retrieveMu.Unlock() + if fn == nil { + delete(retrieveHook, taskID) + return + } + retrieveHook[taskID] = fn +} + +// UnregisterRetrieveHook removes the registered retrieve callback for a task. +func UnregisterRetrieveHook(taskID string) { RegisterRetrieveHook(taskID, nil) } + +// RecordRetrieve invokes the registered retrieve callback for the given task. +func RecordRetrieve(taskID string, c Call) { + retrieveMu.RLock() + fn := retrieveHook[taskID] + retrieveMu.RUnlock() + if fn != nil { + fn(c) + } +} + +// RegisterFoundLocalHook registers a callback to receive found-local counts. +func RegisterFoundLocalHook(taskID string, fn func(int)) { + foundLocalMu.Lock() + defer foundLocalMu.Unlock() + if fn == nil { + delete(foundLocalCb, taskID) + return + } + foundLocalCb[taskID] = fn +} + +// UnregisterFoundLocalHook removes the registered found-local callback. +func UnregisterFoundLocalHook(taskID string) { RegisterFoundLocalHook(taskID, nil) } + +// ReportFoundLocal invokes the registered found-local callback for the task. +func ReportFoundLocal(taskID string, count int) { + foundLocalMu.RLock() + fn := foundLocalCb[taskID] + foundLocalMu.RUnlock() + if fn != nil { + fn(count) + } +} + +// -------- Minimal in-process collectors for events -------------------------- + +// Store session +type storeSession struct { + CallsByIP map[string][]Call + SymbolsFirstPass int + SymbolsTotal int + IDFilesCount int + DurationMS int64 +} + +var storeSessions = struct{ m map[string]*storeSession }{m: map[string]*storeSession{}} + +// RegisterStoreBridge hooks store callbacks into the store session collector. +func StartStoreCapture(taskID string) { + RegisterStoreHook(taskID, func(c Call) { + s := storeSessions.m[taskID] + if s == nil { + s = &storeSession{CallsByIP: map[string][]Call{}} + storeSessions.m[taskID] = s + } + key := c.IP + if key == "" { + key = c.Address + } + s.CallsByIP[key] = append(s.CallsByIP[key], c) + }) +} + +func StopStoreCapture(taskID string) { UnregisterStoreHook(taskID) } + +// SetStoreSummary sets store summary fields for the first pass and totals. +// +// - symbolsFirstPass: number of symbols sent during the first pass +// - symbolsTotal: total symbols available in the directory +// - idFilesCount: number of ID/metadata files included in the first combined batch +// - durationMS: elapsed time of the first-pass store phase +func SetStoreSummary(taskID string, symbolsFirstPass, symbolsTotal, idFilesCount int, durationMS int64) { + if taskID == "" { + return + } + s := storeSessions.m[taskID] + if s == nil { + s = &storeSession{CallsByIP: map[string][]Call{}} + storeSessions.m[taskID] = s + } + s.SymbolsFirstPass = symbolsFirstPass + s.SymbolsTotal = symbolsTotal + s.IDFilesCount = idFilesCount + s.DurationMS = durationMS +} + +// BuildStoreEventPayloadFromCollector builds the store event payload (minimal). +func BuildStoreEventPayloadFromCollector(taskID string) map[string]any { + s := storeSessions.m[taskID] + if s == nil { + return map[string]any{ + "store": map[string]any{ + "duration_ms": int64(0), + "symbols_first_pass": 0, + "symbols_total": 0, + "id_files_count": 0, + "success_rate_pct": float64(0), + "calls_by_ip": map[string][]Call{}, + }, + } + } + // Compute per-call success rate across first-pass store RPC attempts + totalCalls := 0 + successCalls := 0 + for _, calls := range s.CallsByIP { + for _, c := range calls { + totalCalls++ + if c.Success { + successCalls++ + } + } + } + var successRate float64 + if totalCalls > 0 { + successRate = float64(successCalls) / float64(totalCalls) * 100.0 + } + return map[string]any{ + "store": map[string]any{ + "duration_ms": s.DurationMS, + "symbols_first_pass": s.SymbolsFirstPass, + "symbols_total": s.SymbolsTotal, + "id_files_count": s.IDFilesCount, + "success_rate_pct": successRate, + "calls_by_ip": s.CallsByIP, + }, + } +} + +// Retrieve session +type retrieveSession struct { + CallsByIP map[string][]Call + FoundLocal int + RetrieveMS int64 + DecodeMS int64 +} + +var retrieveSessions = struct{ m map[string]*retrieveSession }{m: map[string]*retrieveSession{}} + +// RegisterRetrieveBridge hooks retrieve callbacks into the retrieve collector. +func StartRetrieveCapture(taskID string) { + RegisterRetrieveHook(taskID, func(c Call) { + s := retrieveSessions.m[taskID] + if s == nil { + s = &retrieveSession{CallsByIP: map[string][]Call{}} + retrieveSessions.m[taskID] = s + } + key := c.IP + if key == "" { + key = c.Address + } + s.CallsByIP[key] = append(s.CallsByIP[key], c) + }) + RegisterFoundLocalHook(taskID, func(n int) { + s := retrieveSessions.m[taskID] + if s == nil { + s = &retrieveSession{CallsByIP: map[string][]Call{}} + retrieveSessions.m[taskID] = s + } + s.FoundLocal = n + }) +} + +func StopRetrieveCapture(taskID string) { + UnregisterRetrieveHook(taskID) + UnregisterFoundLocalHook(taskID) +} + +// SetRetrieveSummary sets timing info for retrieve/decode phases. +func SetRetrieveSummary(taskID string, retrieveMS, decodeMS int64) { + if taskID == "" { + return + } + s := retrieveSessions.m[taskID] + if s == nil { + s = &retrieveSession{CallsByIP: map[string][]Call{}} + retrieveSessions.m[taskID] = s + } + s.RetrieveMS = retrieveMS + s.DecodeMS = decodeMS +} + +// BuildDownloadEventPayloadFromCollector builds the download section payload. +func BuildDownloadEventPayloadFromCollector(taskID string) map[string]any { + s := retrieveSessions.m[taskID] + if s == nil { + return map[string]any{ + "retrieve": map[string]any{ + "found_local": 0, + "retrieve_ms": int64(0), + "decode_ms": int64(0), + "calls_by_ip": map[string][]Call{}, + }, + } + } + return map[string]any{ + "retrieve": map[string]any{ + "found_local": s.FoundLocal, + "retrieve_ms": s.RetrieveMS, + "decode_ms": s.DecodeMS, + "calls_by_ip": s.CallsByIP, + }, + } +} + +// -------- Context helpers (dedicated to metrics tagging) -------------------- + +type ctxKey string + +var taskIDKey ctxKey = "p2pmetrics-task-id" + +// WithTaskID returns a child context with the metrics task ID set. +func WithTaskID(ctx context.Context, taskID string) context.Context { + if ctx == nil { + return context.Background() + } + return context.WithValue(ctx, taskIDKey, taskID) +} + +// TaskIDFromContext extracts the metrics task ID from context (or ""). +func TaskIDFromContext(ctx context.Context) string { + if ctx == nil { + return "" + } + if v := ctx.Value(taskIDKey); v != nil { + if s, ok := v.(string); ok { + return s + } + } + return "" +} diff --git a/proto/supernode/supernode.proto b/proto/supernode/supernode.proto index edbff3b0..50597e90 100644 --- a/proto/supernode/supernode.proto +++ b/proto/supernode/supernode.proto @@ -154,6 +154,37 @@ message StatusResponse { repeated BanEntry ban_list = 4; DatabaseStats database = 5; DiskStatus disk = 6; + + // Last handled BatchStoreData requests (most recent first) + message RecentBatchStoreEntry { + int64 time_unix = 1; + string sender_id = 2; + string sender_ip = 3; + int32 keys = 4; + int64 duration_ms = 5; + bool ok = 6; + string error = 7; + } + + // Last handled BatchGetValues requests (most recent first) + message RecentBatchRetrieveEntry { + int64 time_unix = 1; + string sender_id = 2; + string sender_ip = 3; + int32 requested = 4; + int32 found = 5; + int64 duration_ms = 6; + string error = 7; + } + + repeated RecentBatchStoreEntry recent_batch_store = 7; + repeated RecentBatchRetrieveEntry recent_batch_retrieve = 8; + + // Per-IP buckets: last 10 per sender IP + message RecentBatchStoreList { repeated RecentBatchStoreEntry entries = 1; } + message RecentBatchRetrieveList { repeated RecentBatchRetrieveEntry entries = 1; } + map recent_batch_store_by_ip = 9; + map recent_batch_retrieve_by_ip = 10; } P2PMetrics p2p_metrics = 9; diff --git a/sdk/adapters/lumera/adapter.go b/sdk/adapters/lumera/adapter.go index 36038937..8fe7a1fb 100644 --- a/sdk/adapters/lumera/adapter.go +++ b/sdk/adapters/lumera/adapter.go @@ -3,6 +3,7 @@ package lumera import ( "context" "fmt" + "sort" "github.com/LumeraProtocol/supernode/v2/sdk/log" @@ -103,15 +104,26 @@ func (a *Adapter) GetSupernodeWithLatestAddress(ctx context.Context, address str return nil, fmt.Errorf("received nil response for supernode %s", address) } - // Determine latest address/state strictly by max height + // Sort PrevIpAddresses by height in descending order + sort.Slice(resp.PrevIpAddresses, func(i, j int) bool { + return resp.PrevIpAddresses[i].Height > resp.PrevIpAddresses[j].Height + }) + + // Sort States by height in descending order + sort.Slice(resp.States, func(i, j int) bool { + return resp.States[i].Height > resp.States[j].Height + }) + + // Extract latest address latestAddress := "" - if addr, err := getLatestIP(resp); err == nil { - latestAddress = addr + if len(resp.PrevIpAddresses) > 0 { + latestAddress = resp.PrevIpAddresses[0].Address } + // Extract current state currentState := "" - if st, err := getLatestState(resp); err == nil && st != nil { - currentState = st.State.String() + if len(resp.States) > 0 { + currentState = resp.States[0].State.String() } info := &SuperNodeInfo{ @@ -271,19 +283,17 @@ func getLatestState(supernode *sntypes.SuperNode) (*sntypes.SuperNodeStateRecord return nil, fmt.Errorf("no state history exists for the supernode") } - var latest *sntypes.SuperNodeStateRecord - for _, s := range supernode.States { - if s == nil { - continue - } - if latest == nil || s.Height > latest.Height { - latest = s - } - } - if latest == nil { - return nil, fmt.Errorf("no valid state in history") + // Sort by height in descending order to get the latest first + sort.Slice(supernode.States, func(i, j int) bool { + return supernode.States[i].Height > supernode.States[j].Height + }) + + // Access the latest state safely + if supernode.States[0] == nil { + return nil, fmt.Errorf("latest state in history is nil") } - return latest, nil + + return supernode.States[0], nil } func getLatestIP(supernode *sntypes.SuperNode) (string, error) { @@ -296,17 +306,15 @@ func getLatestIP(supernode *sntypes.SuperNode) (string, error) { return "", fmt.Errorf("no ip history exists for the supernode") } - var latest *sntypes.IPAddressHistory - for _, r := range supernode.PrevIpAddresses { - if r == nil { - continue - } - if latest == nil || r.Height > latest.Height { - latest = r - } - } - if latest == nil { - return "", fmt.Errorf("no valid IP address in history") + // Sort by height in descending order to get the latest first + sort.Slice(supernode.PrevIpAddresses, func(i, j int) bool { + return supernode.PrevIpAddresses[i].Height > supernode.PrevIpAddresses[j].Height + }) + + // Access the latest IP address safely + if supernode.PrevIpAddresses[0] == nil { + return "", fmt.Errorf("latest IP address in history is nil") } - return latest.Address, nil + + return supernode.PrevIpAddresses[0].Address, nil } diff --git a/sdk/adapters/supernodeservice/adapter.go b/sdk/adapters/supernodeservice/adapter.go index afb1ac3d..969ace44 100644 --- a/sdk/adapters/supernodeservice/adapter.go +++ b/sdk/adapters/supernodeservice/adapter.go @@ -2,22 +2,19 @@ package supernodeservice import ( "context" + "encoding/json" "fmt" "io" "os" "path/filepath" - "regexp" - "strconv" "time" "github.com/LumeraProtocol/supernode/v2/gen/supernode" "github.com/LumeraProtocol/supernode/v2/gen/supernode/action/cascade" - cascadecommon "github.com/LumeraProtocol/supernode/v2/pkg/cascade" "github.com/LumeraProtocol/supernode/v2/sdk/event" "github.com/LumeraProtocol/supernode/v2/sdk/log" "google.golang.org/grpc" - "google.golang.org/grpc/metadata" ) type cascadeAdapter struct { @@ -87,18 +84,8 @@ func (a *cascadeAdapter) CascadeSupernodeRegister(ctx context.Context, in *Casca phaseCtx, cancel := context.WithCancel(baseCtx) defer cancel() - // Check if we should skip artifact storage on retry - callCtx := phaseCtx - if in.SkipArtifactStorage { - md := metadata.Pairs( - cascadecommon.SkipArtifactStorageHeader, - cascadecommon.SkipArtifactStorageHeaderValue, - ) - callCtx = metadata.NewOutgoingContext(callCtx, md) - } - // Create the client stream - stream, err := a.client.Register(callCtx, opts...) + stream, err := a.client.Register(phaseCtx, opts...) if err != nil { a.logger.Error(ctx, "Failed to create register stream", "error", err) if in.EventLogger != nil { @@ -358,10 +345,27 @@ func (a *cascadeAdapter) CascadeSupernodeRegister(ctx context.Context, in *Casca event.KeyTaskID: in.TaskId, event.KeyActionID: in.ActionID, } - // Extract success rate if provided in message format: "... success_rate=NN.NN%" + // For artefacts stored, parse JSON payload with metrics (new minimal shape) if resp.EventType == cascade.SupernodeEventType_ARTEFACTS_STORED { - if rate, ok := parseSuccessRate(resp.Message); ok { - edata[event.KeySuccessRate] = rate + var payload map[string]any + if err := json.Unmarshal([]byte(resp.Message), &payload); err == nil { + if store, ok := payload["store"].(map[string]any); ok { + if v, ok := store["duration_ms"].(float64); ok { + edata[event.KeyStoreDurationMS] = int64(v) + } + if v, ok := store["symbols_first_pass"].(float64); ok { + edata[event.KeyStoreSymbolsFirstPass] = int64(v) + } + if v, ok := store["symbols_total"].(float64); ok { + edata[event.KeyStoreSymbolsTotal] = int64(v) + } + if v, ok := store["id_files_count"].(float64); ok { + edata[event.KeyStoreIDFilesCount] = int64(v) + } + if v, ok := store["calls_by_ip"]; ok { + edata[event.KeyStoreCallsByIP] = v + } + } } } in.EventLogger(ctx, toSdkEventWithMessage(resp.EventType, resp.Message), resp.Message, edata) @@ -461,11 +465,32 @@ func (a *cascadeAdapter) CascadeSupernodeDownload( a.logger.Info(ctx, "supernode event", "event_type", x.Event.EventType, "message", x.Event.Message, "action_id", in.ActionID) if in.EventLogger != nil { - in.EventLogger(ctx, toSdkEvent(x.Event.EventType), x.Event.Message, event.EventData{ + edata := event.EventData{ event.KeyActionID: in.ActionID, event.KeyEventType: x.Event.EventType, event.KeyMessage: x.Event.Message, - }) + } + // Parse detailed metrics for downloaded event if JSON payload provided (new minimal shape) + if x.Event.EventType == cascade.SupernodeEventType_ARTEFACTS_DOWNLOADED { + var payload map[string]any + if err := json.Unmarshal([]byte(x.Event.Message), &payload); err == nil { + if retrieve, ok := payload["retrieve"].(map[string]any); ok { + if v, ok := retrieve["found_local"].(float64); ok { + edata[event.KeyRetrieveFoundLocal] = int64(v) + } + if v, ok := retrieve["retrieve_ms"].(float64); ok { + edata[event.KeyRetrieveMS] = int64(v) + } + if v, ok := retrieve["decode_ms"].(float64); ok { + edata[event.KeyDecodeMS] = int64(v) + } + if v, ok := retrieve["calls_by_ip"]; ok { + edata[event.KeyRetrieveCallsByIP] = v + } + } + } + } + in.EventLogger(ctx, toSdkEvent(x.Event.EventType), x.Event.Message, edata) } // 3b. Actual data chunk @@ -498,7 +523,7 @@ func (a *cascadeAdapter) CascadeSupernodeDownload( } return &CascadeSupernodeDownloadResponse{ Success: true, - Message: "artifact downloaded", + Message: "artefact downloaded", OutputPath: in.OutputPath, }, nil } @@ -554,20 +579,6 @@ func toSdkEventWithMessage(e cascade.SupernodeEventType, msg string) event.Event return toSdkEvent(e) } -var rateRe = regexp.MustCompile(`success_rate=([0-9]+(?:\.[0-9]+)?)%`) - -func parseSuccessRate(msg string) (float64, bool) { - m := rateRe.FindStringSubmatch(msg) - if len(m) != 2 { - return 0, false - } - f, err := strconv.ParseFloat(m[1], 64) - if err != nil { - return 0, false - } - return f, true -} - func toSdkSupernodeStatus(resp *supernode.StatusResponse) *SupernodeStatusresponse { result := &SupernodeStatusresponse{} result.Version = resp.Version diff --git a/sdk/adapters/supernodeservice/types.go b/sdk/adapters/supernodeservice/types.go index 36a06271..4dbdd7b6 100644 --- a/sdk/adapters/supernodeservice/types.go +++ b/sdk/adapters/supernodeservice/types.go @@ -16,11 +16,10 @@ type LoggerFunc func( ) type CascadeSupernodeRegisterRequest struct { - FilePath string - ActionID string - TaskId string - SkipArtifactStorage bool - EventLogger LoggerFunc + FilePath string + ActionID string + TaskId string + EventLogger LoggerFunc } type CascadeSupernodeRegisterResponse struct { diff --git a/sdk/event/keys.go b/sdk/event/keys.go index 4a6f8eaa..1e012677 100644 --- a/sdk/event/keys.go +++ b/sdk/event/keys.go @@ -15,7 +15,6 @@ const ( KeyProgress EventDataKey = "progress" KeyEventType EventDataKey = "event_type" KeyOutputPath EventDataKey = "output_path" - KeySuccessRate EventDataKey = "success_rate" // Upload/download metrics keys (no progress events; start/complete metrics only) KeyBytesTotal EventDataKey = "bytes_total" @@ -30,4 +29,20 @@ const ( // Task specific keys KeyTaskID EventDataKey = "task_id" KeyActionID EventDataKey = "action_id" + + // Removed legacy cascade storage metrics keys (meta/sym timings and nodes) + + // Combined store metrics (metadata + symbols) — new minimal only + KeyStoreDurationMS EventDataKey = "store_duration_ms" + // New minimal store metrics + KeyStoreSymbolsFirstPass EventDataKey = "store_symbols_first_pass" + KeyStoreSymbolsTotal EventDataKey = "store_symbols_total" + KeyStoreIDFilesCount EventDataKey = "store_id_files_count" + KeyStoreCallsByIP EventDataKey = "store_calls_by_ip" + + // Download (retrieve) detailed metrics — new minimal only + KeyRetrieveFoundLocal EventDataKey = "retrieve_found_local" + KeyRetrieveMS EventDataKey = "retrieve_ms" + KeyDecodeMS EventDataKey = "decode_ms" + KeyRetrieveCallsByIP EventDataKey = "retrieve_calls_by_ip" ) diff --git a/sdk/net/factory.go b/sdk/net/factory.go index 510d1b6b..b9fad9fd 100644 --- a/sdk/net/factory.go +++ b/sdk/net/factory.go @@ -36,12 +36,13 @@ func NewClientFactory(ctx context.Context, logger log.Logger, keyring keyring.Ke logger.Debug(ctx, "Creating supernode client factory", "localAddress", config.LocalCosmosAddress) - // Optimized for streaming 1GB files with 4MB chunks (10 concurrent streams) + // Tuned for 1GB max files with 4MB chunks + // Reduce in-flight memory by aligning windows and msg sizes to chunk size. opts := client.DefaultClientOptions() - opts.MaxRecvMsgSize = 16 * 1024 * 1024 // 16MB to match server - opts.MaxSendMsgSize = 16 * 1024 * 1024 // 16MB to match server - opts.InitialWindowSize = 16 * 1024 * 1024 // 16MB per stream (4x chunk size) - opts.InitialConnWindowSize = 160 * 1024 * 1024 // 160MB (16MB x 10 streams) + opts.MaxRecvMsgSize = 8 * 1024 * 1024 // 8MB: supports 4MB chunks + overhead + opts.MaxSendMsgSize = 8 * 1024 * 1024 // 8MB: supports 4MB chunks + overhead + opts.InitialWindowSize = 4 * 1024 * 1024 // 4MB per-stream window ≈ chunk size + opts.InitialConnWindowSize = 64 * 1024 * 1024 // 64MB per-connection window return &ClientFactory{ logger: logger, diff --git a/sdk/task/cascade.go b/sdk/task/cascade.go index 4e25fc06..c13b94a1 100644 --- a/sdk/task/cascade.go +++ b/sdk/task/cascade.go @@ -14,9 +14,8 @@ import ( type CascadeTask struct { BaseTask - filePath string - actionId string - skipArtifactStorage bool + filePath string + actionId string } // NewCascadeTask creates a new CascadeTask using a BaseTask plus cascade-specific parameters @@ -75,7 +74,6 @@ func (t *CascadeTask) registerWithSupernodes(ctx context.Context, supernodes lum var lastErr error for idx, sn := range supernodes { - req.SkipArtifactStorage = t.skipArtifactStorage // 1 t.LogEvent(ctx, event.SDKRegistrationAttempt, "attempting registration with supernode", event.EventData{ event.KeySupernode: sn.GrpcEndpoint, @@ -118,10 +116,6 @@ func (t *CascadeTask) attemptRegistration(ctx context.Context, _ int, sn lumera. }) req.EventLogger = func(ctx context.Context, evt event.EventType, msg string, data event.EventData) { - if evt == event.SupernodeArtefactsStored { - t.skipArtifactStorage = true - req.SkipArtifactStorage = true - } t.LogEvent(ctx, evt, msg, data) } // Use ctx directly; per-phase timers are applied inside the adapter diff --git a/supernode/node/action/server/cascade/cascade_action_server.go b/supernode/node/action/server/cascade/cascade_action_server.go index c3255614..a99fbf0a 100644 --- a/supernode/node/action/server/cascade/cascade_action_server.go +++ b/supernode/node/action/server/cascade/cascade_action_server.go @@ -7,13 +7,11 @@ import ( "os" pb "github.com/LumeraProtocol/supernode/v2/gen/supernode/action/cascade" - cascadecommon "github.com/LumeraProtocol/supernode/v2/pkg/cascade" "github.com/LumeraProtocol/supernode/v2/pkg/errors" "github.com/LumeraProtocol/supernode/v2/pkg/logtrace" cascadeService "github.com/LumeraProtocol/supernode/v2/supernode/services/cascade" "google.golang.org/grpc" - "google.golang.org/grpc/metadata" ) type ActionServer struct { @@ -76,12 +74,6 @@ func (server *ActionServer) Register(stream pb.CascadeService_RegisterServer) er } ctx := stream.Context() - skipStorage := false - if md, ok := metadata.FromIncomingContext(ctx); ok { - if values := md.Get(cascadecommon.SkipArtifactStorageHeader); len(values) > 0 && values[0] == cascadecommon.SkipArtifactStorageHeaderValue { - skipStorage = true - } - } logtrace.Info(ctx, "client streaming request to upload cascade input data received", fields) const maxFileSize = 1 * 1024 * 1024 * 1024 // 1GB limit @@ -193,15 +185,12 @@ func (server *ActionServer) Register(stream pb.CascadeService_RegisterServer) er // Process the complete data task := server.factory.NewCascadeRegistrationTask() - fields[cascadecommon.LogFieldSkipStorage] = skipStorage - err = task.Register(ctx, &cascadeService.RegisterRequest{ - TaskID: metadata.TaskId, - ActionID: metadata.ActionId, - DataHash: hash, - DataSize: totalSize, - FilePath: targetPath, - SkipArtifactStorage: skipStorage, + TaskID: metadata.TaskId, + ActionID: metadata.ActionId, + DataHash: hash, + DataSize: totalSize, + FilePath: targetPath, }, func(resp *cascadeService.RegisterResponse) error { grpcResp := &pb.RegisterResponse{ EventType: pb.SupernodeEventType(resp.EventType), @@ -303,19 +292,24 @@ func (server *ActionServer) Download(req *pb.DownloadRequest, stream pb.CascadeS } logtrace.Info(ctx, "streaming artefact file in chunks", fields) - restoredFile, err := readFileContentsInChunks(restoredFilePath) + // Open the restored file and stream directly from disk to avoid buffering entire file in memory + f, err := os.Open(restoredFilePath) if err != nil { - logtrace.Error(ctx, "failed to read restored file", logtrace.Fields{ - logtrace.FieldError: err.Error(), - }) + logtrace.Error(ctx, "failed to open restored file", logtrace.Fields{logtrace.FieldError: err.Error()}) + return err + } + defer f.Close() + + fi, err := f.Stat() + if err != nil { + logtrace.Error(ctx, "failed to stat restored file", logtrace.Fields{logtrace.FieldError: err.Error()}) return err } - logtrace.Info(ctx, "file has been read in chunks", fields) // Calculate optimal chunk size based on file size - chunkSize := calculateOptimalChunkSize(int64(len(restoredFile))) + chunkSize := calculateOptimalChunkSize(fi.Size()) logtrace.Info(ctx, "calculated optimal chunk size for download", logtrace.Fields{ - "file_size": len(restoredFile), + "file_size": fi.Size(), "chunk_size": chunkSize, }) @@ -332,58 +326,30 @@ func (server *ActionServer) Download(req *pb.DownloadRequest, stream pb.CascadeS return err } - // Split and stream the file using adaptive chunk size - for i := 0; i < len(restoredFile); i += chunkSize { - end := i + chunkSize - if end > len(restoredFile) { - end = len(restoredFile) - } - - err := stream.Send(&pb.DownloadResponse{ - ResponseType: &pb.DownloadResponse_Chunk{ - Chunk: &pb.DataChunk{ - Data: restoredFile[i:end], - }, - }, - }) - - if err != nil { - logtrace.Error(ctx, "failed to stream chunk", logtrace.Fields{ - logtrace.FieldError: err.Error(), - }) - return err - } - } - - // Cleanup is handled in deferred block above - - logtrace.Info(ctx, "completed streaming all chunks", fields) - return nil -} - -func readFileContentsInChunks(filePath string) ([]byte, error) { - f, err := os.Open(filePath) - if err != nil { - return nil, err - } - defer f.Close() - - buf := make([]byte, 1024*1024) - var fileBytes []byte - + // Stream the file in fixed-size chunks + buf := make([]byte, chunkSize) for { n, readErr := f.Read(buf) if n > 0 { - // Process chunk - fileBytes = append(fileBytes, buf[:n]...) + if err := stream.Send(&pb.DownloadResponse{ + ResponseType: &pb.DownloadResponse_Chunk{ + Chunk: &pb.DataChunk{Data: buf[:n]}, + }, + }); err != nil { + logtrace.Error(ctx, "failed to stream chunk", logtrace.Fields{logtrace.FieldError: err.Error()}) + return err + } } if readErr == io.EOF { break } if readErr != nil { - return nil, fmt.Errorf("chunked read failed: %w", readErr) + return fmt.Errorf("chunked read failed: %w", readErr) } } - return fileBytes, nil + // Cleanup is handled in deferred block above + + logtrace.Info(ctx, "completed streaming all chunks", fields) + return nil } diff --git a/supernode/node/supernode/server/server.go b/supernode/node/supernode/server/server.go index 7ded8eb7..37e8f4dd 100644 --- a/supernode/node/supernode/server/server.go +++ b/supernode/node/supernode/server/server.go @@ -61,13 +61,13 @@ func (server *Server) Run(ctx context.Context) error { // Tuned for 1GB max files with 4MB chunks. Reduce in-flight memory. opts := grpcserver.DefaultServerOptions() - opts.MaxRecvMsgSize = (8 * 1024 * 1024) // 8MB supports 4MB chunks + overhead - opts.MaxSendMsgSize = (8 * 1024 * 1024) // 8MB for download streaming - opts.InitialWindowSize = (4 * 1024 * 1024) // 4MB per-stream window ~ chunk size - opts.InitialConnWindowSize = (64 * 1024 * 1024) // 64MB per-connection window - opts.MaxConcurrentStreams = 20 // Prevent resource exhaustion - opts.ReadBufferSize = (1 * 1024 * 1024) // 1MB TCP buffer - opts.WriteBufferSize = (1 * 1024 * 1024) // 1MB TCP buffer + opts.MaxRecvMsgSize = (16 * 1024 * 1024) // 16MB (supports 4MB chunks + overhead) + opts.MaxSendMsgSize = (16 * 1024 * 1024) // 16MB for download streaming + opts.InitialWindowSize = (16 * 1024 * 1024) // 16MB per stream (4x chunk size) + opts.InitialConnWindowSize = (160 * 1024 * 1024) // 160MB (16MB x 10 streams) + opts.MaxConcurrentStreams = 20 // Limit to prevent resource exhaustion + opts.ReadBufferSize = (8 * 1024 * 1024) // 8MB TCP buffer + opts.WriteBufferSize = (8 * 1024 * 1024) // 8MB TCP buffer for _, address := range addresses { addr := net.JoinHostPort(strings.TrimSpace(address), strconv.Itoa(server.config.Port)) diff --git a/supernode/node/supernode/server/status_server.go b/supernode/node/supernode/server/status_server.go index 5a8cc156..d90b1e3e 100644 --- a/supernode/node/supernode/server/status_server.go +++ b/supernode/node/supernode/server/status_server.go @@ -174,6 +174,69 @@ func (s *SupernodeServer) GetStatus(ctx context.Context, req *pb.StatusRequest) pbpm.Disk.UsedMb = pm.Disk.UsedMB pbpm.Disk.FreeMb = pm.Disk.FreeMB + // Recent batch store + for _, e := range pm.RecentBatchStore { + pbpm.RecentBatchStore = append(pbpm.RecentBatchStore, &pb.StatusResponse_P2PMetrics_RecentBatchStoreEntry{ + TimeUnix: e.TimeUnix, + SenderId: e.SenderID, + SenderIp: e.SenderIP, + Keys: int32(e.Keys), + DurationMs: e.DurationMS, + Ok: e.OK, + Error: e.Error, + }) + } + // Recent batch retrieve + for _, e := range pm.RecentBatchRetrieve { + pbpm.RecentBatchRetrieve = append(pbpm.RecentBatchRetrieve, &pb.StatusResponse_P2PMetrics_RecentBatchRetrieveEntry{ + TimeUnix: e.TimeUnix, + SenderId: e.SenderID, + SenderIp: e.SenderIP, + Requested: int32(e.Requested), + Found: int32(e.Found), + DurationMs: e.DurationMS, + Error: e.Error, + }) + } + + // Per-IP buckets + if pm.RecentBatchStoreByIP != nil { + pbpm.RecentBatchStoreByIp = map[string]*pb.StatusResponse_P2PMetrics_RecentBatchStoreList{} + for ip, list := range pm.RecentBatchStoreByIP { + pbList := &pb.StatusResponse_P2PMetrics_RecentBatchStoreList{} + for _, e := range list { + pbList.Entries = append(pbList.Entries, &pb.StatusResponse_P2PMetrics_RecentBatchStoreEntry{ + TimeUnix: e.TimeUnix, + SenderId: e.SenderID, + SenderIp: e.SenderIP, + Keys: int32(e.Keys), + DurationMs: e.DurationMS, + Ok: e.OK, + Error: e.Error, + }) + } + pbpm.RecentBatchStoreByIp[ip] = pbList + } + } + if pm.RecentBatchRetrieveByIP != nil { + pbpm.RecentBatchRetrieveByIp = map[string]*pb.StatusResponse_P2PMetrics_RecentBatchRetrieveList{} + for ip, list := range pm.RecentBatchRetrieveByIP { + pbList := &pb.StatusResponse_P2PMetrics_RecentBatchRetrieveList{} + for _, e := range list { + pbList.Entries = append(pbList.Entries, &pb.StatusResponse_P2PMetrics_RecentBatchRetrieveEntry{ + TimeUnix: e.TimeUnix, + SenderId: e.SenderID, + SenderIp: e.SenderIP, + Requested: int32(e.Requested), + Found: int32(e.Found), + DurationMs: e.DurationMS, + Error: e.Error, + }) + } + pbpm.RecentBatchRetrieveByIp[ip] = pbList + } + } + response.P2PMetrics = pbpm } diff --git a/supernode/services/cascade/adaptors/mocks/p2p_mock.go b/supernode/services/cascade/adaptors/mocks/p2p_mock.go index 025109b2..4f62a440 100644 --- a/supernode/services/cascade/adaptors/mocks/p2p_mock.go +++ b/supernode/services/cascade/adaptors/mocks/p2p_mock.go @@ -43,12 +43,11 @@ func (m *MockP2PService) EXPECT() *MockP2PServiceMockRecorder { } // StoreArtefacts mocks base method. -func (m *MockP2PService) StoreArtefacts(ctx context.Context, req adaptors.StoreArtefactsRequest, f logtrace.Fields) (adaptors.StoreArtefactsMetrics, error) { +func (m *MockP2PService) StoreArtefacts(ctx context.Context, req adaptors.StoreArtefactsRequest, f logtrace.Fields) error { m.ctrl.T.Helper() ret := m.ctrl.Call(m, "StoreArtefacts", ctx, req, f) - ret0, _ := ret[0].(adaptors.StoreArtefactsMetrics) - ret1, _ := ret[1].(error) - return ret0, ret1 + ret0, _ := ret[0].(error) + return ret0 } // StoreArtefacts indicates an expected call of StoreArtefacts. diff --git a/supernode/services/cascade/adaptors/p2p.go b/supernode/services/cascade/adaptors/p2p.go index 78e9b1bf..116d6810 100644 --- a/supernode/services/cascade/adaptors/p2p.go +++ b/supernode/services/cascade/adaptors/p2p.go @@ -13,6 +13,7 @@ import ( "github.com/LumeraProtocol/supernode/v2/p2p" "github.com/LumeraProtocol/supernode/v2/pkg/logtrace" + cm "github.com/LumeraProtocol/supernode/v2/pkg/p2pmetrics" "github.com/LumeraProtocol/supernode/v2/pkg/storage/rqstore" "github.com/LumeraProtocol/supernode/v2/pkg/utils" "github.com/LumeraProtocol/supernode/v2/supernode/services/common/storage" @@ -20,7 +21,7 @@ import ( ) const ( - loadSymbolsBatchSize = 2500 + loadSymbolsBatchSize = 3000 // Minimum first-pass coverage to store before returning from Register (percent) storeSymbolsPercent = 18 @@ -32,16 +33,8 @@ const ( //go:generate mockgen -destination=mocks/p2p_mock.go -package=cascadeadaptormocks -source=p2p.go type P2PService interface { // StoreArtefacts stores ID files and RaptorQ symbols. - // - // Aggregation model: - // - Each underlying StoreBatch returns (ratePct, requests) where requests is - // the number of node RPCs. The aggregated success rate can be computed as - // a weighted average by requests across metadata and symbol batches, - // yielding a global success view across all node calls attempted for this action. - // See implementation notes for item‑weighted aggregation currently in use. - // - // Returns detailed metrics for both categories along with an aggregated view. - StoreArtefacts(ctx context.Context, req StoreArtefactsRequest, f logtrace.Fields) (StoreArtefactsMetrics, error) + // Metrics are recorded via internal metrics helpers; no metrics are returned. + StoreArtefacts(ctx context.Context, req StoreArtefactsRequest, f logtrace.Fields) error } // p2pImpl is the default implementation of the P2PService interface. @@ -62,40 +55,23 @@ type StoreArtefactsRequest struct { SymbolsDir string } -// StoreArtefactsMetrics captures detailed outcomes of metadata and symbols storage. -type StoreArtefactsMetrics struct { - // Metadata (ID files) - MetaRate float64 // percentage (0–100) - MetaRequests int // number of node RPCs attempted for metadata - MetaCount int // number of metadata files attempted - - // Symbols - SymRate float64 // percentage (0–100) across all symbol batches (item-weighted) - SymRequests int // total node RPCs for symbol batches - SymCount int // total symbols processed - - // Aggregated view - AggregatedRate float64 // item-weighted across metadata and symbols - TotalRequests int // MetaRequests + SymRequests -} +func (p *p2pImpl) StoreArtefacts(ctx context.Context, req StoreArtefactsRequest, f logtrace.Fields) error { + logtrace.Info(ctx, "About to store artefacts (metadata + symbols)", logtrace.Fields{"taskID": req.TaskID, "id_files": len(req.IDFiles)}) + + // Enable per-node store RPC capture for this task + cm.StartStoreCapture(req.TaskID) + defer cm.StopStoreCapture(req.TaskID) -func (p *p2pImpl) StoreArtefacts(ctx context.Context, req StoreArtefactsRequest, f logtrace.Fields) (StoreArtefactsMetrics, error) { - logtrace.Info(ctx, "About to store ID files", logtrace.Fields{"taskID": req.TaskID, "fileCount": len(req.IDFiles)}) - // NOTE: For now we aggregate by item count (ID files + symbol count). - // TODO(move-to-request-weighted): Switch aggregation to request-weighted once - // external consumers and metrics expectations are updated. We already return - // totalRequests so the event/logs can include accurate request counts. - symRate, symCount, symReqs, err := p.storeCascadeSymbolsAndData(ctx, req.TaskID, req.ActionID, req.SymbolsDir, req.IDFiles) + start := time.Now() + firstPassSymbols, totalSymbols, err := p.storeCascadeSymbolsAndData(ctx, req.TaskID, req.ActionID, req.SymbolsDir, req.IDFiles) if err != nil { - return StoreArtefactsMetrics{}, errors.Wrap(err, "error storing raptor-q symbols") + return errors.Wrap(err, "error storing artefacts") } - logtrace.Info(ctx, "raptor-q symbols have been stored", f) - - return StoreArtefactsMetrics{ - SymRate: symRate, - SymRequests: symReqs, - SymCount: symCount, - }, nil + dur := time.Since(start).Milliseconds() + logtrace.Info(ctx, "artefacts have been stored", logtrace.Fields{"taskID": req.TaskID, "symbols_first_pass": firstPassSymbols, "symbols_total": totalSymbols, "id_files_count": len(req.IDFiles)}) + // Record store summary for later event emission + cm.SetStoreSummary(req.TaskID, firstPassSymbols, totalSymbols, len(req.IDFiles), dur) + return nil } // storeCascadeSymbols loads symbols from `symbolsDir`, optionally downsamples, @@ -105,16 +81,16 @@ func (p *p2pImpl) StoreArtefacts(ctx context.Context, req StoreArtefactsRequest, // - the total number of node requests attempted across batches // // Returns (aggRate, totalSymbols, totalRequests, err). -func (p *p2pImpl) storeCascadeSymbolsAndData(ctx context.Context, taskID, actionID string, symbolsDir string, metadataFiles [][]byte) (float64, int, int, error) { +func (p *p2pImpl) storeCascadeSymbolsAndData(ctx context.Context, taskID, actionID string, symbolsDir string, metadataFiles [][]byte) (int, int, error) { /* record directory in DB */ if err := p.rqStore.StoreSymbolDirectory(taskID, symbolsDir); err != nil { - return 0, 0, 0, fmt.Errorf("store symbol dir: %w", err) + return 0, 0, fmt.Errorf("store symbol dir: %w", err) } /* gather every symbol path under symbolsDir ------------------------- */ keys, err := walkSymbolTree(symbolsDir) if err != nil { - return 0, 0, 0, err + return 0, 0, err } totalAvailable := len(keys) @@ -142,10 +118,7 @@ func (p *p2pImpl) storeCascadeSymbolsAndData(ctx context.Context, taskID, action /* stream in fixed-size batches -------------------------------------- */ - sumWeightedRates := 0.0 - totalSymbols := 0 // symbols only - totalItems := 0 // symbols + metadata (for rate weighting) - totalRequests := 0 + totalSymbols := 0 // symbols stored firstBatchProcessed := false for start := 0; start < len(keys); { @@ -168,7 +141,7 @@ func (p *p2pImpl) storeCascadeSymbolsAndData(ctx context.Context, taskID, action // Load just this symbol chunk symBytes, err := utils.LoadSymbols(symbolsDir, batch) if err != nil { - return 0, totalSymbols, totalRequests, fmt.Errorf("load symbols: %w", err) + return 0, 0, fmt.Errorf("load symbols: %w", err) } // Build combined payload: metadata first, then symbols @@ -178,44 +151,30 @@ func (p *p2pImpl) storeCascadeSymbolsAndData(ctx context.Context, taskID, action // Send as the same data type you use for symbols bctx, cancel := context.WithTimeout(ctx, storeBatchContextTimeout) - rate, reqs, err := p.p2p.StoreBatch(bctx, payload, storage.P2PDataRaptorQSymbol, taskID) + bctx = cm.WithTaskID(bctx, taskID) + err = p.p2p.StoreBatch(bctx, payload, storage.P2PDataRaptorQSymbol, taskID) cancel() if err != nil { - agg := 0.0 - if totalItems > 0 { - agg = sumWeightedRates / float64(totalItems) - } - return agg, totalSymbols, totalRequests + reqs, fmt.Errorf("p2p store batch (first): %w", err) + return totalSymbols, totalAvailable, fmt.Errorf("p2p store batch (first): %w", err) } - // Metrics - items := len(payload) // meta + symbols - sumWeightedRates += rate * float64(items) - totalItems += items totalSymbols += len(symBytes) - totalRequests += reqs + // No per-RPC metrics propagated from p2p // Delete only the symbols we uploaded if len(batch) > 0 { if err := utils.DeleteSymbols(ctx, symbolsDir, batch); err != nil { - return rate, totalSymbols, totalRequests, fmt.Errorf("delete symbols: %w", err) + return totalSymbols, totalAvailable, fmt.Errorf("delete symbols: %w", err) } } firstBatchProcessed = true } else { - rate, requests, count, err := p.storeSymbolsInP2P(ctx, taskID, symbolsDir, batch) + count, err := p.storeSymbolsInP2P(ctx, taskID, symbolsDir, batch) if err != nil { - agg := 0.0 - if totalItems > 0 { - agg = sumWeightedRates / float64(totalItems) - } - return agg, totalSymbols, totalRequests, err + return totalSymbols, totalAvailable, err } - sumWeightedRates += rate * float64(count) - totalItems += count totalSymbols += count - totalRequests += requests } start = end @@ -227,17 +186,13 @@ func (p *p2pImpl) storeCascadeSymbolsAndData(ctx context.Context, taskID, action achievedPct = (float64(totalSymbols) / float64(totalAvailable)) * 100.0 } logtrace.Info(ctx, "first-pass achieved coverage (symbols)", - logtrace.Fields{"achieved_symbols": totalSymbols, "achieved_percent": achievedPct, "total_requests": totalRequests}) + logtrace.Fields{"achieved_symbols": totalSymbols, "achieved_percent": achievedPct}) if err := p.rqStore.UpdateIsFirstBatchStored(actionID); err != nil { - return 0, totalSymbols, totalRequests, fmt.Errorf("update first-batch flag: %w", err) + return totalSymbols, totalAvailable, fmt.Errorf("update first-batch flag: %w", err) } - aggRate := 0.0 - if totalItems > 0 { - aggRate = sumWeightedRates / float64(totalItems) - } - return aggRate, totalSymbols, totalRequests, nil + return totalSymbols, totalAvailable, nil } @@ -269,27 +224,28 @@ func walkSymbolTree(root string) ([]string, error) { // storeSymbolsInP2P loads a batch of symbols and stores them via P2P. // Returns (ratePct, requests, count, error) where `count` is the number of symbols in this batch. -func (c *p2pImpl) storeSymbolsInP2P(ctx context.Context, taskID, root string, fileKeys []string) (float64, int, int, error) { +func (c *p2pImpl) storeSymbolsInP2P(ctx context.Context, taskID, root string, fileKeys []string) (int, error) { logtrace.Info(ctx, "loading batch symbols", logtrace.Fields{"count": len(fileKeys)}) symbols, err := utils.LoadSymbols(root, fileKeys) if err != nil { - return 0, 0, 0, fmt.Errorf("load symbols: %w", err) + return 0, fmt.Errorf("load symbols: %w", err) } symCtx, cancel := context.WithTimeout(ctx, storeBatchContextTimeout) + symCtx = cm.WithTaskID(symCtx, taskID) defer cancel() - rate, requests, err := c.p2p.StoreBatch(symCtx, symbols, storage.P2PDataRaptorQSymbol, taskID) - if err != nil { - return rate, requests, len(symbols), fmt.Errorf("p2p store batch: %w", err) + if err := c.p2p.StoreBatch(symCtx, symbols, storage.P2PDataRaptorQSymbol, taskID); err != nil { + return len(symbols), fmt.Errorf("p2p store batch: %w", err) } logtrace.Info(ctx, "stored batch symbols", logtrace.Fields{"count": len(symbols)}) if err := utils.DeleteSymbols(ctx, root, fileKeys); err != nil { - return rate, requests, len(symbols), fmt.Errorf("delete symbols: %w", err) + return len(symbols), fmt.Errorf("delete symbols: %w", err) } logtrace.Info(ctx, "deleted batch symbols", logtrace.Fields{"count": len(symbols)}) - return rate, requests, len(symbols), nil + // No per-RPC metrics propagated from p2p + return len(symbols), nil } diff --git a/supernode/services/cascade/download.go b/supernode/services/cascade/download.go index 3c4e640e..bfcc25a9 100644 --- a/supernode/services/cascade/download.go +++ b/supernode/services/cascade/download.go @@ -3,23 +3,23 @@ package cascade import ( "bytes" "context" + "encoding/json" "fmt" "os" "sort" + "time" actiontypes "github.com/LumeraProtocol/lumera/x/action/v1/types" "github.com/LumeraProtocol/supernode/v2/pkg/codec" "github.com/LumeraProtocol/supernode/v2/pkg/crypto" "github.com/LumeraProtocol/supernode/v2/pkg/errors" "github.com/LumeraProtocol/supernode/v2/pkg/logtrace" + cm "github.com/LumeraProtocol/supernode/v2/pkg/p2pmetrics" "github.com/LumeraProtocol/supernode/v2/pkg/utils" + "github.com/LumeraProtocol/supernode/v2/supernode/services/cascade/adaptors" "github.com/LumeraProtocol/supernode/v2/supernode/services/common" ) -const ( - requiredSymbolPercent = 17 -) - type DownloadRequest struct { ActionID string } @@ -76,7 +76,7 @@ func (task *CascadeRegistrationTask) Download( // Notify: network retrieval phase begins task.streamDownloadEvent(SupernodeEventTypeNetworkRetrieveStarted, "Network retrieval started", "", "", send) - filePath, tmpDir, err := task.downloadArtifacts(ctx, actionDetails.GetAction().ActionID, metadata, fields) + filePath, tmpDir, err := task.downloadArtifacts(ctx, actionDetails.GetAction().ActionID, metadata, fields, send) if err != nil { fields[logtrace.FieldError] = err.Error() return task.wrapErr(ctx, "failed to download artifacts", err, fields) @@ -88,10 +88,15 @@ func (task *CascadeRegistrationTask) Download( return nil } -func (task *CascadeRegistrationTask) downloadArtifacts(ctx context.Context, actionID string, metadata actiontypes.CascadeMetadata, fields logtrace.Fields) (string, string, error) { +func (task *CascadeRegistrationTask) downloadArtifacts(ctx context.Context, actionID string, metadata actiontypes.CascadeMetadata, fields logtrace.Fields, send func(resp *DownloadResponse) error) (string, string, error) { logtrace.Info(ctx, "started downloading the artifacts", fields) - var layout codec.Layout + var ( + layout codec.Layout + layoutFetchMS int64 + layoutDecodeMS int64 + layoutAttempts int + ) for _, indexID := range metadata.RqIdsIds { indexFile, err := task.P2PClient.Retrieve(ctx, indexID) @@ -107,11 +112,14 @@ func (task *CascadeRegistrationTask) downloadArtifacts(ctx context.Context, acti } // Try to retrieve layout files using layout IDs from index file - layout, err = task.retrieveLayoutFromIndex(ctx, indexData, fields) + var netMS, decMS int64 + layout, netMS, decMS, layoutAttempts, err = task.retrieveLayoutFromIndex(ctx, indexData, fields) if err != nil { logtrace.Info(ctx, "failed to retrieve layout from index", fields) continue } + layoutFetchMS = netMS + layoutDecodeMS = decMS if len(layout.Blocks) > 0 { logtrace.Info(ctx, "layout file retrieved via index", fields) @@ -122,8 +130,11 @@ func (task *CascadeRegistrationTask) downloadArtifacts(ctx context.Context, acti if len(layout.Blocks) == 0 { return "", "", errors.New("no symbols found in RQ metadata") } - - return task.restoreFileFromLayout(ctx, layout, metadata.DataHash, actionID) + // Persist layout timing in fields for downstream metrics + fields["layout_fetch_ms"] = layoutFetchMS + fields["layout_decode_ms"] = layoutDecodeMS + fields["layout_attempts"] = layoutAttempts + return task.restoreFileFromLayout(ctx, layout, metadata.DataHash, actionID, send) } func (task *CascadeRegistrationTask) restoreFileFromLayout( @@ -131,6 +142,7 @@ func (task *CascadeRegistrationTask) restoreFileFromLayout( layout codec.Layout, dataHash string, actionID string, + send func(resp *DownloadResponse) error, ) (string, string, error) { fields := logtrace.Fields{ @@ -143,19 +155,44 @@ func (task *CascadeRegistrationTask) restoreFileFromLayout( sort.Strings(allSymbols) totalSymbols := len(allSymbols) - requiredSymbols := (totalSymbols*requiredSymbolPercent + 99) / 100 - fields["totalSymbols"] = totalSymbols - fields["requiredSymbols"] = requiredSymbols - logtrace.Info(ctx, "Symbols to be retrieved", fields) + logtrace.Info(ctx, "Retrieving all symbols for decode", fields) + + // Enable retrieve metrics capture for this action + cm.StartRetrieveCapture(actionID) + defer cm.StopRetrieveCapture(actionID) - // Progressive retrieval moved to helper for readability/testing - decodeInfo, err := task.retrieveAndDecodeProgressively(ctx, layout, actionID, fields) + // Measure symbols batch retrieve duration + retrieveStart := time.Now() + // Tag context with metrics task ID (actionID) + ctxRetrieve := cm.WithTaskID(ctx, actionID) + symbols, err := task.P2PClient.BatchRetrieve(ctxRetrieve, allSymbols, totalSymbols, actionID) if err != nil { fields[logtrace.FieldError] = err.Error() - logtrace.Error(ctx, "failed to decode symbols progressively", fields) + logtrace.Error(ctx, "batch retrieve failed", fields) + return "", "", fmt.Errorf("batch retrieve symbols: %w", err) + } + retrieveMS := time.Since(retrieveStart).Milliseconds() + + // Measure decode duration + decodeStart := time.Now() + decodeInfo, err := task.RQ.Decode(ctx, adaptors.DecodeRequest{ + ActionID: actionID, + Symbols: symbols, + Layout: layout, + }) + if err != nil { + fields[logtrace.FieldError] = err.Error() + logtrace.Error(ctx, "decode failed", fields) return "", "", fmt.Errorf("decode symbols using RaptorQ: %w", err) } + decodeMS := time.Since(decodeStart).Milliseconds() + + // Set minimal retrieve summary and emit event strictly from internal collector + cm.SetRetrieveSummary(actionID, retrieveMS, decodeMS) + if b, err := json.MarshalIndent(cm.BuildDownloadEventPayloadFromCollector(actionID), "", " "); err == nil { + task.streamDownloadEvent(SupernodeEventTypeArtefactsDownloaded, string(b), "", "", send) + } fileHash, err := crypto.HashFileIncrementally(decodeInfo.FilePath, 0) if err != nil { @@ -207,25 +244,35 @@ func (task *CascadeRegistrationTask) parseIndexFile(data []byte) (IndexFile, err } // retrieveLayoutFromIndex retrieves layout file using layout IDs from index file -func (task *CascadeRegistrationTask) retrieveLayoutFromIndex(ctx context.Context, indexData IndexFile, fields logtrace.Fields) (codec.Layout, error) { +func (task *CascadeRegistrationTask) retrieveLayoutFromIndex(ctx context.Context, indexData IndexFile, fields logtrace.Fields) (codec.Layout, int64, int64, int, error) { // Try to retrieve layout files using layout IDs from index file + var ( + totalFetchMS int64 + totalDecodeMS int64 + attempts int + ) for _, layoutID := range indexData.LayoutIDs { + attempts++ + t0 := time.Now() layoutFile, err := task.P2PClient.Retrieve(ctx, layoutID) + totalFetchMS += time.Since(t0).Milliseconds() if err != nil || len(layoutFile) == 0 { continue } + t1 := time.Now() layout, _, _, err := parseRQMetadataFile(layoutFile) + totalDecodeMS += time.Since(t1).Milliseconds() if err != nil { continue } if len(layout.Blocks) > 0 { - return layout, nil + return layout, totalFetchMS, totalDecodeMS, attempts, nil } } - return codec.Layout{}, errors.New("no valid layout found in index") + return codec.Layout{}, totalFetchMS, totalDecodeMS, attempts, errors.New("no valid layout found in index") } func (task *CascadeRegistrationTask) CleanupDownload(ctx context.Context, actionID string) error { diff --git a/supernode/services/cascade/helper.go b/supernode/services/cascade/helper.go index eed20700..fb8c7ef5 100644 --- a/supernode/services/cascade/helper.go +++ b/supernode/services/cascade/helper.go @@ -5,7 +5,6 @@ import ( "context" "encoding/base64" "fmt" - stdmath "math" "strconv" "strings" @@ -17,6 +16,7 @@ import ( "github.com/LumeraProtocol/supernode/v2/pkg/lumera/modules/supernode" "github.com/LumeraProtocol/supernode/v2/pkg/utils" "github.com/LumeraProtocol/supernode/v2/supernode/services/cascade/adaptors" + cm "github.com/LumeraProtocol/supernode/v2/pkg/p2pmetrics" sdk "github.com/cosmos/cosmos-sdk/types" "github.com/golang/protobuf/proto" @@ -25,6 +25,8 @@ import ( "google.golang.org/grpc/status" ) +// layout stats helpers removed to keep download metrics minimal. + func (task *CascadeRegistrationTask) fetchAction(ctx context.Context, actionID string, f logtrace.Fields) (*actiontypes.Action, error) { res, err := task.LumeraClient.GetAction(ctx, actionID) if err != nil { @@ -171,14 +173,8 @@ func (task *CascadeRegistrationTask) generateRQIDFiles(ctx context.Context, meta } // storeArtefacts persists cascade artefacts (ID files + RaptorQ symbols) via the -// P2P adaptor and returns an aggregated network success rate percentage and total -// node requests used to compute it. -// -// Aggregation details: -// - Underlying batches return (ratePct, requests) where `requests` is the number -// of node RPCs attempted. The adaptor computes a weighted average by requests -// across all batches, reflecting the overall network success rate. -func (task *CascadeRegistrationTask) storeArtefacts(ctx context.Context, actionID string, idFiles [][]byte, symbolsDir string, f logtrace.Fields) (adaptors.StoreArtefactsMetrics, error) { +// P2P adaptor. P2P does not return metrics; cascade summarizes and emits them. +func (task *CascadeRegistrationTask) storeArtefacts(ctx context.Context, actionID string, idFiles [][]byte, symbolsDir string, f logtrace.Fields) error { return task.P2P.StoreArtefacts(ctx, adaptors.StoreArtefactsRequest{ IDFiles: idFiles, SymbolsDir: symbolsDir, @@ -203,22 +199,24 @@ func (task *CascadeRegistrationTask) wrapErr(ctx context.Context, msg string, er // emitArtefactsStored builds a single-line metrics summary and emits the // SupernodeEventTypeArtefactsStored event while logging the metrics line. func (task *CascadeRegistrationTask) emitArtefactsStored( - ctx context.Context, - metrics adaptors.StoreArtefactsMetrics, - fields logtrace.Fields, - send func(resp *RegisterResponse) error, + ctx context.Context, + fields logtrace.Fields, + _ codec.Layout, + send func(resp *RegisterResponse) error, ) { - ok := int(stdmath.Round(metrics.AggregatedRate / 100.0 * float64(metrics.TotalRequests))) - fail := metrics.TotalRequests - ok - line := fmt.Sprintf( - "artefacts stored | success_rate=%.2f%% agg_rate=%.2f%% total_reqs=%d ok=%d fail=%d meta_rate=%.2f%% meta_reqs=%d meta_count=%d sym_rate=%.2f%% sym_reqs=%d sym_count=%d", - metrics.AggregatedRate, metrics.AggregatedRate, metrics.TotalRequests, ok, fail, - metrics.MetaRate, metrics.MetaRequests, metrics.MetaCount, - metrics.SymRate, metrics.SymRequests, metrics.SymCount, - ) - fields["metrics"] = line + if fields == nil { + fields = logtrace.Fields{} + } + + // Build payload strictly from internal collector (no P2P snapshots) + payload := cm.BuildStoreEventPayloadFromCollector(task.ID()) + + b, _ := json.MarshalIndent(payload, "", " ") + msg := string(b) + fields["metrics_json"] = msg logtrace.Info(ctx, "artefacts have been stored", fields) - task.streamEvent(SupernodeEventTypeArtefactsStored, line, "", send) + task.streamEvent(SupernodeEventTypeArtefactsStored, msg, "", send) + // No central state to clear; adaptor returns calls inline } // extractSignatureAndFirstPart extracts the signature and first part from the encoded data diff --git a/supernode/services/cascade/progressive_decode.go b/supernode/services/cascade/progressive_decode.go deleted file mode 100644 index 3db58988..00000000 --- a/supernode/services/cascade/progressive_decode.go +++ /dev/null @@ -1,82 +0,0 @@ -package cascade - -import ( - "context" - "fmt" - - "github.com/LumeraProtocol/supernode/v2/pkg/codec" - "github.com/LumeraProtocol/supernode/v2/pkg/logtrace" - "github.com/LumeraProtocol/supernode/v2/supernode/services/cascade/adaptors" -) - -// retrieveAndDecodeProgressively performs a minimal two-step retrieval for a single-block layout: -// 1) Send ALL keys with a minimum required count (requiredSymbolPercent). -// 2) If decode fails, escalate by asking for ALL symbols (required = total). -func (task *CascadeRegistrationTask) retrieveAndDecodeProgressively(ctx context.Context, layout codec.Layout, actionID string, - fields logtrace.Fields) (adaptors.DecodeResponse, error) { - if fields == nil { - fields = logtrace.Fields{} - } - fields[logtrace.FieldActionID] = actionID - - if len(layout.Blocks) == 0 { - return adaptors.DecodeResponse{}, fmt.Errorf("empty layout: no blocks") - } - - // Single-block path - if len(layout.Blocks) == 1 { - blk := layout.Blocks[0] - total := len(blk.Symbols) - if total == 0 { - return adaptors.DecodeResponse{}, fmt.Errorf("empty layout: no symbols") - } - - // Step 1: send ALL keys, require only reqCount - reqCount := (total*requiredSymbolPercent + 99) / 100 - if reqCount < 1 { - reqCount = 1 - } else if reqCount > total { - reqCount = total - } - fields["targetPercent"] = requiredSymbolPercent - fields["targetCount"] = reqCount - fields["total"] = total - logtrace.Info(ctx, "retrieving initial symbols (single block)", fields) - - symbols, err := task.P2PClient.BatchRetrieve(ctx, blk.Symbols, reqCount, actionID) - if err != nil { - fields[logtrace.FieldError] = err.Error() - logtrace.Error(ctx, "failed to retrieve symbols", fields) - return adaptors.DecodeResponse{}, fmt.Errorf("failed to retrieve symbols: %w", err) - } - - decodeInfo, err := task.RQ.Decode(ctx, adaptors.DecodeRequest{ - ActionID: actionID, - Symbols: symbols, - Layout: layout, - }) - if err == nil { - return decodeInfo, nil - } - - // Step 2: escalate to require ALL symbols - fields["escalating"] = true - fields["requiredCount"] = total - logtrace.Info(ctx, "initial decode failed; retrieving all symbols (single block)", fields) - - symbols, err = task.P2PClient.BatchRetrieve(ctx, blk.Symbols, reqCount*2, actionID) - if err != nil { - fields[logtrace.FieldError] = err.Error() - logtrace.Error(ctx, "failed to retrieve all symbols", fields) - return adaptors.DecodeResponse{}, fmt.Errorf("failed to retrieve symbols: %w", err) - } - - return task.RQ.Decode(ctx, adaptors.DecodeRequest{ - ActionID: actionID, - Symbols: symbols, - Layout: layout, - }) - } - - return adaptors.DecodeResponse{}, fmt.Errorf("unsupported layout: expected 1 block, found %d", len(layout.Blocks)) -} diff --git a/supernode/services/cascade/register.go b/supernode/services/cascade/register.go index 89609a08..dd6e1e77 100644 --- a/supernode/services/cascade/register.go +++ b/supernode/services/cascade/register.go @@ -4,19 +4,17 @@ import ( "context" "os" - cascadecommon "github.com/LumeraProtocol/supernode/v2/pkg/cascade" "github.com/LumeraProtocol/supernode/v2/pkg/logtrace" "github.com/LumeraProtocol/supernode/v2/supernode/services/common" ) // RegisterRequest contains parameters for upload request type RegisterRequest struct { - TaskID string - ActionID string - DataHash []byte - DataSize int - FilePath string - SkipArtifactStorage bool + TaskID string + ActionID string + DataHash []byte + DataSize int + FilePath string } // RegisterResponse contains the result of upload @@ -159,24 +157,13 @@ func (task *CascadeRegistrationTask) Register( task.streamEvent(SupernodeEventTypeFinalizeSimulated, "Finalize simulation passed", "", send) /* 11. Persist artefacts -------------------------------------------------------- */ - // Persist artefacts to the P2P network unless explicitly skipped by the client. - // Aggregation model (context): - // - Each underlying StoreBatch returns (ratePct, requests) where requests is - // the number of node RPCs. The aggregated success rate can be computed as a - // weighted average by requests across metadata and symbol batches, yielding - // an overall network success view for the action. - if req.SkipArtifactStorage { - fields[cascadecommon.LogFieldSkipStorage] = true - logtrace.Info(ctx, "Artifact storage skipped at client request", fields) - task.streamEvent(SupernodeEventTypeArtefactsStored, cascadecommon.ArtifactStorageSkippedMessage, "", send) - } else { - metrics, err := task.storeArtefacts(ctx, action.ActionID, rqidResp.RedundantMetadataFiles, encResp.SymbolsDir, fields) - if err != nil { - return err - } - // Emit single-line metrics via helper to keep Register clean - task.emitArtefactsStored(ctx, metrics, fields, send) + // Persist artefacts to the P2P network. P2P interfaces return error only; + // metrics are summarized at the cascade layer and emitted via event. + if err := task.storeArtefacts(ctx, action.ActionID, rqidResp.RedundantMetadataFiles, encResp.SymbolsDir, fields); err != nil { + return err } + // Emit compact analytics payload from centralized metrics collector + task.emitArtefactsStored(ctx, fields, encResp.Metadata, send) resp, err := task.LumeraClient.FinalizeAction(ctx, action.ActionID, rqidResp.RQIDs) if err != nil { diff --git a/supernode/services/cascade/register_test.go b/supernode/services/cascade/register_test.go index 82af32fe..c73b96b7 100644 --- a/supernode/services/cascade/register_test.go +++ b/supernode/services/cascade/register_test.go @@ -104,19 +104,10 @@ func TestCascadeRegistrationTask_Register(t *testing.T) { Metadata: codecpkg.Layout{Blocks: []codecpkg.Block{{BlockID: 1, Hash: "abc"}}}, }, nil) - // 8. Store artefacts (returns detailed metrics) - p2p.EXPECT(). - StoreArtefacts(gomock.Any(), gomock.Any(), gomock.Any()). - Return(adaptors.StoreArtefactsMetrics{ - MetaRate: 96.0, - MetaRequests: 20, - MetaCount: 2, - SymRate: 94.0, - SymRequests: 100, - SymCount: 1000, - AggregatedRate: 95.0, - TotalRequests: 120, - }, nil) + // 8. Store artefacts (no metrics returned; recorded centrally) + p2p.EXPECT(). + StoreArtefacts(gomock.Any(), gomock.Any(), gomock.Any()). + Return(nil) }, expectedError: "", expectedEvents: 12, diff --git a/supernode/services/common/storage/handler.go b/supernode/services/common/storage/handler.go index ae80615b..210dab0f 100644 --- a/supernode/services/common/storage/handler.go +++ b/supernode/services/common/storage/handler.go @@ -14,6 +14,7 @@ import ( "github.com/LumeraProtocol/supernode/v2/p2p" "github.com/LumeraProtocol/supernode/v2/pkg/errors" "github.com/LumeraProtocol/supernode/v2/pkg/logtrace" + "github.com/LumeraProtocol/supernode/v2/pkg/p2pmetrics" "github.com/LumeraProtocol/supernode/v2/pkg/storage/files" "github.com/LumeraProtocol/supernode/v2/pkg/storage/rqstore" "github.com/LumeraProtocol/supernode/v2/pkg/utils" @@ -66,10 +67,6 @@ func (h *StorageHandler) StoreBytesIntoP2P(ctx context.Context, data []byte, typ } // StoreBatch stores into P2P an array of byte slices. -// -// Note: The underlying P2P client returns (successRatePct, requests, err). -// This handler intentionally ignores the metrics and only propagates error, -// as callers of this common storage path historically consumed only errors. func (h *StorageHandler) StoreBatch(ctx context.Context, list [][]byte, typ int) error { val := ctx.Value(logtrace.CorrelationIDKey) taskID := "" @@ -78,9 +75,9 @@ func (h *StorageHandler) StoreBatch(ctx context.Context, list [][]byte, typ int) } logtrace.Info(ctx, "task_id in storeList", logtrace.Fields{logtrace.FieldTaskID: taskID}) - - _, _, err := h.P2PClient.StoreBatch(ctx, list, typ, taskID) - return err + // Add taskID to context for metrics + ctx = p2pmetrics.WithTaskID(ctx, taskID) + return h.P2PClient.StoreBatch(ctx, list, typ, taskID) } // StoreRaptorQSymbolsIntoP2P stores RaptorQ symbols into P2P @@ -170,7 +167,9 @@ func (h *StorageHandler) storeSymbolsInP2P(ctx context.Context, taskID, root str return fmt.Errorf("load symbols: %w", err) } - if _, _, err := h.P2PClient.StoreBatch(ctx, symbols, P2PDataRaptorQSymbol, taskID); err != nil { + // Add taskID to context for metrics + ctx = p2pmetrics.WithTaskID(ctx, taskID) + if err := h.P2PClient.StoreBatch(ctx, symbols, P2PDataRaptorQSymbol, taskID); err != nil { return fmt.Errorf("p2p store batch: %w", err) } diff --git a/supernode/services/common/storage/handler_test.go b/supernode/services/common/storage/handler_test.go index e1be29f1..fd4e0d8e 100644 --- a/supernode/services/common/storage/handler_test.go +++ b/supernode/services/common/storage/handler_test.go @@ -49,7 +49,8 @@ func TestStoreBatch(t *testing.T) { ctx := context.WithValue(context.Background(), "task_id", "123") list := [][]byte{[]byte("a"), []byte("b")} - p2pClient.On("StoreBatch", mock.Anything, list, 3, "").Return(0.0, 0, nil) + // StoreBatch now returns error only + p2pClient.On("StoreBatch", mock.Anything, list, 3, "").Return(nil) err := handler.StoreBatch(ctx, list, 3) assert.NoError(t, err) diff --git a/supernode/services/common/supernode/service.go b/supernode/services/common/supernode/service.go index c167eec2..13d5efe4 100644 --- a/supernode/services/common/supernode/service.go +++ b/supernode/services/common/supernode/service.go @@ -1,16 +1,16 @@ package supernode import ( - "context" - "fmt" - "time" - - "github.com/LumeraProtocol/supernode/v2/p2p" - "github.com/LumeraProtocol/supernode/v2/p2p/kademlia" - "github.com/LumeraProtocol/supernode/v2/pkg/logtrace" - "github.com/LumeraProtocol/supernode/v2/pkg/lumera" - "github.com/LumeraProtocol/supernode/v2/pkg/utils" - "github.com/LumeraProtocol/supernode/v2/supernode/config" + "context" + "fmt" + "time" + + "github.com/LumeraProtocol/supernode/v2/p2p" + "github.com/LumeraProtocol/supernode/v2/p2p/kademlia" + "github.com/LumeraProtocol/supernode/v2/pkg/logtrace" + "github.com/LumeraProtocol/supernode/v2/pkg/lumera" + "github.com/LumeraProtocol/supernode/v2/pkg/utils" + "github.com/LumeraProtocol/supernode/v2/supernode/config" ) // Version is the supernode version, set by the main application @@ -125,130 +125,229 @@ func (s *SupernodeStatusService) GetStatus(ctx context.Context, includeP2PMetric PeerAddresses: []string{}, } - // Prepare P2P metrics container (always present in response) - metrics := P2PMetrics{ - NetworkHandleMetrics: map[string]HandleCounters{}, - ConnPoolMetrics: map[string]int64{}, - BanList: []BanEntry{}, - } - - // Collect P2P network information and metrics (fill when available and requested) - if includeP2PMetrics && s.p2pService != nil { - p2pStats, err := s.p2pService.Stats(ctx) - if err != nil { - // Log error but continue - non-critical - logtrace.Error(ctx, "failed to get p2p stats", logtrace.Fields{logtrace.FieldError: err.Error()}) - } else { - if dhtStats, ok := p2pStats["dht"].(map[string]interface{}); ok { - if peersCount, ok := dhtStats["peers_count"].(int); ok { - resp.Network.PeersCount = int32(peersCount) - } - - // Extract peer addresses - if peers, ok := dhtStats["peers"].([]*kademlia.Node); ok { - resp.Network.PeerAddresses = make([]string, 0, len(peers)) - for _, peer := range peers { - // Format peer address as "ID@IP:Port" - peerAddr := fmt.Sprintf("%s@%s:%d", string(peer.ID), peer.IP, peer.Port) - resp.Network.PeerAddresses = append(resp.Network.PeerAddresses, peerAddr) - } - } else { - resp.Network.PeerAddresses = []string{} - } - } - - // Disk info - if du, ok := p2pStats["disk-info"].(utils.DiskStatus); ok { - metrics.Disk = DiskStatus{AllMB: du.All, UsedMB: du.Used, FreeMB: du.Free} - } else if duPtr, ok := p2pStats["disk-info"].(*utils.DiskStatus); ok && duPtr != nil { - metrics.Disk = DiskStatus{AllMB: duPtr.All, UsedMB: duPtr.Used, FreeMB: duPtr.Free} - } - - // Ban list - if bans, ok := p2pStats["ban-list"].([]kademlia.BanSnapshot); ok { - for _, b := range bans { - metrics.BanList = append(metrics.BanList, BanEntry{ - ID: b.ID, - IP: b.IP, - Port: uint32(b.Port), - Count: int32(b.Count), - CreatedAtUnix: b.CreatedAt.Unix(), - AgeSeconds: int64(b.Age.Seconds()), - }) - } - } - - // Conn pool metrics - if pool, ok := p2pStats["conn-pool"].(map[string]int64); ok { - for k, v := range pool { - metrics.ConnPoolMetrics[k] = v - } - } - - // DHT metrics and database/network counters live inside dht map - if dhtStats, ok := p2pStats["dht"].(map[string]interface{}); ok { - // Database - if db, ok := dhtStats["database"].(map[string]interface{}); ok { - var sizeMB float64 - if v, ok := db["p2p_db_size"].(float64); ok { - sizeMB = v - } - var recs int64 - switch v := db["p2p_db_records_count"].(type) { - case int: - recs = int64(v) - case int64: - recs = v - case float64: - recs = int64(v) - } - metrics.Database = DatabaseStats{P2PDBSizeMB: sizeMB, P2PDBRecordsCount: recs} - } - - // Network handle metrics - if nhm, ok := dhtStats["network"].(map[string]kademlia.HandleCounters); ok { - for k, c := range nhm { - metrics.NetworkHandleMetrics[k] = HandleCounters{Total: c.Total, Success: c.Success, Failure: c.Failure, Timeout: c.Timeout} - } - } else if nhmI, ok := dhtStats["network"].(map[string]interface{}); ok { - for k, vi := range nhmI { - if c, ok := vi.(kademlia.HandleCounters); ok { - metrics.NetworkHandleMetrics[k] = HandleCounters{Total: c.Total, Success: c.Success, Failure: c.Failure, Timeout: c.Timeout} - } - } - } - } - - // DHT rolling metrics snapshot is attached at top-level under dht_metrics - if snap, ok := p2pStats["dht_metrics"].(kademlia.DHTMetricsSnapshot); ok { - // Store success - for _, p := range snap.StoreSuccessRecent { - metrics.DhtMetrics.StoreSuccessRecent = append(metrics.DhtMetrics.StoreSuccessRecent, StoreSuccessPoint{ - TimeUnix: p.Time.Unix(), - Requests: int32(p.Requests), - Successful: int32(p.Successful), - SuccessRate: p.SuccessRate, - }) - } - // Batch retrieve - for _, p := range snap.BatchRetrieveRecent { - metrics.DhtMetrics.BatchRetrieveRecent = append(metrics.DhtMetrics.BatchRetrieveRecent, BatchRetrievePoint{ - TimeUnix: p.Time.Unix(), - Keys: int32(p.Keys), - Required: int32(p.Required), - FoundLocal: int32(p.FoundLocal), - FoundNetwork: int32(p.FoundNet), - DurationMS: p.Duration.Milliseconds(), - }) - } - metrics.DhtMetrics.HotPathBannedSkips = snap.HotPathBannedSkips - metrics.DhtMetrics.HotPathBanIncrements = snap.HotPathBanIncrements - } - } - } - - // Always include metrics (may be empty if not available) - resp.P2PMetrics = metrics + // Prepare P2P metrics container (always present in response) + metrics := P2PMetrics{ + NetworkHandleMetrics: map[string]HandleCounters{}, + ConnPoolMetrics: map[string]int64{}, + BanList: []BanEntry{}, + } + + // Collect P2P network information and metrics (fill when available and requested) + if includeP2PMetrics && s.p2pService != nil { + p2pStats, err := s.p2pService.Stats(ctx) + if err != nil { + // Log error but continue - non-critical + logtrace.Error(ctx, "failed to get p2p stats", logtrace.Fields{logtrace.FieldError: err.Error()}) + } else { + if dhtStats, ok := p2pStats["dht"].(map[string]interface{}); ok { + if peersCount, ok := dhtStats["peers_count"].(int); ok { + resp.Network.PeersCount = int32(peersCount) + } + + // Extract peer addresses + if peers, ok := dhtStats["peers"].([]*kademlia.Node); ok { + resp.Network.PeerAddresses = make([]string, 0, len(peers)) + for _, peer := range peers { + // Format peer address as "ID@IP:Port" + peerAddr := fmt.Sprintf("%s@%s:%d", string(peer.ID), peer.IP, peer.Port) + resp.Network.PeerAddresses = append(resp.Network.PeerAddresses, peerAddr) + } + } else { + resp.Network.PeerAddresses = []string{} + } + } + + // Disk info + if du, ok := p2pStats["disk-info"].(utils.DiskStatus); ok { + metrics.Disk = DiskStatus{AllMB: du.All, UsedMB: du.Used, FreeMB: du.Free} + } else if duPtr, ok := p2pStats["disk-info"].(*utils.DiskStatus); ok && duPtr != nil { + metrics.Disk = DiskStatus{AllMB: duPtr.All, UsedMB: duPtr.Used, FreeMB: duPtr.Free} + } + + // Ban list + if bans, ok := p2pStats["ban-list"].([]kademlia.BanSnapshot); ok { + for _, b := range bans { + metrics.BanList = append(metrics.BanList, BanEntry{ + ID: b.ID, + IP: b.IP, + Port: uint32(b.Port), + Count: int32(b.Count), + CreatedAtUnix: b.CreatedAt.Unix(), + AgeSeconds: int64(b.Age.Seconds()), + }) + } + } + + // Conn pool metrics + if pool, ok := p2pStats["conn-pool"].(map[string]int64); ok { + for k, v := range pool { + metrics.ConnPoolMetrics[k] = v + } + } + + // DHT metrics and database/network counters live inside dht map + if dhtStats, ok := p2pStats["dht"].(map[string]interface{}); ok { + // Database + if db, ok := dhtStats["database"].(map[string]interface{}); ok { + var sizeMB float64 + if v, ok := db["p2p_db_size"].(float64); ok { + sizeMB = v + } + var recs int64 + switch v := db["p2p_db_records_count"].(type) { + case int: + recs = int64(v) + case int64: + recs = v + case float64: + recs = int64(v) + } + metrics.Database = DatabaseStats{P2PDBSizeMB: sizeMB, P2PDBRecordsCount: recs} + } + + // Network handle metrics + if nhm, ok := dhtStats["network"].(map[string]kademlia.HandleCounters); ok { + for k, c := range nhm { + metrics.NetworkHandleMetrics[k] = HandleCounters{Total: c.Total, Success: c.Success, Failure: c.Failure, Timeout: c.Timeout} + } + } else if nhmI, ok := dhtStats["network"].(map[string]interface{}); ok { + for k, vi := range nhmI { + if c, ok := vi.(kademlia.HandleCounters); ok { + metrics.NetworkHandleMetrics[k] = HandleCounters{Total: c.Total, Success: c.Success, Failure: c.Failure, Timeout: c.Timeout} + } + } + } + + // Recent batch store/retrieve (overall lists) + if rbs, ok := dhtStats["recent_batch_store_overall"].([]kademlia.RecentBatchStoreEntry); ok { + for _, e := range rbs { + metrics.RecentBatchStore = append(metrics.RecentBatchStore, RecentBatchStoreEntry{ + TimeUnix: e.TimeUnix, + SenderID: e.SenderID, + SenderIP: e.SenderIP, + Keys: e.Keys, + DurationMS: e.DurationMS, + OK: e.OK, + Error: e.Error, + }) + } + } else if anyList, ok := dhtStats["recent_batch_store_overall"].([]interface{}); ok { + for _, vi := range anyList { + if e, ok := vi.(kademlia.RecentBatchStoreEntry); ok { + metrics.RecentBatchStore = append(metrics.RecentBatchStore, RecentBatchStoreEntry{ + TimeUnix: e.TimeUnix, + SenderID: e.SenderID, + SenderIP: e.SenderIP, + Keys: e.Keys, + DurationMS: e.DurationMS, + OK: e.OK, + Error: e.Error, + }) + } + } + } + if rbr, ok := dhtStats["recent_batch_retrieve_overall"].([]kademlia.RecentBatchRetrieveEntry); ok { + for _, e := range rbr { + metrics.RecentBatchRetrieve = append(metrics.RecentBatchRetrieve, RecentBatchRetrieveEntry{ + TimeUnix: e.TimeUnix, + SenderID: e.SenderID, + SenderIP: e.SenderIP, + Requested: e.Requested, + Found: e.Found, + DurationMS: e.DurationMS, + Error: e.Error, + }) + } + } else if anyList, ok := dhtStats["recent_batch_retrieve_overall"].([]interface{}); ok { + for _, vi := range anyList { + if e, ok := vi.(kademlia.RecentBatchRetrieveEntry); ok { + metrics.RecentBatchRetrieve = append(metrics.RecentBatchRetrieve, RecentBatchRetrieveEntry{ + TimeUnix: e.TimeUnix, + SenderID: e.SenderID, + SenderIP: e.SenderIP, + Requested: e.Requested, + Found: e.Found, + DurationMS: e.DurationMS, + Error: e.Error, + }) + } + } + } + + // Per-IP buckets + if byip, ok := dhtStats["recent_batch_store_by_ip"].(map[string][]kademlia.RecentBatchStoreEntry); ok { + for ip, list := range byip { + bucket := make([]RecentBatchStoreEntry, 0, len(list)) + for _, e := range list { + bucket = append(bucket, RecentBatchStoreEntry{ + TimeUnix: e.TimeUnix, + SenderID: e.SenderID, + SenderIP: e.SenderIP, + Keys: e.Keys, + DurationMS: e.DurationMS, + OK: e.OK, + Error: e.Error, + }) + } + // initialize map if needed + if metrics.RecentBatchStoreByIP == nil { + metrics.RecentBatchStoreByIP = map[string][]RecentBatchStoreEntry{} + } + metrics.RecentBatchStoreByIP[ip] = bucket + } + } + if byip, ok := dhtStats["recent_batch_retrieve_by_ip"].(map[string][]kademlia.RecentBatchRetrieveEntry); ok { + for ip, list := range byip { + bucket := make([]RecentBatchRetrieveEntry, 0, len(list)) + for _, e := range list { + bucket = append(bucket, RecentBatchRetrieveEntry{ + TimeUnix: e.TimeUnix, + SenderID: e.SenderID, + SenderIP: e.SenderIP, + Requested: e.Requested, + Found: e.Found, + DurationMS: e.DurationMS, + Error: e.Error, + }) + } + if metrics.RecentBatchRetrieveByIP == nil { + metrics.RecentBatchRetrieveByIP = map[string][]RecentBatchRetrieveEntry{} + } + metrics.RecentBatchRetrieveByIP[ip] = bucket + } + } + } + + // DHT rolling metrics snapshot is attached at top-level under dht_metrics + if snap, ok := p2pStats["dht_metrics"].(kademlia.DHTMetricsSnapshot); ok { + // Store success + for _, p := range snap.StoreSuccessRecent { + metrics.DhtMetrics.StoreSuccessRecent = append(metrics.DhtMetrics.StoreSuccessRecent, StoreSuccessPoint{ + TimeUnix: p.Time.Unix(), + Requests: int32(p.Requests), + Successful: int32(p.Successful), + SuccessRate: p.SuccessRate, + }) + } + // Batch retrieve + for _, p := range snap.BatchRetrieveRecent { + metrics.DhtMetrics.BatchRetrieveRecent = append(metrics.DhtMetrics.BatchRetrieveRecent, BatchRetrievePoint{ + TimeUnix: p.Time.Unix(), + Keys: int32(p.Keys), + Required: int32(p.Required), + FoundLocal: int32(p.FoundLocal), + FoundNetwork: int32(p.FoundNet), + DurationMS: p.Duration.Milliseconds(), + }) + } + metrics.DhtMetrics.HotPathBannedSkips = snap.HotPathBannedSkips + metrics.DhtMetrics.HotPathBanIncrements = snap.HotPathBanIncrements + } + } + } + + // Always include metrics (may be empty if not available) + resp.P2PMetrics = metrics // Calculate rank from top supernodes if s.lumeraClient != nil && s.config != nil { diff --git a/supernode/services/common/supernode/types.go b/supernode/services/common/supernode/types.go index 032aa0ee..9a6f0953 100644 --- a/supernode/services/common/supernode/types.go +++ b/supernode/services/common/supernode/types.go @@ -3,23 +3,23 @@ package supernode // StatusResponse represents the complete system status information // with clear organization of resources and services type StatusResponse struct { - Version string // Supernode version - UptimeSeconds uint64 // Uptime in seconds - Resources Resources // System resource information - RunningTasks []ServiceTasks // Services with currently running tasks - RegisteredServices []string // All registered/available services - Network NetworkInfo // P2P network information - Rank int32 // Rank in the top supernodes list (0 if not in top list) - IPAddress string // Supernode IP address with port (e.g., "192.168.1.1:4445") - P2PMetrics P2PMetrics // Detailed P2P metrics snapshot + Version string // Supernode version + UptimeSeconds uint64 // Uptime in seconds + Resources Resources // System resource information + RunningTasks []ServiceTasks // Services with currently running tasks + RegisteredServices []string // All registered/available services + Network NetworkInfo // P2P network information + Rank int32 // Rank in the top supernodes list (0 if not in top list) + IPAddress string // Supernode IP address with port (e.g., "192.168.1.1:4445") + P2PMetrics P2PMetrics // Detailed P2P metrics snapshot } // Resources contains system resource metrics type Resources struct { - CPU CPUInfo // CPU usage information - Memory MemoryInfo // Memory usage information - Storage []StorageInfo // Storage volumes information - HardwareSummary string // Formatted hardware summary (e.g., "8 cores / 32GB RAM") + CPU CPUInfo // CPU usage information + Memory MemoryInfo // Memory usage information + Storage []StorageInfo // Storage volumes information + HardwareSummary string // Formatted hardware summary (e.g., "8 cores / 32GB RAM") } // CPUInfo contains CPU usage metrics @@ -54,68 +54,92 @@ type ServiceTasks struct { // NetworkInfo contains P2P network information type NetworkInfo struct { - PeersCount int32 // Number of connected peers in P2P network - PeerAddresses []string // List of connected peer addresses (optional, may be empty for privacy) + PeersCount int32 // Number of connected peers in P2P network + PeerAddresses []string // List of connected peer addresses (optional, may be empty for privacy) } // P2PMetrics mirrors the proto P2P metrics for status API type P2PMetrics struct { - DhtMetrics DhtMetrics - NetworkHandleMetrics map[string]HandleCounters - ConnPoolMetrics map[string]int64 - BanList []BanEntry - Database DatabaseStats - Disk DiskStatus + DhtMetrics DhtMetrics + NetworkHandleMetrics map[string]HandleCounters + ConnPoolMetrics map[string]int64 + BanList []BanEntry + Database DatabaseStats + Disk DiskStatus + RecentBatchStore []RecentBatchStoreEntry + RecentBatchRetrieve []RecentBatchRetrieveEntry + RecentBatchStoreByIP map[string][]RecentBatchStoreEntry + RecentBatchRetrieveByIP map[string][]RecentBatchRetrieveEntry } type StoreSuccessPoint struct { - TimeUnix int64 - Requests int32 - Successful int32 - SuccessRate float64 + TimeUnix int64 + Requests int32 + Successful int32 + SuccessRate float64 } type BatchRetrievePoint struct { - TimeUnix int64 - Keys int32 - Required int32 - FoundLocal int32 - FoundNetwork int32 - DurationMS int64 + TimeUnix int64 + Keys int32 + Required int32 + FoundLocal int32 + FoundNetwork int32 + DurationMS int64 } type DhtMetrics struct { - StoreSuccessRecent []StoreSuccessPoint - BatchRetrieveRecent []BatchRetrievePoint - HotPathBannedSkips int64 - HotPathBanIncrements int64 + StoreSuccessRecent []StoreSuccessPoint + BatchRetrieveRecent []BatchRetrievePoint + HotPathBannedSkips int64 + HotPathBanIncrements int64 } type HandleCounters struct { - Total int64 - Success int64 - Failure int64 - Timeout int64 + Total int64 + Success int64 + Failure int64 + Timeout int64 } type BanEntry struct { - ID string - IP string - Port uint32 - Count int32 - CreatedAtUnix int64 - AgeSeconds int64 + ID string + IP string + Port uint32 + Count int32 + CreatedAtUnix int64 + AgeSeconds int64 } type DatabaseStats struct { - P2PDBSizeMB float64 - P2PDBRecordsCount int64 + P2PDBSizeMB float64 + P2PDBRecordsCount int64 } type DiskStatus struct { - AllMB float64 - UsedMB float64 - FreeMB float64 + AllMB float64 + UsedMB float64 + FreeMB float64 +} + +type RecentBatchStoreEntry struct { + TimeUnix int64 + SenderID string + SenderIP string + Keys int + DurationMS int64 + OK bool + Error string +} + +type RecentBatchRetrieveEntry struct { + TimeUnix int64 + SenderID string + SenderIP string + Requested int + Found int + DurationMS int64 + Error string } // TaskProvider interface defines the contract for services to provide diff --git a/tests/integration/p2p/p2p_integration_test.go b/tests/integration/p2p/p2p_integration_test.go index a689b75d..bce71f58 100644 --- a/tests/integration/p2p/p2p_integration_test.go +++ b/tests/integration/p2p/p2p_integration_test.go @@ -108,7 +108,7 @@ func TestP2PBasicIntegration(t *testing.T) { // Add debug logging log.Printf("Storing batch with keys: %v", expectedKeys) - _, _, err := services[0].StoreBatch(ctx, batchData, 0, taskID) + err := services[0].StoreBatch(ctx, batchData, 0, taskID) require.NoError(t, err) // Add immediate verification