diff --git a/.github/actions/setup-env/action.yml b/.github/actions/setup-env/action.yml index 4d9cfcf5..1b44339d 100644 --- a/.github/actions/setup-env/action.yml +++ b/.github/actions/setup-env/action.yml @@ -38,12 +38,3 @@ runs: shell: bash run: | go mod download - - - name: Tidy Go modules - shell: bash - run: | - # Ensure go.mod/go.sum are up to date to prevent CI build failures - go mod tidy - if [ -f sn-manager/go.mod ]; then - (cd sn-manager && go mod tidy) - fi diff --git a/.gitignore b/.gitignore index 341114a4..685adc58 100644 --- a/.gitignore +++ b/.gitignore @@ -28,3 +28,4 @@ go.work.sum /release /tests/system/data tests/system/**/supernode-data* +AGENTS.md \ No newline at end of file diff --git a/README.md b/README.md index 8edccf03..cca276d0 100644 --- a/README.md +++ b/README.md @@ -207,6 +207,18 @@ enum SupernodeEventType { See docs/gateway.md for the full gateway guide (endpoints, examples, Swagger links). +### Codec Configuration (fixed policy) + +The supernode uses a fixed RaptorQ codec policy (linux/amd64 only): +- Concurrency: 4 +- Symbol size: 65535 +- Redundancy: 5 +- Max memory: detected cgroup/system memory minus 10% headroom + +Status includes these effective values under `codec` in `StatusResponse`. +The HTTP gateway also exposes a minimal view at `GET /api/v1/codec` with: +- `symbol_size`, `redundancy`, `max_memory_mb`, `concurrency`, `headroom_pct`, `mem_limit_mb`, `mem_limit_source`. + ## CLI Commands ### Core Commands diff --git a/docs/cascade-performance.md b/docs/cascade-performance.md new file mode 100644 index 00000000..1cecf566 --- /dev/null +++ b/docs/cascade-performance.md @@ -0,0 +1,191 @@ +# Cascade Downloads & Performance: Concepts, Limits, and Tuning + +This document explains how Cascade encoding/decoding works, the performance and memory factors involved, and practical configuration guidance. It consolidates the “blocks and symbols” primer and expands it with deeper operational tuning, error references, and code pointers — in a concise, professional format. + +## Overview + +- Cascade uses RaptorQ forward error correction to split a file into blocks and symbols that can be stored/fetched from a P2P network. +- Decoding requires enough symbols to reconstruct each block; integrity is verified with hashes recorded in the layout. +- Performance and reliability are driven by four main levers: block size, redundancy, concurrency, and memory headroom. Batching and ordering in the store path, and supernode selection in the download path, also matter. + +## Current Defaults (Implementation) + +- RaptorQ (codec) + - Block cap: 256 MB (encode‑time upper bound per block) + - Decode concurrency: 1 + - Memory headroom: 20% of detected RAM + - Symbol size: ~65,535 bytes + - Redundancy: 5 + +- Store path (foreground adaptor) + - Batch size: 2,500 files per batch (≈156 MiB typical at default symbol size) + - Downsampling: if total files > 2,500, take 10% sorted prefix for initial store + - Per‑batch P2P store timeout: 5 minutes + +- Store path (background worker) + - Batch size: 1,000 files per batch (≈62.5 MiB typical) + +- Download path + - SDK per‑supernode download deadline: 10 minutes + - Supernode ranking: status probe ~2 seconds per node; sorted by available memory (desc) + - P2P exec timeouts (per RPC): + - FindValue: 5s + - BatchFindValues: 60s + - BatchGetValues: 75s + - StoreData: 10s + - BatchStoreData: 75s + - Replicate: 90s + +- Upload constraints + - Max file size: 1 GB (enforced in SDK and server) + - Adaptive upload chunk size: ~64 KB → 4 MB based on file size + +## Core Concepts + +- Block: A contiguous segment of the original file. Think of it as a “chapter”. +- Symbol: A small piece produced by RaptorQ for a block. You only need “enough” symbols to reconstruct the block. +- Layout: Metadata that lists all blocks (block_id, size, original offset, per‑block hash) and the symbol IDs belonging to each block. + +Encode (upload): +- Choose a block size; RaptorQ creates symbols per block; symbols + layout are stored. + +Decode (download): +- Fetch symbols from the network; reconstruct each block independently; write each block back at its original offset; verify hashes; stream the file. + +Key facts: +- Symbols never mix across blocks. +- Peak memory during decode scales roughly with the chosen block size (plus overhead). + +## File Size Limits & Upload Chunking + +- Maximum file size: 1 GB (enforced both in SDK and server handlers). +- Adaptive upload chunk size: ~64 KB → 4 MB depending on total file size for throughput vs memory stability. + +## Encoding/Decoding Workflow (high level) + +1) SDK uploads file to a supernode (gRPC stream). Server writes to a temporary file, validates size and integrity. +2) Server encodes with RaptorQ: produces a symbols directory and a layout JSON. +3) Server stores artefacts: layout/ID files and symbols into P2P in batches. +4) Later, SDK requests download; supernode fetches symbols progressively and decodes to reconstruct the file; integrity is verified. + +## Contexts & Timeouts (download path) + +- SDK: wraps the download RPC with a 10‑minute deadline. +- Server: uses that context; P2P layer applies per‑RPC timeouts (e.g., 5s for single key FindValue, ~75s for BatchGetValues), with internal early cancellation once enough symbols are found. +- RaptorQ: uses the same context for logging; no additional deadline inside decode. + +## Memory Model + +- Decoder memory is primarily a function of block size and concurrency. +- Headroom percentage reduces the usable memory budget to leave safety buffer for the OS and other processes. +- Example formula: usable_memory ≈ TotalRAM × (1 − headroom%). + +## Configuration Levers + +The implementation uses simple fixed constants for safety and predictability. You can adjust them and rebuild. + +1) Block Size Cap (`targetBlockMB`, encode‑time) +- What: Upper bound on block size. Actual used size = min(recommended_by_codec, cap). +- Effect: Smaller cap lowers peak decode memory (more blocks, more symbols/keys). Larger cap reduces block count (faster on big machines) but raises peak memory. +- Current default: 256 MB (good balance on well-provisioned machines). Only affects newly encoded artefacts. + +2) Redundancy (`defaultRedundancy`, encode‑time) +- What: Extra protection (more symbols) to tolerate missing data. +- Effect: Higher redundancy improves recoverability but costs more storage and network I/O. Does not materially change peak memory. +- Current default: 5 (good real‑world trade‑off). + +3) Concurrency (`fixedConcurrency`, decode‑time) +- What: Number of RaptorQ decode workers. +- Effect: Higher is faster but multiplies memory; lower is safer and predictable. +- Current default: 1 (safe default for wide environments). + +4) Headroom (`headroomPct`, decode‑time) +- What: Percentage of detected RAM left unused by the RaptorQ processor. +- Effect: More headroom = safer under load; less headroom = more memory available to decode. +- Current default: 20% (conservative and robust for shared hosts). + +## Batching Strategy (store path) + +Why batching matters: +- Store batches are loaded wholly into memory before sending to P2P. +- A fixed “files‑per‑batch” limit gives variable memory usage because symbol files can differ slightly in size. + +Current defaults: +- Foreground adaptor: `loadSymbolsBatchSize = 2500` → ≈ 2,500 × 65,535 B ≈ 156 MiB per batch (typical). +- Background worker: `loadSymbolsBatchSize = 1000` → ≈ 62.5 MiB per batch. + +Byte‑budget alternative (conceptual, not implemented): +- Cap the total bytes per batch (e.g., 128–256 MiB), with a secondary cap on file count. +- Benefits: predictable peak memory; better throughput on small symbols; avoids spikes on larger ones. + +## Ordering for Throughput (store path) + +- We sort relative file paths before batching (e.g., `block_0/...`, `block_1/...`) to improve filesystem locality and reduce disk seeks. This favors speed. +- Trade‑off: If a process stops mid‑way, earlier blocks (lexicographically smaller) are more likely stored than later ones. For fairness across blocks at partial completion, interleaving could be used at some CPU cost. + +## Supernode Selection (download path) + +- The SDK ranks supernodes by available memory (fast 2s status probe per node) and attempts downloads in that order. +- This increases the chances of successful decode for large files. + +## Defaults & Suggested Settings + +1 GB files (general) +- Block cap: 256 MB (≈4 blocks) +- Concurrency: 1 +- Headroom: 20% +- Redundancy: 5 + +Large‑memory machines (performance‑leaning) +- Block cap: 256 MB (or 512 MB) to reduce block count and increase throughput. +- Concurrency: 1–2. +- Headroom: 15–20% depending on other workloads. +- Redundancy: 5 (or 6 in sparse networks). + +Small‑memory machines +- Block cap: 64–128 MB +- Concurrency: 1 +- Headroom: 20% +- Redundancy: 5 + +## Error Reference + +- memory limit exceeded + - The decoder exceeded its memory budget. Reduce block size or concurrency, increase RAM, or lower headroom. + +- hash mismatch for block X + - Data reconstructed for the block did not match the expected hash. Often indicates wrong/corrupt symbols; can also occur when decoding fails mid‑way under memory pressure. Re‑fetching or re‑encoding may be required. + +- insufficient symbols + - Not enough valid symbols were available; the retriever will fetch more. + +- gRPC Internal on download stream + - The supernode returned an error during decode (e.g., memory failure). The SDK will try the next supernode. + +## Code Pointers + +- Block cap, headroom, concurrency (RaptorQ): `pkg/codec/raptorq.go` +- Store batching (foreground path): `supernode/services/cascade/adaptors/p2p.go` +- Store batching (background worker): `p2p/kademlia/rq_symbols.go` +- Batch symbol loading / deletion: `pkg/utils/utils.go` (LoadSymbols, DeleteSymbols) +- Supernode ranking by memory (download): `sdk/task/download.go` +- File size cap & adaptive upload chunking: SDK and server sides (`sdk/adapters/supernodeservice/adapter.go`, `supernode/node/action/server/cascade/cascade_action_server.go`) + +## Notes & Scope + +- Changing block size only affects new encodes; existing artefacts keep their original layout. +- Tuning should reflect your fleet: prefer safety defaults for heterogeneous environments; be aggressive only on known large‑RAM hosts. + +## FAQ + +- Why might a smaller file decode but a larger file fail? + - Peak memory grows with data size and chosen block size. A smaller file may fit within the decoder’s memory budget on a given machine, while a larger one may exceed it. Smaller blocks and/or more RAM resolve this. + +- Does changing block size affect old files? + - No. It only affects newly encoded content. Existing artefacts retain their original layout. + +- Will smaller blocks slow things down? + - Slightly, due to more pieces and network lookups. For constrained machines, the reliability gain outweighs the small performance cost. + +- What’s the best block size? + - There’s no single best value. 128 MB is a solid default. Use 64 MB for smaller machines and 256–512 MB for large servers when maximizing throughput. diff --git a/docs/gateway.md b/docs/gateway.md index 638e2794..19c43447 100644 --- a/docs/gateway.md +++ b/docs/gateway.md @@ -1,40 +1,45 @@ # Supernode HTTP Gateway -The Supernode exposes its gRPC services via an HTTP/JSON gateway on port `8002`. +The HTTP gateway exposes the gRPC services via REST on port `8002` using grpc-gateway. -- Swagger UI: http://localhost:8002/swagger-ui/ -- OpenAPI Spec: http://localhost:8002/swagger.json +## Endpoints -## Status API +### GET `/api/v1/status` +Returns supernode status: system resources (CPU, memory, storage), service info, and optionally P2P metrics. -GET `/api/v1/status` +- Query `include_p2p_metrics=true` enables detailed P2P metrics and peer info. +- When omitted or false, peer count, peer addresses, and `p2p_metrics` are not included. -Returns the current supernode status including system resources (CPU, memory, storage), running tasks, registered services, network info, and codec configuration. +Examples: -- Query `include_p2p_metrics=true` adds detailed P2P metrics and peer information. - -Example: ```bash +# Lightweight status curl "http://localhost:8002/api/v1/status" -``` -With P2P metrics: -```bash +# Include P2P metrics and peer info curl "http://localhost:8002/api/v1/status?include_p2p_metrics=true" ``` -## Services API - -GET `/api/v1/services` - -Returns the list of available services and methods exposed by this supernode. - -Example: -```bash -curl http://localhost:8002/api/v1/services +Example responses are shown in the main README under the SupernodeService section. + +### GET `/api/v1/codec` +Returns the minimal effective RaptorQ codec configuration used by the node (fixed policy): + +```json +{ + "symbol_size": 65535, + "redundancy": 5, + "max_memory_mb": 12288, + "concurrency": 4, + "headroom_pct": 10, + "mem_limit_mb": 13653, + "mem_limit_source": "cgroupv2:memory.max" +} ``` -## Notes +## API Documentation + +- Swagger UI: `http://localhost:8002/swagger-ui/` +- OpenAPI Spec: `http://localhost:8002/swagger.json` -- The gateway translates between HTTP/JSON and gRPC/protobuf, enabling easy integration with web tooling and monitoring. -- Interactive exploration is available via Swagger UI. +The Swagger UI provides an interactive interface to explore and test all available API endpoints. diff --git a/gen/supernode/action/cascade/service.pb.go b/gen/supernode/action/cascade/service.pb.go index ce3c177b..dd083d04 100644 --- a/gen/supernode/action/cascade/service.pb.go +++ b/gen/supernode/action/cascade/service.pb.go @@ -7,11 +7,10 @@ package cascade import ( - reflect "reflect" - sync "sync" - protoreflect "google.golang.org/protobuf/reflect/protoreflect" protoimpl "google.golang.org/protobuf/runtime/protoimpl" + reflect "reflect" + sync "sync" ) const ( @@ -34,12 +33,15 @@ const ( SupernodeEventType_SIGNATURE_VERIFIED SupernodeEventType = 7 SupernodeEventType_RQID_GENERATED SupernodeEventType = 8 SupernodeEventType_RQID_VERIFIED SupernodeEventType = 9 - // Emitted by supernode when finalize transaction simulation passes - SupernodeEventType_FINALIZE_SIMULATED SupernodeEventType = 10 - SupernodeEventType_ARTEFACTS_STORED SupernodeEventType = 11 - SupernodeEventType_ACTION_FINALIZED SupernodeEventType = 12 - SupernodeEventType_ARTEFACTS_DOWNLOADED SupernodeEventType = 13 - SupernodeEventType_FINALIZE_SIMULATION_FAILED SupernodeEventType = 14 + SupernodeEventType_FINALIZE_SIMULATED SupernodeEventType = 10 + SupernodeEventType_ARTEFACTS_STORED SupernodeEventType = 11 + SupernodeEventType_ACTION_FINALIZED SupernodeEventType = 12 + SupernodeEventType_ARTEFACTS_DOWNLOADED SupernodeEventType = 13 + SupernodeEventType_FINALIZE_SIMULATION_FAILED SupernodeEventType = 14 + // Download phase markers (additive, backward compatible) + SupernodeEventType_NETWORK_RETRIEVE_STARTED SupernodeEventType = 15 // Supernode started pulling symbols from network + SupernodeEventType_DECODE_COMPLETED SupernodeEventType = 16 // File reconstructed and verified; ready on disk + SupernodeEventType_SERVE_READY SupernodeEventType = 17 // File is ready to be streamed to client ) // Enum value maps for SupernodeEventType. @@ -58,8 +60,11 @@ var ( 10: "FINALIZE_SIMULATED", 11: "ARTEFACTS_STORED", 12: "ACTION_FINALIZED", - 13: "ARTEFACTS_DOWNLOADED", - 14: "FINALIZE_SIMULATION_FAILED", + 13: "ARTEFACTS_DOWNLOADED", + 14: "FINALIZE_SIMULATION_FAILED", + 15: "NETWORK_RETRIEVE_STARTED", + 16: "DECODE_COMPLETED", + 17: "SERVE_READY", } SupernodeEventType_value = map[string]int32{ "UNKNOWN": 0, @@ -75,8 +80,11 @@ var ( "FINALIZE_SIMULATED": 10, "ARTEFACTS_STORED": 11, "ACTION_FINALIZED": 12, - "ARTEFACTS_DOWNLOADED": 13, - "FINALIZE_SIMULATION_FAILED": 14, + "ARTEFACTS_DOWNLOADED": 13, + "FINALIZE_SIMULATION_FAILED": 14, + "NETWORK_RETRIEVE_STARTED": 15, + "DECODE_COMPLETED": 16, + "SERVE_READY": 17, } ) @@ -577,8 +585,8 @@ var file_supernode_action_cascade_service_proto_rawDesc = []byte{ 0x61, 0x73, 0x63, 0x61, 0x64, 0x65, 0x2e, 0x53, 0x75, 0x70, 0x65, 0x72, 0x6e, 0x6f, 0x64, 0x65, 0x45, 0x76, 0x65, 0x6e, 0x74, 0x54, 0x79, 0x70, 0x65, 0x52, 0x09, 0x65, 0x76, 0x65, 0x6e, 0x74, 0x54, 0x79, 0x70, 0x65, 0x12, 0x18, 0x0a, 0x07, 0x6d, 0x65, 0x73, 0x73, 0x61, 0x67, 0x65, 0x18, - 0x02, 0x20, 0x01, 0x28, 0x09, 0x52, 0x07, 0x6d, 0x65, 0x73, 0x73, 0x61, 0x67, 0x65, 0x2a, 0xce, - 0x02, 0x0a, 0x12, 0x53, 0x75, 0x70, 0x65, 0x72, 0x6e, 0x6f, 0x64, 0x65, 0x45, 0x76, 0x65, 0x6e, + 0x02, 0x20, 0x01, 0x28, 0x09, 0x52, 0x07, 0x6d, 0x65, 0x73, 0x73, 0x61, 0x67, 0x65, 0x2a, 0xb3, + 0x03, 0x0a, 0x12, 0x53, 0x75, 0x70, 0x65, 0x72, 0x6e, 0x6f, 0x64, 0x65, 0x45, 0x76, 0x65, 0x6e, 0x74, 0x54, 0x79, 0x70, 0x65, 0x12, 0x0b, 0x0a, 0x07, 0x55, 0x4e, 0x4b, 0x4e, 0x4f, 0x57, 0x4e, 0x10, 0x00, 0x12, 0x14, 0x0a, 0x10, 0x41, 0x43, 0x54, 0x49, 0x4f, 0x4e, 0x5f, 0x52, 0x45, 0x54, 0x52, 0x49, 0x45, 0x56, 0x45, 0x44, 0x10, 0x01, 0x12, 0x17, 0x0a, 0x13, 0x41, 0x43, 0x54, 0x49, @@ -598,22 +606,28 @@ var file_supernode_action_cascade_service_proto_rawDesc = []byte{ 0x45, 0x46, 0x41, 0x43, 0x54, 0x53, 0x5f, 0x53, 0x54, 0x4f, 0x52, 0x45, 0x44, 0x10, 0x0b, 0x12, 0x14, 0x0a, 0x10, 0x41, 0x43, 0x54, 0x49, 0x4f, 0x4e, 0x5f, 0x46, 0x49, 0x4e, 0x41, 0x4c, 0x49, 0x5a, 0x45, 0x44, 0x10, 0x0c, 0x12, 0x18, 0x0a, 0x14, 0x41, 0x52, 0x54, 0x45, 0x46, 0x41, 0x43, - 0x54, 0x53, 0x5f, 0x44, 0x4f, 0x57, 0x4e, 0x4c, 0x4f, 0x41, 0x44, 0x45, 0x44, 0x10, 0x0d, 0x32, - 0x98, 0x01, 0x0a, 0x0e, 0x43, 0x61, 0x73, 0x63, 0x61, 0x64, 0x65, 0x53, 0x65, 0x72, 0x76, 0x69, - 0x63, 0x65, 0x12, 0x43, 0x0a, 0x08, 0x52, 0x65, 0x67, 0x69, 0x73, 0x74, 0x65, 0x72, 0x12, 0x18, - 0x2e, 0x63, 0x61, 0x73, 0x63, 0x61, 0x64, 0x65, 0x2e, 0x52, 0x65, 0x67, 0x69, 0x73, 0x74, 0x65, - 0x72, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x1a, 0x19, 0x2e, 0x63, 0x61, 0x73, 0x63, 0x61, - 0x64, 0x65, 0x2e, 0x52, 0x65, 0x67, 0x69, 0x73, 0x74, 0x65, 0x72, 0x52, 0x65, 0x73, 0x70, 0x6f, - 0x6e, 0x73, 0x65, 0x28, 0x01, 0x30, 0x01, 0x12, 0x41, 0x0a, 0x08, 0x44, 0x6f, 0x77, 0x6e, 0x6c, - 0x6f, 0x61, 0x64, 0x12, 0x18, 0x2e, 0x63, 0x61, 0x73, 0x63, 0x61, 0x64, 0x65, 0x2e, 0x44, 0x6f, - 0x77, 0x6e, 0x6c, 0x6f, 0x61, 0x64, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x1a, 0x19, 0x2e, - 0x63, 0x61, 0x73, 0x63, 0x61, 0x64, 0x65, 0x2e, 0x44, 0x6f, 0x77, 0x6e, 0x6c, 0x6f, 0x61, 0x64, - 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x30, 0x01, 0x42, 0x45, 0x5a, 0x43, 0x67, 0x69, - 0x74, 0x68, 0x75, 0x62, 0x2e, 0x63, 0x6f, 0x6d, 0x2f, 0x4c, 0x75, 0x6d, 0x65, 0x72, 0x61, 0x50, - 0x72, 0x6f, 0x74, 0x6f, 0x63, 0x6f, 0x6c, 0x2f, 0x73, 0x75, 0x70, 0x65, 0x72, 0x6e, 0x6f, 0x64, - 0x65, 0x2f, 0x76, 0x32, 0x2f, 0x67, 0x65, 0x6e, 0x2f, 0x73, 0x75, 0x70, 0x65, 0x72, 0x6e, 0x6f, - 0x64, 0x65, 0x2f, 0x61, 0x63, 0x74, 0x69, 0x6f, 0x6e, 0x2f, 0x63, 0x61, 0x73, 0x63, 0x61, 0x64, - 0x65, 0x62, 0x06, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x33, + 0x54, 0x53, 0x5f, 0x44, 0x4f, 0x57, 0x4e, 0x4c, 0x4f, 0x41, 0x44, 0x45, 0x44, 0x10, 0x0d, 0x12, + 0x1e, 0x0a, 0x1a, 0x46, 0x49, 0x4e, 0x41, 0x4c, 0x49, 0x5a, 0x45, 0x5f, 0x53, 0x49, 0x4d, 0x55, + 0x4c, 0x41, 0x54, 0x49, 0x4f, 0x4e, 0x5f, 0x46, 0x41, 0x49, 0x4c, 0x45, 0x44, 0x10, 0x0e, 0x12, + 0x1c, 0x0a, 0x18, 0x4e, 0x45, 0x54, 0x57, 0x4f, 0x52, 0x4b, 0x5f, 0x52, 0x45, 0x54, 0x52, 0x49, + 0x45, 0x56, 0x45, 0x5f, 0x53, 0x54, 0x41, 0x52, 0x54, 0x45, 0x44, 0x10, 0x0f, 0x12, 0x14, 0x0a, + 0x10, 0x44, 0x45, 0x43, 0x4f, 0x44, 0x45, 0x5f, 0x43, 0x4f, 0x4d, 0x50, 0x4c, 0x45, 0x54, 0x45, + 0x44, 0x10, 0x10, 0x12, 0x0f, 0x0a, 0x0b, 0x53, 0x45, 0x52, 0x56, 0x45, 0x5f, 0x52, 0x45, 0x41, + 0x44, 0x59, 0x10, 0x11, 0x32, 0x98, 0x01, 0x0a, 0x0e, 0x43, 0x61, 0x73, 0x63, 0x61, 0x64, 0x65, + 0x53, 0x65, 0x72, 0x76, 0x69, 0x63, 0x65, 0x12, 0x43, 0x0a, 0x08, 0x52, 0x65, 0x67, 0x69, 0x73, + 0x74, 0x65, 0x72, 0x12, 0x18, 0x2e, 0x63, 0x61, 0x73, 0x63, 0x61, 0x64, 0x65, 0x2e, 0x52, 0x65, + 0x67, 0x69, 0x73, 0x74, 0x65, 0x72, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x1a, 0x19, 0x2e, + 0x63, 0x61, 0x73, 0x63, 0x61, 0x64, 0x65, 0x2e, 0x52, 0x65, 0x67, 0x69, 0x73, 0x74, 0x65, 0x72, + 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x28, 0x01, 0x30, 0x01, 0x12, 0x41, 0x0a, 0x08, + 0x44, 0x6f, 0x77, 0x6e, 0x6c, 0x6f, 0x61, 0x64, 0x12, 0x18, 0x2e, 0x63, 0x61, 0x73, 0x63, 0x61, + 0x64, 0x65, 0x2e, 0x44, 0x6f, 0x77, 0x6e, 0x6c, 0x6f, 0x61, 0x64, 0x52, 0x65, 0x71, 0x75, 0x65, + 0x73, 0x74, 0x1a, 0x19, 0x2e, 0x63, 0x61, 0x73, 0x63, 0x61, 0x64, 0x65, 0x2e, 0x44, 0x6f, 0x77, + 0x6e, 0x6c, 0x6f, 0x61, 0x64, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x30, 0x01, 0x42, + 0x45, 0x5a, 0x43, 0x67, 0x69, 0x74, 0x68, 0x75, 0x62, 0x2e, 0x63, 0x6f, 0x6d, 0x2f, 0x4c, 0x75, + 0x6d, 0x65, 0x72, 0x61, 0x50, 0x72, 0x6f, 0x74, 0x6f, 0x63, 0x6f, 0x6c, 0x2f, 0x73, 0x75, 0x70, + 0x65, 0x72, 0x6e, 0x6f, 0x64, 0x65, 0x2f, 0x76, 0x32, 0x2f, 0x67, 0x65, 0x6e, 0x2f, 0x73, 0x75, + 0x70, 0x65, 0x72, 0x6e, 0x6f, 0x64, 0x65, 0x2f, 0x61, 0x63, 0x74, 0x69, 0x6f, 0x6e, 0x2f, 0x63, + 0x61, 0x73, 0x63, 0x61, 0x64, 0x65, 0x62, 0x06, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x33, } var ( diff --git a/gen/supernode/supernode.pb.go b/gen/supernode/supernode.pb.go index a14551a3..966573dd 100644 --- a/gen/supernode/supernode.pb.go +++ b/gen/supernode/supernode.pb.go @@ -606,16 +606,13 @@ type StatusResponse_CodecConfig struct { sizeCache protoimpl.SizeCache unknownFields protoimpl.UnknownFields - SymbolSize uint32 `protobuf:"varint,1,opt,name=symbol_size,json=symbolSize,proto3" json:"symbol_size,omitempty"` // bytes (typically 65535) - Redundancy uint32 `protobuf:"varint,2,opt,name=redundancy,proto3" json:"redundancy,omitempty"` // repair factor (percent-like scalar; 5 = default) - MaxMemoryMb uint64 `protobuf:"varint,3,opt,name=max_memory_mb,json=maxMemoryMb,proto3" json:"max_memory_mb,omitempty"` // memory cap for native decoder - Concurrency uint32 `protobuf:"varint,4,opt,name=concurrency,proto3" json:"concurrency,omitempty"` // native decoder parallelism - Profile string `protobuf:"bytes,5,opt,name=profile,proto3" json:"profile,omitempty"` // selected profile: edge|standard|perf - HeadroomPct int32 `protobuf:"varint,6,opt,name=headroom_pct,json=headroomPct,proto3" json:"headroom_pct,omitempty"` // reserved memory percentage (0-90) - MemLimitMb uint64 `protobuf:"varint,7,opt,name=mem_limit_mb,json=memLimitMb,proto3" json:"mem_limit_mb,omitempty"` // detected memory limit (MB) - MemLimitSource string `protobuf:"bytes,8,opt,name=mem_limit_source,json=memLimitSource,proto3" json:"mem_limit_source,omitempty"` // detection source (cgroup/meminfo) - EffectiveCores int32 `protobuf:"varint,9,opt,name=effective_cores,json=effectiveCores,proto3" json:"effective_cores,omitempty"` // detected cores/quota - CpuLimitSource string `protobuf:"bytes,10,opt,name=cpu_limit_source,json=cpuLimitSource,proto3" json:"cpu_limit_source,omitempty"` // detection source (cgroups/NumCPU) + SymbolSize uint32 `protobuf:"varint,1,opt,name=symbol_size,json=symbolSize,proto3" json:"symbol_size,omitempty"` // bytes (typically 65535) + Redundancy uint32 `protobuf:"varint,2,opt,name=redundancy,proto3" json:"redundancy,omitempty"` // repair factor (percent-like scalar; 5 = default) + MaxMemoryMb uint64 `protobuf:"varint,3,opt,name=max_memory_mb,json=maxMemoryMb,proto3" json:"max_memory_mb,omitempty"` // memory cap for native decoder + Concurrency uint32 `protobuf:"varint,4,opt,name=concurrency,proto3" json:"concurrency,omitempty"` // native decoder parallelism + HeadroomPct int32 `protobuf:"varint,6,opt,name=headroom_pct,json=headroomPct,proto3" json:"headroom_pct,omitempty"` // reserved memory percentage (0-90) + MemLimitMb uint64 `protobuf:"varint,7,opt,name=mem_limit_mb,json=memLimitMb,proto3" json:"mem_limit_mb,omitempty"` // detected memory limit (MB) + MemLimitSource string `protobuf:"bytes,8,opt,name=mem_limit_source,json=memLimitSource,proto3" json:"mem_limit_source,omitempty"` // detection source (cgroup/meminfo) } func (x *StatusResponse_CodecConfig) Reset() { @@ -676,13 +673,6 @@ func (x *StatusResponse_CodecConfig) GetConcurrency() uint32 { return 0 } -func (x *StatusResponse_CodecConfig) GetProfile() string { - if x != nil { - return x.Profile - } - return "" -} - func (x *StatusResponse_CodecConfig) GetHeadroomPct() int32 { if x != nil { return x.HeadroomPct @@ -704,20 +694,6 @@ func (x *StatusResponse_CodecConfig) GetMemLimitSource() string { return "" } -func (x *StatusResponse_CodecConfig) GetEffectiveCores() int32 { - if x != nil { - return x.EffectiveCores - } - return 0 -} - -func (x *StatusResponse_CodecConfig) GetCpuLimitSource() string { - if x != nil { - return x.CpuLimitSource - } - return "" -} - type StatusResponse_Resources_CPU struct { state protoimpl.MessageState sizeCache protoimpl.SizeCache @@ -1435,7 +1411,7 @@ var file_supernode_supernode_proto_rawDesc = []byte{ 0x0a, 0x0b, 0x53, 0x65, 0x72, 0x76, 0x69, 0x63, 0x65, 0x49, 0x6e, 0x66, 0x6f, 0x12, 0x12, 0x0a, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x12, 0x18, 0x0a, 0x07, 0x6d, 0x65, 0x74, 0x68, 0x6f, 0x64, 0x73, 0x18, 0x02, 0x20, 0x03, - 0x28, 0x09, 0x52, 0x07, 0x6d, 0x65, 0x74, 0x68, 0x6f, 0x64, 0x73, 0x22, 0xb4, 0x1c, 0x0a, 0x0e, + 0x28, 0x09, 0x52, 0x07, 0x6d, 0x65, 0x74, 0x68, 0x6f, 0x64, 0x73, 0x22, 0xd9, 0x1b, 0x0a, 0x0e, 0x53, 0x74, 0x61, 0x74, 0x75, 0x73, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x18, 0x0a, 0x07, 0x76, 0x65, 0x72, 0x73, 0x69, 0x6f, 0x6e, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x07, 0x76, 0x65, 0x72, 0x73, 0x69, 0x6f, 0x6e, 0x12, 0x25, 0x0a, 0x0e, 0x75, 0x70, 0x74, 0x69, @@ -1640,7 +1616,7 @@ var file_supernode_supernode_proto_rawDesc = []byte{ 0x74, 0x72, 0x79, 0x12, 0x10, 0x0a, 0x03, 0x6b, 0x65, 0x79, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x03, 0x6b, 0x65, 0x79, 0x12, 0x14, 0x0a, 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x18, 0x02, 0x20, 0x01, 0x28, 0x03, 0x52, 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x3a, 0x02, 0x38, 0x01, 0x1a, - 0xf0, 0x02, 0x0a, 0x0b, 0x43, 0x6f, 0x64, 0x65, 0x63, 0x43, 0x6f, 0x6e, 0x66, 0x69, 0x67, 0x12, + 0x95, 0x02, 0x0a, 0x0b, 0x43, 0x6f, 0x64, 0x65, 0x63, 0x43, 0x6f, 0x6e, 0x66, 0x69, 0x67, 0x12, 0x1f, 0x0a, 0x0b, 0x73, 0x79, 0x6d, 0x62, 0x6f, 0x6c, 0x5f, 0x73, 0x69, 0x7a, 0x65, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0d, 0x52, 0x0a, 0x73, 0x79, 0x6d, 0x62, 0x6f, 0x6c, 0x53, 0x69, 0x7a, 0x65, 0x12, 0x1e, 0x0a, 0x0a, 0x72, 0x65, 0x64, 0x75, 0x6e, 0x64, 0x61, 0x6e, 0x63, 0x79, 0x18, 0x02, @@ -1649,38 +1625,33 @@ var file_supernode_supernode_proto_rawDesc = []byte{ 0x62, 0x18, 0x03, 0x20, 0x01, 0x28, 0x04, 0x52, 0x0b, 0x6d, 0x61, 0x78, 0x4d, 0x65, 0x6d, 0x6f, 0x72, 0x79, 0x4d, 0x62, 0x12, 0x20, 0x0a, 0x0b, 0x63, 0x6f, 0x6e, 0x63, 0x75, 0x72, 0x72, 0x65, 0x6e, 0x63, 0x79, 0x18, 0x04, 0x20, 0x01, 0x28, 0x0d, 0x52, 0x0b, 0x63, 0x6f, 0x6e, 0x63, 0x75, - 0x72, 0x72, 0x65, 0x6e, 0x63, 0x79, 0x12, 0x18, 0x0a, 0x07, 0x70, 0x72, 0x6f, 0x66, 0x69, 0x6c, - 0x65, 0x18, 0x05, 0x20, 0x01, 0x28, 0x09, 0x52, 0x07, 0x70, 0x72, 0x6f, 0x66, 0x69, 0x6c, 0x65, - 0x12, 0x21, 0x0a, 0x0c, 0x68, 0x65, 0x61, 0x64, 0x72, 0x6f, 0x6f, 0x6d, 0x5f, 0x70, 0x63, 0x74, - 0x18, 0x06, 0x20, 0x01, 0x28, 0x05, 0x52, 0x0b, 0x68, 0x65, 0x61, 0x64, 0x72, 0x6f, 0x6f, 0x6d, - 0x50, 0x63, 0x74, 0x12, 0x20, 0x0a, 0x0c, 0x6d, 0x65, 0x6d, 0x5f, 0x6c, 0x69, 0x6d, 0x69, 0x74, - 0x5f, 0x6d, 0x62, 0x18, 0x07, 0x20, 0x01, 0x28, 0x04, 0x52, 0x0a, 0x6d, 0x65, 0x6d, 0x4c, 0x69, - 0x6d, 0x69, 0x74, 0x4d, 0x62, 0x12, 0x28, 0x0a, 0x10, 0x6d, 0x65, 0x6d, 0x5f, 0x6c, 0x69, 0x6d, - 0x69, 0x74, 0x5f, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x18, 0x08, 0x20, 0x01, 0x28, 0x09, 0x52, - 0x0e, 0x6d, 0x65, 0x6d, 0x4c, 0x69, 0x6d, 0x69, 0x74, 0x53, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x12, - 0x27, 0x0a, 0x0f, 0x65, 0x66, 0x66, 0x65, 0x63, 0x74, 0x69, 0x76, 0x65, 0x5f, 0x63, 0x6f, 0x72, - 0x65, 0x73, 0x18, 0x09, 0x20, 0x01, 0x28, 0x05, 0x52, 0x0e, 0x65, 0x66, 0x66, 0x65, 0x63, 0x74, - 0x69, 0x76, 0x65, 0x43, 0x6f, 0x72, 0x65, 0x73, 0x12, 0x28, 0x0a, 0x10, 0x63, 0x70, 0x75, 0x5f, - 0x6c, 0x69, 0x6d, 0x69, 0x74, 0x5f, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x18, 0x0a, 0x20, 0x01, - 0x28, 0x09, 0x52, 0x0e, 0x63, 0x70, 0x75, 0x4c, 0x69, 0x6d, 0x69, 0x74, 0x53, 0x6f, 0x75, 0x72, - 0x63, 0x65, 0x32, 0xd7, 0x01, 0x0a, 0x10, 0x53, 0x75, 0x70, 0x65, 0x72, 0x6e, 0x6f, 0x64, 0x65, - 0x53, 0x65, 0x72, 0x76, 0x69, 0x63, 0x65, 0x12, 0x58, 0x0a, 0x09, 0x47, 0x65, 0x74, 0x53, 0x74, - 0x61, 0x74, 0x75, 0x73, 0x12, 0x18, 0x2e, 0x73, 0x75, 0x70, 0x65, 0x72, 0x6e, 0x6f, 0x64, 0x65, - 0x2e, 0x53, 0x74, 0x61, 0x74, 0x75, 0x73, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x1a, 0x19, - 0x2e, 0x73, 0x75, 0x70, 0x65, 0x72, 0x6e, 0x6f, 0x64, 0x65, 0x2e, 0x53, 0x74, 0x61, 0x74, 0x75, - 0x73, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x22, 0x16, 0x82, 0xd3, 0xe4, 0x93, 0x02, - 0x10, 0x12, 0x0e, 0x2f, 0x61, 0x70, 0x69, 0x2f, 0x76, 0x31, 0x2f, 0x73, 0x74, 0x61, 0x74, 0x75, - 0x73, 0x12, 0x69, 0x0a, 0x0c, 0x4c, 0x69, 0x73, 0x74, 0x53, 0x65, 0x72, 0x76, 0x69, 0x63, 0x65, - 0x73, 0x12, 0x1e, 0x2e, 0x73, 0x75, 0x70, 0x65, 0x72, 0x6e, 0x6f, 0x64, 0x65, 0x2e, 0x4c, 0x69, - 0x73, 0x74, 0x53, 0x65, 0x72, 0x76, 0x69, 0x63, 0x65, 0x73, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, - 0x74, 0x1a, 0x1f, 0x2e, 0x73, 0x75, 0x70, 0x65, 0x72, 0x6e, 0x6f, 0x64, 0x65, 0x2e, 0x4c, 0x69, - 0x73, 0x74, 0x53, 0x65, 0x72, 0x76, 0x69, 0x63, 0x65, 0x73, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, - 0x73, 0x65, 0x22, 0x18, 0x82, 0xd3, 0xe4, 0x93, 0x02, 0x12, 0x12, 0x10, 0x2f, 0x61, 0x70, 0x69, - 0x2f, 0x76, 0x31, 0x2f, 0x73, 0x65, 0x72, 0x76, 0x69, 0x63, 0x65, 0x73, 0x42, 0x36, 0x5a, 0x34, - 0x67, 0x69, 0x74, 0x68, 0x75, 0x62, 0x2e, 0x63, 0x6f, 0x6d, 0x2f, 0x4c, 0x75, 0x6d, 0x65, 0x72, - 0x61, 0x50, 0x72, 0x6f, 0x74, 0x6f, 0x63, 0x6f, 0x6c, 0x2f, 0x73, 0x75, 0x70, 0x65, 0x72, 0x6e, - 0x6f, 0x64, 0x65, 0x2f, 0x76, 0x32, 0x2f, 0x67, 0x65, 0x6e, 0x2f, 0x73, 0x75, 0x70, 0x65, 0x72, - 0x6e, 0x6f, 0x64, 0x65, 0x62, 0x06, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x33, + 0x72, 0x72, 0x65, 0x6e, 0x63, 0x79, 0x12, 0x21, 0x0a, 0x0c, 0x68, 0x65, 0x61, 0x64, 0x72, 0x6f, + 0x6f, 0x6d, 0x5f, 0x70, 0x63, 0x74, 0x18, 0x06, 0x20, 0x01, 0x28, 0x05, 0x52, 0x0b, 0x68, 0x65, + 0x61, 0x64, 0x72, 0x6f, 0x6f, 0x6d, 0x50, 0x63, 0x74, 0x12, 0x20, 0x0a, 0x0c, 0x6d, 0x65, 0x6d, + 0x5f, 0x6c, 0x69, 0x6d, 0x69, 0x74, 0x5f, 0x6d, 0x62, 0x18, 0x07, 0x20, 0x01, 0x28, 0x04, 0x52, + 0x0a, 0x6d, 0x65, 0x6d, 0x4c, 0x69, 0x6d, 0x69, 0x74, 0x4d, 0x62, 0x12, 0x28, 0x0a, 0x10, 0x6d, + 0x65, 0x6d, 0x5f, 0x6c, 0x69, 0x6d, 0x69, 0x74, 0x5f, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x18, + 0x08, 0x20, 0x01, 0x28, 0x09, 0x52, 0x0e, 0x6d, 0x65, 0x6d, 0x4c, 0x69, 0x6d, 0x69, 0x74, 0x53, + 0x6f, 0x75, 0x72, 0x63, 0x65, 0x4a, 0x04, 0x08, 0x05, 0x10, 0x06, 0x4a, 0x04, 0x08, 0x09, 0x10, + 0x0a, 0x4a, 0x04, 0x08, 0x0a, 0x10, 0x0b, 0x32, 0xd7, 0x01, 0x0a, 0x10, 0x53, 0x75, 0x70, 0x65, + 0x72, 0x6e, 0x6f, 0x64, 0x65, 0x53, 0x65, 0x72, 0x76, 0x69, 0x63, 0x65, 0x12, 0x58, 0x0a, 0x09, + 0x47, 0x65, 0x74, 0x53, 0x74, 0x61, 0x74, 0x75, 0x73, 0x12, 0x18, 0x2e, 0x73, 0x75, 0x70, 0x65, + 0x72, 0x6e, 0x6f, 0x64, 0x65, 0x2e, 0x53, 0x74, 0x61, 0x74, 0x75, 0x73, 0x52, 0x65, 0x71, 0x75, + 0x65, 0x73, 0x74, 0x1a, 0x19, 0x2e, 0x73, 0x75, 0x70, 0x65, 0x72, 0x6e, 0x6f, 0x64, 0x65, 0x2e, + 0x53, 0x74, 0x61, 0x74, 0x75, 0x73, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x22, 0x16, + 0x82, 0xd3, 0xe4, 0x93, 0x02, 0x10, 0x12, 0x0e, 0x2f, 0x61, 0x70, 0x69, 0x2f, 0x76, 0x31, 0x2f, + 0x73, 0x74, 0x61, 0x74, 0x75, 0x73, 0x12, 0x69, 0x0a, 0x0c, 0x4c, 0x69, 0x73, 0x74, 0x53, 0x65, + 0x72, 0x76, 0x69, 0x63, 0x65, 0x73, 0x12, 0x1e, 0x2e, 0x73, 0x75, 0x70, 0x65, 0x72, 0x6e, 0x6f, + 0x64, 0x65, 0x2e, 0x4c, 0x69, 0x73, 0x74, 0x53, 0x65, 0x72, 0x76, 0x69, 0x63, 0x65, 0x73, 0x52, + 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x1a, 0x1f, 0x2e, 0x73, 0x75, 0x70, 0x65, 0x72, 0x6e, 0x6f, + 0x64, 0x65, 0x2e, 0x4c, 0x69, 0x73, 0x74, 0x53, 0x65, 0x72, 0x76, 0x69, 0x63, 0x65, 0x73, 0x52, + 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x22, 0x18, 0x82, 0xd3, 0xe4, 0x93, 0x02, 0x12, 0x12, + 0x10, 0x2f, 0x61, 0x70, 0x69, 0x2f, 0x76, 0x31, 0x2f, 0x73, 0x65, 0x72, 0x76, 0x69, 0x63, 0x65, + 0x73, 0x42, 0x36, 0x5a, 0x34, 0x67, 0x69, 0x74, 0x68, 0x75, 0x62, 0x2e, 0x63, 0x6f, 0x6d, 0x2f, + 0x4c, 0x75, 0x6d, 0x65, 0x72, 0x61, 0x50, 0x72, 0x6f, 0x74, 0x6f, 0x63, 0x6f, 0x6c, 0x2f, 0x73, + 0x75, 0x70, 0x65, 0x72, 0x6e, 0x6f, 0x64, 0x65, 0x2f, 0x76, 0x32, 0x2f, 0x67, 0x65, 0x6e, 0x2f, + 0x73, 0x75, 0x70, 0x65, 0x72, 0x6e, 0x6f, 0x64, 0x65, 0x62, 0x06, 0x70, 0x72, 0x6f, 0x74, 0x6f, + 0x33, } var ( diff --git a/gen/supernode/supernode.swagger.json b/gen/supernode/supernode.swagger.json index eadb5fbc..51f22046 100644 --- a/gen/supernode/supernode.swagger.json +++ b/gen/supernode/supernode.swagger.json @@ -338,10 +338,6 @@ "format": "int64", "title": "native decoder parallelism" }, - "profile": { - "type": "string", - "title": "selected profile: edge|standard|perf" - }, "headroomPct": { "type": "integer", "format": "int32", @@ -355,15 +351,6 @@ "memLimitSource": { "type": "string", "title": "detection source (cgroup/meminfo)" - }, - "effectiveCores": { - "type": "integer", - "format": "int32", - "title": "detected cores/quota" - }, - "cpuLimitSource": { - "type": "string", - "title": "detection source (cgroups/NumCPU)" } }, "title": "RaptorQ codec configuration (effective values)" diff --git a/p2p/kademlia/network.go b/p2p/kademlia/network.go index a2dc1e45..366f0f4a 100644 --- a/p2p/kademlia/network.go +++ b/p2p/kademlia/network.go @@ -32,22 +32,14 @@ const ( defaultMaxPayloadSize = 200 // MB errorBusy = "Busy" maxConcurrentFindBatchValsRequests = 25 - // defaultExecTimeout is for small control-plane RPCs; large payload RPCs - // have explicit entries in execTimeouts below. - defaultExecTimeout = 10 * time.Second + defaultExecTimeout = 10 * time.Second ) // Global map for message type timeouts var execTimeouts map[int]time.Duration func init() { - // Initialize the request execution timeout values. - // These defaults are intentionally conservative to accommodate slower - // peers and larger payloads. If future deployments consistently see - // responsive nodes, consider reducing the larger RPC timeouts (e.g., - // BatchStoreData/BatchGetValues to ~45s, BatchFindValues to ~30s) to - // fail fast on degraded nodes. Long, user-dependent operations like - // uploads/downloads are governed at higher layers. + // Initialize the request execution timeout values execTimeouts = map[int]time.Duration{ Ping: 5 * time.Second, FindNode: 10 * time.Second, diff --git a/pkg/codec/README.md b/pkg/codec/README.md index 3ae6557c..8e38b23f 100644 --- a/pkg/codec/README.md +++ b/pkg/codec/README.md @@ -1,26 +1,8 @@ # Codec (RaptorQ) Guide -## Table of Contents -- [Overview](#overview) -- [What It Does](#what-it-does) -- [Behavioural Note](#behavioural-note) -- [What We Don’t Do](#what-we-dont-do) -- [Quick Defaults (no env)](#quick-defaults-no-env) -- [Change Behavior (simple rules)](#change-behavior-simple-rules) -- [Profiles (behavior overview)](#profiles-behavior-overview) -- [Where It Applies](#where-it-applies) -- [Integration](#integration) -- [Notes](#notes) -- [Log Observability](#log-observability) -- [P2P interaction (high‑level)](#p2p-interaction-highlevel) -- [Examples (no env)](#examples-no-env) -- [Minimal Usage (Decode)](#minimal-usage-decode) -- [Minimal Usage (Encode)](#minimal-usage-encode) -- [Troubleshooting](#troubleshooting) - ## Overview - Thin, well‑scoped API around RaptorQ for encoding/decoding cascade artefacts. -- Safe defaults and memory‑conscious behavior tuned for files up to 1 GB. +- Fixed policy (no env vars, no profiles), tuned for predictable behavior. ## What It Does - Per‑request processor: create/free a processor per Encode/Decode to bound native memory. @@ -30,66 +12,20 @@ - Encode → `//` - Decode → `//` (decoded output sits alongside) -## Behavioural Note -- Decode mutates input: it deletes entries from `DecodeRequest.Symbols` after flushing to disk to free memory. - -## What We Don’t Do -- Require env setup: runs fine without any env vars. If set, env vars override the defaults. -- Full pre‑fetch of symbols: progressive retrieval avoids over‑fetching and reduces memory pressure. -- Long‑lived processors: we create/free a processor per request to bound native memory. - -## Quick Defaults (no env) -- Profile: perf (fastest defaults) -- Headroom: `LUMERA_RQ_MEM_HEADROOM_PCT=40` → usable_mem = limit × (1−0.40) -- Max memory: `min(0.6 × usable_mem, 16 GiB)` -- Concurrency: `min(8, effective_cores)`, then reduced so `max_memory_mb / concurrency ≥ 512 MB` -- Symbol size: `65535` • Redundancy: `5` - -## Change Behavior (simple rules) -- Pick profile: set `CODEC_PROFILE=edge|standard|perf` (perf is default) -- Reserve headroom: set `LUMERA_RQ_MEM_HEADROOM_PCT` (0–90, default 40) -- Override knobs directly (take precedence over profile): - - `LUMERA_RQ_MAX_MEMORY_MB`, `LUMERA_RQ_CONCURRENCY`, `LUMERA_RQ_SYMBOL_SIZE`, `LUMERA_RQ_REDUNDANCY` -- Detection sources: memory from cgroups v2/v1 (fallback `/proc/meminfo`), CPU from cgroup quota or `runtime.NumCPU()` - -## Profiles (behavior overview) - -| Profile | Default selection | Max memory (default) | Concurrency (default) | CPU cap | Min per‑worker MB | Symbol size | Redundancy | Env overrides | -|-----------|-------------------|-----------------------------------------------|--------------------------------------|-------------------------------|-------------------|-------------|------------|----------------------------------------| -| `edge` | Only when forced | `min(usable_mem, 1 GiB)` | `2` | Capped by effective cores | `≥ 512` | 65535 | 5 | `LUMERA_RQ_*`, `CODEC_PROFILE=edge` | -| `standard`| Only when forced | `min(0.6 × usable_mem, 4 GiB)` | `4` | Capped by effective cores | `≥ 512` | 65535 | 5 | `LUMERA_RQ_*`, `CODEC_PROFILE=standard`| -| `perf` | Default | `min(0.6 × usable_mem, 16 GiB)` | `8` | Capped by effective cores | `≥ 512` | 65535 | 5 | `LUMERA_RQ_*`, `CODEC_PROFILE=perf` | - -- usable_mem = memory_limit × (1 − `LUMERA_RQ_MEM_HEADROOM_PCT`/100). Default headroom = 40%. -- Effective cores: derived from cgroup CPU quota (v2 `cpu.max`, v1 `cpu.cfs_*`) or `runtime.NumCPU()`. -- Per‑worker memory: if `max_memory_mb / concurrency < 512`, concurrency is reduced until the target is met (down to 1). -- Env overrides: any of `LUMERA_RQ_SYMBOL_SIZE`, `LUMERA_RQ_REDUNDANCY`, `LUMERA_RQ_MAX_MEMORY_MB`, `LUMERA_RQ_CONCURRENCY` supersede the profile defaults. +## Fixed Policy +- Concurrency: `4` +- Symbol size: `65535` +- Redundancy: `5` +- Max memory: system/cgroup memory minus `10%` headroom (no environment overrides) ## Where It Applies - Encode: `pkg/codec/raptorq.go::Encode()` → compute block size, `EncodeFile()`, read layout - Decode: `pkg/codec/decode.go::Decode()` → write symbols + layout to disk, `DecodeSymbols()` -- Progressive decode: `supernode/supernode/services/cascade/progressive_decode.go` escalates required symbols (9%, 25%, 50%, 75%, 100%) - -## Integration -- Adaptors: `supernode/supernode/services/cascade/adaptors/rq.go` bridge the codec to higher‑level services -- Download flow: `supernode/supernode/services/cascade/download.go` uses progressive decode and verifies final file hash ## Notes - Error code vs message: treat the error code as authoritative; message is context only - Disk usage: ensure `` has space for symbols and the final file - Cleanup: callers delete the decode temp dir (`DecodeTmpDir`) -- File size: defaults suit ≤1 GiB; use overrides or profiles for larger files - -## Log Observability -- On processor creation we log: symbol_size, redundancy_factor, max_memory_mb, concurrency, profile, headroom_pct, mem_limit and source, effective_cores and source. - -## P2P interaction (high‑level) -- Codec redundancy and DHT replication are independent; progressive decode needs “enough” distinct symbols, not all of them - -## Examples (no env) -- 16 GiB limit, 8 cores → usable=9.6 GiB → max=5.76 GiB → conc=8 → ~720 MB/worker -- 8 GiB limit, 4 cores → usable=4.8 GiB → max=2.88 GiB → conc=4 → ~720 MB/worker -- 2 GiB limit, 2 cores → usable=1.2 GiB → max=0.72 GiB → conc=1 (to keep ≥512 MB/worker) ## Minimal Usage (Decode) ```go diff --git a/pkg/codec/codec.go b/pkg/codec/codec.go index 23bfbc21..329a0ba5 100644 --- a/pkg/codec/codec.go +++ b/pkg/codec/codec.go @@ -1,18 +1,16 @@ //go:generate mockgen -destination=codec_mock.go -package=codec -source=codec.go // Package codec provides an abstraction over the RaptorQ encoding/decoding engine -// used by the supernode for cascade artefacts. It centralizes resource tuning and -// safe decode/encode workflows. +// used by the supernode for cascade artefacts. It centralizes safe encode/decode +// workflows with a fixed policy (no environment overrides): +// - Concurrency: 4 +// - Symbol size: 65535 +// - Redundancy: 5 +// - Max memory: detected system/cgroup memory minus slight headroom (10%) // -// - Memory/Concurrency: The concrete implementation (raptorQ) reads runtime overrides -// from environment variables to tame memory pressure on different deployments: -// * LUMERA_RQ_SYMBOL_SIZE (uint16, default 65535) -// * LUMERA_RQ_REDUNDANCY (uint8, default 5) -// * LUMERA_RQ_MAX_MEMORY_MB (uint64, default 16384) -// * LUMERA_RQ_CONCURRENCY (uint64, default 4) -// - Decode Memory Hygiene: Symbols passed in memory are written to disk immediately -// and dropped from RAM during decode to minimize overlapping heap and native -// allocations. +// Decode Memory Hygiene: Symbols passed in memory are written to disk immediately +// and dropped from RAM during decode to minimize overlapping heap and native +// allocations. package codec import ( diff --git a/pkg/codec/decode.go b/pkg/codec/decode.go index d54a766c..d2b99091 100644 --- a/pkg/codec/decode.go +++ b/pkg/codec/decode.go @@ -17,8 +17,8 @@ type DecodeRequest struct { } type DecodeResponse struct { - Path string - DecodeTmpDir string + Path string + DecodeTmpDir string } func (rq *raptorQ) Decode(ctx context.Context, req DecodeRequest) (DecodeResponse, error) { @@ -29,7 +29,7 @@ func (rq *raptorQ) Decode(ctx context.Context, req DecodeRequest) (DecodeRespons } logtrace.Info(ctx, "RaptorQ decode request received", fields) - // Use env-configurable processor to allow memory/concurrency tuning per deployment + // Create processor using fixed policy (no env overrides) processor, err := newProcessor(ctx) if err != nil { fields[logtrace.FieldError] = err.Error() @@ -43,38 +43,41 @@ func (rq *raptorQ) Decode(ctx context.Context, req DecodeRequest) (DecodeRespons return DecodeResponse{}, fmt.Errorf("mkdir %s: %w", symbolsDir, err) } - // Write symbols to disk as soon as possible to reduce heap residency. - // Prior versions kept symbols in memory until after decode, which could - // exacerbate memory pressure. We now persist to disk immediately and drop each entry - // from the map to allow GC to reclaim memory sooner. This helps avoid spikes that - // could coincide with RaptorQ allocations (cf. memory limit exceeded reports). - for id, data := range req.Symbols { - symbolPath := filepath.Join(symbolsDir, id) - if err := os.WriteFile(symbolPath, data, 0o644); err != nil { - fields[logtrace.FieldError] = err.Error() - return DecodeResponse{}, fmt.Errorf("write symbol %s: %w", id, err) - } - // Drop the in-memory copy promptly; safe to delete during range - delete(req.Symbols, id) - } - logtrace.Info(ctx, "symbols written to disk", fields) - - // ---------- write layout file ---------- - // Use a conventional filename to match rq-go documentation examples. - // The library consumes the explicit path, so name is not strictly required, but - // aligning with `_raptorq_layout.json` aids operators/debuggers. - layoutPath := filepath.Join(symbolsDir, "_raptorq_layout.json") - layoutBytes, err := json.Marshal(req.Layout) - if err != nil { - fields[logtrace.FieldError] = err.Error() - return DecodeResponse{}, fmt.Errorf("marshal layout: %w", err) - } - if err := os.WriteFile(layoutPath, layoutBytes, 0o644); err != nil { + // Build a reverse index from symbol ID -> block ID from the provided layout + symToBlock := buildSymbolToBlockIndex(req.Layout) + + // Write symbols to disk promptly to reduce heap residency. Place them under + // symbolsDir/block_/ to match rq-go expectations. + if err := writeSymbolsPerBlock(ctx, symbolsDir, symToBlock, req.Symbols); err != nil { + fields[logtrace.FieldError] = err.Error() + return DecodeResponse{}, err + } + logtrace.Info(ctx, "symbols written to disk (per-block)", fields) + + // ---------- write layout file ---------- + // Use a conventional filename to match rq-go documentation examples. + // The library consumes the explicit path, so name is not strictly required, but + // aligning with `_raptorq_layout.json` aids operators/debuggers. + layoutPath, err := writeLayoutFile(req.Layout, symbolsDir) + if err != nil { fields[logtrace.FieldError] = err.Error() - return DecodeResponse{}, fmt.Errorf("write layout file: %w", err) + return DecodeResponse{}, err } logtrace.Info(ctx, "layout.json written", fields) + // ---------- preflight check: ensure at least one symbol exists for each block ---------- + perBlockCounts := computePerBlockCounts(req.Layout, symbolsDir) + fields["per_block_counts"] = perBlockCounts + logtrace.Info(ctx, "pre-decode per-block symbol counts", fields) + + // If any block has zero symbols, fail fast with an "insufficient symbols" message + // so the progressive retriever can escalate and fetch more. + for _, blk := range req.Layout.Blocks { + if perBlockCounts[blk.BlockID] == 0 { + return DecodeResponse{}, fmt.Errorf("insufficient symbols: no symbols found for block %d", blk.BlockID) + } + } + // Decode the symbols into an output file using the provided layout. outputPath := filepath.Join(symbolsDir, "output") if err := processor.DecodeSymbols(symbolsDir, outputPath, layoutPath); err != nil { @@ -86,3 +89,75 @@ func (rq *raptorQ) Decode(ctx context.Context, req DecodeRequest) (DecodeRespons logtrace.Info(ctx, "RaptorQ decoding completed successfully", fields) return DecodeResponse{Path: outputPath, DecodeTmpDir: symbolsDir}, nil } + +// buildSymbolToBlockIndex constructs a lookup from symbol ID to its block ID +// based on the provided layout. +func buildSymbolToBlockIndex(layout Layout) map[string]int { + m := make(map[string]int) + for _, blk := range layout.Blocks { + for _, sid := range blk.Symbols { + m[sid] = blk.BlockID + } + } + return m +} + +// writeSymbolsPerBlock writes symbols to per-block directories to match rq-go expectations. +// It also deletes each symbol from the provided map after persisting to reduce heap residency. +func writeSymbolsPerBlock(ctx context.Context, symbolsDir string, symToBlock map[string]int, symbols map[string][]byte) error { + for id, data := range symbols { + blockID, ok := symToBlock[id] + var destDir string + if ok { + destDir = filepath.Join(symbolsDir, fmt.Sprintf("block_%d", blockID)) + } else { + // Fallback: if symbol not present in layout (unexpected), keep at root. + destDir = symbolsDir + logtrace.Info(ctx, "symbol ID not present in layout; writing at root", logtrace.Fields{"symbol_id": id}) + } + if err := os.MkdirAll(destDir, 0o755); err != nil { + return fmt.Errorf("mkdir %s: %w", destDir, err) + } + symbolPath := filepath.Join(destDir, id) + if err := os.WriteFile(symbolPath, data, 0o644); err != nil { + return fmt.Errorf("write symbol %s: %w", id, err) + } + delete(symbols, id) + } + return nil +} + +// writeLayoutFile marshals and writes the layout JSON into the symbols directory. +func writeLayoutFile(layout Layout, symbolsDir string) (string, error) { + layoutPath := filepath.Join(symbolsDir, "_raptorq_layout.json") + layoutBytes, err := json.Marshal(layout) + if err != nil { + return "", fmt.Errorf("marshal layout: %w", err) + } + if err := os.WriteFile(layoutPath, layoutBytes, 0o644); err != nil { + return "", fmt.Errorf("write layout file: %w", err) + } + return layoutPath, nil +} + +// computePerBlockCounts counts non-directory, non-layout files under each block directory. +func computePerBlockCounts(layout Layout, symbolsDir string) map[int]int { + counts := make(map[int]int) + for _, blk := range layout.Blocks { + blockDir := filepath.Join(symbolsDir, fmt.Sprintf("block_%d", blk.BlockID)) + entries, err := os.ReadDir(blockDir) + if err != nil { + counts[blk.BlockID] = 0 + continue + } + c := 0 + for _, e := range entries { + if e.IsDir() || e.Name() == "_raptorq_layout.json" { + continue + } + c++ + } + counts[blk.BlockID] = c + } + return counts +} diff --git a/pkg/codec/raptorq.go b/pkg/codec/raptorq.go index 3437b466..f2437d15 100644 --- a/pkg/codec/raptorq.go +++ b/pkg/codec/raptorq.go @@ -4,10 +4,8 @@ import ( "context" "encoding/json" "fmt" - "math" "os" "path/filepath" - "runtime" "strconv" "strings" @@ -15,6 +13,18 @@ import ( "github.com/LumeraProtocol/supernode/v2/pkg/logtrace" ) +// Fixed policy (linux/amd64 only): +// - Concurrency: 1 +// - Symbol size: 65535 +// - Redundancy: 5 +// - Max memory: use detected system/cgroup memory with headroom +const ( + fixedConcurrency = 1 + defaultRedundancy = 5 + headroomPct = 20 // simple fixed safety margin + targetBlockMB = 128 // cap block size on encode (MB); 0 means use recommended +) + type raptorQ struct { symbolsBaseDir string } @@ -26,112 +36,40 @@ func NewRaptorQCodec(dir string) Codec { } -// newProcessor constructs a RaptorQ processor using environment overrides when provided. -// -// Environment variables: -// - LUMERA_RQ_SYMBOL_SIZE (uint16, default 65535) -// - LUMERA_RQ_REDUNDANCY (uint8, default 5) -// - LUMERA_RQ_MAX_MEMORY_MB (uint64, default 16384) -// - LUMERA_RQ_CONCURRENCY (uint64, default 4) -// -// The goal is to allow deployments to tune memory and concurrency to avoid pressure spikes -// without changing code paths, while keeping the prior defaults when unset. +// newProcessor constructs a RaptorQ processor using a fixed policy: +// - concurrency=1 +// - symbol size=65535 +// - redundancy=5 +// - max memory = detected system/cgroup memory with slight headroom func newProcessor(ctx context.Context) (*raptorq.RaptorQProcessor, error) { - // 1) Detect resources (memory/CPU) memLimitMB, memSource := detectMemoryLimitMB() - effCores, cpuSource := detectEffectiveCores() - - // 2) Apply headroom knob and compute usable memory - headroomPct := readInt("LUMERA_RQ_MEM_HEADROOM_PCT", 40, 0, 90) usableMemMB := computeUsableMem(memLimitMB, headroomPct) - // 3) Select profile (forced via env or inferred) - profile := selectProfile(os.Getenv("CODEC_PROFILE")) - - // 4) Compute default limits for the chosen profile - defMaxMemMB, defConcurrency := defaultLimitsForProfile(profile, usableMemMB) - - // 5) Adjust concurrency by effective cores/quotas - defConcurrency = adjustConcurrency(defConcurrency, effCores) + // Fixed params + symbolSize := uint16(raptorq.DefaultSymbolSize) + redundancy := uint8(defaultRedundancy) + concurrency := uint64(fixedConcurrency) + maxMemMB := usableMemMB - // 6) Ensure per-worker memory target (>=512MB) by reducing concurrency if needed - defConcurrency = rebalancePerWorkerMem(defMaxMemMB, defConcurrency, 512) - - // 7) Read env overrides (env wins) - symbolSize := uint16(readUint("LUMERA_RQ_SYMBOL_SIZE", uint64(raptorq.DefaultSymbolSize), 1024, 65535)) - redundancy := uint8(readUint("LUMERA_RQ_REDUNDANCY", 5, 1, 32)) - maxMemMB := readUint("LUMERA_RQ_MAX_MEMORY_MB", defMaxMemMB, 256, 1<<20) - concurrency := readUint("LUMERA_RQ_CONCURRENCY", defConcurrency, 1, 1024) - - // 8) Log final configuration + perWorkerMB := uint64(0) + if concurrency > 0 { + perWorkerMB = maxMemMB / concurrency + } logtrace.Info(ctx, "RaptorQ processor config", logtrace.Fields{ "symbol_size": symbolSize, "redundancy_factor": redundancy, "max_memory_mb": maxMemMB, "concurrency": concurrency, - "profile": profile, + "per_worker_mb": perWorkerMB, "headroom_pct": headroomPct, "mem_limit_mb": memLimitMB, "mem_limit_source": memSource, - "effective_cores": effCores, - "cpu_limit_source": cpuSource, }) - // 9) Construct processor return raptorq.NewRaptorQProcessor(symbolSize, redundancy, maxMemMB, concurrency) } -// RaptorQConfig describes the effective codec configuration derived from env, resources and defaults. -type RaptorQConfig struct { - SymbolSize uint16 - Redundancy uint8 - MaxMemoryMB uint64 - Concurrency uint64 - Profile string - HeadroomPct int - MemLimitMB uint64 - MemLimitSource string - EffectiveCores int - CpuLimitSource string -} - -// CurrentConfig computes the current effective RaptorQ configuration without allocating a processor. -func CurrentConfig(ctx context.Context) RaptorQConfig { - // 1) Detect resources - memLimitMB, memSource := detectMemoryLimitMB() - effCores, cpuSource := detectEffectiveCores() - - // 2) Apply headroom and select profile - headroomPct := readInt("LUMERA_RQ_MEM_HEADROOM_PCT", 40, 0, 90) - usableMemMB := computeUsableMem(memLimitMB, headroomPct) - profile := selectProfile(os.Getenv("CODEC_PROFILE")) - - // 3) Compute defaults and adjust by cores and per‑worker budget - defMaxMemMB, defConcurrency := defaultLimitsForProfile(profile, usableMemMB) - defConcurrency = adjustConcurrency(defConcurrency, effCores) - defConcurrency = rebalancePerWorkerMem(defMaxMemMB, defConcurrency, 512) - - // 4) Apply env overrides - symbolSize := uint16(readUint("LUMERA_RQ_SYMBOL_SIZE", uint64(raptorq.DefaultSymbolSize), 1024, 65535)) - redundancy := uint8(readUint("LUMERA_RQ_REDUNDANCY", 5, 1, 32)) - maxMemMB := readUint("LUMERA_RQ_MAX_MEMORY_MB", defMaxMemMB, 256, 1<<20) - concurrency := readUint("LUMERA_RQ_CONCURRENCY", defConcurrency, 1, 1024) - - return RaptorQConfig{ - SymbolSize: symbolSize, - Redundancy: redundancy, - MaxMemoryMB: maxMemMB, - Concurrency: concurrency, - Profile: profile, - HeadroomPct: headroomPct, - MemLimitMB: memLimitMB, - MemLimitSource: memSource, - EffectiveCores: effCores, - CpuLimitSource: cpuSource, - } -} - -// detectMemoryLimitMB attempts to determine the memory limit (MB) from cgroups, falling back to MemTotal. +// detectMemoryLimitMB determines the memory limit (MB) from cgroups, falling back to MemTotal. func detectMemoryLimitMB() (uint64, string) { // cgroup v2: /sys/fs/cgroup/memory.max if b, err := os.ReadFile("/sys/fs/cgroup/memory.max"); err == nil { @@ -147,11 +85,10 @@ func detectMemoryLimitMB() (uint64, string) { s := strings.TrimSpace(string(b)) if v, err := strconv.ParseUint(s, 10, 64); err == nil && v > 0 { // Some systems report a huge number when unlimited; treat > 1PB as unlimited - if v > 1<<50 { - // unlimited; fallthrough to MemTotal - } else { + if v <= 1<<50 { return v / (1024 * 1024), "cgroupv1:memory.limit_in_bytes" } + // unlimited; fallthrough to MemTotal } } // Fallback: /proc/meminfo MemTotal @@ -171,153 +108,12 @@ func detectMemoryLimitMB() (uint64, string) { return 0, "unknown" } -// detectEffectiveCores attempts to determine CPU quota; returns cores and source. -func detectEffectiveCores() (int, string) { - // cgroup v2: /sys/fs/cgroup/cpu.max: "max" or " " - if b, err := os.ReadFile("/sys/fs/cgroup/cpu.max"); err == nil { - parts := strings.Fields(strings.TrimSpace(string(b))) - if len(parts) == 2 && parts[0] != "max" { - if quota, err1 := strconv.ParseUint(parts[0], 10, 64); err1 == nil { - if period, err2 := strconv.ParseUint(parts[1], 10, 64); err2 == nil && period > 0 { - cores := int(float64(quota) / float64(period)) - if cores < 1 { - cores = 1 - } - return cores, "cgroupv2:cpu.max" - } - } - } - } - // cgroup v1: /sys/fs/cgroup/cpu/cpu.cfs_quota_us and cpu.cfs_period_us - if qb, err1 := os.ReadFile("/sys/fs/cgroup/cpu/cpu.cfs_quota_us"); err1 == nil { - if pb, err2 := os.ReadFile("/sys/fs/cgroup/cpu/cpu.cfs_period_us"); err2 == nil { - qStr := strings.TrimSpace(string(qb)) - pStr := strings.TrimSpace(string(pb)) - if qStr != "-1" { - if quota, errA := strconv.ParseUint(qStr, 10, 64); errA == nil { - if period, errB := strconv.ParseUint(pStr, 10, 64); errB == nil && period > 0 { - cores := int(float64(quota) / float64(period)) - if cores < 1 { - cores = 1 - } - return cores, "cgroupv1:cpu.cfs_quota_us/period" - } - } - } - } - } - return 0, "runtime.NumCPU" -} - -func minNonZero(a, b uint64) uint64 { - if a == 0 { - return b - } - if b == 0 { - return a - } - if a < b { - return a - } - return b -} - -// readUint reads an unsigned integer from env with bounds and a default. -func readUint(env string, def uint64, min uint64, max uint64) uint64 { - if v, ok := os.LookupEnv(env); ok { - if n, err := strconv.ParseUint(v, 10, 64); err == nil { - if n < min { - return min - } - if max > 0 && n > max { - return max - } - return n - } - } - return def -} - -// readInt reads an integer from env with bounds and a default. -func readInt(env string, def int, min int, max int) int { - if v, ok := os.LookupEnv(env); ok { - if n, err := strconv.Atoi(v); err == nil { - if n < min { - return min - } - if max > 0 && n > max { - return max - } - return n - } - } - return def -} - -// computeUsableMem applies a headroom percentage to a memory limit. +// computeUsableMem applies a headroom percentage to a memory limit using integer math. func computeUsableMem(memLimitMB uint64, headroomPct int) uint64 { if headroomPct <= 0 || memLimitMB == 0 { return memLimitMB } - return uint64(math.Max(0, float64(memLimitMB)*(1.0-float64(headroomPct)/100.0))) -} - -// selectProfile decides which profile to use based on a forced value or memory limit. -func selectProfile(forced string) string { - p := strings.ToLower(strings.TrimSpace(forced)) - switch p { - case "edge", "standard", "perf": - return p - } - // Default to perf when not forced - return "perf" -} - -// defaultLimitsForProfile returns default max memory and concurrency for a profile. -func defaultLimitsForProfile(profile string, usableMemMB uint64) (uint64, uint64) { - switch profile { - case "edge": - mm := minNonZero(usableMemMB, 1024) - if mm == 0 { - mm = 1024 - } - return mm, 2 - case "standard": - mm := uint64(math.Min(float64(usableMemMB)*0.6, 4*1024)) - if mm < 1024 { - mm = 1024 - } - return mm, 4 - default: // perf - mm := uint64(math.Min(float64(usableMemMB)*0.6, 16*1024)) - return mm, 8 - } -} - -// adjustConcurrency caps concurrency by effective cores. -func adjustConcurrency(defConcurrency uint64, effCores int) uint64 { - cpu := runtime.NumCPU() - if effCores > 0 { - cpu = effCores - } - if cpu < 1 { - cpu = 1 - } - if defConcurrency > uint64(cpu) { - return uint64(cpu) - } - return defConcurrency -} - -// rebalancePerWorkerMem reduces concurrency until each worker has at least minPerWorkerMB. -func rebalancePerWorkerMem(defMaxMemMB uint64, defConcurrency uint64, minPerWorkerMB uint64) uint64 { - if defConcurrency == 0 { - return 1 - } - for defMaxMemMB/defConcurrency < minPerWorkerMB && defConcurrency > 1 { - defConcurrency-- - } - return defConcurrency + return memLimitMB - (memLimitMB*uint64(headroomPct))/100 } func (rq *raptorQ) Encode(ctx context.Context, req EncodeRequest) (EncodeResponse, error) { @@ -330,7 +126,7 @@ func (rq *raptorQ) Encode(ctx context.Context, req EncodeRequest) (EncodeRespons "data-size": req.DataSize, } - // Use env-configurable processor to allow memory/concurrency tuning per deployment + // Create processor using fixed policy (linux/amd64, no env) processor, err := newProcessor(ctx) if err != nil { return EncodeResponse{}, fmt.Errorf("create RaptorQ processor: %w", err) @@ -339,7 +135,20 @@ func (rq *raptorQ) Encode(ctx context.Context, req EncodeRequest) (EncodeRespons logtrace.Info(ctx, "RaptorQ processor created", fields) /* ---------- 1. run the encoder ---------- */ - blockSize := processor.GetRecommendedBlockSize(uint64(req.DataSize)) + // Determine block size with a simple cap (no env): use min(recommended, targetBlockMB) + rec := processor.GetRecommendedBlockSize(uint64(req.DataSize)) + var blockSize int + if targetBlockMB > 0 { + targetBytes := int(targetBlockMB) * 1024 * 1024 + if rec == 0 || rec > targetBytes { + blockSize = targetBytes + } else { + blockSize = rec + } + } else { + blockSize = rec + } + logtrace.Info(ctx, "RaptorQ recommended block size", logtrace.Fields{"block_size": rec, "chosen_block_size": blockSize, "target_block_mb": targetBlockMB}) symbolsDir := filepath.Join(rq.symbolsBaseDir, req.TaskID) if err := os.MkdirAll(symbolsDir, 0o755); err != nil { @@ -356,8 +165,7 @@ func (rq *raptorQ) Encode(ctx context.Context, req EncodeRequest) (EncodeRespons return EncodeResponse{}, fmt.Errorf("raptorq encode: %w", err) } - /* we no longer need the temp file */ - // _ = os.Remove(tmpPath) + // layout will be read from disk below /* ---------- 2. read the layout JSON ---------- */ layoutData, err := os.ReadFile(resp.LayoutFilePath) diff --git a/pkg/codecconfig/config.go b/pkg/codecconfig/config.go index f4ac8f05..55bc41f0 100644 --- a/pkg/codecconfig/config.go +++ b/pkg/codecconfig/config.go @@ -2,173 +2,56 @@ package codecconfig import ( "context" - "math" "os" - "runtime" "strconv" "strings" ) -// Config describes the effective codec configuration derived from env, resources and defaults. +// Config describes the effective codec configuration (fixed policy; no env). type Config struct { SymbolSize uint16 Redundancy uint8 MaxMemoryMB uint64 Concurrency uint64 - Profile string HeadroomPct int MemLimitMB uint64 MemLimitSource string - EffectiveCores int - CpuLimitSource string } // Defaults mirrored from rq-go and current project conventions. const ( - defaultSymbolSize = 65535 - defaultRedundancy = 5 - minPerWorkerMB = 512 + defaultSymbolSize = 65535 + defaultRedundancy = 5 + fixedConcurrency = 4 + headroomPct = 10 ) -// Current computes the current effective codec configuration without requiring cgo. +// Current computes the current effective codec configuration (fixed policy). func Current(ctx context.Context) Config { memLimitMB, memSource := detectMemoryLimitMB() - effCores, cpuSource := detectEffectiveCores() - - headroomPct := readInt("LUMERA_RQ_MEM_HEADROOM_PCT", 40, 0, 90) usableMemMB := computeUsableMem(memLimitMB, headroomPct) - profile := selectProfile(os.Getenv("CODEC_PROFILE")) - // Support alternative name for profile if provided - if p := strings.TrimSpace(os.Getenv("LUMERA_RQ_PROFILE")); p != "" { - profile = selectProfile(p) - } - - defMaxMemMB, defConcurrency := defaultLimitsForProfile(profile, usableMemMB) - defConcurrency = adjustConcurrency(defConcurrency, effCores) - defConcurrency = rebalancePerWorkerMem(defMaxMemMB, defConcurrency, minPerWorkerMB) - - symbolSize := uint16(readUint("LUMERA_RQ_SYMBOL_SIZE", uint64(defaultSymbolSize), 1024, 65535)) - redundancy := uint8(readUint("LUMERA_RQ_REDUNDANCY", uint64(defaultRedundancy), 1, 32)) - maxMemMB := readUint("LUMERA_RQ_MAX_MEMORY_MB", defMaxMemMB, 256, 1<<20) - concurrency := readUint("LUMERA_RQ_CONCURRENCY", defConcurrency, 1, 1024) + symbolSize := uint16(defaultSymbolSize) + redundancy := uint8(defaultRedundancy) + maxMemMB := usableMemMB + concurrency := uint64(fixedConcurrency) return Config{ SymbolSize: symbolSize, Redundancy: redundancy, MaxMemoryMB: maxMemMB, Concurrency: concurrency, - Profile: profile, HeadroomPct: headroomPct, MemLimitMB: memLimitMB, MemLimitSource: memSource, - EffectiveCores: effCores, - CpuLimitSource: cpuSource, } } -func minNonZero(a, b uint64) uint64 { - if a == 0 { - return b - } - if b == 0 { - return a - } - if a < b { - return a - } - return b -} - -func readUint(env string, def uint64, min uint64, max uint64) uint64 { - if v, ok := os.LookupEnv(env); ok { - if n, err := strconv.ParseUint(v, 10, 64); err == nil { - if n < min { - return min - } - if max > 0 && n > max { - return max - } - return n - } - } - return def -} - -func readInt(env string, def int, min int, max int) int { - if v, ok := os.LookupEnv(env); ok { - if n, err := strconv.Atoi(v); err == nil { - if n < min { - return min - } - if max > 0 && n > max { - return max - } - return n - } - } - return def -} - func computeUsableMem(memLimitMB uint64, headroomPct int) uint64 { if headroomPct <= 0 || memLimitMB == 0 { return memLimitMB } - return uint64(math.Max(0, float64(memLimitMB)*(1.0-float64(headroomPct)/100.0))) -} - -func selectProfile(forced string) string { - p := strings.ToLower(strings.TrimSpace(forced)) - switch p { - case "edge", "standard", "perf": - return p - } - // Default to perf when not forced - return "perf" -} - -func defaultLimitsForProfile(profile string, usableMemMB uint64) (uint64, uint64) { - switch profile { - case "edge": - mm := minNonZero(usableMemMB, 1024) - if mm == 0 { - mm = 1024 - } - return mm, 2 - case "standard": - mm := uint64(math.Min(float64(usableMemMB)*0.6, 4*1024)) - if mm < 1024 { - mm = 1024 - } - return mm, 4 - default: // perf - mm := uint64(math.Min(float64(usableMemMB)*0.6, 16*1024)) - return mm, 8 - } -} - -func adjustConcurrency(defConcurrency uint64, effCores int) uint64 { - cpu := runtime.NumCPU() - if effCores > 0 { - cpu = effCores - } - if cpu < 1 { - cpu = 1 - } - if defConcurrency > uint64(cpu) { - return uint64(cpu) - } - return defConcurrency -} - -func rebalancePerWorkerMem(defMaxMemMB uint64, defConcurrency uint64, minPerWorkerMB uint64) uint64 { - if defConcurrency == 0 { - return 1 - } - for defMaxMemMB/defConcurrency < minPerWorkerMB && defConcurrency > 1 { - defConcurrency-- - } - return defConcurrency + return memLimitMB - (memLimitMB*uint64(headroomPct))/100 } // detectMemoryLimitMB attempts to determine the memory limit (MB) from cgroups, falling back to MemTotal. @@ -211,40 +94,4 @@ func detectMemoryLimitMB() (uint64, string) { return 0, "unknown" } -// detectEffectiveCores attempts to determine CPU quota; returns cores and source. -func detectEffectiveCores() (int, string) { - // cgroup v2: /sys/fs/cgroup/cpu.max: "max" or " " - if b, err := os.ReadFile("/sys/fs/cgroup/cpu.max"); err == nil { - parts := strings.Fields(strings.TrimSpace(string(b))) - if len(parts) == 2 && parts[0] != "max" { - if quota, err1 := strconv.ParseUint(parts[0], 10, 64); err1 == nil { - if period, err2 := strconv.ParseUint(parts[1], 10, 64); err2 == nil && period > 0 { - cores := int(float64(quota) / float64(period)) - if cores < 1 { - cores = 1 - } - return cores, "cgroupv2:cpu.max" - } - } - } - } - // cgroup v1: /sys/fs/cgroup/cpu/cpu.cfs_quota_us and cpu.cfs_period_us - if qb, err1 := os.ReadFile("/sys/fs/cgroup/cpu/cpu.cfs_quota_us"); err1 == nil { - if pb, err2 := os.ReadFile("/sys/fs/cgroup/cpu/cpu.cfs_period_us"); err2 == nil { - qStr := strings.TrimSpace(string(qb)) - pStr := strings.TrimSpace(string(pb)) - if qStr != "-1" { - if quota, errA := strconv.ParseUint(qStr, 10, 64); errA == nil { - if period, errB := strconv.ParseUint(pStr, 10, 64); errB == nil && period > 0 { - cores := int(float64(quota) / float64(period)) - if cores < 1 { - cores = 1 - } - return cores, "cgroupv1:cpu.cfs_quota_us/period" - } - } - } - } - } - return 0, "runtime.NumCPU" -} +// No CPU quota detection needed for fixed policy. diff --git a/pkg/common/task/worker.go b/pkg/common/task/worker.go index a62752d9..280b5fb8 100644 --- a/pkg/common/task/worker.go +++ b/pkg/common/task/worker.go @@ -109,15 +109,10 @@ func NewWorker() *Worker { } // cleanupLoop periodically removes tasks that are in a final state for a grace period -// or any task that has been around for too long func (worker *Worker) cleanupLoop(ctx context.Context) { const ( cleanupInterval = 30 * time.Second finalTaskTTL = 2 * time.Minute - // maxTaskAge removes any task entry after this age, regardless of state. - // Keep greater than the largest server-side task envelope (RegisterTimeout ~75m) - // to avoid pruning legitimate long-running tasks from the worker registry. - maxTaskAge = 2 * time.Hour ) ticker := time.NewTicker(cleanupInterval) @@ -134,17 +129,11 @@ func (worker *Worker) cleanupLoop(ctx context.Context) { kept := worker.tasks[:0] for _, t := range worker.tasks { st := t.Status() - if st != nil { - // Remove any task older than 30 minutes, regardless of state - if now.Sub(st.CreatedAt) >= maxTaskAge { + if st != nil && st.SubStatus != nil && st.SubStatus.IsFinal() { + if now.Sub(st.CreatedAt) >= finalTaskTTL { + // drop this finalized task continue } - // Also remove final tasks after 2 minutes - if st.SubStatus != nil && st.SubStatus.IsFinal() { - if now.Sub(st.CreatedAt) >= finalTaskTTL { - continue - } - } } kept = append(kept, t) } diff --git a/proto/supernode/action/cascade/service.proto b/proto/supernode/action/cascade/service.proto index 06148539..9cfdb4c9 100644 --- a/proto/supernode/action/cascade/service.proto +++ b/proto/supernode/action/cascade/service.proto @@ -62,4 +62,8 @@ enum SupernodeEventType { ACTION_FINALIZED = 12; ARTEFACTS_DOWNLOADED = 13; FINALIZE_SIMULATION_FAILED = 14; + // Download phase markers (additive, backward compatible) + NETWORK_RETRIEVE_STARTED = 15; // Supernode started pulling symbols from network + DECODE_COMPLETED = 16; // File reconstructed and verified; ready on disk + SERVE_READY = 17; // File is ready to be streamed to client } diff --git a/proto/supernode/supernode.proto b/proto/supernode/supernode.proto index eb7cba4a..b305756a 100644 --- a/proto/supernode/supernode.proto +++ b/proto/supernode/supernode.proto @@ -164,12 +164,13 @@ message StatusResponse { uint32 redundancy = 2; // repair factor (percent-like scalar; 5 = default) uint64 max_memory_mb = 3; // memory cap for native decoder uint32 concurrency = 4; // native decoder parallelism - string profile = 5; // selected profile: edge|standard|perf + // reserved field 5 (was: profile) + reserved 5; int32 headroom_pct = 6; // reserved memory percentage (0-90) uint64 mem_limit_mb = 7; // detected memory limit (MB) string mem_limit_source = 8;// detection source (cgroup/meminfo) - int32 effective_cores = 9; // detected cores/quota - string cpu_limit_source = 10; // detection source (cgroups/NumCPU) + // reserved fields 9 and 10 (were: effective_cores, cpu_limit_source) + reserved 9, 10; } CodecConfig codec = 10; diff --git a/sdk/README.md b/sdk/README.md index fb56a610..b0aecb20 100644 --- a/sdk/README.md +++ b/sdk/README.md @@ -221,12 +221,6 @@ if err != nil { // taskID can be used to track the download progress ``` -Behavior notes: -- The SDK determines the final filename from on-chain cascade metadata and creates the path: `outputDir//`. -- Events will include both the supernode endpoint (`event.KeySupernode`) and its Cosmos address (`event.KeySupernodeAddress`). -- On success, both `SDKOutputPathReceived` and `SDKDownloadSuccessful` include `event.KeyOutputPath`. -- Failures include reason-coded suffixes in the message (e.g., `| reason=timeout`) and set `event.KeyMessage` accordingly. - **Parameters:** - `ctx context.Context`: Context for the operation - `actionID string`: ID of the action to download @@ -387,7 +381,7 @@ The SDK provides an event system to monitor task progress through event subscrip - `SDKRegistrationAttempt`: Attempting to register with a supernode - `SDKRegistrationFailure`: Registration with supernode failed - `SDKRegistrationSuccessful`: Successfully registered with supernode -- `SDKTaskTxHashReceived`: Transaction hash received from supernode (includes endpoint in `KeySupernode` and Cosmos address in `KeySupernodeAddress`) +- `SDKTaskTxHashReceived`: Transaction hash received from supernode - `SDKTaskCompleted`: Task completed successfully - `SDKTaskFailed`: Task failed with error - `SDKConnectionEstablished`: Connection to supernode established @@ -398,9 +392,9 @@ The SDK provides an event system to monitor task progress through event subscrip - `SDKProcessingFailed`: Processing failed (reason=stream_recv|missing_final_response) - `SDKProcessingTimeout`: Processing exceeded time budget and was cancelled - `SDKDownloadAttempt`: Attempting to download from supernode -- `SDKDownloadFailure`: Download attempt failed (message may include `| reason=timeout|canceled`; `KeyMessage` mirrors the reason) -- `SDKOutputPathReceived`: File download path received (includes `KeyOutputPath` and supernode identity keys) -- `SDKDownloadSuccessful`: Download completed successfully (includes `KeyOutputPath` and supernode identity keys) +- `SDKDownloadFailure`: Download attempt failed +- `SDKOutputPathReceived`: File download path received +- `SDKDownloadSuccessful`: Download completed successfully **Supernode Events (forwarded from supernodes):** - `SupernodeActionRetrieved`: Action retrieved from blockchain @@ -427,8 +421,8 @@ Events may include additional data accessible through these keys: - `event.KeyError`: Error message (for failure events) - `event.KeyCount`: Count of items (e.g., supernodes found) -- `event.KeySupernode`: Supernode gRPC endpoint -- `event.KeySupernodeAddress`: Supernode Cosmos address +- `event.KeySupernode`: Supernode endpoint +- `event.KeySupernodeAddress`: Supernode cosmos address - `event.KeyIteration`: Attempt iteration number - `event.KeyTxHash`: Transaction hash - `event.KeyMessage`: Event message diff --git a/sdk/adapters/supernodeservice/adapter.go b/sdk/adapters/supernodeservice/adapter.go index 8008b277..3b6b61dc 100644 --- a/sdk/adapters/supernodeservice/adapter.go +++ b/sdk/adapters/supernodeservice/adapter.go @@ -88,12 +88,13 @@ func (a *cascadeAdapter) CascadeSupernodeRegister(ctx context.Context, in *Casca // Create the client stream stream, err := a.client.Register(phaseCtx, opts...) if err != nil { - a.logger.Error(ctx, "Failed to create register stream", - "error", err) + a.logger.Error(ctx, "Failed to create register stream", "error", err) if in.EventLogger != nil { - in.EventLogger(baseCtx, event.SDKUploadFailed, "upload failed | reason=stream_open", event.EventData{ + in.EventLogger(baseCtx, event.SDKUploadFailed, "Upload failed", event.EventData{ event.KeyTaskID: in.TaskId, event.KeyActionID: in.ActionID, + event.KeyReason: "stream_open", + event.KeyError: err.Error(), }) } return nil, err @@ -104,9 +105,11 @@ func (a *cascadeAdapter) CascadeSupernodeRegister(ctx context.Context, in *Casca if err != nil { a.logger.Error(ctx, "Failed to open file", "filePath", in.FilePath, "error", err) if in.EventLogger != nil { - in.EventLogger(baseCtx, event.SDKUploadFailed, "upload failed | reason=file_open", event.EventData{ + in.EventLogger(baseCtx, event.SDKUploadFailed, "Upload failed", event.EventData{ event.KeyTaskID: in.TaskId, event.KeyActionID: in.ActionID, + event.KeyReason: "file_open", + event.KeyError: err.Error(), }) } return nil, fmt.Errorf("failed to open file: %w", err) @@ -118,9 +121,11 @@ func (a *cascadeAdapter) CascadeSupernodeRegister(ctx context.Context, in *Casca if err != nil { a.logger.Error(ctx, "Failed to get file stats", "filePath", in.FilePath, "error", err) if in.EventLogger != nil { - in.EventLogger(baseCtx, event.SDKUploadFailed, "upload failed | reason=file_stat", event.EventData{ + in.EventLogger(baseCtx, event.SDKUploadFailed, "Upload failed", event.EventData{ event.KeyTaskID: in.TaskId, event.KeyActionID: in.ActionID, + event.KeyReason: "file_stat", + event.KeyError: err.Error(), }) } return nil, fmt.Errorf("failed to get file stats: %w", err) @@ -149,9 +154,14 @@ func (a *cascadeAdapter) CascadeSupernodeRegister(ctx context.Context, in *Casca // Emit upload started event if in.EventLogger != nil { estChunks := (totalBytes + int64(chunkSize) - 1) / int64(chunkSize) - in.EventLogger(baseCtx, event.SDKUploadStarted, - fmt.Sprintf("upload started | size=%dB chunk_size=%dB est_chunks=%d", totalBytes, chunkSize, estChunks), - event.EventData{event.KeyTaskID: in.TaskId, event.KeyActionID: in.ActionID}) + in.EventLogger(baseCtx, event.SDKUploadStarted, "Upload started", + event.EventData{ + event.KeyTaskID: in.TaskId, + event.KeyActionID: in.ActionID, + event.KeyBytesTotal: totalBytes, + event.KeyChunkSize: chunkSize, + event.KeyEstChunks: estChunks, + }) } uploadStart := time.Now() @@ -160,7 +170,12 @@ func (a *cascadeAdapter) CascadeSupernodeRegister(ctx context.Context, in *Casca uploadTimer := time.AfterFunc(cascadeUploadTimeout, func() { a.logger.Error(baseCtx, "Upload phase timeout reached; cancelling stream") if in.EventLogger != nil { - in.EventLogger(baseCtx, event.SDKUploadFailed, "upload failed | reason=timeout", event.EventData{event.KeyTaskID: in.TaskId, event.KeyActionID: in.ActionID}) + in.EventLogger(baseCtx, event.SDKUploadFailed, "Upload failed", event.EventData{ + event.KeyTaskID: in.TaskId, + event.KeyActionID: in.ActionID, + event.KeyReason: "timeout", + event.KeyError: "upload phase timeout", + }) } cancel() }) @@ -175,9 +190,12 @@ func (a *cascadeAdapter) CascadeSupernodeRegister(ctx context.Context, in *Casca if err != nil { a.logger.Error(ctx, "Failed to read file chunk", "chunkIndex", chunkIndex, "error", err) if in.EventLogger != nil { - in.EventLogger(baseCtx, event.SDKUploadFailed, fmt.Sprintf("upload failed | reason=read_error chunk=%d", chunkIndex), event.EventData{ - event.KeyTaskID: in.TaskId, - event.KeyActionID: in.ActionID, + in.EventLogger(baseCtx, event.SDKUploadFailed, "Upload failed", event.EventData{ + event.KeyTaskID: in.TaskId, + event.KeyActionID: in.ActionID, + event.KeyReason: "read_error", + event.KeyChunkIndex: chunkIndex, + event.KeyError: err.Error(), }) } return nil, fmt.Errorf("failed to read file chunk: %w", err) @@ -195,9 +213,12 @@ func (a *cascadeAdapter) CascadeSupernodeRegister(ctx context.Context, in *Casca if err := stream.Send(chunk); err != nil { a.logger.Error(ctx, "Failed to send data chunk", "chunkIndex", chunkIndex, "error", err) if in.EventLogger != nil { - in.EventLogger(baseCtx, event.SDKUploadFailed, fmt.Sprintf("upload failed | reason=send_error chunk=%d", chunkIndex), event.EventData{ - event.KeyTaskID: in.TaskId, - event.KeyActionID: in.ActionID, + in.EventLogger(baseCtx, event.SDKUploadFailed, "Upload failed", event.EventData{ + event.KeyTaskID: in.TaskId, + event.KeyActionID: in.ActionID, + event.KeyReason: "send_error", + event.KeyChunkIndex: chunkIndex, + event.KeyError: err.Error(), }) } return nil, fmt.Errorf("failed to send chunk: %w", err) @@ -224,9 +245,11 @@ func (a *cascadeAdapter) CascadeSupernodeRegister(ctx context.Context, in *Casca if err := stream.Send(metadata); err != nil { a.logger.Error(ctx, "Failed to send metadata", "TaskId", in.TaskId, "ActionID", in.ActionID, "error", err) if in.EventLogger != nil { - in.EventLogger(baseCtx, event.SDKUploadFailed, "upload failed | reason=send_metadata", event.EventData{ + in.EventLogger(baseCtx, event.SDKUploadFailed, "Upload failed", event.EventData{ event.KeyTaskID: in.TaskId, event.KeyActionID: in.ActionID, + event.KeyReason: "send_metadata", + event.KeyError: err.Error(), }) } return nil, fmt.Errorf("failed to send metadata: %w", err) @@ -237,9 +260,11 @@ func (a *cascadeAdapter) CascadeSupernodeRegister(ctx context.Context, in *Casca if err := stream.CloseSend(); err != nil { a.logger.Error(ctx, "Failed to close stream and receive response", "TaskId", in.TaskId, "ActionID", in.ActionID, "error", err) if in.EventLogger != nil { - in.EventLogger(baseCtx, event.SDKUploadFailed, "upload failed | reason=close_send", event.EventData{ + in.EventLogger(baseCtx, event.SDKUploadFailed, "Upload failed", event.EventData{ event.KeyTaskID: in.TaskId, event.KeyActionID: in.ActionID, + event.KeyReason: "close_send", + event.KeyError: err.Error(), }) } return nil, fmt.Errorf("failed to receive response: %w", err) @@ -258,16 +283,22 @@ func (a *cascadeAdapter) CascadeSupernodeRegister(ctx context.Context, in *Casca if elapsed > 0 { avg = mb / elapsed } - in.EventLogger(baseCtx, event.SDKUploadCompleted, - fmt.Sprintf("upload complete | size=%dB chunks=%d elapsed=%.2fs avg=%.2fMB/s", totalBytes, chunkIndex, elapsed, avg), - event.EventData{event.KeyTaskID: in.TaskId, event.KeyActionID: in.ActionID}) + in.EventLogger(baseCtx, event.SDKUploadCompleted, "Upload completed", + event.EventData{ + event.KeyTaskID: in.TaskId, + event.KeyActionID: in.ActionID, + event.KeyBytesTotal: totalBytes, + event.KeyChunks: chunkIndex, + event.KeyElapsedSeconds: elapsed, + event.KeyThroughputMBS: avg, + }) } // Processing phase timer starts now (waiting for server streamed responses) processingTimer := time.AfterFunc(cascadeProcessingTimeout, func() { a.logger.Error(baseCtx, "Processing phase timeout reached; cancelling stream") if in.EventLogger != nil { - in.EventLogger(baseCtx, event.SDKProcessingTimeout, "processing timeout", event.EventData{event.KeyTaskID: in.TaskId, event.KeyActionID: in.ActionID}) + in.EventLogger(baseCtx, event.SDKProcessingTimeout, "Processing timed out", event.EventData{event.KeyTaskID: in.TaskId, event.KeyActionID: in.ActionID}) } cancel() }) @@ -279,7 +310,7 @@ func (a *cascadeAdapter) CascadeSupernodeRegister(ctx context.Context, in *Casca // Emit processing started if in.EventLogger != nil { - in.EventLogger(baseCtx, event.SDKProcessingStarted, "processing started | awaiting server progress", event.EventData{event.KeyTaskID: in.TaskId, event.KeyActionID: in.ActionID}) + in.EventLogger(baseCtx, event.SDKProcessingStarted, "Processing started", event.EventData{event.KeyTaskID: in.TaskId, event.KeyActionID: in.ActionID}) } // Handle streaming responses from supernode @@ -298,9 +329,8 @@ func (a *cascadeAdapter) CascadeSupernodeRegister(ctx context.Context, in *Casca } } if in.EventLogger != nil { - in.EventLogger(baseCtx, event.SDKProcessingFailed, - fmt.Sprintf("processing failed | reason=stream_recv error=%v", err), - event.EventData{event.KeyTaskID: in.TaskId, event.KeyActionID: in.ActionID}) + in.EventLogger(baseCtx, event.SDKProcessingFailed, "Processing failed", + event.EventData{event.KeyTaskID: in.TaskId, event.KeyActionID: in.ActionID, event.KeyError: err.Error()}) } return nil, fmt.Errorf("failed to receive server response: %w", err) } @@ -337,7 +367,7 @@ func (a *cascadeAdapter) CascadeSupernodeRegister(ctx context.Context, in *Casca return nil, fmt.Errorf("processing timed out or cancelled before final response: %w", phaseCtx.Err()) } if in.EventLogger != nil { - in.EventLogger(baseCtx, event.SDKProcessingFailed, "processing failed | reason=missing_final_response", event.EventData{event.KeyTaskID: in.TaskId, event.KeyActionID: in.ActionID}) + in.EventLogger(baseCtx, event.SDKProcessingFailed, "Processing failed: missing final response", event.EventData{event.KeyTaskID: in.TaskId, event.KeyActionID: in.ActionID}) } return nil, fmt.Errorf("no final response with tx_hash received") } @@ -365,19 +395,15 @@ func (a *cascadeAdapter) GetSupernodeStatus(ctx context.Context) (SupernodeStatu // CascadeSupernodeDownload downloads a file from a supernode gRPC stream func (a *cascadeAdapter) CascadeSupernodeDownload( - ctx context.Context, - in *CascadeSupernodeDownloadRequest, - opts ...grpc.CallOption, + ctx context.Context, + in *CascadeSupernodeDownloadRequest, + opts ...grpc.CallOption, ) (*CascadeSupernodeDownloadResponse, error) { - // Use provided context as-is (no correlation IDs). Add watchdogs: - // - idle timer: reset on every received message (event or chunk). - // - max timer: hard cap for one attempt. - phaseCtx, phaseCancel := context.WithCancel(ctx) - defer phaseCancel() + // Use provided context as-is (no correlation IDs) // 1. Open gRPC stream (server-stream) - stream, err := a.client.Download(phaseCtx, &cascade.DownloadRequest{ + stream, err := a.client.Download(ctx, &cascade.DownloadRequest{ ActionId: in.ActionID, Signature: in.Signature, }, opts...) @@ -400,27 +426,11 @@ func (a *cascadeAdapter) CascadeSupernodeDownload( } defer outFile.Close() - var ( - bytesWritten int64 - chunkIndex int - ) - - // 3. Receive streamed responses with liveness watchdog - // Start with a generous prep idle timeout; tighten after first message - currentIdle := downloadPrepIdleTimeout - idleTimer := time.AfterFunc(currentIdle, func() { - a.logger.Error(ctx, "download idle timeout; cancelling stream", "action_id", in.ActionID) - phaseCancel() - }) - defer idleTimer.Stop() - maxTimer := time.AfterFunc(downloadMaxTimeout, func() { - a.logger.Error(ctx, "download max timeout; cancelling stream", "action_id", in.ActionID) - phaseCancel() - }) - defer maxTimer.Stop() - start := time.Now() - lastActivity := start - firstMsg := false + var ( + bytesWritten int64 + chunkIndex int + startedEmitted bool + ) // 3. Receive streamed responses for { @@ -429,17 +439,6 @@ func (a *cascadeAdapter) CascadeSupernodeDownload( break } if err != nil { - // Classify timeouts for clearer upstream handling - if phaseCtx.Err() != nil { - sinceLast := time.Since(lastActivity) - sinceStart := time.Since(start) - switch { - case sinceLast >= downloadIdleTimeout: - return nil, fmt.Errorf("download idle timeout: %w", context.DeadlineExceeded) - case sinceStart >= downloadMaxTimeout: - return nil, fmt.Errorf("download overall timeout: %w", context.DeadlineExceeded) - } - } return nil, fmt.Errorf("stream recv: %w", err) } @@ -447,71 +446,54 @@ func (a *cascadeAdapter) CascadeSupernodeDownload( // 3a. Progress / event message case *cascade.DownloadResponse_Event: - // On first message, tighten idle window for active transfer - if !firstMsg { - firstMsg = true - currentIdle = downloadIdleTimeout - } - if idleTimer != nil { - idleTimer.Reset(currentIdle) - } - lastActivity = time.Now() a.logger.Info(ctx, "supernode event", "event_type", x.Event.EventType, "message", x.Event.Message, "action_id", in.ActionID) if in.EventLogger != nil { in.EventLogger(ctx, toSdkEvent(x.Event.EventType), x.Event.Message, event.EventData{ - event.KeyTaskID: in.TaskID, event.KeyActionID: in.ActionID, event.KeyEventType: x.Event.EventType, event.KeyMessage: x.Event.Message, }) } - // 3b. Actual data chunk - case *cascade.DownloadResponse_Chunk: - data := x.Chunk.Data - if len(data) == 0 { - // Treat empty chunks as keep-alive; reset idle - if !firstMsg { - firstMsg = true - currentIdle = downloadIdleTimeout - } - if idleTimer != nil { - idleTimer.Reset(currentIdle) - } - lastActivity = time.Now() - continue - } - if _, err := outFile.Write(data); err != nil { - return nil, fmt.Errorf("write chunk: %w", err) - } + // 3b. Actual data chunk + case *cascade.DownloadResponse_Chunk: + data := x.Chunk.Data + if len(data) == 0 { + continue + } + if !startedEmitted { + if in.EventLogger != nil { + in.EventLogger(ctx, event.SDKDownloadStarted, "Download started", event.EventData{event.KeyActionID: in.ActionID}) + } + startedEmitted = true + } + if _, err := outFile.Write(data); err != nil { + return nil, fmt.Errorf("write chunk: %w", err) + } bytesWritten += int64(len(data)) chunkIndex++ - if !firstMsg { - firstMsg = true - currentIdle = downloadIdleTimeout - } - if idleTimer != nil { - idleTimer.Reset(currentIdle) - } - lastActivity = time.Now() + a.logger.Debug(ctx, "received chunk", "chunk_index", chunkIndex, "chunk_size", len(data), "bytes_written", bytesWritten) } } a.logger.Info(ctx, "download complete", "bytes_written", bytesWritten, "path", in.OutputPath, "action_id", in.ActionID) - return &CascadeSupernodeDownloadResponse{ - Success: true, - Message: "artefact downloaded", - OutputPath: in.OutputPath, - }, nil + if in.EventLogger != nil { + in.EventLogger(ctx, event.SDKDownloadCompleted, "Download completed", event.EventData{event.KeyActionID: in.ActionID, event.KeyOutputPath: in.OutputPath}) + } + return &CascadeSupernodeDownloadResponse{ + Success: true, + Message: "artefact downloaded", + OutputPath: in.OutputPath, + }, nil } // toSdkEvent converts a supernode-side enum value into an internal SDK EventType. func toSdkEvent(e cascade.SupernodeEventType) event.EventType { - switch e { + switch e { case cascade.SupernodeEventType_ACTION_RETRIEVED: return event.SupernodeActionRetrieved case cascade.SupernodeEventType_ACTION_FEE_VERIFIED: @@ -534,8 +516,14 @@ func toSdkEvent(e cascade.SupernodeEventType) event.EventType { return event.SupernodeArtefactsStored case cascade.SupernodeEventType_ACTION_FINALIZED: return event.SupernodeActionFinalized - case cascade.SupernodeEventType_ARTEFACTS_DOWNLOADED: - return event.SupernodeArtefactsDownloaded + case cascade.SupernodeEventType_ARTEFACTS_DOWNLOADED: + return event.SupernodeArtefactsDownloaded + case cascade.SupernodeEventType_NETWORK_RETRIEVE_STARTED: + return event.SupernodeNetworkRetrieveStarted + case cascade.SupernodeEventType_DECODE_COMPLETED: + return event.SupernodeDecodeCompleted + case cascade.SupernodeEventType_SERVE_READY: + return event.SupernodeServeReady case cascade.SupernodeEventType_FINALIZE_SIMULATED: return event.SupernodeFinalizeSimulated case cascade.SupernodeEventType_FINALIZE_SIMULATION_FAILED: diff --git a/sdk/adapters/supernodeservice/timeouts.go b/sdk/adapters/supernodeservice/timeouts.go index 884ca83b..83311185 100644 --- a/sdk/adapters/supernodeservice/timeouts.go +++ b/sdk/adapters/supernodeservice/timeouts.go @@ -9,16 +9,3 @@ const cascadeUploadTimeout = 60 * time.Minute // cascadeProcessingTimeout bounds the time waiting for server-side processing // and final response (e.g., tx hash) after upload completes. const cascadeProcessingTimeout = 10 * time.Minute - -// Download timeouts (adapter-level) -// - downloadPrepIdleTimeout: idle window before the first message arrives, -// allowing the server time to prepare (e.g., reconstruct large files). -// - downloadIdleTimeout: cancels if no messages (events/chunks) are received -// after transfer begins; protects against stalls while allowing long transfers. -// - downloadMaxTimeout: hard cap for a single download attempt. -const ( - // Give server prep up to ~5m + cushion without client cancelling. - downloadPrepIdleTimeout = 6 * time.Minute - downloadIdleTimeout = 2 * time.Minute - downloadMaxTimeout = 60 * time.Minute -) diff --git a/sdk/docs/cascade-timeouts.md b/sdk/docs/cascade-timeouts.md index 8a2af30e..716804bc 100644 --- a/sdk/docs/cascade-timeouts.md +++ b/sdk/docs/cascade-timeouts.md @@ -1,60 +1,134 @@ -# Cascade Timeouts — Quick Guide +# Cascade Registration Timeouts and Networking + +This document explains how timeouts and deadlines are applied across the SDK cascade registration flow, including the current split between upload and processing phases and the relevant client/server defaults. + +## Purpose + +- Make slow, user‑network–dependent uploads more tolerant without impacting other stages. +- Keep health checks and connection establishment responsive. +- Enable clearer error categorization: upload vs processing. + +## TL;DR Defaults + +- Upload timeout (adapter): `cascadeUploadTimeout = 60m` — covers client-side file streaming to the supernode. +- Processing timeout (adapter): `cascadeProcessingTimeout = 10m` — covers waiting for server progress/final tx hash after upload completes. +- Health check to supernodes (task): `connectionTimeout = 10s` — per-node probe during discovery. +- gRPC connect (client): + - Adds a default `30s` deadline if caller context has none. + - Connection readiness gate: `ConnWaitTime = 10s` per attempt, with `MaxRetries = 3` and retry backoff. +- ALTS handshake (secure transport): `30s` internal read timeouts (client and server sides). +- Supernode gRPC server: + - No per‑RPC timeout for `Register`/`Download` handlers. + - Keepalive is permissive (idle ping at 1h, ping ack timeout 30m). + - Stream tuning: 16MB message caps, 16MB stream window, 160MB conn window, ~20 concurrent streams. -Concise overview of timeout locations, defaults, and intent. +## Control Flow and Contexts -## Defaults +1) `sdk/action/client.go: ClientImpl.StartCascade(ctx, ...)` + - Forwards `ctx` to Task Manager. -- Register (client → server) - - Upload (SDK adapter): 60m — `cascadeUploadTimeout` - - Processing (SDK adapter): 10m — `cascadeProcessingTimeout` - - Server envelope: 75m — `RegisterTimeout` +2) `sdk/task/manager.go: ManagerImpl.CreateCascadeTask(...)` + - Detaches from caller: `taskCtx := context.WithCancel(context.Background())`. + - All subsequent work uses `taskCtx` (no deadline by default). -- Download (server → client) - - Server preparation: 5m — `DownloadPrepareTimeout` - - Client per‑attempt: 60m — `downloadTimeout` - - Client liveness (SDK adapter): - - Prep idle (pre‑first‑message): 6m — `downloadPrepIdleTimeout` - - Idle (post‑first‑message): 2m — `downloadIdleTimeout` - - Max attempt: 60m — `downloadMaxTimeout` - - Note: File streaming is not server‑bounded; the client governs transfer. +3) `sdk/task/cascade.go: CascadeTask.Run(ctx)` + - Validates file size; fetches healthy supernodes; registers with one. -- Discovery / Connect - - Health probe per supernode: 10s — `connectionTimeout` - - gRPC connect default: 30s if caller provides no deadline - - Keepalives: permissive (idle ping ~1h, ack timeout ~30m) +4) Discovery: `sdk/task/task.go: BaseTask.fetchSupernodes` → `BaseTask.isServing` + - `context.WithTimeout(parent, 10s)` for health probe (create client + `HealthCheck`). -## Intent and Ordering +5) Registration attempt: `sdk/task/cascade.go: attemptRegistration` + - Client connect: uses task context (no deadline); gRPC injects a 30s default at connect if needed. + - No outer registration timeout here; the adapter handles per‑phase timers. -- Register: server envelope (75m) > SDK phases (60m + 10m) so the client surfaces errors first when appropriate. -- Download: server prep is tight (5m). Transfer is governed by the client with a generous per‑attempt window and two‑phase idle watchdogs. +6) RPC staging: + - `sdk/net/impl.go: supernodeClient.RegisterCascade` → + - `sdk/adapters/supernodeservice/adapter.go: CascadeSupernodeRegister` performs client‑stream upload and reads server progress / final tx hash. -## Where They Live +## Where Timeouts Come From (by Layer) -- SDK adapter (upload/download phases): `sdk/adapters/supernodeservice/timeouts.go` -- SDK task (discovery, per‑attempt download): `sdk/task/timeouts.go` -- Supernode service (server envelopes): `supernode/services/cascade/timeouts.go` -- P2P internal RPCs: `p2p/kademlia/network.go` (fixed per‑message timeouts) +- SDK adapter level (registration RPC): + - `cascadeUploadTimeout` (60m): upload phase timer (file chunks + metadata + CloseSend). + - `cascadeProcessingTimeout` (10m): processing phase timer (receive server progress + final tx hash). +- SDK task level: + - `connectionTimeout` (10s): supernode health checks only. -## Notes +- gRPC client (`pkg/net/grpc/client`): + - `defaultTimeout = 30s`: applied to connect if context lacks a deadline. + - `ConnWaitTime = 10s`, `MaxRetries = 3`, backoff configured; keepalives: 30m/30m. -- Health checks use a 10s budget for snappy discovery. +- ALTS handshake (`pkg/net/credentials/alts/handshake`): + - `defaultTimeout = 30s` for handshake read operations (client/server). + +- gRPC server (`pkg/net/grpc/server` and supernode runtime): + - No explicit per‑RPC timeouts; generous keepalives; tuned flow control and message sizes for 4MB chunks. + +## SDK Constants + +Timeout constants are defined in dedicated files for clarity: + +- Upload/Processing: `supernode/sdk/adapters/supernodeservice/timeouts.go` +- Connection/health probe: `supernode/sdk/task/timeouts.go` + +Notes: +- `BaseTask.isServing` keeps a short 10s budget for snappy health checks. - gRPC connect/handshake defaults remain unchanged. -## Events (SDK) -- Upload timeout → `SDKUploadFailed` -- Processing timeout → `SDKProcessingTimeout` -- Download failure (timeout/canceled) → `SDKDownloadFailure` +## Implementation Details + +The split is implemented inside `CascadeSupernodeRegister` where the phases are naturally separated by the client‑stream CloseSend. + +1) Create a cancelable context from the inbound one for the stream lifetime: + +```go +phaseCtx, cancel := context.WithCancel(ctx) +defer cancel() +stream, err := a.client.Register(phaseCtx, opts...) +``` + +2) Upload phase timer: + +```go +uploadTimer := time.AfterFunc(cascadeUploadTimeout, cancel) + +// send chunks... +// send metadata... + +if err := stream.CloseSend(); err != nil { /* ... */ } +uploadTimer.Stop() +``` + +3) Processing phase timer (server progress → final tx hash): + +```go +processingTimer := time.AfterFunc(cascadeProcessingTimeout, cancel) +defer processingTimer.Stop() + +for { + resp, err := stream.Recv() + // handle EOF, errors, progress, final tx hash +} +``` + +4) Error mapping and events: +- If cancellation occurs during Send loop → classify as upload timeout and emit `SDKUploadTimeout`. +- If cancellation occurs during Recv loop → classify as processing timeout and emit `SDKProcessingTimeout`. +- Surface distinct error messages and publish events accordingly. This approach requires no request‑struct changes and preserves existing call sites. It uses a single cancelable context across both phases and phase‑specific timers. -## Minimal Tuning Guidance -- Slow client links: keep download attempt at 60m; adjust idle windows if needed. -- Very large inputs: raise `cascadeUploadTimeout` (keep processing modest at 10m). +## Additional Notes + +- Health checks use `connectionTimeout = 10s` during supernode discovery. +- gRPC client connect behavior: adds a `30s` deadline if none is present, waits up to `ConnWaitTime = 10s` per attempt with retries. +- Downloads use a separate `downloadTimeout = 5m` envelope. + +## Operational Guidance -## Reference Map -- SDK: `sdk/task/timeouts.go`, `sdk/adapters/supernodeservice/timeouts.go`, `sdk/adapters/supernodeservice/adapter.go` -- Server: `supernode/services/cascade/timeouts.go`, server handlers in `supernode/node/action/server/cascade` -- Network: `pkg/net/grpc/client`, `p2p/kademlia/network.go` +- For slow client links: raise `cascadeUploadTimeout` (e.g., 30–120m). Keep processing modest (e.g., 5–10m) unless chain finalization is known to stall. +- Server tuning is already generous; no server change required to support longer uploads. +- Telemetry: differentiate upload vs processing timeout in logs and emitted events for better retry behavior and user messaging. +- Retry policy: on upload timeout, prefer retrying with a different supernode; on processing timeout, consider whether the server might still finalize (idempotency depends on service semantics). ## File/Code Reference Map @@ -71,12 +145,11 @@ This approach requires no request‑struct changes and preserves existing call s - Supernode - `supernode/supernode/node/supernode/server/server.go` — server options (16MB caps, windows, 20 streams). - - `supernode/supernode/node/action/server/cascade/cascade_action_server.go` — server-side handlers. - - `supernode/supernode/services/cascade/timeouts.go` — Register (`RegisterTimeout = 75m`) and Download prep (`DownloadPrepareTimeout = 5m`) timeouts. + - `supernode/supernode/node/action/server/cascade/cascade_action_server.go` — server-side Register/Download handlers (no per‑RPC timeout). ## Events -- Upload phase timeout: classified as `SDKUploadFailed` with message suffix `| reason=timeout`. +- Upload phase timeout: `SDKUploadTimeout`. - Processing phase timeout: `SDKProcessingTimeout`. # Cascade Registration Timeouts and Networking @@ -87,12 +160,10 @@ This document describes how the SDK applies timeouts and deadlines during cascad - Upload (adapter): `cascadeUploadTimeout = 60m` — client-side streaming of file chunks and metadata. - Processing (adapter): `cascadeProcessingTimeout = 10m` — wait for server progress and final tx hash after upload completes. - Discovery (task): `connectionTimeout = 10s` — per-supernode health probe during discovery. -- Download (task): `downloadTimeout = 60m` — per-attempt envelope. Adapter adds - `downloadPrepIdleTimeout = 6m` (pre-first-message), `downloadIdleTimeout = 2m` - (post-first-message), and `downloadMaxTimeout = 60m`. +- Download (task): `downloadTimeout = 5m` — envelope for cascade download. - gRPC client connect: adds a `30s` deadline if none is present; readiness wait per attempt `ConnWaitTime = 10s` with retries and backoff. - ALTS handshake: internal `30s` read timeouts on both client and server sides. -- Supernode gRPC server: task-level timeouts are applied (Register 75m). Download preparation is bounded to 5m; file streaming is client-governed. Keepalive is permissive (idle ping ~1h, ack timeout ~30m); flow-control and message-size tuning supports 4MB chunks. +- Supernode gRPC server: no per-RPC timeout; keepalive is permissive (idle ping ~1h, ack timeout ~30m); flow-control and message-size tuning supports 4MB chunks. ## Control Flow @@ -107,7 +178,7 @@ This document describes how the SDK applies timeouts and deadlines during cascad ## Events -- Upload phase timeout — emitted as `SDKUploadFailed` with `reason=timeout` in the message and `KeyMessage = "timeout"`. +- `SDKUploadTimeout` — emitted when the upload phase exceeds its time budget. - `SDKProcessingTimeout` — emitted when the post-upload processing exceeds its time budget. ## Files and Constants diff --git a/sdk/event/keys.go b/sdk/event/keys.go index c0e24cc8..4a6f8eaa 100644 --- a/sdk/event/keys.go +++ b/sdk/event/keys.go @@ -17,6 +17,16 @@ const ( KeyOutputPath EventDataKey = "output_path" KeySuccessRate EventDataKey = "success_rate" + // Upload/download metrics keys (no progress events; start/complete metrics only) + KeyBytesTotal EventDataKey = "bytes_total" + KeyChunkSize EventDataKey = "chunk_size" + KeyEstChunks EventDataKey = "est_chunks" + KeyChunks EventDataKey = "chunks" + KeyElapsedSeconds EventDataKey = "elapsed_seconds" + KeyThroughputMBS EventDataKey = "throughput_mb_s" + KeyChunkIndex EventDataKey = "chunk_index" + KeyReason EventDataKey = "reason" + // Task specific keys KeyTaskID EventDataKey = "task_id" KeyActionID EventDataKey = "action_id" diff --git a/sdk/event/types.go b/sdk/event/types.go index 2f7be099..635b1e2f 100644 --- a/sdk/event/types.go +++ b/sdk/event/types.go @@ -32,10 +32,10 @@ const ( SDKProcessingFailed EventType = "sdk:processing_failed" SDKProcessingTimeout EventType = "sdk:processing_timeout" - SDKDownloadAttempt EventType = "sdk:download_attempt" - SDKDownloadFailure EventType = "sdk:download_failure" - SDKOutputPathReceived EventType = "sdk:output_path_received" - SDKDownloadSuccessful EventType = "sdk:download_successful" + SDKDownloadAttempt EventType = "sdk:download_attempt" + SDKDownloadFailure EventType = "sdk:download_failure" + SDKDownloadStarted EventType = "sdk:download_started" + SDKDownloadCompleted EventType = "sdk:download_completed" ) const ( @@ -52,6 +52,9 @@ const ( SupernodeArtefactsStored EventType = "supernode:artefacts_stored" SupernodeActionFinalized EventType = "supernode:action_finalized" SupernodeArtefactsDownloaded EventType = "supernode:artefacts_downloaded" + SupernodeNetworkRetrieveStarted EventType = "supernode:network_retrieve_started" + SupernodeDecodeCompleted EventType = "supernode:decode_completed" + SupernodeServeReady EventType = "supernode:serve_ready" SupernodeUnknown EventType = "supernode:unknown" SupernodeFinalizeSimulationFailed EventType = "supernode:finalize_simulation_failed" ) diff --git a/sdk/net/impl.go b/sdk/net/impl.go index f09f8649..ab0f7b28 100644 --- a/sdk/net/impl.go +++ b/sdk/net/impl.go @@ -140,12 +140,12 @@ func (c *supernodeClient) GetSupernodeStatus(ctx context.Context) (*supernodeser // Download downloads the cascade action file func (c *supernodeClient) Download(ctx context.Context, in *supernodeservice.CascadeSupernodeDownloadRequest, opts ...grpc.CallOption) (*supernodeservice.CascadeSupernodeDownloadResponse, error) { - resp, err := c.cascadeClient.CascadeSupernodeDownload(ctx, in, opts...) - if err != nil { - return nil, fmt.Errorf("download failed: %w", err) - } + resp, err := c.cascadeClient.CascadeSupernodeDownload(ctx, in, opts...) + if err != nil { + return nil, fmt.Errorf("get artefacts failed: %w", err) + } - return resp, nil + return resp, nil } // Close closes the connection to the supernode diff --git a/sdk/task/cascade.go b/sdk/task/cascade.go index 3eb0b057..cabb0db1 100644 --- a/sdk/task/cascade.go +++ b/sdk/task/cascade.go @@ -110,7 +110,7 @@ func (t *CascadeTask) attemptRegistration(ctx context.Context, _ int, sn lumera. defer client.Close(ctx) // Emit connection established event for observability - t.LogEvent(ctx, event.SDKConnectionEstablished, "connection established", event.EventData{ + t.LogEvent(ctx, event.SDKConnectionEstablished, "Connection to supernode established", event.EventData{ event.KeySupernode: sn.GrpcEndpoint, event.KeySupernodeAddress: sn.CosmosAddress, }) @@ -127,11 +127,10 @@ func (t *CascadeTask) attemptRegistration(ctx context.Context, _ int, sn lumera. return fmt.Errorf("upload rejected by %s: %s", sn.CosmosAddress, resp.Message) } - t.LogEvent(ctx, event.SDKTaskTxHashReceived, "txhash received", event.EventData{ - event.KeyTxHash: resp.TxHash, - event.KeySupernode: sn.GrpcEndpoint, - event.KeySupernodeAddress: sn.CosmosAddress, - }) + t.LogEvent(ctx, event.SDKTaskTxHashReceived, "txhash received", event.EventData{ + event.KeyTxHash: resp.TxHash, + event.KeySupernode: sn.CosmosAddress, + }) return nil } diff --git a/sdk/task/download.go b/sdk/task/download.go index 7687503d..3e85007a 100644 --- a/sdk/task/download.go +++ b/sdk/task/download.go @@ -2,10 +2,10 @@ package task import ( "context" - stderrors "errors" "fmt" "os" - "path/filepath" + "sort" + "time" "github.com/LumeraProtocol/supernode/v2/sdk/adapters/lumera" "github.com/LumeraProtocol/supernode/v2/sdk/adapters/supernodeservice" @@ -13,6 +13,11 @@ import ( "github.com/LumeraProtocol/supernode/v2/sdk/net" ) +// timeouts +const ( + downloadTimeout = 15 * time.Minute +) + type CascadeDownloadTask struct { BaseTask actionId string @@ -65,41 +70,88 @@ func (t *CascadeDownloadTask) downloadFromSupernodes(ctx context.Context, supern Signature: t.signature, } - // Process supernodes in pairs - var allErrors []error - for i := 0; i < len(supernodes); i += 2 { - // Determine how many supernodes to try in this batch (1 or 2) - batchSize := 2 - if i+1 >= len(supernodes) { - batchSize = 1 + // Remove existing file once before starting attempts to allow overwrite + if _, err := os.Stat(req.OutputPath); err == nil { + if removeErr := os.Remove(req.OutputPath); removeErr != nil { + return fmt.Errorf("failed to remove existing file %s: %w", req.OutputPath, removeErr) } + } - t.logger.Info(ctx, "attempting concurrent download from supernode batch", "batch_start", i, "batch_size", batchSize) + // Optionally rank supernodes by available memory to improve success for large files + // We keep a short timeout per status fetch to avoid delaying downloads. + type rankedSN struct { + sn lumera.Supernode + availGB float64 + hasStatus bool + } + ranked := make([]rankedSN, 0, len(supernodes)) + for _, sn := range supernodes { + ranked = append(ranked, rankedSN{sn: sn}) + } - // Try downloading from this batch concurrently - result, batchErrors := t.attemptConcurrentDownload(ctx, supernodes[i:i+batchSize], clientFactory, req, i) + // Probe supernode status with short timeouts and close clients promptly + for i := range ranked { + sn := ranked[i].sn + // 2s status timeout to keep this pass fast + stx, cancel := context.WithTimeout(ctx, 2*time.Second) + client, err := clientFactory.CreateClient(stx, sn) + if err != nil { + cancel() + continue + } + status, err := client.GetSupernodeStatus(stx) + _ = client.Close(stx) + cancel() + if err != nil { + continue + } + ranked[i].hasStatus = true + ranked[i].availGB = status.Resources.Memory.AvailableGB + } - if result != nil { - // Success! Log and return (include final output path) - t.LogEvent(ctx, event.SDKDownloadSuccessful, "download successful", event.EventData{ - event.KeySupernode: result.SupernodeEndpoint, - event.KeySupernodeAddress: result.SupernodeAddress, - event.KeyIteration: result.Iteration, - event.KeyOutputPath: t.outputPath, - }) - return nil + // Sort: nodes with status first, higher available memory first + sort.Slice(ranked, func(i, j int) bool { + if ranked[i].hasStatus != ranked[j].hasStatus { + return ranked[i].hasStatus && !ranked[j].hasStatus } + return ranked[i].availGB > ranked[j].availGB + }) + + // Rebuild the supernodes list in the sorted order + for i := range ranked { + supernodes[i] = ranked[i].sn + } + + // Try supernodes sequentially, one by one (now sorted) + var lastErr error + for idx, sn := range supernodes { + iteration := idx + 1 + + // Log download attempt + t.LogEvent(ctx, event.SDKDownloadAttempt, "attempting download from super-node", event.EventData{ + event.KeySupernode: sn.GrpcEndpoint, + event.KeySupernodeAddress: sn.CosmosAddress, + event.KeyIteration: iteration, + }) - // Both (or the single one) failed, collect errors - allErrors = append(allErrors, batchErrors...) + if err := t.attemptDownload(ctx, sn, clientFactory, req); err != nil { + // Log failure and continue to next supernode + t.LogEvent(ctx, event.SDKDownloadFailure, "download from super-node failed", event.EventData{ + event.KeySupernode: sn.GrpcEndpoint, + event.KeySupernodeAddress: sn.CosmosAddress, + event.KeyIteration: iteration, + event.KeyError: err.Error(), + }) + lastErr = err + continue + } - // Log batch failure - t.logger.Warn(ctx, "download batch failed", "batch_start", i, "batch_size", batchSize, "errors", len(batchErrors)) + // Success; return to caller + return nil } - // All attempts failed - if len(allErrors) > 0 { - return fmt.Errorf("failed to download from all super-nodes: %v", allErrors) + if lastErr != nil { + return fmt.Errorf("failed to download from all super-nodes: %w", lastErr) } return fmt.Errorf("no supernodes available for download") } @@ -120,31 +172,19 @@ func (t *CascadeDownloadTask) attemptDownload( } defer client.Close(ctx) - // Emit connection established for observability (parity with StartCascade) - t.LogEvent(ctx, event.SDKConnectionEstablished, "connection established", event.EventData{ - event.KeySupernode: sn.GrpcEndpoint, - event.KeySupernodeAddress: sn.CosmosAddress, - }) - req.EventLogger = func(ctx context.Context, evt event.EventType, msg string, data event.EventData) { t.LogEvent(ctx, evt, msg, data) } - resp, err := client.Download(ctx, req) - if err != nil { - return fmt.Errorf("download from %s: %w", sn.CosmosAddress, err) - } - if !resp.Success { - return fmt.Errorf("download rejected by %s: %s", sn.CosmosAddress, resp.Message) - } + resp, err := client.Download(ctx, req) + if err != nil { + return fmt.Errorf("download from %s: %w", sn.CosmosAddress, err) + } + if !resp.Success { + return fmt.Errorf("download rejected by %s: %s", sn.CosmosAddress, resp.Message) + } - t.LogEvent(ctx, event.SDKOutputPathReceived, "file downloaded", event.EventData{ - event.KeyOutputPath: resp.OutputPath, - event.KeySupernode: sn.GrpcEndpoint, - event.KeySupernodeAddress: sn.CosmosAddress, - }) - - return nil + return nil } // downloadResult holds the result of a successful download attempt @@ -152,7 +192,6 @@ type downloadResult struct { SupernodeAddress string SupernodeEndpoint string Iteration int - TempPath string } // attemptConcurrentDownload tries to download from multiple supernodes concurrently @@ -164,11 +203,10 @@ func (t *CascadeDownloadTask) attemptConcurrentDownload( req *supernodeservice.CascadeSupernodeDownloadRequest, baseIteration int, ) (*downloadResult, []error) { - // Remove existing final file if it exists to allow overwrite (once per batch) - finalOutputPath := req.OutputPath - if _, err := os.Stat(finalOutputPath); err == nil { - if removeErr := os.Remove(finalOutputPath); removeErr != nil { - return nil, []error{fmt.Errorf("failed to remove existing file %s: %w", finalOutputPath, removeErr)} + // Remove existing file if it exists to allow overwrite (do this once before concurrent attempts) + if _, err := os.Stat(req.OutputPath); err == nil { + if removeErr := os.Remove(req.OutputPath); removeErr != nil { + return nil, []error{fmt.Errorf("failed to remove existing file %s: %w", req.OutputPath, removeErr)} } } @@ -184,9 +222,6 @@ func (t *CascadeDownloadTask) attemptConcurrentDownload( } resultCh := make(chan attemptResult, len(batch)) - // Track per-attempt temporary output paths for safe concurrent writes - tmpPaths := make([]string, len(batch)) - // Start concurrent download attempts for idx, sn := range batch { iteration := baseIteration + idx + 1 @@ -201,18 +236,14 @@ func (t *CascadeDownloadTask) attemptConcurrentDownload( go func(sn lumera.Supernode, idx int, iter int) { // Create a copy of the request for this goroutine reqCopy := &supernodeservice.CascadeSupernodeDownloadRequest{ - ActionID: req.ActionID, - TaskID: req.TaskID, - // Use a unique temporary path per attempt to avoid concurrent writes - OutputPath: filepath.Join(filepath.Dir(finalOutputPath), fmt.Sprintf(".%s.part.%d", filepath.Base(finalOutputPath), idx)), + ActionID: req.ActionID, + TaskID: req.TaskID, + OutputPath: req.OutputPath, Signature: req.Signature, } - tmpPaths[idx] = reqCopy.OutputPath err := t.attemptDownload(batchCtx, sn, factory, reqCopy) if err != nil { - // Best-effort cleanup of the partial file for this attempt - _ = os.Remove(reqCopy.OutputPath) resultCh <- attemptResult{ err: err, idx: idx, @@ -225,7 +256,6 @@ func (t *CascadeDownloadTask) attemptConcurrentDownload( SupernodeAddress: sn.CosmosAddress, SupernodeEndpoint: sn.GrpcEndpoint, Iteration: iter, - TempPath: reqCopy.OutputPath, }, idx: idx, } @@ -233,24 +263,11 @@ func (t *CascadeDownloadTask) attemptConcurrentDownload( } // Collect results - var errs []error - for i := 0; i < len(batch); i++ { + var errors []error + for i := range len(batch) { select { case result := <-resultCh: if result.success != nil { - // Attempt to move the temp file to the final destination atomically - if err := os.Rename(result.success.TempPath, finalOutputPath); err != nil { - // Treat rename failure as a batch error - cancelBatch() - // Drain remaining results to avoid goroutine leaks - go func() { - for j := i + 1; j < len(batch); j++ { - <-resultCh - } - }() - return nil, []error{fmt.Errorf("finalize download (rename) failed: %w", err)} - } - // Success! Cancel other attempts and return cancelBatch() // Drain remaining results to avoid goroutine leaks @@ -264,23 +281,13 @@ func (t *CascadeDownloadTask) attemptConcurrentDownload( // Log failure sn := batch[result.idx] - // Classify failure reason when possible - data := event.EventData{ + t.LogEvent(ctx, event.SDKDownloadFailure, "download from super-node failed", event.EventData{ event.KeySupernode: sn.GrpcEndpoint, event.KeySupernodeAddress: sn.CosmosAddress, event.KeyIteration: baseIteration + result.idx + 1, event.KeyError: result.err.Error(), - } - msg := "download from super-node failed" - if stderrors.Is(result.err, context.DeadlineExceeded) { - data[event.KeyMessage] = "timeout" - msg += " | reason=timeout" - } else if stderrors.Is(result.err, context.Canceled) { - data[event.KeyMessage] = "canceled" - msg += " | reason=canceled" - } - t.LogEvent(ctx, event.SDKDownloadFailure, msg, data) - errs = append(errs, result.err) + }) + errors = append(errors, result.err) case <-ctx.Done(): return nil, []error{ctx.Err()} @@ -288,5 +295,5 @@ func (t *CascadeDownloadTask) attemptConcurrentDownload( } // All attempts in this batch failed - return nil, errs + return nil, errors } diff --git a/sdk/task/timeouts.go b/sdk/task/timeouts.go index 62ae733d..f6e1e7e6 100644 --- a/sdk/task/timeouts.go +++ b/sdk/task/timeouts.go @@ -2,22 +2,7 @@ package task import "time" -// Connection and health check timeouts -const ( - // connectionTimeout bounds supernode health/connection probing. - // Keep this short to preserve snappy discovery without impacting long uploads. - connectionTimeout = 10 * time.Second -) +// connectionTimeout bounds supernode health/connection probing. +// Keep this short to preserve snappy discovery without impacting long uploads. +const connectionTimeout = 10 * time.Second -// Task execution timeouts -const ( - // downloadTimeout bounds a single download attempt at the task layer. - // This should exceed typical slow-client scenarios; fine-grained - // liveness is enforced by the adapter via an idle timeout. - downloadTimeout = 60 * time.Minute -) - -// Note: Upload and processing timeouts are defined in sdk/adapters/supernodeservice/timeouts.go -// as they are specific to the adapter implementation: -// - cascadeUploadTimeout = 60 * time.Minute (for slow network uploads) -// - cascadeProcessingTimeout = 10 * time.Minute (for server-side processing) diff --git a/sn-manager/internal/updater/updater.go b/sn-manager/internal/updater/updater.go index b0e2f1ea..5bf650c1 100644 --- a/sn-manager/internal/updater/updater.go +++ b/sn-manager/internal/updater/updater.go @@ -28,7 +28,7 @@ const ( updateCheckInterval = 10 * time.Minute // forceUpdateAfter is the age threshold after a release is published // beyond which updates are applied regardless of normal gates (idle, policy) - forceUpdateAfter = 1 * time.Hour + forceUpdateAfter = 30 * time.Minute ) type AutoUpdater struct { diff --git a/supernode/node/action/server/cascade/cascade_action_server.go b/supernode/node/action/server/cascade/cascade_action_server.go index e2fe9c08..b6b6112c 100644 --- a/supernode/node/action/server/cascade/cascade_action_server.go +++ b/supernode/node/action/server/cascade/cascade_action_server.go @@ -308,8 +308,21 @@ func (server *ActionServer) Download(req *pb.DownloadRequest, stream pb.CascadeS "chunk_size": chunkSize, }) - // Split and stream the file using adaptive chunk size - for i := 0; i < len(restoredFile); i += chunkSize { + // Announce: file is ready to be served to the client + if err := stream.Send(&pb.DownloadResponse{ + ResponseType: &pb.DownloadResponse_Event{ + Event: &pb.DownloadEvent{ + EventType: pb.SupernodeEventType_SERVE_READY, + Message: "File available for download", + }, + }, + }); err != nil { + logtrace.Error(ctx, "failed to send serve-ready event", logtrace.Fields{logtrace.FieldError: err.Error()}) + return err + } + + // Split and stream the file using adaptive chunk size + for i := 0; i < len(restoredFile); i += chunkSize { end := i + chunkSize if end > len(restoredFile) { end = len(restoredFile) diff --git a/supernode/node/supernode/gateway/server.go b/supernode/node/supernode/gateway/server.go index 10cf0cc1..db6852b5 100644 --- a/supernode/node/supernode/gateway/server.go +++ b/supernode/node/supernode/gateway/server.go @@ -71,16 +71,34 @@ func (s *Server) Run(ctx context.Context) error { // Register Swagger endpoints httpMux.HandleFunc("/swagger.json", s.serveSwaggerJSON) httpMux.HandleFunc("/swagger-ui/", s.serveSwaggerUI) - // Expose current RaptorQ codec config as part of status surface - httpMux.HandleFunc("/api/v1/codec", func(w http.ResponseWriter, r *http.Request) { - if r.Method != http.MethodGet { - http.Error(w, "method not allowed", http.StatusMethodNotAllowed) - return - } - cfg := codecconfig.Current(r.Context()) - w.Header().Set("Content-Type", "application/json") - _ = json.NewEncoder(w).Encode(cfg) - }) + // Expose minimal RaptorQ codec config (trimmed; no bloat) + httpMux.HandleFunc("/api/v1/codec", func(w http.ResponseWriter, r *http.Request) { + if r.Method != http.MethodGet { + http.Error(w, "method not allowed", http.StatusMethodNotAllowed) + return + } + cfg := codecconfig.Current(r.Context()) + type minimalCodec struct { + SymbolSize uint16 `json:"symbol_size"` + Redundancy uint8 `json:"redundancy"` + MaxMemoryMB uint64 `json:"max_memory_mb"` + Concurrency uint64 `json:"concurrency"` + HeadroomPct int `json:"headroom_pct"` + MemLimitMB uint64 `json:"mem_limit_mb"` + MemLimitSource string `json:"mem_limit_source"` + } + out := minimalCodec{ + SymbolSize: cfg.SymbolSize, + Redundancy: cfg.Redundancy, + MaxMemoryMB: cfg.MaxMemoryMB, + Concurrency: cfg.Concurrency, + HeadroomPct: cfg.HeadroomPct, + MemLimitMB: cfg.MemLimitMB, + MemLimitSource: cfg.MemLimitSource, + } + w.Header().Set("Content-Type", "application/json") + _ = json.NewEncoder(w).Encode(out) + }) httpMux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) { if r.URL.Path == "/" { http.Redirect(w, r, "/swagger-ui/", http.StatusFound) diff --git a/supernode/node/supernode/server/status_server.go b/supernode/node/supernode/server/status_server.go index f4d2a6d6..0710cf4f 100644 --- a/supernode/node/supernode/server/status_server.go +++ b/supernode/node/supernode/server/status_server.go @@ -180,18 +180,15 @@ func (s *SupernodeServer) GetStatus(ctx context.Context, req *pb.StatusRequest) // Populate codec configuration cfg := codecconfig.Current(ctx) - response.Codec = &pb.StatusResponse_CodecConfig{ - SymbolSize: uint32(cfg.SymbolSize), - Redundancy: uint32(cfg.Redundancy), - MaxMemoryMb: cfg.MaxMemoryMB, - Concurrency: uint32(cfg.Concurrency), - Profile: cfg.Profile, - HeadroomPct: int32(cfg.HeadroomPct), - MemLimitMb: cfg.MemLimitMB, - MemLimitSource: cfg.MemLimitSource, - EffectiveCores: int32(cfg.EffectiveCores), - CpuLimitSource: cfg.CpuLimitSource, - } + response.Codec = &pb.StatusResponse_CodecConfig{ + SymbolSize: uint32(cfg.SymbolSize), + Redundancy: uint32(cfg.Redundancy), + MaxMemoryMb: cfg.MaxMemoryMB, + Concurrency: uint32(cfg.Concurrency), + HeadroomPct: int32(cfg.HeadroomPct), + MemLimitMb: cfg.MemLimitMB, + MemLimitSource: cfg.MemLimitSource, + } return response, nil } diff --git a/supernode/services/cascade/adaptors/p2p.go b/supernode/services/cascade/adaptors/p2p.go index dc8c077a..fcaad76a 100644 --- a/supernode/services/cascade/adaptors/p2p.go +++ b/supernode/services/cascade/adaptors/p2p.go @@ -232,15 +232,8 @@ func (c *p2pImpl) storeSymbolsInP2P(ctx context.Context, taskID, root string, fi return 0, 0, 0, fmt.Errorf("load symbols: %w", err) } - // Timeouts: rely on inner per-RPC limits and outer task envelope - // - Each node RPC invoked by StoreBatch uses Network.Call with per-message - // timeouts (BatchStoreData currently ~75s) enforced in kademlia/network.go. - // - The server-side Register handler is wrapped in a long envelope timeout - // (RegisterTimeout) to guarantee eventual completion/cancellation. - // Therefore, we avoid adding another mid-layer timer here to prevent - // premature cancellation of large batches and to let lower-level retries - // and per-RPC deadlines operate as designed. - symCtx := ctx + symCtx, cancel := context.WithTimeout(ctx, 5*time.Minute) + defer cancel() rate, requests, err := c.p2p.StoreBatch(symCtx, symbols, storage.P2PDataRaptorQSymbol, taskID) if err != nil { diff --git a/supernode/services/cascade/download.go b/supernode/services/cascade/download.go index a6851397..78db1758 100644 --- a/supernode/services/cascade/download.go +++ b/supernode/services/cascade/download.go @@ -31,20 +31,13 @@ type DownloadResponse struct { DownloadedDir string } -// Download preparation is bounded by DownloadPrepareTimeout (see timeouts.go). -// The subsequent file streaming to the client is not bounded by this timer. - func (task *CascadeRegistrationTask) Download( ctx context.Context, req *DownloadRequest, send func(resp *DownloadResponse) error, ) (err error) { - // Bound the preparation phase only (metadata/layout/symbols/restore) - ctx, cancel := context.WithTimeout(ctx, DownloadPrepareTimeout) - defer cancel() - - fields := logtrace.Fields{logtrace.FieldMethod: "Download", logtrace.FieldRequest: req} - logtrace.Info(ctx, "cascade-action-download request received", fields) + fields := logtrace.Fields{logtrace.FieldMethod: "Download", logtrace.FieldRequest: req} + logtrace.Info(ctx, "Cascade download request received", fields) // Ensure task status is finalized regardless of outcome defer func() { @@ -61,8 +54,8 @@ func (task *CascadeRegistrationTask) Download( fields[logtrace.FieldError] = err return task.wrapErr(ctx, "failed to get action", err, fields) } - logtrace.Info(ctx, "action has been retrieved", fields) - task.streamDownloadEvent(SupernodeEventTypeActionRetrieved, "action has been retrieved", "", "", send) + logtrace.Info(ctx, "Action retrieved", fields) + task.streamDownloadEvent(SupernodeEventTypeActionRetrieved, "Action retrieved", "", "", send) if actionDetails.GetAction().State != actiontypes.ActionStateDone { err = errors.New("action is not in a valid state") @@ -70,24 +63,27 @@ func (task *CascadeRegistrationTask) Download( fields[logtrace.FieldActionState] = actionDetails.GetAction().State return task.wrapErr(ctx, "action not found", err, fields) } - logtrace.Info(ctx, "action has been validated", fields) - task.streamDownloadEvent(SupernodeEventTypeActionFinalized, "action state has been validated", "", "", send) + logtrace.Info(ctx, "Action state validated", fields) metadata, err := task.decodeCascadeMetadata(ctx, actionDetails.GetAction().Metadata, fields) if err != nil { fields[logtrace.FieldError] = err.Error() return task.wrapErr(ctx, "error decoding cascade metadata", err, fields) } - logtrace.Info(ctx, "cascade metadata has been decoded", fields) - task.streamDownloadEvent(SupernodeEventTypeMetadataDecoded, "metadata has been decoded", "", "", send) + logtrace.Info(ctx, "Cascade metadata decoded", fields) + task.streamDownloadEvent(SupernodeEventTypeMetadataDecoded, "Cascade metadata decoded", "", "", send) - filePath, tmpDir, err := task.downloadArtifacts(ctx, actionDetails.GetAction().ActionID, metadata, fields) - if err != nil { - fields[logtrace.FieldError] = err.Error() - return task.wrapErr(ctx, "failed to download artifacts", err, fields) - } - logtrace.Info(ctx, "artifacts have been downloaded", fields) - task.streamDownloadEvent(SupernodeEventTypeArtefactsDownloaded, "artifacts have been downloaded", filePath, tmpDir, send) + // Notify: network retrieval phase begins + task.streamDownloadEvent(SupernodeEventTypeNetworkRetrieveStarted, "Network retrieval started", "", "", send) + + filePath, tmpDir, err := task.downloadArtifacts(ctx, actionDetails.GetAction().ActionID, metadata, fields) + if err != nil { + fields[logtrace.FieldError] = err.Error() + return task.wrapErr(ctx, "failed to download artifacts", err, fields) + } + logtrace.Info(ctx, "File reconstructed and hash verified", fields) + // Notify: decode completed, file ready on disk + task.streamDownloadEvent(SupernodeEventTypeDecodeCompleted, "Decode completed", filePath, tmpDir, send) return nil } @@ -151,13 +147,15 @@ func (task *CascadeRegistrationTask) restoreFileFromLayout( fields["totalSymbols"] = totalSymbols fields["requiredSymbols"] = requiredSymbols - logtrace.Info(ctx, "symbols to be retrieved", fields) + logtrace.Info(ctx, "Symbols to be retrieved", fields) - // Progressive retrieval moved to helper for readability/testing - decodeInfo, err := task.retrieveAndDecodeProgressively(ctx, allSymbols, layout, actionID, fields) - if err != nil { - return "", "", err - } + // Progressive retrieval moved to helper for readability/testing + decodeInfo, err := task.retrieveAndDecodeProgressively(ctx, layout, actionID, fields) + if err != nil { + fields[logtrace.FieldError] = err.Error() + logtrace.Error(ctx, "failed to decode symbols progressively", fields) + return "", "", fmt.Errorf("decode symbols using RaptorQ: %w", err) + } fileHash, err := crypto.HashFileIncrementally(decodeInfo.FilePath, 0) if err != nil { @@ -171,14 +169,13 @@ func (task *CascadeRegistrationTask) restoreFileFromLayout( return "", "", errors.New("file hash is nil") } - // Validate final payload hash against on-chain data hash err = task.verifyDataHash(ctx, fileHash, dataHash, fields) if err != nil { logtrace.Error(ctx, "failed to verify hash", fields) fields[logtrace.FieldError] = err.Error() return "", decodeInfo.DecodeTmpDir, err } - logtrace.Info(ctx, "file successfully restored and hash verified", fields) + logtrace.Info(ctx, "File successfully restored and hash verified", fields) return decodeInfo.FilePath, decodeInfo.DecodeTmpDir, nil } diff --git a/supernode/services/cascade/events.go b/supernode/services/cascade/events.go index 54006053..0b25d3b8 100644 --- a/supernode/services/cascade/events.go +++ b/supernode/services/cascade/events.go @@ -18,4 +18,8 @@ const ( SupernodeEventTypeActionFinalized SupernodeEventType = 12 SupernodeEventTypeArtefactsDownloaded SupernodeEventType = 13 SupernodeEventTypeFinalizeSimulationFailed SupernodeEventType = 14 + // Download phase markers + SupernodeEventTypeNetworkRetrieveStarted SupernodeEventType = 15 + SupernodeEventTypeDecodeCompleted SupernodeEventType = 16 + SupernodeEventTypeServeReady SupernodeEventType = 17 ) diff --git a/supernode/services/cascade/progressive_decode.go b/supernode/services/cascade/progressive_decode.go index 87c958be..0b46e06d 100644 --- a/supernode/services/cascade/progressive_decode.go +++ b/supernode/services/cascade/progressive_decode.go @@ -1,13 +1,14 @@ package cascade import ( - "context" - "fmt" - "strings" + "context" + "fmt" + "math" + "strings" - "github.com/LumeraProtocol/supernode/v2/pkg/codec" - "github.com/LumeraProtocol/supernode/v2/pkg/logtrace" - "github.com/LumeraProtocol/supernode/v2/supernode/services/cascade/adaptors" + "github.com/LumeraProtocol/supernode/v2/pkg/codec" + "github.com/LumeraProtocol/supernode/v2/pkg/logtrace" + "github.com/LumeraProtocol/supernode/v2/supernode/services/cascade/adaptors" ) // retrieveAndDecodeProgressively progressively retrieves symbols and attempts decode at @@ -22,81 +23,176 @@ import ( // could lead to repeated failures or fetching too many symbols upfront, increasing memory // pressure. This helper escalates in steps (9%, 25%, 50%, 75%, 100%). func (task *CascadeRegistrationTask) retrieveAndDecodeProgressively( - ctx context.Context, - allSymbols []string, - layout codec.Layout, - actionID string, - fields logtrace.Fields, + ctx context.Context, + layout codec.Layout, + actionID string, + fields logtrace.Fields, ) (adaptors.DecodeResponse, error) { - // Ensure base context fields are present for logs - if fields == nil { - fields = logtrace.Fields{} - } - fields[logtrace.FieldActionID] = actionID + // Ensure base context fields are present for logs + if fields == nil { + fields = logtrace.Fields{} + } + fields[logtrace.FieldActionID] = actionID - totalSymbols := len(allSymbols) - // escalate retrieval targets - percents := []int{requiredSymbolPercent, 25, 50, 75, 100} - seen := map[int]struct{}{} - ordered := make([]int, 0, len(percents)) - for _, p := range percents { - if p < requiredSymbolPercent { - continue - } - if _, ok := seen[p]; !ok { - seen[p] = struct{}{} - ordered = append(ordered, p) - } - } + // escalate retrieval targets + percents := []int{requiredSymbolPercent, 25, 50, 75, 100} + seen := map[int]struct{}{} + ordered := make([]int, 0, len(percents)) + for _, p := range percents { + if p < requiredSymbolPercent { + continue + } + if _, ok := seen[p]; !ok { + seen[p] = struct{}{} + ordered = append(ordered, p) + } + } - var lastErr error - for _, p := range ordered { - reqCount := (totalSymbols*p + 99) / 100 - fields["targetPercent"] = p - fields["targetCount"] = reqCount - logtrace.Info(ctx, "retrieving symbols for target percent", fields) + // Build per-block symbol lists for balanced selection and compute total + perBlock, totalSymbols := makePerBlock(layout) - symbols, err := task.P2PClient.BatchRetrieve(ctx, allSymbols, reqCount, actionID) - if err != nil { - fields[logtrace.FieldError] = err.Error() - logtrace.Error(ctx, "failed to retrieve symbols", fields) - return adaptors.DecodeResponse{}, fmt.Errorf("failed to retrieve symbols: %w", err) - } - fields["retrievedSymbols"] = len(symbols) - logtrace.Info(ctx, "symbols retrieved", fields) + var lastErr error + for _, p := range ordered { + reqCount := (totalSymbols*p + 99) / 100 + fields["targetPercent"] = p + fields["targetCount"] = reqCount + logtrace.Info(ctx, "retrieving symbols for target percent", fields) - // Attempt decode - decodeInfo, err := task.RQ.Decode(ctx, adaptors.DecodeRequest{ - ActionID: actionID, - Symbols: symbols, - Layout: layout, - }) - if err == nil { - return decodeInfo, nil - } + // Build a balanced candidate key set ensuring some coverage per block. + candidateKeys := selectBalancedKeys(perBlock, layout, reqCount, totalSymbols) - // Only escalate for probable insufficiency/integrity errors; otherwise, fail fast - errStr := err.Error() - if p >= 100 || !( - strings.Contains(errStr, "decoding failed") || - strings.Contains(strings.ToLower(errStr), "hash mismatch") || - strings.Contains(strings.ToLower(errStr), "insufficient") || - strings.Contains(strings.ToLower(errStr), "symbol")) { - fields[logtrace.FieldError] = errStr - logtrace.Error(ctx, "failed to decode symbols", fields) - return adaptors.DecodeResponse{}, fmt.Errorf("decode symbols using RaptorQ: %w", err) - } + symbols, err := task.P2PClient.BatchRetrieve(ctx, candidateKeys, reqCount, actionID) + if err != nil { + fields[logtrace.FieldError] = err.Error() + logtrace.Error(ctx, "failed to retrieve symbols", fields) + return adaptors.DecodeResponse{}, fmt.Errorf("failed to retrieve symbols: %w", err) + } + fields["retrievedSymbols"] = len(symbols) + logtrace.Info(ctx, "symbols retrieved", fields) - logtrace.Info(ctx, "decode failed; escalating symbol target", logtrace.Fields{ - "last_error": errStr, - }) - lastErr = err - } + // Attempt decode + decodeInfo, err := task.RQ.Decode(ctx, adaptors.DecodeRequest{ + ActionID: actionID, + Symbols: symbols, + Layout: layout, + }) + if err == nil { + return decodeInfo, nil + } - if lastErr != nil { - fields[logtrace.FieldError] = lastErr.Error() - logtrace.Error(ctx, "failed to decode symbols after escalation", fields) - return adaptors.DecodeResponse{}, fmt.Errorf("decode symbols using RaptorQ: %w", lastErr) - } - return adaptors.DecodeResponse{}, fmt.Errorf("decode symbols using RaptorQ: unknown failure") + // Only escalate for probable insufficiency/integrity errors; otherwise, fail fast + errStr := err.Error() + if p >= 100 || !(strings.Contains(errStr, "decoding failed") || + strings.Contains(strings.ToLower(errStr), "hash mismatch") || + strings.Contains(strings.ToLower(errStr), "insufficient") || + strings.Contains(strings.ToLower(errStr), "symbol")) { + fields[logtrace.FieldError] = errStr + logtrace.Error(ctx, "failed to decode symbols", fields) + return adaptors.DecodeResponse{}, fmt.Errorf("decode symbols using RaptorQ: %w", err) + } + + logtrace.Info(ctx, "decode failed; escalating symbol target", logtrace.Fields{ + "last_error": errStr, + }) + lastErr = err + } + + if lastErr != nil { + fields[logtrace.FieldError] = lastErr.Error() + logtrace.Error(ctx, "failed to decode symbols after escalation", fields) + return adaptors.DecodeResponse{}, fmt.Errorf("decode symbols using RaptorQ: %w", lastErr) + } + return adaptors.DecodeResponse{}, fmt.Errorf("decode symbols using RaptorQ: unknown failure") +} + +// makePerBlock returns a copy of per-block symbol slices and the total symbol count. +func makePerBlock(layout codec.Layout) (map[int][]string, int) { + perBlock := make(map[int][]string) + total := 0 + for _, blk := range layout.Blocks { + syms := make([]string, len(blk.Symbols)) + copy(syms, blk.Symbols) + perBlock[blk.BlockID] = syms + total += len(syms) + } + return perBlock, total +} + +// selectBalancedKeys chooses up to reqCount keys ensuring at least 1 per block when possible, +// then distributing the remainder roughly proportional to block sizes, and finally filling +// any gap via round-robin. totalSymbols is the sum of all per-block lengths. +func selectBalancedKeys(perBlock map[int][]string, layout codec.Layout, reqCount int, totalSymbols int) []string { + candidateKeys := make([]string, 0, reqCount) + blocks := layout.Blocks + // seed: at least one per block if possible + for _, blk := range blocks { + if len(candidateKeys) >= reqCount { + break + } + syms := perBlock[blk.BlockID] + if len(syms) > 0 { + candidateKeys = append(candidateKeys, syms[0]) + } + } + remaining := reqCount - len(candidateKeys) + if remaining <= 0 { + return candidateKeys + } + // quotas per block + quotas := make(map[int]int) + sumQuotas := 0 + for _, blk := range blocks { + blkTotal := len(perBlock[blk.BlockID]) + if blkTotal == 0 || totalSymbols == 0 { + quotas[blk.BlockID] = 0 + continue + } + q := int(math.Ceil(float64(remaining) * float64(blkTotal) / float64(totalSymbols))) + if q < 0 { + q = 0 + } + quotas[blk.BlockID] = q + sumQuotas += q + } + // normalize overshoot + for sumQuotas > remaining { + for _, blk := range blocks { + if sumQuotas <= remaining { + break + } + if quotas[blk.BlockID] > 0 { + quotas[blk.BlockID]-- + sumQuotas-- + } + } + } + // select additional symbols according to quotas + for _, blk := range blocks { + want := quotas[blk.BlockID] + if want <= 0 { + continue + } + syms := perBlock[blk.BlockID] + start := 1 // 0th may have been used in seed + for i := start; i < len(syms) && want > 0 && len(candidateKeys) < reqCount; i++ { + candidateKeys = append(candidateKeys, syms[i]) + want-- + } + } + // if still short (rounding/small blocks), round-robin fill + if len(candidateKeys) < reqCount { + for _, blk := range blocks { + if len(candidateKeys) >= reqCount { + break + } + syms := perBlock[blk.BlockID] + for i := 0; i < len(syms) && len(candidateKeys) < reqCount; i++ { + candidateKeys = append(candidateKeys, syms[i]) + } + } + } + if len(candidateKeys) > reqCount { + candidateKeys = candidateKeys[:reqCount] + } + return candidateKeys } diff --git a/supernode/services/cascade/register.go b/supernode/services/cascade/register.go index 640f1a98..403d0428 100644 --- a/supernode/services/cascade/register.go +++ b/supernode/services/cascade/register.go @@ -24,10 +24,6 @@ type RegisterResponse struct { TxHash string } -// RegisterTimeout bounds the execution time of a Register task to prevent tasks -// from lingering in a non-final state if a dependency stalls. Keep greater than -// the SDK's upload+processing budgets so the client cancels first. - // Register processes the upload request for cascade input data. // 1- Fetch & validate action (it should be a cascade action registered on the chain) // 2- Ensure this super-node is eligible to process the action (should be in the top supernodes list for the action block height) @@ -48,12 +44,9 @@ func (task *CascadeRegistrationTask) Register( req *RegisterRequest, send func(resp *RegisterResponse) error, ) (err error) { - // Defensive envelope deadline to guarantee task finalization - ctx, cancel := context.WithTimeout(ctx, RegisterTimeout) - defer cancel() fields := logtrace.Fields{logtrace.FieldMethod: "Register", logtrace.FieldRequest: req} - logtrace.Info(ctx, "cascade-action-registration request received", fields) + logtrace.Info(ctx, "Cascade registration request received", fields) // Ensure task status and resources are finalized regardless of outcome defer func() { @@ -69,9 +62,9 @@ func (task *CascadeRegistrationTask) Register( defer func() { if req != nil && req.FilePath != "" { if remErr := os.RemoveAll(req.FilePath); remErr != nil { - logtrace.Warn(ctx, "error removing file", fields) + logtrace.Warn(ctx, "Failed to remove uploaded file", fields) } else { - logtrace.Info(ctx, "input file has been cleaned up", fields) + logtrace.Info(ctx, "Uploaded file cleaned up", fields) } } }() @@ -85,46 +78,46 @@ func (task *CascadeRegistrationTask) Register( fields[logtrace.FieldCreator] = action.Creator fields[logtrace.FieldStatus] = action.State fields[logtrace.FieldPrice] = action.Price - logtrace.Info(ctx, "action has been retrieved", fields) - task.streamEvent(SupernodeEventTypeActionRetrieved, "action has been retrieved", "", send) + logtrace.Info(ctx, "Action retrieved", fields) + task.streamEvent(SupernodeEventTypeActionRetrieved, "Action retrieved", "", send) /* 2. Verify action fee -------------------------------------------------------- */ if err := task.verifyActionFee(ctx, action, req.DataSize, fields); err != nil { return err } - logtrace.Info(ctx, "action fee has been validated", fields) - task.streamEvent(SupernodeEventTypeActionFeeVerified, "action-fee has been validated", "", send) + logtrace.Info(ctx, "Action fee verified", fields) + task.streamEvent(SupernodeEventTypeActionFeeVerified, "Action fee verified", "", send) /* 3. Ensure this super-node is eligible -------------------------------------- */ fields[logtrace.FieldSupernodeState] = task.config.SupernodeAccountAddress if err := task.ensureIsTopSupernode(ctx, uint64(action.BlockHeight), fields); err != nil { return err } - logtrace.Info(ctx, "current-supernode exists in the top-sn list", fields) - task.streamEvent(SupernodeEventTypeTopSupernodeCheckPassed, "current supernode exists in the top-sn list", "", send) + logtrace.Info(ctx, "Top supernode eligibility confirmed", fields) + task.streamEvent(SupernodeEventTypeTopSupernodeCheckPassed, "Top supernode eligibility confirmed", "", send) /* 4. Decode cascade metadata -------------------------------------------------- */ cascadeMeta, err := task.decodeCascadeMetadata(ctx, action.Metadata, fields) if err != nil { return err } - logtrace.Info(ctx, "cascade metadata decoded", fields) - task.streamEvent(SupernodeEventTypeMetadataDecoded, "cascade metadata has been decoded", "", send) + logtrace.Info(ctx, "Cascade metadata decoded", fields) + task.streamEvent(SupernodeEventTypeMetadataDecoded, "Cascade metadata decoded", "", send) /* 5. Verify data hash --------------------------------------------------------- */ if err := task.verifyDataHash(ctx, req.DataHash, cascadeMeta.DataHash, fields); err != nil { return err } - logtrace.Info(ctx, "data-hash has been verified", fields) - task.streamEvent(SupernodeEventTypeDataHashVerified, "data-hash has been verified", "", send) + logtrace.Info(ctx, "Data hash verified", fields) + task.streamEvent(SupernodeEventTypeDataHashVerified, "Data hash verified", "", send) /* 6. Encode the raw data ------------------------------------------------------ */ encResp, err := task.encodeInput(ctx, req.ActionID, req.FilePath, req.DataSize, fields) if err != nil { return err } - logtrace.Info(ctx, "input-data has been encoded", fields) - task.streamEvent(SupernodeEventTypeInputEncoded, "input data has been encoded", "", send) + logtrace.Info(ctx, "Input encoded", fields) + task.streamEvent(SupernodeEventTypeInputEncoded, "Input encoded", "", send) /* 7. Signature verification + layout decode ---------------------------------- */ layout, signature, err := task.verifySignatureAndDecodeLayout( @@ -133,35 +126,35 @@ func (task *CascadeRegistrationTask) Register( if err != nil { return err } - logtrace.Info(ctx, "signature has been verified", fields) - task.streamEvent(SupernodeEventTypeSignatureVerified, "signature has been verified", "", send) + logtrace.Info(ctx, "Signature verified", fields) + task.streamEvent(SupernodeEventTypeSignatureVerified, "Signature verified", "", send) /* 8. Generate RQ-ID files ----------------------------------------------------- */ rqidResp, err := task.generateRQIDFiles(ctx, cascadeMeta, signature, action.Creator, encResp.Metadata, fields) if err != nil { return err } - logtrace.Info(ctx, "rq-id files have been generated", fields) - task.streamEvent(SupernodeEventTypeRQIDsGenerated, "rq-id files have been generated", "", send) + logtrace.Info(ctx, "RQID files generated", fields) + task.streamEvent(SupernodeEventTypeRQIDsGenerated, "RQID files generated", "", send) /* 9. Consistency checks ------------------------------------------------------- */ if err := verifyIDs(layout, encResp.Metadata); err != nil { return task.wrapErr(ctx, "failed to verify IDs", err, fields) } - logtrace.Info(ctx, "rq-ids have been verified", fields) - task.streamEvent(SupernodeEventTypeRqIDsVerified, "rq-ids have been verified", "", send) + logtrace.Info(ctx, "RQIDs verified", fields) + task.streamEvent(SupernodeEventTypeRqIDsVerified, "RQIDs verified", "", send) /* 10. Simulate finalize to avoid storing artefacts if it would fail ---------- */ if _, err := task.LumeraClient.SimulateFinalizeAction(ctx, action.ActionID, rqidResp.RQIDs); err != nil { fields[logtrace.FieldError] = err.Error() - logtrace.Info(ctx, "finalize action simulation failed", fields) + logtrace.Info(ctx, "Finalize simulation failed", fields) // Emit explicit simulation failure event for client visibility - task.streamEvent(SupernodeEventTypeFinalizeSimulationFailed, "finalize action simulation failed", "", send) + task.streamEvent(SupernodeEventTypeFinalizeSimulationFailed, "Finalize simulation failed", "", send) return task.wrapErr(ctx, "finalize action simulation failed", err, fields) } - logtrace.Info(ctx, "finalize action simulation passed", fields) + logtrace.Info(ctx, "Finalize simulation passed", fields) // Transmit as a standard event so SDK can propagate it (dedicated type) - task.streamEvent(SupernodeEventTypeFinalizeSimulated, "finalize action simulation passed", "", send) + task.streamEvent(SupernodeEventTypeFinalizeSimulated, "Finalize simulation passed", "", send) /* 11. Persist artefacts -------------------------------------------------------- */ // Persist artefacts to the P2P network. @@ -180,13 +173,13 @@ func (task *CascadeRegistrationTask) Register( resp, err := task.LumeraClient.FinalizeAction(ctx, action.ActionID, rqidResp.RQIDs) if err != nil { fields[logtrace.FieldError] = err.Error() - logtrace.Info(ctx, "Finalize Action Error", fields) + logtrace.Info(ctx, "Finalize action error", fields) return task.wrapErr(ctx, "failed to finalize action", err, fields) } txHash := resp.TxResponse.TxHash fields[logtrace.FieldTxHash] = txHash - logtrace.Info(ctx, "action has been finalized", fields) - task.streamEvent(SupernodeEventTypeActionFinalized, "action has been finalized", txHash, send) + logtrace.Info(ctx, "Action finalized", fields) + task.streamEvent(SupernodeEventTypeActionFinalized, "Action finalized", txHash, send) return nil } diff --git a/supernode/services/cascade/timeouts.go b/supernode/services/cascade/timeouts.go deleted file mode 100644 index 110dba3d..00000000 --- a/supernode/services/cascade/timeouts.go +++ /dev/null @@ -1,21 +0,0 @@ -package cascade - -import "time" - -// Server-side task envelope timeouts for Cascade service. -// -// Rationale: -// - RegisterTimeout must exceed the total of SDK upload + processing budgets -// so the client surfaces errors; currently upload=60m and processing=10m. -// - Download uses a split approach: a tight server-side preparation timeout -// and a relaxed client-governed transfer window. -const ( - // RegisterTimeout bounds the entire Register RPC handler lifetime. - // Must be greater than SDK's upload (60m) + processing (10m) budgets. - RegisterTimeout = 75 * time.Minute - - // DownloadPrepareTimeout bounds the server-side preparation phase for - // downloads (fetch metadata, retrieve symbols, reconstruct and verify file). - // This phase is independent of client bandwidth and should be quick. - DownloadPrepareTimeout = 5 * time.Minute -) diff --git a/supernode/services/common/base/supernode_task.go b/supernode/services/common/base/supernode_task.go index 6cde3d4f..937e6013 100644 --- a/supernode/services/common/base/supernode_task.go +++ b/supernode/services/common/base/supernode_task.go @@ -25,21 +25,7 @@ type SuperNodeTask struct { func (task *SuperNodeTask) RunHelper(ctx context.Context, clean TaskCleanerFunc) error { ctx = task.context(ctx) logtrace.Debug(ctx, "Start task", logtrace.Fields{}) - defer func() { - // Log accurate end-state when task finishes - st := task.Status() - if st != nil && st.SubStatus != nil { - if st.SubStatus.IsFailure() { - logtrace.Info(ctx, "Task canceled", logtrace.Fields{}) - } else if st.SubStatus.IsFinal() { - logtrace.Info(ctx, "Task completed", logtrace.Fields{}) - } else { - logtrace.Info(ctx, "Task ended", logtrace.Fields{}) - } - } else { - logtrace.Info(ctx, "Task ended", logtrace.Fields{}) - } - }() + defer logtrace.Info(ctx, "Task canceled", logtrace.Fields{}) defer task.Cancel() task.SetStatusNotifyFunc(func(status *state.Status) { diff --git a/testnet_version_check.sh b/testnet_version_check.sh index e61bf32a..e3a53a4a 100755 --- a/testnet_version_check.sh +++ b/testnet_version_check.sh @@ -7,7 +7,7 @@ set -o pipefail # # Usage: ./check_versions.sh (no args) -TIMEOUT=5 +TIMEOUT=10 API_URL="https://lcd.testnet.lumera.io/LumeraProtocol/lumera/supernode/list_super_nodes?pagination.limit=1000&pagination.count_total=true" TOTAL_CHECKED=0