Skip to content

fix(container-cache): use TCP liveness probe so a busy proxy is never killed #318

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
balajinvda wants to merge 4 commits into
base: main
Choose a base branch
from
Open

fix(container-cache): use TCP liveness probe so a busy proxy is never killed #318

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
9c15516
fix(container-cache): use TCP liveness probe so a busy proxy is never…
balaji-g Jul 21, 2026
6d40435
fix(container-cache): keep nginx workers unblocked under slow backing…
balaji-g Jul 21, 2026
dfa374f
test(container-cache): rendered-probe regression tests; narrow probe …
balaji-g Jul 21, 2026
e555b5a
docs(container-cache): describe maxConnsPerServer scope accurately
balaji-g Jul 21, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions deploy/helm/container-cache/Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
.PHONY: test lint
test:
./tests/render-probes-test.sh
lint:
helm lint deploy
142 changes: 142 additions & 0 deletions deploy/helm/container-cache/_cc-analysis/access-0.jsonl
View file
Open in desktop

Large diffs are not rendered by default.

200 changes: 200 additions & 0 deletions deploy/helm/container-cache/_cc-analysis/access-1.jsonl
View file
Open in desktop

Large diffs are not rendered by default.

235 changes: 235 additions & 0 deletions deploy/helm/container-cache/_cc-analysis/access-2.jsonl
View file
Open in desktop

Large diffs are not rendered by default.

16 changes: 16 additions & 0 deletions deploy/helm/container-cache/_cc-analysis/ds1.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
7 0 loop0 15996 0 1406140 40367 0 0 0 0 0 115067 40367 0 0 0 0 0 0
7 1 loop1 13 0 40 3 0 0 0 0 0 3 3 0 0 0 0 0 0
7 2 loop2 17 0 52 3 0 0 0 0 0 5 3 0 0 0 0 0 0
7 3 loop3 17 0 58 3 0 0 0 0 0 3 3 0 0 0 0 0 0
7 4 loop4 165 0 11352 88 0 0 0 0 0 91 88 0 0 0 0 0 0
7 5 loop5 11 0 24 2 0 0 0 0 0 2 2 0 0 0 0 0 0
7 6 loop6 128468 0 9320422 1254729 0 0 0 0 0 121992 1254729 0 0 0 0 0 0
7 7 loop7 6936 0 637934 27483 0 0 0 0 0 69229 27483 0 0 0 0 0 0
259 0 nvme0n1 3936561 421479 228015376 9340451 36834748 35405637 2560278794 185024315 1 17611487 194364766 0 0 0 0 0 0
259 1 nvme0n1p1 3936169 419982 227998687 9340189 36834746 35405637 2560278792 185024314 1 19293830 194364503 0 0 0 0 0 0
259 2 nvme0n1p14 37 0 544 12 0 0 0 0 0 11 12 0 0 0 0 0 0
259 3 nvme0n1p15 304 1497 14177 229 2 0 2 1 0 54 231 0 0 0 0 0 0
7 8 loop8 11 0 28 0 0 0 0 0 0 1 0 0 0 0 0 0 0
259 4 nvme1n1 48555 158 1995954 31685 1056721 250911 23103432 1068189 0 1079812 1099874 0 0 0 0 0 0
251 768 nvmesh/csi-e6c59d8f-f308-4bca 33240 0 15817600 2075831 4257 0 14414728 43210774 0 1211960 45286605 0 0 0 0 0 0
251 256 nvmesh/csi-c36388b0-e863-40fd 7876 0 1628776 156276218 69110 0 884614936 3595076543 486 13155693 3751352761 0 0 0 0 0 0
16 changes: 16 additions & 0 deletions deploy/helm/container-cache/_cc-analysis/ds2.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
7 0 loop0 16081 0 1413336 40413 0 0 0 0 0 115112 40413 0 0 0 0 0 0
7 1 loop1 13 0 40 3 0 0 0 0 0 3 3 0 0 0 0 0 0
7 2 loop2 17 0 52 3 0 0 0 0 0 5 3 0 0 0 0 0 0
7 3 loop3 17 0 58 3 0 0 0 0 0 3 3 0 0 0 0 0 0
7 4 loop4 165 0 11352 88 0 0 0 0 0 91 88 0 0 0 0 0 0
7 5 loop5 11 0 24 2 0 0 0 0 0 2 2 0 0 0 0 0 0
7 6 loop6 128499 0 9322392 1254751 0 0 0 0 0 122013 1254751 0 0 0 0 0 0
7 7 loop7 6936 0 637934 27483 0 0 0 0 0 69229 27483 0 0 0 0 0 0
259 0 nvme0n1 3944707 422319 228256360 9345400 36835415 35406293 2560293714 185024963 0 17614142 194370364 0 0 0 0 0 0
259 1 nvme0n1p1 3944315 420822 228239671 9345139 36835413 35406293 2560293712 185024962 0 19296629 194370101 0 0 0 0 0 0
259 2 nvme0n1p14 37 0 544 12 0 0 0 0 0 11 12 0 0 0 0 0 0
259 3 nvme0n1p15 304 1497 14177 229 2 0 2 1 0 54 231 0 0 0 0 0 0
7 8 loop8 11 0 28 0 0 0 0 0 0 1 0 0 0 0 0 0 0
259 4 nvme1n1 48556 158 1995962 31686 1056735 250926 23103792 1068201 0 1079818 1099887 0 0 0 0 0 0
251 768 nvmesh/csi-e6c59d8f-f308-4bca 33240 0 15817600 2075831 4257 0 14414728 43210774 0 1211960 45286605 0 0 0 0 0 0
251 256 nvmesh/csi-c36388b0-e863-40fd 7876 0 1628776 156276218 69145 0 885894168 3598239920 451 13167846 3754516138 0 0 0 0 0 0
15 changes: 15 additions & 0 deletions deploy/helm/container-cache/_cc-analysis/n1-a.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
7 0 loop0 17280 0 1525692 118875 0 0 0 0 0 218875 118875 0 0 0 0 0 0
7 1 loop1 14 0 42 5 0 0 0 0 0 5 5 0 0 0 0 0 0
7 2 loop2 18 0 54 6 0 0 0 0 0 5 6 0 0 0 0 0 0
7 3 loop3 18 0 62 6 0 0 0 0 0 7 6 0 0 0 0 0 0
7 4 loop4 166 0 11354 296 0 0 0 0 0 292 296 0 0 0 0 0 0
7 5 loop5 12 0 26 4 0 0 0 0 0 4 4 0 0 0 0 0 0
7 6 loop6 167606 0 12258088 2819236 0 0 0 0 0 236721 2819236 0 0 0 0 0 0
7 7 loop7 7938 0 727196 14582 0 0 0 0 0 67948 14582 0 0 0 0 0 0
259 0 nvme0n1 4273054 583314 226208165 15414079 49302568 48401096 3034131938 209751360 0 22339978 225165439 0 0 0 0 0 0
259 1 nvme0n1p1 4272770 582696 226192367 15413751 49302565 48401096 3034131928 209751358 0 28371092 225165110 0 0 0 0 0 0
259 2 nvme0n1p14 37 0 544 13 0 0 0 0 0 11 13 0 0 0 0 0 0
259 3 nvme0n1p15 196 618 13286 292 3 0 10 1 0 66 294 0 0 0 0 0 0
7 8 loop8 11 0 28 0 0 0 0 0 0 0 0 0 0 0 0 0 0
251 256 nvmesh/csi-5321bf45-df13-4975 20938 0 4987552 308453578 75835 0 1175039112 3868429865 387 12425029 4176883443 0 0 0 0 0 0
251 512 nvmesh/csi-ef9ba400-87f9-41e4 62118 0 25197384 4754754 39542 0 68437152 284403806 0 2393935 289158560 0 0 0 0 0 0
15 changes: 15 additions & 0 deletions deploy/helm/container-cache/_cc-analysis/n1-b.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
7 0 loop0 17280 0 1525692 118875 0 0 0 0 0 218875 118875 0 0 0 0 0 0
7 1 loop1 14 0 42 5 0 0 0 0 0 5 5 0 0 0 0 0 0
7 2 loop2 18 0 54 6 0 0 0 0 0 5 6 0 0 0 0 0 0
7 3 loop3 18 0 62 6 0 0 0 0 0 7 6 0 0 0 0 0 0
7 4 loop4 166 0 11354 296 0 0 0 0 0 292 296 0 0 0 0 0 0
7 5 loop5 12 0 26 4 0 0 0 0 0 4 4 0 0 0 0 0 0
7 6 loop6 167606 0 12258088 2819236 0 0 0 0 0 236721 2819236 0 0 0 0 0 0
7 7 loop7 7938 0 727196 14582 0 0 0 0 0 67948 14582 0 0 0 0 0 0
259 0 nvme0n1 4273303 583325 226237589 15414444 49305158 48401804 3035064666 209791809 0 22342388 225206253 0 0 0 0 0 0
259 1 nvme0n1p1 4273019 582707 226221791 15414116 49305155 48401804 3035064656 209791807 0 28373660 225205924 0 0 0 0 0 0
259 2 nvme0n1p14 37 0 544 13 0 0 0 0 0 11 13 0 0 0 0 0 0
259 3 nvme0n1p15 196 618 13286 292 3 0 10 1 0 66 294 0 0 0 0 0 0
7 8 loop8 11 0 28 0 0 0 0 0 0 0 0 0 0 0 0 0 0
251 256 nvmesh/csi-5321bf45-df13-4975 20938 0 4987552 308453578 75842 0 1175264648 3869343520 405 12436252 4177797098 0 0 0 0 0 0
251 512 nvmesh/csi-ef9ba400-87f9-41e4 62118 0 25197384 4754754 39542 0 68437152 284403806 0 2393935 289158560 0 0 0 0 0 0
16 changes: 16 additions & 0 deletions deploy/helm/container-cache/_cc-analysis/n2-a.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
7 0 loop0 90701 0 8090656 9392178 0 0 0 0 0 3054832 9392178 0 0 0 0 0 0
7 1 loop1 13 0 40 4 0 0 0 0 0 5 4 0 0 0 0 0 0
7 2 loop2 20 0 58 7 0 0 0 0 0 8 7 0 0 0 0 0 0
7 3 loop3 18 0 62 7 0 0 0 0 0 9 7 0 0 0 0 0 0
7 4 loop4 165 0 11352 89 0 0 0 0 0 87 89 0 0 0 0 0 0
7 5 loop5 12 0 26 3 0 0 0 0 0 5 3 0 0 0 0 0 0
7 6 loop6 838061 0 61685928 95619662 0 0 0 0 0 5020524 95619662 0 0 0 0 0 0
7 7 loop7 24886 0 2356640 1613160 0 0 0 0 0 647253 1613160 0 0 0 0 0 0
259 0 nvme1n1 1170880 13739 212153350 1029578 5842489 2466301 1024673840 10021645 0 4219469 11051224 0 0 0 0 0 0
259 1 nvme0n1 32839434 4289621 2301457816 496220345 111884785 148167689 10423328554 349338515 0 42084300 845558861 0 0 0 0 0 0
259 3 nvme0n1p1 32839042 4288124 2301441127 496220076 111884783 148167689 10423328552 349338514 0 54044183 845558590 0 0 0 0 0 0
259 4 nvme0n1p14 37 0 544 21 0 0 0 0 0 24 21 0 0 0 0 0 0
259 5 nvme0n1p15 304 1497 14177 218 2 0 2 1 0 83 220 0 0 0 0 0 0
7 8 loop8 11 0 28 0 0 0 0 0 0 0 0 0 0 0 0 0 0
251 256 nvmesh/csi-21d4b107-85d5-4a84 24325 0 12230008 1773683 372 0 41344 255721 0 940317 2029404 0 0 0 0 0 0
251 512 nvmesh/csi-d8cca2c9-87ae-48df 2304 0 609072 34699226 43867 0 260188680 3889660555 1852 3719204 3924359781 0 0 0 0 0 0
16 changes: 16 additions & 0 deletions deploy/helm/container-cache/_cc-analysis/n2-b.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
7 0 loop0 90701 0 8090656 9392178 0 0 0 0 0 3054832 9392178 0 0 0 0 0 0
7 1 loop1 13 0 40 4 0 0 0 0 0 5 4 0 0 0 0 0 0
7 2 loop2 20 0 58 7 0 0 0 0 0 8 7 0 0 0 0 0 0
7 3 loop3 18 0 62 7 0 0 0 0 0 9 7 0 0 0 0 0 0
7 4 loop4 165 0 11352 89 0 0 0 0 0 87 89 0 0 0 0 0 0
7 5 loop5 12 0 26 3 0 0 0 0 0 5 3 0 0 0 0 0 0
7 6 loop6 838078 0 61687606 95619669 0 0 0 0 0 5020528 95619669 0 0 0 0 0 0
7 7 loop7 24886 0 2356640 1613160 0 0 0 0 0 647253 1613160 0 0 0 0 0 0
259 0 nvme1n1 1170880 13739 212153350 1029578 5842508 2466310 1024676088 10021671 0 4219480 11051249 0 0 0 0 0 0
259 1 nvme0n1 32839669 4289621 2301467104 496220492 111885200 148168037 10423336050 349338912 0 42084470 845559405 0 0 0 0 0 0
259 3 nvme0n1p1 32839277 4288124 2301450415 496220223 111885198 148168037 10423336048 349338911 0 54044424 845559134 0 0 0 0 0 0
259 4 nvme0n1p14 37 0 544 21 0 0 0 0 0 24 21 0 0 0 0 0 0
259 5 nvme0n1p15 304 1497 14177 218 2 0 2 1 0 83 220 0 0 0 0 0 0
7 8 loop8 11 0 28 0 0 0 0 0 0 0 0 0 0 0 0 0 0
251 256 nvmesh/csi-21d4b107-85d5-4a84 24325 0 12230008 1773683 372 0 41344 255721 0 940317 2029404 0 0 0 0 0 0
251 512 nvmesh/csi-d8cca2c9-87ae-48df 2304 0 609072 34699226 43867 0 260188680 3889660555 1870 3730291 3924359781 0 0 0 0 0 0
26 changes: 26 additions & 0 deletions deploy/helm/container-cache/_cc-analysis/nvcf-container-config.json
View file
Open in desktop

Large diffs are not rendered by default.

26 changes: 26 additions & 0 deletions deploy/helm/container-cache/_cc-analysis/nvcf-default-config.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
{
"apiVersion": "v1",
"data": {
"default.conf": "# SPDX-FileCopyrightText: Copyright (c) 2023-2025 NVIDIA CORPORATION \u0026 AFFILIATES. All rights reserved.\n# SPDX-License-Identifier: Apache-2.0\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n # Health check\n server {\n listen 13129;\n access_log off;\n # Health check endpoint\n location /healthz {\n add_header Content-Type text/plain;\n return 200 \"The Container Caching Proxy is running!\";\n }\n\n # reload nginx\n location /reload {\n allow 127.0.0.1;\n deny all;\n content_by_lua_block {\n local resty_signal = require \"resty.signal\"\n local file, err = io.open(\"/var/run/nginx.pid\", \"r\")\n if file == nil then\n ngx.status = 500\n ngx.say(\"Failed to reload nginx process: \", err)\n ngx.exit(ngx.status)\n end\n local pid = file:read(\"*n\")\n file:close()\n ngx.say(\"Nginx PID\", pid)\n ngx.say(\"Reloading nginx\", pid)\n local ok, err = resty_signal.kill(pid, \"HUP\")\n if not ok then\n ngx.log(ngx.ERR, \"failed to reload nginx process of pid \", pid, \": \", err)\n ngx.say(\"Failed to reload nginx process: \", err)\n ngx.status = 500\n ngx.exit(ngx.status)\n end\n }\n }\n }\n\n # Metrics\n server {\n listen 9145;\n access_log off;\n location /cache_metrics {\n content_by_lua_block {\n metric_connections:set(ngx.var.connections_reading, {\"reading\"})\n metric_connections:set(ngx.var.connections_waiting, {\"waiting\"})\n metric_connections:set(ngx.var.connections_writing, {\"writing\"})\n prometheus:collect()\n }\n }\n\n location = /metrics {\n return 404;\n }\n\n location /nginx_status {\n stub_status on;\n allow 127.0.0.1;\n allow ::1;\n deny all;\n }\n }\n"
},
"kind": "ConfigMap",
"metadata": {
"annotations": {
"argocd.argoproj.io/tracking-id": "nvcf-container-cache:/ConfigMap:container-caching/nvcf-default-config",
"kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"v1\",\"data\":{\"default.conf\":\"# SPDX-FileCopyrightText: Copyright (c) 2023-2025 NVIDIA CORPORATION \\u0026 AFFILIATES. All rights reserved.\\n# SPDX-License-Identifier: Apache-2.0\\n#\\n# Licensed under the Apache License, Version 2.0 (the \\\"License\\\");\\n# you may not use this file except in compliance with the License.\\n# You may obtain a copy of the License at\\n#\\n# http://www.apache.org/licenses/LICENSE-2.0\\n#\\n# Unless required by applicable law or agreed to in writing, software\\n# distributed under the License is distributed on an \\\"AS IS\\\" BASIS,\\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\\n# See the License for the specific language governing permissions and\\n# limitations under the License.\\n # Health check\\n server {\\n listen 13129;\\n access_log off;\\n # Health check endpoint\\n location /healthz {\\n add_header Content-Type text/plain;\\n return 200 \\\"The Container Caching Proxy is running!\\\";\\n }\\n\\n # reload nginx\\n location /reload {\\n allow 127.0.0.1;\\n deny all;\\n content_by_lua_block {\\n local resty_signal = require \\\"resty.signal\\\"\\n local file, err = io.open(\\\"/var/run/nginx.pid\\\", \\\"r\\\")\\n if file == nil then\\n ngx.status = 500\\n ngx.say(\\\"Failed to reload nginx process: \\\", err)\\n ngx.exit(ngx.status)\\n end\\n local pid = file:read(\\\"*n\\\")\\n file:close()\\n ngx.say(\\\"Nginx PID\\\", pid)\\n ngx.say(\\\"Reloading nginx\\\", pid)\\n local ok, err = resty_signal.kill(pid, \\\"HUP\\\")\\n if not ok then\\n ngx.log(ngx.ERR, \\\"failed to reload nginx process of pid \\\", pid, \\\": \\\", err)\\n ngx.say(\\\"Failed to reload nginx process: \\\", err)\\n ngx.status = 500\\n ngx.exit(ngx.status)\\n end\\n }\\n }\\n }\\n\\n # Metrics\\n server {\\n listen 9145;\\n access_log off;\\n location /cache_metrics {\\n content_by_lua_block {\\n metric_connections:set(ngx.var.connections_reading, {\\\"reading\\\"})\\n metric_connections:set(ngx.var.connections_waiting, {\\\"waiting\\\"})\\n metric_connections:set(ngx.var.connections_writing, {\\\"writing\\\"})\\n prometheus:collect()\\n }\\n }\\n\\n location = /metrics {\\n return 404;\\n }\\n\\n location /nginx_status {\\n stub_status on;\\n allow 127.0.0.1;\\n allow ::1;\\n deny all;\\n }\\n }\\n\"},\"kind\":\"ConfigMap\",\"metadata\":{\"annotations\":{\"argocd.argoproj.io/tracking-id\":\"nvcf-container-cache:/ConfigMap:container-caching/nvcf-default-config\"},\"labels\":{\"app\":\"nvcf-container-cache\",\"app.kubernetes.io/instance\":\"nvcf-container-cache\",\"app.kubernetes.io/managed-by\":\"Helm\",\"app.kubernetes.io/name\":\"nvcf-container-cache\",\"app.kubernetes.io/version\":\"1.2.1\",\"helm.sh/chart\":\"nvcf-container-cache-0.25.23\"},\"name\":\"nvcf-default-config\",\"namespace\":\"container-caching\"}}\n"
},
"creationTimestamp": "2025-11-13T14:18:39Z",
"labels": {
"app": "nvcf-container-cache",
"app.kubernetes.io/instance": "nvcf-container-cache",
"app.kubernetes.io/managed-by": "Helm",
"app.kubernetes.io/name": "nvcf-container-cache",
"app.kubernetes.io/version": "1.2.1",
"helm.sh/chart": "nvcf-container-cache-0.25.23"
},
"name": "nvcf-default-config",
"namespace": "container-caching",
"resourceVersion": "535192435",
"uid": "48201897-fc5f-45f5-bf86-3eb652080d55"
}
}
Loading
Loading