Skip to content

Commit 903580c

Browse files
Code2Collapseclaude
andcommitted
Add C2C Farm render farm (port of feat 3f6e4bf..6ee59b0)
Tractor-style multi-cloud spooler + dashboard: SQLite audit log (async writer), priority queue with per-user/backend caps, storage couriers (S3/R2/MinIO, Azure Blob, GCS - lazy), ComfyUI-native backend adapter with compute-profile gateway routing + websocket progress/preview tap, pre-flight node validation, C2C_Submit / C2C_ClusterStatus / C2C_JobHistory nodes, /c2c/farm/* routes, C2C Farm sidebar dashboard, farm Docker image + ACR/ECR CI template, 26 pytest cases. Verified on feat: 26/26 tests, live loopback end-to-end (dispatch -> render -> result download -> audit row), pre-flight negative test, dashboard in-browser. Worktree port: py_compile 25 files + js + tests green against main. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 parent 5a18963 commit 903580c

33 files changed

Lines changed: 2827 additions & 0 deletions

.github/workflows/build_docker.yml

Lines changed: 82 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,82 @@
1+
# C2C Farm backend image — CI template.
2+
# Builds the full-ComfyUI farm image (renderfarm/docker/Dockerfile + the
3+
# pinned custom_nodes_manifest.txt) and pushes to ACR and/or ECR.
4+
#
5+
# TEMPLATE: wire the secrets below in repo settings before enabling.
6+
# ACR: ACR_LOGIN_SERVER, ACR_USERNAME, ACR_PASSWORD
7+
# ECR: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_REGION, ECR_REPOSITORY
8+
# Runs on manual dispatch (images are ~20GB — don't build per-push).
9+
10+
name: build-c2c-farm-backend-image
11+
12+
on:
13+
workflow_dispatch:
14+
inputs:
15+
comfyui_ref:
16+
description: "ComfyUI git ref to bake"
17+
default: "master"
18+
required: true
19+
image_tag:
20+
description: "Image tag"
21+
default: "latest"
22+
required: true
23+
push_acr:
24+
description: "Push to Azure Container Registry"
25+
type: boolean
26+
default: true
27+
push_ecr:
28+
description: "Push to AWS ECR"
29+
type: boolean
30+
default: false
31+
32+
jobs:
33+
build:
34+
runs-on: ubuntu-latest
35+
steps:
36+
- uses: actions/checkout@v4
37+
38+
- uses: docker/setup-buildx-action@v3
39+
40+
- name: Login to ACR
41+
if: ${{ inputs.push_acr }}
42+
uses: docker/login-action@v3
43+
with:
44+
registry: ${{ secrets.ACR_LOGIN_SERVER }}
45+
username: ${{ secrets.ACR_USERNAME }}
46+
password: ${{ secrets.ACR_PASSWORD }}
47+
48+
- name: Configure AWS credentials
49+
if: ${{ inputs.push_ecr }}
50+
uses: aws-actions/configure-aws-credentials@v4
51+
with:
52+
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
53+
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
54+
aws-region: ${{ secrets.AWS_REGION }}
55+
56+
- name: Login to ECR
57+
if: ${{ inputs.push_ecr }}
58+
id: ecr
59+
uses: aws-actions/amazon-ecr-login@v2
60+
61+
- name: Compute tags
62+
id: tags
63+
run: |
64+
TAGS=""
65+
if [ "${{ inputs.push_acr }}" = "true" ]; then
66+
TAGS="${{ secrets.ACR_LOGIN_SERVER }}/comfy-c2c-farm-backend:${{ inputs.image_tag }}"
67+
fi
68+
if [ "${{ inputs.push_ecr }}" = "true" ]; then
69+
[ -n "$TAGS" ] && TAGS="$TAGS,"
70+
TAGS="$TAGS${{ steps.ecr.outputs.registry }}/${{ secrets.ECR_REPOSITORY }}:${{ inputs.image_tag }}"
71+
fi
72+
echo "tags=$TAGS" >> "$GITHUB_OUTPUT"
73+
74+
- name: Build & push
75+
uses: docker/build-push-action@v6
76+
with:
77+
context: .
78+
file: renderfarm/docker/Dockerfile
79+
push: true
80+
tags: ${{ steps.tags.outputs.tags }}
81+
build-args: |
82+
COMFYUI_REF=${{ inputs.comfyui_ref }}

.gitignore

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -231,3 +231,10 @@ _deprecated/
231231

232232
# MEC error assistant runtime data
233233
user/
234+
235+
# C2C Farm render-farm runtime data (SQLite audit log — WAL siblings too)
236+
renderfarm/config/audit.db
237+
renderfarm/config/audit.db-wal
238+
renderfarm/config/audit.db-shm
239+
# but the C2C Farm render-farm test suite IS part of the source tree
240+
!renderfarm/tests/

__init__.py

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -520,8 +520,23 @@ def _register_tcl_routes(_server): # type: ignore
520520
group="nodes",
521521
)
522522

523+
# ── C2C Farm render-farm spooler (Tractor-style multi-cloud dispatch) ──
524+
_FARM_MAPPINGS, _FARM_DISPLAY = {}, {}
525+
try:
526+
from .nodes import render_farm as _farm
527+
_FARM_MAPPINGS.update(_farm.NODE_CLASS_MAPPINGS)
528+
_FARM_DISPLAY.update(_farm.NODE_DISPLAY_NAME_MAPPINGS)
529+
except Exception as _farm_exc: # pragma: no cover
530+
_c2c_rec_fail(
531+
"render_farm", _farm_exc,
532+
hint="C2C Farm spooler (C2C_Submit / ClusterStatus / JobHistory); "
533+
"check renderfarm/config/*.json is intact.",
534+
group="nodes",
535+
)
536+
523537
NODE_CLASS_MAPPINGS = {
524538
**_FOLDER_MAPPINGS,
539+
**_FARM_MAPPINGS,
525540
**_FLUID_MAPPINGS,
526541
**_MEC_MAPPINGS,
527542
**_MA_MAPPINGS,
@@ -551,6 +566,7 @@ def _register_tcl_routes(_server): # type: ignore
551566
}
552567
NODE_DISPLAY_NAME_MAPPINGS = {
553568
**_FOLDER_DISPLAY,
569+
**_FARM_DISPLAY,
554570
**_FLUID_DISPLAY,
555571
**_MEC_DISPLAY,
556572
**_MA_DISPLAY,

0 commit comments

Comments
 (0)