From 2d76149270a6f2fd5c79b49eb853855d44b182e7 Mon Sep 17 00:00:00 2001 From: Dimitar Atanasov Date: Wed, 20 May 2026 11:08:33 +0300 Subject: [PATCH 1/5] fix(bash): env.sh fails silently on first run when secrets.env / resolved.env are missing MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit env.sh used the `[[ -f file ]] && source file` short-circuit pattern to conditionally load two optional config files. When either file does not exist, the failed `[[ ]]` test returns exit 1 and the `&&` chain inherits that status. Because env.sh is the last file sourced through common.sh from every numbered script (which all use `set -euo pipefail`), the non-zero return propagates back up the source chain and triggers `errexit` in the parent script — silently, before any command can print output. The bug fires on every truly clean first-run install, which is exactly the flow the README promises will work. It was not caught in development because the maintainers' working trees always had `secrets.env` left over from previous runs, so the test never failed. Replace the short-circuit with explicit `if [[ ]]; then ... fi` blocks. The `if` construct is specifically exempt from `set -e` propagation, so a missing optional file no longer kills the parent. Reproduces on bash 3.2.57 (Apple stock) and bash 5.3.9 (Homebrew); affects all Linux + macOS + WSL/Git Bash users. Native Windows users using the PowerShell scripts are unaffected. --- scripts/bash/env.sh | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/scripts/bash/env.sh b/scripts/bash/env.sh index 1abb58e..2715da9 100644 --- a/scripts/bash/env.sh +++ b/scripts/bash/env.sh @@ -49,7 +49,13 @@ export CONTAINER_NAME="${CONTAINER_NAME:-lightchain-worker}" # AI_CONFIG_ADDRESS and JOB_REGISTRY_ADDRESS get populated by 01-resolve-addresses.sh # and persisted to scripts/bash/resolved.env. We source that file here if present. _SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" -[[ -f "$_SCRIPT_DIR/resolved.env" ]] && source "$_SCRIPT_DIR/resolved.env" +if [[ -f "$_SCRIPT_DIR/resolved.env" ]]; then + # shellcheck source=/dev/null + source "$_SCRIPT_DIR/resolved.env" +fi # Secrets are loaded from scripts/bash/secrets.env (gitignored). -[[ -f "$_SCRIPT_DIR/secrets.env" ]] && source "$_SCRIPT_DIR/secrets.env" +if [[ -f "$_SCRIPT_DIR/secrets.env" ]]; then + # shellcheck source=/dev/null + source "$_SCRIPT_DIR/secrets.env" +fi From 4a8800a007ebd5921428a6ef6c99934a7c69fe1e Mon Sep 17 00:00:00 2001 From: Dimitar Atanasov Date: Wed, 20 May 2026 11:23:12 +0300 Subject: [PATCH 2/5] fix(bash): mark phase + ops scripts as executable MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The bash scripts shipped as mode 100644, so on a fresh clone every direct invocation (./scripts/bash/00-generate-key.sh, status.sh, tail-jobs.sh, etc.) fails with `zsh: permission denied`. Users had to either `chmod +x scripts/bash/*.sh` manually or fall back to `bash ./scripts/bash/...` — neither is documented in the README, both contradict the documented one-liner usage. Mark the 14 scripts the README invokes directly as 100755. Leave the three files that are only sourced or copied at 100644: - common.sh: sourced from every phase script - env.sh: sourced from common.sh - secrets.example.sh: copied (not executed) by 00-generate-key.sh --- scripts/bash/00-generate-key.sh | 0 scripts/bash/01-resolve-addresses.sh | 0 scripts/bash/02-prepare-ollama.sh | 0 scripts/bash/03-pull-image.sh | 0 scripts/bash/04-import-key.sh | 0 scripts/bash/05-generate-ecdh.sh | 0 scripts/bash/06-fund-worker.sh | 0 scripts/bash/07-register.sh | 0 scripts/bash/08-run-worker.sh | 0 scripts/bash/deregister.sh | 0 scripts/bash/status.sh | 0 scripts/bash/stop.sh | 0 scripts/bash/sweep-rewards.sh | 0 scripts/bash/tail-jobs.sh | 0 14 files changed, 0 insertions(+), 0 deletions(-) mode change 100644 => 100755 scripts/bash/00-generate-key.sh mode change 100644 => 100755 scripts/bash/01-resolve-addresses.sh mode change 100644 => 100755 scripts/bash/02-prepare-ollama.sh mode change 100644 => 100755 scripts/bash/03-pull-image.sh mode change 100644 => 100755 scripts/bash/04-import-key.sh mode change 100644 => 100755 scripts/bash/05-generate-ecdh.sh mode change 100644 => 100755 scripts/bash/06-fund-worker.sh mode change 100644 => 100755 scripts/bash/07-register.sh mode change 100644 => 100755 scripts/bash/08-run-worker.sh mode change 100644 => 100755 scripts/bash/deregister.sh mode change 100644 => 100755 scripts/bash/status.sh mode change 100644 => 100755 scripts/bash/stop.sh mode change 100644 => 100755 scripts/bash/sweep-rewards.sh mode change 100644 => 100755 scripts/bash/tail-jobs.sh diff --git a/scripts/bash/00-generate-key.sh b/scripts/bash/00-generate-key.sh old mode 100644 new mode 100755 diff --git a/scripts/bash/01-resolve-addresses.sh b/scripts/bash/01-resolve-addresses.sh old mode 100644 new mode 100755 diff --git a/scripts/bash/02-prepare-ollama.sh b/scripts/bash/02-prepare-ollama.sh old mode 100644 new mode 100755 diff --git a/scripts/bash/03-pull-image.sh b/scripts/bash/03-pull-image.sh old mode 100644 new mode 100755 diff --git a/scripts/bash/04-import-key.sh b/scripts/bash/04-import-key.sh old mode 100644 new mode 100755 diff --git a/scripts/bash/05-generate-ecdh.sh b/scripts/bash/05-generate-ecdh.sh old mode 100644 new mode 100755 diff --git a/scripts/bash/06-fund-worker.sh b/scripts/bash/06-fund-worker.sh old mode 100644 new mode 100755 diff --git a/scripts/bash/07-register.sh b/scripts/bash/07-register.sh old mode 100644 new mode 100755 diff --git a/scripts/bash/08-run-worker.sh b/scripts/bash/08-run-worker.sh old mode 100644 new mode 100755 diff --git a/scripts/bash/deregister.sh b/scripts/bash/deregister.sh old mode 100644 new mode 100755 diff --git a/scripts/bash/status.sh b/scripts/bash/status.sh old mode 100644 new mode 100755 diff --git a/scripts/bash/stop.sh b/scripts/bash/stop.sh old mode 100644 new mode 100755 diff --git a/scripts/bash/sweep-rewards.sh b/scripts/bash/sweep-rewards.sh old mode 100644 new mode 100755 diff --git a/scripts/bash/tail-jobs.sh b/scripts/bash/tail-jobs.sh old mode 100644 new mode 100755 From 01f28370cece42099f5f2a726a9bd1b4246397ce Mon Sep 17 00:00:00 2001 From: Kostadin Buglov Date: Wed, 20 May 2026 13:05:21 +0300 Subject: [PATCH 3/5] Update ollama default host --- scripts/powershell/env.ps1 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/powershell/env.ps1 b/scripts/powershell/env.ps1 index d8f8192..820ff5e 100644 --- a/scripts/powershell/env.ps1 +++ b/scripts/powershell/env.ps1 @@ -40,7 +40,7 @@ $env:SUPPORTED_MODELS = "llama3-8b" # tends to lock onto the IPv6 address and the Ollama health-check ends up # stuck (worker_ollama_up stays at 0). Pinning to the IPv4 address that Docker # Desktop exposes for the Windows host fixes that. -$env:OLLAMA_URL = "http://192.168.65.254:11434" +$env:OLLAMA_URL = "http://host.docker.internal:11434" # Local paths. The Windows host folder is mounted into the container at /data. $env:KEYS_DIR = "$env:USERPROFILE\lightchain-worker\keys" From 5934cb8ea635474de509000f8f5c04efb06a4b17 Mon Sep 17 00:00:00 2001 From: Dimitar Atanasov Date: Wed, 20 May 2026 14:31:19 +0300 Subject: [PATCH 4/5] docs(operations): expand 'Upgrading the worker image' with NETWORK export + 01-resolve MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The previous three-line stop/pull/run snippet was correct but assumed the operator was in the same shell session as their original onboarding. In practice, operators frequently come back to upgrade days or weeks later in a fresh terminal where NETWORK is no longer exported — the scripts then silently default to mainnet and pull the wrong image / connect to the wrong gateway. This change: * Adds an explicit NETWORK re-export as step 1 so the procedure works from any shell. * Adds 01-resolve-addresses as a defensive step 2 — idempotent in the happy path (just re-writes resolved.env with the same on-chain addresses), self-healing if resolved.env got nuked, and protective against the rare scenario where Lightchain replaces a proxy. * Documents what every other step would do and why no earlier phase (00, 02, 04, 05, 06, 07) needs to be re-run — pre-empts the natural question "if 01 then why not all of them". * Adds a 'Verifying the new image' subsection listing the three health-signal log lines to look for after restart. * Adds a 'Checking whether there's actually a new image to pull' subsection with the manifest-inspect + RepoDigest commands so operators can compare digests without doing a full pull. No code change. --- docs/operations.md | 23 +++++++++++++++++++++-- 1 file changed, 21 insertions(+), 2 deletions(-) diff --git a/docs/operations.md b/docs/operations.md index 4182786..ac84300 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -221,21 +221,40 @@ Then re-run from Phase 01: ## Upgrading the worker image -Lightchain occasionally pushes new versions of the worker image. To upgrade: +Lightchain occasionally pushes new versions of the worker image. First, check whether there's actually a new image to pull (so you don't restart the container for no reason): + +```bash +# Registry's current :latest digest (use the mainnet URL if you're on mainnet) +curl -sSI "https://us-central1-docker.pkg.dev/v2/lightchain/lightchain-testnet-public-docker/worker/manifests/latest" \ + -H "Accept: application/vnd.docker.distribution.manifest.list.v2+json" \ + -H "Accept: application/vnd.oci.image.index.v1+json" \ + -H "Accept: application/vnd.docker.distribution.manifest.v2+json" \ + | awk 'tolower($1)=="docker-content-digest:" {print $2}' | tr -d '\r' + +# Your running container's image digest +docker image inspect "$(docker inspect lightchain-worker --format '{{.Image}}')" \ + --format '{{range .RepoDigests}}{{.}}{{"\n"}}{{end}}' | head -1 +``` + +If the digests match, you're already current — nothing to do. If they differ, upgrade: ```powershell +$env:NETWORK = "testnet" # or "mainnet" — required if this is a fresh terminal +.\scripts\powershell\01-resolve-addresses.ps1 .\scripts\powershell\stop.ps1 .\scripts\powershell\03-pull-image.ps1 .\scripts\powershell\08-run-worker.ps1 -NoTail ``` ```bash +export NETWORK=testnet # or mainnet — required if this is a fresh terminal +./scripts/bash/01-resolve-addresses.sh ./scripts/bash/stop.sh ./scripts/bash/03-pull-image.sh ./scripts/bash/08-run-worker.sh --no-tail ``` -No re-registration needed — your on-chain state is independent of the binary version. +No re-registration needed — your on-chain state is independent of the binary version. No other onboarding phase (00, 02, 04-07) needs to be re-run. ## Multi-worker on one machine From 4c4987aab04e897e0aeeac727124e2c6fec61993 Mon Sep 17 00:00:00 2001 From: Kris Date: Wed, 20 May 2026 15:11:39 +0300 Subject: [PATCH 5/5] docs(macos): warn that scripts require bash 4+ (macOS ships 3.2) macOS's default /bin/bash is 3.2 due to GPLv3 licensing. Two scripts (04-import-key.sh, 06-fund-worker.sh) use the ${var,,} lowercase expansion introduced in bash 4.0 and fail with `bad substitution` on a stock Mac, leaving the operator in a half-done state. - README: blockquote callout in the Linux/macOS quick-start - docs/installation-macos.md: required-software row + install step + verification line + gotcha entry with the actual error message - docs/installation-linux.md: bash 4.0+ row with a note that any mainstream distro from the last decade satisfies it --- README.md | 2 ++ docs/installation-linux.md | 3 +++ docs/installation-macos.md | 26 ++++++++++++++++++++++++++ 3 files changed, 31 insertions(+) diff --git a/README.md b/README.md index c3e5277..ab6d628 100644 --- a/README.md +++ b/README.md @@ -99,6 +99,8 @@ $env:FUNDER_PRIVKEY = "0xYOUR_FUNDER_KEY" ### Linux / macOS (Bash) +> **macOS:** Apple ships bash 3.2 (from 2007). Two of the scripts use bash 4+ syntax and will error with `bad substitution`. Install a modern bash before running the toolkit: `brew install bash && echo 'export PATH="/opt/homebrew/bin:$PATH"' >> ~/.zshrc && source ~/.zshrc`. See [docs/installation-macos.md](docs/installation-macos.md) for details. + ```bash # One-time install (Ubuntu/Debian) sudo apt-get install -y docker.io diff --git a/docs/installation-linux.md b/docs/installation-linux.md index bf07ddf..e250b16 100644 --- a/docs/installation-linux.md +++ b/docs/installation-linux.md @@ -10,6 +10,9 @@ Tested on **Ubuntu 24.04 LTS** with an NVIDIA RTX 4070. Should work identically | **NVIDIA Container Toolkit** | latest | Only if you want Ollama in Docker (we won't) | | **Ollama** | ≥ 0.5 | Local LLM runtime | | **Foundry (cast)** | ≥ 1.7.1 | EVM wallet ops, contract reads | +| **Bash** | ≥ 4.0 | Required for several scripts | + +> Bash 4.0+ is required, but any mainstream distro from the last decade satisfies this (Ubuntu ≥ 10.04, RHEL ≥ 7, etc.). Run `bash --version` to confirm if unsure. ## Install steps (Ubuntu / Debian) diff --git a/docs/installation-macos.md b/docs/installation-macos.md index 26d8efb..ffd66f0 100644 --- a/docs/installation-macos.md +++ b/docs/installation-macos.md @@ -24,6 +24,7 @@ If you're on an 8 GB Mac, you can run the worker but expect to lose timeouts on | **Docker Desktop** | ≥ 4.30 | Runs the worker container | `brew install --cask docker` | | **Ollama** | ≥ 0.5 | Local LLM runtime | `brew install ollama` | | **Foundry (cast)** | ≥ 1.7.1 | EVM wallet ops, contract reads | `curl -L https://foundry.paradigm.xyz \| bash && foundryup` | +| **Bash** | ≥ 4.0 | Required for several scripts | `brew install bash` | ## Install steps @@ -44,6 +45,12 @@ curl -L https://foundry.paradigm.xyz | bash ~/.foundry/bin/foundryup echo 'export PATH="$HOME/.foundry/bin:$PATH"' >> ~/.zshrc source ~/.zshrc + +# Modern bash (macOS ships bash 3.2; several scripts use bash 4+ syntax) +brew install bash +echo 'export PATH="/opt/homebrew/bin:$PATH"' >> ~/.zshrc # Apple Silicon +# On Intel Macs use: echo 'export PATH="/usr/local/bin:$PATH"' >> ~/.zshrc +source ~/.zshrc ``` ## Post-install verification @@ -54,10 +61,29 @@ docker ps # should return an empty table ollama --version ollama list # may be empty cast --version # cast Version: 1.7.1 or newer +bash --version # GNU bash, version 5.x ``` ## macOS-specific gotchas +### Apple's `/bin/bash` is version 3.2 — too old for the toolkit + +macOS hasn't shipped a newer bash since 2007 because bash 4+ is GPLv3 (Apple won't bundle GPLv3 code). Several scripts (e.g. `04-import-key.sh`, `06-fund-worker.sh`) use the `${var,,}` lowercase parameter expansion introduced in bash 4.0. On the stock bash you'll see: + +``` +./04-import-key.sh: line 35: ${reported,,}: bad substitution +``` + +The error fires **after** the keystore is already imported but before the address-match sanity check, leaving you in a half-done state with no clear diagnostic. Install a modern bash via Homebrew (see "Install steps" above) and make sure it's first on your `PATH`: + +```bash +which bash # should print /opt/homebrew/bin/bash (Apple Silicon) + # or /usr/local/bin/bash (Intel) +bash --version # should print 5.x +``` + +If `which bash` still returns `/bin/bash` after install, prepend the Homebrew bin dir to `PATH` in `~/.zshrc` and open a new terminal tab. + ### `host.docker.internal` works out of the box Unlike Windows, Docker Desktop for Mac sets up `host.docker.internal` resolution cleanly and uses IPv4 by default. The toolkit's default `OLLAMA_URL=http://host.docker.internal:11434` works without any tweaks.