Install a fresh GC stack using "recover from backup"

[IamJeffG]

Goal

Avoid the many classes of transient API errors we see from gc-stack-deploy by making the full GC stack install operation more or less atomic from the user's point of view.

Closes #30

Benefits:

• no longer requires caprover CLI on the VM (caprover package not available on VM upon fresh install with Azure template #122)
• may be run anywhere -- on VM or not. You just need to get the generated backup.tar file onto the VM.

What I changed and why

Adds a new Python project gen_backup. It

1. packages its own template "backup" files
2. reads the same stack.yaml as gc-stack-deploy where user can configure their new deployment
3. injects those variables from stack.yaml into the template file
4. creates a TAR of the modified files.

Then outside the Python project, this TAR can be used when installing caprover to pre-populate all the apps, env vars, volume mounts. I added the bare-bones instructions to a new README.

Notes to Reviewers

Best intro to this PR is to read https://github.com/ConservationMetrics/gc-deploy/blob/install-from-caprover-backup/caprover/gen-backup/README.md

How I convinced myself this is right

New unittests ensure we're doing variable substitution correctly.

Additions to the makefile allow you to spin up a new caprover locally from backup.

What I have NOT yet done are (1) review the script code itself, (2) ensure the stack that gets deployed is fully working: the Docker services are running but I have not dug into the services further than this.

What I'm not doing here

Concerns with this approach, not easily resolved:

• gc-stack-deploy interspersed postgresql database setup with app deployment: deploy postgres; CREATE DATABASE warehouse; deploy explorer; create windmill users; deploy windmill. But we cannot do that from within the backup; restoring from backup is from our point of view, an atomic operation.
  • Need a follow-up script, and it would also mean that some of the new deployment's apps will be failing until that database DDL is done. ("Recover from backup" does not populate data volumes)
  • This can get a bit weird. Say we want windmill to run as non-superuser.. You'd create the backup with the new user that doesn't exist yet, to prevent windmill from being able to login to postgres at all until the finalizer script is done. The finalizer script creates the user and the database as that user, at which Windmill would begin to work.
• Not sure how easily you can find the filebrowser admin password (TODO: check)

Things to probably change (without a lot of effort):

• cli.py basically bakes in a translation layer between one-click-app variables vs env variable names, of which the one-click-app defns need to be the source of truth. This needs to be DRY'd.
• I was not able to reliably enable SSL on brand new domains from backup. After launching caprover to recover from backup, you will need to go in to the Caprover web portal and enable and force SSL on all domains, or we can write a script that uses the API to do this.
• DRY with gc-stack-deploy, and I think the ultimate solution is to unify them into a single Python package, possibly even with gen-backup REPLACING most of the stack_deploy.py script. Just one example: gen-backup relies on the stack.yaml written by gc-stack-deploy init, so for now you do need to install both packages.

LLM use disclosure

Claude code wrote most of this code, I haven't looked closely at the code (yet) -- want to ensure this POC is worth future investment first.

[rudokemper]

I think I am missing something:

In the readme, I note the section IV. Mock a "backup" file that contains your stack and install Caprover from it which provides two steps to run gen-backup and then start the container: https://github.com/ConservationMetrics/gc-deploy/blob/install-from-caprover-backup/caprover/gen-backup/README.md#iv-mock-a-backup-file-that-contains-your-stack-and-install-caprover-from-it

Does that docker incantation already "import" the backup? Is that because gen-backup constructs config-captain.json and other files and puts them in the right directory, which CapRover will implicitly use when starting up for the first time?

Or is there an extra step in the CapRover UI or CLI to import the backup TAR?

(Might be worth clarifying this in one of the documentation files, in addition to, or as a substitution for, answering my question here on the issue.)

[IamJeffG]

There is no extra step. The docker run will start caprover and initialize the app stack all at the same time. (Basically caprover's first-time startup knows to look for a backup.tar and if it finds it it installs it.) I can make that more clear.

My next steps for today:

1. add a derivative PR that does what install-from-backup cannot: database setup, SSL
2. confirm this all works for a real stack deploy on a fresh cloud VM, i.e. the apps start correctly and reach their database

[IamJeffG]

On one hand, this is a nicer experience not having to deal with Caprover-API and server errors and timeouts beget by doing everything iteratively. The TAR generation and subsequent install and caprover and the apps of the Guardian Connector stack "just works" every time -- up to the limitations we already know of:

• no SSL
• no ability to customize the database (especially users and roles)
• all apps must be installed in one go.

Reluctantly, it's exactly for those limitations I don't think we should keep this the way forward.

• The necessary follow-up is another manual step
• Tying database server deployment to application deployment is limiting, it will disallow us from doing things we might want to do.
• it's "too big" an operation, so if it did happen to fail, there's no recovery short of blowing it all away and starting over.

Instead, I believe our focus should be on

1. hardening gc-stack-deploy as an interative experience (which this effort would have replaced), which has benefits of maintaining fidelity to the one-click-app defintiions and not introducing the shortcomings above.
2. Documenting a broader plan to streamline the entire new-VM deployment -- most friction is not in gc-stack-deploy but all the clicking to set up auth0, and in post-hoc Windmill configuration.

---

I will close this without merge. I'm also merging the intended follow-up branch into this branch -- the semantic distinction starts with commit 71481f0. That and follow-up is the "finalize" step.