Live bootstrap to guix

[vxtls]

No description provided.

[Googulator]

With CONFIG_USER_NS=y in kconfig, util-linux's tests now pass, and Guix moves on to building Valgrind.

[vxtls]

With CONFIG_USER_NS=y in kconfig, util-linux's tests now pass, and Guix moves on to building Valgrind.

Wait, what stage are you at right now? Are you on guix pull or guix system image?

[Googulator]

guix pull still, but past the initial bootstrap AKA commencement.scm

[Googulator]

Ran into and debugged another issue: when building guix-manual, I get the following error:

In unknown file:
           0 (copy-file "/gnu/store/1n4lagn25hylvrn9x9v2qjf0r0dj9sby-doc/os-config-desktop.texi" "./os-config-desktop.texi")

ERROR: In procedure copy-file:
In procedure copy-file: Permission denied

The cause: when this call is executed, ./os-config-desktop.texi already exists in the build directory, with read-only permissions.
That copy is generated from /gnu/store/fcw6rj38k5g3cwhdqqr4yfwlhfjzr81q-examples/desktop.tmpl, which in turn comes from subdirectory gnu/system/examples/ of the local repository created in preparation for calling guix pull, using the contents of the Guix 1.5.0 tarball. The upstream Guix repository also contains this file. So this copy is fine.

However, /gnu/store/1n4lagn25hylvrn9x9v2qjf0r0dj9sby-doc, on a normal Guix installation, contains no os-config-desktop.texi. Normally, the Guix channel being pulled is a fork of Guix's upstream repository in one way or the other, and that repository has an explicit gitignore set for this file. But in our case, we initialize the channel repository with the contents of the tarball, which contains some pregenerated documentation & other files, included for user convenience. And that happens to include a copy of os-config-desktop.texi in doc, which ends up conflicting with the one generated from desktop.tmpl.

Fix, option 1: in guix-daemon-and-pull.sh, insert rm doc/os-config*; rm doc/*.??*.*; rm doc/version*.texi; rm doc/stamp*; rm doc/*.1; rm doc/*.info; rm doc/guix.info*; rm doc/images/*.eps doc/images/*.pdf doc/images/bootstrap-*.png doc/images/coreutils-bag-graph.png doc/images/coreutils-graph.png doc/images/gcc-core-mesboot0-graph.png doc/images/service-graph.png doc/images/shepherd-graph.png before git init. (Ideally, do a more thorough cleanup of pregenerated files.)

Fix, option 2: Use a clone or snapshot of the Guix Git repository, rather than a release tarball intended for human use, to prepare the local channel. (Preferably I would also switch the actual build of the Guix package to be based on a Git repository, although live-bootstrap has a preference for release tarballs.)

For now, I have locally implemented option 1, and guix pull seems to be proceeding fine.

[Googulator]

Successful guix pull - moving on to ISO build.

EDIT: "disable-authentication: unrecognized option" - See Googulator/guix@ca0114e for a workaround; unfortunately this causes it to rerun the entire bootstrap :(

[Googulator]

Before the ISO build, one also needs to cp /var/guix/profiles/per-user/root/current-guix/manifest /usr/manifest - otherwise it won't be able to find the local channel.

[vxtls]

cp /var/guix/profiles/per-user/root/current-guix/manifest /usr/manifest

just to make sure, what I need to do is add this before iso build in guix-build-iso.sh? copy it to the live-bootstrap's /usr dir?

[Googulator]

Yes, exactly.

One more thing: "--disable-authentication" on "guix system image" is invalid, and will error out unless removed.

[Googulator]

During the ISO build, I got an error complaining about a missing aux-files/logo.txt file. It's in gnu/installer in the tarball & channel repo, so before building the ISO, one needs to test -d /usr/share/guile/site/3.0/gnu/installer/aux-files || cp -r /var/lib/guix/local-channels/guix/gnu/installer/aux-files /usr/share/guile/site/3.0/gnu/installer/

[Googulator]

In the "build-native-utils-earlier" patch, the procedure mesboot0-package needs to read ˙(supported-systems '("i686-linux" "x86_64-linux")) instead of just (supported-systems '("i686-linux")), otherwise ISO generation will fail, thinking that there is no way to build 64-bit packages on the bootstrapped system.

It might also be necessary to change the target in the guix pull script to x86_64-linux, to avoid running the Guix bootstrap twice.

EDIT: Oops...

[Googulator]

And... bitrot strikes.

Guix tries to download 0001-unordered-fix-copy-assign.patch from boost.org, and then check it against a specific hash. But Boost decided to "clean up" their copy of this patch, removing trailing whitespace - breaking the hash.

The original one with the right hash can still be found @ https://cdn.jsdelivr.net/npm/@openziti/ziti-sdk-nodejs@0.14.2/vcpkg/ports/boost-unordered/0001-unordered-fix-copy-assign.patch - manually copying it to /gnu/store/chzch29xi2gzdxbzk77sb432klkddra1-boost-0001-unordered-fix-copy-assign.patch seems to work (EDIT: it doesn't).

[vxtls]

The original one with the right hash can still be found @ https://cdn.jsdelivr.net/npm/@openziti/ziti-sdk-nodejs@0.14.2/vcpkg/ports/boost-unordered/0001-unordered-fix-copy-assign.patch - manually copying it to /gnu/store/chzch29xi2gzdxbzk77sb432klkddra1-boost-0001-unordered-fix-copy-assign.patch seems to work.

Should we patch Guix to fix this problem? Just like the bootstrap binary, use a local web server to serve this file

[Googulator]

A local web server is probably a good idea, yes.

[vxtls]

Wait a minute, what do you mean by "it doesn't", you mean, the file didn't match or we can't copy it to /gnu/store?

[Googulator]

Copying to the store directly didn't work. Modifying the package definition to use the alternative URL did.

But now, I'm facing a new problem, where one of the "module-import-compiled" packages fails to build, apparently because contents of the host Guile (3.0.11) leak into the Guix environment, where the local Guile (3.0.9) can't execute them. I'm still working on solving this.

[vxtls]

But now, I'm facing a new problem, where one of the "module-import-compiled" packages fails to build, apparently because contents of the host Guile (3.0.11) leak into the Guix environment, where the local Guile (3.0.9) can't execute them. I'm still working on solving this.

What's the status of this issue? One thing that puzzles me is, aren't all Guix builds supposed to run in a chroot or namespace environment?
EDIT: can i have the log?
EDIT2: Currently I am running ISO build, see if I can reproduce the problem.

[vxtls]

Copying to the store directly didn't work. Modifying the package definition to use the alternative URL did.

But now, I'm facing a new problem, where one of the "module-import-compiled" packages fails to build, apparently because contents of the host Guile (3.0.11) leak into the Guix environment, where the local Guile (3.0.9) can't execute them. I'm still working on solving this.

  GEN      etc/init.d/guix-daemon
  GEN      etc/guix-daemon.conf
  GEN      etc/guix-publish.conf
make[2]: Leaving directory '/tmp/guix-build-guix-1.5.0rc1.drv-0/source'
make[1]: Leaving directory '/tmp/guix-build-guix-1.5.0rc1.drv-0/source'
phase `build' succeeded after 4432.3 seconds
starting phase `copy-bootstrap-guile'
accepted connection from pid 8350, user nixbld
accepted connection from pid 8378, user nixbld
Backtrace:
In ice-9/boot-9.scm:
  1752:10 10 (with-exception-handler _ _ #:unwind? _ # _)
In unknown file:
           9 (apply-smob/0 #<thunk 7ffff77352a0>)
In ice-9/boot-9.scm:
    724:2  8 (call-with-prompt _ _ #<procedure default-prompt-handle…>)
In ice-9/eval.scm:
    619:8  7 (_ #(#(#<directory (guile-user) 7ffff773ac80>)))
In ice-9/command-line.scm:
   185:19  6 (_ #<input: string 7ffff7734850>)
In unknown file:
           5 (eval (begin (use-modules (guix)) (with-store store …)) #)
In ice-9/boot-9.scm:
  1752:10  4 (with-exception-handler _ _ #:unwind? _ # _)
In guix/store.scm:
   690:37  3 (thunk)
In ice-9/eval.scm:
    619:8  2 (_ #(#(#(#(#<directory (guile-user) 7ffff773a…>) …) …) …))
In unknown file:
           1 (symlink "/tmp/guix-tests/store/r26xz0l0wb2x0kil3c539f…" …)
In ice-9/boot-9.scm:
  1685:16  0 (raise-exception _ #:continuable? _)

ice-9/boot-9.scm:1685:16: In procedure raise-exception:
In procedure symlink: File exists
error: in phase 'copy-bootstrap-guile': uncaught exception:
%exception #<&invoke-error program: "./test-env" arguments: ("guile" "-c" "(begin (use-modules (guix)) (with-store store (let* ((item (add-to-store store \"guile-static-stripped-2.0.9-i686-linux.tar.xz\" #f
phase `copy-bootstrap-guile' failed after 0.6 seconds
command "./test-env" "guile" "-c" "(begin (use-modules (guix)) (with-store store (let* ((item (add-to-store store \"guile-static-stripped-2.0.9-i686-linux.tar.xz\" #f \"sha256\" \"/gnu/store/29dkam4a6sdz1pn81
build process 12 exited with status 256
bash-5.3# ls  /tmp/guix-tests
ls: cannot access '/tmp/guix-tests': No such file or directory

is it something like this?

[Googulator]

No, it was a different issue. Sourcing the Guix profile (/var/guix/profiles/per-user/root/current-guix/etc/profile) solved that, but it causes a new, strange issue where the ISO build will fail instantly when called by the main build script, or manually in the primary TTY1 terminal. Switching to TTY2 and calling it there makes it work.

I did encounter the issue you describe - I solved it differently, by making the 2nd copy operation conditional on the i686 and x86_64 bootstrap guile paths being different.

Now, I'm facing another issue, where tests fail for the "guix-1.5.0rc1.drv" derivation. The cause seems to be that it's trying to download the official bootstrap binaries to run the tests with, but is failing to do so. IMO the right way to fix this is to inject the patches we use when building our Guix into this derivation, so it too uses the locally built bootstrap binaries instead.

[vxtls]

I did encounter the issue you describe - I solved it differently, by making the 2nd copy operation conditional on the i686 and x86_64 bootstrap guile paths being different.

Where I am doing is to copy the original Bootstrap Guile directory, then add i686.txt and x86_64.txt to it, ensuring that their hash values are different. This way, guix won't fail during the copy operation because a file already exists at the destination (currently being tested; it just failed because the patch wasn't written correctly).

Now, I'm facing another issue, where tests fail for the "guix-1.5.0rc1.drv" derivation. The cause seems to be that it's trying to download the official bootstrap binaries to run the tests with, but is failing to do so. IMO the right way to fix this is to inject the patches we use when building our Guix into this derivation, so it too uses the locally built bootstrap binaries instead.

Where is the definition file for this?

And another question: Do you think we should make the generated ISO to be installed offline? (Because the default ISO image downloads external substitutes during the installation process, which undermines our assumption of a fully source-code-based build.)