Feature: add option --disable-apt-cacher (following #249) by AbcSxyZ · Pull Request #253 · devrandom/gitian-builder

AbcSxyZ · 2021-09-16T21:34:12Z

Following #249 from @micaelmalta, this PR creates the option --disable-apt-cacher,
to disable the use of apt cacher for dependencies cache.

It also adds some documentation on the README about the option. Documentation would need at least an update for none ubuntu/debian distros.

…ocker

AbcSxyZ · 2021-09-18T18:16:14Z

Do not work on lxc, within the container, source.list is done between following lines :

gitian-builder/libexec/make-clean-vm

Line 66 in 32b1145

on-target -u root bash < target-bin/bootstrap-fixup

The file bootstrap-fixup.in contain source.list configuration, and expect to use a mirror.

Fix needed

AbcSxyZ · 2021-09-18T20:36:41Z

Just added some code to configure lxc container apt/source.list to avoid using mirror.

Within a running container, I ended up with :

deb http://archive.ubuntu.com/ubuntu trusty main universe
deb http://security.ubuntu.com/ubuntu trusty-security main universe
deb http://archive.ubuntu.com/ubuntu trusty-updates main universe

instead of :

deb http://$MIRROR_HOST:PORT/archive.ubuntu.com/ubuntu trusty main universe
deb http://$MIRROR_HOST:PORT/security.ubuntu.com/ubuntu trusty-security main universe
deb http://$MIRROR_HOST:PORT/archive.ubuntu.com/ubuntu trusty-updates main universe

Actually, it's removing references to apt cacher server, but I'm unable to test it. It looks like having lxc container connected to Internet is a bit tricky, I crash because of the internet connection on apt commands.

I'm not sure if lxc containers are more used without internet, or packages installed when building base, I don't know the software and ignore how a configuration like that should work...
I'm trying to use lxc, not necessary need it but would be great, but I'm unable to do it for now

AbcSxyZ · 2021-09-19T21:16:33Z

Add bug fix for apt cacher configuration within the Dockerfile, see following comment for details.

AbcSxyZ · 2021-09-19T22:37:18Z

libexec/config-bootstrap-fixup

+if [ $APT_CACHER = "1" ]; then
+    sed "s;HOSTIP;$MIRROR_HOST;g" < target-bin/bootstrap-fixup.in > target-bin/bootstrap-fixup
+else
+    sed "s;HOSTIP:3142/;;g" < target-bin/bootstrap-fixup.in > target-bin/bootstrap-fixup
+fi


This is actually not working.

config-boostrap-fixup is called with bin/make-base-vm, which have APT_CACHER variable through --disable-apt-cacher.
It is called also with bin/gbuild, who use it with make-clean-vm. At this moment, they do not have APT_CACHER variable anymore.

Creating the following error :

bash: [: =: unary operator expected

APT_CACHER will be replaced by an unquoted empty value.

As a possible solution, would it be possible to configure bootstrap-fixup.in only when using make-base-vm ? And then reuse generated bootstrap-fixup file instead of recreating one in make-clean-vm.

The feature was implemented by commit 5785dfc, I suppose we can do it as I suggest in the case GITIAN_HOST_IP doesn't change between make-base-vm & gbuild.
Actually, without this situation, it looks like it's creating twice the same file.

It can make sure configuration is more done through make-base-vm.
Maybe deleting the following line can do the job (?) :

gitian-builder/libexec/make-clean-vm

Line 65 in 4609325

libexec/config-bootstrap-fixup

Not sure if it's the right solution, but if it is I just added the commit d8d370d to implement this fix.

AbcSxyZ · 2021-09-20T17:24:12Z

bin/make-base-vm

    base_image="$DISTRO:$SUITE"
  fi

+  apt_cacher=""


Variable declaration useless

AbcSxyZ · 2021-10-23T15:43:16Z

Hi 👋,

I wanted to know if you are planning to review this PR ? You were really reactive for the feature, but I do not have any reply since September.

Actually, the PR need tests for LXC & KVM, both with and without --disable-apt-cacher + I had a question in a previous comment about the implementation. If anyone can test it with his build, I wasn't able to install/use lxc & kvm...

We did it while working on Dogecoin, there is a pending PR reworking the gitian process and it uses this feature, would be great to know what you want to do 🙃

devrandom · 2021-11-02T18:28:58Z

I do intend to review, but have to allocate some time for testing with the different configurations...

devrandom

I found a couple of issues right away.

devrandom · 2021-11-22T09:16:49Z

libexec/config-bootstrap-fixup

 fi

-sed "s;HOSTIP;$MIRROR_HOST;g" < target-bin/bootstrap-fixup.in > target-bin/bootstrap-fixup
+if [ $APT_CACHER = "1" ]; then


two issues:

missing quotes around $APT_CACHER

APT_CACHER is not actually exported as an environment variable

devrandom · 2021-11-22T09:23:34Z

Thinking about this further, and given that Bitcoin is moving to a different build system, I am planning to move this repo to maintenance mode. Only serious bugs and security issues will be considered going forward.

AbcSxyZ · 2021-11-22T11:39:14Z

Can you tell me what is this new build system, please ? I do not find information related to this change, in the current gitian docs from bitcoin-core repo is still related to your repo, and it's redirecting to a gitian-build.py on bitcoin/bitcoin which do not exists anymore.

EDIT: saw your readme indicating guix, thx.

micaelmalta and others added 5 commits July 13, 2021 23:42

Docker apt-cacher

f9aa177

make-base-vm: switch apt cacher option, using cacher enabled by default

0b10618

make-base-vm: allow disabling apt cacher for lxc & kvm

f9a993e

apt-cacher: reset default mirror and avoid use of host network with d…

f58f681

…ocker

apt-cacher: add readme doc for --disable-apt-cacher

68dd756

AbcSxyZ requested a review from devrandom as a code owner September 16, 2021 21:34

apt-cacher: avoid use of host network with docker

7568db8

AbcSxyZ changed the title ~~Feature: disable apt cacher default (following #249)~~ Feature: add option --disable-apt-cacher (following #249) Sep 17, 2021

apt-cache: avoid use mirror with lxc

3793b65

AbcSxyZ mentioned this pull request Sep 19, 2021

Docs4Doge: update of gitian builder script + documentation (follow #2396) dogecoin/dogecoin#2579

Closed

apt-cacher: fix configuration error of the Dockerfile

525e33a

AbcSxyZ commented Sep 19, 2021

View reviewed changes

apt-cacher: avoid configuration of bootstrap-fixup.in in make-clean-vm

d8d370d

AbcSxyZ commented Sep 20, 2021

View reviewed changes

bin/make-base-vm

base_image="$DISTRO:$SUITE"

fi

apt_cacher=""

Copy link

Contributor Author

AbcSxyZ Sep 20, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Variable declaration useless

AbcSxyZ mentioned this pull request Nov 2, 2021

Open items for 1.14.5 release dogecoin/dogecoin#2655

Closed

27 tasks

devrandom reviewed Nov 22, 2021

View reviewed changes

devrandom closed this Nov 22, 2021

AbcSxyZ mentioned this pull request Nov 27, 2021

gitian-builder: add patches to disable apt-cacher dogecoin/dogecoin#2723

Closed

Conversation

AbcSxyZ commented Sep 16, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AbcSxyZ commented Sep 18, 2021

Uh oh!

AbcSxyZ commented Sep 18, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AbcSxyZ commented Sep 19, 2021

Uh oh!

AbcSxyZ Sep 19, 2021

Choose a reason for hiding this comment

Uh oh!

AbcSxyZ Sep 20, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AbcSxyZ Sep 20, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AbcSxyZ Sep 20, 2021

Choose a reason for hiding this comment

Uh oh!

AbcSxyZ commented Oct 23, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

devrandom commented Nov 2, 2021

Uh oh!

devrandom left a comment

Choose a reason for hiding this comment

Uh oh!

devrandom Nov 22, 2021

Choose a reason for hiding this comment

Uh oh!

devrandom commented Nov 22, 2021

Uh oh!

AbcSxyZ commented Nov 22, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

AbcSxyZ commented Sep 16, 2021 •

edited

Loading

AbcSxyZ commented Sep 18, 2021 •

edited

Loading

AbcSxyZ Sep 20, 2021 •

edited

Loading

AbcSxyZ Sep 20, 2021 •

edited

Loading

AbcSxyZ commented Oct 23, 2021 •

edited

Loading

AbcSxyZ commented Nov 22, 2021 •

edited

Loading