-
Notifications
You must be signed in to change notification settings - Fork 247
Update Gitlab Version
Toil does CI through a Gitlab instance at ucsc-ci.com.
The instance lives in AWS EC2, in the us-west-2 region of the ucsc-platform-dev account. It is connected to a domain in AWS Route 53, by having the instance be assigned a static Elastic IP and hardcoding the domain to point to it. The instance was (manually???) configured to terminate SSL itself.
The setup was originally built out using a Terraform template for the "allspark" system, which set it up with a load balancer, but the load balancer has since been ripped out manually.
We've also applied a custom Netplan configuration to work around the AWS DHCP servers not having 100% uptime; if systemd-networkd goes to renew the DHCP lease and the server doesn't answer right away, it lets the lease expire. We have both a static and a dynamic IP set to try and keep the instance online without breaking the ability to reach it at a new IP assigned by AWS.
cat >./99-static-ip-and-dhcp.yaml <<'EOF'
network:
ethernets:
eth0:
dhcp4: true
dhcp6: false
match:
macaddress: 0a:e4:cd:73:f0:a3
set-name: eth0
addresses:
- 172.31.10.143/20
gateway4: 172.31.0.1
nameservers:
search:
- us-west-2.compute.internal
addresses: [172.31.0.2]
version: 2
EOF
netplan try --config-file 99-static-ip-and-dhcp.yaml --timeout 60
mv 99-static-ip-and-dhcp.yaml /etc/netplan/
See also:
- Gitlab CI Autoscaling Setup, where the runners are discussed
- Gitlab Object Storage, for how Gitlab stores its artifacts and caches
- Renew Gitlab License, for how and when to renew the Gitlab license
Updating gitlab for ucsc-ci.com:
-
EC2 Instance is "allspark"
-
AWS account: platform-dev
-
Region: us-west-2
-
SSH Key is an AWS secret (TODO: which?) or your SSH key might be authorized
To connect (note that the server IP has probably changes; check the AWS EC2 console!):
ssh -i "allspark.pem" ubuntu@ec2-34-220-182-252.us-west-2.compute.amazonaws.com
Or:
ssh ubuntu@ucsc-ci.com
Then run:
sudo apt-get update
If the GPG Key is out of date, you may have to refresh it before running sudo apt-get update. If you need to adjust the /etc/apt/sources/list.d files, you may need to account for the transition from .list format to .sources format.
Don't run sudo apt-get upgrade to update gitlab.
Gitlab may require multiple installs to upgrade.
Follow the upgrade paths here:
https://docs.gitlab.com/ee/update/upgrade_paths.html
To see gitlab-ee versions:
sudo apt-cache policy gitlab-ee | less
Double-check the versions. Do not try to downgrade Gitlab.
Install each version in the order described by the upgrade path and then update the server. For example:
sudo apt-get install gitlab-ee=13.0.14-ee.0
sudo gitlab-ctl reconfigure && sudo gitlab-ctl restart
sudo gitlab-ctl restart redis
sudo apt-get install gitlab-ee=13.1.11-ee.0
sudo gitlab-ctl reconfigure && sudo gitlab-ctl restart
sudo gitlab-ctl restart redis
sudo apt-get install gitlab-ee=13.8.4-ee.0
sudo gitlab-ctl reconfigure && sudo gitlab-ctl restart
sudo gitlab-ctl restart redis
Check for extra data.## directories in /var/opt/gitlab/postgresql. Each time the Gitlab PostgreSQL database server version upgrades, a copy of the data directory for the old PostgreSQL version is left behind. You don't want to leave too many of these around, especially if the database is too big. Sufficiently old ones can be removed with rm -Rf.
Check for too many old backups in /var/opt/gitlab/backups.
If you cannot find the shared SSH private key, and your own SSH public key is not authorized on the server, you can use "EC2 Instance Connect" in the AWS console to get a shell on the machine to recover from.
If you manage to type the wrong version of Gitlab to install, you might see this:
The following packages will be DOWNGRADED:
gitlab-ee
You should stop and install the right version instead.
If you abort Gitlab deb package installation in the middle, you may need to sudo dpkg --configure -a to get dpkg back into a good state.
After that, you can install Gitlab versions as normal, but you might see things like:
dpkg: warning: unable to delete old directory '/opt/gitlab/embedded/service/gitlab-rails/public/assets/sql.js/gh-pages/documentation/stylesheets': Directory not empty
Having the wrong stuff in /opt/gitlab/embedded can break the Gitlab version migration scripts, which do things like install gems in there.
You can replace the files in there with:
sudo mv /opt/gitlab/embedded /opt/gitlab/embedded.bak
sudo apt install --reinstall gitlab-ee
You might need to disable the Gitlab automatic backup and version check logic if the existing Gitlab installation is sufficiently broken, in order to let the package install. This is very dangerous if you have not already made a backup!. You can do this by creating files /etc/gitlab/skip-auto-backup and /etc/gitlab/skip-unmigrated-data-check. Remember to remove them when you are done.
Do not stop the gitlab-runsvdir systemctl service. The service has to be running for the upgrade process to work. This is because it is responsible for running the database server, and the upgrade scripts really want to talk to the database.
If you get a failure like:
* Mixlib::ShellOut::ShellCommandFailed occurred in delayed notification: runit_service[gitlab-kas] (gitlab-kas::enable line 122) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/gitlab-kas ----
STDOUT: fail: /opt/gitlab/service/gitlab-kas: runsv not running
STDERR:
---- End output of /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/gitlab-kas ----
Ran /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/gitlab-kas returned 1
(Note the runsv not running)
Then you need to start the service:
sudo systemctl start gitlab-runsvdir
Then make sure it is actually running in sudo systemctl status.
The Gitlab package installation or reconfiguration scripts might fail like this:
* Mixlib::ShellOut::ShellCommandFailed occurred in Chef Infra Client run: rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash[migrate gitlab-rails database] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) had an error: Mixlib::ShellOut::ShellCommandFailed: Command execution failed. STDOUT/STDERR suppressed for sensitive resource
It's not clear how to make the logs be revealed to you, but you can run the Rails migrations separately and see what they say:
sudo gitlab-rake db:migrate
If the result is complaints about NoMethodErrors or other things that should never happen, in Ruby code in /opt/gitlab/embedded, then you might need to fix a partial or corrupted Gitlab installation.
If you want to watch the server logs as they happen, you can run:
sudo gitlab-ctl tail
If the pull mirroring for a project is stuck in an updating state, the Sidekiq job that was doing the import may have gone away, perhaps due to running out of memory. There is a StuckProjectImportJobsWorker that runs every 24 hours to find and fix this problem, but you can run it right away:
- Go into the Gitlab Rails console:
sudo gitlab-rails console - Ask Gitlab to run the task soon:
Gitlab::Import::StuckProjectImportJobsWorker.perform_at(Time.now + 5 * 60)
This will create a job on the Scheduled tab of https://ucsc-ci.com/admin/background_jobs and after it runs, the mirroring should no longer be stuck running. (You still might not be able to manually start it right away, though.)
Sometimes, du -hs /var/opt/gitlab/postgresql will tell you that the database is quite large.
If you've already removed extra old data directories for old PostgresQL versions, you can try and vacuum tables.
You can sudo gitlab-psql to connect to the database. An unknown AI has suggested that you can run these queries to look at disk space usage and dead tuples (i.e. garbage):
SELECT
relname AS "Table",
pg_size_pretty(pg_total_relation_size(relid)) AS "Total Size",
pg_size_pretty(pg_relation_size(relid)) AS "Table Size",
pg_size_pretty(pg_total_relation_size(relid) - pg_relation_size(relid)) AS "Index Size"
FROM
pg_catalog.pg_statio_user_tables
ORDER BY pg_total_relation_size(relid) DESC
LIMIT 10;
SELECT relname, n_dead_tup, last_autovacuum, last_analyze
FROM pg_stat_user_tables
WHERE n_dead_tup > 0
ORDER BY n_dead_tup DESC;
Note that these only list the table name; you also probably need the database name:
SELECT * from pg_catalog.pg_statio_user_tables WHERE relname = 'pm_package_version_licenses';
Then you can VACUUM public.pm_package_version_licenses; to remove the dead tuples, and VACUUM FULL public.pm_package_version_licenses; to lock the table, copy all the live data to a new file, and unlock the table (which seems to be the only way to actually free disk space). These will be slow, so you will probably need to SET statement_timeout = 0; first.
The Gitlab robots.txt has wildcard rules against bots asking for things like blames, but Facebook's meta-externalagent/1.1 operating from Facebook-owned IP space seems to ignore them. It also doesn't go away even if it finds itself closing connections before getting responses.
You can prohibit it completely by adding a custom robots.txt.
sudo mkdir /etc/gitlab/custom
sudo cp /opt/gitlab/embedded/service/gitlab-rails/public/robots.txt /etc/gitlab/custom/robots.txt
Edit /etc/gitlab/custom/robots.txt:
# meta-externalagent is known to request blames despite the wildcard rule against this, so tell it to go away
User-agent: meta-externalagent
Disallow: /
Edit /etc/gitlab/gitlab.rb:
nginx['custom_gitlab_server_config'] = "\nlocation =/robots.txt { alias /etc/gitlab/custom/robots.txt; }\n"
Reconfigure Gitlab:
sudo gitlab-ctl reconfigure
Unfortunately, the bot only promises to read robots.txt every 24 hours, so you might want to block it with more nginx rules. Note that nginx if directives can really only do a couple things, like return inside a location. See https://stackoverflow.com/a/79716883 for an explanation of the way location rules match and https://nginx.org/en/docs/http/request_processing.html for information on the 444 close-the-connection magic code. Note that sets and ifs are processed in order and location rules are not, and that a location for / is already defined and can't be adjusted here. Note also that nginx does not support boolean operators in its conditionals.
nginx['custom_gitlab_server_config'] = "\nlocation =/robots.txt { alias /etc/gitlab/custom/robots.txt; }\nset $is_lost_robot 0;\nif ($http_user_agent ~* \"meta-externalagent\") { set $is_lost_robot 1; }\nif ($uri ~ \".*/robots.txt$\") { set $is_lost_robot 0 ; }\nif ($is_lost_robot = 1) { return 444; }\n"
To debug your configuration:
sudo less /var/opt/gitlab/nginx/conf/service_conf/gitlab-rails.conf
sudo less /var/log/gitlab/nginx/current