This is opinionated for gitlab.xiph.org, the old VM is a buster, the new one a bookworm.

Our situation is that due to some failed spam bot cleanup some legitimate users where deleted, we want to restore their issue data, but we only have an old DB backup from just before the occurence. We will restore this and then upgrade our way up to the version on "prod", after which we will restore everything except the DB to get a more complete setup, we then need to find a way to restore the objects for the problematic users(which we have to identity - we know some).

  • Get a VM
  • Follow https://about.gitlab.com/install/#debian until just before installing gitlab itself
  • Make sure to downgrade the apt sources list as needed if your OS is too new for the old gitlab version(for me bullseye worked) :/
  • Do apt update
  • Install the gitlab version from the backup i.e. if the Backupfile is named "/var/opt/gitlab/backups/1687588575_2023_06_24_16.0.5_gitlab_backup.tar" you want 16.0.5-ce.0
  • Copy over the requesite file from /etc/gitlab/config_backup/gitlab_config_2023_06_24 and extract to /etc/gitlab/
tar -C / -xf gitlab_config_*2023_06_24*.tar
  • Adapt config to new hostnames.
  • Install acmetool and run:
acmetool want newhostname.newdomain.newtld
  • Reconfigure gitlab
gitlab-ctl reconfigure
  • Adapt acmetool config to work with the now running nginx(we needed the cert so nginx would start, so it could do the redirects AAAAAAARRRGGHHH before we could have acmetool place the files \o/):
# /var/lib/acme/conf/target
request:
  challenge:
    webroot-paths:
      - /var/www/.well-known/acme-challenge
    http-self-test: false
  • Adapt cron for acmetool
# cat /etc/cron.d/acmetool
SHELL=/bin/sh
PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin
MAILTO=root
6 23 * * * root SHELL=/bin/sh
PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin
MAILTO=root
6 23 * * * root /usr/bin/acmetool --batch reconcile && gitlab-ctl restart
  • Touch /var/lib/acme/live/newhost.newdomain.newtld/revoked to force acmetool to renew
  • Run /usr/bin/acmetool --batch reconcile && gitlab-ctl restart
  • Wait half an hour until your gitlab is back up, you now have a fresh instance
  • Copy the backup and change perms
cp 1687588575_2023_06_24_16.0.5_gitlab_backup.tar /var/opt/gitlab/backups/
chown git:git /var/opt/gitlab/backups/1687588575_2023_06_24_16.0.5_gitlab_backup.tar
  • Stop processes
sudo gitlab-ctl stop puma
sudo gitlab-ctl stop sidekiq
  • Verify
sudo gitlab-ctl status
  • Perform the magic
gitlab-backup restore BACKUP=1687588575_2023_06_24_16.0.5
  • Start again and verify
gitlab-ctl status
gitlab-ctl start
  • Wait....
  • The frontpage of your instance should now look the same as on the original
  • Remember we just restored the database from an auto db backup, we have NO data!
  • This means we need to cheat a bit more and now upgrade the instance to the same version as in "prod" - currently 16.9.1-ce.0
  • To do this we need to check Upgrade Paths \o/ - follow this https://docs.gitlab.com/ee/update/
  • To get from 16.0.5 to 16.9.1 we will need to do: 16.0.8->16.1.6->16.2.9->16.3.7->16.7.6->16.9.1 \o/
  • We will first tackle 16.0.8
apt install gitlab-ce=16.0.8-ce.0
  • oh you are no admin? follow - https://forum.gitlab.com/t/how-do-i-change-my-profile-to-admin/35888
# gitlab-rails console -e production
--------------------------------------------------------------------------------
 Ruby:         ruby 3.0.6p216 (2023-03-30 revision 23a532679b) [x86_64-linux]
 GitLab:       16.0.8 (267ff69e3cc) FOSS
 GitLab Shell: 14.20.0
 PostgreSQL:   13.11
------------------------------------------------------------[ booted in 41.50s ]
Loading production environment (Rails 6.1.7.2)
irb(main):001:0>
irb(main):002:0> user = User.find_by(username: 'stephan48')
=> #<User id:633 @stephan48>
irb(main):003:0> user.admin = true
=> true
irb(main):004:0> user.save!
=> true
irb(main):005:0>
  • Check for background migrations - https://gitlab-restore.xiph.org/admin/background_migrations
  • Before the next step change the /etc/apt/sources.list.d/gitlab_gitlab-ce.list back to bookworm
  • Perform the next upgrade
apt update
apt install gitlab-ce=16.1.6-ce.0
  • Wait
  • In the meantime do a backup on "prod"
gitlab-backup create
  • Gitlab wants a redis restart after the upgrade
gitlab-ctl restart redis

  • Wait for the instance coming up and check background migrations - it actually had to do stuff here

  • Do the next step (hint you can add "-d" to already download the package while waiting...)

apt install gitlab-ce=16.2.9-ce.0
  • Restart redis again
gitlab-ctl restart redis

  • Wait for the instance coming up and check background migrations - it actually had to do stuff here

  • Do the next step (hint you can add "-d" to already download the package while waiting...)

apt install -d gitlab-ce=16.3.7-ce.0
apt install gitlab-ce=16.3.7-ce.0

  • Wait for the instance coming up and check background migrations - it actually had to do stuff here

  • Do the next step (hint you can add "-d" to already download the package while waiting...)

apt install -d gitlab-ce=16.7.6-ce.0
apt install -d gitlab-ce=16.9.1-ce.0
apt install gitlab-ce=16.7.6-ce.0
  • Redis upgrade time \o/
gitlab-ctl restart redis

  • Wait for the instance coming up and check background migrations - it actually had to do stuff here again - this repetition is getting boring? ;)

  • Do the next step (hint you can add "-d" to already download the package while waiting...)

apt install gitlab-ce=16.9.1-ce.0
  • Wait until its done
  • You are now on the same version as the "prod" instance
  • Copy over the Backup
  • Place it inside /var/opt/gitlab/backups and chown it
mv 1709683168_2024_03_05_16.9.1_gitlab_backup.tar /var/opt/gitlab/backups/
chown git: /var/opt/gitlab/backups/1709683168_2024_03_05_16.9.1_gitlab_backup.tar
  • Before doing anything more, do a backup of the restored VM before the restore
gitlab-backup create

  • We now have the backup 1709722674_2024_03_06_16.9.1_gitlab_backup.tar in /var/opt/gitlab/backups/

  • We now restore the copied backup EXCEPT the db, this gives us some more logical context for our data

gitlab-backup restore BACKUP=1709683168_2024_03_05_16.9.1 SKIP=db
  • After this is done you should have all repodata back \o/
  • Do another backup just to be safe
gitlab-backup create

  • We now get 1709724053_2024_03_06_16.9.1_gitlab_backup.tar

  • We are done \o/

We now got a point in time restore for the given old backup which was created by a gitlab before a upgrade. We only had a database backup, because thats all gitlab does.

The next steps are to find out how to reimport the previously lost issues back into the main instance. Maybe that will be another blog post.

...