Migrating & Upgrading Gitlab from 8.x to 10.x
As part of my inheritance at a new role, I've got a Gitlab server running on an Ubuntu VM. The program I support at the college has student teams managing their projects on this Git server. I was hoping to wait until next summer to migrate to a different Git solution (like a University-hosted GitHub Enterprise solution that's been floated around), but due to security concerns, my Gitlab server can't wait that long. In this post, I need to update the Gitlab package and move it to a new VM. I'll be updating to Gitlab v10 and moving the entire thing to a RHEL v7 VM in our University's hosting environment.
The process of simply moving from one server to another is relatively simple and is well documented in Gitlab's Documentation. A lot of the difficulty comes from the fact that anything you can export (whether it's an individual repository or a full backup tarball) can only be imported to the EXACT same version, type, and install. If your source server is version 8.16.3, Community Edition, Omnibus installation, then your destination server has to be the same version, type, and same installation method.
The version number can be found in the Admin page of your git server's web interface, as shown below. This will also indicate the type, Community Edition or Enterprise Edition, CE or EE, respectively. If you're using the CE, then your version number will be just X.X.X. If you're using the EE, then your version number will look like X.X.X-ee.
In order to verify the installation method (omnibus vs source) then look in /etc/gitlab for a gitlab.rb file. This is the main configuration file for the omnibus installations. From my understanding, if you can find the gitlab.rb file in /etc/gitlab, then it's an omnibus installation.
Initially, I had the idea of taking a backup on the old server and restoring on the new server. That's when I found out about the stipulation that backups from a Gitlab server can ONLY be restored on a Gitlab server of the same version, type, and install-method.
Second, I had the idea of exporting the repositories individually from the source server and using the import feature to pull in the exported repositories to the new server. This would work fine in my case since my git server only has ~100 users, 1 group, and 20ish projects that I need to keep. Moving the project data is what's important to me. I can manually recreate the user accounts and the group. That idea failed as well. This too WOULD work, if the import/export tools are the same versions.
Third, I had the idea of just copying the Git Data manually from the source server to the destination server, using SCP or RSYNC. The default location for the repositories are in /var/opt/gitlab/git-data/repositories. While the operation doesn't error out, you will not see the repositories you moved over showing up in the web interface, even after restarting Gitlab. I had hope that if I could dump the old DB using pg-dump and use the resulting SQL file to write the data to the new DB. After a LOT of Googling and trial & error, I decided to give up hope on this plan.
I finally decided that installing the old version on my new server, restoring a backup from the source server, and then upgrading the destination server with all of the data would be my path of least resistance.
Installing an older version isn't difficult once you figure it out. Go HERE and choose the instructions for your distro up to the point of actually installing gitlab. In my case, I'm using RHEL/CentOS v7.
Install the prerequisites using the commands below.
sudo yum install -y curl policycoreutils-python openssh-server
sudo systemctl enable sshd
sudo systemctl start sshd
sudo firewall-cmd --permanent --add-service=http
sudo systemctl reload firewalld
sudo yum install postfix
sudo systemctl enable postfix
sudo systemctl start postfix
The given code above for adding the package repo references Gitlab-ce, which is what it'll try to install when you go to apply updates with your package manager. If you're using Gitlab-EE, you'll need change. If you already ran this command for one type and you need to install the other, you can delete the gitlab_gitlab-XX.repo file from /etc/yum.repos.d and rerun the command above to add the proper Gitlab package repository.
Don't forget to run the following command to configure Gitlab after installation!
sudo gitlab-ctl reconfigure
At this point, I'm assuming you've already got a backup tarball from your source server. If you don't, the backup process is relatively straightforward, well documented, and linked above. The restore process isn't particularly tricky, but I wanted add some explanation in addition to the existing documentation.
First, you'll need to stop the unicorn and sidekiq Gitlab services. The commands to do so will be below along with a command to verify that they're down.
sudo gitlab-ctl stop unicorn
sudo gitlab-ctl stop sidekiq
sudo gitlab-ctl status
How to Determine The Version, Type, & Installation
The version number can be found in the Admin page of your git server's web interface, as shown below. This will also indicate the type, Community Edition or Enterprise Edition, CE or EE, respectively. If you're using the CE, then your version number will be just X.X.X. If you're using the EE, then your version number will look like X.X.X-ee.
In order to verify the installation method (omnibus vs source) then look in /etc/gitlab for a gitlab.rb file. This is the main configuration file for the omnibus installations. From my understanding, if you can find the gitlab.rb file in /etc/gitlab, then it's an omnibus installation.
What Did NOT Work
Initially, I had the idea of taking a backup on the old server and restoring on the new server. That's when I found out about the stipulation that backups from a Gitlab server can ONLY be restored on a Gitlab server of the same version, type, and install-method.
Second, I had the idea of exporting the repositories individually from the source server and using the import feature to pull in the exported repositories to the new server. This would work fine in my case since my git server only has ~100 users, 1 group, and 20ish projects that I need to keep. Moving the project data is what's important to me. I can manually recreate the user accounts and the group. That idea failed as well. This too WOULD work, if the import/export tools are the same versions.
Third, I had the idea of just copying the Git Data manually from the source server to the destination server, using SCP or RSYNC. The default location for the repositories are in /var/opt/gitlab/git-data/repositories. While the operation doesn't error out, you will not see the repositories you moved over showing up in the web interface, even after restarting Gitlab. I had hope that if I could dump the old DB using pg-dump and use the resulting SQL file to write the data to the new DB. After a LOT of Googling and trial & error, I decided to give up hope on this plan.
I finally decided that installing the old version on my new server, restoring a backup from the source server, and then upgrading the destination server with all of the data would be my path of least resistance.
Installing an Older version of Gitlab
Installing an older version isn't difficult once you figure it out. Go HERE and choose the instructions for your distro up to the point of actually installing gitlab. In my case, I'm using RHEL/CentOS v7.
Install the prerequisites using the commands below.
sudo yum install -y curl policycoreutils-python openssh-server
sudo systemctl enable sshd
sudo systemctl start sshd
sudo firewall-cmd --permanent --add-service=http
sudo systemctl reload firewalld
sudo yum install postfix
sudo systemctl enable postfix
sudo systemctl start postfix
Add the Gitlab repository by running
curl https://packages.gitlab.com/install/repositories/gitlab/gitlab-ce/script.rpm.sh | sudo bash
The given code above for adding the package repo references Gitlab-ce, which is what it'll try to install when you go to apply updates with your package manager. If you're using Gitlab-EE, you'll need change. If you already ran this command for one type and you need to install the other, you can delete the gitlab_gitlab-XX.repo file from /etc/yum.repos.d and rerun the command above to add the proper Gitlab package repository.
Based on your Gitlab type (CE vs EE), go to the Gitlab Package Archive and find the version that matches your source server. At the time of this writing, 10.2.0 is the latest version.
When you've selected your package to download, you'll be given the option to add the repository and install the package through your package manager. It may have been user error on my end but doing that didn't work, so I opted to download the RPM file, move to my destination server via SCP, and install the local package with yum.
Don't forget to run the following command to configure Gitlab after installation!
sudo gitlab-ctl reconfigure
Restoring a Backup of your Gitlab server
At this point, I'm assuming you've already got a backup tarball from your source server. If you don't, the backup process is relatively straightforward, well documented, and linked above. The restore process isn't particularly tricky, but I wanted add some explanation in addition to the existing documentation.
First, you'll need to stop the unicorn and sidekiq Gitlab services. The commands to do so will be below along with a command to verify that they're down.
sudo gitlab-ctl stop unicorn
sudo gitlab-ctl stop sidekiq
sudo gitlab-ctl status
The output will look similar to the image below.
Next, we want to restore from our backup. I took mine from my source server and SCP'd it to the same directory on my destination server. Your backup tarball needs to be in whatever directory is specified in your gitlab.rb file. The default is /var/opt/gitlab/backups.
To kick off a restore, you'll need to use the following command.
sudo gitlab-rake gitlab:backup:restore BACKUP=##########_[year]_[month]_[day]
I initially had some trouble because I was specifying the the full file name, rather than just the backup. In my case, my working backup restore command looked like
sudo gitlab-rake gitlab:backup:restore BACKUP=1510180042_2017_11_08
Obviously, you'll need to specify the proper backup on your system. The output should look similar to this.
The overall restore time will vary depending on a wide variety of variables. How big the backup is will be one of the largest factors. Once the backup is finished unpacking, you'll be prompted with a warning that this will overwrite all of the tables in the embedded PostgreSQL DB. You'll need to type "yes" as opposed to "y".
It's worth noting that the restore procedure doesn't seem to affect your gitlab.rb configuration file. You'll want to SCP that from your source server, so you can pick n' pull bits that you need from it. In my case, all I really need from it was the LDAP configuration. As mentioned previously, you'll need to reconfigure Gitlab using the command below.
sudo gitlab-ctl reconfigure
After you've verified that the backup worked, take a moment to celebrate that it's working.
If you're running Gitlab on a virtual machine, now would be a good time to make a checkpoint/snapshot/etc. We might break it again.
Upgrading Gitlab will be the easy part. Use your package manager to upgrade to the latest release available in your repo. In my case, it's as simple as
yum upgrade
If you're upgrading from a similarly older version of Gitlab to v10+, you'll see an error message saying that your version of PostgreSQL is too old, like the image below.
In order to resolve this, we'll need to upgrade PostgreSQL, which can be done following Gitlab's documentation. As per the documentation, I'll run the command below.
sudo gitlab-ctl pg-upgrade
The output should look similar to the screenshot below.
The pg-upgrade process will also run a gitlab-ctl reconfigure. It's worth getting in the habit of checking the status of gitlab's services using the gitlab-ctl status command afterwards.
When we try a upgrading again, it shouldn't have any trouble now. It will likely take a while during the upgrade, sitting at the spot below.
Once complete, the upgrade process will reconfigure Gitlab. When checking the status (remember our habit, mentioned above), you'll see some new services running that we haven't seen before.
At this point, you should now be able to access the web interface of your Gitlab install and check the Admin panel. You can celebrate that it's the latest version and you should see all of your users, groups, and projects. W00t!
It's worth noting that the restore procedure doesn't seem to affect your gitlab.rb configuration file. You'll want to SCP that from your source server, so you can pick n' pull bits that you need from it. In my case, all I really need from it was the LDAP configuration. As mentioned previously, you'll need to reconfigure Gitlab using the command below.
sudo gitlab-ctl reconfigure
After you've verified that the backup worked, take a moment to celebrate that it's working.
If you're running Gitlab on a virtual machine, now would be a good time to make a checkpoint/snapshot/etc. We might break it again.
Upgrading Gitlab & PostgreSQL
Upgrading Gitlab will be the easy part. Use your package manager to upgrade to the latest release available in your repo. In my case, it's as simple as
yum upgrade
If you're upgrading from a similarly older version of Gitlab to v10+, you'll see an error message saying that your version of PostgreSQL is too old, like the image below.
In order to resolve this, we'll need to upgrade PostgreSQL, which can be done following Gitlab's documentation. As per the documentation, I'll run the command below.
sudo gitlab-ctl pg-upgrade
The output should look similar to the screenshot below.
The pg-upgrade process will also run a gitlab-ctl reconfigure. It's worth getting in the habit of checking the status of gitlab's services using the gitlab-ctl status command afterwards.
When we try a upgrading again, it shouldn't have any trouble now. It will likely take a while during the upgrade, sitting at the spot below.
Once complete, the upgrade process will reconfigure Gitlab. When checking the status (remember our habit, mentioned above), you'll see some new services running that we haven't seen before.
At this point, you should now be able to access the web interface of your Gitlab install and check the Admin panel. You can celebrate that it's the latest version and you should see all of your users, groups, and projects. W00t!
Comments
Post a Comment