vSphere 5.1 upgrade phase 3: Converting to the VCSA & Wrap-Up
In the previous parts of this upgrade project, I discussed upgrading vCenter itself followed by upgrading the hosts & VMs themselves and I'll end with converting my Windows vCenter install to VMware's own Linux-based vCenter Server Appliance.
Before we jump into converting anything, we need to do some pre-requisite work. I'm converting to the VCSA by using the VCS-to-VCSA Converter, which is a Fling for converting a compatible Windows-based vCenter server to the VCSA. Looking at the requirements for this Fling, my vCenter DB will need to be on a remote SQL server so I'll need to move my vCenter DB from my local vCenter server to one of our other SQL servers temporarily. (I say temporarily because the VCS-to-VCSA fling will migrate our remote SQL DB into the appliance's own vPostgres SQL DB.)
Once the DBs are moved over and everything on your vCenter server is pointed at the remote SQL server, you're ready to start up your vCenter services and stop/disable the SQL services local to your vCenter server. Restart the vCenter services in the reverse order from how you stopped them (IE, make sure the VirtualCenter service is started first) and you're done.
Before we jump into converting anything, we need to do some pre-requisite work. I'm converting to the VCSA by using the VCS-to-VCSA Converter, which is a Fling for converting a compatible Windows-based vCenter server to the VCSA. Looking at the requirements for this Fling, my vCenter DB will need to be on a remote SQL server so I'll need to move my vCenter DB from my local vCenter server to one of our other SQL servers temporarily. (I say temporarily because the VCS-to-VCSA fling will migrate our remote SQL DB into the appliance's own vPostgres SQL DB.)
MIgrating The DBs To A Remote SQL Server
There's 2 parts to this that happen kind of simultaneously. First is that I need to "move" the DBs by taking a full backup and restoring the DB to the new server and then point vCenter services to the DB on the remote SQL server. My steps for doing that are as follows. (Before jumping in, I highly recommend making a clone of your vCenter VM before doing anything else. I think having a solid backout plan in case your change goes sideways is just as important than having a good change plan any day. Making mistakes is perfectly fine; Not being able to recover from them is not. Food for thought.)
Before we begin, you'll need to 1) Take a full backup of your vCenter DB and (and really any other DB you're moving) and 2) Stop all vCenter Server related services, stopping the VirtualCenter service last. VMware has a KB article for this if you're not sure how to go about stopping services though I didn't find it particularly helpful. If you've ever stopped a Windows service before, you're already an expert.
Moving the Databases
Before we begin, you'll need to 1) Take a full backup of your vCenter DB and (and really any other DB you're moving) and 2) Stop all vCenter Server related services, stopping the VirtualCenter service last. VMware has a KB article for this if you're not sure how to go about stopping services though I didn't find it particularly helpful. If you've ever stopped a Windows service before, you're already an expert.
Stop the DBs you're going to move (assuming the only DBs on the vCenter server are the vCenter DBs you want to move.) According to VMware, on the remote SQL server you'll want to go ahead and create some new, blank databases. Then, restore the backed up vcenter DBs to the new, blank DBs with the option of "Overwrite existing database with replace."
The part about creating new DBs and overwriting with your backups was a bit confusing to my supervisor and I but it did come from VMware directly in KB 7960893, which I also linked to below. What ended up being easier for us to do was to use System Center Data Protection Manager (DPM) to restore the backed up database to the remote SQL server, which was how I was protecting vCenter for file-level backups prior to the migration. You should be able to follow the same procedure for restoring a DB using your tool of choice. Doing a full restore to another server didn't work for me so I restored the MDF and LDF files to a network location on the SQL server, attached them to the SQL server and recreated the login for the DB. It's a little scary the first time you do it but it's very simple actually.
Modifying the ODBC Data Source
Prior to changing the ODBC DSN connection information, you'll need to move restore the backed up DB over to your remote SQL server and configure the SQL login for that DB (This is so that when you go through with setting up the new DSN connection, it'll have something to actually connect to.) With the backup of our DBs complete, the vCenter services stopped, DB restored to the remote SQL server, open the 64-bit OBDC Connections and replace the current SQL server with the new server name and named instance (if you're using a named instance). Select the name, click the Configure button and go through the wizard to complete the change. (Since my VUM DB has always been on a remote server and I run VUM services on a separate server from my vCenter server, I'm not going through the motions of moving the VUM DB. However, if you need to move the VUM DB in your environment, you'll want to make the same change as described above in the 32-bit ODBC Connections).
Changing the SSO DB Location
This is actually where I ran into some confusion and had to open a ticket with VMware so I made sure I wasn't about to ruin my SSO config. Nearly all of the official and unofficial information I found for updating the SSO configuration all harken back to KB 2045528 and can be summed up as "run this ssocli command and then check these two specific files in their specific locations to see if the change you made actually stuck." My problem was that when I went poking around my vCenter server, I couldn't find either if the specific directories mentioned, nor a trace of the files I needed to verify. According to the VMware Support rep who assisted me, the files and directories specified in KB 2045528 were for vCenter SSO for v5.1, and since I'm on v5.5, I should be using KB 1028601, which doesn't give any specific info for updating the SSO configuration. In v5.5, the SSO and vCenter DBs are one (as opposed to two separate DBs like they were for v5.1), so no additional modification is required for updating the configuration for SSO for v5.5. KB 2045528 only applies for vSphere v5.1.
Changing the vCenter WebServices DB Location
Now that the vCenter DB changes are out of the way, let's open explorer, browse to C:\ProgramData\VMware\VMware VirtualCenter\ and open the vcdb.properties (assuming "vcdb" is the name of your vCenter database). We'll want to change the following entry to reflect the new DB modifications.
url=jdbc:sqlserver://NewDBHostname;databaseName\=database_name;integratedSecurity\=true
Now that the DB is on the new SQL server and everything on the vCenter server is pointing at the new SQL server, we're almost there.
Verifying the Jobs and Stored Procedures.
vCenter has jobs and stored procedures handled by the SQL Server Agent for things like cleaning out old data, moving data between tables, etc. Instructions for verifying everything is still there can be found in KB 2033096. If they didn't survive the move, or they don't look correct, you may need to delete and recreate them before calling it done. To do that, reference KB 1004382.
Once the DBs are moved over and everything on your vCenter server is pointed at the remote SQL server, you're ready to start up your vCenter services and stop/disable the SQL services local to your vCenter server. Restart the vCenter services in the reverse order from how you stopped them (IE, make sure the VirtualCenter service is started first) and you're done.
Links I found helpful for the DB migration steps above are
Vlad Madaras' guide to changing the vCenter DB (ignore the part about updating the SSO configuration if you're on v5.0 (which didn't have SSO) or if you're on a version of vCenter newer than v5.1. SSO was only it's own DB in v5.1.)
Before you make the jump to the VCSA, make sure you know what else your vCenter server is serving aside from just vCenter. My group in the university is rather large and supports the IT infrastructure for several smaller groups. Some of those smaller groups are very independent and have their own cluster managed by our vCenter server. Others are less independent and we support all the things for them. The more independent group uses Horizon View clients in their environment. Horizon View is VMware's VDI solution. Horizon View has clients, agents, a connection broker and a composer service. In the DB move described above, I broke View Composer service because it's DB was on the local SQL server. (who's services I disabled after the move. We used to have View clients in our environment so i thought that View Composer DB was our old one and no longer being used. My bad.) We got it fixed in less than an hour (most of which was figuring out it was their DB. Our documentation wasn't up to date.)
In order to proceed with migrating to the VCSA, the View Composer application and SQL database had to be moved. The plan was to set it up just like I did with VUM. VUM lives on a miscellaneous server, with a database on a remote SQL server. VUM has an ODBC DSN connection to it's remote SQL database, and it can access vCenter, even if it's not co-located with vCenter on the same server. The good news here if you have something similar is that the View Composer is a relatively simple installation and you're a pro at moving DBs to remote SQL servers now.
Because nothing ever goes perfect...
Before you make the jump to the VCSA, make sure you know what else your vCenter server is serving aside from just vCenter. My group in the university is rather large and supports the IT infrastructure for several smaller groups. Some of those smaller groups are very independent and have their own cluster managed by our vCenter server. Others are less independent and we support all the things for them. The more independent group uses Horizon View clients in their environment. Horizon View is VMware's VDI solution. Horizon View has clients, agents, a connection broker and a composer service. In the DB move described above, I broke View Composer service because it's DB was on the local SQL server. (who's services I disabled after the move. We used to have View clients in our environment so i thought that View Composer DB was our old one and no longer being used. My bad.) We got it fixed in less than an hour (most of which was figuring out it was their DB. Our documentation wasn't up to date.)
In order to proceed with migrating to the VCSA, the View Composer application and SQL database had to be moved. The plan was to set it up just like I did with VUM. VUM lives on a miscellaneous server, with a database on a remote SQL server. VUM has an ODBC DSN connection to it's remote SQL database, and it can access vCenter, even if it's not co-located with vCenter on the same server. The good news here if you have something similar is that the View Composer is a relatively simple installation and you're a pro at moving DBs to remote SQL servers now.
Using the VCS-to-VCSA Converter Fling
If you've never heard of Flings from VMware Labs, you should check out what people have made. Flings are side projects from engineers at VMware and others in the VMware community. It's where I found the VCS to VCSA Fling, which is what I'll be using to migrate my Windows-hosted vCenter server to the Linux-based vCenter Server Appliance. I can't link directly to the instructions for using the Fling but they're downloadable in a DOC format.
Before we get to migrating anything, download the VCS_to_VCSA_Converter.OVA and deploy it in your environment. Then, download the VCSA OVA from VMware (taking care to make sure you're downloading the same version of vCenter, EG, v5.5 u2e) and deploy that with the same hostname & IP Address as your Windows vCenter Server (but don't turn it on yet). You'll also want to collect some information from your environment. Account credentials & such. You'll need
Before we get to migrating anything, download the VCS_to_VCSA_Converter.OVA and deploy it in your environment. Then, download the VCSA OVA from VMware (taking care to make sure you're downloading the same version of vCenter, EG, v5.5 u2e) and deploy that with the same hostname & IP Address as your Windows vCenter Server (but don't turn it on yet). You'll also want to collect some information from your environment. Account credentials & such. You'll need
- An IP Address to assign to the converter appliance (optionally, you could use DHCP)
- SSO Admin credentials
- Windows Administrator credentials for the Windows vCenter Server
- A root password to assign to the VCSA during configuration (the default is "vmware" sans quotes)
- Credentials that vCenter uses to access it's database (in my environment, a service account is used to login to the DB)
Once you have these, you're ready to rock & roll.
When you first power on the converter appliance, you'll be asked to configure it's networking. You can use DCHP or you can set everything manually. Personally, I like static IPs for this sort of thing.
Once that's done, the next screen will start the migration wizard. If you haven't already deployed your destination VCSA, now's the time to do so. Once deployed, take a snapshot in case problems happen during the migration. This way, you can jump back to a clean slate if the migration gets hairy. Remember not to power on your VCSA yet.
With your VCSA destination appliance deployed and snapshotted, you can move on to the next step in the migration wizard. It'll ask you for the hostname or IP of the source Windows vCenter server and credentials for a Windows Administrator account (not to be confused with a vCenter Administrator). Once supplied, the migration appliance will pull over some vCenter data and config information.
Once it's done, you'll be prompted to turn off the source Windows vCenter Server and power on the VCSA you deployed earlier. After hitting next, the migration appliance will wait for the VCSA to finish booting. Once it's ready, the migration appliance will ask you to verify the SSH key presented by the VCSA, If it looks good to you then click "Yes" to proceed.
Next, you'll need to enter the root password you configured for the VCSA. (If you didn't change it, it's "vmware", sans quotes, by default.) After that, you'll specify the DNS Domain Suffix (eg yourorgization.com). You can update this or delete it if you don't want to specify one at this time.
For me, after deploying the VCSA, it came up with no network configured, even though I'd specified the network settings during the OVA deployment process. I had to manually set the network settings each time, as shown in the above screenshot. While it wasn't difficult to do, I had to do it multiple times, as explained later.
The next step is to log in to your destination VCSA and do some basic configuration. Accept the EULA, joining the VMware Customer Experience Program or not, etc. You'll want to select the "Set Custom Configuration" and click Next. Set the default "Embedded" for the database and click Next. Set the default "embedded" SSO deployment type. Enter the password for the SSO Administrator. If you're using AD (which, you likely are), and you'd like to join the VCSA to your domain, tick the box for "Active Directory Enabled" and enter your AD domain & domain admin credentials. (You may need to remove the old AD object from your AD database).
Once the VCSA has finished configuring, close the browser tab you have to the VCSA and ,back in the migration appliance, select OK to the next step. Once a connection can be made to the VCSA, you'll be prompted to accept the fingerprint of the SSL certificate. Click "Yes" to continue.
Next, you'll need to enter the SSO Administrator credentials. The migration appliance will pull over some SSL certs from the source Windows vCenter server. If you've joined the VCSA to your AD domain, you have the option of configure the AD domain as an Identity Source in SSO. I don't see why you wouldn't do this unless you only use local logins in vCenter. Your environment may vary. This is usually a manual step performed in the VCSA configuration but the migration appliance can handle it by merely confirming the AD domain name is correct.
At this point, the migration appliance will be asking you for the DB information. You'll need to specify the hostname or IP Address of your remote SQL server hosting the vCenter DB. You'll also need to specify the DB name, schema name (only if it's not dbo) and credentials for the DB (I'm using a service account but your environment may vary). This is where I messed up the credentials and botched my DB migration the first time. Be extra sure you know what account vCenter is using to access the DB. I initially thought it just wasn't taking the credentials but after a few tries, I got taken to a screen that told me a migration was already in progress. It promptly failed after that. I reverted to my snapshot and soldiered on. You'll also have the option of including statistics, events and tasks along with the rest of your DB information. (Checking this box will make the migration longer and if you have more than 100k entries in these tables, the makers of the Fling recommend not migrating that data.) During this step, all of the data from the SQL database will be copied to the embedded vPostgres database running on the VCSA. How long this takes varies on the size of your DB, if you're including stats/events/tasks, etc.
My vCenter DB migration took FOREVER! Once I had the DB credentials sorted out, I finally got the DB migration going about noonish. It finished somewhere between 7PM that night and 1AM the next morning. (that range is for VM-level backup jobs that run and the one at 7PM failed but the larger string of backup jobs that I have running between 1AM and 4AM all ran fine.) My vCenter DB was 88Gb by the way.
When it's finished, you'll be brought back to the beginning of the migration wizard. Before logging into the vSphere Web Client, start all your plugins you disabled in the beginning. In my case that's just VUM and View Composer which I have running on a separate server as talked about in my post in phase 2. (If you're running NSX, you may need to reconnect the plugin to vCenter after starting the plugin.) The first time you login to the vSphere Web Client, it will take a little longer because it will have to download the plugin from the vSphere Web Server.
Once it's done, you'll be prompted to turn off the source Windows vCenter Server and power on the VCSA you deployed earlier. After hitting next, the migration appliance will wait for the VCSA to finish booting. Once it's ready, the migration appliance will ask you to verify the SSH key presented by the VCSA, If it looks good to you then click "Yes" to proceed.
Next, you'll need to enter the root password you configured for the VCSA. (If you didn't change it, it's "vmware", sans quotes, by default.) After that, you'll specify the DNS Domain Suffix (eg yourorgization.com). You can update this or delete it if you don't want to specify one at this time.
For me, after deploying the VCSA, it came up with no network configured, even though I'd specified the network settings during the OVA deployment process. I had to manually set the network settings each time, as shown in the above screenshot. While it wasn't difficult to do, I had to do it multiple times, as explained later.
The next step is to log in to your destination VCSA and do some basic configuration. Accept the EULA, joining the VMware Customer Experience Program or not, etc. You'll want to select the "Set Custom Configuration" and click Next. Set the default "Embedded" for the database and click Next. Set the default "embedded" SSO deployment type. Enter the password for the SSO Administrator. If you're using AD (which, you likely are), and you'd like to join the VCSA to your domain, tick the box for "Active Directory Enabled" and enter your AD domain & domain admin credentials. (You may need to remove the old AD object from your AD database).
Once the VCSA has finished configuring, close the browser tab you have to the VCSA and ,back in the migration appliance, select OK to the next step. Once a connection can be made to the VCSA, you'll be prompted to accept the fingerprint of the SSL certificate. Click "Yes" to continue.
Next, you'll need to enter the SSO Administrator credentials. The migration appliance will pull over some SSL certs from the source Windows vCenter server. If you've joined the VCSA to your AD domain, you have the option of configure the AD domain as an Identity Source in SSO. I don't see why you wouldn't do this unless you only use local logins in vCenter. Your environment may vary. This is usually a manual step performed in the VCSA configuration but the migration appliance can handle it by merely confirming the AD domain name is correct.
At this point, the migration appliance will be asking you for the DB information. You'll need to specify the hostname or IP Address of your remote SQL server hosting the vCenter DB. You'll also need to specify the DB name, schema name (only if it's not dbo) and credentials for the DB (I'm using a service account but your environment may vary). This is where I messed up the credentials and botched my DB migration the first time. Be extra sure you know what account vCenter is using to access the DB. I initially thought it just wasn't taking the credentials but after a few tries, I got taken to a screen that told me a migration was already in progress. It promptly failed after that. I reverted to my snapshot and soldiered on. You'll also have the option of including statistics, events and tasks along with the rest of your DB information. (Checking this box will make the migration longer and if you have more than 100k entries in these tables, the makers of the Fling recommend not migrating that data.) During this step, all of the data from the SQL database will be copied to the embedded vPostgres database running on the VCSA. How long this takes varies on the size of your DB, if you're including stats/events/tasks, etc.
My vCenter DB migration took FOREVER! Once I had the DB credentials sorted out, I finally got the DB migration going about noonish. It finished somewhere between 7PM that night and 1AM the next morning. (that range is for VM-level backup jobs that run and the one at 7PM failed but the larger string of backup jobs that I have running between 1AM and 4AM all ran fine.) My vCenter DB was 88Gb by the way.
When it's finished, you'll be brought back to the beginning of the migration wizard. Before logging into the vSphere Web Client, start all your plugins you disabled in the beginning. In my case that's just VUM and View Composer which I have running on a separate server as talked about in my post in phase 2. (If you're running NSX, you may need to reconnect the plugin to vCenter after starting the plugin.) The first time you login to the vSphere Web Client, it will take a little longer because it will have to download the plugin from the vSphere Web Server.
Wrap-Up
I have some documentation to write/update since I've made some big changes to our virtual infrastructure throughout this project. I wrote my about my experience through this work so I can hopefully help some other SysAdmin who needs to do similar work. If I help even one person, it was all worth it. This project took me 6+ months to go through but a lot of it was planning and delays due to other projects competing for my time. It's really weird to put all this work into something nobody can see or touch but it definitely exists and aside from my team and the IT departments my team supports, nobody will even notice much of a difference, sadly. Regardless, I put a ton of sweat equity into this, learned a ton and am much more comfortable working with VMware. This project has been awesome and I'm very proud. My boss, Dave helped me out a lot, my team was very understanding while I took down vCenter more times than I care to admit and my wife listened to me talk about this for months. I may as well have been speaking Klingon but she listened nonetheless. My virtual infrastructure is not done yet. I've still got plans to resolve some storage issues and implement Storage-DRS, deploy a SysLog appliance, and add some cool new plugins to our web-client. Those are all posts for another day though. I'd like to thank you for reading my rambling posts and I hope you enjoyed following along.
Comments
Post a Comment