vSphere 5.1 upgrade phase 1: Uninstalling Heartbeat
As our new virtualization admin, I've taken over the day-to-day administration of our UCS & vSphere infrastructure. I'm also in charge of the upgrade project for it. We're currently rockin' vSphere v5.1 on 2x Windows Server 2k8r2 vCenter VMs in a Heartbeat v6.6 configuration. We want to get to a single vCenter Server Appliance (VCSA) v5.5. I've broken this up into 3 distinct phases.
In the first phase, we need to get to a single vCenter configuration. That means tearing down the Heartbeat configuration & decommissioning our secondary vCenter VM. In the second phase, I'll do a simple upgrade to the vCenter server to v5.5, followed by the hosts, VM hardware version & VMtools upgrade (with either VUM or pounding out something in PowerCLI). In the third phase, I'll use the VCS to VCSA fling to convert our Windows vCenter server to the Linux-based VCSA.
Last week, I began phase 1 of our upgrade by uninstalling Heartbeat v6.6 & decommissioning our secondary vCenter VM. In typical bad decision fashion, I decided to perform this new procedure on a Friday afternoon on the week of my boss's 3-week vacation. Uninstalling HB was relatively straight-forward. I used this KB article from VMware. What I didn't realize was that when the uninstallation wizard asks you if you want to restart the server, there's no delay. The server immediately restarted and I was unable to login to vCenter for a solid 45 minutes. If i'd had a lump of coal during that time, I could've made a diamond. Pucker factor 11.
It came back up but SSO wasn't working properly. One of our teammates could login but I couldn't. When I tried logging in via the web client using Windows session credentials like I always do, I'd get an error saying "The authentication server returned an unexpected error: ns0:RequestFailed: Internal Error while creating SAML 2.0 Token. The error may be caused by a malfunctioning identity source." After poking around a bit, I was able to login to the fat client but not the web client. Deeming it no longer an emergency, I went home for the weekend.
On Monday, I contacted VMware Support (putting in a ticket was painless and the TC that helped me was great.) We looked at some logs, got in via the SSO admin account & re-added the identity source. After that, I was able to login if I typed out my credentials rather than ticking the "Use Windows Session Credentials" box like we always do. This pointed to an issue with the vCenter client integration plugin, which is what passes the session credentials to the web client. The fact that I could login without issue into the fat client supports this. We uninstalled & reinstalled the client integration plugin but no cigar. The tech said he'd said he'd do some research on his end and get back to me. When he got back to me, he said the my version of vCenter had the first release of SSO which had some weaknesses. He suggested we upgrade which we're already planning to do.
Last week, I began phase 1 of our upgrade by uninstalling Heartbeat v6.6 & decommissioning our secondary vCenter VM. In typical bad decision fashion, I decided to perform this new procedure on a Friday afternoon on the week of my boss's 3-week vacation. Uninstalling HB was relatively straight-forward. I used this KB article from VMware. What I didn't realize was that when the uninstallation wizard asks you if you want to restart the server, there's no delay. The server immediately restarted and I was unable to login to vCenter for a solid 45 minutes. If i'd had a lump of coal during that time, I could've made a diamond. Pucker factor 11.
It came back up but SSO wasn't working properly. One of our teammates could login but I couldn't. When I tried logging in via the web client using Windows session credentials like I always do, I'd get an error saying "The authentication server returned an unexpected error: ns0:RequestFailed: Internal Error while creating SAML 2.0 Token. The error may be caused by a malfunctioning identity source." After poking around a bit, I was able to login to the fat client but not the web client. Deeming it no longer an emergency, I went home for the weekend.
On Monday, I contacted VMware Support (putting in a ticket was painless and the TC that helped me was great.) We looked at some logs, got in via the SSO admin account & re-added the identity source. After that, I was able to login if I typed out my credentials rather than ticking the "Use Windows Session Credentials" box like we always do. This pointed to an issue with the vCenter client integration plugin, which is what passes the session credentials to the web client. The fact that I could login without issue into the fat client supports this. We uninstalled & reinstalled the client integration plugin but no cigar. The tech said he'd said he'd do some research on his end and get back to me. When he got back to me, he said the my version of vCenter had the first release of SSO which had some weaknesses. He suggested we upgrade which we're already planning to do.
For now, I'm considering phase one complete and I've learned not to piss off the SSO gods.
Comments
Post a Comment