A Multi-Site vSphere/SDDC Upgrade – A Step by Step Guide

I wanted to share with you the steps required to upgrade a typical multi-site vSphere/ SDDC implementation. This particular implementation had the following products.

 

Product	Current version	Planned Version
vCenter Server	6.0 U3	6.5 U1
ESXi	6.0 U3	6.5 U1
vSAN	6.2	6.6.1
NSX	6.2.4	6.3.5
vROPS	6.3.0	6.6.1
vRLI	3.3.1	4.5
vDP	6.1.0.173	6.1.5
vSphere Replication	6.1.1	6.5.1
VMware Tools	6.0 U3	6.5 U1

 

In order to understand the upgrade order here is some background information on this fictitious environment.

The environment has a Production site and a DR site. NSX is deployed across both sites using universal objects, SRM and vSphere Replication are in use with vSAN being used locally at each site. It also has a deployment of vROPS and vRLI at both sites.

Note. The following should be taken as a rough guide only please make sure you check the appropriate VMware Guides/KB’s for your product versions before commencing any upgrade.

	Action	Site	Impact to vSphere	Required	VM downtime
1.	Carry out Health Check of VC & PSC before starting (Go or No Go)	Both Sites	N/A	Yes	No
2.	Backup All Components	Both Sites	N/A	Yes	No
3.	Backup PSC’s with vDP & Data Protect	Both Sites	N/A	Yes	No
4.	Deploy second PSC at each site – Configure replication see KB 2131191 for justification	Both Sites	vCenter management of ESXi hosts unavailable during upgrade.	Yes	No
5.	Disable vSphere Replication	Both Sites	No protection of VMs (Backup is the only method of restoring)	Yes	No
6.	Upgrade the external Platform Services Controller server 6.0.x to vCenter 6.5 for both sites	Both Sites	vCenter management of ESXi hosts unavailable during upgrade.	Yes (if using an external Platform Services Controller)	No
7.	Upgrade vDP at both sites	Both Sites	Backups unavailable during upgrade	Yes in order to restore in to 6.5 during the upgrade	No
8.	Upgrade NSX Manager at the DR Site	DR Site	No Changes to DR NSX during Upgrade	Yes	No
9.	Upgrade NSX Controller Cluster at the DR Site	DR Site	NSX reverts to read only mode Change Window Required	Yes	Yes – No Disruption as long as VM’s don’t move or any changes made
10.	Upgrade NSX Host preperation at the DR Site	DR Site	Hosts require Reboot	Yes	No
11.	Upgrade NSX DLR’s at the DR Site	DR Site	Disruption to Service	Yes	Yes
12.	NSX Edges at the DR Site	DR Site	Outage required while edge is redeployed and upgraded	Yes	Yes
13.	Upgrade vCenter from vCenter 6.0.x to vCenter 6.5. at the DR Site	DR Site	vCenter management of ESXi hosts unavailable during upgrade.	Yes	No
14.	Upgrade vROPS	Both Sites	N/A	Yes	No
15.	Upgrade Log Insight	Both Sites	N/A	Yes	No
16.	Use vSphere Update Manger to scan and remediate an ESXi host.	DR Site	1 host not available, reduced capacity.	Yes	No
17.	Repeat steps for remaining hosts in the cluster.	DR Site	One host will always be unavailable while it is being upgraded with vSphere Update Manager.	Yes	No
18.	Upgrade vSAN at the DR Site	DR Site	Possible Performance and increased risk of failure during upgrade – upgrade could take several days.	Yes	No
19.	Upgrade NSX Manager at the Prod Site	Prod Site	No Changes to DR NSX during Upgrade	Yes	No
20.	Upgrade NSX Controller Cluster at the Prod Site	Prod Site	NSX reverts to read only mode Change Window Required	Yes	Yes – No Disruption as long as VM’s don’t move or any changes made
21.	Upgrade NSX Host preperation at the Prod Site	Prod Site	Hosts require Reboot	Yes	No
22.	Upgrade NSX DLR’s at the Prod Site	Prod Site	Disruption to Service	Yes	Yes
23.	NSX Edges at the Prod Site	Prod Site	Outage required while edge is redeployed and upgraded	Yes	Yes
24.	Upgrade vCenter from vCenter 6.0.x to vCenter 6.5. at the Prod Site	Prod Site	vCenter management of ESXi hosts unavailable during upgrade.	Yes	No
25.	Upgrade vSphere Replication at Both Sites	Both Sites	Replication unavailable until this points it has to be disabled before starting the upgrade process	Yes	No vSphere Replication protection during upgrade
26.	Use vSphere Update Manger to scan and remediate an ESXi host.	Prod Site	1 host not available, reduced capacity.	Yes	No
27.	Repeat steps for remaining hosts in the cluster.	Prod Site	One host will always be unavailable while it is being upgraded with vSphere Update Manager.	Yes	No
28.	Upgrade vSAN at the Prod Site	Prod Site	Possible Performance and increased risk of failure during upgrade – upgrade could take several days.	Yes	No
29.	Optional: Update VMware Tools on each VM with a vSphere Update Manager baseline.	Prod Site	Virtual machine reboot.	Recommended	Reboot
30.	Optional: Update virtual hardware on each VM with a vSphere Update Manager baseline.	Prod Site	Virtual machine shutdown, 1+ reboots.	No	Yes; upgrade during shutdown

 

Further Upgrade Information

vSphere

 

1. Upgrade all external Platform Services Controller servers running 6.0.x to 6.5.
2. Upgrade vCenter Server 6.0.x to vCenter 6.5.

This step includes the upgrade of all components and installation of new components that were not previously addressed.

During the upgrade, the vCenter Server will be unavailable to perform any provisioning operations or functions such as vSphere vMotion and vSphere DRS. Once started, the vCenter database is upgraded first, followed by the vCenter binary upgrade. After the upgrade, the hosts are auto-reconnected.

3. Upgrade vSphere Update Manager(Windows Only, on vSphere 6.5 Update Manager is included with the vCenter Appliance). In the same way as vCenter, the database is upgraded first, followed by binaries. With vSphere 6.5, vSphere Update Manager is configured and administered through the vSphere Web Client.
4. Use vSphere Update Manager to create a baseline, scan the hosts, and then upgrade the ESXi hosts to version 6.5. A copy of the ESXi 6.5 installation media must be available and added to vSphere Update Manager.
5. Use vSphere Update Manager to create a baseline for VMware Tools, scan virtual machines, and remediate to upgrade VMware Tools.
6. Upgrade VMware Virtual Machine Hardware to version 13 (if not already completed previously) to take advantage of new features in vSphere 6.5.

• This step should not be performed until all hosts have been upgraded to ESXi 6.5, because VMware Virtual Machine Hardware version 13 cannot be run on older hosts.

7. If applicable, upgrade VMFS volumes on any ESXi host that has already been upgraded, but is not yet running VMFS6.

• Only VMFS3 and above can be upgraded to VMFS5 on an ESXi 6.x host. Verify that older volumes are upgraded to at least VMFS3 before upgrading to ESXi 6.x.
• Once a VMFS file system update has been completed, it is not possible to roll back this change. Therefore, hosts not running ESXi 5.x or 6.x will be unable to read these volumes.

 

 

vSAN

 

Before starting the vSAN upgrade process, ensure that the following requirements are met:

1. The vSphere environment is up to date:
   • The vCenter Server managing the hosts must be at an equal or higher patch level than the ESXi hosts it manages.
   • All hosts should be running the same build of ESXi before vSAN cluster upgrade is started.
     • If the ESXi host versions are not matched, the hosts should be patched to the same build before upgrading.

2. All vSAN disks should be healthy
   • No disk should be failed or absent
   • This can be determined via the vSAN Disk Management view in the vSphere Web Client
3. There should be no inaccessible vSAN objects
   • This can be verified with the vSAN Health Service in vSAN 6.0 and above, or with the Ruby vSphere Console(RVC) in all releases.
4. There should not be any active resync at the start of the upgrade process.
   • Some resync activity is expected during the upgrade process, as data needs to be synchronized following host reboots.
5. Ensure that there are no known compatibility issues between your current vSAN version and the desired target vSAN version. For information on upgrade requirements, see vSAN upgrade requirements (2145248).
   • If required, update the vSAN cluster to the required build before undertaking the upgrade process to avoid compatibility concerns.

Host Preparation

Ensure you choose the right maintenance mode option. When you move a host into maintenance mode in vSAN, you have three options to choose:

• Ensure availability
  If you select Ensure availability, vSAN allows you to move the host into maintenance mode faster than Full data migration and allows access to the virtual machines in the environment.
• Full data migration
• No data migration
  If you select No data migration, vSAN does not evacuate any data from this host. If you power off or remove the host from the cluster, some virtual machines might become inaccessible.

Exit maintenance mode and resync

• When the ESXi host is upgraded and moved out of maintenance mode, a resync will occur. You can see this through the web client.
• Ensure this is complete before moving onto the next host. A resync is occurring as the host that has been updated can now contribute to the vSAN Datastore again. It is vital to wait till this resync is complete to ensure there is no data loss.

 

 

NSX

 

For detailed information please refer to the NSX Upgrade guide.https://docs.vmware.com/en/VMware-NSX-for-vSphere/6.3/nsx_63_upgrade.pdf

 

vROPS

 

For detailed information please refer to the vROPS Upgrade guide.

https://docs.vmware.com/en/vRealize-Operations-Manager/6.6/vrealize-operations-manager-66-vapp-deploy-guide.pdf

 

vRLI

 

For detailed information please refer to the vRLI Upgrade guide.

https://docs.vmware.com/en/vRealize-Log-Insight/4.5/com.vmware.log-insight.administration.doc/GUID-4F6ACCE8-73F4-4380-80DE-19885F96FECB.html

 

vDP

 

For detailed information please refer to the vDP Upgrade guide.

https://docs.vmware.com/en/VMware-vSphere/6.5/vmware-data-protection-administration-guide-61.pdf