vCenter 6.7 upgrade thumbprint mismatch

I had a rather frustrating experience the other day with a customers vCenter Upgrade. As many of you will know, there is a bit of prep work that goes into these upgrades, so having them not go to plan can be a little disheartening at times, especially when it’s something simple that you missed (Spoiler: it was not DNS).

To set the scene, I tend to do most of my work from an “admin workstation” which has all the tools I need installed. Here, I stepped through the wizard, verified and accepted the SHA1 thumbprints when presented and went onto stage 2 of the upgrade. Shortly after the pre-upgrade checks failed with an “internal error”. Upon checking the upgrade logs, I was presented with the following:

File "/usr/lib/vmware/cis_upgrade_runner/libs/pyVmomi.zip/pyVmomi/SoapAdapter.py", line 981, in _VerifyThumbprint
raise ThumbprintMismatchException(thumbprint, sha1Digest)
pyVmomi.SoapAdapter.ThumbprintMismatchException: Server has wrong SHA1 thumbprint:65abc597698285900c37f009a5c11ab45c03e123 (required) != ba4b9b745034c61785fdc33ee123d87397ea999c (server)
2019-07-12T23:16:36.999Z INFO root Exiting with exit-code 1

As most would, I did a sanity check in my browser to validate the certificate thumbprint and everything was matching up and I could not find the thumbprint that the installer was referencing. I did notice however that the proxy server had injected itself into the certificate chain, which reminded me of a new agent this customer had recently deployed to their fleet (this is getting more common, as it allows the company to inspect outbound SSL traffic). I checked the thumbprints again from a server that did not not have this agent and sure enough the thumbprint was what the installer was referencing (The correct thumbprint).

Logically, I closed the wizard and re-started stage 2 of the upgrade from the server that was getting the correct thumbprint. My frustration grew as I was presented with the same error message shortly after. There was a good hour spent on re-checking things and inspecting logs.

What I discovered was (may be obvious to some) that it appears that the upgrade wizard writes the thumbprints into the pre-check configuration from Stage 1 when the new appliance is first deployed. In stage two it then validates against this from the newly deployed appliance.

The fix is pretty obvious by now, restarting the upgrade from the server without the proxy agent saw the process go through smoothly. Pretty trivial, but something to be aware of.

TL;DR – Make sure there is nothing (enterprise proxy) injecting itself into your browsers certificate chain on the workstation you’re using the upgrade wizard on as it will throw out the SHA1 thumbprint.

Advertisements

vSphere focused Powershell snippets

How many times have you came across a scenario where you know you’ve solved with some Powershell code in the past, but can’t remember which cmdlet, flag or syntax you used?

I’ve done this to myself countless times, so over the last couple of months I’ve made a conscious effort to centrally write down snippets of code that I’ve used, so future me will be grateful for (helping me save time and some sanity also).

Below are some snippets of code that you may find useful in your day to day tasks as a vSphere admin. Some of these examples would’ve been pinched from other blogs and others I’ve crafted myself. I’ve tried to reference the original blog where possible.


Virtual Machine Operations

Removing snapshots:
The csv should contain headers vmname and snapname which I just grab from RVTools. This will go through and delete one snapshot at a time, if you’re a gambler, you can do them all at once by adding -RunAsync but I would not recommend it in production.

$vms = Import-csv .\snaps.csv
$vms | %{Get-Snapshot $_.vmname -name $_.snapname | Remove-snapshot -confirm:$false}

The above code is good if you want to target specific snapshots for VMs, but if you want to remove all snapshots against a VM, a slight modification to the above code, show below will work. The CSV you pass in here should only contain a list of VM names:


foreach ($vm in $vmList) {Get-Snapshot $vm | Remove-Snapshot}

Note: Previously we had a $vm | % {… instead of a foreach statement. % is short for foreach and we are passing in the list of VMs into it.

Deploy OVF with customizatons
If you need to deploy a few instances of the same appliance with some customizations, this is really helpful.

First you need to import the OVF and extract the varaibles that are available for customization

$ovfpath = ".\VendorAppliance.ova"
$ovfconfig = Get-OvfConfiguration -ovf $ovfpath
$ovfconfig.ToHashTable() | FT -autosize

Once you have this, you can pick out which ones you need to customize and declrate the values as below.

$ovfconfig.NetworkMapping.primary.Value = "CLS01_VL3"
$ovfconfig.NetworkMapping.mon0.Value = "CLS01_Monitor"
$datastore = "cls01_fcp"
$esxhost = "esxi01.vkarps.local"
$vmname ="AppServer"
$DRSRulename = "AppServer1tohost1"
$HostRule = "Host1"
$cluster = "CLS01"

Once that it all done, it’s time to deploy the appliance
Import-VApp $ovfpath -OvfConfiguration $ovfconfig -name $vmname -VMHost $esxhost -Datastore $datastore

Prevent tools from re-sizing display
(https://vkarps.wordpress.com/2018/07/30/vm-console-display-resizing/)


$vmname = "KioskVM"
$spec = New-Object VMware.Vim.VirtualMachineConfigSpec
$opt = New-Object VMware.Vim.OptionValue
$opt.Key = "guestInfo.svga.wddm.modeset"
$opt.Value = "FALSE"
$spec.ExtraConfig += $opt
$vm = Get-VM -Name $vmName
$vm.ExtensionData.ReconfigVM($spec)

Move single VM to defined folder

$vmfolder = "SQL"
Move-VM -vm SQL01 -destination $vmfolder

Move multiple VMs to defined folder

$vmFolder = "SQL"
$vms = import-csv vms.csv
Foreach ($vm in $vms){Move-VM -vm $vm -destination $vmfolder}

Move VMs from one datastore to another

$vms = Get-datastore "cls01_vol01" | Get-VM
foreach($vm in $vms){
Move-VM $vm.name -Datastore "cls01_vol02"
}

Move list of virtual machines to a defined host

$vms = Import-csv .\SQLVMs.csv
foreach($vm in $vms){
Move-VM $vm.name -Destination "esxi01.vkarps.local"
}

AutoDeploy

Create new deploy rule with image profile
If you want to create a new AutoDeploy rule, there is no commandlet that I’ve found (as of writing this) that will list out the image profile objects from the repository. The way we get around this is by grabbing the image profile from an existing host that already has the image applied, then passing that object into the New-DeployRule command.

$img = Get-VMHostImageProfile esxi01.vkarps.local
New-DeployRule -Name DeployRuleName -Item $img,Cluster1 -Pattern "ipv4=10.1.1.100-10.1.1.104"

Copy and existing Autodeploy rule and set some new patterns
This is useful if you want to use an existing image or host profile and apply it to a new target / pattern. You could also use this to solve the same issue that the previous code did.

Copy-DeployRule -DeployRule "testrule" -ReplaceItem host_profile,targetcluster -ReplacePattern "ipv4=192.XXX.1.10-192.XXX.1.20"

Forcing a compliance update
So, if you’ve ever played with AutoDeploy you will know it has some annoying ‘features’. One is when you’re changing an image profile for a deploy rule, the host won’t automatically pick up that change on the next reboot (in my experience anyway). The below snippet will force the AutoDeploy service to refresh it’s mappings for the target host.

Test-DeployRuleSetCompliance -VMhost esxi01.vkarps.local | Repair-DeployRuleSetCompliance

Host Operations

Create a new VMKernel port
Enabling vMotion with JumboFrames in this example

New-VMHostNetworkAdapter -VMHost esxi01.vkarps.local -PortGroup Cluster1_VMK_10.1.2.0_VL2_vmotion -VirtualSwitch Cluster1_vds -IP 10.1.2.100 -SubnetMask 255.255.255.0 -Mtu 9000 -vmotionenabled $true

Mounting an NFS datastore to a host
New-Datastore -Nfs -VMHost esxi01.vkarps.local -Name cls1_syno_vol01 -Path "cls1_syno_vol01" -NfsHost 10.1.3.10

Mounting an NFS datastore to all hosts in cluster

$hosts = Get-Cluster "CLS01" | Get-VMHost
foreach($hst in $hosts){
New-Datastore -Nfs -VMHost $hst.name -Name cls1_syno_vol01 -Path "cls1_syno_vol01" -NfsHost 10.1.3.10
}

Set dump collector configuration for defined cluster

$cluster = CLS01
Get-Cluster $clsuter | Get-VMHost | Set-VMHostDumpCollector -HostVNic "vmk0" -NetworkServerIP "10.1.1.111" -NetworkServerPort 6500

Configure Multipathing policy variants
You may have a scenario that a number of hosts across clusters have a particular lun set to fixed path and you need to set it to round robin. The below snippet will do that for the defined scsi lun

$hosts = Import-csv .\FixedPathHosts.csv
Foreach ($h in $hosts){
Get-VMhost $h.name | Get-ScsiLun naa.6000155000000010b000801ebeaaaa50 | Set-ScsiLun -MultipathPolicy "roundrobin"
}

Similar to above, but for all hosts in the clusters
Get-Cluster CLS01 | Get-VMhost | Get-ScsiLun naa.6000155000000010b000801ebeaaaa50 | Set-ScsiLun -MultipathPolicy "roundrobin"

For a single ESXi host, but any lun stating with naa.600

Get-VMHost esxi01.vkarps.local | Get-ScsiLun -CanonicalName "naa.600*" | Set-ScsiLun -MultipathPolicy "roundrobin"

Reporting

We all often get asked to report on various and perhaps obscure things. Here are some that I’ve found I either regularly get asked for or have found handy to extract data quickly.

VM Properties
Sometimes you may need to get some metadata about a list of VMs for folks, below is an example of taking a list of VMs and finding out which hosts they are currently running on and spitting out the result to a file. It can be easily modified to find different VM properties.

$vmList = Get-Content vm.txt
$output = foreach ($vmName in $vmList) {Get-VM $vmName | Select-Object -Property Name,VMHost}
$output | Export-Csv c:\temp\VMHostList.csv

DRS Groups
Say you need to get the specs (or attributes) of virtual machines in a DRS Group. Below is an example of how to grab a list of objects in a DRS group and find the specs we are interested in (vCPU count and Memory allocation).

$a=(get-DRSClusterGroup -cluster Cluster2 -Name "SQL VMS")
$b=$a.vm
$c= foreach ($vm in $b) {get-VM $vm | Select Name, NumCPU,MemoryGB }
$c | export-csv Cluster2_sqlVMs.csv

compare object (DRS group to what’s on the host)
If you have some “License restrictions” in your environment, you may have to force a number of VMs onto a single or gorup of hosts using DRS groups. It’s quite hard to maintain a DRS rule that negates the first rule. I.e: Anything not in group “SQL” group, do not put it onto “SQL Host” host.
I found the below snippet helpful to compare the VM DRS Group to what is actually running on the host. It is much quicker than uing a spreadsheet or going through it manually. The compare-object cmdlet can be used for a number of use cases.


$a = get-cluster CLS01 | Get-DrsClusterGroup -name "SQL Guests" | Select member
$b = Get-VMHost esxi01.vkarps.local | Get-VM | Select name
$a.member.name.count (This line and the one below are not required, just gives you an indication if you need to run the last line or not. I.e if they're the same, you're good; in this scenario).
$b.count
compare-object $a.member.name $b.name

Get datastore mount path (NFS)

Get-VMHost esxi01.vkarps.local | get-Datastore | Select Name,@{n='RemoteHost';e={$_.RemoteHost[0]}},RemotePath | Export-csv CLS01_NFS_DatastoreDetails.csv -NoTypeInformation

Get datastores / datastore cluster relationship

Get-Datastore -Location DC2 | Select @{N=’DSC’;E={Get-DatastoreCluster -Datastore $_ | Select -ExpandProperty Name}},Name

Check which port groups have netflow enabled on vDS

Get-vDSwitch clos01_vds | Get-vDPortGroup | Select Name,@{Name="NetflowEnabled";Expression={$_.Extensiondata.Config.defaultPortConfig.ipfixEnabled.Value}}

Get dump collector config for cluster

Get-Cluster CLS01| Get-VMHost | Get-VMHostDumpCollector | FT VMHost,Hostvnic,NetworkServerIP,NetworkserverPort,Enabled

List VMs with ISO mounted in Cluster

Get-Cluster CLS01 | Get-VM | Select Name, @{Label="ISO file"; Expression = { ($_ | Get-CDDrive).ISOPath }} | Export-csv -NoTypeInformation CLS01MountedMedia.csv

Get Power on events for defined VM
This can be adjusted to look at other tasks. Great writeup here with some more detail http://www.lucd.info/2009/11/18/events-dear-boy-events-part-2/


Get-VM SQL01 | Get-VIEvent -MaxSamples ([int]::MaxValue) | where { $_.fullFormattedMessage -like "Task: Power on*" }

Get VM DNS name for VMs with particular string in the name

Get-VM | Where {$_.name -like "*SQL*"} | Select name, @{N="DnsName"; E={$_.ExtensionData.Guest.Hostname}} | FT -autosize

Get VM count for each host and export to csv

Get-VMHost | Select @{N=“Cluster“;E={Get-Cluster -VMHost $_}}, Name, @{N=“NumVM“;E={($_ | Get-VM).Count}} | Sort Cluster, Name | Export-Csv -NoTypeInformation c:\HostVMCount.csv`

Get vMotion events for particular Virtual Machines and export to csv
Ensure you have the Get-VMotion script installed on your machine (see http://www.brianbunke.com/blog/2017/01/03/get-vmotion/)

Import-Module Get-vMotion
$vms =@('SQL01','SQL02');
$array = @()
foreach($vm in $vms){
$array += get-VM $vm | Get-vMotion -days 30 | Select Name,srcHost,dstHost,Duration,StartTime,EndTime
}
$array | Export-csv -NoTypeInformation .\vMotionEvents.csv

Get VMs Sync Time With Host Setting

Get-VM * | Select @{N='VM Name';E={$_.Name}},@{N='GuestOS';E={$_.ExtensionData.Guest.GuestFullName}},@{N='SyncWithHost';E={$_.ExtensionData.Config.tools.SyncTimeWithHost}} | Export-csv .\timeSync.csv -NoTypeInformation

vDS Operations

Creating a new port group on an existing vDS
In this example we are setting the “Allow Promiscuous” flag to the port group.
$vds = Get-VDSwitch "cls01_vds"
$pg = "cls01_monitor"
$vdspg = $vds | New-VDPortgroup -name $pg
$vdspg | Get-VDSecurityPolicy | Set-VDSecurityPolicy -AllowPromiscuous $true

Export vDS config

$vDSConfLocation = "C:\vDSConfigs"
$vDS = "CLS01_vds"
Get-VDSwitch -name $vDS | Export-VDSwitch -Description “Backup of $($_.Name) VDS” -Destination "$vDSConfLocation\$vDS$($_.Name).Zip" -Force

Create new vDS based off existing one in different vCenter
This can be easily adjusted to copy from within the same vCenter

$srcvCenter = "vc01.vkarps.local"
$dstvCenter = "vc02.vkarps.local"
$DCLocation = "DC2"
$vDS = "CLS01_vds"
$srcVDS = Get-vDSwitch $vDS
$srcPG = $srcvDS | Get-VDPortgroup
New-VDSwitch -Server $dstvCenter -name $vds -Location (get-Datacenter -Server $dstvCenter $DCLocation) -LinkDiscoveryProtocol CDP -LinkDiscoveryProtocolOperation Listen -Mtu $srcVDS.mtu -NumUplinkPorts $srcVDS.NumUplinkPorts -Version 6.5.0
Foreach ($pg in $srcPG)
{
$pgVLAN = $pg.Extensiondata.Config.DefaultPortConfig.Vlan.VlanID
If ($pg.IsUplink -eq "True"){Write-Host "Skipping Uplink PortGroup" -ForegroundColor yellow}
#If it is not the uplink pg, create it
else{
Get-VDSwitch -Server $dstvCenter -name $vDS | New-VDPortgroup -Name $pg.name -NumPorts $pg.numPorts -VLanId $pgVLAN
}
}

Cluster Operations

Create VM DRS Group and add a VM

$vmname = "SQL01"
New-DrsClusterGroup -Name "SQL VMs" -cluster $cluster -vm $vmname

Create Host DRS group and add host

$esxhost = "esxi01.vkarps.local"
New-DrsClusterGroup -Name "Host1" -cluster $cluster -VMHost $esxhost

Create new VM/Host rule from previously created groups

$DRSRulename = "SQL Licensing"
$vmGroup = "SQL VMS"
$hostRule = "Host1"
New-DrsVMHostRule -Name $DRSRulename -Cluster $cluster -VMGroup $vmGroup -VMHostGroup $HostRule -Type "MustRunOn"

Create new cluster based off a cluster in different vCenter

$cluster = "CLS01"
$srcCluster = Get-cluster -name $cluster
$DCLocation = "DC2"
$dstvCenter = "vc01.karps.local"
$srcvCenter = "vc02.vkarps.local"
New-Cluster -Server $dstvCenter -name $srcCluster.name -Location $DClocation -DRSEnabled -DRSAutomationLevel $srcCluster.DRSAutomationLevel -EVCMode $srcCluster.EVCMode -HAEnabled

Datastore Operations

Create Datastore Cluster and disable SIOC

New-DatastoreCluster -Name $dsCluster -Location $DCLocation

Set SDRS to manual and disable SIOC on Datastore Cluster

Set-Datastorecluster -sdrsAutomationLevel Manual -IOLoadBalanceEnabled $false

Create folder under datastores view under existing folder

(get-view (get-view -ViewType datacenter -Filter @{"name"="DC2"}).DatastoreFolder).CreateFolder("SQL")

I hope that some of these examples have been useful to you or perhaps given you a better idea on how you can use powershell for particular use cases, where you can then adopt it to your use case.

That’s all I have for now, I’ll eventually put this into github and keep adding to it there. If you have any feedback, please leave a comment below.