Host profile removing vmk0 management port

Up until recently I had very minimal exposure and experience with host profiles and had the pleasure of getting better acquainted with the feature. There is a well documented “bug” with the host profiles & auto deploy where the vMotion kernel port takes the vmk0 port and the host disconnects from vCenter (eg. here & here) however my problem was slightly different and the many blog posts I found on the problem did not solve it for me.

Here is what went down.

I was targeting a host to upgrade from 5.1 to 5.5 which is deployed via Auto deploy, I pointed the host at the 5.5 image, rebooted it – all good. I made some slight tweaks to the config, updated the answer file and applied the profile. On the subsequent reboot the host came back up with part of its new config and then I watched vCenter apply the remaining settings. What happened next was that I noticed the task was attempting to reconnect with the host. Flicking over to the IMM showed that the host had now been re-configured with the vMotion kernel port IP address and soon enough vCenter marked the host as disconnected.

Some head scratching ensued and many hours went by as I tried to get to the source of the problem. From furious googling to validating the host profiles against other clusters and a number of pointless reboots. I found myself at a bit of a “chicken or the egg” scenario as the host profile prevented me from modifying the management network IP and I couldn’t manage the host from vCenter :(. I soon discovered that I had a brief window in where the host was connected to vCenter and where I could remove the host profile from it. After two reboots I was back to a host without the host profile attached, where I could again manage the host.

I decided it was time to configure the host manually and create a new host profile. It all went swimmingly and my host was now complaint, woo hoo! Happy days… until I went to re-build the next host in the cluster. As I went to apply the profile, the summary page stated something along the lines of “Remove vmk0 from vSwitch 0”. Damn, same issue!

The same headaches eventuated and I wasted another couple of hours. After some deliberating with colleagues I figured I would create the vmkernel ports manually and then try applying the profile. This time around, no messages about removing vmk0, I felt a slight hint of confidence and rebooted the host. This time around the host came back complaint and configured as per the host profile, finally!

So in conclusion, create all your VM Kernel ports prior to applying the host profile.

This may be common knowledge to some, but it wasn’t to me and I didn’t see it called out anywhere through my searches, although I may have missed it.

Thanks for reading!

17/08/18 – Update: If the above does not work for you, try creating the VMkernel ports manually, updating the answer file and reboot the host (don’t apply or remediate the host manually). Applies for vSphere 6.x.


O365 / Exchange one-liners

This post is way overdue, but better late than never right?

I spent a few good months late last year working on a O365 migration project and I thought I would share some of the powershell commands that I used for various tasks throughout the project. I have gathered some of these from various blogs (many from and the Microsoft KB articles. I thought it might be useful to have them in once place and could prove useful to you folk in the wild.

First thing’s first – Connecting. Here is a write up of the process and the per-requisite software you need, but below is what I’ve used to connect my ps session.

Set-ExecutionPolicy Unrestricted -Force   
$O365CREDS = Get-Credential
$ONPREMCREDS = Get-Credential
$SESSION = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri -Credential $O365CREDS -Authentication Basic -AllowRedirection
Import-PSSession $SESSION
Connect-MsolService -Credential $O365CREDS

#Test connectivity
Get-Mailbox yourmailboxname
Disable Clutter

This was a real annoyance for our users

Get-Mailbox | Set-Clutter -Enable $False
Set maximum send / receive size limit for all users with @domainname address

Note: This command might need to change based on the size of your tenant (There are a few documented methods out there). I found that if you have a large number of users, you will constantly get prompted to authenticate or the command will fail once it reaches the threshold.

Get-Mailbox * | Set-Mailbox -MaxReceiveSize 150MB -MaxSendSize 150MB

Another thing to keep in mind with this one is that you’re limited to your Send / Receive connectors, so keep an eye out on those limits.

Setup Out of Office
Set-MailboxAutoReplyConfiguration -Identity -AutoReplyStateEnabled -ExternalMessage "I'm on vacation" -InternalMessage "I'm on vacation"
Migration batch operations
Get current migraiton batches in your tenant
Get the migration progress of each mailbox in a migraiton batch
Get-MigrationUser -BatchId "vKARPS-Batch3-Finance" | Get-MigrationUserStatistics | FT -AutoSize
Create migration batch from csv, but dont start syncing.

Here I have allowed for a bad item limit of 100 and set allowincremental syncs to true. This is so that when the batch it started, it will not automatically cutover.

 New-MigrationBatch -Name "vKARPS-Batch4-Legal" -SourceEndpoint Australia -TargetDeliveryDomain -CSVData ([System.IO.File]::ReadAllBytes("c:ScriptsLegalUsers.csv")) -AllowIncrementalSyncs $True -BadItemLimit 100 -NotificationEmails
Start a migration batch
Start-MigrationBatch -Identity "vKARPS-Batch4-Legal"
Stop a migration batch
Stop-MigrationBatch -Identity "vKARPS-Batch4-Legal"
Remove a user from a migration batch, without deleting the batch.

Note that the batch must be in a stopped state.

Remove-MigrationUser -Identity
Get details about a user that is been migrated
Get-MigraitonUser -identity
Get detailed migraiton log for a user.

This is useful when troubleshooting migration issues

Get-MigrationUserStatistics -IncludeReport -Diagnostics | Export-Clixml
Mailbox Assessment

These are some one-liners to extract information to help target mailboxes for assesment to see if theyre ready to migrate to O365. This was also helpful to find relationships between user mailboxes and shared mailboxes. Note that these are to be run on the on-premises exchange servers.

Get last logon time

This was useful to identify mailboxes which have not been used in a while or are too large to be migrated. In my case most these were candidates for deletion, saving us licenses in Exchange Online or having difficult conversations to reduce mailbox sizes. Placing in a delimiter allows you to easily separate the columns in excel (text to columns function).

Get-Mailbox | Get-MailboxStatistics | Select-Object DisplayName,LastLoggedOnUserAccount,LastLogonTime,LastLogoffTime,ItemCount,TotalItemSize | Export-Csv -Delimiter ~ -NoTypeInformation C:\temp\MailboxLastLogon.csv
Get Mailbox permissions

This command will list the permissions for each mailbox if one or more accounts have access to it, apart from itself. You may have to do some further filtering on accounts, depending on your environment

Get-Mailbox -ResultSize unlimited | Get-MailboxPermission | where {$_.user.tostring() -ne "NT AUTHORITY\SELF" -and $_.IsInherited -eq $false} | Select Identity,User,@{Name='Access Rights';Expression={[string]::join(', ', $_.AccessRights)}} | Export-Csv -NoTypeInformation C:\Temp\MailboxPermissions.csv
Get Mailbox Type

This is helpful to quickly filter on the types of mailboxes and identity any that are not setup correctly. For example if a shared mailbox is setup as a user mailbox.

 Get-Mailbox -ResultSize unlimited | Select-Object SamAccountName,DisplayName,RecipientTypeDetails,PrimarySMTPAddress | Export-CSV C:\temp\MailboxType.csv
MSOL user operations
Assign License

This will obviously vary depending on what license you wish to apply, but here is the syntax

Set-MSolUserLicense -UserPrincipalName -UsageLocation AU
Set-MSolUserLicense -UserPrincipalName -AddLicenses vkarps:EXCHANGESTANDARD

Remove License
Set-MSolUserLicense -UserPrincipalName -RemoveLicenses vkarps:EXCHANGESTANDARD

Check License Status for users with @DOMAINNAME address
Get-MsolUser -All | Where-Object {$_.UserPrincipalName -like "*"}

Check license status for list of users in a .csv file

The csv file needs to name ‘name’ in the first line, each following line should contain a valid UPN.

Import-CSV C:\Scripts\NewUsers.csv | ForEach {Get-MsolUser -UserPrincipalName "$($_.Name)"}
Change UPN for MSOL user

I came across instances where user accounts were created with UPNs for other domains / namespaces which were replicated created in the cloud. It can also happen if the UPN is changed after the account has been provisioned in Exchange Online. This issue becomes apparent when you try to migrate the user to Exchange Online. Here is an article which explains the issue:

Set-MsolUserPrincipalName -UserPrincipalName "" -NewUserPrincipalName ""

Set-MsolUserPrincipalName -UserPrincipalName "" -NewUserPrincipalName ""
Deleting mailbox and user from Exchange Online

This may be required for a number of reasons, in my case it was due to duplicate mailboxes (one on-premises and the other on O365). Note: The mailbox will be deleted and cannot be recovered.

Remove-msoluser –userprincipalname
Get-msoluser –userprincipalname -ReturnDeletedUsers  fl *objectID*
Remove-MsolUser -ObjectID -RemoveFromRecycleBin -Force

You may have to give it a couple of minutes before running the next command

Get-mailbox -SoftdeletedMailbox | Remove-Mailbox -PermanentlyDelete -Force -Confirm:$fals


Disconnected from Host. Agent is out of date and needs a manual upgrade

I recently performed an upgrade of vCenter from 5.0 to 5.5 and had an issue with just one of my hosts connecting to vCenter.

Agent out of date
First I tried to re-connect it manually with no success. I restarted the management agents from the iLO (not sure why I didn’t try go via ssh first as I normally would) and tried to connect again as this worked for other issues I came across in the past, still nothing.

OK, let’s uninstall the FDM client manually and try re-connecting the host (KB1003714):

cp /opt/vmware/uninstallers/ /tmp
chmod +x /tmp/

This time upon connecting the host I was prompted for the root credentials and proceeded through the wizard; enter next error:

Cannot contact the specified host (host1.lab.local) 
The host may not be available on the network, a network configuration 
problem may exist, or the management services on this host may not be

Here, I did a sanity check and confirmed that DNS resolution was working and that I could ping the host, all good. Hrmm…

Next I went over to take a look at the fdm.log and the vpxa.log which were not giving me much to go off. So I went over to the events tab in the c# client to have another look at the error and noticed I was also getting and incorrect username and password error, dafaq ?

So I decided to ssh to the host to confirm that I wasn’t mistyping the password and then realized that I couldn’t initiate a session:

"Network Error: Connection Refused"

I checked lockdown mode was disabled, SSH service was running and the host firewall rules were okay. Hrmm, why on earth is SSH refusing my connection?

Some quick google-fu came up with the following KB article (KB1039095).
Back to the iLO I went and sure enough, the inetd.conf file was blank
I copied the contents of the configuration file from another host in the cluster, restarted the ssh daemon along with the management agents.

Here is my inetd.conf incase you don’t have another host to copy from:

# Internet server configuration database

# Remote shell access

ssh      stream   tcp   nowait   root   /usr/lib/vmware/openssh/bin/sshd       sshd ++swap,group=host/vim/vimuser/terminal/ssh -i
ssh      stream   tcp6  nowait   root   /usr/lib/vmware/openssh/bin/sshd       sshd ++swap,group=host/vim/vimuser/terminal/ssh -i

# VMware authentication daemon
authd   stream    tcp   nowait   root   /sbin/authd           authd
authd   stream    tcp6  nowait   root   /sbin/authd           authd

I could now ssh to the host, great! I proceeded to try connect the host and voila the host was able to connect back into the cluster.

As vCenter needs to copy over the installation files to the host for the installation of the FDM agent it was unable to do so as SSH was busted.

Here is another useful article I used during my troubleshooting (KB2004429).