vSphere Replication ‘Solution user detail’ certificate is invalid

I came across an odd ball today when upgrading a customers vSphere Replication appliance from 5.8.1 to 6.0. For those who have done a VR in place upgrade before, you will know that there is really not much to it; Mount ISO, Install Update, Reboot, done. When VR 6.0 was introduced an additional step is required after you perform the upgrade, you had to go in and register it with your lookup service. All this requires is the lookup service url and the SSO administrator credentials and you’re done as per the documentation here.

Instead I was presented with this error message:

'Solution user detail' certificate is invalid - certificateException java.security.cert.CertificateExpiredException: NotAfter: Fri Jun 17 23:01:55 UTC 2016

img-alternative-text

Its probably worth noting that this has come off the back of a vCenter upgrade to 6.5, so ‘solution user’ automatically got me looking at SSO solution users in the web client and the vCenter extension manager to validate that the certificate thumbprints matched up.

Sure enough, the thumbprint that was registered with vCenter matched the one of the vSphere Replication appliance. My google-fu didnt get me any furhter either, next thing that came to mind was ssh and logs. I saw the secuiry tab and figured I would enable ssh from there…nope, no enable ssh option there dummy. What I did see on the secuirty tab was this.

Right, expity date matches the error message!

So here is what you need to do:

Head back over on the configuration page there is a “Install a new SSL Certificate” section where you can generate and install a self signed or your own certificate.

Hit the Generate and Install button, validate the warning that it will overwrite the existing one and let it do its thing.

img-alternative-text

 

Once it is done, you will be prompted by the following message.

img-alternative-text

Reload your brower like it states, even though it seems as if does it when it takes you do the login page.

Once you’re back in, populate your Lookup Service URL and SSO Administrator credentials at the configuration tab and hit Save and Restart service. If all went well, you should get the below message.

img-alternative-text

Advertisements

Migrating Windows vCenter 5.5 to VCSA 6.5

So it’s that time again, evaluating how us vSphere admins are going to get our customers/employers up to the latest and greatest version. If you’ve ever done upgrades from previous versions to 5.5, you will immensely appreciate the work that the engineering team over at VMware has done to help us migrate over to 6.5. The Migration Assistant is really amazing and does a lot of the heavy lifting.

In saying that, there are a few things you’re going to have to validate and prepare for if you want to have a smooth migration. I recently went through the steps in a lab environment and will outline the steps I went through to both prepare for the migration and the migration itself. Yes – there may be blogs out there that go through the process already. However, I found that they only covered basic deployment models and this is also an opportunity for me to have some long overdue blog writing practice. Hopefully someone finds this useful.

If there is anything that I’ve missed, messed up or fat fingered please let me know in the comments below.

In the my scenario, vCenter was separated between two windows servers. One held the SSO, Inventory and update manager and web client services, while the other held vCenter. Autodeploy is also in the picture, but on it’s own server. Now, if you’ve been good and read through the upgrade documentation, you will notice that as the configuration currently stands, you won’t be able to start the migration. Here are links to useful aricles that got me started with planning out the upgrade.

Important info before upgrading to 6.5: KB2147548
Best Practices in migrating to vCenter 6.5: KB2147686
FAQ’s about the migraiton (Moving to 6.0 but still relevant and helpful): KB2146439
Preparing for migration blog
William Lam has a great collection of most links here:

Update: @emad_younis has a good summary here

Before I started anything else, I wanted to make sure my current deployment was re-aligned to a model which was supported by the migration assistant. The first thing that I needed to do was to get the Inventory service and update manger moved off the server with SSO installed and moved over to the vCenter server.

Moving Inventory Service

Moving the inventoy service is quite simple and this KB article outlines it quite well.
Tip #1 Make sure you have the right version of the 5.5 installer mounted. If this doesnt match your vCenter server you will have a bad time (I know this as I have made this mistake myself and spent too much time that I wish to admit in troubleshooting why my vCenter wouldnt talk to my inventory service after the move).

You may want to consider backing up your inventory service database and restoring it onto the vCenter server. See this link will give you instructions on how to do so. Tip #2 Make sure you have enough space available on the drive which you are creating the backup file on. The database in my environment was about 8GB, which almost filled up my system drive, oops.

Once you’ve re-installed it you will just need to re-point your vCenter to the inventory service with the below command.

cd C:\Program Files\VMware\Infrastructure\VirtualCenter Server\isregtool
register-is.bat vCenter_Server_URL Inventory_Service_URL Lookup_Service_URL
Example:
register-is.bat https://vc01.vkarps.local:443/sdk https://vc01.vkarps.local:10443 https://vcsso01.vkarps.local:7444/lookupservice/sdk  

Moving Update Manager

Next, we need to move Update Manager, This article sums up the process quite well. Tip #3 Remember that the DSN needs to be 32bit.

Current and Target State

So once things have been shuffled around, it should look something like this:

VM1: Single Sign On
VM2: vCenter, Inventory Service, Update Manager, Web Client

After the upgrade, VM1 will become the PSC and VM2 will become the VCSA.

Prep tasks

Before you go any further, make yourself a check list to ensure youve got everything ready for the migraiton. These are some of the things that I picked up during the planing phase and had in my check list. They are not in any particular order:

  • Give your colleagues and other teams who use vCenter a heads up, chances are that they are unfamiliar with the web client
  • Test your local administrative credentials on the source Windows servers, in case of roll back
  • Have your vCenter service account and SSO admin passwords handy
  • Validate that you have permissions to domain join permisions.
  • Validate that your vSphere component versions are supported by the migration (Hosts earlier than 5.5 are not) 
  • List out any 3rd party extensions to vCenter or anything that interfaces with it and their versions to confirm their support for 6.5. Ie. Backup software, storage plugins or orchestration software. It is likely that you will need to update these also. 
  • Prepare your sql database guide 
  • Check the size of your database to estimate the size your VCSA needs to be here
  • Estimate your migration time to 6.5 ( Will help with determining and justifying your outage window) KB214620
  • NTP is configured throughout the entire vSphere environment and is in sync.
  • Forward and reverse DNS records work for vSphere components
  • Target vCenter Cluster DRS setting cannot be set to Fully Automated, partial or manual will do.
  • Verify that certificate checking is enabled in vCenter here
  • Lockdown mode must be disabled on at least one host in the cluster which you’re deploying the appliances to
  • The vCenter service account has ‘replace process level token’ rights on the computer object or OU that your vCenter server resides. Policy Path: “Computer Configuration\Windows Settings\Security Settings\Local Policies\User Rights Assignment\Replace a process level token”
  • If you’re using DHCP for your vCenter address, ensure that the port group or vSwitch accepts mac address changes. ( This was not the case in my scenario but something worth pointing out)
  • Validate that you have enough free disk space on your source server as it will stage data as it extracts it. The Migration Assistant will call this out when it does it’s checks. This size will vary on your database size.
  • Fill out the required installation requirements which vmware has provided here
  • If you have another vCenter server in the mix with it’s own SSO domain, you may consider consolidating SSO domains prior to going to 6.5 as you won’t be able to later KB2033620.
  • Backups, backups and more backups

Deploying new PSC appliance

Alright, time to get the PSC appliance deployed so we can have it ready for migrating the SSO service.
On VM1, which hosts the SSO service, mount the 6.5 VCSA iso and launch the Migration Assistant. The migration assistant is located in X:\migration-assistant\VMware-Migration-Assistant.exe.

A console window will show up and ask you for your SSO password, afterwards it will begin to perform some pre-checks.

You should see something similar to the above. If you’ve missed anything in the pre-checks or something doesn’t stack up, it will let you know. I found that when I first ran this I didn’t have enough storage space for the migration. It should prompt you to enter another location for the migration data.
Have a read through it, as you may miss something important, I found its quite informative and notifies you of certain extensions you may have that will not work after the migration. If you’ve been paying attention, you will notice this screenshot if from a server which had the web client installed on it which will not work after the SSO service is migrated. (This is from another lab environment, but I was happy to not have the web client available during the migration). If all is well, you will see “Waiting for migration to start…”

Next from a workstation or server, mount the same ISO but this time launch the ui-installer (I haven’t had a play with the cli-installer, but that’s an option too for the ones who want to script the migration). Note that this can be run from Unix and OSX machines too!

Click on migrate and proceed to the next step.

Click Next, sign your life away and hit Next again.

Enter in the SSO server fqdn and the SSO Administrator credentials

Next you will be prompted to verify the thumbprint, which you can find in the Migration Assistant console. Make sure this matches up.

Next we will be defining where we wish to deploy our PSC to. Enter in your vCenter server name and credentials to an account which has administrative access.

Select the folder which you wish to place the new appliance

Select your target cluster/ESXi host

Supply your target appliance name and a root password. Note that the new appliance will inherit the OS name of the source server. This might not conform to your naming standard in the enterprise but in my experience I’ve found that it’s easier to have the VM object name the same as the name in the OS in large environments. The installer won’t let you name the VM object the same as your source server as it already exists, so you can either rename your source VM object or rename the appliance VM object later and perform a svMotion to make sure the vm files are consistent.

Select the datastore you wish to deploy to and check whether or not you want it thin provisioned

Now you will need to populate the temporary IP details that the appliance will use during the initial parts of the migration.

Next you will be presented with a summary page outlining the the steps we just went through. Make sure it all stacks up and hit Finish. The wizard will now deploy your new Platform Service Controller.

If all went well, you will see the below screen and have a newly deployed PSC appliance.

 

Migrating to PSC

So far we have deployed the target PSC appliance but have not migrated anything. In stage 2 of the migration wizard we will be doing just that. When you hit next, so that the wizard can go off and connect to the source SSO server we specified earlier.

You will be prompted to enter in credentials of an AD account which has permissions to join the domain

Choose if you want to opt in or out of the Customer Experience Improvement Program

Review the summary and check the box at the bottom confirming that you have backed up your data.

When you click next, you will be presented with warning advising you that the soruce machine will be powered off during this process. You will have an outage from this point, so if you’re doing production make sure you are in your outage window before clicking OK.

Now the migration will kick off and hopefully finish succesfully in a short while. After you get the success message you will be given some URLs for your new PSC. They should something as follows:

You should now be able to hit the PSC client at https://pscname:443/psc

Deploying VCSA

Now that we have a new external PSC, it is now time to move on to getting vCenter migrated over to 6.5. Make sure that you have upgraded all your SSO servers prior to upgrading vCenter.

Firstly we need to deploy the new vCenter appliance which is quite similar to the steps we took with the PSC appliance. Launching the same ISO and starting the wizard as with the PSC, although this time around the wizard will prompt for the SSO and vCenter service account credentials. You might see a little more going on as the wizard performs it’s health checks as it is more than likely that you have some extensions hooked in with your vCenter (i.e.Update Manager, Backup or Storage software). If you’ve done your prep there should be no suprises here, right? :).
Once the migration assistant has finished and you’re happy to proceed, launch the UI installer from another workstation as in the previous steps. Enter in the required details for the source, target, name, etc. We will skip ahead to the deployment size part.

In this step you will need to select the deployment size of your new VCSA. Note that you can pick a storage size seperately from the compute resrouces. Go back to your checklist and see if your existing database size will fit within the default storage size for the deployment you’ve selected. In my case I needed to increase the size.

The next few steps are the same as before, specify a target datastore, temp IP details and wait for the wizard to finish the deployment. If it all went to plan, you should get a successful completion message. Go on and check out the appliance management page at https://tempIPyouProvided:5480.

Migrating to VCSA

Here is the part where most of the sitting around will happen while the wizard does the heavy lifting. From the previous completion page, click Continue and then next at the Stage 2 screen.

The wizard will now go through and do some pre-migration checks. If you got alerted about extensions in the migration assistant, you will get similar ones in the migration wizard. Have a good read of that’s there as there might be a few things that you may need to fix prior to proceeding. After you’ve acknowledged the warnings you will be prompted to specify AD Domain credentials.

Next you need to decide which data you want to copy over, in my case I wanted to copy all the historical data for no other reason but curiosity. If you have any auditing/compliance requirements in your environment, it might be a good idea to select the same.

The next page gives you a summary of the settings we chose along with a checkbox confirming you’ve performed all the relevant backups. Note that from here onwards the migration will start and shortly after your vCenter will be unavailable. So once again, if you’re in production make sure you’re within your outage window before proceeding. As before, the wizard will warn you of this:

Now it’s time for a well deserved coffee, so go off for a while and enjoy yourself. This part can take a few hours, depending on your source and your database size. You should have a rough idea from the calculations you did in the prep tasks. 🙂

Once it’s done without any errors you should have a smile on your face and be looking at a screen like below:

The new 6.5 VCSA should be up and running with all your configuration in tact. The wizard will display links to your new vcsa management pages:

vSphere Client: https://vcsa.vkarps.local:443/vsphere-client
HTML5 Client ( Partial features available): https://vcsa.vkarps.local:443/ui
Appliance Management:https://vcsa.vkarps.local:5480

Be sure to get familiar with all the clients, especially the HTML5 client which will give you a good idea as to what you will see more of in the future. Don’t forget to poke around the appliance management client. It now gives insight into what’s going on within the vPostgres database which I’m sure you will appreciate.

Rollback

I’m hoping that everything went well in your migration, however things go do wrong. Fortunately this is one of the easiest rollbacks I’ve had to do in my experience. Due to the nature of the way the migration is done, rolling back is a matter of switching off the appliances and rejoining your windows machines to the network.

Here are the steps that I followed to rollback:

1. Power off the VCSA and PSC appliances
2. Power on your Virtiual Center and SSO servers
3. Logon as Local administrator
4. Ensure the VM is connected to the network
5. In an elevated powershell run the following command to reset the computer password on each server:

Reset-ComputerMachinePassword -Server <YourDomainController> -Credential <yourAdminusername>

6.Reboot both servers
7. Go do your health checks to ensure all services have started and that your VC deployment is back to where you left it.

Conclusion

The migration to 6.5 is one of the easiest ones to date in my opinion and I hope that I’ve done a good job in outlining what’s required and will help you out in your migration. Feel free to give me any type of feedback in the comments section below.

Get on it and #migrate2vcsa.

Host profile removing vmk0 management port

Up until recently I had very minimal exposure and experience with host profiles and had the pleasure of getting better acquainted with the feature. There is a well documented “bug” with the host profiles & auto deploy where the vMotion kernel port takes the vmk0 port and the host disconnects from vCenter (eg. here & here) however my problem was slightly different and the many blog posts I found on the problem did not solve it for me.

Here is what went down.

I was targeting a host to upgrade from 5.1 to 5.5 which is deployed via Auto deploy, I pointed the host at the 5.5 image, rebooted it – all good. I made some slight tweaks to the config, updated the answer file and applied the profile. On the subsequent reboot the host came back up with part of its new config and then I watched vCenter apply the remaining settings. What happened next was that I noticed the task was attempting to reconnect with the host. Flicking over to the IMM showed that the host had now been re-configured with the vMotion kernel port IP address and soon enough vCenter marked the host as disconnected.

Some head scratching ensued and many hours went by as I tried to get to the source of the problem. From furious googling to validating the host profiles against other clusters and a number of pointless reboots. I found myself at a bit of a “chicken or the egg” scenario as the host profile prevented me from modifying the management network IP and I couldn’t manage the host from vCenter :(. I soon discovered that I had a brief window in where the host was connected to vCenter and where I could remove the host profile from it. After two reboots I was back to a host without the host profile attached, where I could again manage the host.

I decided it was time to configure the host manually and create a new host profile. It all went swimmingly and my host was now complaint, woo hoo! Happy days… until I went to re-build the next host in the cluster. As I went to apply the profile, the summary page stated something along the lines of “Remove vmk0 from vSwitch 0”. Damn, same issue!

The same headaches eventuated and I wasted another couple of hours. After some deliberating with colleagues I figured I would create the vmkernel ports manually and then try applying the profile. This time around, no messages about removing vmk0, I felt a slight hint of confidence and rebooted the host. This time around the host came back complaint and configured as per the host profile, finally!

So in conclusion, create all your VM Kernel ports prior to applying the host profile.

This may be common knowledge to some, but it wasn’t to me and I didn’t see it called out anywhere through my searches, although I may have missed it.

Thanks for reading!

17/08/18 – Update: If the above does not work for you, try creating the VMkernel ports manually, updating the answer file and reboot the host (don’t apply or remediate the host manually). Applies for vSphere 6.x.

O365 / Exchange one-liners

This post is way overdue, but better late than never right?

I spent a few good months late last year working on a O365 migration project and I thought I would share some of the powershell commands that I used for various tasks throughout the project. I have gathered some of these from various blogs (many from exchangeserverpro.com) and the Microsoft KB articles. I thought it might be useful to have them in once place and could prove useful to you folk in the wild.

First thing’s first – Connecting. Here is a write up of the process and the per-requisite software you need, but below is what I’ve used to connect my ps session.

Set-ExecutionPolicy Unrestricted -Force   
$O365CREDS = Get-Credential
$ONPREMCREDS = Get-Credential
$SESSION = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri https://ps.outlook.com/powershell -Credential $O365CREDS -Authentication Basic -AllowRedirection
Import-PSSession $SESSION
Connect-MsolService -Credential $O365CREDS

#Test connectivity
Get-Mailbox yourmailboxname
Disable Clutter

This was a real annoyance for our users

Get-Mailbox User@vkarps.com | Set-Clutter -Enable $False
Set maximum send / receive size limit for all users with @domainname address

Note: This command might need to change based on the size of your tenant (There are a few documented methods out there). I found that if you have a large number of users, you will constantly get prompted to authenticate or the command will fail once it reaches the threshold.

Get-Mailbox *@vkarps.com | Set-Mailbox -MaxReceiveSize 150MB -MaxSendSize 150MB

Another thing to keep in mind with this one is that you’re limited to your Send / Receive connectors, so keep an eye out on those limits.

Setup Out of Office
Set-MailboxAutoReplyConfiguration -Identity user@vkarps.com -AutoReplyStateEnabled -ExternalMessage "I'm on vacation" -InternalMessage "I'm on vacation"
Migration batch operations
Get current migraiton batches in your tenant
Get-MigrationBatch
Get the migration progress of each mailbox in a migraiton batch
Get-MigrationUser -BatchId "vKARPS-Batch3-Finance" | Get-MigrationUserStatistics | FT -AutoSize
Create migration batch from csv, but dont start syncing.

Here I have allowed for a bad item limit of 100 and set allowincremental syncs to true. This is so that when the batch it started, it will not automatically cutover.

 New-MigrationBatch -Name "vKARPS-Batch4-Legal" -SourceEndpoint Australia -TargetDeliveryDomain vkarps.mail.onmicrosoft.com -CSVData ([System.IO.File]::ReadAllBytes("c:ScriptsLegalUsers.csv")) -AllowIncrementalSyncs $True -BadItemLimit 100 -NotificationEmails admin@vkarps.com
Start a migration batch
Start-MigrationBatch -Identity "vKARPS-Batch4-Legal"
Stop a migration batch
Stop-MigrationBatch -Identity "vKARPS-Batch4-Legal"
Remove a user from a migration batch, without deleting the batch.

Note that the batch must be in a stopped state.

Remove-MigrationUser -Identity noob.user@vkarps.com
Get details about a user that is been migrated
Get-MigraitonUser -identity financeteam@vkarps.com
Get detailed migraiton log for a user.

This is useful when troubleshooting migration issues

Get-MigrationUserStatistics noob@vkarps.com -IncludeReport -Diagnostics | Export-Clixml noob@vkarps.com.xml
Mailbox Assessment

These are some one-liners to extract information to help target mailboxes for assesment to see if theyre ready to migrate to O365. This was also helpful to find relationships between user mailboxes and shared mailboxes. Note that these are to be run on the on-premises exchange servers.

Get last logon time

This was useful to identify mailboxes which have not been used in a while or are too large to be migrated. In my case most these were candidates for deletion, saving us licenses in Exchange Online or having difficult conversations to reduce mailbox sizes. Placing in a delimiter allows you to easily separate the columns in excel (text to columns function).

Get-Mailbox | Get-MailboxStatistics | Select-Object DisplayName,LastLoggedOnUserAccount,LastLogonTime,LastLogoffTime,ItemCount,TotalItemSize | Export-Csv -Delimiter ~ -NoTypeInformation C:\temp\MailboxLastLogon.csv
Get Mailbox permissions

This command will list the permissions for each mailbox if one or more accounts have access to it, apart from itself. You may have to do some further filtering on accounts, depending on your environment

Get-Mailbox -ResultSize unlimited | Get-MailboxPermission | where {$_.user.tostring() -ne "NT AUTHORITY\SELF" -and $_.IsInherited -eq $false} | Select Identity,User,@{Name='Access Rights';Expression={[string]::join(', ', $_.AccessRights)}} | Export-Csv -NoTypeInformation C:\Temp\MailboxPermissions.csv
Get Mailbox Type

This is helpful to quickly filter on the types of mailboxes and identity any that are not setup correctly. For example if a shared mailbox is setup as a user mailbox.

 Get-Mailbox -ResultSize unlimited | Select-Object SamAccountName,DisplayName,RecipientTypeDetails,PrimarySMTPAddress | Export-CSV C:\temp\MailboxType.csv
MSOL user operations
Assign License

This will obviously vary depending on what license you wish to apply, but here is the syntax

Set-MSolUserLicense -UserPrincipalName user@vkarps.com -UsageLocation AU
Set-MSolUserLicense -UserPrincipalName user@vkarps.com -AddLicenses vkarps:EXCHANGESTANDARD

Remove License
Set-MSolUserLicense -UserPrincipalName user@vkarps.com -RemoveLicenses vkarps:EXCHANGESTANDARD

Check License Status for users with @DOMAINNAME address
Get-MsolUser -All | Where-Object {$_.UserPrincipalName -like "*@vkarps.com"}

Check license status for list of users in a .csv file

The csv file needs to name ‘name’ in the first line, each following line should contain a valid UPN.

Import-CSV C:\Scripts\NewUsers.csv | ForEach {Get-MsolUser -UserPrincipalName "$($_.Name)"}
Change UPN for MSOL user

I came across instances where user accounts were created with UPNs for other domains / namespaces which were replicated created in the cloud. It can also happen if the UPN is changed after the account has been provisioned in Exchange Online. This issue becomes apparent when you try to migrate the user to Exchange Online. Here is an article which explains the issue:

Set-MsolUserPrincipalName -UserPrincipalName "noob.user@lab.vkarps.com" -NewUserPrincipalName "noob.user@vkarps.onmicrosoft.com"

Set-MsolUserPrincipalName -UserPrincipalName "noob.user@vkarps.onmicrosoft.com" -NewUserPrincipalName "noob.user@vkarps.com"
Deleting mailbox and user from Exchange Online

This may be required for a number of reasons, in my case it was due to duplicate mailboxes (one on-premises and the other on O365). Note: The mailbox will be deleted and cannot be recovered.

Remove-msoluser –userprincipalname noob.user@vkarps.com
Get-msoluser –userprincipalname noob.user@vkarps.com -ReturnDeletedUsers  fl *objectID*
Remove-MsolUser -ObjectID -RemoveFromRecycleBin -Force

You may have to give it a couple of minutes before running the next command

Get-mailbox -SoftdeletedMailbox noob.user@vkarps.com | Remove-Mailbox -PermanentlyDelete -Force -Confirm:$fals

 

Disconnected from Host. Agent is out of date and needs a manual upgrade

I recently performed an upgrade of vCenter from 5.0 to 5.5 and had an issue with just one of my hosts connecting to vCenter.

Agent out of date
First I tried to re-connect it manually with no success. I restarted the management agents from the iLO (not sure why I didn’t try go via ssh first as I normally would) and tried to connect again as this worked for other issues I came across in the past, still nothing.

OK, let’s uninstall the FDM client manually and try re-connecting the host (KB1003714):

cp /opt/vmware/uninstallers/VMware-fdm-uninstall.sh /tmp
chmod +x /tmp/VMware-fdm-uninstall.sh
/tmp/VMware-fdm-uninstall.sh

This time upon connecting the host I was prompted for the root credentials and proceeded through the wizard; enter next error:

Cannot contact the specified host (host1.lab.local) 
The host may not be available on the network, a network configuration 
problem may exist, or the management services on this host may not be
responding

Here, I did a sanity check and confirmed that DNS resolution was working and that I could ping the host, all good. Hrmm…

Next I went over to take a look at the fdm.log and the vpxa.log which were not giving me much to go off. So I went over to the events tab in the c# client to have another look at the error and noticed I was also getting and incorrect username and password error, dafaq ?

So I decided to ssh to the host to confirm that I wasn’t mistyping the password and then realized that I couldn’t initiate a session:

"Network Error: Connection Refused"

I checked lockdown mode was disabled, SSH service was running and the host firewall rules were okay. Hrmm, why on earth is SSH refusing my connection?

Some quick google-fu came up with the following KB article (KB1039095).
Back to the iLO I went and sure enough, the inetd.conf file was blank
I copied the contents of the configuration file from another host in the cluster, restarted the ssh daemon along with the management agents.

Here is my inetd.conf incase you don’t have another host to copy from:

# Internet server configuration database

# Remote shell access

ssh      stream   tcp   nowait   root   /usr/lib/vmware/openssh/bin/sshd       sshd ++swap,group=host/vim/vimuser/terminal/ssh -i
ssh      stream   tcp6  nowait   root   /usr/lib/vmware/openssh/bin/sshd       sshd ++swap,group=host/vim/vimuser/terminal/ssh -i

# VMware authentication daemon
authd   stream    tcp   nowait   root   /sbin/authd           authd
authd   stream    tcp6  nowait   root   /sbin/authd           authd

I could now ssh to the host, great! I proceeded to try connect the host and voila the host was able to connect back into the cluster.

As vCenter needs to copy over the installation files to the host for the installation of the FDM agent it was unable to do so as SSH was busted.

Here is another useful article I used during my troubleshooting (KB2004429).