Disconnected from Host. Agent is out of date and needs a manual upgrade

I recently performed an upgrade of vCenter from 5.0 to 5.5 and had an issue with just one of my hosts connecting to vCenter.

Agent out of date
First I tried to re-connect it manually with no success. I restarted the management agents from the iLO (not sure why I didn’t try go via ssh first as I normally would) and tried to connect again as this worked for other issues I came across in the past, still nothing.

OK, let’s uninstall the FDM client manually and try re-connecting the host (KB1003714):

cp /opt/vmware/uninstallers/VMware-fdm-uninstall.sh /tmp
chmod +x /tmp/VMware-fdm-uninstall.sh

This time upon connecting the host I was prompted for the root credentials and proceeded through the wizard; enter next error:

Cannot contact the specified host (host1.lab.local) 
The host may not be available on the network, a network configuration 
problem may exist, or the management services on this host may not be

Here, I did a sanity check and confirmed that DNS resolution was working and that I could ping the host, all good. Hrmm…

Next I went over to take a look at the fdm.log and the vpxa.log which were not giving me much to go off. So I went over to the events tab in the c# client to have another look at the error and noticed I was also getting and incorrect username and password error, dafaq ?

So I decided to ssh to the host to confirm that I wasn’t mistyping the password and then realized that I couldn’t initiate a session:

"Network Error: Connection Refused"

I checked lockdown mode was disabled, SSH service was running and the host firewall rules were okay. Hrmm, why on earth is SSH refusing my connection?

Some quick google-fu came up with the following KB article (KB1039095).
Back to the iLO I went and sure enough, the inetd.conf file was blank
I copied the contents of the configuration file from another host in the cluster, restarted the ssh daemon along with the management agents.

Here is my inetd.conf incase you don’t have another host to copy from:

# Internet server configuration database

# Remote shell access

ssh      stream   tcp   nowait   root   /usr/lib/vmware/openssh/bin/sshd       sshd ++swap,group=host/vim/vimuser/terminal/ssh -i
ssh      stream   tcp6  nowait   root   /usr/lib/vmware/openssh/bin/sshd       sshd ++swap,group=host/vim/vimuser/terminal/ssh -i

# VMware authentication daemon
authd   stream    tcp   nowait   root   /sbin/authd           authd
authd   stream    tcp6  nowait   root   /sbin/authd           authd

I could now ssh to the host, great! I proceeded to try connect the host and voila the host was able to connect back into the cluster.

As vCenter needs to copy over the installation files to the host for the installation of the FDM agent it was unable to do so as SSH was busted.

Here is another useful article I used during my troubleshooting (KB2004429).