Everybody probably knows the basics of VMware HA so I’m not going to explain how to set it up or that is uses a heartbeat for monitoring outages or isolation.
However I do want to explain the different kind of nodes, roles and isolation response. Here we go…
Primary and Secondary nodes
A VMware HA Cluster consists of nodes, primary and secondary nodes. Primary nodes hold cluster settings and all “node states” which are synced between primaries. Secondary nodes send their state info(resource occupation) to the primary nodes.
Nodes send a heartbeat to each other, which is the mechanism to detect possible outages. Primary nodes send heartbeats to primary nodes only. Secondary nodes also send their heartbeats only to primary nodes. Nodes send out these heartbeats every second by default. However this is a changeable value: das.failuredetectioninterval. (Advanced Settings on your HA-Cluster)
The first 5 hosts that join the VMware HA cluster are automatically selected as primary nodes. All the others are automatically selected as secondary nodes. When you do a reconfigure for HA the primary nodes and secondary nodes are selected again, this is random. When a primary node fails one might expect that an automatic re-election or promotion occurs. Be aware that this isn’t the case! A re-election/promotion of a single host only occurs when the failed host is either put in “Maintenance Mode” or removed from the cluster!
You are probably wondering what will happen when for instance all your primary nodes fail at the same time… No HA restart will take place! This is an unaddressed issue, that’s the reason why you can only account for 4 host failures within a cluster! Remember five primaries, four hosts failures = one primary left!
The fail-over coordinator
You need at least one primary because the “fail-over coordinator” role will be assigned to this primary. The fail-over coordinator coordinates the restart of virtual machines on the remaining primary and secondary hosts. The coordinator takes restart priorities in account. Keep in mind, when two hosts fail at the same time it will handle the restart sequentially. In other words, restart the VM’s of the first failed host(taking restart priorities in account) and then restart the VM’s of the host that failed as second(again taking restart priorities in account). If the fail-over coordinator fails one of the primaries will take over.
Isolation Response
Now that we are talking about restarts…. As of ESX 3.5 U3 / vCenter 3.5 U3 the new default isolation response value is “leave powered on”. In other words, when a host network isolation occurs the VM’s remain running. The default setting used to be “Power Off VM”, which meant a restart of the VM on a different host.
One might argue this setting. I prefer “Power off vm” because I don’t want to use a deprecated host. But to each his own. When the isolation response is set to “Leave powered on” it still restarts VM’s when a host completely dies.
I guess most of you would like to know how HA knows if the host is isolated or completely unavailable. This is because of the VMDK file lock, no other host will be able to boot a VM when the files are locked. Yes indeed, they will try to boot them because these hosts won’t know that the “missing” host is isolated, it could be completely dead. So if all network connections are dead, including the vSwitch for the VM’s these VM’s will not be restarted on a different host if they are using FC that is. They will only be restarted if for whatever reason also the storage connection fails and the VMDK file locks are timed out.
This also means that when using iSCSI you should never ever set this setting to “Leave powered on” cause this will cause a split brain scenario for sure. (VM’s will be restarted on a new host because the lock timed out, while the VM’s also are still running on the original host.)
So what happens when a host is completely dead, yes you’ve probably guesed it by now the VMDK file locks times out and the VM’s are restarted on another host.
No comments:
Post a Comment