Limitations of SMW Failover

The SMW HA failover feature has the following limitations:

  • Both SMWs must be of the same type, either Dell PowerEdge 815s, Dell PowerEdge 630s, or Dell PowerEdge 640s.
  • Both SMWs must run the same versions of SLES and SMW/HSS software.
  • System administration of an SMW HA environment is more complex than administration of a system with a single SMW.
  • HSS Commands on the SMW do not work immediately after a failover. The failover takes time to complete. If an HSS command is issued on the active SMW before the HA system has settled, it will fail. The system must complete the failover before HSS commands are executed.

    After a failover, there is a delay before the cluster is fully available. In the first 30 seconds after failover, resources may appear to be started, then change to another state. Although logging in via the virtual IP address is possible before this period is over, the cluster is not ready for use until all resources are fully started. Cray recommends waiting for 30 - 60 seconds before using a command that interacts with the HSS daemons, to ensure that all cluster resources have started and that the shared file systems are mounted on the active SMW.

  • SMW and CLE upgrades in an HA environment require some duplication of effort, with portions of the procedure done individually to each SMW. System down-time requirements for operating system upgrades are somewhat longer as a result.
  • There is no support for seamless failover (also called double failure) if errors occur while the system is doing error handling for another system component. If an HSS daemon or other SMW process were doing some type of error handling that got interrupted by an (unrelated) failover, when that daemon restarts on the new SMW it may not be able to resume operation where it left off and complete the recovery from the first error. In this case, even though a failover occurs, manual intervention might still be required to return the system to an operational state.
  • There is no support for seamless failover during operational commands. All user commands that were started from the active SMW are terminated. These commands must be restarted on the new active SMW. The restarted commands might not start with the same internal states, if those commands do not provide persistent capabilities. An interrupted operation such as xtbootsys, shutdown, dump, warm-swap, or flash will need to be reissued after failover has completed and the other SMW becomes active.
  • Partial migration of managed resources is not supported. For example, the SMW HA system does not support migration of individual HSS daemons or resources to the other SMW. A particular SMW is either active, with complete responsibility for all HSS daemons, or passive with no HSS daemons running.
  • If both SMWs are started (powered on) at the same time, a race condition can develop that could result in one SMW being powered off via the STONITH capability. Before starting the second SMW, wait until the first SMW has completed startup and initialized all cluster resources.
  • During failover, if there is no communication between the SMW and the Cray mainframe for about 30 seconds, workload throttling can occur; therefore, auto-throttling of applications is likely while an actual SMW failover is taking place. Blades begin to auto-throttle if essential HSS daemons (erd, state-manager, or xtnlrd) are unavailable and lasts until those daemons resume operation on the other SMW. On a single-cabinet system, the throttled period was fairly consistent, lasting approximately 120 seconds. The throttled period may increase for larger systems.