Debug Ansible Failures in init

Look for cray-ansible failures during init in the SMW logs and in the Ansible logs on the individual nodes.

Check the console log on the SMW to find out which nodes failed. Ansible failures in init drop a node into debug shell. The boot process is not allowed to continue until cray-ansible during init is successful on a node.
  1. Look for cray-ansible failures in the SMW console log.
    crayadm@smw~> /var/opt/cray/log/p0-current> cat console-20160523 | grep 'FAILED in init phase'
    <158>1 2016-05-23T12:01:22.576591-05:00 c0-0c0s0n1 xtconsole 31798 p0-20160523t115109 [console@34] cray-ansible: serial start of play type cle FAILED in init phase
<158>1 2016-05-23T12:01:22.576634-05:00 c0-0c0s0n1 xtconsole 31798 p0-20160523t115109 [console@34] cray-ansible: serial start of play type cle FAILED in init phase
<158>1 2016-05-23T12:01:34.411653-05:00 c0-0c0s1n2 xtconsole 31798 p0-20160523t115109 [console@34] cray-ansible: serial start of play type cle FAILED in init phase
<158>1 2016-05-23T12:01:34.411699-05:00 c1-0c2s1n2 xtconsole 31798 p0-20160523t115109 [console@34] cray-ansible: serial start of play type cle FAILED in init phase
  2. Access the debug shell with xtcon from the SMW.
    smw# xtcon c1-0c2s1n2
    DEBUG(err_ansible)#
  3. Inspect Ansible logs on the node in /var/opt/cray/log/ansible, make a configuration change in the config set, or do some other corrective action. Exiting from the debug shell causes cray-ansible to run again in init.