Applies to SUSE OpenStack Cloud 7

J Recovering Clusters to a Healthy State

If one node in your cluster refuses to rejoin the cluster, it is most likely that the node has not been shut down cleanly. This can either happen because of manual intervention or because the node has been fenced (shut down) by the STONITH mechanism of the cluster. Fencing is used to protect the integrity of data in case of a split-brain scenario.

The following sections refer to problems with the Control Nodes cluster and show how to recover your degraded cluster to full strength. This takes the following basic steps:

  1. Re-adding the Node to the Cluster

  2. Recovering Crowbar and Chef

  3. In addition, you may need to reset resource failcounts to allow resources to start on the node you have re-added to the cluster. See Section J.4, “Cleaning Up Resources”.

  4. In addition, you may need to manually remove the maintenance mode flag from a node. See Section J.5, “Removing the Maintenance Mode Flag from a Node”.

For a list of possible symptoms that help you to diagnose a degraded cluster, see Section J.1, “Symptoms of a Degraded Control Node Cluster”.

J.1 Symptoms of a Degraded Control Node Cluster

The following incidents may occur if a Control Node in your cluster has been shut down in an unclean state:

  • A VM reboots although the SUSE OpenStack Cloud administrator did not trigger this action.

  • One of the Control Node in the Crowbar Web interface is in status Problem, signified by a red dot next to the node.

  • The Hawk Web interface stops responding on one of the Control Nodes, while it is still responding on the others.

  • The SSH connection to one of the Control Nodes freezes.

  • The OpenStack components stop responding for a short while.

J.2 Re-adding the Node to the Cluster

  1. Reboot the node.

  2. Connect to the node via SSH from the Administration Server.

  3. If you have a 2-node cluster with the STONITH: Do not start corosync on boot after fencing option set to Automatic, remove the block file that is created on a node during start of the cluster service:

    root # rm /var/spool/corosync/block_automatic_start

    The block file avoids STONITH deathmatches for 2-node clusters (where each node kills the other one, resulting in both nodes rebooting all the time). When Corosync shuts down cleanly, the block file is automatically removed. Otherwise the block file is still present and prevents the cluster service from (re-)starting on that node.

    Alternatively, bypass the block file by starting the cluster service on the cluster node before reconnecting the node to Crowbar in the next section (see Section J.3, “Recovering Crowbar and Chef”):

    root # systemctl start pacemaker

J.3 Recovering Crowbar and Chef

Making the Pacemaker node rejoin the cluster is not enough. All nodes in the cloud (including the Administration Server) need to be aware that this node is back online. This requires the following steps for Crowbar and Chef:

  1. Log in to the node you have re-added to the cluster.

  2. Re-register the node with Crowbar by executing:

    root # systemctl start crowbar_join
  3. Log in to each of the other Control Nodes.

  4. Trigger a Chef run:

    root # chef-client

J.4 Cleaning Up Resources

A resource will be automatically restarted if it fails, but each failure increases the resource's failcount. If a migration-threshold has been set for the resource, the node will no longer run the resource when the number of failures reaches the migration threshold. To allow the resource to start again on the node, reset the resource's failcount by cleaning up the resource manually. You can clean up individual resources by using the Hawk Web interface or all in one go as described below:

  1. Log in to any one of the cluster nodes which is currently online in the cluster. (You can check this via crm_mon.)

  2. Clean-up all stopped resources with the following command:

    root # crm_resource -o | \
      awk '/\tStopped |Timed Out/ { print $1 }' | \
      xargs -r -n1 crm resource cleanup

J.5 Removing the Maintenance Mode Flag from a Node

During normal operation, chef-client sometimes needs to place a node into maintenance mode. The node is kept in maintenance mode until the chef-client run finishes. However, if the chef-client run fails, the node may be left in maintenance mode. In that case, the cluster management tools like crmsh or Hawk will show all resources on that node as unmanaged. To remove the maintenance flag:

  1. Log in the cluster node.

  2. Disable the maintenance mode with:

    root # crm node ready

J.6 Recovering From an Unresolvable DRBD Split Brain Situation

Although policies to automatically resolve a DRBD split brain situations exist, there are situations which require to be resolved manually. Such a situation is indicated by a Kernel message like:

kernel: block drbd0: Split-Brain detected, dropping connection!

To resolve the split brain you need to choose a node which data modifications will be discarded. These modifications will be replaced by the data from the healthy node and will not be recoverable, so make sure to choose the right node. If in doubt, make a backup of the node before starting the recovery process. Proceed as follows:

  1. Put the cluster in maintenance mode:

    root # crm configure property maintenance-mode=true
  2. Check if the chose node is in primary role:

    root # systemctl status drbd
  3. If it is in primary role, stop all services using this resource and switch it to secondary role. If the node already is in secondary role, skip this step.

    root # drbdadm secondary RESOURCE
  4. Check if a node is in state WFConnection by looking at the output of systemctl status drbd.

  5. If the node is in state WFConnection disconnect the resource:

    root # drbdadm disconnect RESOURCE
  6. Discard all modifications on the chosen node. This step is irreversible, the modifications on the chosen node will be lost!

    root # drbdadm -- --discard-my-data connect RESOURCE
  7. If the other (healthy) node is in state WFConnection, synchronization to the chosen node will start automatically. If not, reconnect the healthy node to start the synchronization:

    root # drbdadm connect RESOURCE

    During the synchronization all data modifications on the chosen node will be overwritten with the data from the healthy node.

  8. When the synchronization has finished, reset the cluster to normal mode:

    root # crm configure property maintenance-mode=false
Print this page