Sierra Toolkit  Version of the Day
Sierra Toolkit Rebalance Use Cases
Collaboration diagram for Sierra Toolkit Rebalance Use Cases:

Use Case 1: Unequal element weights

This use case demonstrates unequal element weights. A single-element-thick column of hex elements is constructed in the x-direction. Weights are assigned to elements such that a perfect rebalance is possible with Proc_Rank+1 elements placed on each processor. This is achieved when running on #procs < 4, but Zoltan does not produce the optimal new distribution for 4 or more procs.

The following hex8 mesh is used in this use case:

  Global node and element numbering
 <pre>
      3       7      11      15      19                 
      +-------+-------+-------+-------+                +-------+
     /       /       /       /       /|               /       /|
   4/      8/     12/     16/     20/ |              /       / |    Z  Y
   +-------+-------+-------+-------+  |     ......  +-------+  |    | / 
   |       |       |       |       |  +18           |       |  /    |/  
   |  e1   |  e2   |  e3   |  e4   | /              |  eN   | /     *--X       
   |       |       |       |       |/               |       |/         
   +-------+-------+-------+-------+                +-------+
   1       5      9       13      17                  
 </pre>
  where N = #elements = (#procs)(#procs+1)/2

  Local node numbering
 <pre>
      8       7
      +-------+
     /       /|
   5/      6/ |
   +-------+  |
   |       |  +3
   |  e1   | /
   |       |/
   +-------+
   1       2
 </pre>

The mesh is constructed on proc 0 using the HexFixture class and unequal element weights are assigned as follows:

// Assign weights so that a perfect rebalance is possible so long as the rebalancer can figure out
// to put p_rank+1 elements on proc = p_rank based on these weights.
unsigned nslabs = 0;
if( 0 == p_rank ) {
for ( unsigned l = 1 ; l <= p_size ; ++l ) {
for ( unsigned k = 0 ; k < nz ; ++k ) {
for ( unsigned j = 0 ; j < ny ; ++j ) {
for ( unsigned i = 0 ; i < l ; ++i ) {
const stk_classic::mesh::EntityId elem_id = 1 + nslabs + i + j*ngx + k*ngx*ny;
stk_classic::mesh::Entity * elem = bulk.get_entity(element_rank, elem_id);
double * const e_weight = stk_classic::mesh::field_data( weight_field , *elem );
*e_weight = double(ngx) / double(l);
}
}
}
nslabs += l;
}
}
// end assign weights

See UseCase_Rebal_1.cpp for the complete source listing.

Use Case 2: Node, Edge, Face and Element weights on subset

This use case demonstrates element weights comprised of contributions from nodes, edges, faces and elements over a subset of the mesh. A 3x3x3 cube of hex8 elements is constructed on proc 0 and weights are assigned to a subset of mesh entities. The mesh and weights are assigned as follows:

  Global node and element numbering
 <pre>

           +-------+-------+-------+               +                   +                 +
          /       /       /       /|              /|                  /|         
         /       /       /       / |             / |                 / |        
        +-------+-------+-------+  |            +  |                +  |    
       /       /       /       /|  +           /|  +               /   +   
      /       /       /       / | /|          / | /|              /    |  
     +-------+-------+-------+  |/ |         +  |/ |             +     | 
    /       /       /       /|  +  |        /|  +  |            /      |
   /       /       /       / | /|  +       / | /|  +           /       +
  +-------+-------+-------+  |/ | /|      +  |/ | /|          +        |        +
  |       |       |       |  +  |/ |      |  +  |/ |          |        |    
  |  e1   |  e2   |  e3   | /|  +  |      | /|  +  |          |        |        
  |       |       |       |/ | /|  +      |/ | /|  +          |        +                 +
  +-------+-------+-------+  |/ | /       +  |/ | /           +       /         
  |       |       |       |  +  |/        |  +  |/            |      /      
  |  e1   |  e2   |  e3   | /|  +         | /|  +             |     +      
  |       |       |       |/ | /          |/ | /              |    /      
  +-------+-------+-------+  |/           +  |/               +   /      
  |       |       |       |  +            |  +                |  +      
  |  e1   |  e2   |  e3   | /             | /                 | /      
  |       |       |       |/              |/                  |/      
  +-------+-------+-------+               +                   +                 +
x = 0
 </pre>
  Weight_elems = 1.0                 Z  Y      Local node numbering
  Weight_faces = 10.0                | /      
  Weight_edges = 100.0               |/            8       7       
  Weight_nodes = 1000.0              *--X          +-------+              
                                                  /       /|       
                                                5/      6/ |       
                                                +-------+  |   
                                                |       |  +3   
                                                |  e1   | /
                                                |       |/
                                                +-------+
                                                1       2
  

where all 27 elements are assigned weights along with the 9 faces, 12 edges and 4 nodes on the plane at x = 0.

// Assign entity weights
if( 0 == p_rank )
{
// Get the faces on the x=0 plane and give them a characteristic weight
stk_classic::mesh::EntityVector selected_nodes;
stk_classic::mesh::EntityVector selected_faces;
stk_classic::mesh::EntityVector one_face;
for ( unsigned j = 0 ; j < ny; ++j )
for ( unsigned k = 0 ; k < nz; ++k )
{
selected_nodes.clear();
selected_nodes.push_back( fixture.node(0, j, k ) );
selected_nodes.push_back( fixture.node(0, j+1, k ) );
selected_nodes.push_back( fixture.node(0, j, k+1) );
selected_nodes.push_back( fixture.node(0, j+1, k+1) );
stk_classic::mesh::get_entities_through_relations(selected_nodes, face_rank, one_face);
selected_faces.push_back(one_face[0]);
}
for( size_t iface = 0; iface < selected_faces.size(); ++iface )
{
stk_classic::mesh::Entity * face = selected_faces[iface];
double * const weight = stk_classic::mesh::field_data( weight_field, *face );
weight[0] = 10.0;
}
// Get the edges on the boundary of the x=0 plane and give them a characteristic weight
stk_classic::mesh::EntityVector selected_edges;
stk_classic::mesh::EntityVector one_edge;
for ( unsigned j = 0 ; j < ny; ++j )
{
selected_nodes.clear();
selected_nodes.push_back( fixture.node(0, j, 0) );
selected_nodes.push_back( fixture.node(0, j+1, 0) );
stk_classic::mesh::get_entities_through_relations(selected_nodes, edge_rank, one_edge);
selected_edges.push_back(one_edge[0]);
selected_nodes.clear();
selected_nodes.push_back( fixture.node(0, j, nz) );
selected_nodes.push_back( fixture.node(0, j+1, nz) );
stk_classic::mesh::get_entities_through_relations(selected_nodes, edge_rank, one_edge);
selected_edges.push_back(one_edge[0]);
}
for ( unsigned k = 0 ; k < nz; ++k )
{
selected_nodes.clear();
selected_nodes.push_back( fixture.node(0, 0, k) );
selected_nodes.push_back( fixture.node(0, 0, k+1) );
stk_classic::mesh::get_entities_through_relations(selected_nodes, edge_rank, one_edge);
selected_edges.push_back(one_edge[0]);
selected_nodes.clear();
selected_nodes.push_back( fixture.node(0, ny, k) );
selected_nodes.push_back( fixture.node(0, ny, k+1) );
stk_classic::mesh::get_entities_through_relations(selected_nodes, edge_rank, one_edge);
selected_edges.push_back(one_edge[0]);
}
for( size_t iedge = 0; iedge < selected_edges.size(); ++iedge )
{
stk_classic::mesh::Entity * edge = selected_edges[iedge];
double * const weight = stk_classic::mesh::field_data( weight_field, *edge );
weight[0] = 100.0;
}
// Finally, give the corner nodes of the x=0 plane a characteristic weight
selected_nodes.clear();
double * weight = stk_classic::mesh::field_data( weight_field, *fixture.node(0, 0, 0) );
weight[0] = 1000.0;
weight = stk_classic::mesh::field_data( weight_field, *fixture.node(0, ny, 0) );
weight[0] = 1000.0;
weight = stk_classic::mesh::field_data( weight_field, *fixture.node(0, 0, nz) );
weight[0] = 1000.0;
weight = stk_classic::mesh::field_data( weight_field, *fixture.node(0, ny, nz) );
weight[0] = 1000.0;
// Assign element weights
for( size_t i = 0; i < my_element_ids.size(); ++i )
{
stk_classic::mesh::Entity * elem = bulk.get_entity(element_rank, my_element_ids[i]);
double * const e_weight = stk_classic::mesh::field_data( weight_field , *elem );
*e_weight = 1.0;
}
//
// Get the elements on the x=0 plane and sum in weights from relations
selected_nodes.clear();
for ( unsigned j = 0 ; j < ny+1; ++j )
for ( unsigned k = 0 ; k < nz+1; ++k )
selected_nodes.push_back( fixture.node(0, j, k) );
std::vector<stk_classic::mesh::EntityRank> ranks;
ranks.push_back(face_rank);
ranks.push_back(edge_rank);
ranks.push_back(node_rank);
stk_classic::mesh::EntityVector selected_elems;
for ( unsigned j = 0 ; j < ny; ++j )
for ( unsigned k = 0 ; k < nz; ++k )
{
selected_nodes.clear();
selected_nodes.push_back( fixture.node(0, j, k ) );
selected_nodes.push_back( fixture.node(0, j+1, k ) );
selected_nodes.push_back( fixture.node(0, j, k+1) );
selected_nodes.push_back( fixture.node(0, j+1, k+1) );
stk_classic::mesh::get_entities_through_relations(selected_nodes, element_rank, one_face);
selected_elems.push_back(one_face[0]);
}
sum_element_weights_through_relations(selected_elems, weight_field, ranks);
}

The use case passes if the amount of imbalance following a rebalance is below 1.45 for 3 procs and below 1.1 for 2 or 4 procs.

See UseCase_Rebal_2.cpp for the complete source listing.

Use Case 3: Periodic Boundary via Constraint Relations

This use case sets up a 2D mesh of quad4 elements and then establishes a constraint realtion between the top and bottom of the mesh as would be needed to enforce periodic boundary conditions.

The following quad4 mesh is manually constructed on proc 0:

  Global node and element numbering
 <pre>

   21      22      23      24      25
   +-------+-------+-------+-------+   y = top
   |       |       |       |       |
   |  e13  |  e14  |  e15  |  e16  |
   |       |       |       |5      |
16 +-------+-------+-------+-------+ 20
   |       |       |       |       |      
   |  e9   |  e10  |  e11  |  e12  |      
   |       |       |       |       |      
11 +-------+-------+-------+-------+ 15
   |       |       |       |       |
   |  e5   |  e6   |  e7   |  e8   |
   |       |5      |       |       |  
 6 +-------+-------+-------+-------+ 10
   |       |       |       |       |  
   |  e1   |  e2   |  e3   |  e4   |    
   |       |       |       |       | 
   +-------+-------+-------+-------+   y = bottom
   1       2       3       4       5
 </pre>

Local node numbering:

       3       4
       +-------+     Y
       |       |     |
       |  e1   |     |
       |       |     *--> X
       +-------+
       1       2
     

and the two sets of nodes at y=bottom and y=top are related through constraint relations as follows:

// Assign constraint relations between nodes at top and bottom of mesh
{
const unsigned iy_bottom = 0;
const unsigned iy_top = ny;
stk_classic::mesh::PartVector add(1, &fem_meta.locally_owned_part());
for ( unsigned ix = 0 ; ix <= nx ; ++ix ) {
stk_classic::mesh::EntityId nid_bottom = 1 + ix + iy_bottom * nnx ;
stk_classic::mesh::EntityId nid_top = 1 + ix + iy_top * nnx ;
stk_classic::mesh::Entity * n_bottom = bulk_data.get_entity( node_rank, nid_bottom );
stk_classic::mesh::Entity * n_top = bulk_data.get_entity( node_rank, nid_top );
const stk_classic::mesh::EntityId constraint_entity_id = 1 + ix + nny * nnx;
stk_classic::mesh::Entity & c = bulk_data.declare_entity( constraint_rank, constraint_entity_id, add );
bulk_data.declare_relation( c , *n_bottom , 0 );
bulk_data.declare_relation( c , *n_top , 1 );
}
} // end snippet

The use case passes if the load imbalance of the new partition is below the nominal value of 1.5.

See UseCase_Rebal_3.cpp for the complete source listing.

Use Case 4: User-customization following default rebalance

This use case demonstrates additional user customization following a default rebalance in order to enforce constraints for new partitions. In this case, the constraint is that two quad4 elements sharing edge #7 be collocated on the same proc following rebalance. This is enforced using a greedy sideset class which inherits the determine_new_partition method.

The following quad4 mesh is used in this use case:

Global node and element numbering

    13      14      15      16
    +-------+-------+-------+
    |       |       |       |
    |  e7   |  e8   |  e9   |
    |       |       |       |
  9 +-------+-------+-------+ 12    Y
    |       |       |       |       |
    |  e4   |  e5   |  e6   |       |
    |       |       |       |       *--> X
  5 +-------+-------+-------+ 8
    |       |       |       | 
    |  e1   |  e2   |  e3   |
    |       |       |       | 
    +-------+-------+-------+ 
    1       2       3       4

Local node numbering

          3       4
          +-------+
          |       |
          |  e1   |
          |       |
          +-------+
          1       2

See UseCase_Rebal_4.cpp for the complete source listing.