SELF_SetInterface_3D_t Module

  1. SELF_SetInterface_3D_t.f90
  2. SELF_SetInterface_3D_t

Interface map and trace exchange between element sets.

An interface face is a face of the global mesh whose two elements belong to different element sets (SELF_ElementSetMesh_3D). In the sub-mesh of each set it is tagged SELF_BC_SET_INTERFACE, and its exterior state must come from the other set. For every interface face owned by this rank, this module receives the partner element's face trace,

ghost(i,j,v,f) = b_p(i2,j2,s2,e2,v), v = 1..nvar(p),

where p is the partner set, (e2,s2) the partner element and side, b_p the partner's boundary trace, and (i2,j2) the receiver's flip mapping of (i,j). The mapping is the one SideExchange applies for a face with flip = mod(sideInfo(4),10), so a ghost value equals the exterior state the unsplit mesh would have had on that face.

Faces. Local interface faces are numbered f = 1..nFaces, grouped by receiving set: the faces of set k are setFaceStart(k) .. setFaceStart(k+1)-1, in global element order. Each face both sends its own trace to the partner's rank and receives the partner's trace.

Messages. Faces whose partner is on another rank are grouped by partner rank. Within each group the faces are ordered by global side id, which both ranks know, and each face carries NpNpnvar values of the sending set (Np = N+1). The receiver knows the partner set's nvar, so both ends agree on the layout without negotiation. The exchange uses persistent point-to-point requests with a constant tag on a duplicate of the global mesh communicator, so it cannot match any halo message of the sub-meshes. Faces whose partner is on the same rank are copied directly.

Each exchange runs in two parts. GatherOutbox copies every local interface face's own trace, unflipped, into outbox(:,:,v,f); a GPU backend overrides it to gather on the device and copy the outbox to the host. ExchangeOutbox then sends, receives and flips on the host, so both backends fill ghost with the same values.

Collectives. BuildMap duplicates the communicator and Free releases it; both are collective over the global mesh communicator. Exchange posts point-to-point messages only, and only between ranks that share interface faces.


Uses

  • module~~self_setinterface_3d_t~~UsesGraph module~self_setinterface_3d_t SELF_SetInterface_3D_t module~self_constants SELF_Constants module~self_setinterface_3d_t->module~self_constants module~self_mesh_3d SELF_Mesh_3D module~self_setinterface_3d_t->module~self_mesh_3d mpi mpi module~self_setinterface_3d_t->mpi iso_c_binding iso_c_binding module~self_setinterface_3d_t->iso_c_binding module~self_constants->iso_c_binding iso_fortran_env iso_fortran_env module~self_constants->iso_fortran_env module~self_mesh_3d->iso_c_binding module~self_mesh_3d_t SELF_Mesh_3D_t module~self_mesh_3d->module~self_mesh_3d_t module~self_gpu SELF_GPU module~self_mesh_3d->module~self_gpu module~self_mesh_3d_t->module~self_constants module~self_mesh_3d_t->iso_c_binding module~self_supportroutines SELF_SupportRoutines module~self_mesh_3d_t->module~self_supportroutines module~self_mesh SELF_Mesh module~self_mesh_3d_t->module~self_mesh module~self_quadrature SELF_Quadrature module~self_mesh_3d_t->module~self_quadrature module~self_hdf5 SELF_HDF5 module~self_mesh_3d_t->module~self_hdf5 HDF5 HDF5 module~self_mesh_3d_t->HDF5 module~self_domaindecomposition SELF_DomainDecomposition module~self_mesh_3d_t->module~self_domaindecomposition module~self_mesh_2d_t SELF_Mesh_2D_t module~self_mesh_3d_t->module~self_mesh_2d_t module~self_lagrange~3 SELF_Lagrange module~self_mesh_3d_t->module~self_lagrange~3 module~self_gpu->iso_c_binding module~self_gpu_enums~2 SELF_GPU_enums module~self_gpu->module~self_gpu_enums~2 module~self_supportroutines->module~self_constants module~self_supportroutines->iso_fortran_env module~self_mesh->module~self_constants module~self_mesh->iso_c_binding module~self_mesh->module~self_domaindecomposition module~self_quadrature->module~self_constants module~self_quadrature->iso_fortran_env module~self_hdf5->module~self_constants module~self_hdf5->mpi module~self_hdf5->iso_fortran_env module~self_hdf5->HDF5 module~self_domaindecomposition->mpi module~self_domaindecomposition->iso_c_binding module~self_domaindecomposition->iso_fortran_env module~self_domaindecomposition_t SELF_DomainDecomposition_t module~self_domaindecomposition->module~self_domaindecomposition_t module~self_mesh_2d_t->module~self_constants module~self_mesh_2d_t->iso_c_binding module~self_mesh_2d_t->module~self_supportroutines module~self_mesh_2d_t->module~self_mesh module~self_mesh_2d_t->module~self_quadrature module~self_mesh_2d_t->module~self_hdf5 module~self_mesh_2d_t->HDF5 module~self_mesh_2d_t->module~self_domaindecomposition module~self_mesh_2d_t->module~self_lagrange~3 module~self_lagrange~3->module~self_constants module~self_lagrange~3->iso_c_binding module~self_lagrange~3->iso_fortran_env module~self_lagrange~3->module~self_gpu module~self_lagrange_t SELF_Lagrange_t module~self_lagrange~3->module~self_lagrange_t module~self_gpu_enums~2->iso_c_binding module~self_domaindecomposition_t->module~self_constants module~self_domaindecomposition_t->mpi module~self_domaindecomposition_t->iso_c_binding module~self_domaindecomposition_t->module~self_supportroutines module~self_domaindecomposition_t->module~self_lagrange~3 module~self_lagrange_t->module~self_constants module~self_lagrange_t->iso_c_binding module~self_lagrange_t->iso_fortran_env module~self_lagrange_t->module~self_supportroutines module~self_lagrange_t->module~self_quadrature module~self_lagrange_t->module~self_hdf5 module~self_lagrange_t->HDF5

Used by

  • module~~self_setinterface_3d_t~~UsedByGraph module~self_setinterface_3d_t SELF_SetInterface_3D_t module~self_setinterface_3d SELF_SetInterface_3D module~self_setinterface_3d->module~self_setinterface_3d_t module~self_setinterface_3d~2 SELF_SetInterface_3D module~self_setinterface_3d~2->module~self_setinterface_3d_t module~self_multisetmodel3d_t SELF_MultiSetModel3D_t module~self_multisetmodel3d_t->module~self_setinterface_3d module~self_multisetmodel3d SELF_MultiSetModel3D module~self_multisetmodel3d->module~self_setinterface_3d module~self_multisetmodel3d->module~self_multisetmodel3d_t module~self_multisetmodel3d~2 SELF_MultiSetModel3D module~self_multisetmodel3d~2->module~self_multisetmodel3d_t

Contents

Variables

Derived Types

Subroutines


Variables

TypeVisibilityAttributesNameInitial
integer, private, parameter:: SET_INTERFACE_TAG =0

Derived Types

type, public :: SetInterface3D_t

Components

TypeVisibilityAttributesNameInitial
integer, public :: N =0
integer, public :: comm =MPI_COMM_NULL
integer, public, allocatable:: faceElem(:)

element in set k's sub-mesh
integer, public, allocatable:: faceFlip(:)

mod(sideInfo(4),10) of the receiving side
integer, public, allocatable:: facePartnerElem(:)

partner element in set p's sub-mesh; 0 if remote
integer, public, allocatable:: facePartnerFace(:)

local face index of the partner; 0 if remote Own and partner traces, (1:Np,1:Np,1:nvarMax,1:nFaces). Pointers so a GPU backend can copy them to and from device memory with c_loc.
integer, public, allocatable:: facePartnerRank(:)

rank owning the partner element
integer, public, allocatable:: facePartnerSet(:)

partner set p
integer, public, allocatable:: facePartnerSide(:)

partner local side 1..6
integer, public, allocatable:: faceSet(:)

receiving set k
integer, public, allocatable:: faceSide(:)

local side 1..6
integer, public, allocatable:: faceSideId(:)

|global side id|
real(kind=prec), public, pointer, contiguous:: ghost(:,:,:,:)=> null()

Neighbour ranks and their faces
integer, public :: mpiPrec
integer, public :: nFaces =0
integer, public :: nNbr =0
integer, public :: nSets =0
integer, public, allocatable:: nbrFace(:)

local face index, ordered by neighbour then side id
integer, public, allocatable:: nbrFaceStart(:)

(1:nNbr+1)
integer, public, allocatable:: nbrRank(:)
integer, public :: nvarMax =0

Local interface faces, f = 1..nFaces
integer, public, allocatable:: nvarOfSet(:)
real(kind=prec), public, pointer, contiguous:: outbox(:,:,:,:)=> null()
integer, public :: rankId =0
real(kind=prec), public, allocatable:: recvBuf(:)
integer, public, allocatable:: recvStart(:)

(1:nNbr+1), offsets into recvBuf
integer, public, allocatable:: requests(:)
real(kind=prec), public, allocatable:: sendBuf(:)
integer, public, allocatable:: sendStart(:)

(1:nNbr+1), offsets into sendBuf
integer, public, allocatable:: setFaceStart(:)

(1:nSets+1)

Type-Bound Procedures

procedure, public :: BuildMap => BuildMap_SetInterface3D_t
procedure, public :: Exchange => Exchange_SetInterface3D_t
procedure, public :: ExchangeOutbox => ExchangeOutbox_SetInterface3D_t
procedure, public :: Free => Free_SetInterface3D_t
procedure, public :: GatherOutbox => GatherOutbox_SetInterface3D_t

type, public :: SetTraceRef

Face trace of one set: trace(1:Np,1:Np,1:6,1:nElem,1:nvar), for example solution%boundary of the set's model.

Components

TypeVisibilityAttributesNameInitial
real(kind=prec), public, pointer, contiguous:: trace(:,:,:,:,:)=> null()

Device copy of the same trace (GPU backend); left null on the host path.
type(c_ptr), public :: trace_gpu =c_null_ptr

Subroutines

public subroutine BuildMap_SetInterface3D_t(this, globalMesh, setOfElem, setGlobalId, setOffsetThisRank, nvarOfSet, N)

Build the interface map from the global mesh and the set numbering, without communication beyond duplicating the communicator. Collective over globalMesh%decomp%mpiComm.

Read more…

Arguments

TypeIntentOptionalAttributesName
class(SetInterface3D_t), intent(inout) :: this
type(Mesh3D), intent(in) :: globalMesh
integer, intent(in) :: setOfElem(1:)
integer, intent(in) :: setGlobalId(1:)
integer, intent(in) :: setOffsetThisRank(1:)
integer, intent(in) :: nvarOfSet(1:)
integer, intent(in) :: N

public subroutine ExchangeOutbox_SetInterface3D_t(this)

Send each remote face's outbox to the partner's rank, and fill

Read more…

Arguments

TypeIntentOptionalAttributesName
class(SetInterface3D_t), intent(inout) :: this

public subroutine Exchange_SetInterface3D_t(this, traces)

Fill ghost(:,:,1:nvar(p),f) for every local interface face f with the partner's face trace, flipped into the receiving face's orientation.

Read more…

Arguments

TypeIntentOptionalAttributesName
class(SetInterface3D_t), intent(inout) :: this
type(SetTraceRef), intent(in) :: traces(1:)

public pure subroutine FlipIndex(i, j, flip, N, i2, j2)

Index (i2,j2) on the sending face that supplies node (i,j) of the receiving face, for the receiving side's flip in 0..7. Identical to the mapping in SideExchange_MappedScalar3D_t and ApplyFlip_MappedScalar3D_t.

Arguments

TypeIntentOptionalAttributesName
integer, intent(in) :: i
integer, intent(in) :: j
integer, intent(in) :: flip
integer, intent(in) :: N
integer, intent(out) :: i2
integer, intent(out) :: j2

public subroutine Free_SetInterface3D_t(this)

Release the persistent requests, buffers and the duplicate communicator. Collective over the communicator the map was built on.

Arguments

TypeIntentOptionalAttributesName
class(SetInterface3D_t), intent(inout) :: this

public subroutine GatherOutbox_SetInterface3D_t(this, traces)

outbox(:,:,v,f) = trace of set k on face f, v = 1..nvar(k), unflipped, for every local interface face f of every set k. Host traces.

Arguments

TypeIntentOptionalAttributesName
class(SetInterface3D_t), intent(inout) :: this
type(SetTraceRef), intent(in) :: traces(1:)

public subroutine HeapSortByKey(keys, vals)

Sort keys(1:n) ascending in place, permuting vals(1:n) alongside. Heap sort: O(n log n), no extra storage. Keys are global side ids, distinct within one neighbour's face list, so stability does not matter.

Arguments

TypeIntentOptionalAttributesName
integer, intent(inout) :: keys(1:)
integer, intent(inout) :: vals(1:)