Ensures the non-blocking request and status scratch can hold nMsg messages
in flight, growing it when it cannot.
GenerateDecomposition sizes the scratch from the mesh alone (one entry per unique side), but an exchange posts a send and a receive per rank-remote side per variable -- and, for vectors, per direction. The message count is therefore a multiple of the side count that the mesh cannot know, and a solution with enough variables overruns the scratch. Each exchange counts the sides it will message (CountRemoteSides / CountRemoteMortarSides) and reserves from that count before posting; the call is a comparison once the scratch is large enough.
Reserving from the rank-remote side count rather than from the local side
count matters at scale: the remote sides are the partition boundary, while the
local sides are the whole subdomain, and each entry carries an
MPI_STATUS_SIZE row of stats with it.
No message may be in flight when this is called: the scratch is reallocated, not grown in place. Every exchange posts and then waits (through FinalizeMPIExchangeAsync) before another can start, so a reservation at the head of an exchange is safe.
| Type | Intent | Optional | Attributes | Name | ||
|---|---|---|---|---|---|---|
| class(DomainDecomposition_t), | intent(inout) | :: | this | |||
| integer, | intent(in) | :: | nMsg |
subroutine ReserveMessages_DomainDecomposition_t(this,nMsg)
!! Ensures the non-blocking request and status scratch can hold `nMsg` messages
!! in flight, growing it when it cannot.
!!
!! GenerateDecomposition sizes the scratch from the mesh alone (one entry per
!! unique side), but an exchange posts a send and a receive per rank-remote side
!! *per variable* -- and, for vectors, per direction. The message count is
!! therefore a multiple of the side count that the mesh cannot know, and a
!! solution with enough variables overruns the scratch. Each exchange counts the
!! sides it will message (CountRemoteSides / CountRemoteMortarSides) and reserves
!! from that count before posting; the call is a comparison once the scratch is
!! large enough.
!!
!! Reserving from the rank-remote side count rather than from the local side
!! count matters at scale: the remote sides are the partition boundary, while the
!! local sides are the whole subdomain, and each entry carries an
!! MPI_STATUS_SIZE row of `stats` with it.
!!
!! No message may be in flight when this is called: the scratch is reallocated,
!! not grown in place. Every exchange posts and then waits (through
!! FinalizeMPIExchangeAsync) before another can start, so a reservation at the
!! head of an exchange is safe.
implicit none
class(DomainDecomposition_t),intent(inout) :: this
integer,intent(in) :: nMsg
if(nMsg <= this%maxMsg) return
if(allocated(this%requests)) deallocate(this%requests)
if(allocated(this%stats)) deallocate(this%stats)
allocate(this%requests(1:nMsg))
allocate(this%stats(MPI_STATUS_SIZE,1:nMsg))
this%maxMsg = nMsg
endsubroutine ReserveMessages_DomainDecomposition_t