ExchangeOldWindow Subroutine

public subroutine ExchangeOldWindow(decomp, Np, nvar, nLocalOld, uLocal, winFirst, winLast, wFirst, wLast, uWin, nBytesRecv, nBytesSent, nElemRemote)

Migrate the pre-regrid solution into this rank's old-element window with point-to-point messages: the Stage-5 v2 replacement for allgathering the whole old field onto every rank.

The schedule itself lives in SELF_SolutionMigration, on flat buffers, so that one copy of it serves both dimensions and every backend - in particular so that the GPU backend can run the same message set with the window in device memory. This wrapper is the 3-D host form: it exists because the plan-window tests and the portable path read and write the solution as a rank-5 array, and passing the whole array to the flat routine's assumed-size dummy is sequence association over storage that is contiguous in every case SELF constructs.

Runs once per adapting epoch, between time steps - never inside the time-stepping loop.

Arguments

TypeIntentOptionalAttributesName
type(DomainDecomposition), intent(in) :: decomp
integer, intent(in) :: Np

nodes per direction (N+1)

integer, intent(in) :: nvar
integer, intent(in) :: nLocalOld

elements this rank owned before the epoch

real(kind=prec), intent(in) :: uLocal(1:Np,1:Np,1:Np,1:nLocalOld,1:nvar)
integer, intent(in) :: winFirst(1:decomp%nRanks)
integer, intent(in) :: winLast(1:decomp%nRanks)
integer, intent(in) :: wFirst

this rank's window, normalized (wFirst > wLast if empty)

integer, intent(in) :: wLast
real(kind=prec), intent(inout) :: uWin(1:Np,1:Np,1:Np,wFirst:wLast,1:nvar)

intent(inout), not out: the receives write it through MPI rather than through the dummy, and an intent(out) dummy would license a compiler to treat it as undefined on entry.

integer(kind=int64), intent(inout) :: nBytesRecv
integer(kind=int64), intent(inout) :: nBytesSent
integer(kind=int64), intent(inout) :: nElemRemote

Calls

proc~~exchangeoldwindow~2~~CallsGraph proc~exchangeoldwindow~2 ExchangeOldWindow proc~exchangeoldwindowflat ExchangeOldWindowFlat proc~exchangeoldwindow~2->proc~exchangeoldwindowflat proc~postoldwindowexchange PostOldWindowExchange proc~exchangeoldwindowflat->proc~postoldwindowexchange proc~ownedrun OwnedRun proc~exchangeoldwindowflat->proc~ownedrun proc~finisholdwindowexchange FinishOldWindowExchange proc~exchangeoldwindowflat->proc~finisholdwindowexchange proc~postoldwindowexchange->proc~ownedrun mpi_isend mpi_isend proc~postoldwindowexchange->mpi_isend mpi_irecv mpi_irecv proc~postoldwindowexchange->mpi_irecv mpi_waitall mpi_waitall proc~finisholdwindowexchange->mpi_waitall

Contents

Source Code


Source Code

  subroutine ExchangeOldWindow(decomp,Np,nvar,nLocalOld,uLocal,winFirst,winLast, &
                               wFirst,wLast,uWin,nBytesRecv,nBytesSent,nElemRemote)
    !! Migrate the pre-regrid solution into this rank's old-element window with point-to-point
    !! messages: the Stage-5 v2 replacement for allgathering the whole old field onto every rank.
    !!
    !! The schedule itself lives in SELF_SolutionMigration, on flat buffers, so that one copy of it
    !! serves both dimensions and every backend - in particular so that the GPU backend can run
    !! the same message set with the window in device memory. This wrapper is the 3-D host form:
    !! it exists because the plan-window tests and the portable path read and write the solution as
    !! a rank-5 array, and passing the whole array to the flat routine's assumed-size dummy is
    !! sequence association over storage that is contiguous in every case SELF constructs.
    !!
    !! Runs once per adapting epoch, between time steps - never inside the time-stepping loop.
    implicit none
    type(DomainDecomposition),intent(in) :: decomp
    integer,intent(in) :: Np !! nodes per direction (N+1)
    integer,intent(in) :: nvar
    integer,intent(in) :: nLocalOld !! elements this rank owned before the epoch
    real(prec),intent(in) :: uLocal(1:Np,1:Np,1:Np,1:nLocalOld,1:nvar)
    integer,intent(in) :: winFirst(1:decomp%nRanks)
    integer,intent(in) :: winLast(1:decomp%nRanks)
    integer,intent(in) :: wFirst !! this rank's window, normalized (wFirst > wLast if empty)
    integer,intent(in) :: wLast
    real(prec),intent(inout) :: uWin(1:Np,1:Np,1:Np,wFirst:wLast,1:nvar)
    !! intent(inout), not out: the receives write it through MPI rather than through the dummy,
    !! and an intent(out) dummy would license a compiler to treat it as undefined on entry.
    integer(int64),intent(inout) :: nBytesRecv
    integer(int64),intent(inout) :: nBytesSent
    integer(int64),intent(inout) :: nElemRemote

    ! Whole arrays, not first elements: either may be zero-sized (an empty window on a rank that
    ! owns no new elements; no old elements on a rank the previous partition left empty), and a
    ! zero-sized whole array is a legal actual argument for an assumed-size dummy where the
    ! element reference uWin(1,1,1,wFirst,1) would be out of bounds.
    call ExchangeOldWindowFlat(decomp,Np*Np*Np,nvar,nLocalOld,uLocal,winFirst,winLast, &
                               wFirst,wLast,uWin,nBytesRecv,nBytesSent,nElemRemote)

  endsubroutine ExchangeOldWindow