SELF_SolutionMigration Module

  1. SELF_SolutionMigration.f90
  2. SELF_SolutionMigration

Point-to-point migration of the pre-regrid solution into a rank's old-element WINDOW: the memory-space-agnostic core of AMR Stage-5 v2, shared by every backend and both dimensions.

An adapting epoch repartitions the mesh, so the rank that will own a new element and the rank that owned its old source need not be the same. Both partitions are contiguous ranges of the same space-filling-curve leaf order and the transfer plan is rank-replicated, so each rank can compute the contiguous window of old elements its own new range references (PlanWindows, which needs the plan and therefore stays with the AMR controllers) and then receive exactly that window - rather than allgathering the whole old field, which is what v1 did.

Two properties make the schedule cheap, and both are worth understanding before editing this module:

  1. Routing needs NO communication. Send and receive runs are the same intersection (OwnedRun) evaluated from the same replicated offsetElem and window tables on both ends of every pair, so the two schedules match by construction: no handshake, no count exchange, no collective. Reaching for MPI_Alltoallv here would pay for a negotiation that replication has already answered.

  2. NOTHING needs packing. A run of elements at a fixed variable is contiguous in both the source field and the window, so each message reads and writes the real storage directly. That costs one message per (peer,variable) instead of one per peer, and it removes a pack buffer, an unpack loop and two whole-field copies.

Everything here works on FLAT buffers, indexed as (perElem, nElem, nvar) in Fortran order with perElem = (N+1)2 in 2-D and (N+1)3 in 3-D. That is what lets one copy of the schedule serve both dimensions and both backends: the GPU backend hands these routines Fortran descriptors over DEVICE allocations and MPI receives straight into device memory (the idiom of SideExchangeStart in SELF_MappedScalar_3D.f90; GPU-aware MPI is already an unconditional requirement of any multi-rank GPU run in SELF), while the portable backend hands them host allocatables.

The buffer arguments are assumed-size, so a caller with a rank-4 or rank-5 window passes the whole array and relies on sequence association. Every such actual argument in SELF is contiguous (explicit-shape dummies, allocatables, or pointer,contiguous), so no copy-in temporary can appear - which matters because MPI is handed the addresses.

Runs once per adapting epoch, between time steps - never inside the time-stepping loop.


Uses

  • module~~self_solutionmigration~~UsesGraph module~self_solutionmigration SELF_SolutionMigration mpi mpi module~self_solutionmigration->mpi iso_fortran_env iso_fortran_env module~self_solutionmigration->iso_fortran_env module~self_domaindecomposition~2 SELF_DomainDecomposition module~self_solutionmigration->module~self_domaindecomposition~2 module~self_constants SELF_Constants module~self_solutionmigration->module~self_constants module~self_domaindecomposition_t SELF_DomainDecomposition_t module~self_domaindecomposition~2->module~self_domaindecomposition_t module~self_constants->iso_fortran_env iso_c_binding iso_c_binding module~self_constants->iso_c_binding module~self_domaindecomposition_t->mpi module~self_domaindecomposition_t->module~self_constants module~self_domaindecomposition_t->iso_c_binding module~self_lagrange SELF_Lagrange module~self_domaindecomposition_t->module~self_lagrange module~self_supportroutines SELF_SupportRoutines module~self_domaindecomposition_t->module~self_supportroutines module~self_lagrange->iso_fortran_env module~self_lagrange->module~self_constants module~self_lagrange->iso_c_binding module~self_lagrange_t SELF_Lagrange_t module~self_lagrange->module~self_lagrange_t module~self_supportroutines->iso_fortran_env module~self_supportroutines->module~self_constants module~self_lagrange_t->iso_fortran_env module~self_lagrange_t->module~self_constants module~self_lagrange_t->iso_c_binding module~self_lagrange_t->module~self_supportroutines module~self_quadrature SELF_Quadrature module~self_lagrange_t->module~self_quadrature HDF5 HDF5 module~self_lagrange_t->HDF5 module~self_hdf5 SELF_HDF5 module~self_lagrange_t->module~self_hdf5 module~self_quadrature->iso_fortran_env module~self_quadrature->module~self_constants module~self_hdf5->mpi module~self_hdf5->iso_fortran_env module~self_hdf5->module~self_constants module~self_hdf5->HDF5

Used by

  • module~~self_solutionmigration~~UsedByGraph module~self_solutionmigration SELF_SolutionMigration module~self_dgmodel2d_t SELF_DGModel2D_t module~self_dgmodel2d_t->module~self_solutionmigration module~self_dgmodel3d_t SELF_DGModel3D_t module~self_dgmodel3d_t->module~self_solutionmigration module~self_amrcontroller_3d SELF_AMRController_3D module~self_amrcontroller_3d->module~self_solutionmigration module~self_amrcontroller_3d->module~self_dgmodel3d_t module~self_amrcontroller_2d SELF_AMRController_2D module~self_amrcontroller_2d->module~self_solutionmigration module~self_amrcontroller_2d->module~self_dgmodel2d_t module~self_dgmodel2d SELF_DGModel2D module~self_dgmodel2d->module~self_dgmodel2d_t module~self_dgmodel3d SELF_DGModel3D module~self_dgmodel3d->module~self_dgmodel3d_t module~self_dgmodel2d~2 SELF_DGModel2D module~self_dgmodel2d~2->module~self_dgmodel2d_t module~self_dgmodel3d~2 SELF_DGModel3D module~self_dgmodel3d~2->module~self_dgmodel3d_t module~self_ecdgmodel2d_t SELF_ECDGModel2D_t module~self_ecdgmodel2d_t->module~self_dgmodel2d module~self_nulldgmodel2d_t self_NullDGModel2D_t module~self_nulldgmodel2d_t->module~self_dgmodel2d module~self_linearshallowwater2d_t self_LinearShallowWater2D_t module~self_linearshallowwater2d_t->module~self_dgmodel2d module~self_advection_diffusion_2d_t self_advection_diffusion_2d_t module~self_advection_diffusion_2d_t->module~self_dgmodel2d module~self_lineareuler2d_t self_LinearEuler2D_t module~self_lineareuler2d_t->module~self_dgmodel2d module~self_lineareuler2d_pml_t self_LinearEuler2D_PML_t module~self_lineareuler2d_pml_t->module~self_dgmodel2d module~self_lineareuler2d_pml_t->module~self_lineareuler2d_t module~self_ecdgmodel3d_t SELF_ECDGModel3D_t module~self_ecdgmodel3d_t->module~self_dgmodel3d module~self_advection_diffusion_3d_t self_advection_diffusion_3d_t module~self_advection_diffusion_3d_t->module~self_dgmodel3d module~self_lineareuler3d_t self_LinearEuler3D_t module~self_lineareuler3d_t->module~self_dgmodel3d module~self_nulldgmodel3d_t self_NullDGModel3D_t module~self_nulldgmodel3d_t->module~self_dgmodel3d module~self_lineareuler2d~2 self_LinearEuler2D module~self_lineareuler2d~2->module~self_lineareuler2d_t module~self_esatmo3d SELF_ESAtmo3D module~self_esatmo3d->module~self_ecdgmodel3d_t module~self_esatmo3d_t SELF_ESAtmo3D_t module~self_esatmo3d->module~self_esatmo3d_t module~self_ecdgmodel2d~2 SELF_ECDGModel2D module~self_ecdgmodel2d~2->module~self_ecdgmodel2d_t module~self_ecdgmodel2d SELF_ECDGModel2D module~self_ecdgmodel2d->module~self_ecdgmodel2d_t module~self_nulldgmodel2d self_NullDGModel2D module~self_nulldgmodel2d->module~self_nulldgmodel2d_t module~self_esatmo2d SELF_ESAtmo2D module~self_esatmo2d->module~self_ecdgmodel2d_t module~self_esatmo2d_t SELF_ESAtmo2D_t module~self_esatmo2d->module~self_esatmo2d_t module~self_ecadvection2d~2 SELF_ECAdvection2D module~self_ecadvection2d~2->module~self_ecdgmodel2d_t module~self_ecadvection2d_t SELF_ECAdvection2D_t module~self_ecadvection2d~2->module~self_ecadvection2d_t module~self_nulldgmodel2d~2 self_NullDGModel2D module~self_nulldgmodel2d~2->module~self_nulldgmodel2d_t module~self_linearshallowwater2d self_LinearShallowWater2D module~self_linearshallowwater2d->module~self_linearshallowwater2d_t module~self_linearshallowwater2d~2 self_LinearShallowWater2D module~self_linearshallowwater2d~2->module~self_linearshallowwater2d_t module~self_advection_diffusion_2d self_advection_diffusion_2d module~self_advection_diffusion_2d->module~self_advection_diffusion_2d_t module~self_advection_diffusion_2d~2 self_advection_diffusion_2d module~self_advection_diffusion_2d~2->module~self_advection_diffusion_2d_t module~self_lineareuler3d~2 self_LinearEuler3D module~self_lineareuler3d~2->module~self_lineareuler3d_t module~self_ecadvection3d SELF_ECAdvection3D module~self_ecadvection3d->module~self_ecdgmodel3d_t module~self_ecadvection3d_t SELF_ECAdvection3D_t module~self_ecadvection3d->module~self_ecadvection3d_t module~self_lineareuler2d self_LinearEuler2D module~self_lineareuler2d->module~self_lineareuler2d_t module~self_lineareuler2d_pml~2 self_LinearEuler2D_PML module~self_lineareuler2d_pml~2->module~self_lineareuler2d_pml_t module~self_advection_diffusion_3d self_advection_diffusion_3d module~self_advection_diffusion_3d->module~self_advection_diffusion_3d_t module~self_ecdgmodel3d~2 SELF_ECDGModel3D module~self_ecdgmodel3d~2->module~self_ecdgmodel3d_t module~self_ecdgmodel3d SELF_ECDGModel3D module~self_ecdgmodel3d->module~self_ecdgmodel3d_t module~self_lineareuler2d_pml self_LinearEuler2D_PML module~self_lineareuler2d_pml->module~self_lineareuler2d_pml_t module~self_advection_diffusion_3d~2 self_advection_diffusion_3d module~self_advection_diffusion_3d~2->module~self_advection_diffusion_3d_t module~self_lineareuler3d self_LinearEuler3D module~self_lineareuler3d->module~self_lineareuler3d_t module~self_nulldgmodel3d~2 self_NullDGModel3D module~self_nulldgmodel3d~2->module~self_nulldgmodel3d_t module~self_nulldgmodel3d self_NullDGModel3D module~self_nulldgmodel3d->module~self_nulldgmodel3d_t module~self_ecadvection2d_t->module~self_ecdgmodel2d~2 module~self_ecadvection3d_t->module~self_ecdgmodel3d~2 module~self_esatmo3d_t->module~self_ecdgmodel3d~2 module~self_esatmo2d_t->module~self_ecdgmodel2d~2 module~self_ecadvection2d SELF_ECAdvection2D module~self_ecadvection2d->module~self_ecadvection2d_t module~self_esatmo3d~2 SELF_ESAtmo3D module~self_esatmo3d~2->module~self_esatmo3d_t module~self_esatmo2d~2 SELF_ESAtmo2D module~self_esatmo2d~2->module~self_esatmo2d_t module~self_ecadvection3d~2 SELF_ECAdvection3D module~self_ecadvection3d~2->module~self_ecadvection3d_t

Contents

Subroutines


Subroutines

public subroutine ExchangeOldWindowFlat(decomp, perElem, nvar, nLocalOld, uLocal, winFirst, winLast, wFirst, wLast, uWin, nBytesRecv, nBytesSent, nElemRemote)

The complete host-memory migration: post the messages, copy the part of the window this rank already owns while they are in flight, then wait. This is the portable backend's whole implementation; the GPU backend calls PostOldWindowExchange / FinishOldWindowExchange itself so that it can serve the local part with a device-to-device copy instead.

Arguments

TypeIntentOptionalAttributesName
type(DomainDecomposition), intent(in) :: decomp
integer, intent(in) :: perElem
integer, intent(in) :: nvar
integer, intent(in) :: nLocalOld
real(kind=prec), intent(in) :: uLocal(*)
integer, intent(in) :: winFirst(1:decomp%nRanks)
integer, intent(in) :: winLast(1:decomp%nRanks)
integer, intent(in) :: wFirst
integer, intent(in) :: wLast
real(kind=prec), intent(inout) :: uWin(*)
integer(kind=int64), intent(inout) :: nBytesRecv
integer(kind=int64), intent(inout) :: nBytesSent
integer(kind=int64), intent(inout) :: nElemRemote

public subroutine FinishOldWindowExchange(requests, msgCount)

Wait on the messages PostOldWindowExchange posted. A no-op when nothing was posted, which is the common case for a symmetric refinement: under a contiguous space-filling-curve repartition each rank's new range is sourced from its own old range, so the window is entirely local and not a single byte moves.

Arguments

TypeIntentOptionalAttributesName
integer, intent(inout) :: requests(:)
integer, intent(in) :: msgCount

public subroutine OwnedRun(offsetElem, r, first, last, a, b)

The run of elements that rank r-1 owns and that also lies in [first,last]: a..b, empty when b < a. offsetElem is a decomposition's contiguous ownership table, so rank r-1 owns offsetElem(r)+1 .. offsetElem(r+1).

Read more…

Arguments

TypeIntentOptionalAttributesName
integer, intent(in) :: offsetElem(:)
integer, intent(in) :: r

1-based table index; the rank is r-1
integer, intent(in) :: first
integer, intent(in) :: last
integer, intent(out) :: a
integer, intent(out) :: b

public subroutine PostOldWindowExchange(decomp, perElem, nvar, nLocalOld, uLocal, winFirst, winLast, wFirst, wLast, uWin, requests, msgCount, nBytesRecv, nBytesSent, nElemRemote)

Post every message of this rank's migration schedule and return without waiting: the receives that fill the remote runs of its window, then the sends that serve the runs of its old range that other ranks' windows need. Receives are posted before sends, as everywhere else in SELF.

Read more…

Arguments

TypeIntentOptionalAttributesName
type(DomainDecomposition), intent(in) :: decomp
integer, intent(in) :: perElem

reals per element per variable: (N+1)2 in 2-D, (N+1)3 in 3-D
integer, intent(in) :: nvar
integer, intent(in) :: nLocalOld

elements this rank owned before the epoch (uLocal's stride)
real(kind=prec), intent(in) :: uLocal(*)

(perElem,nLocalOld,nvar), first element = this rank's first
integer, intent(in) :: winFirst(1:decomp%nRanks)
integer, intent(in) :: winLast(1:decomp%nRanks)
integer, intent(in) :: wFirst

this rank's window, normalized (wFirst > wLast if empty)
integer, intent(in) :: wLast
real(kind=prec), intent(inout) :: uWin(*)

(perElem,nWinElem,nvar), first element = global old wFirst intent(inout), not out: the receives write it through MPI rather than through the dummy, and an intent(out) dummy would license a compiler to treat it as undefined on entry.
integer, intent(inout) :: requests(:)

at least 2nvardecomp%nRanks entries
integer, intent(inout) :: msgCount

0 on entry; the number of posted messages on exit
integer(kind=int64), intent(inout) :: nBytesRecv
integer(kind=int64), intent(inout) :: nBytesSent
integer(kind=int64), intent(inout) :: nElemRemote