List of all members
walberla::gather::MPIGatherScheme Class Reference

Detailed Description

Collects / Gathers data from multiple blocks to a single process.

Usage / Restrictions:

  • each block sends a fixed amount of data to a collection process
  • this amount has to be the same for all timesteps (i.e. calls of communicate() )
  • to use the Scheme implement the GatherPackInfo interface or use one of the existing implementations
  • The collect operation is done every time the communicate() method is called. If the result is not needed immediately, but only at the end of the simulation consider using the FileCollectorScheme.

Implementation:

  • when communicate() is called first, the amount of data that each process sends is determined
  • then an MPI_Allgather operation over all processes has to be done, to determine which processes participate this potentially very expensive operation is done only once in the setup phases
  • a MPI communicator is created for all processes participating in the gather operation (i.e. that packed something ), and the amount of data that each process sends is sent to the gathering process
  • subsequent calls of communicate() use that communicator for an MPI_Gatherv operation

#include <MPIGatherScheme.h>

Private Attributes

Communication parameters initialized by runSetupPhase
bool setupPhaseDone_
 true if runSetupPhase() was called More...
 
MPI_Comm gatherCommunicator_
 communicator containing only participating processes More...
 
int gatherRank_
 rank in gatherCommunicator_ that gathers the data More...
 
std::vector< int > displacementVector_
 encodes how much each participating process sends (see MPI_Gatherv) ( only on gather process ) More...
 
std::vector< int > sendBytesPerProcess_
 For each process in gatherCommunicator_ the number of bytes to send ( only on gather process ) More...
 
int gatherMsgSize_
 total size of gather message ( only on gather process ) More...
 
int bytesToSend_
 number of bytes sent by this process ( on all processes ) More...
 
uint_t everyNTimestep_
 

Construction & Destruction


using PackInfoVector = std::vector< shared_ptr< GatherPackInfo > >
 
domain_decomposition::BlockStorage & blocks_
 
PackInfoVector packInfos_
 all registered PackInfos More...
 
int gatherRankInGlobalComm_
 gather processes rank in mpiManager->comm() More...
 
 MPIGatherScheme (domain_decomposition::BlockStorage &blockStorage, int gatherRank=0, uint_t everyNTimestep=1)
 
 ~MPIGatherScheme ()
 
void addPackInfo (const shared_ptr< GatherPackInfo > &pi)
 Registering PackInfo's. More...
 
void communicate ()
 Performs the gather operation. More...
 
void operator() ()
 Similar to communicate but only executes everyNTimestep ( see constructor ) More...
 
void runSetupPhase ()
 
void setupGatherCommunicator (bool thisProcessParticipates, MPI_Comm &commOut, int &newRank)
 

Member Typedef Documentation

◆ PackInfoVector

using walberla::gather::MPIGatherScheme::PackInfoVector = std::vector<shared_ptr<GatherPackInfo> >
private

Constructor & Destructor Documentation

◆ MPIGatherScheme()

walberla::gather::MPIGatherScheme::MPIGatherScheme ( domain_decomposition::BlockStorage &  blockStorage,
int  gatherRank = 0,
uint_t  everyNTimestep = 1 
)

◆ ~MPIGatherScheme()

walberla::gather::MPIGatherScheme::~MPIGatherScheme ( )

Member Function Documentation

◆ addPackInfo()

void walberla::gather::MPIGatherScheme::addPackInfo ( const shared_ptr< GatherPackInfo > &  pi)

Registering PackInfo's.

  • The ownership of the passed pack info is transferred to the MPICollectorScheme, i.e. the pack info is deleted by the scheme
  • when calling addPackInfo after communicate(), the expensive setupPhase has to be done again -> better first call all addPackInfo() then start calling communicate()

◆ communicate()

void walberla::gather::MPIGatherScheme::communicate ( )

Performs the gather operation.

Collects all data on sending processes according to information given in the pack infos and unpacks them on the process that holds the root block (1,1,1)

◆ operator()()

void walberla::gather::MPIGatherScheme::operator() ( )
inline

Similar to communicate but only executes everyNTimestep ( see constructor )

◆ runSetupPhase()

void walberla::gather::MPIGatherScheme::runSetupPhase ( )
private

◆ setupGatherCommunicator()

void walberla::gather::MPIGatherScheme::setupGatherCommunicator ( bool  thisProcessParticipates,
MPI_Comm &  commOut,
int &  newRank 
)
private

Member Data Documentation

◆ blocks_

domain_decomposition::BlockStorage& walberla::gather::MPIGatherScheme::blocks_
private

◆ bytesToSend_

int walberla::gather::MPIGatherScheme::bytesToSend_
private

number of bytes sent by this process ( on all processes )

◆ displacementVector_

std::vector<int> walberla::gather::MPIGatherScheme::displacementVector_
private

encodes how much each participating process sends (see MPI_Gatherv) ( only on gather process )

◆ everyNTimestep_

uint_t walberla::gather::MPIGatherScheme::everyNTimestep_
private

◆ gatherCommunicator_

MPI_Comm walberla::gather::MPIGatherScheme::gatherCommunicator_
private

communicator containing only participating processes

◆ gatherMsgSize_

int walberla::gather::MPIGatherScheme::gatherMsgSize_
private

total size of gather message ( only on gather process )

◆ gatherRank_

int walberla::gather::MPIGatherScheme::gatherRank_
private

rank in gatherCommunicator_ that gathers the data

◆ gatherRankInGlobalComm_

int walberla::gather::MPIGatherScheme::gatherRankInGlobalComm_
private

gather processes rank in mpiManager->comm()

◆ packInfos_

PackInfoVector walberla::gather::MPIGatherScheme::packInfos_
private

all registered PackInfos

◆ sendBytesPerProcess_

std::vector<int> walberla::gather::MPIGatherScheme::sendBytesPerProcess_
private

For each process in gatherCommunicator_ the number of bytes to send ( only on gather process )

◆ setupPhaseDone_

bool walberla::gather::MPIGatherScheme::setupPhaseDone_
private

true if runSetupPhase() was called

The documentation for this class was generated from the following files: