template<typename RecvBuffer_T = RecvBuffer, typename SendBuffer_T = SendBuffer>
class walberla::mpi::GenericBufferSystem< RecvBuffer_T, SendBuffer_T >
Manages MPI Communication with a set of known communication partners.
Features / Restrictions:
- usable in every case where normal MPI_Send, MPI_Recv is needed
- communication partners have to be known, message size not necessarily
- unknown message sizes possible ( -> automatic extra message to exchange sizes )
- Implemented with non-blocking MPI calls, and MPI_Waitany to process a message as soon as it was received while still waiting for other messages
Example:
bs.sendBuffer( neighborRank1 ) << 42;
bs.sendBuffer( neighborRank2 ) << 4242;
bs.setReceiverInfoFromSendBufferState( true, false );
bs.sendAll();
for ( auto i = bs.begin(); i != bs.end(); ++i ) {
cout << "Message received from " << i.rank() << endl;
int messageContent;
i.buffer() >> messageContent;
cout << "Received " << messageContent << endl;
}
Usage:
- Setup:
- define receive information using setReceiverInfo() or setReceiverInfoFromSendBufferState() With these functions one defines the processes that we receive from and optionally the size of the received messages. If the message sizes are unknown, they have to be communicated first. One also defines if the sizes stay constant or if they change in each communication step (size message is then sent before every content message)
- The receiver and send information can be changed, if no communication is currently running.
- Communication Step:
- Optionally call scheduleReceives() -> starts communication step and causes MPI_IRecv's to be called. This is also automatically called on first send operation.
- fill send buffers
- call send() for each buffer after filling the buffer, or call sendAll() after filling all buffers. The send*() functions return immediately, internally MPI_ISend's are called. The send*() functions start the communication step if it was not already started by scheduleReceives()
- Receiving: iterate over incoming messages using begin() and end(). Internally a MPI_Waitany is executed that returns as soon as a single message was received. Then this message can be processed while waiting for the other messages.
- Attention
- Even if the current process does not receive anything, call begin() and end() to finish the communication step
- When iteration has reached the end the communication step is finished.
- when communication has finished all Send- and RecvBuffers are automatically cleared
When running multiple BufferSystems concurrently different MPI tags have to be used for the systems: the tag can be passed in the constructor.
|
|
| | GenericBufferSystem (const MPI_Comm &communicator, int tag=0) |
| |
| | GenericBufferSystem (const GenericBufferSystem &other) |
| |
| GenericBufferSystem & | operator= (const GenericBufferSystem &other) |
| |
| | ~GenericBufferSystem ()=default |
| |
|
|
| template<typename RankIter > |
| void | setReceiverInfo (RankIter begin, RankIter end, bool changingSize) |
| |
| template<typename Range > |
| void | setReceiverInfo (const Range &range, bool changingSize) |
| |
| void | setReceiverInfo (const std::set< MPIRank > &ranksToRecvFrom, bool changingSize) |
| | Sets receiver information, when message sizes are unknown. More...
|
| |
| void | setReceiverInfo (const std::map< MPIRank, MPISize > &ranksToRecvFrom) |
| | Sets receiver information, when message sizes are known. More...
|
| |
| void | setReceiverInfoFromSendBufferState (bool useSizeFromSendBuffers, bool changingSize) |
| | Sets receiver information, using SendBuffers (symmetric communication) More...
|
| |
| void | sizeHasChanged (bool alwaysChangingSize=false) |
| | Notifies that GenericBufferSystem that message sizes have changed ( and optionally are changing in all following steps) More...
|
| |
|
| void | scheduleReceives () |
| |
| SendBuffer_T & | sendBuffer (MPIRank rank) |
| | Returns an existing SendBuffer, or creates a new one (only if !isCommunicationRunning() ) More...
|
| |
| SendBuffer_T & | sendBuffer (uint_t rank) |
| |
| size_t | size () const |
| |
| void | sendAll () |
| | Sends filled ( nonzero length) SendBuffers to their destination ranks. More...
|
| |
| void | send (MPIRank rank) |
| | Sends a single SendBuffer to its destination rank. More...
|
| |
| iterator | begin () |
| |
| iterator | end () |
| |
|
|
| bool | isSizeCommunicatedInNextStep () const |
| |
| bool | isCommunicationRunning () const |
| |
| bool | isReceiverInformationSet () const |
| |
| void | useIProbe (const bool use) |
| |
| bool | isIProbedUsed () const |
| |
| int64_t | getBytesSent () const |
| | Bytes sent during the current or last communication. More...
|
| |
| int64_t | getBytesReceived () const |
| | Bytes received during the current or last communication. More...
|
| |
| int64_t | getNumberOfSends () const |
| | Communication partners during current or last send operation. More...
|
| |
| int64_t | getNumberOfReceives () const |
| | Communication partners during current or last receive operation. More...
|
| |
|
|
| using | RankRange = std::set< MPIRank > |
| |
| class | GenericOpenMPBufferSystem< RecvBuffer_T, SendBuffer_T > |
| |
| internal::KnownSizeCommunication< RecvBuffer_T, SendBuffer_T > | knownSizeComm_ |
| |
| internal::UnknownSizeCommunication< RecvBuffer_T, SendBuffer_T > | unknownSizeComm_ |
| |
| internal::UnknownSizeCommunicationIProbe< RecvBuffer_T, SendBuffer_T > | unknownSizeCommIProbe_ |
| |
| internal::NoMPICommunication< RecvBuffer_T, SendBuffer_T > | noMPIComm_ |
| |
| internal::AbstractCommunication< RecvBuffer_T, SendBuffer_T > * | currentComm_ |
| |
| bool | sizeChangesEverytime_ |
| |
| bool | communicationRunning_ |
| |
| std::map< MPIRank, typename internal::AbstractCommunication< RecvBuffer_T, SendBuffer_T >::ReceiveInfo > | recvInfos_ |
| | Info about the message to be received from a certain rank: information holds the buffer and, if known, the message size. More...
|
| |
| std::map< MPIRank, SendInfo > | sendInfos_ |
| |
| bool | useIProbe_ = false |
| | switch between IProbe and two message communication for varying size communication More...
|
| |
| int64_t | bytesSent_ = 0 |
| | number of bytes sent during last communication More...
|
| |
| int64_t | bytesReceived_ = 0 |
| | number of bytes received during last communication More...
|
| |
| int64_t | numberOfSends_ = 0 |
| | number of communication partners during last send More...
|
| |
| int64_t | numberOfReceives_ = 0 |
| | number of communication partners during last receive More...
|
| |
| static std::set< int > | activeTags_ |
| |
| static RankRange | noRanks () |
| |
| static RankRange | allRanks () |
| |
| static RankRange | allRanksButRoot () |
| |
| static RankRange | onlyRoot () |
| |
| static RankRange | onlyRank (MPIRank includedRank) |
| |
| void | startCommunication () |
| | Starts communication. More...
|
| |
| void | endCommunication () |
| | Cleanup after communication. More...
|
| |
| RecvBuffer_T * | waitForNext (MPIRank &fromRank) |
| | Helper function for iterator. More...
|
| |
| void | setCommunicationType (const bool knownSize) |
| | Sets the communication type to known size, unknown size or NoMPI comm. More...
|
| |
template<typename Rb , typename Sb >
Sends filled ( nonzero length) SendBuffers to their destination ranks.
If some of the SendBuffers have been sent manually before using send(int rank) they are skipped, only the remaining buffers are sent.
If communication was not started before, it is started with this function.
template<typename Rb , typename Sb >
Sets receiver information, using SendBuffers (symmetric communication)
Gives the GenericBufferSystem the information that messages are received from the same processes that we send to (i.e. from all ranks where SendBuffers were already filled ) sendBuffer() has to be called before, and corresponding SendBuffers have to be filled.
- Parameters
-
| useSizeFromSendBuffers | If true, the sizes are expected to be known and equal to the size of the SendBuffers. SendBuffers with zero size are ignored. If false, all SendBuffers (also with zero size) are registered as ranks where messages are received from. The size is unknown, and communicated before. |
| changingSize | if true the size is communicated before every communication step. if useSizeFromSendBuffer==true and changingSize==true, the size is not communicated in the first step but in all following steps. |