通过MPI_Get和MPI_Type_vector,可以使用跨式远程内存访问。然而,我只看到过vector类型用于访问源内存的例子,而不是目标内存:
MPI_Datatype type_origin;
MPI_Type_vector(..., &type_origin);
MPI_Get(myBuf, size, MPI_DOUBLE, rank_origin, disp_origin, 1, type_origin, win);
我想知道使用vector类型(=strided)访问源内存和目标内存是否符合规范:
MPI_Datatype type_origin, type_target;
MPI_Type_vector(..., &type_origin);
MPI_Type_vector(..., &type_target);
MPI_Get(myBuf + disp_target, 1, type_target, rank_origin, disp_origin, 1, type_origin, win);
我试用了它,在我的机器上运行良好,使用Fedora 36、g++ v.12.2.1、OpenMPI v.3.1和Eigen v.3.4
但我不知道这是否符合规范和可移植性。请指示。
以下是我的完整示例:
#define OMPI_SKIP_MPICXX 1
#include <mpi.h>
#include <Eigen/Dense>
#include <iostream>
using namespace Eigen;
MPI_Datatype getVectorType(
const Ref<const Array2i>& totalBlockSize,
const Ref<const Array2i>& subBlockSize,
Index nComponents
){
MPI_Datatype vec;
MPI_Type_vector(
subBlockSize.y(),
subBlockSize.x() * nComponents,
totalBlockSize.x() * nComponents,
MPI_DOUBLE,
&vec
);
return vec;
}
int getDisp(
const Ref<const Array2i>& start,
const Ref<const Array2i>& size,
Index nComponents
){
return ( start.y() * size.x() + start.x() ) * nComponents;
}
int main(int argc, char* argv[]){
MPI_Init(&argc,&argv);
MPI_Comm comm {MPI_COMM_WORLD};
int nRanks, rank;
MPI_Comm_size(comm, &nRanks);
MPI_Comm_rank(comm, &rank);
/* let's just say it's ranks 1 and 2 that have to communicate */
int
rank_origin {1},
rank_target {2};
/* and what they have to communicate is a block of data,
* which is not contiguous on either rank */
Array2i
size_origin { 8,12},
size_target { size_origin + 1 },
start_block { 3, 4},
size_block { 4, 6};
ArrayXXd arr_origin, arr_target;
/* number of components per cell, equals number of rows in arrays */
/* to make it simple, it's set to 1 here, so it can be ignored below */
Index nComp {1};
auto reshaped = [&](ArrayXXd& arr, const Array2i& size){
return arr.reshaped( nComp * size.x(), size.y() );
};
auto reshapedBlock = [&](auto& resh,
const Array2i& start_block,
const Array2i& size_block
){
return resh.block(
nComp * start_block.x(), start_block.y(),
nComp * size_block .x(), size_block .y()
);
};
auto print = [&](const auto& resh){
std::cout
<< "On rank " << rank
<< ", array content (reshaped):\n" << resh
<< "\n";
};
if ( rank == rank_origin ){
arr_origin.resize( nComp, size_origin.prod() );
/* set here as a default value so that we know where it's from */
arr_origin = -rank_origin;
auto resh { reshaped(arr_origin, size_origin) };
auto reshBlock { reshapedBlock(resh, start_block, size_block) };
reshBlock = rank_origin;
print(resh);
}
MPI_Datatype type_origin, type_target;
if ( rank == rank_target ){
arr_target.resize( nComp, size_target.prod() );
arr_target= -rank_target;
type_origin = getVectorType(size_origin, size_block, nComp);
type_target = getVectorType(size_target, size_block, nComp);
MPI_Type_commit(&type_origin);
MPI_Type_commit(&type_target);
}
MPI_Win win;
constexpr int disp { sizeof(double) };
MPI_Win_create(
arr_origin.data(), arr_origin.size() * disp, disp,
MPI_INFO_NULL, comm, &win
);
MPI_Win_fence(0, win);
if ( rank == rank_target ){
int
disp_origin { getDisp(start_block, size_origin, nComp) },
disp_target { getDisp(start_block, size_target, nComp) };
MPI_Get(
arr_target.data() +
disp_target, 1, type_target,
rank_origin,
disp_origin, 1, type_origin,
win
);
MPI_Type_free(&type_origin);
MPI_Type_free(&type_target);
}
MPI_Win_fence(0, win);
if ( rank == rank_target ){
print( reshaped(arr_target, size_target) );
}
MPI_Win_free(&win);
MPI_Finalize();
return 0;
}
1条答案
按热度按时间ttvkxqim1#
是的,这是法律的的。对于MPI_Get可以使用的数据类型没有限制。尽管非连续的源缓冲区比远程缓冲区更容易实现,但这可能会影响性能。