c - Can you transpose array when sending using MPI_Type_create_subarray? -
i'm trying transpose matrix using mpi in c. each process has square submatrix, , want send right process (the 'opposite' 1 on grid), transposing part of communication.
i'm using mpi_type_create_subarray has argument order, either mpi_order_c or mpi_order_fortran row-major , column-major respectively. thought if sent 1 of these, , received other, matrix transposed part of communication. however, doesn't seem happen - stays non-transposed.
the important part of code below, , whole code file available @ this gist. have ideas why isn't working? should approach doing transpose work? i'd have thought would, having read descriptions of mpi_order_c , mpi_order_fortran, maybe not.
/* ----------- transpose ----------- */ /* find opposite co-ordinates (as know it's square) */ coords2[0] = coords[1]; coords2[1] = coords[0]; /* rank process */ mpi_cart_rank(cart_comm, coords2, &rank2); /* send these new coordinates */ tag = (coords[0] + 1) * (coords[1] + 1); /* create new derived type receive */ /* mpi_type_vector(rows_in_core, cols_in_core, cols_in_core, mpi_double, &vector_type); */ sizes[0] = rows_in_core; sizes[1] = cols_in_core; subsizes[0] = rows_in_core; subsizes[1] = cols_in_core; starts[0] = 0; starts[1] = 0; mpi_type_create_subarray(2, sizes, subsizes, starts, mpi_order_fortran, mpi_double, &send_type); mpi_type_commit(&send_type); mpi_type_create_subarray(2, sizes, subsizes, starts, mpi_order_c, mpi_double, &recv_type); mpi_type_commit(&recv_type); /* we're sending in row-major form, it's rows_in_core * cols_in_core lots of mpi_double */ mpi_send(&array[0][0], 1, send_type, rank2, tag ,cart_comm); /* receive these new coordinates */ mpi_recv(&new_array[0][0], 1, recv_type, rank2, tag, cart_comm, &status);
i have thought work, too, apparently not.
if slog through relevant bit of mpi standard defines resulting typemap, reason becomes clear -- mpi_type_create_subarray maps out region subarray takes in full array, marches through memory in linear order, data layout doesn't change. in other words, when sizes equal subsizes, subarray contiguous block of memory; , subarray strictly smaller whole array, you're changing subregion being sent/received to, not data ordering. can see effect when choosing subregion:
int sizes[]={cols,rows}; int subsizes[]={2,4}; int starts[]={1,1}; mpi_type_create_subarray(2, sizes, subsizes, starts, mpi_order_fortran, mpi_int, &ftype); mpi_type_commit(&ftype); mpi_type_create_subarray(2, sizes, subsizes, starts, mpi_order_c, mpi_int, &ctype); mpi_type_commit(&ctype); mpi_isend(&(send[0][0]), 1, ctype, 0, 1, mpi_comm_world,&reqc); mpi_recv(&(recvc[0][0]), 1, ctype, 0, 1, mpi_comm_world, &statusc); mpi_isend(&(send[0][0]), 1, ctype, 0, 1, mpi_comm_world,&reqf); mpi_recv(&(recvf[0][0]), 1, ftype, 0, 1, mpi_comm_world, &statusf); /*...*/ printf("original:\n"); printarr(send,rows,cols); printf("\nreceived -- c order:\n"); printarr(recvc,rows,cols); printf("\nreceived: -- fortran order:\n"); printarr(recvf,rows,cols); gives this:
0 1 2 3 4 5 6 10 11 12 13 14 15 16 20 21 22 23 24 25 26 30 31 32 33 34 35 36 40 41 42 43 44 45 46 50 51 52 53 54 55 56 60 61 62 63 64 65 66 received -- c order: 0 0 0 0 0 0 0 0 11 12 13 14 0 0 0 21 22 23 24 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 received: -- fortran order: 0 0 0 0 0 0 0 0 11 12 0 0 0 0 0 13 14 0 0 0 0 0 21 22 0 0 0 0 0 23 24 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 so same data getting sent , received; that's happening arrays sizes, subsizes , starts being reversed.
you can transpose mpi datatypes -- standard gives couple of examples, 1 of i've transliterated c here -- have create types yourself. news it's no longer subarray stuff:
mpi_type_vector(rows, 1, cols, mpi_int, &col); mpi_type_hvector(cols, 1, sizeof(int), col, &transpose); mpi_type_commit(&transpose); mpi_isend(&(send[0][0]), rows*cols, mpi_int, 0, 1, mpi_comm_world,&req); mpi_recv(&(recv[0][0]), 1, transpose, 0, 1, mpi_comm_world, &status); mpi_type_free(&col); mpi_type_free(&transpose); printf("original:\n"); printarr(send,rows,cols); printf("received\n"); printarr(recv,rows,cols); $ mpirun -np 1 ./transpose2 original: 0 1 2 3 4 5 6 10 11 12 13 14 15 16 20 21 22 23 24 25 26 30 31 32 33 34 35 36 40 41 42 43 44 45 46 50 51 52 53 54 55 56 60 61 62 63 64 65 66 received 0 10 20 30 40 50 60 1 11 21 31 41 51 61 2 12 22 32 42 52 62 3 13 23 33 43 53 63 4 14 24 34 44 54 64 5 15 25 35 45 55 65 6 16 26 36 46 56 66
Comments
Post a Comment