> Yes, the delay seems to be coming here: > > err = hpmp_rdmacm->rdma_connect(id, NULL); > if (err) { > hpmp_printf("rdma_connect() failed"); > return NULL; > } > t1 = MPI_Wtime(); > > retry3: > err = > hpmp_rdmacm->rdma_get_cm_event(hpmp_rdmacm->connect_cm_channel, &event); > if (err) { > if (errno == EINTR) goto retry3; > hpmp_printf("rdma_get_cm_event() failed"); > return NULL; > } > > if (event->event != RDMA_CM_EVENT_ESTABLISHED) { > hpmp_printf("rdma_get_cm_event() unexpected event (%d vs > %d)" > "while connecting to %d\n", > event->event, RDMA_CM_EVENT_ESTABLISHED, > port); > return NULL; > } > > t2 = MPI_Wtime(); > fprintf(stderr, "CONNECTION ESTABLISHED ON CONNECT %lf\n", t2-t1); > hpmp_rdmacm->rdma_ack_cm_event(event); > > > > I get output such as: > > [ 1] CONNECTION ESTABLISHED ON CONNECT 0.001447 > [ 9] CONNECTION ESTABLISHED ON CONNECT 6.145778 > [ 6] CONNECTION ESTABLISHED ON CONNECT 5.233660 > [ 0] CONNECTION ESTABLISHED ON CONNECT 0.001343 > [ 6] CONNECTION ESTABLISHED ON CONNECT 0.001155 > [ 7] CONNECTION ESTABLISHED ON CONNECT 4.517944 > [11] CONNECTION ESTABLISHED ON CONNECT 0.001445 > [ 3] CONNECTION ESTABLISHED ON CONNECT 0.001558 > [ 7] CONNECTION ESTABLISHED ON CONNECT 0.001627 > [ 5] CONNECTION ESTABLISHED ON CONNECT 6.145470 > [ 2] CONNECTION ESTABLISHED ON CONNECT 5.657639 > [ 9] CONNECTION ESTABLISHED ON CONNECT 0.001602 > [10] CONNECTION ESTABLISHED ON CONNECT 6.188743 > [ 1] CONNECTION ESTABLISHED ON CONNECT 0.001500 > [ 6] CONNECTION ESTABLISHED ON CONNECT 0.001061 > [ 1] CONNECTION ESTABLISHED ON CONNECT 0.001183 > [11] CONNECTION ESTABLISHED ON CONNECT 0.001213 > [ 5] CONNECTION ESTABLISHED ON CONNECT 0.210666 What version of Linux is this running? Is the remote side responding to the connection request in a timely manner? The connect call cannot complete until the remote side accepts the connection. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html