issue with MPI communication with two MIC cards and xeon processor

Hello,

I am running a MPI application (involving 5 ranks) which runs smoothly when all ranks are on Xeon processor but when i put two ranks on MIC0 and MIC1 there is following issue and the program just hangs and gives me segmentation fault.

setup:-

using (blocking MPI send and non blocking MPI recv)

rank0, rank1 on MIC0,MIC1

rank2,rank3,rank4 on xeon

issue:-

rank1-->sends 100 packets and reaches finalize() 

rank2-->only receives 60 packets and then hangs

some things i tried:-

I added a sleep(1) before rank1 sends packets and this solved the issue as rank2 could get all the packets

but for large number of packets (>100) adding sleep doesnt solve the issue and the system hangs

any suggestions

thank you