Error when setting "preschedule=FALSE" and using more than 2 nodes

[arodri7]

Hi Drew,
Hope you are doing well! I am using task_tools for our submission, but when I set preschedule=FALSE and used 25 nodes with more than 300K tasks to execute, I got the following error:

Error in spmd.recv.integer(x.buffer = integer(1), rank.source = anysource()) :
  MPI_ERR_TRUNCATE: message truncated
Calls: mpi_napply -> mpi_napply_nopreschedule -> spmd.recv.integer

I tried the same with 2 nodes and a list of tasks to execute of 219 and it did work.

I don't get errors when setting preschedule=TRUE. I can go ahead with that, but I'd like to set it to FALSE as the jobs execute at different time lengths and some resources would be left idle.

Thanks,
Alex

[madduri]

Hey Drew - Just wanted to ping about this issue. It looks like this is issue is lowering the overall robustness in our run. Is athere any additional information we should capture that will help?