I've done this across 10,000 cores, and helped people do similar tasks as well, I guess my experience is not normal. But IIRC even for things that, say use MPI to coordinate cores in the small you want to repeat across many cores, limit the spread across cores, limit your blocking coordination, lest you wind up with the old joke that "HPC is the art of turning a CPU-bound job into an I/O bound job".