Use a map-reduce style pattern for your work allocation and you lose some small amount of efficiency, but the design becomes much much easier. You can even mix and match different types of work in your reduce stage to keep the different types of hardware busy.