Not switching to the CPU - switching to the ANE (Neural Cores) - if you read the research papers Apple has released - the example I gave is pretty much how it's being used - small image classification models running on the ANE, alongside a graphics app that needs the GPU to be free.
Yes, hammering the GPU too hard can affect the display server, but no, switching to the CPU is not a good alternative