Was the on-device local LLM stack that you tried llama.cpp or something like MLC? I've seen better performance with MLC than llama.cpp in the past - but it has been probably at a least a year since I tested iphones and androids for local inference
I looked into using https://github.com/mybigday/llama.rn. Ultimately, it was too slow to be conversational. The demands of the rendering the WebGL would likely not help.
