It does fundamentally divide humanity into those that can imagine, and those that can't. Its not unreasonable to guess others experience the world similarly to oneself; not at all interesting that folks would be surprised by that.
It's perhaps an exaggeration to describe them as 'those that can't imagine': Galton noted that inability to visualise was much more prevalent among his colleagues, the scientific men of the day, than among the general public.
