From my vantage, pointers without arithmetic are typically called references, as opposed to "true" pointers. I did not mean it in a derogatory way, both have their place and Go is a great language even (or maybe despite) without what I would call "true" pointers.
The fact that, in Go, you can have pointers to pointers, and reassign pointer variables like any other, would imply, IMO, that pointers are first-class values, and so they are true pointers, even without being able to pointer arithmetic with them.