You're right. In fact, I have my vimperator setup so that it automatically does this for Google Reader and GMail, and only for the relevant tab (i.e. if I hit "v" to view a Reader article, the pass-through turns off on the new tab). I did it with this statement in my .vimperatorrc (Linux):

autocmd LocationChange .* js modes.passAllKeys = /(google\.com\/reader|mail\.google\.com)/.test(buffer.URL)