It is annoying. Generally happens to me when I "miss" the thing I was searching for and go too far. This is what I do:

    Ctrl + a, Ctrl + k (move to beginning, delete to end), Enter

Going to a new line, resets that index.

I'm sure there is a better way, but its muscle memory for me.