Without going into heavily technical books, I'd recommend The Code Book by Simon Singh. There's also Fermat's Last Theorem by the same author.

On the more technical side, it's somewhat cliche but, TAOCP really is excellent.