See my above comment for a library that requires no geospatial database, just DEM files in gridfloat format. By recording information about each gridfloat file (bounding box, cellsize in lat/lng) you can do some simple math and only one disk operation (in the optimal case) to get an elevation value. Even with my crappy code in the library I mentioned in the other comment, the service is very fast on even spinning disks.