mmap a file into memory. As you touch a page the CPU generates a page fault and invokes the kernel. The kernel merges the partially written page with the underlying data. As there is no way to avoid the merge you end up doing write() instead.

The page fault is a synchronous read. Horrible for performance.

What do you mean by "The kernel merges the partially written page". A pagefault traps the execution just before any CPU write succeeds, then the kernel optionally loads the page, maps it into process' space, flushes TLBs and resumes process execution. mmap() style disk writes happen on page-level granularity, usually not before the kernel runs out of free memory frames.

I agree on the fact that using mmap() reads vs read() reads leaves the kernel with no possibility to reorder requests, as any thread trapped in a pagefault is obviously unable to generate more I/O requests. Reordering can lead to better performance. This is not the case with asynchronous read().

I might have expressed myself somewhat unclear. Let me try again. varnishd tries to write to a page not in memory and gets a page fault while doing so. As the kernel has no idea of your intentions to overwrite the whole page it will read the page into memory and then return control to varnish who completes the memory write, merging the write request just made onto the preexisting page. varnish does not utilize any asynchronous reads. All IO is blocking.

I understand now. Not enough that the kernel unnecessarily loads a page that is about to be fully rewritten, it also blocks the thread that is trapped in a pagefault, while it could potentially go and serve another connection. I can see how write() can be faster than mmap(), thanks.

Which is exactly the thinking as to why squid cache used write()...