I love articles like this! They took the extra step o find the reason for the behavior they saw rather than just saying "yup ...that's how it is".
I've also had multi-millisecond pauses on ext4. In my case, the buffer fills up and we wait for sync_dirty_buffers to do it's job. We tried adjusting the various settings to tune it, but at the end, it's always there. We ended up buffering our own writes in the application so we could get the behavior we wanted.
Have you considered putting the journal on a separate physical device? There have been a lot of reports through the years of some pretty substantial boosts in performance because the drives don't need to bounce back and forth between journaling and writing.
> Here it's clear that xfs gives better results, with the average write beating all ext4 modes by a few hundred microseconds.
Am I wrong that the graph above that statement does not show that? While it shows xfs having better latency than ext4, the axis is labelled as "write latency nanoseconds", and the difference between ext4 and xfs is ~400ns, or .4µs; not "a few hundred microseconds."
I saw the same thing a couple of months ago with a key/value datastore that synchronously writes each update to disk. Switching from ext4 to xfs fixed periodic 100ms+ latency spikes.
Be aware that XFS will lose more data on a kernel panic than ext4; for example, all vim buffers saved in the last few seconds before the crash will be of size zero on reboot. With ext4 and reiserfs I find that I'm more likely to have either the old or new data in the files.
I've also had multi-millisecond pauses on ext4. In my case, the buffer fills up and we wait for sync_dirty_buffers to do it's job. We tried adjusting the various settings to tune it, but at the end, it's always there. We ended up buffering our own writes in the application so we could get the behavior we wanted.