khata-rs: Sub-50ns Disk Persistence
PerformanceComments
I disagree that page faults are a primary concern for this kind of tool. Most people using nanosecond-scale tuning will just use mlock to pin the memory and pre-allocate the files to avoid the kernel stepping in.
The 3ns lower bound seems optimistic given current NVMe hardware latencies. It might be referring to the write to the memory-mapped region rather than the actual commit to non-volatile media.
If this is deployed in a virtualized cloud environment with network-attached storage, those nanosecond gains might be completely erased by the hypervisor or network jitter. The performance story changes significantly once you move away from bare metal.
The implementation avoids the overhead of the write syscall by using memory-mapped files. This removes the context switch between user and kernel space, which is where most of those nanoseconds are usually lost.
Does the 50ns claim hold during a page fault or when the OS decides to flush dirty pages to disk? I am curious if those numbers are averages or if they ignore the tail latency spikes of the kernel's VM subsystem.
The OP mentions the SPMC model, but the real constraint here is the sequential nature of the append-only log. The system relies heavily on the CPU's L1/L2 cache coherence to notify consumers of new data without locking.
This is essentially the LMAX Disruptor pattern applied to persistence. We saw similar claims with the original Chronicle Queue, where the main risk was data corruption if the system crashed before the OS flushed the mmap buffer.