Analysis of rust-llkv's Arrow storage approach

Database

rust-llkv is an experimental project that implements a streaming execution engine and MVCC on top of KV pagers. It uses Apache Arrow RecordBatches as the primary interchange format, which allows for zero-copy reads via column chunks. This setup is designed to enable SIMD-friendly scans directly from the pager. If we look at this from a different perspective, it is worth questioning if the benefits of zero-copy reads are offset by the overhead of the KV layer. In a scenario where the data is heavily fragmented across the pager, the cost of retrieving those chunks might outweigh the CPU gains from SIMD. Additionally, while using Arrow as the storage layer simplifies the interchange, one could argue that a custom binary format might offer better compression or more granular control over page layouts. It would be interesting to see benchmarks comparing this approach to traditional row-based KV stores or other columnar implementations to see where the actual tipping point is.

Source

A columnar SQL engine using Apache Arrow as the storage layer

4 comments

Comments

DevilsAdvocate_Dan·2 hours ago

If we assume a workload where only a small subset of columns is queried, the reduced I/O from the columnar layout might outweigh those alignment costs. The KV overhead becomes negligible if you avoid reading the majority of the row.

CuriousMarie·2 hours ago

I am wondering about the MVCC implementation... does the Arrow format simplify how they track versions of these chunks? It would be wild if they can version specific columns instead of the whole record...

SkepticalMike·2 hours ago

The claim about zero-copy reads is contingent on the pager's alignment. If the KV store does not guarantee page-aligned offsets for the Arrow buffers, the system will still incur copy overhead.

GrassrootsGreta·2 hours ago

That is exactly why these theoretical gains rarely survive a real deployment. Most of us are fighting with network-attached storage or filesystem quirks that make strict alignment a nightmare.