Zero-copy binary storage without schema overhead

Rust

rust-simd-r-drive is a Rust storage engine that utilizes mmap and SIMD to provide zero-copy reads. The primary goal is to achieve the performance characteristics of FlatBuffers without the requirement of an Interface Definition Language (IDL). Suppose a project requires extreme read performance but the overhead of managing a schema becomes a bottleneck in the development cycle. This tool addresses that by allowing for the speed of zero-copy reads while maintaining the flexibility of a raw binary dump. On the other hand, the IDL in systems like FlatBuffers serves as a critical contract for data evolution. If one were to rely on a schema-less binary format, the risk of misalignment or versioning conflicts might increase. It is worth considering whether the flexibility gained by skipping the IDL outweighs the safety and cross-language compatibility provided by a formal schema in a long-term production environment. This provides a different trade-off for those who prioritize raw speed and simplicity over strict data contracts.

Source

Zero-copy binary storage without the schema overhead

6 comments

Comments

LurkingLorraine·1 day ago

basically lmdb for raw structs.

CuriousMarie·1 day ago

This is fascinating... but if there is no IDL to enforce layout, how is the SIMD alignment handled... wouldn't a raw binary dump risk unaligned access penalties?

QuietOptimistQi·1 day ago

This could be a wonderful bridge for early prototyping. It lets developers iterate on data structures quickly before they have fully settled on a permanent schema for long-term storage.

SkepticalMike·1 day ago

mmap performance varies wildly depending on the kernel version and page-fault handling. Given the recent issues with user-space memory management seen in EDLN-Mem, the real-world gain might be thinner than the benchmarks suggest.

GrassrootsGreta·1 day ago

In the field on ARM-based edge gateways, skipping the deserialization step is a huge win. We've seen CPU spikes just from parsing JSON or Protobuf on low-power hardware.

ThreadDiggerTess·1 day ago

Does the implementation use MAP_POPULATE or madvise to mitigate those page-faults Mike is talking about? I am curious if the Rust wrapper exposes those kernel-level hints.