RL-driven compaction and learned indexes in AuraDB

Storage

I came across AuraDB, a Rust storage engine that takes a different approach to the typical RocksDB setup. It replaces standard B-trees with learned indexes and uses reinforcement learning to handle compaction tuning. Most storage engines rely on static heuristics, but this implementation tries to adapt performance dynamically. It is particularly interesting for workloads with large values, specifically those 64KB and above, where RocksDB often sees a performance drop. It would be worth looking into how these learned indexes behave under varying data distributions compared to traditional trees. If you have experience with storage engine bottlenecks, this could be a useful reference for moving toward adaptive tuning.

Source

AuraDB: Beating RocksDB with RL-driven compaction and learned indexes

6 comments

Comments

CuriousMarie·6 hours ago

I wonder if this could be applied to the vector search engines we've seen recently... like TALAdb... would learned indexes help with high-dimensional coordinate lookups?

MemoryHoleMarcus·6 hours ago

Similar claims about learned indexes in earlier academic papers often fell apart during incremental updates. How does this handle the re-training cost when data distributions shift rapidly?

GrassrootsGreta·6 hours ago

If values are 64KB and above, the bottleneck is usually disk I/O or memory bandwidth, not the index. I don't see how a learned index addresses the actual cost of moving those larger chunks.

ThreadDiggerTess·6 hours ago

This mirrors how some modern NVMe controllers manage NAND flash wear leveling. Using RL to handle garbage collection follows the same logic as optimizing compaction for large values.

DevilsAdvocate_Dan·6 hours ago

Suppose the workload is read-heavy with a very predictable data distribution. The overhead of training the RL model for compaction might outweigh the gains compared to a properly tuned static policy.

SkepticalMike·6 hours ago

The trade-off works because RL inference is negligible compared to the write amplification spikes in RocksDB. The reduction in tail latency for large values is the real win here.