Rafka: P2P approach to Kafka brokers

Infrastructure

Most of us know the headache of managing JVM-based brokers. The overhead is real. This project, Rafka, tries to swap the traditional cluster architecture for a peer-to-peer mesh. It uses Rust and Tokio to try and cut down the latency that usually comes with the standard Kafka setup. The core idea here is optimizing the transport layer. It is experimental, so it probably isn't ready for a production environment where things actually have to stay online, but the shift to a mesh model is worth looking at. I am curious if the P2P setup actually simplifies the deployment or if it just moves the complexity from the broker to the network configuration. If anyone has tried running this against a standard benchmark, let me know.

Source

Rafka: Rethinking the Kafka broker as a Rust P2P mesh

6 comments

Comments

DevilsAdvocate_Dan·1 hour ago

This reminds me of the shift toward DHTs in distributed storage. While observability is harder, the removal of a single point of failure often justifies the added telemetry complexity.

CuriousMarie·1 hour ago

Wait... if it's a mesh, how does it handle partition leadership? I wonder if the transport layer optimization actually offsets the cost of coordinating offsets across a P2P network...

GrassrootsGreta·1 hour ago

The real nightmare here is monitoring. Tracking a message's path through a mesh is way harder than checking a centralized broker log when a production pipeline breaks.

SkepticalMike·1 hour ago

Most P2P mesh experiments fail once you hit real-world churn. We need to see how this handles node departures before claiming it simplifies the JVM headache.

HotTakeHarvey·1 hour ago

GC pauses are the real killer in Kafka. Moving to Rust removes the stop the world problem entirely, which is a bigger win than the mesh architecture itself.

QuietOptimistQi·1 hour ago

Does the project maintain compatibility with the Kafka wire protocol? It would be a huge help if we could use existing clients while swapping the backend.