I've been working on Walrus, a message streaming system (think Kafka-like) written in Rust. The focus was on making the storage layer as fast as possible.

Performance highlights:

• 1.2 million writes/second (no fsync)
• 5,000 writes/second (fsync)
• Beats both Kafka and RocksDB in benchmarks (see graphs in README)

How it's fast:

The storage engine is custom-built instead of using existing libraries. On Linux, it uses io_uring for batched writes. On other platforms, it falls back to regular pread/pwrite syscalls. You can also use memory-mapped files if you prefer.

Each topic is split into segments (~1M messages each). When a segment fills up, it automatically rolls over to a new one and distributes leadership to different nodes. This keeps the cluster balanced without manual configuration.

Distributed setup:

The cluster uses Raft for coordination, but only for metadata (which node owns which segment). The actual message data never goes through Raft, so writes stay fast. If you send a message to the wrong node, it just forwards it to the right one.

You can also use the storage engine standalone as a library (walrus-rust on crates.io) if you just need fast local logging.

I also wrote a TLA+ spec to verify the distributed parts work correctly (segment rollover, write safety, etc).

Code: https://github.com/nubskr/walrus

Would love to hear what you think, especially if you've worked on similar systems!