Getting Started

Walrus provides a durable, topic-aware write-ahead log for Rust services that need predictable latency under load. This guide walks through installation, core APIs, and the configuration surfaces you are likely to touch first. For a full tour of every knob, see Configuration.

Key Features

• High performance append and read paths optimized for concurrent producers and consumers
• Topic-based isolation with independent cursors and persistent read offsets
• Configurable durability through read consistency and fsync scheduling controls
• Atomic batch operations for both writes and reads with io_uring acceleration on Linux
• Benchmark tooling that captures throughput, latency, and resource usage

Install the Crate

Add Walrus to your Cargo.toml dependencies:

[dependencies]
walrus-rust = "0.1.0"

Then run cargo build to fetch the crate.

First WAL Instance

use walrus_rust::{Walrus, ReadConsistency};

fn main() -> std::io::Result<()> {
    let wal = Walrus::new()?;                         // StrictlyAtOnce by default
    wal.append_for_topic("events", b"hello walrus")?; // append a payload

    if let Some(entry) = wal.read_next("events", true)? {
        println!("read {:?}", String::from_utf8_lossy(&entry.data));
    }

    Ok(())
}

• Passing checkpoint = true advances the persisted cursor according to the configured consistency mode.
• Use checkpoint = false to peek without consuming the entry.

Read Consistency Modes

Walrus ships with two read models. Choose what matches your durability needs:

• ReadConsistency::StrictlyAtOnce — persists the read offset every time you checkpoint. Guarantees at-most-once delivery after crashes at the cost of extra I/O.
• ReadConsistency::AtLeastOnce { persist_every } — buffers cursor updates and persists every N reads. Higher throughput, but the last window of entries may repeat after restarts.

let wal = Walrus::with_consistency(
    ReadConsistency::AtLeastOnce { persist_every: 5_000 },
)?;

Fsync Scheduling

Tune when Walrus flushes dirty data to disk:

• FsyncSchedule::Milliseconds(n) — background thread flushes every n milliseconds (default: 1000 ms).
• FsyncSchedule::SyncEach — fsync immediately after every write; slowest but safest.
• FsyncSchedule::NoFsync — rely on the OS page cache for durability (fastest, least safe).

use walrus_rust::{FsyncSchedule, ReadConsistency, Walrus};

let wal = Walrus::with_consistency_and_schedule(
    ReadConsistency::StrictlyAtOnce,
    FsyncSchedule::Milliseconds(2_000),
)?;

Batch Operations

• batch_append_for_topic writes up to 2,000 entries atomically. On Linux, the fd backend drives io_uring submissions; otherwise writes fall back to sequential mmap I/O.
• batch_read_for_topic streams entries in commit order, respecting both a caller-provided byte cap and the same 2,000-entry ceiling.

See the API reference for details on these batch APIs.

Environment Variables

Fine-tune behaviour without changing code:

• WALRUS_DATA_DIR — relocate the entire wal_files/ tree.
• WALRUS_INSTANCE_KEY — namespace default constructors under wal_files/<sanitized-key>/.
• WALRUS_QUIET — silence debug logging when toggling backends.
• WALRUS_FSYNC — configure benchmark fsync cadence (sync-each, no-fsync, async, <n>ms).
• WALRUS_THREADS — control benchmark thread counts (<n> or <start-end>).
• WALRUS_DURATION, WALRUS_WRITE_DURATION, WALRUS_READ_DURATION — set benchmark runtimes.
• WALRUS_BATCH_SIZE — override entries per batch for batch benchmarks.

Performance Profiles

Mix and match read consistency and fsync scheduling to suit your workload:

Maximum Throughput (No Durability)

let wal = Walrus::with_consistency_and_schedule(
    ReadConsistency::AtLeastOnce { persist_every: 10_000 },
    FsyncSchedule::NoFsync,
)?;

High Throughput (Some Durability)

let wal = Walrus::with_consistency_and_schedule(
    ReadConsistency::AtLeastOnce { persist_every: 10_000 },
    FsyncSchedule::Milliseconds(5_000),
)?;

Maximum Durability

let wal = Walrus::with_consistency_and_schedule(
    ReadConsistency::StrictlyAtOnce,
    FsyncSchedule::SyncEach,
)?;

Balanced Performance

let wal = Walrus::with_consistency_and_schedule(
    ReadConsistency::AtLeastOnce { persist_every: 1_000 },
    FsyncSchedule::Milliseconds(1_000),
)?;

Configuration	Throughput	Durability	Typical Use Case
NoFsync	Highest	None	Performance experiments, disposable data
Milliseconds(5000)	High	Low	High-volume logging
Milliseconds(1000)	Medium	Medium	Balanced production workloads
Milliseconds(100)	Lower	High	Near real-time guarantees
SyncEach	Lowest	Highest	Financial/safety-critical systems

Benchmark Quick Start

Walrus ships with Make targets and plotting scripts:

pip install pandas matplotlib  # plotting dependencies
make bench-and-show-reads      # run read benchmarks and open graphs

Additional targets such as make show-writes, make show-reads, and make show-scaling visualize CSV output from the benchmarking suite.