// HACKER NEWS — CYBERSECURITY
We Replaced MMAP with Io_uring in Our Rust Query Engine. It Got Slower
In the beginning, there was mmap. It was convenient: it let us lazily read huge numbers of Arrow IPC files from disk without managing memory ourselves. It fit our file format perfectly — Arrow IPC’s layout is designed for zero-copy random access, and mmap gives you exactly that.
Then we deployed to production, ran real concurrent query loads, and mmap became a real problem.
At Conviva, we analyze trillions of events a day to pinpoint and diagnose end user experience. At the core of our architecture is an event and pattern analysis engine built on DataFusion, Arrow, Rust, Rayon, and Tokio. Raw events get transformed, encoded in a proprietary mostly-numeric format, and stored in the cloud. We copy them to local NVMe and read large (~3–5 GB) Arrow IPC files. We chose Arrow IPC for simplicity and speed — its memory and disk layouts are identical, so decode cost is minimal, and mmap gives us zero-copy reads natively supported by arrow-rust. A typical query touches 6 columns across 8 batch files (one batch per file), ~1.6 GB per batch, ~13 GB total per day of data.
Hardware: 192-core box, ~750 GB RAM. Two disk configs during the investigation: 2× NVMe LVM-striped (~5.5 GB/s fio ceiling) and 32× NVMe RAID-0 (~21 GB/s fio ceiling). Kernel 5.15 during investigation, 6.x in production.
At lighter loads, mmap worked well — fast, serving queries from raw events in seconds. The trouble started under heavier concurrency. Some latency increase under load is expected — more queries competing for the same CPU. But we saw p95s and p99s spike well beyond what linear scaling would predict, with rows scanned per core dropping sharply even after accounting for concurrency:
That pointed at mmap page-cache thrashing under memory pressure.
To isolate the effect, we ran a controlled test: 1 pod vs. 4 pods on the same host, same concurrent query load. We expected 4 pods to win — more parallelism, better isolation. We were wrong.
For 14-day queries — long enough to fill the page cache — 1 pod beat 4 pods by a real margin: 41% faster at max, >20% at p95. The mmap page cache lives on the host and is shared across pods, so the 4 pods weren’t fighting each other for CPU — they were fighting for page cache. perf record on the same run showed 100% lock contention at the kernel level.
The core issue: mmap’s page cache is implicit shared state. Every process on the host shares one cache, one lock hierarchy, one eviction policy. No single pod controls the resource that matters most for read latency, and as concurrency rises, everyone’s slice of it shrinks.
We didn’t want to just guess, so we dug into the mmap mechanics and the page fault stats. When you mmap a file, the application gets a region of memory backed by the file on disk. Here’s what happens on access: