RocksDB Storage Engine Architecture: LSM-Tree Mechanics and Write Amplification
RocksDB WAL, MemTables, SSTs, and compaction trade write durability, read cost, and space. Amplification and stalls depend on options, version, and workload.
Publication & corrections: infostoa · Editorial standards
Individual author and specialist reviewer information is not provided for this article.
RocksDB LSM write path and compaction conditions
RocksDB is an embedded key-value storage engine. Its LSM write path uses a WAL and in-memory MemTables, then stores flushed sorted data in SST files. Whether writes are logged and synchronized depends on WriteOptions and database configuration; do not assume every acknowledged write has reached stable storage. A flush creates new SST files from a memtable, while compaction merges SSTs and can remove obsolete versions and deletion tombstones. This does not mean all storage operations are sequential: reads and compactions can repeatedly read and rewrite data.
Expert perspective: RocksDB engineer Jay Zhuang on compaction
RocksDB engineer Jay Zhuang analyzes how aligning compaction output boundaries with files in the next level can reduce repeated work in a particular level-compaction strategy. The post says the change is enabled by default in RocksDB 7.8 and later, but its reported write-amplification improvement comes from a particular design and db_bench setup. The simulator assumes equal key-value sizes and does not model deletions, compression, or write stalls during compaction. Those benchmark results are not a performance guarantee for arbitrary production workloads.
Compaction styles and the read path
With leveled compaction, L0 key ranges can overlap, while lower levels are generally maintained with non-overlapping ranges within each level. Other styles, such as universal or tiered compaction, have different overlap and read-cost behavior. It is therefore inaccurate to say every LSM tree has levels that grow by exactly 10x or follows one compaction rule. Reads can consult memtables, multiple SSTs, block cache, indexes, and filters. Bloom filters can help skip files that do not contain a key, but false positives are possible and filters consume memory, CPU, and storage. There is no fixed “99% less I/O” result independent of data and filter configuration.
Metrics and workload tests for tuning
LSM engines trade off write amplification, read amplification, and space amplification. If incoming writes exceed background compaction capacity for long enough, pending compaction can grow and write stalls can occur. Evaluate write_buffer_size, background jobs, level-size multipliers, compaction style, compression, block cache, and Bloom filter bits together. Option names, defaults, and effects can change by RocksDB release; use documentation for the actual library version. Benchmark representative key distributions, update/delete ratios, value sizes, and concurrent read/write traffic. Compare p95/p99 latency, pending compaction bytes, stall duration, device write volume, and space usage.
Read evidence and choose the next test in a hypothetical workload
The workload and values below are hypothetical teaching examples, not RocksDB benchmark results or recommended defaults. Define, for example, 16-byte keys, 1-KiB values, 70% point reads, 25% overwrites, 5% deletes, and fixed concurrency for a 30-minute run. Record whether the dataset exceeds block cache, exclude warm-up, and measure database input bytes and device writes over the same interval. Comparisons become ambiguous when those conditions drift.
| Hypothetical observed evidence | First interpretation to test | Next single-variable validation |
|---|---|---|
| DB input 10 MB/s and device writes 30 MB/s in the same interval | Under these assumptions, total writes/input are about 3. Separate WAL and other-process writes before attributing the ratio to compaction. | Align the DB-stat and device-stat windows and isolate other-process writes to establish a baseline. Do not change an option yet. |
L0 file count and pending compaction bytes rise together, while STALL_MICROS also increases | Compaction may be falling behind and causing stalls; this alone does not prove background-job count is too low. | Hold workload and baseline settings fixed; change only background compaction job count by one step. Compare stall time, p99 write latency, and device utilization. |
| Point-read p99 worsens while write stalls stay low and block-cache hit rate falls | This workload may be under read-path/cache pressure; compaction is not established as the cause. | Hold everything else fixed and change only block-cache capacity. Compare hit rate, physical reads, and p99. |
Repeat each run with the same seed and dataset; record settings, RocksDB version, hardware, warm-up, and measurement interval. One metric can improve while another regresses, so define the target and acceptable costs first. The RocksDB tuning guide likewise recommends inspecting statistics and system bottlenecks before changing workload-specific options (RocksDB Tuning Guide, Compaction Stats and DB Status).