Using LVM with a cache for a root drive

TLDR. This post details how to setup a cache drive using LVM on Arch Linux.

I have long been a fan of Intel’s (sadly now dead) Optane technology. Optane was pitched as a new layer sitting in between RAM and traditional NAND flash SSD storage, that had the non-volatile properties of SSDs, but much lower access latency and higher write endurance when compared with NAND. Optane seemingly never found its market niche, and was discontinued in between 2022-2024.

I have owned a couple of Intel’s 900p PCI-e add in cards for years and they have served a variety of purposes in a couple of different systems. Last year I built a new AMD-based workstation and wanted to find a use for one of these curiosities. I considered a few options and ultimately settled on using the Optane drive as a cache for my main OS drive (A Samsung 990 Pro). This was made possible using Linux’s Logical Volume Management (LVM) tools. LVM provides a flexible block-level abstraction layer between physical storage devices (such as SSDs or HDDs) and filesystems like ext4. With LVM, physical disks are first grouped into a Volume Group (VG), from which Logical Volumes (LVs) can be created. These logical volumes behave like virtual block devices and can be resized, extended across multiple disks, or combined in flexible ways.

LVM works at the block layer, beneath any filesystem. At this level, storage is simply addressed as fixed-size blocks of bytes (typically 512 bytes or 4096 bytes, depending on the device). These blocks have no knowledge of files or directories, they are simply raw regions of storage. The filesystem sits above this layer and translates file operations into block reads and writes.

Importantly for this blog, LVM also supports block-level caching. Internally, this functionality is implemented using the Linux kernel’s dm-cache. LVM acts as a management layer that configures and maintains this cache, while the actual logic is handled by the kernel. This allows a faster device (an NVMe SSD for example) to act as a cache for a slower device. Because caching occurs at the block layer, it is completely transparent to the filesystem.

So in this blog I will show the steps that I took to setup LVM with my two drives on Arch Linux. We will then look at some quick benchmark numbers to what the impact in performance is.

I feel I should state that I am well aware that ‘caching’ an extremely fast drive like the Samsung 990 Pro with a technically slower drive like the Optane is not what this feature was designed for. I just did it for fun and would not recommend it.

---

Before getting started, I will assume:

• You have two drives (nvme0n1 and nvme1n1)
• These drives have been partitioned beforehand
• You have reserved a separate, non-LVM partition for /boot (required for GRUB or systemd-boot)

In this example:

• /dev/nvme0n1p2 → main data device (in my case, the Samsung 990 Pro)
• /dev/nvme1n1p1 → cache device (Intel Optane 900p)

I would also recommend reading the excellent Arch wiki page on LVM as well before getting started for yourself.

Creating Physical Volumes

First, initialise the partitions as LVM Physical Volumes (PVs):

pvcreate /dev/nvme0n1p2 /dev/nvme1n1p1

Notice that we use partitions rather than whole disks. This allows us to reserve space for /boot outside of LVM as I mentioned before.

The command pvdisplay can be used to see details of your physical volumes.

Creating a Volume Group

Next, create a Volume Group (VG). Here we call it vg0:

vgcreate vg0 /dev/nvme0n1p2 /dev/nvme1n1p1

This aggregates both devices into a single logical storage pool from which Logical Volumes (LVs) can be created.

The command vgdisplay can be used to see details of your new VG.

Creating a Logical Volume for root

Now create the root logical volume on the larger device:

lvcreate -l 100%FREE -n root vg0 /dev/nvme0n1p2

Here:

• -n root names the LV
• -l 100%FREE uses all free extents on /dev/nvme0n1p2
• The LV is created only on the main device (not the cache device)

At this stage, /dev/vg0/root is a normal LV, which is not being cached.

Adding Cache Logical Volumes

We now create two LVs on the second device:

• One for cache data
• One for cache metadata

lvcreate -L 440G -n cache_data vg0 /dev/nvme1n1p1
lvcreate -L 1G   -n cache_meta vg0 /dev/nvme1n1p1

In this example, the sizes are tuned to fit an Intel 900p drive. You should adjust these for your own hardware.

What does the cache metadata do?

The cache metadata is where the information required to make the cache worked is stored. This includes a mapping from the original block to the location of the cached block. The size of the metadata cache should scale with the amount of space that is used for the cache.

Creating the Cache Pool

We can now create the cache pool from the two LV’s we just made:

lvconvert --type cache-pool --poolmetadata vg0/cache_meta vg0/cache_data

Behind the scenes LVM will:

• Wipe the contents of both LVs
• Convert them into a dm-cache pool
• Create a metadata spare LV automatically (to allow a copy in case of issues with the main one)

After this step, LVM internally renames the pool to something like cache_data_cpool.

Attaching the Cache to the Root LV

Now we can attach the cache pool to the root LV:

lvconvert --type cache \
  --cachepool vg0/cache_data_cpool \
  --cachemode writeback \
  vg0/root

At this point, root has been converted into a cached LV. The cache pool is now attached to root and will begin accelerating block I/O operations.

About Writeback Mode

In this setup, we use:

--cachemode writeback

LVM cache supports two main modes:

Writethrough (safest option)

• Writes go to both the origin and the cache immediately
• Safest but slowest options
• No risk of losing acknowledged writes if the cache device fails

Writeback (faster but technically less safe approach used here)

• Writes are acknowledged once they land in the cache not the main storage
• Higher write performance
• Small risk: if the cache device fails or power is lost before dirty blocks are flushed, recent writes may be lost.

You can check cache status with:

lvs -a -o lv_name,cache_mode,cache_dirty_blocks,cache_read_hits,cache_read_misses vg0

Which in my case looks like this:

 LV                       CacheMode CacheDirtyBlocks CacheReadHits    CacheReadMisses
  [cache_data_cpool]       writeback                0            40472            22479
  [cache_data_cpool_cdata]
  [cache_data_cpool_cmeta]
  [lvol0_pmspare]
  root                     writeback                0            40472            22479
  [root_corig]

Using the New Logical Volume

Once everything is complete, the LV appears at:

/dev/vg0/root

You can now create a filesystem and install Linux:

mkfs.ext4 /dev/vg0/root

The filesystem is unaware of the cache — it simply sees a block device. The caching happens transparently behind the scenes. This LV can then be used to install Linux onto.

NOTE: For Arch, you must add lvm2 to the list of hooks in mkinitcpio.conf for LVM to be properly enabled in the initramfs.

---

In a follow up blog, I may benchmark the impact this has had on read/write performance as well as latency.