Proxmox storage: Ceph vs ZFS for a cluster

Teams leaving VMware after Broadcom usually settle on Proxmox quickly. Then they hit the real decision: what backs the VM disks. On a single host it does not matter much. On a cluster it decides your failover behaviour, your network bill and how a dead node actually feels at 2 a.m. The two serious options are ZFS and Ceph, and they are not two flavours of the same thing.

The useful question is not “which is faster”. It is: what failure do you need to survive, how many nodes and how much network can you spend, and how much data loss is acceptable on a node failure. Answer those and the choice mostly makes itself.

What each one is in a Proxmox cluster

ZFS — local storage with scheduled replication

ZFS on Proxmox is local to each node. It is a superb single-host filesystem — end-to-end checksums, self-healing on mirror/raidz, snapshots, compression, and an in-RAM cache (ARC) that makes reads fly. But it is not shared. To get VMs onto a second node you use Proxmox's built-in ZFS replication, which sends incremental snapshots on a schedule (as often as every minute). That is asynchronous: when a node dies, you promote the replica on another node and lose everything written since the last replication run. For many workloads a minute of loss is fine; for a database mid-transaction it may not be.

Ceph — distributed, shared, synchronous

Ceph is a distributed storage cluster that Proxmox manages natively (as RBD). Every write is replicated synchronously to multiple nodes (default three copies, minimum two to stay writable). Storage is shared: any node can run any VM, live migration is instant, and a node failure means the VM simply restarts elsewhere with zero data loss. Ceph self-heals when a disk dies by re-replicating its data. The price is that Ceph wants scale and a real network: at least three nodes (five is calmer), fast dedicated links, enterprise NVMe, and more operational knowledge than ZFS.

The comparison that matters

Failure domains: what actually happens when a node dies

This is the whole point of a cluster, so be concrete about it. With ZFS replication, a dead node means its VMs are down until you start the last replicated copy on a surviving node; those VMs come back having lost whatever happened since the last snapshot send. A disk failure inside a node is handled locally by the mirror or raidz — no cluster involvement. With Ceph, a dead node loses nothing: the other copies are already current, affected VMs restart elsewhere immediately, and Ceph rebuilds redundancy in the background. A single OSD (disk) failure is a non-event — Ceph re-replicates automatically, provided enough nodes stay up to satisfy the minimum copy count.

Ceph's resilience assumes you keep min_size nodes alive. On a tight three-node cluster with size=3/min_size=2, losing one node is fine but losing two stops writes. If you cannot run five nodes, size the cluster and the maintenance windows around that reality.

The network is the Ceph decision

More Ceph clusters are ruined by the network than by anything else. Every write goes over the wire two extra times, and recovery traffic after a failure competes with live VM I/O. Ceph on a shared 1 GbE link will be miserable and will make people blame Ceph for a cabling problem. The baseline is a dedicated 10 GbE network, ideally with the Ceph public and cluster (replication) traffic separated, and 25 GbE or more once you push real I/O. If a fast, dedicated network is not in the budget, that single fact is a strong argument for ZFS.

Hardware each one demands

ZFS is forgiving: mirrored or raidz vdevs on whatever disks you have, plenty of RAM for ARC, and — if you run sync-heavy workloads — a small, power-loss-protected SSD as a separate log device. Skip deduplication unless you have measured that you need it. Ceph is less forgiving: use enterprise NVMe with power-loss protection, because consumer SSDs without PLP collapse under Ceph's sync writes, budget roughly a few GB of RAM and a chunk of CPU per OSD, and do not mix wildly different disk sizes or speeds. The cheapest reliable Ceph is more expensive than the cheapest reliable ZFS.

Performance, honestly

For a single VM doing latency-sensitive work, local ZFS on NVMe wins — there is no network hop and no triple write. Ceph trades per-operation latency for distributed durability and for aggregate throughput that grows as you add nodes. So the honest framing is: ZFS is faster for one busy VM on a small cluster; Ceph is about surviving failure and scaling out, not about beating local disk on latency. Benchmark your own workload, but do not expect Ceph to feel like local NVMe, and do not expect ZFS replication to give you Ceph's zero-loss failover.

How we choose

For a two- or three-node cluster on a sensible budget, where a minute of potential data loss on a rare node failure is acceptable, we use ZFS with replication. It is cheap, fast for local reads, easy to reason about, and the failure behaviour is simple to explain to whoever is on call.

When the requirement is genuine high availability — zero data loss, instant failover, live migration — and there are at least three (preferably five) nodes with a dedicated 10/25 GbE network and enterprise NVMe, Ceph is the right tool and worth its complexity. A common middle path also works: ZFS for the bulk of VMs, and Ceph (or a separate shared target) only for the handful that truly cannot lose a transaction.

If you are still deciding between hypervisors at all, start with VMware vs Proxmox vs Hyper-V and what Broadcom's VMware actually costs now — the storage choice only matters once Proxmox is the answer.

Bottom line

• •2—3 nodes, budget-conscious, a minute of failover data loss is acceptable → ZFS + replication.
• •True HA (zero loss, instant failover), ≥3 nodes, dedicated 10/25 GbE, enterprise NVMe → Ceph.
• •The network, not the software, decides whether Ceph is a good idea — no fast dedicated link, no Ceph.
• •Mixed estate? ZFS for most VMs, Ceph only for the few that cannot lose a transaction.

Planning the move off VMware end to end? We walk through what actually transfers and what breaks in leaving VMware after Broadcom.