What is Ceph?
Ceph is an open-source, software-defined storage system that serves object, block and file storage from one cluster built on commodity servers. Every interface sits on RADOS, a distributed object store in which OSD daemons store, replicate, recover and rebalance data on their own. Placement is calculated with the CRUSH algorithm rather than looked up in a central table, so clients and daemons compute where a placement group lives and talk to the right OSDs directly. The cluster components page describes the five daemon types: Monitors, Managers, OSDs, Metadata Servers for CephFS and the RADOS Gateway for S3 and Swift.
The project began in 2004 as research at the Storage Systems Research Center of the University of California, Santa Cruz, where Sage Weil completed his PhD thesis on Ceph in 2007. Inktank was founded in 2012 to sell support, the first long-term stable series (Argonaut) shipped the same year, and Red Hat acquired Inktank in 2014. Since 2018 the Ceph Foundation has operated as a directed fund under the Linux Foundation, as summarized on the official history page. Development happens in the open on GitHub, with mailing lists, a Foundation Slack and public developer meetings.
Ceph is distributed as Linux packages from download.ceph.com and as the container image quay.io/ceph/ceph, which has no latest tag and is meant to be pinned to an explicit version. cephadm is the recommended deployment tool for bare metal and virtual machines; Rook is recommended for Kubernetes. The OS recommendations list Rocky Linux 10, CentOS Stream 9 and Ubuntu 22.04 and 24.04 as tested platforms for Umbrella, with untested packages for Debian 12 and 13. A native Windows client for RBD and CephFS exists, but its packages are built outside the core team.
The current stable release is v21.2.0 Umbrella, published October 28, 2026, with an estimated end of life in mid-2028. It adds an optimized erasure-coding I/O path, a new default CephX cipher with online key rotation for daemons, Monitor database backups, AES-256-GCM server-side encryption and Glacier-style cloud restore in RGW, a userspace FSCrypt implementation for CephFS and a multi-threaded cephfs-mirror. Its container image is built on Rocky Linux 10, so x86 hosts need CPUs with the x86-64-v3 instruction level. Tentacle (v20.2.5) and Squid (v19.2.7) are still maintained according to the release index.
The core is written in C++, with the Manager modules, cephadm and the Dashboard backend in Python and the Dashboard frontend in Angular. Most code is dual-licensed under LGPL 2.1 or LGPL 3.0, a few headers carry GPL terms, and the documentation is CC BY-SA 3.0. Ceph is not a single-node NAS appliance or a backup application: it assumes several hosts, dedicated drives per OSD and an operator who plans failure domains, networks and capacity before data arrives.
Ceph screenshots
Explore the interface. Open a screenshot to view it at full size.
Ceph Dashboard landing page from the official documentation, showing inventory, capacity and IOPS, latency and throughput graphs. The screenshot was taken on a Reef development build, so the current layout differs.
Official architecture diagram: applications use LIBRADOS or the S3 and Swift gateway, hosts and VMs use RBD, clients use CephFS, all on the RADOS object store.
Grafana cluster dashboard shipped in the Ceph repository, fed by the Prometheus Manager module: health, OSD and Monitor counts, PG states, latency, I/O and recovery rate.
cephfs-top output from the CephFS documentation, showing per-client metrics for two file systems mounted through FUSE.
Source: official Ceph project page.
Key capabilities
- RADOS distributed object store All data is stored as objects in pools, hashed to placement groups and mapped to OSDs with CRUSH. Pools can be replicated (three copies is the usual production choice) or erasure-coded. OSDs heartbeat each other, mark failed peers and rebuild missing copies without operator intervention, as long as enough capacity remains in the surviving failure domains.
- Erasure coding with several plugins Erasure-coded pools trade CPU and latency for capacity efficiency. The erasure-code documentation covers the jerasure, ISA, LRC, SHEC and CLAY plugins. Umbrella adds an optimized "Fast EC" path with direct reads and OMAP support, rejects non-4k-aligned chunk sizes for that path and raises health warnings for deprecated plugins and techniques.
- S3 and Swift object storage through RGW The RADOS Gateway (
radosgw) is an HTTP service with S3-compatible and Swift-compatible APIs, run as one instance or several behind a load balancer. It supports bucket policies, IAM-style accounts, STS with OIDC, bucket notifications, lifecycle rules, Lua scripting and server-side encryption. Check the S3 API compatibility page for the operations your applications depend on. - Multisite object replication RGW realms, zonegroups and zones replicate buckets asynchronously between clusters, which the multisite guide describes for active-active and disaster-recovery layouts. Umbrella fixes several full-sync and startup issues in this code. Replication is eventual, so applications that read immediately after writing in another zone need to tolerate lag.
- RBD block devices RADOS Block Device images are thin-provisioned and striped across the cluster. Linux hosts map them with the
rbdkernel module, QEMU and libvirt use librbd directly, and the docs cover OpenStack, CloudStack, Nomad and Kubernetes. Images support snapshots, copy-on-write clones, exclusive locks, live migration and LUKS-based encryption. - RBD mirroring between clusters Images can be mirrored asynchronously to a second cluster in journal-based or snapshot-based mode. The mirroring guide notes that journaling roughly doubles write latency because every write lands in a journal first, while snapshot mode only ships deltas between scheduled mirror snapshots. Failover with snapshot mode rolls back any partially applied delta.
- NVMe over TCP gateway The NVMe-oF gateway exports RBD images as NVMe/TCP targets for Linux and VMware initiators, where they appear as block devices or VMFS datastores. Umbrella extends Dashboard and cephadm support with TLS, mTLS, DH-HMAC-CHAP authentication and KMIP fields. The older iSCSI gateway has been in maintenance since November 2022 and receives no new features.
- CephFS POSIX file system CephFS stores file data in RADOS and keeps metadata in Metadata Server daemons, with at least one active MDS per file system plus a standby. It offers a Linux kernel client and a FUSE client, multiple active MDS ranks, quotas, subvolumes and snapshots with scheduled retention. Umbrella adds a userspace FSCrypt implementation and checkpointed, multi-threaded cephfs-mirror.
- NFS and SMB exports for CephFS The Manager can deploy NFS-Ganesha clusters with the nfs module and Samba clusters with the smb module. Ceph-managed Samba supports SMB2 and SMB3 only, not SMB1, and requires cephadm orchestration. Umbrella adds per-share QoS, custom ports and TLS options for these gateways.
- cephadm orchestration Cephadm deploys every daemon as a container managed by systemd, connects to hosts over SSH and is driven through
ceph orchcommands and YAML service specifications. Requirements are Python 3.9 or later, systemd, Podman or Docker, time synchronization and LVM2, per the cephadm install guide. It also performs rolling upgrades and, since Umbrella, manages certificates for gateways and monitoring. - Rook for Kubernetes Rook is the recommended way to run Ceph inside Kubernetes, or to connect Kubernetes to an external cluster. Its operator owns the OSD lifecycle in that mode, so
ceph orchcannot add or remove OSDs there. Persistent volumes are provisioned through the Ceph CSI drivers for RBD and CephFS; see rook.io. - Dashboard and REST API The Manager hosts a web Dashboard for health, capacity, pools, hosts, OSDs, RGW buckets and users, CephFS, NFS, SMB and NVMe-oF. The same service exposes the Ceph RESTful API under
/api, authenticated with JWT tokens and versioned through media types. The docs warn that some endpoints are still under development and may change between releases. - Prometheus and Grafana monitoring The prometheus Manager module exports cluster, pool, OSD and daemon metrics, and the repository ships Grafana dashboards and alert rules. cephadm can deploy Prometheus, Grafana, Alertmanager and node-exporter alongside the cluster. Health checks are also available from
ceph health detailfor scripts and paging. - Command-line tooling Operators work mostly with the
cephCLI, plusrbd,radosgw-adminandcephfs-topfor client statistics. Most settings live in the Monitors central configuration database and are changed withceph config set, rather than in per-host files. Language bindings in librados cover C, C++ and Python. - Packages and container images Release builds are published as APT and DNF packages from download.ceph.com and as
quay.io/ceph/cephimages with tags such asv21.2andv21.2.0; the container images page explains the moving and pinned tags. Ubuntu, CentOS Stream, Fedora and SUSE also carry cephadm in their own repositories. Legacyceph/daemonimages are no longer built. - Windows client A Windows client maps RBD images as local disks through the WNBD driver and mounts CephFS through ceph-dokan, which needs Dokany 2.0.5 or later. The Windows install guide supports Windows Server 2019 and 2022, and requires Secure Boot to be disabled because the WNBD driver is not signed by Microsoft. These client packages are not maintained or tested by the core team.
- Hardware sizing guidance The minimum hardware table asks for at least 4 GiB of RAM per OSD, 5 GB of RAM and 100 GB of SSD-backed storage per Monitor and 8 GiB of RAM per MDS. It suggests 3 CPU threads per HDD OSD and 6 per NVMe OSD, 10 Gb/s networking and enterprise drives with power-loss protection. These are floors, not production targets.
Where Ceph fits
Private S3 object storage
Deploy RGW on two or more hosts behind a load balancer, place the bucket index pool on SSDs and choose replication or erasure coding for the data pool based on object size and capacity targets. Create accounts and users with radosgw-admin or the Dashboard, then run the S3 calls your applications use, including multipart uploads and presigned URLs. Check that the S3 compatibility list covers versioning, object lock or lifecycle rules if backup software depends on them.
Virtual machine and private cloud storage
RBD is the usual backend for OpenStack Cinder and Glance, for libvirt and QEMU hosts and for hypervisor platforms that integrate Ceph. Keep VM pools replicated with three copies, separate the public and cluster networks when bandwidth is tight, and test live migration between hypervisors. Measure latency under recovery as well as under normal load, because backfill after a host failure competes with guest I/O.
Persistent volumes for Kubernetes
Run Ceph inside Kubernetes with the Rook operator, or point Ceph CSI at an existing cluster managed by cephadm. RBD-backed volumes suit ReadWriteOnce workloads such as databases; CephFS-backed volumes serve ReadWriteMany workloads. Confirm that StorageClasses, snapshot classes and volume expansion behave as expected with the Kubernetes integration guide before migrating stateful applications.
Shared file system for teams and clusters
Create a CephFS volume with ceph fs volume create, give each project a subvolume with quotas and set snapshot schedules. Linux clients mount with the kernel driver, while Windows and macOS users reach the same data through Ceph-managed SMB or NFS-Ganesha gateways. Check how the SMB module maps users and authentication in your directory setup, and size MDS memory for the number of files clients keep open.
Disaster recovery between sites
Use RGW multisite for buckets, RBD mirroring for VM images and cephfs-mirror for directories, each running asynchronously to a second cluster. Pick snapshot-based RBD mirroring when write latency matters and journal-based mode when you need finer recovery points. Rehearse a promotion of the secondary site and a failback, and document how long resync takes for a realistic data volume. A stretch-mode cluster across two sites with a tiebreaker Monitor is the alternative for synchronous replication.
Rolling upgrades and day-two operations
With cephadm, ceph orch upgrade start walks through Managers, Monitors, OSDs and gateways host by host while the cluster stays online, as described in the upgrade guide. Read the release notes first: Umbrella raises the PG-per-OSD targets of the autoscaler, so existing pools may split into more placement groups after the upgrade, and its x86 container image needs x86-64-v3 CPUs. Keep HEALTH_OK as the gate between upgrade steps.
Install
curl --silent --remote-name --location https://download.ceph.com/rpm-umbrella/el9/noarch/cephadm && chmod +x cephadm && ./cephadm bootstrap --mon-ip <mon-ip>Reviews of Ceph
Be the first to share your experience with Ceph.
Write a review
Loading review form…
Alternatives to Ceph
GarageAGPLv3 object store written in Rust that implements the S3 API, replicates data across zones over ordinary Internet links and runs as a single binary. Built for small to medium self-hosted clusters, not for maximum throughput.
SeaweedFSApache-licensed distributed storage for S3 objects, POSIX-style files and Iceberg or Lance tables, with O(1) blob access, horizontal scaling and cloud tiering.
ProxmoxProxmox VE is an open-source server virtualization platform that combines KVM virtual machines, LXC containers, storage and networking in one web interface.