Example: Ceph Storage (Rook)

The ceph-rook plugin provides block, shared file and S3-compatible object storage for in-cluster workloads by installing and managing the Rook-Ceph operator and a singleton CephCluster.

Verifying it works

This page describes the design. To actually exercise the plugin against a local cluster — from empty machine to a pod writing to a Ceph-backed volume — follow the verification runbook.

What it does

1. Install: Installs the Rook-Ceph operator via Helm (from charts.rook.io/release) and bootstraps a single CephCluster resource
2. Discover: Rook discovers available block devices on cluster nodes and publishes them as Disk CRs (cluster-scoped inventory)
3. Provision: Operators select disks in the console and create a DiskPool CR, specifying which disks to use
4. Consume: Operators create a BlockStorage, FileStorage or ObjectStorage CR, which derives the StorageClass — a replication strategy on the consumer, not the pool
5. Reconcile: The plugin folds every DiskPool’s disks into the singleton CephCluster as OSDs, and for each BlockStorage/FileStorage/ObjectStorage creates a CephBlockPool/CephFilesystem/CephObjectStore and a StorageClass for in-cluster PVC (or ObjectBucketClaim) provisioning
6. Console: Serves list and detail HTML for Disk, DiskPool, BlockStorage, FileStorage and ObjectStorage resources, allowing operators to inspect discovered devices and manage storage

Object map

How the plugin’s objects relate: what an operator creates, what the plugin publishes and derives, and what everything feeds into:

File structure

Decision logic lives in its own file next to its test, so the parts worth getting right (replication, discovery parsing, claim precedence, rendering) are unit testable without a cluster. The reconcilers are glue over those functions.

plugins/storage/ceph-rook/
├── main.go                  # Entry point: NewPlugin, call pluginruntime.Run()
├── plugin.go                # Start: scheme, install, manager, reconciler registration
├── console.go               # Embeds console/ as http.FileSystem (ConsoleProvider)
├── config.go                # FUNP_-prefixed environment configuration
├── install.go               # Helm install, CRD apply, CephCluster bootstrap
├── definition.yaml          # Metadata, permissions, menu, customComponents, allowedResources
├── api/v1alpha1/
│   ├── disk_types.go         # Disk CR: discovered block device inventory
│   ├── diskpool_types.go  # DiskPool CR: operator's disk contribution
│   ├── blockstorage_types.go # BlockStorage CR: RBD (RWO) consumer
│   ├── filestorage_types.go  # FileStorage CR: CephFS (RWX) consumer
│   ├── objectstorage_types.go # ObjectStorage CR: RGW (S3) consumer
│   ├── groupversion_info.go  # Scheme registration + go:generate directives
│   └── zz_generated.deepcopy.go
├── crds/                    # Generated CRD YAML, embedded and applied at install
├── diskinventory_controller.go  # Rook discovery ConfigMaps -> Disk CRs
├── diskpool_controller.go    # DiskPool -> CephCluster OSDs
├── consumer_controller.go       # Generic consumer reconciler: BlockStorage -> CephBlockPool + RBD StorageClass, FileStorage -> CephFilesystem + CephFS StorageClass, ObjectStorage -> CephObjectStore + bucket StorageClass
├── union.go                 # Shared disk union: every live DiskPool's disks, deduplicated
├── claims.go                # Derived names + which pool owns a contested disk
├── replication.go           # spec.replicas -> replica count and CRUSH failure domain
├── discovery.go             # Parses Rook's device JSON; deterministic Disk names
├── cephcluster.go           # Disk statuses -> spec.storage.nodes
├── rookvalues.go            # Helm values + the CephCluster bootstrap object
├── blockpool.go             # Renders CephBlockPool
├── filesystem.go            # Renders CephFilesystem
├── objectstore.go           # Renders CephObjectStore
├── storageclass.go          # Renders the RBD, CephFS and bucket StorageClasses
├── storageclass_apply.go    # Create/reconcile a StorageClass its owner already owns
├── console/                 # Hand-written console pages (no build step)
│   ├── _shared.js           # SDK loader, escaping, navigation helpers
│   ├── consumer-pages.js    # List/detail page factory and create/edit forms for the consumer kinds
│   ├── bucket-pages.js      # Buckets section (ObjectBucketClaims) on the ObjectStorage detail page, and its create/detail forms
│   ├── diskpool-forms.js    # DiskPool create/edit forms
│   ├── sheet.{html,js}      # Every sheet: picks the form by kind and view
│   ├── disk-picker.js       # Shared disk-selection widget (pool/consumer create+edit)
│   ├── disks-list.{html,js}
│   ├── disks-detail.{html,js}
│   ├── diskpools-{list,detail}.{html,js}
│   ├── blockstorages-{list,detail}.{html,js}
│   ├── filestorages-{list,detail}.{html,js}
│   └── objectstorages-{list,detail}.{html,js}
├── test-resources.yaml      # Sample DiskPool and consumer kinds for sandbox verification
└── Dockerfile               # Multi-stage build (Go build + alpine with helm)

Most *.go files above have a matching *_test.go. The exceptions are union.go, storageclass_apply.go and consumer_controller.go — all three are exercised through the per-kind controller test files (blockstorage_controller_test.go, filestorage_controller_test.go, objectstorage_controller_test.go, diskpool_controller_test.go) rather than a test file of their own.

What the console can do

Resource	Pages
DiskPool	list, detail, create, edit
BlockStorage	list, detail, create, edit
FileStorage	list, detail, create, edit
ObjectStorage	list, detail, create, edit, embedded Buckets section
Disk	list, detail

Create and edit open in a console sheet over the list and detail pages, with fundament.openSheet() (see Sheets). Every create and edit in customComponents names the same sheet.html, whose sheet.js picks the form from the kind and view the console opens it with. The forms are built from the shared NLDD Design System bundle the console serves per FUN-18. The create pages work on their routed create page too. The Buckets section opens the ObjectBucketClaim create and detail pages the same way, handing the create page the store’s StorageClass in params. DiskPool edits its disk selection with the same picker the create form uses; BlockStorage edits replicas and whether its StorageClass is the cluster default; FileStorage edits replicas and metadata server count; ObjectStorage edits replicas and gateway instance count. Every edit saves a merge-patch of spec only, so status is never clobbered, and DiskPool’s disk array is replaced wholesale rather than merged element-wise.

Unchecking a disk on a DiskPool warns what the reconciler will and will not do: the device leaves the CephCluster list, but its OSD keeps running until it is purged from Ceph manually, and data may rebalance in the meantime.

None of the five kinds offer delete from the console — allowedResources grants no delete verb. Deleting a DiskPool, BlockStorage, FileStorage or ObjectStorage is a kubectl delete; owner references still cascade the derived CephBlockPool/CephFilesystem/CephObjectStore/StorageClass, and OSDs still survive it, same as ever.

The host has its own generic delete for plugin resources, but resource-detail.component.html renders either a custom detail component or the generic view, never both — so a plugin with a custom detail page loses the host’s delete unless it implements one itself. None of these five do.

Console pages

console.go is what makes the pages reachable: pluginruntime.Run only serves /console/ for a plugin that implements ConsoleProvider, and each page must also be named under spec.customComponents in definition.yaml or the host has no route to it. definition_test.go asserts both directions — every referenced file exists, and every .html file is referenced.

Pages are served under the plugin CSP (script-src 'self'; style-src 'self', no unsafe-inline), so they carry no inline onclick handlers and no inline style attributes — both are blocked. Events are wired with addEventListener and styling comes from the .plugin-* classes in plugin-sdk.css. Navigation goes through _shared.js’s navigateToDetail(), which posts to window.fundament.parentOrigin rather than *; from a consumer or DiskPool create sheet the console closes the sheet and opens the new resource. A bucket create sheet instead opens the new claim’s detail sheet in its place. Create and edit are opened with fundament.openSheet(), and the page under the sheet lists again on fundament.onResourcesChanged().

Disk → DiskPool → consumer-kind flow

The plugin follows a four-step workflow:

1. Device Discovery (Disk CR): Rook continually scans raw, unpartitioned disks on cluster nodes. The DiskInventoryReconciler publishes these as cluster-scoped Disk objects, annotated with node, device path, size, type (HDD/SSD/NVMe), and availability status.

Device identity

A Disk reports two paths. status.path is the kernel name (/dev/sdb) — what an operator matches against lsblk, but the kernel is free to reassign it on the next boot. status.stablePath is the /dev/disk/by-id/ link, picked from rook’s devLinks in preference order (wwn-, nvme-eui., scsi-, ata-, …); links naming a logical layer (lvm-pv-uuid-, dm-, md-uuid-) are ignored, because those can be rebuilt on top of a different physical device.

The stable path is what goes into CephCluster.spec.storage.nodes[].devices[].name — Rook documents the by-id form there as the one that “will not change after reboots” — and it is what the Disk CR’s name is hashed from (DeviceKey: stable path, else WWN, else serial, else kernel path). That matters because a Disk whose name moves drops out of every DiskPool that lists it: the old name resolves to nothing, and the device silently leaves the CephCluster.

Loop devices, and some virtual disks, expose no stable identity at all. They fall back to the kernel path and are only as stable as the kernel’s ordering — acceptable for the k3d dev flow, which is the only place they occur.

2. Pool Selection (DiskPool CR): Operators use the console to select which disks to pool together. Creating a DiskPool specifies a list of disk names. The DiskPoolReconciler watches the DiskPool and folds every live pool’s disks into the singleton CephCluster’s OSDs.

3. Consumer creation (BlockStorage / FileStorage / ObjectStorage CR): Operators create a BlockStorage for block (RWO) volumes, a FileStorage for shared (RWX) volumes or an ObjectStorage for S3 buckets, each specifying a replication strategy (see Replication below); FileStorage additionally sets metadataServers, ObjectStorage sets gatewayInstances. Each kind’s reconciler watches its kind, plus Disk and DiskPool changes that affect the shared OSD set.

4. Storage Class (CephBlockPool/CephFilesystem/CephObjectStore + StorageClass): For each BlockStorage, the plugin creates a CephBlockPool and an RBD StorageClass. For each FileStorage, it creates a CephFilesystem (MDS active + standby) and a CephFS StorageClass. For each ObjectStorage, it creates a CephObjectStore (RGW gateways) and a bucket StorageClass. All sit over the OSDs contributed by every DiskPool; the first two are what in-cluster workloads reference in a PersistentVolumeClaim spec, the last in an ObjectBucketClaim.

Derived object names

Both derived objects are named ceph-<name> for a BlockStorage (RBD), cephfs-<name> for a FileStorage (CephFS), or cephobj-<name> for an ObjectStorage (RGW) — not <name>. The name is operator-chosen and a StorageClass is cluster-scoped, so an unprefixed name would collide with whatever else is on the cluster — a BlockStorage called local-path would otherwise take over k3d’s default StorageClass and garbage-collect it when it was deleted. status.storageClassName reports the real name, which is what a PersistentVolumeClaim should reference.

The prefix reduces collisions; ownership is what prevents damage. Before writing either object the reconciler checks for a controller reference back to this consumer, and refuses to adopt anything else — it goes Degraded with the conflicting object named, rather than silently taking it over.

Contested disks

Nothing stops two DiskPools from listing the same disk. When that happens the older pool keeps it (ties break on name), the younger pool skips it and says so in status.message, and the disk is counted once. The same precedence populates Disk.status.claimedBy, which is what the console’s disk picker filters on, so the inventory and the reconciler never disagree about who owns what.

Single CephCluster, multiple DiskPool model

The plugin follows a singleton CephCluster pattern: only one CephCluster is deployed per cluster. Multiple DiskPool CRs contribute disks to the same shared OSD set; multiple BlockStorage/FileStorage/ObjectStorage CRs each derive their own CephBlockPool/CephFilesystem/CephObjectStore and StorageClass over that same set.

This design supports:

• Multi-tenancy: Different consumer objects can have different replication (and their own metadataServers / gatewayInstances counts), allowing per-workload customization.
• Shared infrastructure: All storage in the cluster flows through a single Ceph cluster, simplifying backup, disaster recovery, and capacity planning.

Tiering by device type is not supported: no renderer sets a deviceClass or a per-pool CRUSH rule, so a pool of SSD disks and a pool of HDD disks feed one OSD set that every volume spans; see Pools share one OSD set.

Replication strategy

BlockStorage, FileStorage and ObjectStorage each carry an optional replicas field that controls how many copies Ceph maintains (for FileStorage and ObjectStorage, both the metadata and data pool replicate at the same size):

• absent (default): derives the replica count from the number of nodes contributing disks (via DiskPool), capped at 3 — min(3, nodes). Two nodes give 2 replicas, five nodes still give 3.

• 1, 2, 3: explicit replica count, e.g. to sacrifice redundancy for capacity on a test cluster.

An explicit count is clamped down to the number of nodes contributing disks to the cluster rather than leaving the object unsatisfiable: asking for 3 replicas on a 2-node cluster yields 2, and the reason is recorded in status.message. The BlockStorage/FileStorage provisions either way; it does not sit degraded waiting for disks that may never arrive — unless no DiskPool contributes any disks at all, in which case it goes Degraded with “no DiskPool contributes disks”.

The node count is the cluster’s, not any single object’s — see Pools share one OSD set.

The failure domain follows from the result: host when replication ends up with 2 or more replicas across 2 or more nodes, otherwise osd. A single-node cluster therefore still provisions — with osd domain and, at 1 replica, requireSafeReplicaSize: false, which Ceph needs to accept a size-1 pool at all.

The resulting replica count, failure domain and any clamping message are recorded in the consumer’s status.

Status fields

A consumer kind’s status describes its derived StorageClass, not live Ceph state:

• storageClassName — the derived StorageClass’s name (ceph-<name>, cephfs-<name> or cephobj-<name>); what a PersistentVolumeClaim (or, for ObjectStorage, an ObjectBucketClaim) should reference.
• replicas, failureDomain — the result of replication sizing.
• phase — Provisioning until the backing Rook object reports Ready; Degraded when it cannot be reconciled without operator action, including when no DiskPool contributes any disks.

A DiskPool’s status is different: it describes only its own disk contribution, not any derived object, and carries no replication of its own:

• selectedDiskCount — how many of spec.disks resolved to a usable Disk. Not the number of OSDs Ceph is running; Rook creates those asynchronously.
• rawCapacity — the summed size of those disks. This is not usable capacity: see Pools share one OSD set. Use ceph df for real free space.
• phase — Ready once it resolves at least one disk; Degraded when it resolves none. DiskPool has no Provisioning phase.

Pools share one OSD set

Every DiskPool feeds the same CephCluster, and the Rook object a consumer derives carries no CRUSH rule confining it to any one DiskPool’s disks. Ceph places a volume’s data across every OSD in the cluster.

Three consequences, all of them load-bearing:

• Multiple pools do not isolate or tier storage. A second consumer object gives you a second StorageClass over the same disks. Real separation needs a device class plus a per-pool CRUSH rule, which this version does not implement.
• rawCapacity / replicas is not usable capacity. With more than one DiskPool it is not even an upper bound.
• Replication is sized against the cluster, not any one pool. auto uses the number of nodes contributing disks cluster-wide. This is deliberate: sizing off a single DiskPool’s disks would cap that pool’s consumers at replicas: 1 — waiving Ceph’s size-1 safety check to advertise no redundancy — while Ceph replicated that data across all hosts regardless.

Removing disks

Taking a disk out of a DiskPool’s spec.disks removes it from the CephCluster’s device list, but does not remove the OSD: Rook needs an explicit purge job for that. Deleting a DiskPool behaves the same way. Treat disk removal as a two-step operation — update the pool, then purge the OSD through Ceph.

FileStorage (CephFS)

FileStorage rides the same shared OSD set as BlockStorage — no separate disks, no separate DiskPool. Where BlockStorage derives an RBD StorageClass (ReadWriteOnce), FileStorage derives a CephFS one (ReadWriteMany), prefixed cephfs-<name> rather than ceph-<name>.

Its CephFilesystem runs spec.metadataServers active MDS daemons, each paired with a standby (activeStandby: true) so a crashed MDS fails over without an operator manually promoting one. preserveFilesystemOnDelete is always true: deleting the FileStorage CR (or its owner) tears down the CephFilesystem object, but never the underlying filesystem data.

ObjectStorage (RGW)

ObjectStorage rides the same shared OSD set and derives a cephobj-<name> pair: a CephObjectStore running spec.gatewayInstances RGW pods behind the rook-ceph-rgw-cephobj-<name> Service (port 80; TLS is a gateway concern), and a StorageClass that drives Rook’s ObjectBucketClaim provisioner (<cluster namespace>.ceph.rook.io/bucket — it follows the CephCluster’s namespace, not the Rook operator’s) rather than a CSI driver.

Workloads consume it by creating an ObjectBucketClaim that references the class — by hand, or through the Buckets section on the ObjectStorage detail page (list, create and inspect claims for that store, with optional per-claim quotas and an exact-name collision pre-check): Rook provisions the bucket and writes the S3 endpoint to a ConfigMap and the credentials to a Secret, both named after the claim. A claim can cap its own bucket via spec.additionalConfig (maxSize, maxObjects). Buckets advertise the signing-only region label from the S3_REGION install config (default nl-1; RGW enforces no region). preservePoolsOnDelete is always true, same stance as FileStorage: deleting an ObjectStorage keeps its seven cephobj-<name>.rgw.* pools and their data. Reclaiming the space is a deliberate second step in the Ceph toolbox: ceph osd pool ls | grep cephobj-<name>, then ceph osd pool rm per pool (requires mon_allow_pool_delete=true).

The S3 endpoint is in-cluster only. External exposure (a Gateway/HTTPRoute via the gateway-api plugin) and standalone S3 users (CephObjectStoreUser) are deliberate non-goals of this version.

Why it needs cluster-admin-equivalent RBAC

The Rook-Ceph operator requires broad, cluster-admin-level permissions to:

• Discover and manage block devices at the node/host level (privileged kernel access)
• Create and update Kubernetes resources across all namespaces (StorageClass, PersistentVolume, etc.)
• Install its own cluster-wide RBAC (ClusterRoles/Bindings with escalate+bind) and the Ceph CSI driver

The plugin declares this in its definition.yaml under spec.permissions.rbac (a cluster-admin-equivalent wildcard rule). The plugin-controller copies those rules verbatim into a ClusterRole bound to the plugin’s ServiceAccount. Nothing narrows them: the plugin pod runs with cluster-admin on the tenant cluster. FUN-17’s SubjectAccessReview does not apply here — it is a per-request check in kube-api-proxy on the console iframe path, and the plugin pod uses its in-cluster config, which never traverses it. There is no separate clusterRoles field on the PluginInstallation — it references a published definition:

# Example PluginInstallation (references a published plugin version; see FUN-19)
# metadata.name must be "<organizationName>--<pluginName>", and pluginVersion is
# the registry's form without the "v" prefix.
apiVersion: plugins.fundament.io/v1
kind: PluginInstallation
metadata:
  name: system--ceph-rook
spec:
  definitionRef:
    organizationName: system
    pluginName: ceph-rook
    pluginVersion: "0.3.0"
    definitionHash: sha256:<hash printed by `just plugins publish storage/ceph-rook`>

Host prerequisites

For Rook to discover and claim disks, cluster nodes must meet these requirements:

1. Raw, unpartitioned block devices: Rook will only claim empty disks with no filesystems or partition tables. If a disk is already in use or partitioned, it will be marked unavailable in the Disk CR.

2. RBD kernel module: The rbd module must be available on nodes. This is typically included in standard Linux distributions but may need to be loaded explicitly on some systems:

   modprobe rbd

3. LVM2: The lvm2 package must be installed on nodes for OSD device management:

   # Ubuntu/Debian
   apt-get install lvm2
   
   # CentOS/RHEL
   yum install lvm2

Ensure these prerequisites are in place on all nodes before creating a DiskPool; otherwise, disks may remain unavailable or OSDs may fail to initialize.

Plugin lifecycle

  Container starts
       │
       ▼
  pluginruntime.Run()
       │
       ├─ HTTP server on :8080
       │
       ▼
  Start()
       │
       ├─ Install Rook-Ceph Helm chart
       ├─ Bootstrap singleton CephCluster CR
       ├─ Create k8s client
       ├─ Register DiskInventoryReconciler, DiskPoolReconciler and the
       │  BlockStorage/FileStorage/ObjectStorage consumer reconcilers
       ├─ Start controller-runtime manager
       ├─ ReportReady()
       ├─ ReportStatus("running", "rook-ceph storage plugin running")
       └─ Block until SIGTERM
              │
              ▼
  Reconcile loops (event-driven)
       │
       ├─ DiskInventoryReconciler: reacts to rook-discover ConfigMaps
       │   │                       AND to DiskPool changes
       │   ├─ publish/update Disk CRs from the probed devices
       │   ├─ recompute claimedBy from the current pools
       │   └─ mark vanished disks unavailable (soft delete; never Delete)
       │
       ├─ DiskPoolReconciler: reacts to DiskPool/Disk changes
       │   ├─ On delete: recompute the CephCluster union without this pool
       │   └─ Otherwise, for the pool:
       │       ├─ Resolve spec.disks, skipping missing and contested disks
       │       ├─ Fold every live pool's disks into CephCluster OSDs (deduped)
       │       └─ Update status (phase, selected disks, raw capacity)
       │
       └─ Consumer reconcilers: react to their own kind plus Disk/DiskPool
           changes (either can change the node count and therefore replication)
           ├─ No OSDs in the shared union -> Degraded, no derived objects made
           ├─ Create/update CephBlockPool ceph-<name>      (BlockStorage)
           │             or CephFilesystem cephfs-<name>   (FileStorage)
           │             or CephObjectStore cephobj-<name> (ObjectStorage)
           │             — refuse to adopt any if not ours
           ├─ Create/update StorageClass  ceph-<name> / cephfs-<name> / cephobj-<name>
           │             — refuse to adopt if not ours
           ├─ Update status (phase, storageClassName, replicas, failure domain)
           └─ Re-check every 30 s while phase is Provisioning
              (until the derived Rook object reports Ready)

A reconcile that fails writes phase: Degraded with the cause in status.message before returning the error, so the reason shows up in the console rather than only in the plugin’s logs.