Runbook: verifying the Ceph/Rook plugin

End-to-end verification of the ceph-rook plugin against a local k3d sandbox, from an empty machine to a pod writing to a Ceph-backed volume.

The unit tests cover the plugin’s decision logic and its reconcilers against a fake client. They cannot tell you whether Rook actually builds an OSD out of a real block device, whether the CSI driver can map an RBD image, or whether the console pages render. That is what this runbook is for.

Budget 45–60 minutes for a first run; most of it is Ceph pulling images and waiting for OSDs.

What each phase proves

Phase	Proves
1 · Environment	The node has real block devices and a live /dev
2 · Baseline	Rook + Ceph work here without any Fundament code
3 · Publish	The image builds and the definition reaches organization-api
4 · Install	The plugin’s install path runs: Helm, CRDs, CephCluster bootstrap
5 · Discovery	The DiskInventory reconciler turns real probes into Disk CRs
6 · Pool	The DiskPool reconciler produces OSDs; BlockStorage/FileStorage/ObjectStorage each produce a StorageClass
7 · PVC	The whole chain actually stores data
8 · Regressions	The specific behaviours fixed in review hold on a real cluster

Phase 2 is the one people skip and shouldn’t. If it fails, the environment is broken and nothing you learn in phases 4–8 means anything.

Before you start

• Docker host must be a VM. storage-disks.sh refuses on bare metal, because a stuck RBD mapping is an unkillable kernel thread and the privileged node enumerates your real drives. On macOS the Docker host is already a VM. See Block devices for k3d.
• At least 8 GiB of VM memory. Ceph needs roughly 2 GiB per OSD plus its daemons. On colima: colima start --cpus 4 --memory 8. Memory cannot be changed on a running VM.
• The management cluster (k3d-fundament) must be up, because publishing goes through organization-api. The sandbox cluster (k3d-fundament-plugin) is where the plugin runs.

Check your kubectl context

just dev and skaffold deploy to the current context, and they do not switch for you. Every command below passes --context explicitly for that reason. If you drop the flag, confirm your context first.

Phase 0 · Clean slate

Two things commonly survive from an earlier session and quietly poison a run: a sandbox cluster created without the block-device binds, and a leftover rook-ceph namespace.

# Does the sandbox node have the binds? Empty output means no.
docker inspect k3d-fundament-plugin-server-0 \
  --format '{{range .Mounts}}{{.Destination}}{{"\n"}}{{end}}' | grep -E '^/dev$|^/run/udev$'

# Is there leftover Rook state?
kubectl --context k3d-fundament-plugin -n rook-ceph get cephcluster,cephblockpool,pod

Docker fixes a container’s mounts at creation, so a cluster without the binds cannot be fixed in place — it has to be recreated. Loop devices in the node’s /dev do not prove otherwise: without the bind that directory is a private tmpfs snapshot, and the entries in it are stale artifacts.

If a previous Ceph cluster is still running, take it down in Rook’s order before deleting the k3d cluster, so its finalizers get cleared:

deploy/k3d/rook-smoke.sh down          # if the leftovers came from the smoke script
just plugins uninstall system--ceph-rook       # if they came from the plugin
just plugins cluster-delete

cluster-delete drains Ceph consumers first, so it is safe even mid-experiment.

Phase 1 · Environment

just plugins cluster-create-storage    # binds /dev and /run/udev into the node
just plugins storage-disks doctor      # inspect the kernel Docker actually runs on
just plugins storage-disks attach      # 3 × 20 GiB sparse images -> /dev/loop0p1 .. /dev/loop2p1

doctor is worth reading rather than skimming. The line that matters most:

  rbd      PRESENT (loaded or built-in)

If rbd is MISSING, csi-rbdplugin will CrashLoopBackOff and no RBD PVC will ever mount. It fatals at startup, before it reads a StorageClass, so mounter: rbd-nbd does not rescue it. Ceph itself still runs — mons, mgr and OSDs do not use the module — so phases 4–6 remain testable and only phase 7 is blocked. On macOS, colima provides it.

Confirm the devices reached the node:

just plugins storage-disks status

Expect /dev/loop0p1, /dev/loop1p1, /dev/loop2p1 both on the host and in the node. Three is the useful floor — it is what lets a pool hold 3 replicas across OSDs.

Recreating the sandbox cluster invalidates the management cluster’s wiring

plugin-proxy and kube-api-proxy reach this cluster through the plugin-sandbox-kubeconfig Secret, and the user-half SubjectAccessReview depends on a ClusterRoleBinding inside it — both set up by just plugin-sandbox-kubeconfig, both gone or stale after a cluster-delete/cluster-create-storage. Re-run it now (it is idempotent and restarts both proxies). Skipping it surfaces later as plugin installation not found when the console mints a plugin token, or as 403 denied:user-sar on every plugin API request.

After a reboot

The backing files survive reboots; the loop devices and loaded kernel modules do not, because those are kernel state. Recovery is just plugins storage-disks attach plus a doctor run, which reloads rbd as a side effect — without it, csi-rbdplugin CrashLoopBackOffs on modprobe: FATAL: Module rbd not found (it cannot load modules from inside the node).

Phase 2 · Baseline without the plugin

deploy/k3d/rook-smoke.sh up      # upstream chart + a hand-written CephCluster
deploy/k3d/rook-smoke.sh status  # expect HEALTH_OK (or HEALTH_WARN about redundancy)
deploy/k3d/rook-smoke.sh test    # 1Gi PVC + a pod that writes and verifies -> SMOKE-OK

rook-smoke.sh contains no Fundament code, so it cleanly separates “the environment is broken” from “the plugin is broken”. SMOKE-OK means disks, OSDs, the CSI driver and the kernel are all fine, and any later failure is the plugin’s.

CephFS is skipped on kernels without the ceph module

Mapping a block device needs the rbd module; mounting CephFS needs ceph. A kernel can ship one without the other — OrbStack has rbd but not ceph — so the test drops CephFS and says so rather than failing:

/!\ no 'ceph' kernel module on the Docker host: CephFS cannot mount here.
    Testing RBD only. Block storage -- all the ceph-rook plugin provides -- is
    unaffected, so this is not a failure of the environment.

This plugin is block-only, so RBD is the whole story. just plugins storage-disks has-module ceph answers the question on its own.

One FailedMount on first attach is normal

ceph-csi maps with --options noudev, so rbd map does not wait for udev to create the device node and the first attempt can lose the race:

rbd: mapping succeeded but /dev/rbd0 is not accessible, is host /dev mounted?

kubelet retries and it succeeds. Only treat it as real if it repeats — a genuinely missing /dev bind fails every time. Confirm with docker exec k3d-fundament-plugin-server-0 ls -l /dev/rbd0.

Then free the namespace — the plugin wants the same one:

deploy/k3d/rook-smoke.sh down
just plugins storage-disks reset    # wipe stale OSD metadata, reattach

reset is not optional between installs. Stale BlueStore metadata on the images, or Rook state under dataDirHostPath, is the usual reason a second install fails: the OSD prepare job completes in seconds with skipping osd.N: ... belonging to a different ceph cluster, no OSDs appear, and every consumer eventually degrades (an ObjectStorage shows it as CephObjectStore ... reports Failure because radosgw-admin hangs without OSDs).

Do not substitute a hand wipe for reset. Ceph v19 keeps backup BlueStore labels at the 1 GiB and 10 GiB offsets, so zeroing the start of a partition (or wipefs) leaves an OSD that ceph-volume still detects. reset deletes the backing files outright, which is immune — and it is also why it must not run against a live cluster: it removes /var/lib/rook, the running mons’ store.

Phase 3 · Build and publish

Publishing resolves the plugin’s catalog id by name, so the appstore seed must be applied in the management cluster. This PR adds the ceph-rook catalog entry and a Storage category, so the seed must be re-applied — it is idempotent (ON CONFLICT DO UPDATE), and the db-migrations Job runs it:

kubectl config use-context k3d-fundament
just dev                            # or: just deploy — re-runs db-migrations

Bridge the sandbox controller to marketplace-catalog-api (re-run after recreating either cluster, since it resolves the node IP fresh):

just plugins sandbox-catalog

Now build, push and publish:

Publish as platform-admin@fundament.io, not your usual dev login

CreatePluginVersion requires can_edit on the plugin, which resolves to admin of the owning organization. First-party plugins are owned by the seeded system org, and alice@acme-corp.com and friends are not members of it — publishing as them fails authorization.

db/seed/0100-system-org.sql says the system org “has no members”, and in production that is true. Local dev is the exception: db/testdata/033_0101-content.sql adds platform-admin@fundament.io (…1000-…008) as its admin precisely so first-party definitions can be published, and values-local.yaml gives it a dex login with the same shared password as the other static users. It is applied only when TREK_INSERT_TEST_DATA=true.

Because testdata is keyed to migration 033, a database that was already past 033 when that change landed never receives the row, and trek apply will not backfill it. Check before you start:

kubectl --context k3d-fundament -n fundament exec pod/db-1 -c postgres -- \
  psql -U postgres -d fundament -c "SELECT u.email, ou.permission, ou.status
    FROM tenant.users u JOIN tenant.organizations_users ou ON ou.user_id = u.id
    WHERE ou.organization_id = '019b4000-0000-7000-8000-000000000000';"

One admin/accepted row is what you need. If it is missing, re-run the migrations with db.reset: true, or insert db/testdata/033_0101-content.sql by hand.

export PLUGIN_REGISTRY=localhost:5112

# Publishing goes through `functl plugin publish` (marketplace-registry-api).
# One-time setup: authenticate and select the publishing organization.
functl auth login
functl org set 019b4000-0000-7000-8000-000000000000   # seeded "system" org

just plugins publish storage/ceph-rook --create

The registry endpoint comes from functl’s config (registry_url, or FUNCTL_REGISTRY_URL, which mise sets for this repo). --create reserves the listing on the first publish; versions are create-only, so republishing means bumping metadata.version in the definition.

Authorization also depends on two OpenFGA tuples, written by the authz-worker from the database outbox. If publishing fails with a permission error while the SQL above looks right, confirm the sync landed:

kubectl --context k3d-fundament -n fundament exec pod/db-1 -c postgres -- psql -U postgres -d openfga \
  -c "SELECT object_type, object_id, relation, _user FROM tuple
      WHERE (object_type='plugin' AND object_id='019b4000-3000-7000-8000-000000000011')
         OR (object_type='organization' AND object_id='019b4000-0000-7000-8000-000000000000');"

Expect plugin:…011 owner organization:…000 and organization:…000 admin user:…008.

Expected last line — copy the hash, phase 4 needs it:

published plugin=ceph-rook version=0.3.0 hash=sha256:... id=... version_id=... status=SUBMISSION_STATUS_DRAFT

The registry stores versions without the v prefix, so metadata.version: v0.3.0 publishes as 0.3.0, which is what a PluginInstallation must reference.

Versions are create-only on the registry — an approved version’s hash is a consent record — so there is no --replace. To publish the same content again, bump metadata.version in definition.yaml (a -dev prerelease suffix works) and re-run the publish.

If it fails with no listing named "ceph-rook", pass --create to reserve the listing, seeded from the definition’s metadata.

Phase 4 · Install

The plugin needs local-development config: without it the default 3 mons never reach quorum on a single node, and the real-disk filter would ignore the loop devices.

Three naming rules, all enforced at apply or reconcile time: metadata.name must be <organizationName>--<pluginName> (system--ceph-rook: first-party plugins are published by the seeded system org), organizationName is required, and pluginVersion is the registry’s form without the v prefix (0.3.0, as printed by the publish in phase 3).

Once the appstore install form ships (configSchema support), this config can be entered in the Console at install time instead of hand-writing the CR below.

kubectl --context k3d-fundament-plugin apply -f - <<'YAML'
apiVersion: plugins.fundament.io/v1
kind: PluginInstallation
metadata:
  name: system--ceph-rook
spec:
  definitionRef:
    organizationName: system
    pluginName: ceph-rook
    pluginVersion: "0.3.0"
    definitionHash: sha256:PASTE_THE_HASH_FROM_PHASE_3
  config:
    DEV_LOOP_DEVICES: "true"        # discover ONLY /dev/loopNpN; ignore the host's real disks
    MON_COUNT: "1"
    MGR_COUNT: "1"
    ALLOW_MULTIPLE_PER_NODE: "true"
    CEPHFS_MOUNTER: "fuse"          # see below — required to mount CephFS on a module-less local VM
YAML

CEPHFS_MOUNTER: "fuse" makes the CephFS StorageClass mount via ceph-fuse instead of the kernel client. Local Docker VMs (OrbStack, colima, Docker Desktop) ship a kernel without the ceph module, so a kernel mount fails with modprobe ceph errors and every CephFS PVC hangs in ContainerCreating; /dev/fuse is present on all of them, so ceph-fuse works. It only affects FileStorage — RBD/BlockStorage is unaffected. Leave it unset on a real cluster (or node with the module), where the kernel client is faster. Confirm your VM’s situation with docker run --rm --privileged alpine sh -c 'modprobe ceph && echo OK || echo MISSING'.

CEPH_IMAGE already defaults to the build rook-smoke.sh validates, so leave it unless you are testing a different one. ALLOW_UNSUPPORTED_CEPH defaults to false and should stay there: it turns off Rook’s check that it knows how to drive the Ceph release it is given, and Rook v1.21 supports the default image’s Squid (v19) line. Set it only when pairing the plugin with a Ceph build outside that table.

DEV_LOOP_DEVICES is a safety control, not a convenience

A k3d node is privileged and enumerates the Docker host’s real disks, and they do reach the discovery ConfigMap. ParseDiscoveredDevices is the only thing keeping them out of an OSD. With this flag the filter is replaced by loop-partitions-only, not extended.

Watch it come up:

just plugins status
just plugins logs system--ceph-rook

Expect PHASE=Running, READY=true, and a log line rook-ceph storage plugin running. The first install pulls the Ceph image and takes several minutes.

kubectl --context k3d-fundament-plugin -n rook-ceph get pods
kubectl --context k3d-fundament-plugin get crd | grep ceph.fundament.io
kubectl --context k3d-fundament-plugin -n rook-ceph get cephcluster

Expect the operator, the discover DaemonSet, the CSI pods, both CRDs, and a rook-ceph CephCluster whose spec.storage.nodes is still empty — nothing is consumed until an operator opts disks in.

Phase 5 · Disk discovery

kubectl --context k3d-fundament-plugin get disks

Expect exactly three, one per loop partition:

NAME                                  NODE                            PATH           SIZE      AVAILABLE
k3d-fundament-plugin-server-0-1a2b3c  k3d-fundament-plugin-server-0   /dev/loop0p1   20478Mi   true
...

Two things to check, both of which have bitten this plugin:

# 1. No real host disks leaked in. Every path must be /dev/loopNpN.
kubectl --context k3d-fundament-plugin get disks \
  -o jsonpath='{range .items[*]}{.status.path}{"\n"}{end}'

# 2. Nothing is claimed yet.
kubectl --context k3d-fundament-plugin get disks \
  -o jsonpath='{range .items[*]}{.metadata.name}{" claimedBy="}{.status.claimedBy}{"\n"}{end}'

If the disk list is empty, check the discovery ConfigMaps the reconciler reads:

kubectl --context k3d-fundament-plugin -n rook-ceph get cm -l app=rook-discover
kubectl --context k3d-fundament-plugin -n rook-ceph get cm local-device-k3d-fundament-plugin-server-0 \
  -o jsonpath='{.data.devices}' | jq .

An empty devices list means the discover daemon saw nothing — go back to phase 1. Devices present but no Disk CRs means the filter rejected them: with DEV_LOOP_DEVICES: "true" only entries with "type": "part" on a /dev/loopNpN path survive.

Phase 6 · DiskPool → BlockStorage/FileStorage/ObjectStorage → StorageClass

Disk names are cluster-specific (node name + a hash of the device’s stable identity), so they cannot be hard-coded. Build the pool from whatever was discovered:

DISKS=$(kubectl --context k3d-fundament-plugin get disks -o jsonpath='{range .items[*]}    - name: {.metadata.name}{"\n"}{end}')
kubectl --context k3d-fundament-plugin apply -f - <<YAML
apiVersion: ceph.fundament.io/v1alpha1
kind: DiskPool
metadata:
  name: test-pool
spec:
  disks:
$DISKS
YAML

A DiskPool only contributes disks; it derives no StorageClass on its own. Create a BlockStorage (RWO, RBD), a FileStorage (RWX, CephFS) and an ObjectStorage (S3, RGW) to turn that capacity into StorageClasses — all three ride the OSD set the pool above just contributed to:

kubectl --context k3d-fundament-plugin apply -f - <<'YAML'
apiVersion: ceph.fundament.io/v1alpha1
kind: BlockStorage
metadata:
  name: test-pool
spec: {}
---
apiVersion: ceph.fundament.io/v1alpha1
kind: FileStorage
metadata:
  name: test-pool
spec:
  metadataServers: 1
---
apiVersion: ceph.fundament.io/v1alpha1
kind: ObjectStorage
metadata:
  name: test-pool
spec:
  gatewayInstances: 1
YAML

The consumer kinds are named test-pool here too — a different kind, so no collision with the DiskPool above — which is why every ceph-test-pool reference below still holds. Name it something else and the derived StorageClass follows: see Derived object names.

Watch them provision — the first OSD takes a few minutes:

kubectl --context k3d-fundament-plugin get diskpool test-pool -w
kubectl --context k3d-fundament-plugin get blockstorage test-pool -w

kubectl --context k3d-fundament-plugin get diskpool test-pool -o yaml | yq .status
kubectl --context k3d-fundament-plugin get blockstorage test-pool -o yaml | yq .status

Expected, on this single-node cluster. The DiskPool reports only its disk contribution:

phase: Ready
selectedDiskCount: 3
rawCapacity: 61434Mi                 # this pool's contribution, before replication

The BlockStorage reports the derived StorageClass and the replica count it resolved:

phase: Ready
storageClassName: ceph-test-pool     # note the ceph- prefix
replicas: 1                          # no spec.replicas = min(3, cluster nodes with disks); one node -> 1
failureDomain: osd                   # host domain needs >=2 replicas across >=2 nodes

rawCapacity is the raw size of the disks the pool contributes, not anyone’s capacity — all consumers share one OSD set and Ceph places data across every OSD in it. Ask Ceph for real free space (ceph df in the toolbox). A BlockStorage/FileStorage that finds no DiskPool contributing disks reports Degraded with "no DiskPool contributes disks" in status.message, and creates no StorageClass.

The plugin owns the CephCluster’s continueUpgradeAfterChecksEvenIfNotHealthy: it is true while all contributed disks sit on one node, where pools keep one replica and Ceph never reports an OSD ok-to-stop, and false otherwise. A value set by hand is reverted at once: a CephCluster spec change, like a Disk or DiskPool event, reconciles the pools. Two consequences:

• A consumer with replicas: 1 on two or more nodes still stalls the CephCluster in Configuring Ceph OSDs after its first reconcile. To let that reconcile through, set skipUpgradeChecks: true on the CephCluster by hand (the plugin leaves it alone) and remove it once the CephCluster is Ready.
• On one node, a Ceph version upgrade may restart the MDS daemons back to back: a short CephFS outage.

selectedDiskCount is how many of the DiskPool’s spec.disks resolved, not how many OSDs are running. Check the OSDs separately:

kubectl --context k3d-fundament-plugin -n rook-ceph get pods -l app=rook-ceph-osd
kubectl --context k3d-fundament-plugin -n rook-ceph get cephcluster rook-ceph \
  -o jsonpath='{.spec.storage.nodes}' | jq .

Expect three OSD pods and all three devices listed under the node. Then confirm the derived objects and their ownership:

kubectl --context k3d-fundament-plugin get storageclass ceph-test-pool
kubectl --context k3d-fundament-plugin -n rook-ceph get cephblockpool ceph-test-pool
kubectl --context k3d-fundament-plugin get storageclass ceph-test-pool \
  -o jsonpath='{.metadata.ownerReferences}' | jq .

The owner reference must name test-pool with kind: BlockStorage and controller: true — that is what makes deletion cascade.

FileStorage behaves the same way, over the same OSDs, producing a cephfs-test-pool StorageClass and CephFilesystem with one active and one standby MDS:

kubectl --context k3d-fundament-plugin get filestorage test-pool -o yaml | yq .status
kubectl --context k3d-fundament-plugin get storageclass cephfs-test-pool
kubectl --context k3d-fundament-plugin -n rook-ceph get cephfilesystem cephfs-test-pool

ObjectStorage differs only in what the derived StorageClass is for: it drives Rook’s ObjectBucketClaim provisioner (rook-ceph.ceph.rook.io/bucket), not a CSI driver, so nothing mounts it — Phase 7 claims a bucket instead. Verify the cephobj-test-pool pair and the RGW pod:

kubectl --context k3d-fundament-plugin get objectstorage test-pool -o yaml | yq .status
kubectl --context k3d-fundament-plugin get storageclass cephobj-test-pool
kubectl --context k3d-fundament-plugin -n rook-ceph get cephobjectstore cephobj-test-pool
kubectl --context k3d-fundament-plugin -n rook-ceph get pods -l app=rook-ceph-rgw

Phase 7 exercises all three paths. CephFS mounting needs either the ceph kernel module or — on a module-less local VM — CEPHFS_MOUNTER: "fuse" in the install config (Phase 4); without one of those a CephFS PVC hangs in ContainerCreating with modprobe ceph errors. RBD is unaffected either way.

Phase 7 · Bind a PVC

This is the only step that proves data actually reaches a disk.

kubectl --context k3d-fundament-plugin apply -f - <<'YAML'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ceph-rook-verify
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: ceph-test-pool
  resources: { requests: { storage: 1Gi } }
---
apiVersion: v1
kind: Pod
metadata:
  name: ceph-rook-verify
spec:
  restartPolicy: Never
  containers:
    - name: w
      image: alpine:3.21
      command: ["sh","-c"]
      args:
        - |
          dd if=/dev/urandom of=/data/blob bs=1M count=64 2>/dev/null
          sha256sum /data/blob | cut -d' ' -f1 > /data/sum
          sync
          want=$(cat /data/sum); have=$(sha256sum /data/blob | cut -d' ' -f1)
          [ "$want" = "$have" ] || { echo MISMATCH; exit 1; }
          df -h /data | tail -1
          echo PLUGIN-VERIFY-OK
      volumeMounts: [{ name: v, mountPath: /data }]
  volumes:
    - { name: v, persistentVolumeClaim: { claimName: ceph-rook-verify } }
YAML

kubectl --context k3d-fundament-plugin wait --for=jsonpath='{.status.phase}'=Succeeded \
  pod/ceph-rook-verify --timeout=5m
kubectl --context k3d-fundament-plugin logs pod/ceph-rook-verify

PLUGIN-VERIFY-OK is the end-to-end pass for RBD. If the PVC stays Pending:

kubectl --context k3d-fundament-plugin describe pvc ceph-rook-verify | tail -20
kubectl --context k3d-fundament-plugin -n rook-ceph logs -l app=rook-ceph.rbd.csi.ceph.com-ctrlplugin -c csi-provisioner --tail=50

Now CephFS, whose whole point is ReadWriteMany — so the test is two pods sharing one volume: a writer creates a file, a reader that never wrote it reads the same file back.

kubectl --context k3d-fundament-plugin apply -f - <<'YAML'
apiVersion: v1
kind: PersistentVolumeClaim
metadata: { name: cephfs-verify }
spec:
  accessModes: [ReadWriteMany]
  storageClassName: cephfs-test-pool
  resources: { requests: { storage: 1Gi } }
---
apiVersion: v1
kind: Pod
metadata: { name: cephfs-writer }
spec:
  restartPolicy: Never
  containers:
    - name: w
      image: alpine:3.21
      command: ["sh","-c","echo PLUGIN-CEPHFS-OK > /data/shared; sync; sleep 5; cat /data/shared"]
      volumeMounts: [{ name: v, mountPath: /data }]
  volumes: [{ name: v, persistentVolumeClaim: { claimName: cephfs-verify } }]
---
apiVersion: v1
kind: Pod
metadata: { name: cephfs-reader }
spec:
  restartPolicy: Never
  containers:
    - name: r
      image: alpine:3.21
      command: ["sh","-c","for i in $(seq 90); do [ -s /data/shared ] && cat /data/shared && exit 0; sleep 1; done; echo TIMEOUT; exit 1"]
      volumeMounts: [{ name: v, mountPath: /data }]
  volumes: [{ name: v, persistentVolumeClaim: { claimName: cephfs-verify } }]
YAML

kubectl --context k3d-fundament-plugin wait --for=jsonpath='{.status.phase}'=Succeeded \
  pod/cephfs-reader --timeout=5m
kubectl --context k3d-fundament-plugin logs pod/cephfs-reader   # PLUGIN-CEPHFS-OK

PLUGIN-CEPHFS-OK from the reader (which never wrote the file) proves RWX shared access. If both pods sit in ContainerCreating and the CSI log shows modprobe args: [ceph], the mount is trying the kernel client — set CEPHFS_MOUNTER: "fuse" (Phase 4) and reinstall:

kubectl --context k3d-fundament-plugin -n rook-ceph logs -l app=rook-ceph.cephfs.csi.ceph.com-nodeplugin -c csi-cephfsplugin --tail=30

Finally object storage, which is consumed through an ObjectBucketClaim rather than a PVC: Rook provisions the bucket and writes the endpoint to a ConfigMap and the credentials to a Secret, both named after the claim. The pod below round-trips an object through the S3 API with them:

kubectl --context k3d-fundament-plugin apply -f - <<'YAML'
apiVersion: objectbucket.io/v1alpha1
kind: ObjectBucketClaim
metadata: { name: s3-verify }
spec:
  generateBucketName: s3-verify
  storageClassName: cephobj-test-pool
YAML

kubectl --context k3d-fundament-plugin wait --for=jsonpath='{.status.phase}'=Bound \
  objectbucketclaim/s3-verify --timeout=5m

kubectl --context k3d-fundament-plugin apply -f - <<'YAML'
apiVersion: v1
kind: Pod
metadata: { name: s3-verify }
spec:
  restartPolicy: Never
  containers:
    - name: s3
      image: amazon/aws-cli:2.22.35
      command: ["sh","-c"]
      args:
        - |
          echo PLUGIN-S3-OK > /tmp/blob
          aws --endpoint-url "http://$BUCKET_HOST:$BUCKET_PORT" s3 cp /tmp/blob "s3://$BUCKET_NAME/blob"
          aws --endpoint-url "http://$BUCKET_HOST:$BUCKET_PORT" s3 cp "s3://$BUCKET_NAME/blob" -
      envFrom:
        - configMapRef: { name: s3-verify }
        - secretRef: { name: s3-verify }
YAML

kubectl --context k3d-fundament-plugin wait --for=jsonpath='{.status.phase}'=Succeeded \
  pod/s3-verify --timeout=5m
kubectl --context k3d-fundament-plugin logs pod/s3-verify   # PLUGIN-S3-OK

PLUGIN-S3-OK read back through the gateway proves bucket provisioning, credentials and the RGW data path. The ConfigMap carries BUCKET_HOST/BUCKET_PORT/BUCKET_NAME; the Secret carries AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY.

Clean up before phase 8, so the pool can be deleted later:

kubectl --context k3d-fundament-plugin delete pod/ceph-rook-verify pvc/ceph-rook-verify
kubectl --context k3d-fundament-plugin delete pod/cephfs-writer pod/cephfs-reader pvc/cephfs-verify
kubectl --context k3d-fundament-plugin delete pod/s3-verify objectbucketclaim/s3-verify

Phase 8 · Regression checks

These target behaviours that were wrong before review and are covered by unit tests. Run them to confirm they hold against a real API server.

8a · The console is actually served

Previously the pages were embedded but never routed, and the iframe 404’d.

kubectl --context k3d-fundament-plugin -n plugin-system--ceph-rook \
  port-forward deploy/plugin 8080:8080 &
curl -sS -o /dev/null -w '%{http_code}\n' localhost:8080/console/diskpools-list.html
curl -sS -o /dev/null -w '%{http_code}\n' localhost:8080/console/diskpools-detail.html
kill %1

Both must return 200. Then open the console UI and confirm the pages render styled and that clicking a pool row navigates — the previous version’s inline styles and onclick were silently dropped by the plugin CSP:

https://console.fundament.localhost:8443/

Check the browser console for CSP violations. There should be none.

8a-2 · Pool/consumer editing and disk details

These exercise the console CRUD pages. Keep devtools open throughout — a CSP violation anywhere here is a failure, not cosmetic.

1. Open the BlockStorage → Edit → change Replicas → Save. The detail view returns and status.replicas reflects the new value.

2. Open the DiskPool → Edit → uncheck a disk. The OSD-retirement warning must be visible. Save, then confirm status.selectedDiskCount drops and the device leaves spec.storage.nodes:

   kubectl --context k3d-fundament-plugin -n rook-ceph get cephcluster rook-ceph \
     -o jsonpath='{.spec.storage.nodes}' | jq .

   The OSD pod for that device is still Running. That is correct, not a bug.

3. Edit → uncheck every disk → Save is refused with “Select at least one disk.”

4. Re-check the disk and Save to restore the pool before phase 7/8d rely on it again.

5. Note that none of DiskPool, BlockStorage, FileStorage or ObjectStorage offer delete from the console — see What the console can do. kubectl delete is the only way; phase 8d below uses it.

6. Open Disks → click a path. The detail page shows every field, and Claimed by is set for a pooled disk (plain text — cross-kind links are not expressible in the host contract).

8b · Foreign objects are not adopted

The old code would adopt a same-named StorageClass and delete it when the pool went away.

kubectl --context k3d-fundament-plugin apply -f - <<'YAML'
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ceph-squatter
provisioner: rancher.io/local-path
YAML

kubectl --context k3d-fundament-plugin apply -f - <<'YAML'
apiVersion: ceph.fundament.io/v1alpha1
kind: BlockStorage
metadata:
  name: squatter
spec: {}
YAML

kubectl --context k3d-fundament-plugin get blockstorage squatter -o jsonpath='{.status.phase}{"\n"}{.status.message}{"\n"}'

squatter resolves the OSDs test-pool already contributed, so the reconciler gets far enough to collide with ceph-squatter. Expect Degraded and a message naming the conflict. Critically, the foreign object must be untouched:

kubectl --context k3d-fundament-plugin get storageclass ceph-squatter \
  -o jsonpath='{.provisioner}{" owners="}{.metadata.ownerReferences}{"\n"}'
# rancher.io/local-path owners=      <- unchanged, no owner reference

kubectl --context k3d-fundament-plugin delete blockstorage squatter
kubectl --context k3d-fundament-plugin get storageclass ceph-squatter   # must still exist
kubectl --context k3d-fundament-plugin -n rook-ceph get cephblockpool ceph-squatter  # NotFound (it was owned, GC'd with the BlockStorage)
kubectl --context k3d-fundament-plugin delete storageclass ceph-squatter

8b-2 · Upgrading from a pre-split install

This is a pre-GA breaking change with no automated migration. A pre-split install used the kind StoragePool in the old storage.fundament.io group; a StoragePool created then had already derived a CephBlockPool and StorageClass named ceph-<poolname>. After upgrading, that StoragePool no longer reconciles either — they simply persist, orphaned, until the StoragePool itself is deleted, which garbage collects them via the owner reference. Creating a BlockStorage with the same name derives the same ceph-<name>, so it collides with the still-present foreign-looking objects and goes Degraded refusing adoption, exactly as in 8b above. To recover, either delete the stale CephBlockPool/StorageClass pair first or pick a different BlockStorage name.

Device contributions do not migrate either: the first DiskPool create or delete after the upgrade rewrites the CephCluster’s spec.storage.nodes from DiskPools alone, so devices a pre-split StoragePool contributed silently leave the spec. OSDs already provisioned on those devices keep running, but Rook stops managing and reprovisioning them. To keep them managed, create a DiskPool selecting the same disks (kubectl get disks lists the discovered names) before creating or deleting any other DiskPool.

Deleting a pre-split StoragePool deletes its Ceph pool

A pre-split StoragePool still owns its ceph-<name> CephBlockPool via the owner reference, so deleting the StoragePool garbage-collects the Ceph pool that holds live volumes. 8d’s “deleting a pool does not cascade” holds only for DiskPools created after the split. Delete the volumes (or accept losing them) before deleting a pre-split StoragePool.

8b-3 · Installing over another Rook version

The plugin installs only its pinned Rook, and refuses to install over another version: Rook upgrades one minor version at a time. It reads the version from the rook-ceph release or, with no release, from the rook-version label on the CephCluster’s daemons, and reports Degraded naming it.

Moving an existing cluster to the plugin’s version is a manual upgrade, minor by minor, along Rook’s upgrade guides. It is a security task, not housekeeping: a cluster on Rook v1.16 with Ceph v19.2.3 stays exposed to CVE-2025-30156 until it runs Rook v1.20.6 or later and Ceph v19.2.6 or later, and its daemon keys are rotated (security.cephx.daemon.keyRotationPolicy: KeyGeneration, keyGeneration: 2 on the CephCluster).

8c · claimedBy updates immediately

It used to wait for the discovery daemon’s next sweep (~60m), long enough to hand one disk to two pools through the UI.

kubectl --context k3d-fundament-plugin get disks \
  -o jsonpath='{range .items[*]}{.metadata.name}{" claimedBy="}{.status.claimedBy}{"\n"}{end}'

Every disk in test-pool must already show claimedBy=test-pool — no waiting. This is also what makes the create form’s picker correct, so re-open it and confirm it offers no disks.

8c-2 · A disk awaiting discovery is reported, not miscounted

A Disk is created in two API calls (create, then status), and the old code counted a disk caught in between as selected: a pool could report Ready with selectedDiskCount=N while contributing nothing to the CephCluster — enough to satisfy a kubectl wait --for=condition=Ready before any device was recorded. A Disk with an empty status reproduces that in-between state durably:

kubectl --context k3d-fundament-plugin apply -f - <<'YAML'
apiVersion: ceph.fundament.io/v1alpha1
kind: Disk
metadata:
  name: manual-pending
YAML

kubectl --context k3d-fundament-plugin patch diskpool test-pool --type=json \
  -p='[{"op":"add","path":"/spec/disks/-","value":"manual-pending"}]'

kubectl --context k3d-fundament-plugin get diskpool test-pool -o yaml | yq .status

Expect the pool still Ready, selectedDiskCount still 3 (the pending disk is not counted), and — the operator-visible part — the message naming it:

message: 'awaiting discovery: manual-pending'

The CephCluster’s spec.storage.nodes must be unchanged (the same check as in 8a-2). Restore the pool; manual-pending was appended last, so it is index 3:

kubectl --context k3d-fundament-plugin patch diskpool test-pool --type=json \
  -p='[{"op":"remove","path":"/spec/disks/3"}]'
kubectl --context k3d-fundament-plugin delete disk manual-pending

8c-3 · Rook’s spec defaults survive reconciliation

The plugin renders only a few spec fields on the derived CephBlockPool/ CephFilesystem; the API server defaults the rest (erasureCoded, mirroring, statusCheck, … — the exact set depends on the Rook version). The old code compared and replaced the whole spec, so those defaults made every reconcile issue an Update that stripped them, which the defaulter re-added, in a watch-driven loop. Now the rendered fields are merged in and the defaults are left alone:

# Note the defaulted fields and the resourceVersion.
kubectl --context k3d-fundament-plugin -n rook-ceph get cephblockpool ceph-test-pool \
  -o jsonpath='{.spec}{"\n"}{.metadata.resourceVersion}{"\n"}'

# Force a reconcile: any BlockStorage change enqueues one; an annotation is enough.
kubectl --context k3d-fundament-plugin annotate blockstorage test-pool verify=8c-3 --overwrite

kubectl --context k3d-fundament-plugin -n rook-ceph get cephblockpool ceph-test-pool \
  -o jsonpath='{.spec}{"\n"}{.metadata.resourceVersion}{"\n"}'

Both lines must be identical across the two reads: the defaulted fields are still in the spec, and the unchanged resourceVersion proves no write was issued at all. The CephFilesystem behaves the same way (get cephfilesystem cephfs-test-pool).

While you are here, skim the plugin logs from this whole phase:

kubectl --context k3d-fundament-plugin -n plugin-system--ceph-rook logs deploy/plugin --tail=200

No Operation cannot be fulfilled messages should appear as pool or consumer status — write conflicts are now retried silently instead of flapping objects to Degraded. The remaining fixes of this round (conflict requeue itself, watch fan-out error logging, the discovery watch predicate, the CRD establishment wait, owner-reference and naming cleanups) have no operator-visible surface to probe here; they are covered by the unit tests.

8d · Deleting a pool shrinks the CephCluster

Nothing else recomputes this: the CephCluster carries no owner reference. DiskPool and BlockStorage/FileStorage are independent objects, so deleting the pool does not cascade to the derived StorageClasses — only to the CephCluster’s device list. (This holds only for pools created after the split; a pre-split pool still owns its derived objects, see 8b-2.)

kubectl --context k3d-fundament-plugin -n rook-ceph get cephcluster rook-ceph \
  -o jsonpath='{.spec.storage.nodes}' | jq .      # 3 devices

kubectl --context k3d-fundament-plugin delete diskpool test-pool

kubectl --context k3d-fundament-plugin -n rook-ceph get cephcluster rook-ceph \
  -o jsonpath='{.spec.storage.nodes}' | jq .      # now empty
kubectl --context k3d-fundament-plugin get blockstorage test-pool \
  -o jsonpath='{.status.phase}{"\n"}{.status.message}{"\n"}'
# Degraded / no DiskPool contributes disks
kubectl --context k3d-fundament-plugin get storageclass ceph-test-pool   # still exists
kubectl --context k3d-fundament-plugin -n rook-ceph get cephblockpool ceph-test-pool  # still exists

The CephBlockPool/CephFilesystem/CephObjectStore and their StorageClasses are owned by BlockStorage/FileStorage/ObjectStorage, not by the DiskPool, so they only disappear when those are deleted:

kubectl --context k3d-fundament-plugin delete blockstorage test-pool
kubectl --context k3d-fundament-plugin delete filestorage test-pool
kubectl --context k3d-fundament-plugin delete objectstorage test-pool

kubectl --context k3d-fundament-plugin get storageclass ceph-test-pool     # NotFound (cascade)
kubectl --context k3d-fundament-plugin -n rook-ceph get cephblockpool ceph-test-pool  # NotFound
kubectl --context k3d-fundament-plugin get storageclass cephfs-test-pool   # NotFound (cascade)
kubectl --context k3d-fundament-plugin -n rook-ceph get cephfilesystem cephfs-test-pool  # NotFound
kubectl --context k3d-fundament-plugin get storageclass cephobj-test-pool  # NotFound (cascade)
kubectl --context k3d-fundament-plugin -n rook-ceph get cephobjectstore cephobj-test-pool  # NotFound

The OSDs stay

The devices leave the CephCluster, but Rook does not retire the OSDs — that needs an explicit Ceph purge. kubectl -n rook-ceph get pods -l app=rook-ceph-osd will still show them. This is expected and documented; it is why disk removal is a two-step operation.

8d-2 · A missing CephCluster is named, not hidden

Run this last: it deletes the CephCluster, so it is effectively the start of teardown.

Nothing can provision without the singleton CephCluster, but the old code reconciled a BlockStorage/FileStorage anyway — it created the derived pair and reported Provisioning forever, publishing a StorageClass whose PVCs pend with no hint at the real cause (only the DiskPool named it). Now the consumer refuses to derive anything and names the missing cluster itself.

# Blocks until Rook's finalizer has torn the daemons down — expect a minute or two.
kubectl --context k3d-fundament-plugin -n rook-ceph delete cephcluster rook-ceph --wait=true

kubectl --context k3d-fundament-plugin apply -f - <<'YAML'
apiVersion: ceph.fundament.io/v1alpha1
kind: BlockStorage
metadata:
  name: orphan
spec: {}
YAML

kubectl --context k3d-fundament-plugin get blockstorage orphan \
  -o jsonpath='{.status.phase}{"\n"}{.status.message}{"\n"}'
# Degraded
# CephCluster rook-ceph/rook-ceph not found: nothing can be provisioned until it exists

kubectl --context k3d-fundament-plugin get storageclass ceph-orphan
# NotFound — nothing was derived

kubectl --context k3d-fundament-plugin delete blockstorage orphan

No watch covers the CephCluster for consumers, so they poll for it every 30 seconds. To verify the recovery half instead of proceeding to teardown, restart the plugin — its install path re-bootstraps the CephCluster — and watch orphan move off the missing- cluster message within a poll:

kubectl --context k3d-fundament-plugin -n plugin-system--ceph-rook rollout restart deploy/plugin

Phase 9 · Teardown

Order matters. Ceph consumers must release volumes while Ceph is still running to service the unmounts:

just plugins uninstall system--ceph-rook
just plugins cluster-delete            # drains first
just plugins storage-disks purge       # detach and delete the backing images, freeing the space

To keep the disks for a future run, use just plugins storage-disks reset instead of purge.

Troubleshooting

Symptom	Cause	Fix
REFUSING: the Docker host is not a virtual machine, Detected: unknown, on macOS	VM probes could not answer	Fixed — an aarch64 guest has no DMI and often no systemd, so detection now also reads the device tree and the virtio bus. If it recurs, just plugins storage-disks doctor prints what was detected
losetup: unrecognized option: show (or : j)	Docker host ships BusyBox, not util-linux	Fixed — association no longer uses --show/-j. OrbStack and colima both ship BusyBox
attach exits 1 printing nothing	A failing test as a loop body’s last command, under set -e	Fixed. If a similar silent exit appears, re-run with bash -x
Smoke test times out; events show modprobe args: [ceph] repeating	No ceph kernel module — CephFS cannot mount	Not a failure. The plugin is block-only; the smoke test now skips CephFS automatically
rbd: mapping succeeded but /dev/rbd0 is not accessible once	udev race under --options noudev	Ignore — kubelet retries. Only a concern if it repeats
no catalog entry for "ceph-rook"	Appstore seed not applied	Re-run db-migrations against k3d-fundament
Publish fails with a permission error	Publishing as a user who is not an admin of the owning (system) org	Use a platform-admin@fundament.io token — see the caution in phase 3
No Disk CRs, ConfigMap has devices	Filter rejected them	DEV_LOOP_DEVICES: "true" needs type: part on /dev/loopNpN
OSD prepare job: unsupported diskType loop	allowLoopDevices did not take	Confirm DEV_LOOP_DEVICES is set on the PluginInstallation
Mons never reach quorum	3 mons on one node	Set MON_COUNT: "1" and ALLOW_MULTIPLE_PER_NODE: "true"
CephCluster rejected on version	Ceph release outside Rook’s table	Only if you overrode CEPH_IMAGE: set ALLOW_UNSUPPORTED_CEPH: "true" (default false)
csi-rbdplugin CrashLoopBackOff	rbd kernel module missing	just plugins storage-disks doctor; use colima
PVC Pending, provisioner logs quiet	No OSD is up	kubectl -n rook-ceph get pods -l app=rook-ceph-osd
CephFS PVC stuck ContainerCreating, CSI log shows modprobe args: [ceph]	No ceph kernel module; CSI is kernel-mounting	Set CEPHFS_MOUNTER: "fuse" on the PluginInstallation and reinstall (RBD is unaffected)
rbd: mapping succeeded but /dev/rbd0 is not accessible	No /dev bind	Recreate with just plugins cluster-create-storage
Second install fails after a first attempt	Stale OSD metadata	just plugins storage-disks reset
Node container will not stop	Stale RBD mapping	just plugins storage-disks unmap-stale
Pool stuck Provisioning	CephBlockPool not Ready	kubectl -n rook-ceph describe cephblockpool ceph-<pool>
Pool Degraded	Drift, missing CephCluster, or Rook reporting Failure (write conflicts are retried silently, never reported)	Read status.message — it names the action

Related

• Block devices for k3d — how the loop devices work, persistence, cleanup
• Example: Ceph Storage (Rook) — architecture, replication, RBAC rationale