diff --git a/README.md b/README.md index 7e216c477d8..07d5825b3fc 100644 --- a/README.md +++ b/README.md @@ -35,9 +35,9 @@ attaching them to workloads. Independent CSI plugins are provided to support RBD and CephFS backed volumes, - For details about configuration and deployment of RBD plugin, please refer - [rbd doc](https://github.com/ceph/ceph-csi/blob/devel/docs/deploy-rbd.md) and + [rbd doc](https://github.com/ceph/ceph-csi/blob/devel/docs/rbd/deploy.md) and for CephFS plugin configuration and deployment please - refer [cephFS doc](https://github.com/ceph/ceph-csi/blob/devel/docs/deploy-cephfs.md). + refer [cephFS doc](https://github.com/ceph/ceph-csi/blob/devel/docs/cephfs/deploy.md). - For example usage of the RBD and CephFS CSI plugins, see examples in `examples/`. - Stale resource cleanup, please refer [cleanup doc](docs/resource-cleanup.md). @@ -75,7 +75,7 @@ Ceph CSI version, they will be asked to upgrade when requesting support. ### Ceph-CSI features and available versions -Please refer [rbd nbd mounter](./docs/rbd-nbd.md#support-matrix) +Please refer [rbd nbd mounter](./docs/design/proposals/rbd-nbd.md#support-matrix) for its support details. | Plugin | Features | Feature Status | CSI Driver Version | CSI Spec Version | Ceph Cluster Version | Kubernetes Version | diff --git a/charts/ceph-csi-cephfs/README.md b/charts/ceph-csi-cephfs/README.md index 1fdca5b29b2..6c010db214c 100644 --- a/charts/ceph-csi-cephfs/README.md +++ b/charts/ceph-csi-cephfs/README.md @@ -66,6 +66,33 @@ version. We recommend not to use `--reuse-values` in case there are new defaults AND compare your currently used values with the new default values. +### Enabling encryption support + +To enable FSCrypt support, you will need to include the KMS configuration in +`encryptionKMSConfig`. + +Here is a `values.yaml` example using a Kubernetes secret (`kubernetes` KMS) + +```yaml +encryptionKMSConfig: + encryptionKMSType: "metadata" + secretName: "cephfs-encryption-passphrase" # This secret needs to contain the passphrase as the key `encryptionPassphrase` + secretNamespace: "my-namespace" +storageClass: + encrypted: true + encryptionKMSID: kubernetes +``` + +#### Least privilege secret access + +If you use the `metadata` and let RBAC created by the chart, permissions +will be given to access **only** the secret referenced in the +`encryptionKMSConfig`. This is something important to keep in mind, as a +manual change to the config to point to another secret or add further KMS +config will not be authorized. If you wish to give CephCSI a global secret +access to the cluster, you may set `rbac.leastPrivileges` to `false`, and +permissions will be granted globally via a *ClusterRole*. + #### Known Issues Upgrading - When upgrading to version >=3.7.0, you might encounter an error that the @@ -110,11 +137,13 @@ charts and their default values. | Parameter | Description | Default | | ---------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------- | | `rbac.create` | Specifies whether RBAC resources should be created | `true` | +| `rbac.leastPrivileges` | Specifies whether RBAC resources should be created with a restricted scope when supported (only secrets supported currently) | `true` | | `serviceAccounts.nodeplugin.create` | Specifies whether a nodeplugin ServiceAccount should be created | `true` | | `serviceAccounts.nodeplugin.name` | The name of the nodeplugin ServiceAccount to use. If not set and create is true, a name is generated using the fullname | "" | | `serviceAccounts.provisioner.create` | Specifies whether a provisioner ServiceAccount should be created | `true` | | `serviceAccounts.provisioner.name` | The name of the provisioner ServiceAccount of provisioner to use. If not set and create is true, a name is generated using the fullname | "" | | `csiConfig` | Configuration for the CSI to connect to the cluster | [] | +| `encryptionKMSConfig` | Configuration for the encryption KMS | `{}` | | `commonLabels` | Labels to apply to all resources | `{}` | | `logLevel` | Set logging level for csi containers. Supported values from 0 to 5. 0 for general useful logs, 5 for trace level verbosity. | `5` | | `sidecarLogLevel` | Set logging level for csi sidecar containers. Supported values from 0 to 5. 0 for general useful logs, 5 for trace level verbosity. | `1` | @@ -174,7 +203,7 @@ charts and their default values. | `provisionerSocketFile` | The filename of the provisioner socket | `csi-provisioner.sock` | | `pluginSocketFile` | The filename of the plugin socket | `csi.sock` | | `readAffinity.enabled` | Enable read affinity for CephFS subvolumes. Recommended to set to true if running kernel 5.8 or newer. | `false` | -| `readAffinity.crushLocationLabels` | Define which node labels to use as CRUSH location. This should correspond to the values set in the CRUSH map. For more information, click [here](https://github.com/ceph/ceph-csi/blob/v3.9.0/docs/deploy-cephfs.md#read-affinity-using-crush-locations-for-cephfs-subvolumes)| `[]` | +| `readAffinity.crushLocationLabels` | Define which node labels to use as CRUSH location. This should correspond to the values set in the CRUSH map. For more information, click [here](https://github.com/ceph/ceph-csi/blob/devel/docs/cephfs/deploy.md#read-affinity-using-crush-locations-for-cephfs-subvolumes)| `[]` | | `kubeletDir` | Kubelet working directory | `/var/lib/kubelet` | | `driverName` | Name of the csi-driver | `cephfs.csi.ceph.com` | | `configMapName` | Name of the configmap which contains cluster configuration | `ceph-csi-config` | @@ -184,6 +213,8 @@ charts and their default values. | `storageClass.name` | Specifies the cephFS StorageClass name | `csi-cephfs-sc` | | `storageClass.annotations` | Specifies the annotations for the cephFS storageClass | `[]` | | `storageClass.clusterID` | String representing a Ceph cluster to provision storage from | `` | +| `storageClass.encrypted` | Specifies whether volume should be encrypted. Set it to true if you want to enable encryption | `""` | +| `storageClass.encryptionKMSID` | Specifies the encryption kms id | `""` | | `storageClass.fsName` | CephFS filesystem name into which the volume shall be created | `myfs` | | `storageClass.pool` | Ceph pool into which volume data shall be stored | `""` | | `storageClass.fuseMountOptions` | Comma separated string of Ceph-fuse mount options | `""` | diff --git a/charts/ceph-csi-cephfs/templates/encryptionkms-configmap.yaml b/charts/ceph-csi-cephfs/templates/encryptionkms-configmap.yaml new file mode 100644 index 00000000000..543297b62ab --- /dev/null +++ b/charts/ceph-csi-cephfs/templates/encryptionkms-configmap.yaml @@ -0,0 +1,15 @@ +apiVersion: v1 +kind: ConfigMap +metadata: + name: {{ .Values.kmsConfigMapName | quote }} + namespace: {{ .Release.Namespace }} + labels: + app: {{ include "ceph-csi-cephfs.name" . }} + chart: {{ include "ceph-csi-cephfs.chart" . }} + component: {{ .Values.nodeplugin.name }} + release: {{ .Release.Name }} + heritage: {{ .Release.Service }} + {{- with .Values.commonLabels }}{{ toYaml . | trim | nindent 4 }}{{- end }} +data: + config.json: |- +{{ toJson .Values.encryptionKMSConfig | indent 4 -}} diff --git a/charts/ceph-csi-cephfs/templates/nodeplugin-clusterrole.yaml b/charts/ceph-csi-cephfs/templates/nodeplugin-clusterrole.yaml index 632672ebd54..dfd2ea9f84f 100644 --- a/charts/ceph-csi-cephfs/templates/nodeplugin-clusterrole.yaml +++ b/charts/ceph-csi-cephfs/templates/nodeplugin-clusterrole.yaml @@ -14,10 +14,14 @@ rules: - apiGroups: [""] resources: ["nodes"] verbs: ["get"] + # allow to read Vault Token and connection options from the Tenants namespace - apiGroups: [""] resources: ["configmaps"] verbs: ["get"] +{{- if and .Values.encryptionKMSConfig .Values.encryptionKMSConfig.secretNamespace (not .Values.rbac.leastPrivileges) }} + # allow to read the encryption key used with the metadata KMS - apiGroups: [""] resources: ["secrets"] verbs: ["get"] {{- end -}} +{{- end -}} diff --git a/charts/ceph-csi-cephfs/templates/nodeplugin-role.yaml b/charts/ceph-csi-cephfs/templates/nodeplugin-role.yaml new file mode 100644 index 00000000000..a9a243c7d1a --- /dev/null +++ b/charts/ceph-csi-cephfs/templates/nodeplugin-role.yaml @@ -0,0 +1,22 @@ +{{- if and .Values.rbac.create .Values.rbac.leastPrivileges -}} +{{- if and .Values.encryptionKMSConfig (eq .Values.encryptionKMSConfig.encryptionKMSType "metadata") .Values.encryptionKMSConfig.secretNamespace .Values.encryptionKMSConfig.secretName -}} +kind: Role +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: {{ include "ceph-csi-cephfs.nodeplugin.fullname" . }} + namespace: {{ .Values.encryptionKMSConfig.secretNamespace }} + labels: + app: {{ include "ceph-csi-cephfs.name" . }} + chart: {{ include "ceph-csi-cephfs.chart" . }} + component: {{ .Values.nodeplugin.name }} + release: {{ .Release.Name }} + heritage: {{ .Release.Service }} + {{- with .Values.commonLabels }}{{ toYaml . | trim | nindent 4 }}{{- end }} +rules: + # allow to read the encryption key used with the metadata KMS + - apiGroups: [""] + resources: ["secrets"] + verbs: ["get"] + resourceNames: [{{ .Values.encryptionKMSConfig.secretName | quote }}] +{{- end -}} +{{- end -}} diff --git a/charts/ceph-csi-cephfs/templates/nodeplugin-rolebinding.yaml b/charts/ceph-csi-cephfs/templates/nodeplugin-rolebinding.yaml new file mode 100644 index 00000000000..0be11b5422e --- /dev/null +++ b/charts/ceph-csi-cephfs/templates/nodeplugin-rolebinding.yaml @@ -0,0 +1,24 @@ +{{- if and .Values.rbac.create .Values.rbac.leastPrivileges -}} +{{- if and .Values.encryptionKMSConfig (eq .Values.encryptionKMSConfig.encryptionKMSType "metadata") .Values.encryptionKMSConfig.secretNamespace -}} +kind: RoleBinding +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: {{ include "ceph-csi-cephfs.nodeplugin.fullname" . }} + namespace: {{ .Values.encryptionKMSConfig.secretNamespace }} + labels: + app: {{ include "ceph-csi-cephfs.name" . }} + chart: {{ include "ceph-csi-cephfs.chart" . }} + component: {{ .Values.nodeplugin.name }} + release: {{ .Release.Name }} + heritage: {{ .Release.Service }} + {{- with .Values.commonLabels }}{{ toYaml . | trim | nindent 4 }}{{- end }} +subjects: + - kind: ServiceAccount + name: {{ include "ceph-csi-cephfs.serviceAccountName.nodeplugin" . }} + namespace: {{ .Release.Namespace }} +roleRef: + kind: Role + name: {{ include "ceph-csi-cephfs.nodeplugin.fullname" . }} + apiGroup: rbac.authorization.k8s.io +{{- end -}} +{{- end -}} diff --git a/charts/ceph-csi-cephfs/templates/storageclass.yaml b/charts/ceph-csi-cephfs/templates/storageclass.yaml index a21c998244a..35e5c5e43f7 100644 --- a/charts/ceph-csi-cephfs/templates/storageclass.yaml +++ b/charts/ceph-csi-cephfs/templates/storageclass.yaml @@ -20,6 +20,12 @@ parameters: {{- if .Values.storageClass.pool }} pool: {{ .Values.storageClass.pool }} {{- end }} +{{- if .Values.storageClass.encrypted }} + encrypted: "{{ .Values.storageClass.encrypted }}" +{{- end }} +{{- if .Values.storageClass.encryptionKMSID }} + encryptionKMSID: {{ .Values.storageClass.encryptionKMSID }} +{{- end }} {{- if .Values.storageClass.fuseMountOptions }} fuseMountOptions: "{{ .Values.storageClass.fuseMountOptions }}" {{- end }} diff --git a/charts/ceph-csi-cephfs/values.yaml b/charts/ceph-csi-cephfs/values.yaml index 814b6a41830..fa611cae312 100644 --- a/charts/ceph-csi-cephfs/values.yaml +++ b/charts/ceph-csi-cephfs/values.yaml @@ -2,6 +2,9 @@ rbac: # Specifies whether RBAC resources should be created create: true + # When possible try and reduce the scope of permission to only give + # access to resources defined in the config. See the README for more info + leastPrivileges: true serviceAccounts: nodeplugin: @@ -31,6 +34,20 @@ serviceAccounts: # radosNamespace: "csi" csiConfig: [] +# Configuration for the encryption KMS +# yamllint disable-line rule:line-length +# Ref: https://github.com/ceph/ceph-csi/blob/devel/docs/deploy-cephfs.md#cephfs-volume-encryption +# Example: +# encryptionKMSConfig: +# encryptionKMSType: vault +# vaultAddress: https://vault.example.com +# vaultAuthPath: /v1/auth/kubernetes/login +# vaultRole: csi-kubernetes +# vaultPassphraseRoot: /v1/secret +# vaultPassphrasePath: ceph-csi/ +# vaultCAVerify: "true" +encryptionKMSConfig: {} + # Labels to apply to all resources commonLabels: {} @@ -329,6 +346,18 @@ storageClass: # If omitted, defaults to "csi-vol-". # volumeNamePrefix: "foo-bar-" volumeNamePrefix: "" + + # (optional) Instruct the plugin it has to encrypt the volume + # By default it is disabled. Valid values are "true" or "false". + # A string is expected here, i.e. "true", not true. + # encrypted: "true" + encrypted: "" + + # (optional) Use external key management system for encryption passphrases by + # specifying a unique ID matching KMS ConfigMap. The ID is only used for + # correlation to configmap entry. + encryptionKMSID: "" + # The secrets have to contain user and/or Ceph admin credentials. provisionerSecret: csi-cephfs-secret # If the Namespaces are not specified, the secrets are assumed to @@ -400,6 +429,8 @@ configMapName: ceph-csi-config externallyManagedConfigmap: false # Name of the configmap used for ceph.conf cephConfConfigMapName: ceph-config +# Name of the configmap used for encryption kms configuration +kmsConfigMapName: ceph-csi-encryption-kms-config # CephFS RadosNamespace used to store CSI specific objects and keys. # radosNamespaceCephFS: csi # Unique ID distinguishing this instance of Ceph CSI among other instances, diff --git a/charts/ceph-csi-rbd/README.md b/charts/ceph-csi-rbd/README.md index 4bf4694e7bd..b1156e57993 100644 --- a/charts/ceph-csi-rbd/README.md +++ b/charts/ceph-csi-rbd/README.md @@ -187,7 +187,7 @@ charts and their default values. | `topology.enabled` | Specifies whether topology based provisioning support should be exposed by CSI | `false` | | `topology.domainLabels` | DomainLabels define which node labels to use as domains for CSI nodeplugins to advertise their domains | `{}` | | `readAffinity.enabled` | Enable read affinity for RBD volumes. Recommended to set to true if running kernel 5.8 or newer. | `false` | -| `readAffinity.crushLocationLabels` | Define which node labels to use as CRUSH location. This should correspond to the values set in the CRUSH map. For more information, click [here](https://github.com/ceph/ceph-csi/blob/v3.9.0/docs/deploy-rbd.md#read-affinity-using-crush-locations-for-rbd-volumes)| `[]` | +| `readAffinity.crushLocationLabels` | Define which node labels to use as CRUSH location. This should correspond to the values set in the CRUSH map. For more information, click [here](https://github.com/ceph/ceph-csi/blob/devel/docs/rbd/deploy.md#read-affinity-using-crush-locations-for-rbd-volumes)| `[]` | | `provisionerSocketFile` | The filename of the provisioner socket | `csi-provisioner.sock` | | `pluginSocketFile` | The filename of the plugin socket | `csi.sock` | | `kubeletDir` | kubelet working directory | `/var/lib/kubelet` | diff --git a/charts/ceph-csi-rbd/values.yaml b/charts/ceph-csi-rbd/values.yaml index 16142e32daa..aa17fed9570 100644 --- a/charts/ceph-csi-rbd/values.yaml +++ b/charts/ceph-csi-rbd/values.yaml @@ -46,7 +46,7 @@ csiConfig: [] csiMapping: [] # Configuration for the encryption KMS -# Ref: https://github.com/ceph/ceph-csi/blob/devel/docs/deploy-rbd.md +# Ref: https://github.com/ceph/ceph-csi/blob/devel/docs/rbd/deploy.md # Example: # encryptionKMSConfig: # vault-unique-id-1: @@ -415,7 +415,7 @@ storageClass: # By default host-path /var/log/ceph of node is bind-mounted into # csi-rbdplugin pod at /var/log/ceph mount path. This is to configure # target bindmount path used inside container for ceph clients logging. - # See docs/rbd-nbd.md for available configuration options. + # See docs/design/proposals/rbd-nbd.md for available configuration options. # cephLogDir: /var/log/ceph cephLogDir: "" diff --git a/docs/resource-cleanup.md b/docs/resource-cleanup.md index 69ca86acb59..23162afa99f 100644 --- a/docs/resource-cleanup.md +++ b/docs/resource-cleanup.md @@ -28,7 +28,7 @@ a. get pv_name ### 2. Get omap key/value a. get omapkey (suffix of csi.volumes.default is value used for the CLI option - [--instanceid](deploy-rbd.md#configuration) in the provisioner deployment.) + [--instanceid](rbd/deploy.md#configuration) in the provisioner deployment.) ``` rados listomapkeys csi.volumes.default -p pool_name | grep pv_name @@ -56,7 +56,7 @@ b. get omapval ### 3. Delete the RBD image or CephFS subvolume -a. remove rbd image(csi-vol-omapval, the prefix csi-vol is value of [volumeNamePrefix](deploy-rbd.md#configuration)) +a. remove rbd image(csi-vol-omapval, the prefix csi-vol is value of [volumeNamePrefix](rbd/deploy.md#configuration)) ``` rbd remove rbd_image_name -p pool_name diff --git a/docs/snap-clone.md b/docs/snap-clone.md index e6c970738eb..c1106c57283 100644 --- a/docs/snap-clone.md +++ b/docs/snap-clone.md @@ -56,7 +56,7 @@ The snapshot is created on/for an existing PVC. You should have a PVC in bound state before creating snapshot from it. It is recommended to create a volume snapshot or a PVC clone only when the PVC is not in use. -Please refer pvc creation [doc](https://github.com/ceph/ceph-csi/blob/devel/docs/deploy-cephfs.md) +Please refer pvc creation [doc](https://github.com/ceph/ceph-csi/blob/devel/docs/cephfs/deploy.md) for more information on how to create a PVC. - Verify if PVC is in Bound state diff --git a/examples/rbd/storageclass.yaml b/examples/rbd/storageclass.yaml index 5d2fc5ebae7..9d87b03ef05 100644 --- a/examples/rbd/storageclass.yaml +++ b/examples/rbd/storageclass.yaml @@ -101,7 +101,7 @@ parameters: # By default host-path /var/log/ceph of node is bind-mounted into # csi-rbdplugin pod at /var/log/ceph mount path. This is to configure # target bindmount path used inside container for ceph clients logging. - # See docs/rbd-nbd.md for available configuration options. + # See docs/design/proposals/rbd-nbd.md for available configuration options. # cephLogDir: /var/log/ceph # (optional) ceph client log strategy diff --git a/scripts/install-helm.sh b/scripts/install-helm.sh index 429e633b147..eae2d51e606 100755 --- a/scripts/install-helm.sh +++ b/scripts/install-helm.sh @@ -188,6 +188,7 @@ install_cephcsi_helm_charts() { # issue when installing ceph-csi-rbd kubectl_retry delete cm ceph-csi-config --namespace "${NAMESPACE}" kubectl_retry delete cm ceph-config --namespace "${NAMESPACE}" + kubectl_retry delete cm ceph-csi-encryption-kms-config --namespace "${NAMESPACE}" # shellcheck disable=SC2086 "${HELM}" install --namespace ${NAMESPACE} --set provisioner.fullnameOverride=csi-rbdplugin-provisioner --set nodeplugin.fullnameOverride=csi-rbdplugin --set configMapName=ceph-csi-config --set provisioner.replicaCount=1 --set-json='commonLabels={"app.kubernetes.io/name": "ceph-csi-rbd", "app.kubernetes.io/managed-by": "helm"}' ${SET_SC_TEMPLATE_VALUES} ${RBD_SECRET_TEMPLATE_VALUES} ${RBD_CHART_NAME} "${SCRIPT_DIR}"/../charts/ceph-csi-rbd --set topology.domainLabels="{${NODE_LABEL_REGION},${NODE_LABEL_ZONE}}" --set provisioner.maxSnapshotsOnImage=3 --set provisioner.minSnapshotsOnImage=2 ${READ_AFFINITY_VALUES} --set provisioner.snapshotter.args.enableVolumeGroupSnapshots=true