Chapter 6. Security considerations – IBM Spectrum Scale CSI Driver for Container Persistent Storage

Security considerations
Ensuring an appropriate security architecture is an essential step while planning an enterprise container environment.
In this chapter we will cover security topics related to IBM Spectrum Scale CSI:
Understanding the security-related aspects will help for IBM Spectrum Scale CSI driver planning, deployment, and usage. This chapter is recommended for architects planning the environment and security administrators.
6.1 Secure administration and deployment
Secure administration is one of the important aspects for any deployment. Based on IBM Spectrum Scale security features for administration, IBM Spectrum Scale CSI ensures the administration and deployment are secure. In this section, we describe the different aspects that one is required to consider for this purpose.
6.1.1 Network deployment and firewall configuration
From a security perspective it is recommended to use separate cluster/data and admin networks. The cluster network is used for the IBM Spectrum Scale cluster metadata and data traffic while the admin network is used for administrator access to the IBM Spectrum Scale GUI and CLI as well as for REST API communication.
All Kubernetes worker nodes need access to the IBM Spectrum Scale cluster network. The Kubernetes Infrastructure nodes need access to the IBM Spectrum Scale admin network (as they are expected to interact with IBM Spectrum Scale GUI nodes). One will need to likewise set the firewalls based on the deployment network topology. see the “Firewall recommendations for internal communication among nodes” in IBM Knowledge Center for more details:
 
Note: For the firewall requirements from Kubernetes deployment, see the Kubernetes distribution documentation. For example, for OpenShift see the OpenShift documentation:
6.1.2 Secure communication with IBM Spectrum Scale GUI server
The IBM Spectrum Scale CSI driver communicates with IBM Spectrum Scale through the REST API provided by the IBM Spectrum Scale GUI node. If Kubernetes node labeling in combination with a nodeSelector is used to place the CSI controller plug-in (provisioner and attacher containers) onto the Kubernetes cluster infrastructure nodes, only the infrastructure nodes need access to the IBM Spectrum Scale GUI node using the admin network. To ensure the proper firewall rules are set, see the “Firewall recommendations for IBM Spectrum Scale GUI” entry in IBM Knowledge Center:
In Figure 6-1 on page 41 there are three security parameters that are required for secure communication between the CSI plugin and the GUI server:
Secure SSL Mode It specifies whether certificate validation should be done. If secure SSL Mode is true, providing of GUI CA certificate becomes mandatory. If false, the certificate validation is ignored.
Certificate This is a CA certificate for GUI server. It is required if SSL mode is true.
User credential In Plug-in configuration, we provide a Kubernetes secret object which contains the IBM Spectrum Scale GUI “username” and “Password” as base64 encoded values.
Figure 6-1 Security parameters for secure communication between CSI plugin and GUI server
These parameters are specified as part of operator specification. For more details about how to set up these security parameters, see IBM Knowledge Center:
6.1.3 SELinux considerations
Typically, SELinux prerequisites on the nodes are driven by the Kubernetes distribution. As an example, OpenShift deployment the worker nodes in the cluster are required to have SELinux enabled in enforcing mode with targeted policies while the control plane nodes runs on RHCOS (RHCOS is the immutable container host version of Red Hat Enterprise Linux (RHEL) and features a RHEL kernel with SELinux enabled by default).
On SELinux enabled systems, file system objects will have SELinux context as user:role:type:level. For access decisions governed by SELinux policies, mainly the type label is used and currently IBM Spectrum Scale file system is unlabeled. So, in order to have access to a IBM Spectrum Scale File Set or directory inside a container, a Container Orchestrator like OpenShift potentially might be required to perform relabeling. This setup enables file objects to be labelled with a label accessible to the domain in which the container process is running.
 
Note: The relabeling will not disrupt the regular data access. The GPFS daemon and CES services are running in an unconfined domain so they will have access to the file set or directory which are relabelled by the container orchestrator.
6.1.4 Configuring UID/GID ownership to ensure Persistent Volume access for non-root containers
For accessing a Persistent Volume (PV) inside a container where the main process is running as non-root user, proper ownership needs to be set for a PV and POD security context needs to be properly set.
For changing ownership of a PV, we can use storage class and set UID/GID of a PV. When a PV is created, the ownership of a volume will be dynamically set based on storage class parameters.
Setting of PV ownership using storage class is applicable for dynamic provisioning. For static provisioning, the admin must take care of setting permissions and ownership on the PV. In case UID/GID is not provided in the storage class, by default the file set/directory created for PV will have owner as root.
In Example 6-1 we have set the UID as 4000 and GID as 5555 in a storage class csi-spectrum-scale-file set-uid. This storage class will be used while creating a PV dynamically.
Example 6-1 Sample storageclass with volume ownership
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: csi-spectrum-scale-file set-uid
provisioner: spectrumscale.csi.ibm.com
parameters:
volBackendFs: "fs1"
clusterId: "17934173018790868908"
gid: 5555
uid: 4000
reclaimPolicy: Delete
Using the storage class csi-spectrum-scale-file set-uid shown in Example 6-1, we create a PVC. For this example, the PVC name is scale-fset-pvc-sample.
After the PVC is created it will bind to a PV. On the IBM Spectrum Scale side, the PV will appear as a file set with permission 771 and ownership 4000:5555. This means that any process with uid of 4000 or which belongs to group 5555 will have access to the PV.
For the case in Example 6-2, we have created a POD with a security context such that it will run as user 4001 (which is not a member of group 5555). Because UID 4001 doesn’t have access to PV, we will get a Permission Denied error.
Example 6-2 Definition of POD which will run with UID 4001
# cat podfileset-uid.yaml
kind: Pod
apiVersion: v1
metadata:
name: sanity-pod # Pod name
spec:
nodeSelector:
spectrumscalenode1: "yes"
securityContext:
runAsUser: 4001
containers:
- name: container1 # Container name
image: alpine:latest
command: [ "/bin/sh", "-c", "--" ]
args: [ "while true; do sleep 30; done;" ]
volumeMounts:
- name: vol1
mountPath: "/data" # Where to mount the vol1(pvc1)
restartPolicy: "Never"
volumes:
- name: vol1
persistentVolumeClaim:
claimName: scale-fset-pvc-sample
Using the following command, we can see that UID 4001 is unable to create a file in a PV that is mounted at /data inside a container:
# oc create -f podfileset-uid.yamlpod/sanity-pod created
Using the command in Example 6-3, we can see that UID 4001 is unable to create a file in a PV that is mounted at /data inside a container.
Example 6-3 Example of write from a container on a PVC with different UID
# oc exec -it sanity-pod -- sh -c "id && echo "hello" > /data/file"
uid=4001(4001) gid=0(root)
sh: can't create /data/file: Permission denied
command terminated with exit code 1
In Example 6-4, we have created a POD with a security context such that it will run as user 4000. Because UID 4000 has access to the PV, the write will succeed.
Example 6-4 Definition of POD which will run with UID 4000
# cat podfileset-uid.yaml
kind: Pod
apiVersion: v1
metadata:
name: sanity-pod # Pod name
spec:
nodeSelector:
spectrumscalenode1: "yes"
securityContext:
runAsUser: 4000
containers:
- name: container1 # Container name
image: alpine:latest
command: [ "/bin/sh", "-c", "--" ]
args: [ "while true; do sleep 30; done;" ]
volumeMounts:
- name: vol1
mountPath: "/data" # Where to mount the vol1(pvc1)
restartPolicy: "Never"
volumes:
- name: vol1
persistentVolumeClaim:
claimName: scale-fset-pvc-sample
The following command will create a POD sanity-pod running with UID 4000:
# oc create -f podfileset-uid.yaml
pod/sanity-pod created
Because POD is running with UID 4000, it will be able to write to a PV, because the owner of the PV is set to 4000:
# oc exec -it sanity-pod -- sh -c "id && echo "hello" > /data/file"
uid=4000(4000) gid=0(root)
#
In Example 6-5, we have created a security context such that POD becomes part of supplemental group 5555. Therefore, the container process will have a secondary group 5555. Because the 5555 group has access to the PV, the write will succeed.
Example 6-5 Definition of POD with supplemental group
# cat podfileset-uid.yaml
kind: Pod
apiVersion: v1
metadata:
name: sanity-pod # Pod name
spec:
nodeSelector:
spectrumscalenode1: "yes"
securityContext:
runAsUser: 4001
supplementalGroups: [5555]
containers:
- name: container1 # Container name
image: alpine:latest
command: [ "/bin/sh", "-c", "--" ]
args: [ "while true; do sleep 30; done;" ]
volumeMounts:
- name: vol1
mountPath: "/data" # Where to mount the vol1(pvc1)
restartPolicy: "Never"
volumes:
- name: vol1
persistentVolumeClaim:
claimName: scale-fset-pvc-sample
The following command in will create a POD sanity-pod which will be part of supplementary group 5555:
# oc create -f podfileset-uid.yaml
pod/sanity-pod created
Because group 5555 has write permission on the PV, the write operation will succeed. We can also see in Example 6-6 that POD is made part of supplemental group 5555.
Example 6-6 Example of write from a container with required supplemental group
# oc exec -it sanity-pod -- sh -c "id && echo "hello" > /data/file1"
uid=4001(4001) gid=0(root) groups=5555
#
6.1.5 Privileged CSI PODs
Currently in IBM Spectrum Scale CSI Driver, the attacher and provisioner Statefulset run Pods in privileged mode on OpenShift. Also, the Daemonset Pod for driver registrar runs in privileged mode; however, the Daemonset Pod for the CSI plugin runs in non-privileged mode.
On a Kubernetes cluster, all CSI driver pods run in non-privileged mode.
6.2 Secure data for containers
It is important for workloads adopting to containerization to ensure that the solution provides end-to-end security. From the data point of view, it is vital to have the capabilities for secure data at rest and secure data in transit capabilities, especially if the workloads deal with sensitive data or are under regulatory compliances. In this section, we will go through the security features available with IBM Spectrum Scale that can be leveraged for securing data for containers.
6.2.1 Secure data at rest for container data
In real world deployments there will be a need for some containers to encrypt the data that they write to the attached persistent volumes, where other containers might not have that need. This requires the flexibility for the underlying storage to have policies to enable that functionality. Additionally, some compliance-driven workloads or microservices running inside the container might require FIPS compliant secure data at rest.
IBM Spectrum Scale supports FIPS compliant secure data at rest, which enables granular encryption of data when written to the file system depending upon the policy. It supports standard key managers, such as IBM SKLM and Vormetric, required for managing the keys used for encryption. The encryption policies enable administrators to control what needs to be encrypted and what is not required to be encrypted.
This IBM Spectrum Scale feature easily integrates to help secure the data at rest for containers where the persistent volume attached to the containers is nothing but either a file set or a directory on IBM Spectrum Scale file system. As a hypothetical example, suppose we need to run a workload inside a container associated with a POD that creates two sets of files
*.doc
*.txt
There is a need that all *.doc files are encrypted when a given container writes them to the attached volume, but the .txt files remain unencrypted. Following are sample pseudo steps which one can use to achieve this requirement using static provisioning:
1. Configure IBM Spectrum Scale with file system encryption using a supported key manager. For details, see IBM Knowledge Center for IBM Spectrum Scale regarding Preparation for encryption:
2. Create an independent file set called FilesetA.
3. Set the required encryption polices on FilesetA as shown in Example 6-7.
Example 6-7 Example of encryption policy rules
RULE 'MYENCRULE1' ENCRYPTION 'E1' IS ALGO 'DEFAULTNISTSP800131A' KEYS('1:RKM_1', '2:RKM_2')
 
RULE 'Encrypt files with extension doc with rule E1' SET ENCRYPTION 'E1' FOR file set('FilesetA') WHERE NAME LIKE '%.doc'
4. Create a PV by providing the FilesetA details.
5. Bind a PVC to the created PV.
6. Attach the POD to the above created PVC ensuring the specific container in the POD is associated with that PVC.
This will meet the previously described need for granular secure data at rest required for a given workload. To make such a requirement more generic, one can create storage classes associated with independent File sets that are configured with IBM Spectrum Scale encryption policies. These storage classes can then be used to dynamically provision persistent volumes associated with that encryption policy. In this case, the files adhering to the encryption policy will be encrypted before they are stored to the disk.
6.2.2 Secure data in transit
When volumes are provisioned to containers/POD via external file storage, a question that comes up is if the data in transit between the attached volume and back-end storage is secured or not. In an IBM Spectrum Scale setup, the volumes attached to containers/POD are local bind mounted from the underlying IBM Spectrum Scale client running on the host. Therefore, the data being written on the volume by the container/POD translates locally and not “over the wire”. Now, the IBM Spectrum Scale client further stripes and stores this data on the disks via the IBM Spectrum Scale network shared disk (NSD) servers.
There might be a need to encrypt all of the communication between the IBM Spectrum Scale clients and servers, which will include the data from the containers/POD. To cater to this requirement, IBM Spectrum Scale supports secure data over wire. IBM Spectrum Scale cluster configuration has different security modes that can be changed using the mmchconfig cipherlist=security_mode command. When the security mode is set to one of the supported ciphers, all of the inter-node communication within the IBM Spectrum Scale cluster will be encrypted using the mentioned cipher.
Example: mmchconfig cipherList=AES256-SHA256
 
Note: The default mode of cipherList is AUTHONLY, which ensures that the inter-node traffic is authenticated. This caters to requirements for most deployments. Note that changing the cipherList to one of the supported encryption ciphers might impact the performance of the workload.
For more information, see IBM Knowledge Center for IBM Spectrum Scale in the Security mode section:
In case of remote cluster mounts, IBM Spectrum Scale supports secure data communication between clusters using mmauth commands. For more information see the “Managing a Remote GPFS file system” topic in IBM Knowledge Center for IBM Spectrum Scale:
By securing inter-node IBM Spectrum Scale cluster communication, one can ensure secure data in transit for containerized workloads.