Ensuring Suitable Kubernetes Cluster Configuration

CSI plugins must be able to perform system-level operations to provide other pods running on Kubernetes with usable storage volumes - either raw block devices or file system mounts. Some of these operations, like mounting file systems or accessing the sysfs pseudo-file system, require elevated privileges not normally granted to the other pods.

Therefore, to successfully deploy any CSI storage plugin—including the Lightbits CSI plugin—on a Kubernetes cluster, some security-related restrictions on Kubernetes pods must be relaxed by allowing Kubernetes to run privileged pods. Also, depending on a Kubernetes version, a number of Kubernetes “feature gates” might need to be enabled or disabled. Details about configuring the restrictions and feature gates are in the Ensuring Suitable Kubernetes API Server Configuration and Ensuring Suitable kubelet Configuration on All Cluster Nodes sections of this guide.

Additionally, different methods of deploying Kubernetes clusters may enable or disable feature gates or security features in non-standard ways, and some Kubernetes users modify some of the relevant settings of their clusters after deployment. You must confirm that the settings described in this section are set to the correct values on the Kubernetes cluster in question.

The required modifications apply to two Kubernetes components:

• The Kubernetes API server.
• The kubelet daemon runs on each cluster node.

The following sections detail the required modifications for each component.

Configuring the Kubernetes API Server

In many Kubernetes deployments, the Kubernetes API server itself runs as a static pod on the master node(s). The process of enabling privileged containers and feature gates in the API server requires updating the API server pod Kubernetes spec and ensuring that the Kubernetes API server restarts. Typically, the spec location is at the following path on the master node(s):

/etc/kubernetes/manifests/kube-apiserver.yaml

If your Kubernetes cluster uses a different method of hosting the Kubernetes API server, or if the master node(s) in your Kubernetes cluster does not have the kube-apiserver.yaml file at the indicated path, employ one of the following suggestions:

• See the Kubernetes deployment tooling documentation used to deploy your cluster.
• Contact Lightbits Support for assistance.

To enable privileged containers and feature gates in the Kubernetes API server, open the kube-apiserver.yaml in a text editor, and make sure that the command spec entry of the Kubernetes API server container (spec.containers[0].command) conforms to the following requirements:

• The --allow-privileged command line parameter:

  • Starting with Kubernetes v1.13, privileged pods are enabled by default in Kubernetes.
  • If this parameter is present in the file, it should be specified only once, with a value of true.
  • -allow-privileged=true
  • If this parameter is absent - do not add it, it will take on the value of true by default, as it should.

• In all supported Kubernetes versions (v1.16 and later), the below five --feature-gates parameter entries are enabled by default. If this parameter is present and includes any of the following feature gates, they must be set to a value of true or removed from the list: BlockVolume=true CSIBlockVolume=true CSIPersistentVolume=true KubeletPluginsWatcher=true MountPropagation=true

If the --feature-gates command line parameter is absent or does not include any of these feature gates, they will take on the value of true by default, as they should.

The below shows part of the kube-apiserver.yaml for reference:

apiVersion: v1

kind: Pod

metadata:

  annotations: 

    scheduler.alpha.kubernetes.io/critical-pod: '' 

  creationTimestamp: null

  labels: 

    component: kube-apiserver

    tier: control-plane 

  name: kube-apiserver

  namespace: kube-system

spec:

  containers:

  - command:

    - kube-apiserver

    ...

    - --allow-privileged=true

    ...

    - --feature-gates=CSIDriverRegistry=true,CSINodeInfo=true

    image: registry.k8s.io/kube-apiserver:v1.22.5

    ...

If you have made any changes to the kube-apiserver.yaml file, save and close the file. The kubelet daemon on that Kubernetes node should be monitoring the static pod manifests directory, detect the modification, and automatically restart the Kubernetes API server container with the updated parameters.

Configuring kubelet on All Cluster Nodes

In most Kubernetes deployments, kubelet runs as a daemon or service on each of the cluster nodes. The process of enabling privileged containers and feature gates in kubelet comprises updating the kubelet configuration on each of the Kubernetes cluster nodes and ensuring that all the relevant kubelet daemons are restarted.

Unfortunately, the method of passing command-line parameters to the kubelet daemon depends to a significant degree on the OS running on the Kubernetes cluster nodes, the “init” system used to manage services on those nodes, and the method used to deploy the Kubernetes cluster. The following instructions are for one sample way to update the kubelet configuration. You will need to consult the documentation for the tool used to deploy your Kubernetes cluster for instructions pertinent to your Kubernetes cluster. If you require further assistance, contact Lightbits Support.

To perform this configuration update, you need to connect to each of the Kubernetes cluster nodes; e.g., using SSH, and ensuring that the kubelet startup script or configuration file passes in the relevant command-line flags. As indicated in the preceding note, the exact location of such a startup script or configuration can vary. Often the kubelet parameters are controlled by a number of environment variables that can be set in one or more of the files that are picked up by the OS component used to launch the kubelet service.

For instance, many RHEL/CentOS v7.x based deployments using systemd use one or more of the following files to configure kubelet:

/etc/sysconfig/kubelet

/etc/default/kubelet

/etc/kubernetes/kubelet.env

Some systems can have a dedicated KUBE_ALLOW_PRIV environment variable reserved for just that use case in such files. Other systems use the more generic KUBELET_ARGS or KUBELET_EXTRA_ARGS, and some systems might have both or neither.

In any case, after you have identified the method used to pass the command-line parameters to kubelet, you need to confirm that the following parameters match the requirements specified in the Kubernetes API Server Configuration subsection:

• The --allow-privileged command line parameter.
• The --feature-gates command line parameter and the comma-separated list of its values.

If you had to make any changes to the kubelet configuration, you must restart the kubelet service on the affected Kubernetes cluster node—or reboot the node altogether—to complete the process. No changes take effect until you restart the kubelet service.

Often there are multiple locations or environment variables used to specify the kubelet command-line parameter. Therefore, to avoid misconfiguration, after restarting kubelet check that the command-line flags listed above are passed to kubelet only once and with correct values, or not passed at all if you are relying on defaults instead. For example, you can use the following command and examine its output to see the actual parameters passed, and verify their presence of the correct parameters:

ps aux | grep kubelet