Extending the Lightbits Cluster

The lb-csi-plugin is stateless and holds no persistent information between operations.

Kubernetes API calls that invoke the lb-csi (using Lightbits storage), require information of the Lightbits management endpoints so that the plugin can access the Lightbits cluster.

The way to update this management endpoint list to the plugin today is via the StorageClass.Parameters.mgmt-endpoint field.

The CSI API does not pass the request. Parameters in some types of API calls to the plugin, This results in situations where the plugin needs to access the Lightbits API but does not have the required information to do so.

In order to overcome this limitation, we defined an internal resource named ResourceID which holds this information as the resource ID passed to Kubernetes.

Such a ResourceID is utilized for representing the PersistentVolume.spec.volumeHandle and VolumeSnapshotContent.status.snapshotHandle resources.

ResourceID is guarantied to pass on every API call from CSI - meaning that we can use it to hold limited state.

ResourceID format is: mgmt:<host>:<port>[,<host>:<port>...]|nguid:<nguid>|proj:<proj>|scheme:<grpc|grpcs>

When the Lightbits cluster is expanded (i.e., adding/changing servers to an existing cluster), the management-endpoints list is updated, ensuring that the lb-csi-plugin can access any of the Lightbits api-servers.

This is because the PersistentVolume.spec.volumeHandle and VolumeSnapshotContent.status.snapshotHandle fields are immutable, and they cannot be changed post-resource creation.

To mitigate this problem, we provide a one-shot script that accesses a Kubernetes cluster via standard kubectl calls, and patches the following resource with the updated information:

• StorageClass - will modify the parameters.mgmt-endpoint field with the new endpoint list.
• PersistentVolume - will modify the resource spec.volumeHandle by using the kubectl replace call.
• VolumeSnapshotContent - will replace the resource's spec.source.volumeHandle with the new updated spec.source.snapshotHandle - which will contain the new endpoint list.

This behavior of relying on 'ResourceID' will be fixed in future versions of lb-csi-plugin, once the Lightbits cluster supports VIP. All resources will point to a single endpoint, which will not change during Lightbits cluster updates.

This script should be idempotent, and should be safe to run on resources that were already updated.

Usage

Below is the patcher help output we provide to patch the existing resources in the cluster:

lightos-patcher.sh --help

​

Usage: lightos-patcher.sh [-s <storage_class>] [-e <endpoints>] [-d <backup_directory>]

-v <storage_class>              name of the storage class and all related PVs to update

-s <snapshot_storage_class>     name of the snapshot storage class and all related SnapshotContents to update

-e <endpoints>                  new endpoint list in the form of: <host:port>,<host:port>,...

-d <backup_directory>           folder to backup before and after resources

Examples:

    Suppose we have LightOS Cluster los1 with the following mgmt-endpoints:

    192.168.17.2:443,192.168.18.3:443,192.168.20.4:443

    After extending this cluster by adding a new server (192.168.20.5:443) we will have the following mgmt-endpoints:

    192.168.17.2:443,192.168.18.3:443,192.168.20.4:443,192.168.20.5:443

​

    # patch example-sc StorageClass and all PVs related to that StorageClass

    ./lightos-patcher.sh -v example-sc -e 192.168.17.2:443,192.168.18.3:443,192.168.20.4:443,192.168.20.5:443 -d ~/backup

    # patch example-sc VolumeSnapshotClass and all VolumeSnapshotContents related to that class

    ./lightos-patcher.sh -s example-snap-sc -e 192.168.17.2:443,192.168.18.3:443,192.168.20.4:443,192.168.20.5:443 -d ~/backup

The order of the commands should be:

1. Apply the script against all StorageClasss with the -v option. Verify that all StorageClass and PVs are updated.
2. If there are VolumeSnapshots on the cluster, apply the script with the -s option.

On a cluster that has existing PVs before expanding the Lightbits cluster, run the following:

./lightos-patcher.sh -v <storage-class-name> -e <new-comma-separated-endpoint-list> -d <backup-folder>

This command will:

1. Patch the StorageClass.Parameters.mgmt-endpoint with the new-comma-separated-endpoint-list value.
2. Look up all PVs in the StorageClass, and patch the PersistentVolume.spec.volumeHandle value with the new-comma-separated-endpoint-list.

On a cluster that has VolumeSnapshots created before expanding the Lightbits cluster, run the following:

./lightos-patcher.sh -s <storage-class-name> -e <new-comma-separated-endpoint-list> -d <backup-folder>

This command will:

1. Look up all VolumeSnapshotContents in this VolumeSnapshotClass and replace the VolumeSnapshotContent.spec.source.volumeHandle value with the VolumeSnapshotContent.spec.source.snapshotHandle.