lbcli Upgrade Server/Cluster

The upgrade can be run on live systems. You can upgrade the Lightbits version by using the lbcli tool. You can upgrade the servers one by one with the lbcli upgrade server command, or the entire cluster at once with the lbcli upgrade cluster command.

During a server upgrade, the server will reboot once to complete the upgrade and properly boot into the new version. During a cluster upgrade, the servers will upgrade one by one, and reboot at separate times to maintain volume connectivity.

This mandates that the cluster be healthy prior to the start of the upgrade operation. As a result, the cluster is ensured to be healthy before continuing with the next server upgrade.

Starting from Lightbits version 3.6.1, there will be no server reboots. For additional information, see the Lightbits Release Notes.

lbcli Upgrade Server

Get the cluster status.

Bash
    
 
lbcli get cluster -o yaml
Copy

Before running the lbcli upgrade command, test the repo-uri with wget and append .rpm to make sure it is reachable. For example: wget <repo-uri>.rpm.

Sample Output

Bash
    
 
lbcli get cluster -o yamlETag: "0"MaxAllowedVersion: 3.3.X  <--- Users can upgrade until this version.MinAllowedVersion: 3.2.X  <--- The minimum version that this cluster can support.MinVersionInCluster: 3.2.1~b1177 <--- The minimum version that is currently installed.UUID: 62b84119-1f4d-47fe-87d7-d0515a6c49f7apiEndpoints:- 10.10.10.100:443- 192.168.17.223:443- 10.10.10.101:443- 192.168.17.220:443- 10.10.10.102:443- 192.168.17.221:443clusterName: My-ClustercurrentMaxReplicas: 3discoveryEndpoints:- 10.10.10.100:8009- 10.10.10.101:8009- 10.10.10.102:8009health:  numDegradedVolumes: 0  numInactiveNodes: 0  numNotAvailableVolumes: 0  numReadOnlyVolumes: 0  state: OK                       <--- The health must be OK in order to perform the upgrade.statistics:  compressionRatio: 1  effectivePhysicalStorage: "13809663442943"  estimatedFreeLogicalStorage: "13809663442943"  estimatedLogicalStorage: "13809663442943"  freePhysicalStorage: "13809663442943"  installedPhysicalStorage: "45679219826688"  logicalStorage: "8589934592"  logicalUsedStorage: "0"  managedPhysicalStorage: "21164236111872"  physicalUsedStorage: "0"  physicalUsedStorageIncludingParity: "0"subsystemNQN: nqn.2016-01.com.lightbitslabs:uuid:fbe4a065-4b9e-4772-9666-e83691bacc63supportedMaxReplicas: 3
Copy

Check the status of the nodes (all nodes must be in Active state):

Bash
    
 
lbcli list nodes
Copy

Sample Output

Bash
    
lbcli list nodesName         UUID                                   State      NVMe endpoint       Failure domains   Local rebuild progressserver00-0   d3994d3b-d3c4-565f-8e73-54b5e0c34ab7   Active     10.10.10.100:4420   [server00]        Noneserver01-0   e964223a-fab4-51f6-b31f-6abbc0d1f9c3   Active     10.10.10.101:4420   [server01]        Noneserver02-0   edb8bd43-8090-59af-8d2d-aa8b2375793a   Active     10.10.10.102:4420   [server02]        None
Copy

Check the status of the servers (all nodes must be in Enabled state):

Bash
    
 
lbcli list servers
Copy

Sample Output

Bash
    
 
lbcli list serverNAME       UUID                                   State      RiskOfServiceLoss State     LightOSVersionserver02   f1171a62-7f41-5330-9d5d-b826f779ad2a   Enabled    NoRiskOfServiceLoss         2.3.17~b923server00   82dd376e-9b12-5b0b-86ea-d5b5dfeb8961   Enabled    NoRiskOfServiceLoss         2.3.17~b923server01   2b2767c3-802b-58d8-8810-55d525bd624b   Enabled    NoRiskOfServiceLoss         2.3.17~b923
Copy

Upgrade Server from Remote Repo

Run the upgrade command:

Bash
    
 
lbcli upgrade server --install-pkg-uri="<repo-uri>" --uuid=<server-uuid>
Copy

When issuing commands, a number of checks are run, and an error is returned if any of the following occurs:

Lightbits cannot perform an upgrade from the current version to a specified version.
The cluster is not healthy, and the upgrade operation causes a risk of loss of service.
Additional checks will be run in the background at a later stage of the upgrade operation execution. If any of these checks fail and the upgrade operation is failed or skipped, a detailed error will be returned via an event.
To monitor for the completion of the upgrade operation, either poll list servers and check for the change in the server version, or poll list events.
The cluster is healthy, but some volumes will not be fully protected (this will block the upgrade).
Yum is locked.

Example

Bash
    
 
lbcli list events --component-type=server -o json
Copy

Example

Bash
    
lbcli upgrade server --install-pkg-uri="https://dl.lightbitslabs.com/<YOUR_TOKEN>/lightos-3-<Minor Ver>-x-ga/rpm/el/7/x86_64/lightos-3.0.1~b1004-1.x86_64" --uuid=ac57df6e-4fe9-5ba2-980c-388de6cb4740
Copy

The repo URL must include the lightos version (in this example: lightos-3.0.1b1004-1.x86_64). The repo URI should be a concatenation of the path to the repo that stores the Lightbits packages and the name of the package; i.e: hosting repo: "https:///lightbits", package name: "lightos-3.2.1b1178-1.x86_64.rpm", URI: "https:///lightbits/lightos-3.2.1~b1178-1.x86_64.rpm"
Until the end of the upgrade process, the server will be in a disable state and with SourceOfRiskOfServiceLoss warning, and the node will be in an Inactive state.
The upgrade should take 10-40 minutes to complete. It is composed of a few stages that could vary in time from system to system. Fetch the RPMs from the specified repository, reboot the time of the server, and power up the time of the Lightbits node.
Some servers may enter a state of InRiskOfServiceLoss (in clusters of three nodes, all other servers will be InRiskOfServiceLoss. In larger clusters it may only affect some of the other servers).
The upgrade operation will run in the background. The output of this command is {} in case it passes initial checks. You should then wait and poll until the server is up, the version has updated to the desired one, and the server's nodes are in Active state.
Upgrading a single-server cluster is supported only through the upgrade server command and not via the upgrade cluster command.

Upgrade Server from File

On each server in the cluster, copy the Lightbits RPM files to an empty folder on the local server, the repo directory.

Then run: createrepo (path of the folder). @

Note that this command will create a "repodata" directory that contains metadata. The RPM files must sit alongside the "repodata" directory and not within a sub-directory. If there are updates inside of the repo directory, rerun the full createrepo command to update the metadata within the repodata.

Bash
    
 
createrepo <rpms folder name>
Copy

Example

Bash
    
 
createrepo lightos_rpms/
Copy

Run the upgrade command:

Bash
    
 
lbcli upgrade server --install-pkg-uri="<lightos file location>" --uuid=<server-uuid>
Copy

Example

Bash
    
lbcli upgrade server --install-pkg-uri=file:///root/lightos_rpms/lightos-3.0.1~b1004-1.x86_64 --uuid=82dd376e-9b12-5b0b-86ea-d5b5dfeb8961
Copy

The file location must include the full lightos rpm name without the .rpm (in this example: lightos-3.0.1b1004-1.x86_64). The repo URI should be a concatenation of the path to the repo that stores the Lightbits packages and the name of the package; i.e: hosting repo: "https:///lightbits", package name: "lightos-3.2.1b1178-1.x86_64.rpm", URI: "https:///lightbits/lightos-3.2.1~b1178-1.x86_64.rpm"

Until the end of the upgrade process, the server will be in a disable state and with SourceOfRiskOfServiceLoss warning and the node will be in an Inactive state.

The upgrade should take 10-40 minutes to complete. It is composed of a few stages that could vary in time from system to system. Fetch the RPMs from the specified repository, reboot the time of the server, and power up the time of the Lightbits node.

Some servers may enter a state of InRiskOfServiceLoss (in clusters of three nodes, all other servers will be InRiskOfServiceLoss. In larger clusters it may only affect some of the other servers).

The upgrade operation will run in the background. The output of this command is {} in case it passes initial checks. You should then wait and poll until the server is up, the version has updated to the desired one, and the server's nodes are in Active state.

When the upgrade is complete, check the status of the cluster, servers, and nodes.

Sample Output

Bash
    
lbcli get cluster -o yamlETag: "0"MaxAllowedVersion: 3.3.X  <--- Users can upgrade until this version.MinAllowedVersion: 3.2.X  <--- The minimum version that this cluster can support.MinVersionInCluster: 3.2.1~b1177  <--- The minimum version will still have two servers with 3.2.1. When all the servers are upgraded, this will change.UUID: 62b84119-1f4d-47fe-87d7-d0515a6c49f7apiEndpoints:- 10.10.10.100:443- 192.168.17.223:443- 10.10.10.101:443- 192.168.17.220:443- 10.10.10.102:443- 192.168.17.221:443clusterName: My-ClustercurrentMaxReplicas: 3discoveryEndpoints:- 10.10.10.100:8009- 10.10.10.101:8009- 10.10.10.102:8009health:  numDegradedVolumes: 0  numInactiveNodes: 0  numNotAvailableVolumes: 0  numReadOnlyVolumes: 0  state: OK                       <--- Need to return to OK statestatistics:  compressionRatio: 1  effectivePhysicalStorage: "13809663442943"  estimatedFreeLogicalStorage: "13809663442943"  estimatedLogicalStorage: "13809663442943"  freePhysicalStorage: "13809663442943"  installedPhysicalStorage: "45679219826688"  logicalStorage: "8589934592"  logicalUsedStorage: "0"  managedPhysicalStorage: "21164236111872"  physicalUsedStorage: "0"  physicalUsedStorageIncludingParity: "0"subsystemNQN: nqn.2016-01.com.lightbitslabs:uuid:fbe4a065-4b9e-4772-9666-e83691bacc63supportedMaxReplicas: 3
Copy

Bash
    
lbcli list nodeName         UUID                                   State     NVMe endpoint       Failure domains   Local rebuild progressserver00-0   d3994d3b-d3c4-565f-8e73-54b5e0c34ab7   Active    10.10.10.100:4420   [server00]        Noneserver01-0   e964223a-fab4-51f6-b31f-6abbc0d1f9c3   Active    10.10.10.101:4420   [server01]        Noneserver02-0   edb8bd43-8090-59af-8d2d-aa8b2375793a   Active    10.10.10.102:4420   [server02]        None
Copy

Bash
    
 
lbcli list serverNAME       UUID                                   State     RiskOfServiceLoss State   LightOSVersionserver01   2b2767c3-802b-58d8-8810-55d525bd624b   Enabled   NoRiskOfServiceLoss       2.3.17~b923server02   f1171a62-7f41-5330-9d5d-b826f779ad2a   Enabled   NoRiskOfServiceLoss       3.0.1~b1004server00   82dd376e-9b12-5b0b-86ea-d5b5dfeb8961   Enabled   NoRiskOfServiceLoss       2.3.17~b923
Copy

lbcli Upgrade Cluster

Run the upgrade command:

Bash
    
 
lbcli upgrade cluster --install-pkg-uri="<repo-uri>"
Copy

Before running the lbcli upgrade command, test the repo-uri with wget and append .rpm to make sure it is reachable. For example: wget <repo-uri>.rpm.

Example

Bash
    
lbcli upgrade cluster --install-pkg-uri="https://dl.lightbitslabs.com/<YOUR_TOKEN>/lightos-3-<Minor Ver>-x-ga/rpm/el/7/x86_64/lightos-3.0.1~b1004-1.x86_64"
Copy

Until the end of the upgrade process, the server will be in a disable state and with SourceOfRiskOfServiceLoss warning and the node will be in an Inactive state.
The upgrade should take 10-40 minutes to complete. It is composed of a few stages that could vary in time from system to system. Fetch the RPMs from the specified repository, reboot the time of the server, and power up the time of the Lightbits node.
Some servers may enter a state of InRiskOfServiceLoss (in clusters of three nodes, all other servers will be InRiskOfServiceLoss. In larger clusters it may only affect some of the other servers).
The upgrade operation will run in the background. The output of this command is {} in case it passes initial checks. You should then wait and poll until the server is up, the version has updated to the desired one, and the server's nodes are in Active state.
When the upgrade is complete, check the status of the cluster, servers, and nodes.

Sample Output

Bash
    
 
lbcli get cluster -o yamlETag: "0"MaxAllowedVersion: 3.1.XMinAllowedVersion: 3.0.XMinVersionInCluster: 3.0.1~b1004   <--- The new minimum version.UUID: 62b84119-1f4d-47fe-87d7-d0515a6c49f7apiEndpoints:- 10.10.10.100:443- 192.168.17.223:443- 10.10.10.101:443- 192.168.17.220:443- 10.10.10.102:443- 192.168.17.221:443clusterName: My-ClustercurrentMaxReplicas: 3discoveryEndpoints:- 10.10.10.100:8009- 10.10.10.101:8009- 10.10.10.102:8009health:  numDegradedVolumes: 0  numInactiveNodes: 0  numNotAvailableVolumes: 0  numReadOnlyVolumes: 0  state: OK    <--- Need to return to OK state.nleState: DisabledNLEstatistics:  compressionRatio: 1  effectivePhysicalStorage: "13809663442943"  estimatedFreeLogicalStorage: "13809663442943"  estimatedLogicalStorage: "13809663442943"  freePhysicalStorage: "13809663442943"  installedPhysicalStorage: "45679219826688"  logicalStorage: "8589934592"  logicalUsedStorage: "0"  managedPhysicalStorage: "21164236111872"  physicalUsedStorage: "0"  physicalUsedStorageIncludingParity: "0"subsystemNQN: nqn.2016-01.com.lightbitslabs:uuid:fbe4a065-4b9e-4772-9666-e83691bacc63supportedMaxReplicas: 3
Copy

Bash
    
lbcli list nodeName         UUID                                   State     NVMe endpoint       Failure domains   Local rebuild progressserver00-0   d3994d3b-d3c4-565f-8e73-54b5e0c34ab7   Active    10.10.10.100:4420   [server00]        Noneserver01-0   e964223a-fab4-51f6-b31f-6abbc0d1f9c3   Active    10.10.10.101:4420   [server01]        Noneserver02-0   edb8bd43-8090-59af-8d2d-aa8b2375793a   Active    10.10.10.102:4420   [server02]        None
Copy

Bash
    
 
lbcli list serverNAME       UUID                                   State     RiskOfServiceLoss State   LightbitsVersionserver00   82dd376e-9b12-5b0b-86ea-d5b5dfeb8961   Enabled   NoRiskOfServiceLoss       3.0.1~b1004server02   f1171a62-7f41-5330-9d5d-b826f779ad2a   Enabled   NoRiskOfServiceLoss       3.0.1~b1004server01   2b2767c3-802b-58d8-8810-55d525bd624b   Enabled   NoRiskOfServiceLoss       3.0.1~b1004
Copy

lbcli Upgrade Server for Red Hat

To upgrade a Lightbits cluster on a Red Hat system, you can upgrade each server one by one.

Refer to the kernel mapping in the General System Requirements article in the Lightbits Installation Guide, to see which kernel maps to the latest Lightbits version.

The example below is for upgrading to Lightbits version 3.0.1 build 1007, which requires Red Hat kernel version 4.18.0-372.19.1. Note that this kernel version can work with Lightbits version 2.3.20 build 988 and Lightbits version 3.0.1 build 1007 (for mapping, refer to the General System Requirements article in the Lightbits Installation Guide).

Follow these steps one server at a time. Move on to the next server when you fully complete the steps.

Step 1

Upgrade the kernel on the server. Refer to Red Hat's documentation to perform the kernel upgrade. Any method is supported (for example: rpm, yum, patching, etc.); below is an example using yum. However, do not reboot to a new kernel; just set the new kernel as the next boot.

First edit /etc/yum.conf and comment out the exclude statement (which prevents the kernel from upgrading).

Bash
    
 
[main]gpgcheck=1installonly_limit=3clean_requirements_on_remove=Truebest=Trueskip_if_unavailable=False# exclude=redhat-release* kernel* kmod-kvdo*
Copy

Then update the kernel with the command below. Note that it will ask to remove and delete packages and some modules or old kernels. This is ok.

Bash
    
 
yum install kernel-4.18.0-372.19.1.el8_6.x86_64
Copy

After the command completes, do not reboot. Edit /etc/yum.conf back to the original format and uncomment the exclude statement.

Step 2

Upgrade the Lightbits software on a server.

Bash
    
lbcli upgrade server --install-pkg-uri=https://dl.lightbitslabs.com/<YOUR_TOKEN>/lightos-3-0-1-rhl-86/rpm/el/7/x86_64/lightos-3.0.1~b1007-1.x86_64 --uuid=ac57df6e-4fe9-5ba2-980c-388de6cb4740
Copy

Before running the upgrade command, you can test the repo by appending .rpm to wget: Example: wget https://dl.lightbitslabs.com/<YOUR_TOKEN>/lightos-3-0-1-rhl-86/rpm/el/7/x86_64/lightos-3.0.1~b1007-1.x86_64.rpm

The upgrade will perform a reboot.

Step 3

Move on to the next server.

lbcli Upgrade Cluster for Red Hat

To upgrade a Lightbits cluster on a Red Hat system, you can upgrade the cluster with one command.

Refer to the kernel mapping in the General System Requirements article in the Lightbits Installation Guide, to see which kernel maps to the latest Lightbits version.

Follow these steps one server at a time. Move on to the next server when fully completing the steps.

Step 1

Upgrade the kernel on each server. Refer to Red Hat's documentation to perform the kernel upgrade. Any method is supported (for example: rpm, yum, patching, etc.); below is an example using yum. However, do not reboot the servers to a new kernel. Just set the new kernel as the next boot.

First edit /etc/yum.conf and comment out the exclude statement (which prevents the kernel from upgrading).

Bash
    
 
[main]gpgcheck=1installonly_limit=3clean_requirements_on_remove=Truebest=Trueskip_if_unavailable=False# exclude=redhat-release* kernel* kmod-kvdo*
Copy

Then update the kernel with the command below. Note that it will ask to remove and delete packages and some modules or old kernels. This is ok.

Bash
    
 
yum install kernel-4.18.0-372.19.1.el8_6.x86_64
Copy

After the command completes, do not reboot. Edit /etc/yum.conf back to the original format and uncomment the exclude statement.

Step 2

Upgrade the Lightbits software on the servers.

Bash
    
lbcli upgrade cluster --install-pkg-uri=https://dl.lightbitslabs.com/<YOUR_TOKEN>/lightos-3-0-1-rhl-86/rpm/el/7/x86_64/lightos-3.0.1~b1007-1.x86_64
Copy

The upgrade will upgrade a server and reboot it. Once that server is done, it will move on to the next server and repeat. If there are any issues, it will stop.

Last updated on

Was this page helpful?