lbcli Upgrade Server/Cluster

AI Tools

The upgrade can be run on live systems. You can upgrade the Lightbits version by using the lbcli tool. You can upgrade the servers one by one with the lbcli upgrade server command, or the entire cluster at once with the lbcli upgrade cluster command.

During a server upgrade, the server will reboot once to complete the upgrade (only when needed) and properly boot into the new version. During a cluster upgrade, the servers will upgrade one by one, and reboot at separate times to maintain volume connectivity.

This mandates that the cluster be healthy prior to the start of the upgrade operation. As a result, the cluster is ensured to be healthy before continuing with the next server upgrade.

Note

Starting from Lightbits version 3.6.1, there will be no server reboots. For additional information, see the Lightbits Release Notes.

lbcli Upgrade Server

Get the cluster status.

lbcli get cluster

Note

Before running the lbcli upgrade command, test the repo-uri with wget and append .rpm to make sure it is reachable. For example: wget <repo-uri>.rpm.

Sample Output

lbcli get cluster UUID Subsystem NQN Current max replicas Supported max replicas MinVersionInCluster MinAllowedVersion MaxAllowedVersion f2f31b84-6955-5f5d-a363-aca4d35a3efd nqn.2016-01.com.lightbitslabs:uuid:ddf7d805-cc01-4f74-81d1-00f8f276c6d6 3 3 3.16.1~b15961757728 3.16.X 3.19.X [

Check the status of the nodes (all nodes must be in Active state):

lbcli list nodes

Sample Output

lbcli list nodes Name UUID State NVMe endpoint Failure domains Capacity Used Local rebuild progress server00-0 4f1a4ef8-0af5-5e08-af99-6b882a218b03 Active 172.17.2.1:4420 [server00] 15 TiB 399 GiB None server00-1 3f37e9ae-20e8-5b37-94a1-4a2cd999ce6d Active 172.17.1.1:4420 [server00] 15 TiB 490 GiB None server01-0 e987ce54-db6d-57e2-9116-0a68a8f633bd Active 172.17.2.2:4420 [server01] 15 TiB 455 GiB None server01-1 fc4444d9-008a-593f-aeef-fe370b86aa77 Active 172.17.1.2:4420 [server01] 15 TiB 425 GiB None server02-0 5b2f1c0f-1c07-5add-9a17-9a59d771d05e Active 172.17.2.3:4420 [server02] 15 TiB 352 GiB None server02-1 cb0e9419-dede-54d6-910e-6ae3544d52d2 Active 172.17.1.3:4420 [server02] 15 TiB 490 GiB None

Check the status of the servers (all nodes must be in Enabled state):

lbcli list servers

Sample Output

[root@18-03 ~]# lbcli list servers NAME UUID State RiskOfServiceLoss State LightOSVersion server00 589a688d-b92d-536c-b684-8aa6d1ee130e Enabled NoRiskOfServiceLoss 3.16.1~b15961757728 server01 720a40b1-89fb-5ec0-95d2-5ef6ccef3bea Enabled NoRiskOfServiceLoss 3.16.1~b15961757728 server02 01289a96-4719-5ec2-86c7-90f992713975 Enabled NoRiskOfServiceLoss 3.16.1~b15961757728

Upgrade Server from Remote Repo

Run the upgrade command:

lbcli upgrade server --install-pkg-uri="<repo-uri>" --uuid=<server-uuid>

When issuing commands, a number of checks are run, and an error is returned if any of the following occurs:

Lightbits cannot perform an upgrade from the current version to a specified version.
The cluster is not healthy, and the upgrade operation causes a risk of loss of service.
Additional checks will be run in the background at a later stage of the upgrade operation execution. If any of these checks fail and the upgrade operation is failed or skipped, a detailed error will be returned via an event.
To monitor for the completion of the upgrade operation, either poll list servers and check for the change in the server version, or poll list events.
The cluster is healthy, but some volumes will not be fully protected (this will block the upgrade).
Yum is locked.

Example

lbcli list events --component-type=server -o json

Example

lbcli upgrade server --install-pkg-uri="https://dl.lightbitslabs.com/<YOUR_TOKEN>/lightos-3-<Major-Minor>-rhl-<8/9>/rpm/el/any-version/x86_64/lightos-3.17.1b19460224059-1.el8.x86_64.rpm" --uuid=ac57df6e-4fe9-5ba2-980c-388de6cb4740

Notes

Until the end of the upgrade process, the server will be in a disable state and with SourceOfRiskOfServiceLoss warning, and the node will be in an Inactive state.
The upgrade should take 10-40 minutes to complete. It is composed of a few stages that could vary in time from system to system. Fetch the RPMs from the specified repository, reboot (when needed) the time of the server, and power up the time of the Lightbits node.
Some servers may enter a state of InRiskOfServiceLoss (in clusters of three nodes, all other servers will be InRiskOfServiceLoss. In larger clusters it may only affect some of the other servers).
The upgrade operation will run in the background. The output of this command is {} in case it passes initial checks. You should then wait and poll until the server is up, the version has updated to the desired one, and the server's nodes are in Active state.
Upgrading a single-server cluster is supported only through the upgrade server command and not via the upgrade cluster command.

Upgrade Server from File

On each server in the cluster, copy the Lightbits RPM files to an empty folder on the local server - the repo directory.

Then run: createrepo (path of the folder).

Note that this command will create a "repodata" directory that contains metadata. The RPM files must sit alongside the "repodata" directory and not within a sub-directory. If there are updates inside of the repo directory, rerun the full createrepo command to update the metadata within the repodata.

createrepo <rpms folder name>

Example

createrepo lightos_rpms/

Run the upgrade command:

lbcli upgrade server --install-pkg-uri="<lightos file location>" --uuid=<server-uuid>

Example

lbcli upgrade server --install-pkg-uri=file:///root/lightos_rpms/lightos-3.17.1b19460224059-1.el8.x86_64.rpm --uuid=82dd376e-9b12-5b0b-86ea-d5b5dfeb8961

Notes

Until the end of the upgrade process, the server will be in a disable state and with SourceOfRiskOfServiceLoss warning, and the node will be in an Inactive state.
The upgrade should take 10-40 minutes to complete. It is composed of a few stages that could vary in time from system to system. Fetch the RPMs from the specified repository, reboot (when needed) the time of the server, and power up the time of the Lightbits node.
Some servers may enter a state of InRiskOfServiceLoss (in clusters of three nodes, all other servers will be InRiskOfServiceLoss. In larger clusters it may only affect some of the other servers).
The upgrade operation will run in the background. The output of this command is {} in case it passes initial checks. You should then wait and poll until the server is up, the version has updated to the desired one, and the server's nodes are in Active state.

When the upgrade is complete, check the status of the cluster, servers, and nodes.

Sample Output

lbcli Upgrade Cluster

Run the upgrade command:

lbcli upgrade cluster --install-pkg-uri="<repo-uri>"

Note

Before running the lbcli upgrade command, test the repo-uri with wget and append .rpm to make sure it is reachable. For example: https://dl.lightbitslabs.com/<YOUR_TOKEN>/lightos-3-17-1-rhl-8/rpm/el/7/x86-64/lightos-3.17.1b19460224059-1.el8.x86_ 64.rpm

Example

lbcli upgrade cluster --install-pkg-uri="https://dl.lightbitslabs.com/<YOUR_TOKEN>/lightos-3-<Major-Minor>-rhl-<8/9>/rpm/el/any-version/x86_64/lightos-3.17.1b19460224059-1.el8.x86_64.rpm"

Notes

Until the end of the upgrade process, the server will be in a disable state and with a SourceOfRiskOfServiceLoss warning - and the node will be in an Inactive state.
The upgrade should take 10-40 minutes to complete. It is composed of a few stages that could vary in time from system to system. Fetch the RPMs from the specified repository, reboot (when needed) the time of the server, and power up the time of the Lightbits node.
Some servers may enter a state of InRiskOfServiceLoss (in clusters of three nodes, all other servers will be InRiskOfServiceLoss. In larger clusters it may only affect some of the other servers).
The upgrade operation will run in the background. The output of this command is {} in case it passes initial checks. You should then wait and poll until the server is up, the version has updated to the desired one, and the server's nodes are in Active state.
When the upgrade is complete, check the status of the cluster, servers, and nodes.

Sample Output

[root@18-03 ~]# lbcli get cluster UUID Subsystem NQN Current max replicas Supported max replicas MinVersionInCluster MinAllowedVersion MaxAllowedVersion f2f31b84-6955-5f5d-a363-aca4d35a3efd nqn.2016-01.com.lightbitslabs:uuid:ddf7d805-cc01-4f74-81d1-00f8f276c6d6 3 3 3.16.1~b15961757728 3.16.X 3.19.X [

lbcli list node Name UUID State NVMe endpoint Failure domains Local rebuild progress server00-0 d3994d3b-d3c4-565f-8e73-54b5e0c34ab7 Active 10.10.10.100:4420 [server00] None server01-0 e964223a-fab4-51f6-b31f-6abbc0d1f9c3 Active 10.10.10.101:4420 [server01] None server02-0 edb8bd43-8090-59af-8d2d-aa8b2375793a Active 10.10.10.102:4420 [server02] None

lbcli list server NAME UUID State RiskOfServiceLoss State LightbitsVersion server00 82dd376e-9b12-5b0b-86ea-d5b5dfeb8961 Enabled NoRiskOfServiceLoss 3.0.1~b1004 server02 f1171a62-7f41-5330-9d5d-b826f779ad2a Enabled NoRiskOfServiceLoss 3.0.1~b1004 server01 2b2767c3-802b-58d8-8810-55d525bd624b Enabled NoRiskOfServiceLoss 3.0.1~b1004

Note

Before running the upgrade command, you can test the repo with wget: Example: wget https://dl.lightbitslabs.com/<YOUR_TOKEN>/lightos-3-17-1-rhl-8/rpm/el/7/x86-_64/lightos-3.17.1b19460224059-1.el8.x86_64.rpm

Step 3

Move on to the next server.

Tracking Upgrade Status

To track upgrade status, use lbcli list upgrade status.

lbcli list upgrade-status Cluster Upgrade Status UUID : f2f31b84-6955-5f5d-a363-aca4d35a3efd Min Version : 3.16.1~b15961757728 Upgrade Status : None Target Version : Start Time : Progress : 0/0 End Time : Error Message : NAME UUID Current Version Upgrade Status Target Version Start Time Progress End Time Error Message server00 589a688d-b92d-536c-b684-8aa6d1ee130e 3.16.1~b15961757728 None 0/4 server01 720a40b1-89fb-5ec0-95d2-5ef6ccef3bea 3.16.1~b15961757728 None 0/4 server02 01289a96-4719-5ec2-86c7-90f992713975 3.16.1~b15961757728 None 0/4

Last updated on

Was this page helpful?