The upgrade can be run on live systems. You can upgrade the Lightbits version by using the lbcli tool. You can upgrade the servers one by one with the lbcli upgrade server command, or the entire cluster at once with the lbcli upgrade cluster command.
During a server upgrade, the server will reboot once to complete the upgrade (only when needed) and properly boot into the new version. During a cluster upgrade, the servers will upgrade one by one, and reboot at separate times to maintain volume connectivity.
This mandates that the cluster be healthy prior to the start of the upgrade operation. As a result, the cluster is ensured to be healthy before continuing with the next server upgrade.
Starting from Lightbits version 3.6.1, there will be no server reboots. For additional information, see the Lightbits Release Notes.
lbcli Upgrade Server
Get the cluster status.
lbcli get cluster- Before running the lbcli upgrade command, test the repo-uri with wget and append
.rpmto make sure it is reachable. For example:wget <repo-uri>.rpm.
Sample Output
lbcli get clusterUUID Subsystem NQN Current max replicas Supported max replicas MinVersionInCluster MinAllowedVersion MaxAllowedVersionf2f31b84-6955-5f5d-a363-aca4d35a3efd nqn.2016-01.com.lightbitslabs:uuid:ddf7d805-cc01-4f74-81d1-00f8f276c6d6 3 3 3.16.1~b15961757728 3.16.X 3.19.X[Check the status of the nodes (all nodes must be in Active state):
lbcli list nodesSample Output
lbcli list nodesName UUID State NVMe endpoint Failure domains Capacity Used Local rebuild progressserver00-0 4f1a4ef8-0af5-5e08-af99-6b882a218b03 Active 172.17.2.1:4420 [server00] 15 TiB 399 GiB Noneserver00-1 3f37e9ae-20e8-5b37-94a1-4a2cd999ce6d Active 172.17.1.1:4420 [server00] 15 TiB 490 GiB Noneserver01-0 e987ce54-db6d-57e2-9116-0a68a8f633bd Active 172.17.2.2:4420 [server01] 15 TiB 455 GiB Noneserver01-1 fc4444d9-008a-593f-aeef-fe370b86aa77 Active 172.17.1.2:4420 [server01] 15 TiB 425 GiB Noneserver02-0 5b2f1c0f-1c07-5add-9a17-9a59d771d05e Active 172.17.2.3:4420 [server02] 15 TiB 352 GiB Noneserver02-1 cb0e9419-dede-54d6-910e-6ae3544d52d2 Active 172.17.1.3:4420 [server02] 15 TiB 490 GiB NoneCheck the status of the servers (all nodes must be in Enabled state):
lbcli list serversSample Output
[root@18-03 ~]# lbcli list serversNAME UUID State RiskOfServiceLoss State LightOSVersionserver00 589a688d-b92d-536c-b684-8aa6d1ee130e Enabled NoRiskOfServiceLoss 3.16.1~b15961757728server01 720a40b1-89fb-5ec0-95d2-5ef6ccef3bea Enabled NoRiskOfServiceLoss 3.16.1~b15961757728server02 01289a96-4719-5ec2-86c7-90f992713975 Enabled NoRiskOfServiceLoss 3.16.1~b15961757728Upgrade Server from Remote Repo
Run the upgrade command:
lbcli upgrade server --install-pkg-uri="<repo-uri>" --uuid=<server-uuid>When issuing commands, a number of checks are run, and an error is returned if any of the following occurs:
- Lightbits cannot perform an upgrade from the current version to a specified version.
- The cluster is not healthy, and the upgrade operation causes a risk of loss of service.
- Additional checks will be run in the background at a later stage of the upgrade operation execution. If any of these checks fail and the upgrade operation is failed or skipped, a detailed error will be returned via an event.
- To monitor for the completion of the upgrade operation, either poll list servers and check for the change in the server version, or poll list events.
- The cluster is healthy, but some volumes will not be fully protected (this will block the upgrade).
- Yum is locked.
Example
lbcli list events --component-type=server -o jsonExample
lbcli upgrade server --install-pkg-uri="https://dl.lightbitslabs.com/<YOUR_TOKEN>/lightos-3-<Major-Minor>-rhl-<8/9>/rpm/el/any-version/x86_64/lightos-3.17.1b19460224059-1.el8.x86_64.rpm" --uuid=ac57df6e-4fe9-5ba2-980c-388de6cb4740- Until the end of the upgrade process, the server will be in a disable state and with SourceOfRiskOfServiceLoss warning, and the node will be in an Inactive state.
- The upgrade should take 10-40 minutes to complete. It is composed of a few stages that could vary in time from system to system. Fetch the RPMs from the specified repository, reboot (when needed) the time of the server, and power up the time of the Lightbits node.
- Some servers may enter a state of InRiskOfServiceLoss (in clusters of three nodes, all other servers will be InRiskOfServiceLoss. In larger clusters it may only affect some of the other servers).
- The upgrade operation will run in the background. The output of this command is {} in case it passes initial checks. You should then wait and poll until the server is up, the version has updated to the desired one, and the server's nodes are in Active state.
- Upgrading a single-server cluster is supported only through the
upgrade servercommand and not via theupgrade clustercommand.
Upgrade Server from File
On each server in the cluster, copy the Lightbits RPM files to an empty folder on the local server - the repo directory.
Then run: createrepo (path of the folder).
Note that this command will create a "repodata" directory that contains metadata. The RPM files must sit alongside the "repodata" directory and not within a sub-directory. If there are updates inside of the repo directory, rerun the full createrepo command to update the metadata within the repodata.
createrepo <rpms folder name>Example
createrepo lightos_rpms/Run the upgrade command:
lbcli upgrade server --install-pkg-uri="<lightos file location>" --uuid=<server-uuid>Example
lbcli upgrade server --install-pkg-uri=file:///root/lightos_rpms/lightos-3.17.1b19460224059-1.el8.x86_64.rpm --uuid=82dd376e-9b12-5b0b-86ea-d5b5dfeb8961- Until the end of the upgrade process, the server will be in a disable state and with SourceOfRiskOfServiceLoss warning, and the node will be in an Inactive state.
- The upgrade should take 10-40 minutes to complete. It is composed of a few stages that could vary in time from system to system. Fetch the RPMs from the specified repository, reboot (when needed) the time of the server, and power up the time of the Lightbits node.
- Some servers may enter a state of InRiskOfServiceLoss (in clusters of three nodes, all other servers will be InRiskOfServiceLoss. In larger clusters it may only affect some of the other servers).
- The upgrade operation will run in the background. The output of this command is {} in case it passes initial checks. You should then wait and poll until the server is up, the version has updated to the desired one, and the server's nodes are in Active state.
When the upgrade is complete, check the status of the cluster, servers, and nodes.
Sample Output
lbcli get clusterUUID Subsystem NQN Current max replicas Supported max replicas MinVersionInCluster MinAllowedVersion MaxAllowedVersionf2f31b84-6955-5f5d-a363-aca4d35a3efd nqn.2016-01.com.lightbitslabs:uuid:ddf7d805-cc01-4f74-81d1-00f8f276c6d6 3 3 3.16.1~b15961757728 3.16.X 3.19.X lbcli list nodesName UUID State NVMe endpoint Failure domains Capacity Used Local rebuild progressserver00-0 4f1a4ef8-0af5-5e08-af99-6b882a218b03 Active 172.17.2.1:4420 [server00] 15 TiB 399 GiB Noneserver00-1 3f37e9ae-20e8-5b37-94a1-4a2cd999ce6d Active 172.17.1.1:4420 [server00] 15 TiB 490 GiB Noneserver01-0 e987ce54-db6d-57e2-9116-0a68a8f633bd Active 172.17.2.2:4420 [server01] 15 TiB 455 GiB Noneserver01-1 fc4444d9-008a-593f-aeef-fe370b86aa77 Active 172.17.1.2:4420 [server01] 15 TiB 425 GiB Noneserver02-0 5b2f1c0f-1c07-5add-9a17-9a59d771d05e Active 172.17.2.3:4420 [server02] 15 TiB 352 GiB Noneserver02-1 cb0e9419-dede-54d6-910e-6ae3544d52d2 Active 172.17.1.3:4420 [server02] 15 TiB 490 GiB None[root@18-03 ~]# lbcli list serversNAME UUID State RiskOfServiceLoss State LightOSVersionserver00 589a688d-b92d-536c-b684-8aa6d1ee130e Enabled NoRiskOfServiceLoss 3.16.1~b15961757728server01 720a40b1-89fb-5ec0-95d2-5ef6ccef3bea Enabled NoRiskOfServiceLoss 3.16.1~b15961757728server02 01289a96-4719-5ec2-86c7-90f992713975 Enabled NoRiskOfServiceLoss 3.16.1~b15961757728lbcli Upgrade Cluster
Run the upgrade command:
lbcli upgrade cluster --install-pkg-uri="<repo-uri>"Before running the lbcli upgrade command, test the repo-uri with wget and append .rpm to make sure it is reachable. For example: https://dl.lightbitslabs.com/<YOUR_TOKEN>/lightos-3-17-1-rhl-8/rpm/el/7/x86-64/lightos-3.17.1b19460224059-1.el8.x86_ 64.rpm
Example
lbcli upgrade cluster --install-pkg-uri="https://dl.lightbitslabs.com/<YOUR_TOKEN>/lightos-3-<Major-Minor>-rhl-<8/9>/rpm/el/any-version/x86_64/lightos-3.17.1b19460224059-1.el8.x86_64.rpm"- Until the end of the upgrade process, the server will be in a disable state and with a SourceOfRiskOfServiceLoss warning - and the node will be in an Inactive state.
- The upgrade should take 10-40 minutes to complete. It is composed of a few stages that could vary in time from system to system. Fetch the RPMs from the specified repository, reboot (when needed) the time of the server, and power up the time of the Lightbits node.
- Some servers may enter a state of InRiskOfServiceLoss (in clusters of three nodes, all other servers will be InRiskOfServiceLoss. In larger clusters it may only affect some of the other servers).
- The upgrade operation will run in the background. The output of this command is {} in case it passes initial checks. You should then wait and poll until the server is up, the version has updated to the desired one, and the server's nodes are in Active state.
- When the upgrade is complete, check the status of the cluster, servers, and nodes.
Sample Output
[root@18-03 ~]# lbcli get clusterUUID Subsystem NQN Current max replicas Supported max replicas MinVersionInCluster MinAllowedVersion MaxAllowedVersionf2f31b84-6955-5f5d-a363-aca4d35a3efd nqn.2016-01.com.lightbitslabs:uuid:ddf7d805-cc01-4f74-81d1-00f8f276c6d6 3 3 3.16.1~b15961757728 3.16.X 3.19.X[ lbcli list nodeName UUID State NVMe endpoint Failure domains Local rebuild progressserver00-0 d3994d3b-d3c4-565f-8e73-54b5e0c34ab7 Active 10.10.10.100:4420 [server00] Noneserver01-0 e964223a-fab4-51f6-b31f-6abbc0d1f9c3 Active 10.10.10.101:4420 [server01] Noneserver02-0 edb8bd43-8090-59af-8d2d-aa8b2375793a Active 10.10.10.102:4420 [server02] Nonelbcli list serverNAME UUID State RiskOfServiceLoss State LightbitsVersionserver00 82dd376e-9b12-5b0b-86ea-d5b5dfeb8961 Enabled NoRiskOfServiceLoss 3.0.1~b1004server02 f1171a62-7f41-5330-9d5d-b826f779ad2a Enabled NoRiskOfServiceLoss 3.0.1~b1004server01 2b2767c3-802b-58d8-8810-55d525bd624b Enabled NoRiskOfServiceLoss 3.0.1~b1004###
lbcli upgrade server --install-pkg-uri="https://dl.lightbitslabs.com/<YOUR_TOKEN>/lightos-3-<Major-Minor>-rhl-<8/9>/rpm/el/any-version/x86_64/lightos-3.17.1b19460224059-1.el8.x86_64.rpm" --uuid=ac57df6e-4fe9-5ba2-980c-388de6cb4740Before running the upgrade command, you can test the repo with wget: Example: wget https://dl.lightbitslabs.com/<YOUR_TOKEN>/lightos-3-17-1-rhl-8/rpm/el/7/x86-_64/lightos-3.17.1b19460224059-1.el8.x86_64.rpm
Step 3
Move on to the next server.
Tracking Upgrade Status
To track upgrade status, use lbcli list upgrade status.
x
lbcli list upgrade-statusCluster Upgrade StatusUUID : f2f31b84-6955-5f5d-a363-aca4d35a3efdMin Version : 3.16.1~b15961757728Upgrade Status : NoneTarget Version :Start Time :Progress : 0/0End Time :Error Message :NAME UUID Current Version Upgrade Status Target Version Start Time Progress End Time Error Messageserver00 589a688d-b92d-536c-b684-8aa6d1ee130e 3.16.1~b15961757728 None 0/4 server01 720a40b1-89fb-5ec0-95d2-5ef6ccef3bea 3.16.1~b15961757728 None 0/4 server02 01289a96-4719-5ec2-86c7-90f992713975 3.16.1~b15961757728 None 0/4Was this page helpful?