Prior to release v3.18.1, to restart a Duroslight/GFTL/node instance, you were required to restart the node manager service. This operation takes time, and in the case of a dual-node server, if for example there is a problem with one Duroslight, you still need to restart the node manager service, which will impact both nodes.
The new API documented below allows you to restart only a single node-instance control plane (running inside the NM service) and only its own Duroslight service and its gftl-instance, without impacting the GFTL service (i.e., without impacting the other node’s GFTL instance). You can also perform the restart only for the node’s control plane and Duroslight instances, without its GFTL instance. If you are not restarting the GFTL instance, the restart time is shorter, and this reduces the rebuild time after the restart. The API can be executed remotely, so you can run the API on Server A to restart a node on any other server in the cluster.
In general, the API will fail if the restart causes a PG to go into read-only mode. If you must restart, in this case, you should use the --force flag to override.
There are a few options here, detailed in the table below:
| Node Instance | Duroslight | GFTL | Duration |
|---|
Restart a node | Yes | Yes | No | ~30sec |
Restart a node (and GFTL) | Yes | Yes | Yes | ~60sec |
Stop a node | Yes | Yes | No | ~3sec |
Start a node | Yes | Yes | No | ~30sec |
lbcli Examples
Status of the node: the node-manager server is up and running:
# systemctl status node-manager
● node-manager.service - Node-Manager Service
Loaded: loaded (/usr/lib/systemd/system/node-manager.service; enabled; preset: disabled)
Active: active (running) since Sun 2026-02-08 17:45:17 UTC; 1h 15min ago
Main PID: 13876 (node-manager)
Tasks: 11 (limit: 76093)
Memory: 34.0M
CPU: 4min 56.228s
Validate that the other services are running (GFTL and Duroslight):
# ps -ef | egrep 'lbe|duros'
root 14656 1 0 17:45 ? 00:00:00 /opt/backend/lbe lbe.max_num_read_errors=10 lbe.read_errors_timeframe_msec=5000 lbe.max_num_blkdev_reset_ctrl=5 lbe.blkdev_reset_ctrl_timeframe_msec=120000 lbe.develop_trim=1 lf.lf_mode=2 main.debug_level=6
root 14662 14656 99 17:45 ? 07:45:47 /opt/backend/tmp_extract/staticx-ENfOGP/lbedyn lbe max_num_read_errors 10 lbe read_errors_timeframe_msec 5000 lbe max_num_blkdev_reset_ctrl 5 lbe blkdev_reset_ctrl_timeframe_msec 120000 lbe develop_trim 1 lf lf_mode 2 main debug_level 6
root 15483 1 99 17:46 ? 02:33:59 /opt/duroslight/duroslight --poll-mode --lock-memory=true --reactor-backend=epoll --blocked-reactor-notify-ms=200 --blocked-reactor-reports-per-minute=300 --logger-stdout-timestamps=none --hugepages /mnt/huge --cpuset 7-8 --abort-on-seastar-bad-alloc -m2147483648 --instance-uuid c5575862-a797-54e9-ab99-10b2a869c3b7 --instance-id 0 --conf-file /etc/duroslight/conf.yaml --base_ctrl_id 3 --ip-addr 10.18.82.74 --port 4420 --replicator-port 22226 --admin-core 7 --journal-device /dev/nvme1n1 --dax-device /dev/nvme1n1 --nvram-size 1073741824 --nvram-offset 0 --subsysnqn nqn.2016-01.com.lightbitslabs:uuid:d630288d-f1c6-42ee-a531-01ef29fd8d9e --use-adq false --rx-dram false --disable-header false --lbe-usermode true --develop-trim false --encrypt-user-data true
Example of restarting a node instance (Prompt Y to verify):
[root@rack08-server71-vm04 ~]# lbcli restart node-instance --node-uuid c5575862-a797-54e9-ab99-10b2a869c3b7
WARNING: Restarting node instance c5575862-a797-54e9-ab99-10b2a869c3b7
This is a RISKY operation that will restart the node instance for this node.
Safety checks will be performed to prevent data unavailability.
Backend instance will NOT be restarted (duroslight and control plane only).
Type 'yes' or 'y' to confirm, or anything else to cancel: y
Example of restarting a node instance and GFTL (--consider-backend=true):
lbcli restart node-instance --node-uuid c5575862-a797-54e9-ab99-10b2a869c3b7 --consider-backend=true
WARNING: Restarting node instance c5575862-a797-54e9-ab99-10b2a869c3b7
This is a RISKY operation that will restart the node instance for this node.
Safety checks will be performed to prevent data unavailability.
Type 'yes' or 'y' to confirm, or anything else to cancel: y
Example of how to restart a node, even if it will cause a PG to become read-only:
# lbcli restart node-instance --node-uuid c5575862-a797-54e9-ab99-10b2a869c3b7 --force
WARNING: Restarting node instance c5575862-a797-54e9-ab99-10b2a869c3b7
This is a RISKY operation that will restart the node instance for this node.
FORCE mode enabled: volumes may temporarily become unavailable or read-only!
Backend instance will NOT be restarted (duroslight and control plane only).
Type 'yes' or 'y' to confirm, or anything else to cancel: y
If you try to restart or stop a node instance when another node is down (which could cause PGs to go into read-only state), the API will fail. This is a safeguard, and if it is required, you should use the --force flag:
#lbcli list nodes
Name UUID State NVMe endpoint Failure domains Capacity Used Local rebuild progress
server_0-0 c5..b7 Active 10.18.82.74:4420 [vm04] 14 GiB 0 B
server_1-0 4d..da Inactive 10.18.82.5:4420 [vm05] 14 GiB 0 B
server_2-0 9a..4f Active 10.18.82.100:4420 [vm03] 14 GiB 0 B
# lbcli stop node-instance --node-uuid c5575862-a797-54e9-ab99-10b2a869c3b7
WARNING: Stopping node instance c5575862-a797-54e9-ab99-10b2a869c3b7
This is a RISKY operation that will stop the node instance for this node.
Safety checks will be performed to prevent data unavailability.
Type 'yes' or 'y' to confirm, or anything else to cancel: y
Stopping node instance c5575862-a797-54e9-ab99-10b2a869c3b7 failed: rpc error: code = FailedPrecondition desc = cannot stop node c5575862-a797-54e9-ab99-10b2a869c3b7: volumes are not fully protected. Use --force to override this check
# lbcli restart node-instance --node-uuid c5575862-a797-54e9-ab99-10b2a869c3b7
WARNING: Restarting node instance c5575862-a797-54e9-ab99-10b2a869c3b7
This is a RISKY operation that will restart the node instance for this node.
Safety checks will be performed to prevent data unavailability.
Backend instance will NOT be restarted (duroslight and control plane only).
Type 'yes' or 'y' to confirm, or anything else to cancel: y
Restarting node instance c5575862-a797-54e9-ab99-10b2a869c3b7 failed: rpc error: code = FailedPrecondition desc = cannot stop node c5575862-a797-54e9-ab99-10b2a869c3b7: volumes are not fully protected. Use --force to override this check.
Example of stopping a node instance (Prompt Y to verify):
# lbcli restart node-instance --node-uuid c5575862-a797-54e9-ab99-10b2a869c3b7
WARNING: Restarting node instance c5575862-a797-54e9-ab99-10b2a869c3b7
This is a RISKY operation that will restart the node instance for this node.
Safety checks will be performed to prevent data unavailability.
Backend instance will NOT be restarted (duroslight and control plane only).
Type 'yes' or 'y' to confirm, or anything else to cancel: y
Successfully stopped node instance c5575862-a797-54e9-ab99-10b2a869c3b7
Example of starting a node instance (Prompt Y to verify):
# lbcli start node-instance --node-uuid c5575862-a797-54e9-ab99-10b2a869c3b7
WARNING: Starting node instance c5575862-a797-54e9-ab99-10b2a869c3b7
This operation will start the node instance for this node.
Type 'yes' or 'y' to confirm, or anything else to cancel: y
Successfully started node instance c5575862-a797-54e9-ab99-10b2a869c3b7
