Title
Create new category
Edit page index title
Edit category
Edit link
Node Instance Restart
Prior to release v3.18.1, to restart a Duroslight/GFTL/node instance, you were required to restart the node manager service. This operation takes time, and in the case of a dual-node server, if for example there is a problem with one Duroslight, you still need to restart the node manager service, which will impact both nodes.
The new API documented below allows you to restart only a single node-instance control plane (running inside the NM service) and only its own Duroslight service and its gftl-instance, without impacting the GFTL service (i.e., without impacting the other node’s GFTL instance). You can also perform the restart only for the node’s control plane and Duroslight instances, without its GFTL instance. If you are not restarting the GFTL instance, the restart time is shorter, and this reduces the rebuild time after the restart. The API can be executed remotely, so you can run the API on Server A to restart a node on any other server in the cluster.
In general, the API will fail if the restart causes a PG to go into read-only mode. If you must restart, in this case, you should use the --force flag to override.
For additional information on this API, see lbcli restart node-instance.
There are a few options here, detailed in the table below:
| Node Instance | Duroslight | GFTL | Duration | |
|---|---|---|---|---|
| Restart a node | Yes | Yes | No | ~30sec |
| Restart a node (and GFTL) | Yes | Yes | Yes | ~60sec |
| Stop a node | Yes | Yes | No | ~3sec |
| Start a node | Yes | Yes | No | ~30sec |
lbcli Examples
Status of the node: the node-manager server is up and running:
# systemctl status node-manager ● node-manager.service - Node-Manager Service Loaded: loaded (/usr/lib/systemd/system/node-manager.service; enabled; preset: disabled) Active: active (running) since Sun 2026-02-08 17:45:17 UTC; 1h 15min ago Main PID: 13876 (node-manager) Tasks: 11 (limit: 76093) Memory: 34.0M CPU: 4min 56.228sValidate that the other services are running (GFTL and Duroslight):
# ps -ef | egrep 'lbe|duros'root 14656 1 0 17:45 ? 00:00:00 /opt/backend/lbe lbe.max_num_read_errors=10 lbe.read_errors_timeframe_msec=5000 lbe.max_num_blkdev_reset_ctrl=5 lbe.blkdev_reset_ctrl_timeframe_msec=120000 lbe.develop_trim=1 lf.lf_mode=2 main.debug_level=6root 14662 14656 99 17:45 ? 07:45:47 /opt/backend/tmp_extract/staticx-ENfOGP/lbedyn lbe max_num_read_errors 10 lbe read_errors_timeframe_msec 5000 lbe max_num_blkdev_reset_ctrl 5 lbe blkdev_reset_ctrl_timeframe_msec 120000 lbe develop_trim 1 lf lf_mode 2 main debug_level 6root 15483 1 99 17:46 ? 02:33:59 /opt/duroslight/duroslight --poll-mode --lock-memory=true --reactor-backend=epoll --blocked-reactor-notify-ms=200 --blocked-reactor-reports-per-minute=300 --logger-stdout-timestamps=none --hugepages /mnt/huge --cpuset 7-8 --abort-on-seastar-bad-alloc -m2147483648 --instance-uuid c5575862-a797-54e9-ab99-10b2a869c3b7 --instance-id 0 --conf-file /etc/duroslight/conf.yaml --base_ctrl_id 3 --ip-addr 10.18.82.74 --port 4420 --replicator-port 22226 --admin-core 7 --journal-device /dev/nvme1n1 --dax-device /dev/nvme1n1 --nvram-size 1073741824 --nvram-offset 0 --subsysnqn nqn.2016-01.com.lightbitslabs:uuid:d630288d-f1c6-42ee-a531-01ef29fd8d9e --use-adq false --rx-dram false --disable-header false --lbe-usermode true --develop-trim false --encrypt-user-data trueExample of restarting a node instance (Prompt Y to verify):
[root@rack08-server71-vm04 ~]# lbcli restart node-instance --node-uuid c5575862-a797-54e9-ab99-10b2a869c3b7WARNING: Restarting node instance c5575862-a797-54e9-ab99-10b2a869c3b7This is a RISKY operation that will restart the node instance for this node.Safety checks will be performed to prevent data unavailability.Backend instance will NOT be restarted (duroslight and control plane only).Type 'yes' or 'y' to confirm, or anything else to cancel: yExample of restarting a node instance and GFTL (--consider-backend=true):
lbcli restart node-instance --node-uuid c5575862-a797-54e9-ab99-10b2a869c3b7 --consider-backend=trueWARNING: Restarting node instance c5575862-a797-54e9-ab99-10b2a869c3b7This is a RISKY operation that will restart the node instance for this node.Safety checks will be performed to prevent data unavailability.Type 'yes' or 'y' to confirm, or anything else to cancel: yExample of how to restart a node, even if it will cause a PG to become read-only:
# lbcli restart node-instance --node-uuid c5575862-a797-54e9-ab99-10b2a869c3b7 --forceWARNING: Restarting node instance c5575862-a797-54e9-ab99-10b2a869c3b7This is a RISKY operation that will restart the node instance for this node.FORCE mode enabled: volumes may temporarily become unavailable or read-only!Backend instance will NOT be restarted (duroslight and control plane only).Type 'yes' or 'y' to confirm, or anything else to cancel: yIf you try to restart or stop a node instance when another node is down (which could cause PGs to go into read-only state), the API will fail. This is a safeguard, and if it is required, you should use the --force flag:
#lbcli list nodesName UUID State NVMe endpoint Failure domains Capacity Used Local rebuild progressserver_0-0 c5..b7 Active 10.18.82.74:4420 [vm04] 14 GiB 0 Bserver_1-0 4d..da Inactive 10.18.82.5:4420 [vm05] 14 GiB 0 Bserver_2-0 9a..4f Active 10.18.82.100:4420 [vm03] 14 GiB 0 B# lbcli stop node-instance --node-uuid c5575862-a797-54e9-ab99-10b2a869c3b7WARNING: Stopping node instance c5575862-a797-54e9-ab99-10b2a869c3b7This is a RISKY operation that will stop the node instance for this node.Safety checks will be performed to prevent data unavailability.Type 'yes' or 'y' to confirm, or anything else to cancel: yStopping node instance c5575862-a797-54e9-ab99-10b2a869c3b7 failed: rpc error: code = FailedPrecondition desc = cannot stop node c5575862-a797-54e9-ab99-10b2a869c3b7: volumes are not fully protected. Use --force to override this check# lbcli restart node-instance --node-uuid c5575862-a797-54e9-ab99-10b2a869c3b7WARNING: Restarting node instance c5575862-a797-54e9-ab99-10b2a869c3b7This is a RISKY operation that will restart the node instance for this node.Safety checks will be performed to prevent data unavailability.Backend instance will NOT be restarted (duroslight and control plane only).Type 'yes' or 'y' to confirm, or anything else to cancel: yRestarting node instance c5575862-a797-54e9-ab99-10b2a869c3b7 failed: rpc error: code = FailedPrecondition desc = cannot stop node c5575862-a797-54e9-ab99-10b2a869c3b7: volumes are not fully protected. Use --force to override this check.Example of stopping a node instance (Prompt Y to verify):
# lbcli restart node-instance --node-uuid c5575862-a797-54e9-ab99-10b2a869c3b7WARNING: Restarting node instance c5575862-a797-54e9-ab99-10b2a869c3b7This is a RISKY operation that will restart the node instance for this node.Safety checks will be performed to prevent data unavailability.Backend instance will NOT be restarted (duroslight and control plane only).Type 'yes' or 'y' to confirm, or anything else to cancel: ySuccessfully stopped node instance c5575862-a797-54e9-ab99-10b2a869c3b7Example of starting a node instance (Prompt Y to verify):
# lbcli start node-instance --node-uuid c5575862-a797-54e9-ab99-10b2a869c3b7WARNING: Starting node instance c5575862-a797-54e9-ab99-10b2a869c3b7This operation will start the node instance for this node.Type 'yes' or 'y' to confirm, or anything else to cancel: ySuccessfully started node instance c5575862-a797-54e9-ab99-10b2a869c3b7© 2026 Lightbits Labs™