Node Instance Restart

Prior to release v3.18.1, to restart a Duroslight/GFTL/node instance, you were required to restart the node manager service. This operation takes time, and in the case of a dual-node server, if for example there is a problem with one Duroslight, you still need to restart the node manager service, which will impact both nodes.

The new API documented below allows you to restart only a single node-instance control plane (running inside the NM service) and only its own Duroslight service and its gftl-instance, without impacting the GFTL service (i.e., without impacting the other node’s GFTL instance). You can also perform the restart only for the node’s control plane and Duroslight instances, without its GFTL instance. If you are not restarting the GFTL instance, the restart time is shorter, and this reduces the rebuild time after the restart. The API can be executed remotely, so you can run the API on Server A to restart a node on any other server in the cluster.

In general, the API will fail if the restart causes a PG to go into read-only mode. If you must restart, in this case, you should use the --force flag to override.

Note

For additional information on this API, see lbcli restart node-instance.

There are a few options here, detailed in the table below:

	Node Instance	Duroslight	GFTL	Duration
Restart a node	Yes	Yes	No	~30sec
Restart a node (and GFTL)	Yes	Yes	Yes	~60sec
Stop a node	Yes	Yes	No	~3sec
Start a node	Yes	Yes	No	~30sec

lbcli Examples

Status of the node: the node-manager server is up and running:

# systemctl status node-manager

● node-manager.service - Node-Manager Service

Loaded: loaded (/usr/lib/systemd/system/node-manager.service; enabled; preset: disabled)

Active: active (running) since Sun 2026-02-08 17:45:17 UTC; 1h 15min ago

Main PID: 13876 (node-manager)

Tasks: 11 (limit: 76093)

Memory: 34.0M

CPU: 4min 56.228s

Validate that the other services are running (GFTL and Duroslight):

# ps -ef | egrep 'lbe|duros'

root 14656 1 0 17:45 ? 00:00:00 /opt/backend/lbe lbe.max_num_read_errors=10 lbe.read_errors_timeframe_msec=5000 lbe.max_num_blkdev_reset_ctrl=5 lbe.blkdev_reset_ctrl_timeframe_msec=120000 lbe.develop_trim=1 lf.lf_mode=2 main.debug_level=6

root 14662 14656 99 17:45 ? 07:45:47 /opt/backend/tmp_extract/staticx-ENfOGP/lbedyn lbe max_num_read_errors 10 lbe read_errors_timeframe_msec 5000 lbe max_num_blkdev_reset_ctrl 5 lbe blkdev_reset_ctrl_timeframe_msec 120000 lbe develop_trim 1 lf lf_mode 2 main debug_level 6

root 15483 1 99 17:46 ? 02:33:59 /opt/duroslight/duroslight --poll-mode --lock-memory=true --reactor-backend=epoll --blocked-reactor-notify-ms=200 --blocked-reactor-reports-per-minute=300 --logger-stdout-timestamps=none --hugepages /mnt/huge --cpuset 7-8 --abort-on-seastar-bad-alloc -m2147483648 --instance-uuid c5575862-a797-54e9-ab99-10b2a869c3b7 --instance-id 0 --conf-file /etc/duroslight/conf.yaml --base_ctrl_id 3 --ip-addr 10.18.82.74 --port 4420 --replicator-port 22226 --admin-core 7 --journal-device /dev/nvme1n1 --dax-device /dev/nvme1n1 --nvram-size 1073741824 --nvram-offset 0 --subsysnqn nqn.2016-01.com.lightbitslabs:uuid:d630288d-f1c6-42ee-a531-01ef29fd8d9e --use-adq false --rx-dram false --disable-header false --lbe-usermode true --develop-trim false --encrypt-user-data true

Example of restarting a node instance (Prompt Y to verify):

[root@rack08-server71-vm04 ~]# lbcli restart node-instance --node-uuid c5575862-a797-54e9-ab99-10b2a869c3b7

WARNING: Restarting node instance c5575862-a797-54e9-ab99-10b2a869c3b7

This is a RISKY operation that will restart the node instance for this node.

Safety checks will be performed to prevent data unavailability.

Backend instance will NOT be restarted (duroslight and control plane only).

Type 'yes' or 'y' to confirm, or anything else to cancel: y

Example of restarting a node instance and GFTL (--consider-backend=true):

lbcli restart node-instance --node-uuid c5575862-a797-54e9-ab99-10b2a869c3b7 --consider-backend=true

WARNING: Restarting node instance c5575862-a797-54e9-ab99-10b2a869c3b7

This is a RISKY operation that will restart the node instance for this node.

Safety checks will be performed to prevent data unavailability.

Type 'yes' or 'y' to confirm, or anything else to cancel: y

Example of how to restart a node, even if it will cause a PG to become read-only:

# lbcli restart node-instance --node-uuid c5575862-a797-54e9-ab99-10b2a869c3b7 --force

WARNING: Restarting node instance c5575862-a797-54e9-ab99-10b2a869c3b7

This is a RISKY operation that will restart the node instance for this node.

FORCE mode enabled: volumes may temporarily become unavailable or read-only!

Backend instance will NOT be restarted (duroslight and control plane only).

Type 'yes' or 'y' to confirm, or anything else to cancel: y

If you try to restart or stop a node instance when another node is down (which could cause PGs to go into read-only state), the API will fail. This is a safeguard, and if it is required, you should use the --force flag:

#lbcli list nodes

Name UUID State NVMe endpoint Failure domains Capacity Used Local rebuild progress

server_0-0 c5..b7 Active 10.18.82.74:4420 [vm04] 14 GiB 0 B

server_1-0 4d..da Inactive 10.18.82.5:4420 [vm05] 14 GiB 0 B

server_2-0 9a..4f Active 10.18.82.100:4420 [vm03] 14 GiB 0 B

# lbcli stop node-instance --node-uuid c5575862-a797-54e9-ab99-10b2a869c3b7

WARNING: Stopping node instance c5575862-a797-54e9-ab99-10b2a869c3b7

This is a RISKY operation that will stop the node instance for this node.

Safety checks will be performed to prevent data unavailability.

Type 'yes' or 'y' to confirm, or anything else to cancel: y

Stopping node instance c5575862-a797-54e9-ab99-10b2a869c3b7 failed: rpc error: code = FailedPrecondition desc = cannot stop node c5575862-a797-54e9-ab99-10b2a869c3b7: volumes are not fully protected. Use --force to override this check

Example of stopping a node instance (Prompt Y to verify):

# lbcli restart node-instance --node-uuid c5575862-a797-54e9-ab99-10b2a869c3b7

WARNING: Restarting node instance c5575862-a797-54e9-ab99-10b2a869c3b7

This is a RISKY operation that will restart the node instance for this node.

Safety checks will be performed to prevent data unavailability.

Backend instance will NOT be restarted (duroslight and control plane only).

Type 'yes' or 'y' to confirm, or anything else to cancel: y

Successfully stopped node instance c5575862-a797-54e9-ab99-10b2a869c3b7

Example of starting a node instance (Prompt Y to verify):

# lbcli start node-instance --node-uuid c5575862-a797-54e9-ab99-10b2a869c3b7

WARNING: Starting node instance c5575862-a797-54e9-ab99-10b2a869c3b7

This operation will start the node instance for this node.

Type 'yes' or 'y' to confirm, or anything else to cancel: y

Successfully started node instance c5575862-a797-54e9-ab99-10b2a869c3b7

Last updated on

Was this page helpful?