Volume Management

Understanding a Volume Location in the Lightbits Cluster

The basic work with a block device is to create and manage volumes. This section explains the volumes in a Lightbits cluster - including how to identify the placement of the volumes and how a node failure impacts the volume protection state.

You can use the lbcli List Volumes command to see which node in the cluster holds the data of a specific volume. Note that volume placement may change during the life cycle of a volume due to dynamic rebalancing.

Sample Command

Bash
Copy

Sample Output

YAML
Copy

You can also run the nvme list-subsys command on the application server, on a specific block-device that corresponds to a volume on the cluster.

Sample Command

Bash
Copy

Sample Output

Bash
Copy

This command’s output includes the cluster’s primary node (optimized) IP address for the volume, the secondary node (inaccessible) IP address, and the other nodes that do not hold data for that volume (block device nvme0n1).

Volume Placement

With Volume Placement, you can specify which failure domain to place a volume on upon its creation. The idea behind volume placement is to statically define, separate, and manage how volumes and data can tolerate a failure and provide availability - through failure of a server, rack, row, power grid, etc. By default, Lightbits will define server level failure domains.

Note that this feature is associated with the create volume command.

Current feature limitations of Volume Placement include the following:

  • Available only for single replica volumes.
  • Only failure domains labels are currently matched (e.g., fd:server00).
  • Up to 25 affinities can be specified.
  • Value (failure domain name) is limited to up to 100 characters.
  • Volume Placement cannot be specified for a clone (a clone is always placed on the same nodes as the parent volume/snapshot).
  • Dynamic Rebalancing must be disabled.
  • The entire Lightbits cluster must be upgraded to at least release version 2.3.8.

In Lightbits 2.3.8 and above, a new flag is available - 'placement-affinity' - which can be used as follows:

lbcli create volume

Bash
Copy

In the example above, you can ask the system to place the volume as follows:

  1. On a node that includes Server0 in its failure domains. Note that the server name is used as a default failure domain configuration in node-manager.yaml. You will be responsible for determining if you want anything else other than the default yaml that Lightbits provides.
  2. On a node that includes rack1 in its failure domains.
  3. On a node that includes rack0 in its failure domains.

If the requirement cannot be satisfied because Lightbits did not find such a node, Lightbits will fail the request.

If Lightbits cannot find active nodes with failure domains that match the volume placement request, create volume will fail and will not place the volume on other nodes.

Node Failure and Volume Protection State

The Lightbits cluster software continuously monitors the cluster nodes' health and connectivity and responds to changes in the nodes’ status. In AWS, the Auto Healing feature will take preventive measures in case of AWS notification of an instance outage, and remedial measures on abrupt failure. See the Auto Maintenance Overview section for additional information.

If a node fails, volumes that have data stored on that node can be affected. For a volume with a replication factor of 3, a single node failure may cause the volume protection state to become Degraded. If another node fails, the volume’s state may become ReadOnly.

Although it is not recommended in AWS, RF2 volumes will become ReadOnly with a single node failure, and RF1 volumes will become Inaccessible if the node the data is on fails.

In case all nodes that hold a volume’s replica fail, the volume becomes Inaccessible.

You can view the volume’s protection state by issuing the lbcli List Volumes command.

Sample Command

Bash
Copy

Sample Output

Bash
Copy

As you can see in the output, vol2 and vol3 are not in a FullyProtected volume protection state.

Now, you can use the lbcli List Nodes command to identify which node has failed (the Cluster Admin command). In this command’s output you will see one of the following node states:

Sample Command

Bash
Copy

Sample Output

Bash
Copy
Type to search, ESC to discard
Type to search, ESC to discard
Type to search, ESC to discard
Next to read:
Connecting Application Servers to Lightbits

© 2025 Lightbits Labs™

On This Page
Volume ManagementUnderstanding a Volume Location in the Lightbits ClusterVolume PlacementNode Failure and Volume Protection State