Connecting the Cluster Client to Lightbits

AI Tools

After creating a volume on the Lightbits storage server, log in to one or more of your application clients and use the Lightbits NVMe CLI utility to make a connection to the Lightbits cluster.

Before you begin, enter a Linux ping command to check the TCP/IP connectivity between your application client and the Lightbits storage servers. In the example below, the client has a data NIC connected to reach the 10.10.10.x network. The IP 10.10.10.100 is data NIC of one of the Lightbits storage servers.

Sample Command

$ ping -c 1 10.10.10.100

Sample Ouptut

PING 10.10.10.100 (10.10.10.100) 56(84) bytes of data. 64 bytes from 10.10.10.100: icmp_seq=1 ttl=255 time=0.032 ms --- 10.20.20.10 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 0.032/0.032/0.032/0.000 ms

This output indicates that this application client has a connection to the data NIC IP address on the Lightbits storage server where volumes were created.

Repeat this ping check for the other Lightbits cluster servers: 10.10.10.101 and 10.10.10.102.

After you have checked the TCP/IP connectivity between your application client and the Lightbits storage servers, use the nvme CLI utility to connect the application client via NVMe/TCP to the Lightbits storage server.

To use the nvme CLI utility on your application client, you will need the following details.

Required Lightbits Storage Cluster Connection Details

Required Data	Description	Connect Command Parameter
Lightbits Data NIC IP address	The data NIC IP address of each Lightbits cluster node. These values can be retrieved from the Lightbits management server using the `lbcli list nodes` command.	-a
ACL string	The ACL string you used when you created the volume on the Lightbits storage server node.	-q
Subsystem NQN	The Lightbits cluster subsystem NQN value can be retrieved from the Lightbits management server using the `lbcli get cluster` command.	-n
TCP port	The data TCP port for each of the Lightbits cluster nodes can be retrieved from the Lightbits management server using the lbcli `list nodes` command.	-s

Enter the lbcli get cluster command on any Lightbits storage server to identify the subsytem NQN.

Sample Command

$ lbcli -J $LIGHTOS_JWT get cluster -o yaml

Note

By default, the LIGHTOS_JWT is generated during the Lightbits installation on the Ansible installation host and saved to ~/lightos-system-jwt. See Post-Installation Steps for one way to get LIGHTOS_JWT.

Sample Output

UUID: 95a251b6-0885-4f5b-a0eb-90e90a2009a3 currentMaxReplicas: 3 ... subsystemNQN: nqn.2014-08.org.nvmexpress:NVMf:uuid:b5fe744a-b919-465a-953a-a8a0df7b9d31 <--- subsystem NQN supportedMaxReplicas: 3

Enter the lbcli list nodes command to identify the NIC IP address and TCP port.

Sample Command

$ lbcli -J $LIGHTOS_JWT list nodes

Sample Output

Name UUID State NVMe endpoint Failure domains Local rebuild progress server00-0 08fdb3bd-925a-5e73-adde-8daf881969d3 Active 10.10.10.100:4420 [server00] None server01-0 112a555f-8168-5f07-a4e0-bf8f5b59c740 Active 10.10.10.101:4420 [server01] None server02-0 bc759c13-856d-5521-9ba2-752259abf8f0 Active 10.10.10.102:4420 [server02] None

With the IP, port, subsystem NQN and ACL values for the volume, you can execute the nvme connect command to connect to all of the nodes in the cluster.

Sample NVMe Connect Commands

$ nvme connect -t tcp -a 10.10.10.100 --ctrl-loss-tmo -1 -n \ nqn.2014-08.org.nvmexpress:NVMf:uuid:b5fe744a-b919-465a-953a-a8a0df7b9d31 -s 4420 -q acl3 $ nvme connect -t tcp -a 10.10.10.101 --ctrl-loss-tmo -1 -n \ nqn.2014-08.org.nvmexpress:NVMf:uuid:b5fe744a-b919-465a-953a-a8a0df7b9d31 -s 4420 -q acl3 $ nvme connect -t tcp -a 10.10.10.102 --ctrl-loss-tmo -1 -n \ nqn.2014-08.org.nvmexpress:NVMf:uuid:b5fe744a-b919-465a-953a-a8a0df7b9d31 -s 4420 -q acl3

Notes

We are using an ACL value/hostnqn of "acl3", so that we can connect to the volume created, as detailed in Creating a Volume on the Lightbits Storage Server.
Use the client procedure for each node in the cluster. Remember to use the correct NVME-Endpoint for each node.
Using the --ctrl-loss-tmo -1 flag allows for infinite attempts to reconnect nodes, and prevents a timeout from occurring when attempting to connect with a node in a failure state.
Starting from Version 3.1.1, data IP can be IPv6.
See the discovery-client documentation in the Lightbits Administration Guide. Like nvme connect, this can connect to NVMe over TCP volumes. However, it can also monitor the nodes and if new nodes/paths are created or removed, it will properly maintain those.

Last updated on

Was this page helpful?