Connecting the Cluster Client to Lightbits
After creating a volume on the Lightbits storage server, log in to one or more of your application clients and use the Lightbits NVMe CLI utility to make a connection to the Lightbits cluster.
Before you begin, enter a Linux ping command to check the TCP/IP connectivity between your application client and the Lightbits storage servers. In the example below, the client has a data NIC connected to reach the 10.10.10.x network. The IP 10.10.10.100 is data NIC of one of the Lightbits storage servers.
Sample Command
$ ping -c 1 10.10.10.100Sample Ouptut
PING 10.10.10.100 (10.10.10.100) 56(84) bytes of data.64 bytes from 10.10.10.100: icmp_seq=1 ttl=255 time=0.032 ms--- 10.20.20.10 ping statistics ---1 packets transmitted, 1 received, 0% packet loss, time 0msrtt min/avg/max/mdev = 0.032/0.032/0.032/0.000 msThis output indicates that this application client has a connection to the data NIC IP address on the Lightbits storage server where volumes were created.
Repeat this ping check for the other Lightbits cluster servers: 10.10.10.101 and 10.10.10.102.
After you have checked the TCP/IP connectivity between your application client and the Lightbits storage servers, use the nvme CLI utility to connect the application client via NVMe/TCP to the Lightbits storage server.
To use the nvme CLI utility on your application client, you will need the following details.
Required Lightbits Storage Cluster Connection Details
| Required Data | Description | Connect Command Parameter |
|---|---|---|
| Lightbits Data NIC IP address | The data NIC IP address of each Lightbits cluster node. These values can be retrieved from the Lightbits management server using the lbcli list nodes command. | -a |
| ACL string | The ACL string you used when you created the volume on the Lightbits storage server node. | -q |
| Subsystem NQN | The Lightbits cluster subsystem NQN value can be retrieved from the Lightbits management server using the lbcli get cluster command. | -n |
| TCP port | The data TCP port for each of the Lightbits cluster nodes can be retrieved from the Lightbits management server using the lbcli list nodes command. | -s |
Enter the lbcli get cluster command on any Lightbits storage server to identify the subsytem NQN.
Sample Command
$ lbcli -J $LIGHTOS_JWT get cluster -o yamlBy default the LIGHTOS_JWT is generated during the Lightbits installation on the ansible installation host and is saved to ~/lightos-system-jwt. See Post-Installation Steps for one way to get LIGHTOS_JWT.
Sample Output
UUID: 95a251b6-0885-4f5b-a0eb-90e90a2009a3currentMaxReplicas: 3...subsystemNQN: nqn.2014-08.org.nvmexpress:NVMf:uuid:b5fe744a-b919-465a-953a-a8a0df7b9d31 <--- subsystem NQNsupportedMaxReplicas: 3Enter the lbcli list nodes command to identify the NIC IP address and TCP port.
Sample Command
$ lbcli -J $LIGHTOS_JWT list nodesSample Output
Name UUID State NVMe endpoint Failure domains Local rebuild progressserver00-0 08fdb3bd-925a-5e73-adde-8daf881969d3 Active 10.10.10.100:4420 [server00] Noneserver01-0 112a555f-8168-5f07-a4e0-bf8f5b59c740 Active 10.10.10.101:4420 [server01] Noneserver02-0 bc759c13-856d-5521-9ba2-752259abf8f0 Active 10.10.10.102:4420 [server02] NoneWith the IP, port, subsystem NQN and ACL values for the volume, you can execute the nvme connect command to connect to all of the nodes in the cluster.
Sample NVMe Connect Commands
$ nvme connect -t tcp -a 10.10.10.100 --ctrl-loss-tmo -1 -n \nqn.2014-08.org.nvmexpress:NVMf:uuid:b5fe744a-b919-465a-953a-a8a0df7b9d31 -s 4420 -q acl3$ nvme connect -t tcp -a 10.10.10.101 --ctrl-loss-tmo -1 -n \nqn.2014-08.org.nvmexpress:NVMf:uuid:b5fe744a-b919-465a-953a-a8a0df7b9d31 -s 4420 -q acl3$ nvme connect -t tcp -a 10.10.10.102 --ctrl-loss-tmo -1 -n \nqn.2014-08.org.nvmexpress:NVMf:uuid:b5fe744a-b919-465a-953a-a8a0df7b9d31 -s 4420 -q acl3- We are using an ACL value/hostnqn of "acl3", so that we can connect to the volume created, as detailed in Creating a Volume on the Lightbits Storage Server.
- Use the client procedure for each node in the cluster. Remember to use the correct NVME-Endpoint for each node.
- Using the
--ctrl-loss-tmo -1flag allows for infinite attempts to reconnect nodes, and prevents a timeout from occurring when attempting to connect with a node in a failure state. - Starting from Version 3.1.1, data IP can be IPv6.
- See the discovery-client documentation in the Lightbits Administration Guide. Like
nvme connect, this can connect to NVMe over TCP volumes. However, it can also monitor the nodes and if new nodes/paths are created or removed, it will properly maintain those.