Discovery-client Deployment
Discovery-client is a deployable service running under systemd. The service is responsible for discovering, connecting to, and handling changes in Lighbits clusters. It also provides an ongoing nvme connect-all functionality to a remote cluster of Lightbits nvme controllers.
This means the following:
- The discovery-client maintains an updated list of NVMe-over-Fabrics discovery controllers. A change of the controllers in the Lightbits cluster will be reflected automatically in this list.
- Discovery-client discovers available nvme-over-fabrics subsystems by running nvme discover commands against these discovery controllers. Discover commands are triggered either by an AEN (Asynchronous Event Notification) received from a remote discovery controller, or by a configuration file from the user that specifies new discovery endpoint(s).
- Discovery-client automatically connects to available nvme-over-fabrics subsystems by running nvme connect commands.
You can configure the service via a configuration file.
Setting Up the discovery-client on the Client Server
- Verify that the nvme-tcp related modules are loaded.
Run the command:
lsmod | grep nvme
Expected result:
nvme_tcp 36864 0 nvme_fabrics 24576 1 nvme_tcpnvme_core 118784 2 nvme_tcp,nvme_fabricst10_pi 16384 1 nvme_core- In case the kernel module is not loaded, load it using the command:
modprobe nvme-tcp - Install the discovery client on client host:
yum install -y discovery-client - Start the discovery-client service:
systemctl start discovery-client
Connect all or set up the discovery-client configuration file (for persistent discovery):
> discovery-client connect-all -t tcp -a ofir-Netwo-KCNFHE07SDVA-5a5f14234bd98447.elb.us-east-1.amazonaws.com -q client1 -pExpected output example:
[ { "Instance": 2, "Cntlid": 257, "Device": "/dev/nvme2" }, { "Instance": 3, "Cntlid": 513, "Device": "/dev/nvme3" }, { "Instance": 4, "Cntlid": 769, "Device": "/dev/nvme4" }]List connected nvme devices:
> nvme listExpected output example:
Node SN Model Namespace Usage Format FW Rev/dev/nvme0n1 vol0edb72d077feb8d69 Amazon Elastic Block Store 1 21.47 GB / 21.47 GB 512 B + 0 B 2.0/dev/nvme1n1 AWS436D8B424F5F5F8CC Amazon EC2 NVMe Instance Storage 1 3.75 TB / 3.75 TB 512 B + 0 B 0/dev/nvme2n1 513ee00237f59136 Lightbits LightOS 1 1.10 TB / 1.10 TB 4 KiB + 0 B 2.3You can now create a filestystem on the block device and mount or use it as a direct block device. Below is an example of a FIO command to run to perform reads and writes to the block device from the client.
> fio --name=rw --ioengine=libaio --iodepth=256 --rw=randread --bs=4k --direct=1 --filename=/dev/nvme2n1 --size=10G --numjobs=8 --runtime=20 --group_reportingPersistent discovery-client Configuration
In order to have the NVME-tcp discovery be persistent, you should create and fill in the configuration in a configuration file.
The configuration file should be placed under /etc/discovery-client/discovery.d/<name>.conf
The defined file name must be a proper file name that does not start with "tmp.dc."
Because the discovery-client can work with multiple Lightbits clusters, it needs to know which discovery-service belongs to which cluster. This grouping is achieved by specifying the subsysnqn as an identifier for each entry.
The following is an example input file:
-t tcp -a 172.16.176.11 -s 8009 -q host1 -n nqn.2016-01.com.lightbitslabs:uuid:3715aea0-2705-4a01-9357-c6a9f8009f09-t tcp -a 172.16.175.11 -s 8009 -q host1 -n nqn.2016-01.com.lightbitslabs:uuid:3715aea0-2705-4a01-9357-c6a9f8009f09-t tcp -a 172.16.175.10 -s 8009 -q host1 -n nqn.2016-01.com.lightbitslabs:uuid:3715aea0-2705-4a01-9357-c6a9f8009f09-t tcp -a 172.16.176.12 -s 8009 -q host1 -n nqn.2016-01.com.lightbitslabs:uuid:3715aea0-2705-4a01-9357-c6a9f8009f09-t tcp -a 172.16.176.10 -s 8009 -q host1 -n nqn.2016-01.com.lightbitslabs:uuid:3715aea0-2705-4a01-9357-c6a9f8009f09-t tcp -a 172.16.175.12 -s 8009 -q host1 -n nqn.2016-01.com.lightbitslabs:uuid:3715aea0-2705-4a01-9357-c6a9f8009f09Where:
| Item | Description |
|---|---|
| -t | Transport type (tcp) |
| -a | The IP of the Lightbits storage instance. All of the IPs can be retrieved from the command ‘lbcli list nodes’, under the NVMe endpoint column. |
| -s | The port used by the discovery-service (8009). |
| -q | The hostnqn/ACL name of the client. You can use the value stored in the file /etc/nvme/hostnqn, or specify any string (in the example the client is host1). This value must match the ACL attribute set in the ‘lbcli create volume’ command. |
| -n | The Lightbits cluster hostnqn (can be retrieved from the get cluster API under the Subsystem NQN column). |
- Restart the discovery-client service:
systemctl start discovery-client - Once the service is up, you will be able to see the Lightbits nodes connected by running the command:
discovery-client list ctrl. - In order to see the volumes attached to the client, you should run the command:
nvme list.
Updating the discovery-client Package
To update the discovery-client package on the clients, perform the following steps.
Using yum (RedHat):
- Update the lightbits.repo file under
/etc/yum.repos.d/to the updated repository, to point to the right version. - Execute the Linux command:
yum update -y discovery-client.x86_64.
Using RPM:
To set up the client repository, refer to the Client Software Installation section of the Lightbits Installation Guide.