Discovery-client Deployment and Usage

Overview

The discovery-client is an open-source service managed by systemd. Discovery-client plays a role in the orchestration of your Lightbits storage environment. Specifically, it serves as the mechanism for discovering nodes and volumes within Lightbits clusters.

Once discovered, it establishes and manages connections to these volumes, ensuring seamless data transfers via NVMe-over-Fabrics. The discovery-client is also designed to adapt to dynamic environments, automatically updating connections whenever nodes are added or removed from the cluster. This adaptability ensures that your host maintains effective communication with Lightbits storage clusters and its volumes.

The discovery-client performs several tasks:

Maintains an updated list of NVMe-over-Fabrics discovery controllers. Any changes in the Lightbits cluster controllers will automatically update this list.
Executes nvme discover commands against these discovery controllers to discover available nvme-over-fabrics subsystems. These commands are triggered either by an Asynchronous Event Notification (AEN) received from a remote discovery controller, or by a user-specified configuration file.
Automatically connects to available NVMe-over-Fabrics subsystems by running nvme connect commands.

You can configure the discovery-client via configuration files or using command-line options. This article will cover both.

Setting up Repos for RPM and Debian

For Debian-Based Systems (Ubuntu, Debian, etc.)

Gather reference information about the client's OS version and codename:

Bash
    
 
lsb_release -a
Copy

Install the required packages:

Bash
    
​x
 
apt-get install -y debian-keyring # debian only​apt-get install -y debian-archive-keyring # debian only​apt-get install -y apt-transport-https
Copy

Set the keyring location variable, depending on the OS version:

Bash
    
 
# For Debian Stretch, Ubuntu 16.04 and laterkeyring_location=/usr/share/keyrings/lightbits-discovery-client-archive-keyring.gpg​# For Debian Jessie, Ubuntu 15.10 and earlierkeyring_location=/etc/apt/trusted.gpg.d/lightbits-discovery-client.gpg
Copy

Download the GPG key:

Bash
    
curl -1sLf 'https://dl.lightbitslabs.com/public/discovery-client/gpg.014E5C7FAFD89AEE.key' | gpg --dearmor > ${keyring_location}
Copy

Download the discovery-client repo file:

Bash
    
curl -1sLf 'https://dl.lightbitslabs.com/public/discovery-client/config.deb.txt?distro=ubuntu&codename=xenial' | sudo tee /etc/apt/sources.list.d/lightbits-discovery-client.list
Copy

Note: If required, replace the distribution and codename with your actual operating system and codename (based on the output of lsb_release -a in the first step). For example, for Ubuntu 22, replace xenial with jammy.

Update the repo:

Bash
    
 
apt-get update
Copy

For RPM-based systems (Red Hat, etc.)

Install the prerequisites:

Bash
    
 
yum install -y yum-utilsyum install -y pygpgme
Copy

Continue with the instructions if pygpgme is not found. Newer releases do not require it.

Download and import the GPG key:

Bash
    
 
rpm --import 'https://dl.lightbitslabs.com/public/discovery-client/gpg.014E5C7FAFD89AEE.key'
Copy

Download the repo file:

Bash
    
curl -1sLf 'https://dl.lightbitslabs.com/public/discovery-client/config.rpm.txt?distro=el&codename=7' | sudo tee /etc/yum.repos.d/lightbits-discovery-client.repo
Copy

Update the repo cache:

Bash
    
 
yum makecache -y --disablerepo='*' --enablerepo='lightbits-discovery-client'
Copy

Installation

For RPM-Based Systems:

Bash
    
 
yum install -y discovery-client
Copy

For Debian-Based Systems:

Bash
    
 
apt-get install -y discovery-client
Copy

Initial Setup

Check if the nvme-tcp module is loaded:

Bash
    
 
lsmod | grep nvme
Copy

If not, load it:

Bash
    
 
modprobe nvme-tcp
Copy

Use the following guide to enable the nvme-tcp module persistently through reboots: Client Module Configurations.

Check if nvme-core multipath is enabled:

Bash
    
 
cat /sys/module/nvme_core/parameters/multipath
Copy

If enabled ("Y"), continue to the next initial setup step.

If disabled ("N"), use the following guide to enable nvme-core multipath for the current session and persistently through reboots: Client Module Configurations.

Start the discovery-client service:

Bash
    
 
systemctl start discovery-client
Copy

Find or generate the host NQN:

To find it, run:

Bash
    
 
cat /etc/nvme/hostnqn
Copy

To generate one, run:

Bash
    
 
nvme gen-hostnqn > /etc/nvme/hostnqn
Copy

Volume Creation

From the Lightbits side, create a volume and set its ACL to the hostnqn. Additionally, obtain the cluster NQN, which will be required for connection methods 1 and 2 below.

Create a volume with HostNQN ACL:

Bash
    
 
lbcli create volume <parameters> --acl <hostnqn>
Copy

Find the Lightbits cluster NQN:

Bash
    
 
lbcli get cluster
Copy

Configuration For Persistence Across Reboots

You can configure the discovery-client in one of three ways.

Method 1: Using a Configuration File

This connection method will connect to a Lightbits volume based on the config file when the discovery-client service starts. Therefore if the service is enabled to start on boot, the connection will also start on boot, making it a reboot persistent connection.

Create a Configuration File

The configuration file should be placed under /etc/discovery-client/discovery.d/<name>.conf. In the file, specify the entries for the Lightbits clusters that you want to connect to. Each entry will specify the target, address, port, and other relevant details. If the directory does not exist, create it first with mkdir /etc/discovery-client/discovery.d/

Below is a sample configuration that connects to all of the current nodes of an example cluster. Technically, only one connection line is required for discovery to work. However, listing all does not have negative effects and ensures high availability and redundancy.

Bash
    
 
-t tcp -a 172.16.176.11 -s 8009 -q host1 -n nqn.2016-01.com.lightbitslabs:uuid:3715aea0-2705-4a01-9357-c6a9f8009f09-t tcp -a 172.16.175.11 -s 8009 -q host1 -n nqn.2016-01.com.lightbitslabs:uuid:3715aea0-2705-4a01-9357-c6a9f8009f09-t tcp -a 172.16.175.10 -s 8009 -q host1 -n nqn.2016-01.com.lightbitslabs:uuid:3715aea0-2705-4a01-9357-c6a9f8009f09-t tcp -a 172.16.176.12 -s 8009 -q host1 -n nqn.2016-01.com.lightbitslabs:uuid:3715aea0-2705-4a01-9357-c6a9f8009f09-t tcp -a 172.16.176.10 -s 8009 -q host1 -n nqn.2016-01.com.lightbitslabs:uuid:3715aea0-2705-4a01-9357-c6a9f8009f09-t tcp -a 172.16.175.12 -s 8009 -q host1 -n nqn.2016-01.com.lightbitslabs:uuid:3715aea0-2705-4a01-9357-c6a9f8009f09
Copy

Where:

-t: Transport type (tcp)

-a: the IP of the Lightbits data network. All of the IPs can be retrieved from the command "lbcli list nodes" output, under the NVMe endpoint column.

-s: The port used by the discovery-service (8009).

-q: The hostnqn/ACL name of the client. You can use the value stored in the file /etc/nvme/hostnqn, or specify any string. This value must match the acl attribute set in the "lbcli create volume" command.

-n The Lightbits cluster hostnqn, This parameter must match the value from the "lbcli get cluster" command output, under the Subsystem NQN column.

Restart the Discovery-Client Service

Bash
    
 
systemctl restart discovery-client
Copy

Method 2: Using the Command Line to add hostnqn

You can use the discovery-client add-hostnqn command to establish a connection that will persist reboots. This command - like Method 1 above - will create a configuration file.

Bash
    
 
discovery-client add-hostnqn -a <ip>:8009 -n <clusternqn> -q <hostnqn> --name <config-file-name>
Copy

Method 3: Using the Command Line to connect-all

You can use the discovery-client connect-all command to establish a connection. This method is not reboot persistent.

The following illustrates a sample usage:

Bash
    
 
discovery-client connect-all -t tcp -a <ip> -s 8009 -q <hostnqn> -p
Copy

Where:

-t: Transport type (tcp)

-a: the IP of the Lightbits data network. All of the IPs can be retrieved from the command "lbcli list nodes" output, under the NVMe endpoint column.

-s: The port used by the discovery-service (8009).

-p: The discovery connection will remain persistent after the initial volume connection - thus monitoring for changes. This is the default behavior for connection methods 1 and 2 above.

Usage

List the connected Lightbits volumes:

Bash
    
 
nvme list
Copy

Disconnect from a given controller:

Bash
    
 
discovery-client disconnect -d <dev>
Copy

Disconnect all controllers:

Bash
    
 
discovery-client disconnect-all
Copy

Logs and Troubleshooting

Logs are maintained in the /var/log/discovery-client directory. Tail the logs to identify issues:

Bash
    
 
tail -f /var/log/discovery-client/discovery-client.log
Copy

Last updated on

Was this page helpful?