Discovery-client Deployment and Usage

Overview

The discovery-client is an open-source service managed by systemd. Discovery-client plays a role in the orchestration of your Lightbits storage environment. Specifically, it serves as the mechanism for discovering nodes and volumes within Lightbits clusters.

Once discovered, it establishes and manages connections to these volumes, ensuring seamless data transfers via NVMe-over-Fabrics. The discovery-client is also designed to adapt to dynamic environments, automatically updating connections whenever nodes are added or removed from the cluster. This adaptability ensures that your host maintains effective communication with Lightbits storage clusters and its volumes.

The discovery-client performs several tasks:

• Maintains an updated list of NVMe-over-Fabrics discovery controllers. Any changes in the Lightbits cluster controllers will automatically update this list.

• Executes nvme discover commands against these discovery controllers to discover available nvme-over-fabrics subsystems. These commands are triggered either by an Asynchronous Event Notification (AEN) received from a remote discovery controller, or by a user-specified configuration file.

• Automatically connects to available NVMe-over-Fabrics subsystems by running nvme connect commands.

The NVMe-over-Fabrics specification defines the concept of a discovery controller that an NVMe host can query on a fabric network to discover NVMe subsystems. These subsystems are contained in NVMe targets that can be connected from the network.

The discovery-client is a deployable service running under systemd. Every five seconds (configurable), it will poll the discoveryservices in the cluster.

The discovery-client command requires several parameters that you can query for the NVMe/TCP storage server to be discovered and connected to. Note that the base NVMe specification defines the NQN (NVMe Qualified Name) format, which an NVMe endpoint (device, subsystem, etc) must follow to guarantee a unique name under the NVMe standard. In particular, the host NQN uniquely identifies the NVMe host, and can be used by the discovery controller to control what NVMe target resources are allocated to the NVMe host for a connection.

You can configure the discovery-client via configuration files or using command-line options. This article will cover both.

Setting up Repos for RPM and Debian

For Debian-Based Systems (Ubuntu, Debian, etc.)

1. Gather reference information about the client's OS version and codename:

2. Install the required packages:

3. Set the keyring location variable, depending on the OS version:

4. Download the GPG key:

5. Download the discovery-client repo file:

6. Update the repo:

For RPM-based systems (Red Hat, etc.)

1. Install the prerequisites:

2. Download and import the GPG key:

3. Download the repo file:

4. Update the repo cache:

Installation

For RPM-Based Systems:

For Debian-Based Systems:

Initial Setup

1. Check if the nvme-tcp module is loaded:

2. If not, load it:

Use the following guide to enable the nvme-tcp module persistently through reboots: Client Module Configurations.

3. Check if nvme-core multipath is enabled:

If enabled ("Y"), continue to the next initial setup step.

If disabled ("N"), use the following guide to enable nvme-core multipath for the current session and persistently through reboots: Client Module Configurations.

4. Start the discovery-client service:

5. Find or generate the host NQN:

To find it, run:

To generate one, run:

Volume Creation

From the Lightbits side, create a volume and set its ACL to the hostnqn. Additionally, obtain the cluster NQN, which will be required for connection methods 1 and 2 below.

1. Create a volume with HostNQN ACL:

2. Find the Lightbits cluster NQN:

Configuration For Persistence Across Reboots

You can configure the discovery-client in one of three ways.

Method 1: Using a Configuration File

This connection method will connect to a Lightbits volume based on the config file when the discovery-client service starts. Therefore if the service is enabled to start on boot, the connection will also start on boot, making it a reboot persistent connection.

Create a Configuration File

The configuration file should be placed under /etc/discovery-client/discovery.d/<name>.conf. In the file, specify the entries for the Lightbits clusters that you want to connect to. Each entry will specify the target, address, port, and other relevant details. If the directory does not exist, create it first with mkdir /etc/discovery-client/discovery.d/

Below is a sample configuration that connects to all of the current nodes of an example cluster. Technically, only one connection line is required for discovery to work. However, listing all does not have negative effects and ensures high availability and redundancy.

Where:

-t: Transport type (tcp)

-a: the IP of the Lightbits data network. All of the IPs can be retrieved from the command "lbcli list nodes" output, under the NVMe endpoint column.

-s: The port used by the discovery-service (8009).

-q: The hostnqn/ACL name of the client. You can use the value stored in the file /etc/nvme/hostnqn, or specify any string. This value must match the acl attribute set in the "lbcli create volume" command.

-n The Lightbits cluster hostnqn, This parameter must match the value from the "lbcli get cluster" command output, under the Subsystem NQN column.

Restart the Discovery-Client Service

Method 2: Using the Command Line to add hostnqn

You can use the discovery-client add-hostnqn command to establish a connection that will persist reboots. This command - like Method 1 above - will create a configuration file.

Method 3: Using the Command Line to connect-all

You can use the discovery-client connect-all command to establish a connection. This method is not reboot persistent.

The following illustrates a sample usage:

Where:

-t: Transport type (tcp)

-a: the IP of the Lightbits data network. All of the IPs can be retrieved from the command "lbcli list nodes" output, under the NVMe endpoint column.

-s: The port used by the discovery-service (8009).

-q: The hostnqn/ACL name of the client. You can use the value stored in the file /etc/nvme/hostnqn, or specify any string. This value must match the acl attribute set in the "lbcli create volume" command.

-p: The discovery connection will remain persistent after the initial volume connection - thus monitoring for changes. This is the default behavior for connection methods 1 and 2 above.

Usage

1. List the connected Lightbits volumes:

2. Disconnect from a given controller:

3. Disconnect all controllers:

Logs and Troubleshooting

Logs are maintained in the /var/log/discovery-client.log directory. Tail the logs to identify issues: