NVMe Discover Command

The NVMe-over-Fabrics specification defines the concept of a discovery controller that an NVMe host can query on a fabric network to discover NVMe subsystems. These subsystems are contained in NVMe targets that can be connected from the network. Achieving a persistent discovery connection is possible with Kernel Supporting AEN (kernel >= 5.5 on the client). The nvme discover command requires several parameters for the NVMe/TCP storage server to be discovered and connected to.

Note that the base NVMe specification defines the NQN (NVMe Qualified Name) format, which an NVMe endpoint (device, subsystem, etc.) must follow to guarantee a unique name under the NVMe standard. In particular, the host NQN uniquely identifies the NVMe host, and can be used by the discovery controller to control what NVMe target resources are allocated to the NVMe host for a connection.

Get the hostnqn:

cat /etc/nvme/hostnqn

In case there is no hostnqn, you can generate one by using the following command:

nvme gen-hostnqn > /etc/nvme/hostnqn

The simplified syntax of this command is:

nvme discover -t tcp -a <ip> -s 8009 -q <hostnqn> -p

These sets of parameters are given as an example. See the NVMe documentation for a complete list and usage of the nvme discover command. We have selected these parameters as an example of the minimum required to connect to and use an NVMe/TCP volume.

The hostnqn parameter is configured in a txt file that sits in /etc/nvme/hostnqn. When this is modified and the nvme discover command is invoked, a systemd unit will get called to run connect-all.

After the application client connects to the Lightbits storage cluster, the volumes that this application client can access appear as local block devices. The following example shows how to use the Linux nvme command to list all block devices after the connect command finishes.

Sample Command

$ nvme list

Sample Output

Node SN Model Namespace Usage Format FW Rev

​

---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- --------

​

/dev/nvme7n1 b23f8854fe984771 LightBox 6 4.29 GB / 4.29 GB 4 KiB + 0 B 1.0

​

/dev/nvme7n2 b23f8854fe984771 LightBox 7 4.29 GB / 4.29 GB 4 KiB + 0 B 1.0

​

/dev/nvme7n3 b23f8854fe984771 LightBox 8 4.29 GB / 4.29 GB 4 KiB + 0 B 1.0

​

/dev/nvme7n4 b23f8854fe984771 LightBox 9 4.29 GB / 4.29 GB 4 KiB + 0 B 1.0

​

/dev/nvme7n5 b23f8854fe984771 LightBox 10 10.74 GB / 10.74 GB 4 KiB + 0 B 1.0

In this example, the Lightbits storage server serves four volumes via NVMe/TCP. These volumes are mapped to the /dev Linux file system path. The NVMe/TCP block device names are listed in the Node column.