Loading the NVMe/TCP Host Software and Enabling Multipath
For a client to connect to an NVMe over TCP volume on a Lightbits cluster, the nvme-tcp modules must be loaded. To fully utilize the multipathing features of Lightbits, multipath must then be enabled on the nvme-core module.
Loading the NVME TCP Module
To use NVMe/TCP, you must load the NVMe host modules by entering the following command in the system’s command shell.
modprobe nvme-tcplsmod | grep nvmeThe output is similar to the following example:
nvme_tcp 24576 0nvme_fabrics 20480 1 nvme_tcpnvme_core 49152 4 nvme_fabrics,nvme_tcpMake the setting boot persistent by loading the module on boot with this setting:
echo nvme_tcp > /etc/modules-load.d/nvme_tcp.confMultipath
By default, multipath should be enabled with the nvme_core module.
However, you can run the following command to check:
grep -r "" /sys/module/nvme_core/parametersIf you see /sys/modules/nvme_core/parameters/multipath:N, then multipath is not enabled.
- There is device mapping multipath and NVMe native multipathing. Lightbits requires NVMe native multipathing.
- NVMe native multipathing is enabled by default in RHEL 9.x and is the recommended option by Red Hat. For more on how to enable this, see the relevant Red Hat documentation.
Enable multipath using one of the following methods.
Enabling Multipath Using Grubby
Figure out your current loaded kernel with grubby:
grubby --default-kernelThe output should show the full path of the kernel in the format of /boot/vmlinuz-...
Now configure the kernel boot arguments to load enable multipath. Make sure to put the full path of the default kernel into the command below:
grubby --args=nvme_core.multipath=Y --update-kernel /boot/vmlinuz-...grubby is available out of the box on AlmaLinux and Red Hat based flavours. For distributions that do not have grubby, use the next method.
Enabling Multipath Using a Configuration File
First, create a configuration file that enables multipath:
echo "options nvme_core multipath=Y" > /etc/modprobe.d/50-nvme_core.confThen, update the initramfs, which OSes use to load and configure modules on boot. Use the appropriate tool for the OS:
- On Red Hat (including AlmaLinux), run
dracut -f. - On Debian/Ubuntu systems, run
update-initramfs -u.
Rebooting the Client
It is recommended to reboot the client to make sure that all of the settings are loaded properly. Make sure that the nvme_tcp modules are loaded on boot and that multipath is enabled.
lsmod | grep nvme; grep -r "" /sys/module/nvme_core/parameters;nvme_tcp 24576 0nvme_fabrics 20480 1 nvme_tcpnvme_core 49152 4 nvme_fabrics,nvme_tcp.../sys/modules/nvme_core/parameters/multipath:Y...