This section covers various configuration examples and how they affect the host configuration files. The first host configuration file will be shown server00.yml, and the host configuration files for the remaining servers will follow the same pattern. For more complex examples, two host configuration files will be shown to help better understand the configuration.
Host configuration with no data interfaces (data_ifaces) provided. The playbook does not need to configure the IP on the data interface. The user configured the interfaces prior to running the playbook.
---
nodes:
- instanceID: 0
data_ip: 10.10.10.100
failure_domains:
- server00
ec_enabled: true
lightfieldMode: SW_LF
storageDeviceLayout:
initialDeviceCount: 6
maxDeviceCount: 12
allowCrossNumaDevices: false
deviceMatchers:
# - model =~ ".*"
- partition == false
- size >= gib(300)
Example 2: Data Network Interface Automatically Configured
Host configuration with a single data interface. The playbook configured the interface. The user did not configure interface with the data IP.
---
data_ifaces:
- bootproto: static
conn_name: ens1
ifname: ens1
ip4: 10.10.10.100/24
nodes:
- instanceID: 0
data_ip: 10.10.10.100
failure_domains:
- server00
ec_enabled: true
lightfieldMode: SW_LF
storageDeviceLayout:
initialDeviceCount: 6
maxDeviceCount: 12
allowCrossNumaDevices: false
deviceMatchers:
# - model =~ ".*"
- partition == false
- size >= gib(300)
Note
The example above shows the format for setting IPv4 addresses using ip4: ip/subnet. IPv6 addresses can be set using ip6: ip/prefix.
Example 3: Override the Lightbits Configurations
Host configuration with Lightbits override. The provided value overrides the key listen-client-urls.
---
nodes:
- instanceID: 0
data_ip: 10.10.10.100
failure_domains:
- server00
ec_enabled: true
lightfieldMode: SW_LF
storageDeviceLayout:
initialDeviceCount: 6
maxDeviceCount: 12
allowCrossNumaDevices: false
deviceMatchers:
# - model =~ ".*"
- partition == false
- size >= gib(300)
etcd_settings_user:
listen-client-urls: http://127.0.0.1:2379,http://10.16.173.14:2379,http://192.168.16.219:2379
Example 4: Provide Custom Datapath Configuration
Host configuration with custom datapath configuration provided.
By default, the playbook inspects the remote machine and determines the directory containing the specific configuration for Duroslight and backend services (datapath configuration). The excluding node-manager configuration uses the following logic:
<system_vendor>-<processor_count>-processor-<processor_cores>-cores
---
nodes:
- instanceID: 0
data_ip: 10.10.10.100
failure_domains:
- server00
ec_enabled: true
lightfieldMode: SW_LF
storageDeviceLayout:
initialDeviceCount: 6
maxDeviceCount: 12
allowCrossNumaDevices: false
deviceMatchers:
# - model =~ ".*"
- partition == false
- size >= gib(300)
datapath_config: custom-datapath-config
Example 5: Use the Linux Volume Manager (LVM) Partition for etcd Data
Host configuration with custom lvm partition for etcd data.
---
nodes:
- instanceID: 0
data_ip: 10.10.10.100
failure_domains:
- server00
ec_enabled: true
lightfieldMode: SW_LF
storageDeviceLayout:
initialDeviceCount: 6
maxDeviceCount: 12
allowCrossNumaDevices: false
deviceMatchers:
# - model =~ ".*"
- partition == false
- size >= gib(300)
use_lvm_for_etcd: true
etcd_lv_name: etcd
#etcd_settings_user:
etcd_lv_size: 15GiB
etcd_vg_name: Alma
Example 6: Profile-Generator Overrides
Enable humans to override profile-generator output and provide for each server a custom file that will be taken by profile-generator as the system-profile.
Each host can be different so each host can specify its own override file.
nodes:
- instanceID: 0
data_ip: 10.10.10.100
failure_domains:
- server00
ec_enabled: true
storageDeviceLayout:
initialDeviceCount: 6
maxDeviceCount: 12
allowCrossNumaDevices: false
deviceMatchers:
- partition == false
- size >= gib(300)
profile_generator_overrides_dir: /tmp/overrides.d/server00
In case the cluster is homogeneous and we want to apply the same override to all nodes, we can provide a single setting in the groups/all.yml file or via the cmd with:
ansible-playbook -i ansible/inventories/cluster_example/hosts playbooks/deploy-lightos.yml -e profile_generator_overrides_dir=/tmp/overrides.d
In the above example, we specify profile_generator_overrides_dir, which is a directory on the Ansible Controller that will be copied to the target machine.
Example 7: Dual Instance Configuration
In a dual instance configuration. Lightbits is run from both NUMAs of the CPU.
Below is an example configuration for server00.yml and server01.yml.
In this configuration, both servers NUMA0 and 1 have 12 NVME drives, for a total of 24 SSD drives in the server.
Each server has a single management IP configured (not shown below) and two data IPs (one for each NUMA). Each instance has a unique data IP set for it.
server00.yml
---
name: server00
nodes:
- instanceID: 0
data_ip: 172.16.10.10
failure_domains:
- server00
ec_enabled: true
lightfieldMode: SW_LF
storageDeviceLayout:
initialDeviceCount: 12
maxDeviceCount: 12
allowCrossNumaDevices: false
deviceMatchers:
# - model =~ ".*"
- partition == false
- size >= gib(300)
# - name =~ "nvme0n1"
- instanceID: 1
data_ip: 172.16.20.10
failure_domains:
- server00
ec_enabled: true
lightfieldMode: SW_LF
storageDeviceLayout:
initialDeviceCount: 12
maxDeviceCount: 12
allowCrossNumaDevices: false
deviceMatchers:
# - model =~ ".*"
- partition == false
- size >= gib(300)
server01.yml
---
name: server01
nodes:
- instanceID: 0
data_ip: 172.16.10.11
failure_domains:
- server01
ec_enabled: true
lightfieldMode: SW_LF
storageDeviceLayout:
initialDeviceCount: 12
maxDeviceCount: 12
allowCrossNumaDevices: false
deviceMatchers:
# - model =~ ".*"
- partition == false
- size >= gib(300)
# - name =~ "nvme0n1"
- instanceID: 1
data_ip: 172.16.20.11
failure_domains:
- server01
ec_enabled: true
lightfieldMode: SW_LF
storageDeviceLayout:
initialDeviceCount: 12
maxDeviceCount: 12
allowCrossNumaDevices: false
deviceMatchers:
# - model =~ ".*"
- partition == false
- size >= gib(300)
Example 8: Single IP Dual NUMA Configuration
This example is for Single IP Dual Instance configuration. For more information, see Single-IP-Dual-NUMA Configuration.
The below is an example of a full server00.yml, with a similar physical configuration as the above example. However, instead of using two data IPs, it only uses one data IP for both of its instances.
Note
Unlike dual instance configuration from the example above, which had a unique data IP per instance, each instance on a server has the same IP.
---
name: server00
nodes:
- instanceID: 0
data_ip: 10.10.10.100
failure_domains:
- server00
ec_enabled: true
lightfieldMode: SW_LF
storageDeviceLayout:
initialDeviceCount: 12
maxDeviceCount: 12
allowCrossNumaDevices: false
deviceMatchers:
# - model =~ ".*"
- partition == false
- size >= gib(300)
# - name =~ "nvme0n1"
- instanceID: 1
data_ip: 10.10.10.100
failure_domains:
- server00
ec_enabled: true
lightfieldMode: SW_LF
storageDeviceLayout:
initialDeviceCount: 12
maxDeviceCount: 12
allowCrossNumaDevices: false
deviceMatchers:
# - model =~ ".*"
- partition == false
- size >= gib(300)
The following are examples for setting Journal SSD configurations, as detailed in the Setting the Journal SSD Configuration __article.
Example 1
In this example, the data SSD devices are Micron XTR and the Journal is Kioxia FL6. The model number should be in the matcher so the cluster will know which devices are for data and which are for journaling:
name: server00
nodes:
- instanceID: 0
data_ip: 10.20.155.2
ec_enabled: false
failure_domains:
- server00
storageDeviceLayout:
allowCrossNumaDevices: true
deviceMatchers:
- model =~ "Micron XTR"
- partition == false
initialDeviceCount: 6
maxDeviceCount: 11
journalDeviceLayout:
deviceMatchers:
- model =~ "Kioxia FL6"
- partition == false
numDevices: 1
Example 2
The following is an example of a dual-instance node, using the size matcher so the larger devices are used for data and the smaller ones are used for journaling:
name: server00
nodes:
- instanceID: 0
data_ip: 10.20.155.2
ec_enabled: false
failure_domains:
- server00
storageDeviceLayout:
allowCrossNumaDevices: false
deviceMatchers:
- partition == false
- size > tib(1)
initialDeviceCount: 4
maxDeviceCount: 11
journalDeviceLayout:
deviceMatchers:
- partition == false
- size < gib(800)
- size > gib(400)
numDevices: 2
- instanceID: 1
data_ip: 10.20.155.3
ec_enabled: false
failure_domains:
- server00
storageDeviceLayout:
allowCrossNumaDevices: false
deviceMatchers:
- partition == false
- size > tib(1)
initialDeviceCount: 4
maxDeviceCount: 11
journalDeviceLayout:
deviceMatchers:
- partition == false
- size < gib(800)
- size > gib(400)
numDevices: 2
