Lightbits TRIM Support

The Lightbits TRIM feature allows filesystems to reclaim unused storage space by issuing commands to the storage system (called trim/discard operations). Lightbits handles trim/discard operations issued by the filesystem on its volumes. Combined with Lightbits’ thin provisioning capabilities, this improves storage performance and capacity efficiency.

For instance, the 'fstrim' operation on an XFS filesystem mounted on a Lightbits volume can reduce storage utilization (reduce volume usage size), when invoked with the TRIM feature enabled.

Lightbits has implemented the NVMe deallocate as specified in the NVMe command set specification, to improve capacity utilization and planning. This capability works with the TRIM command, allowing the operating system to inform Lightbits about unused data blocks. For more information, refer to sections 3.3.3 and 3.3.3.2.1 of the NVMe command set specification.

When the OS deletes a file and sends a TRIM command, it is translated to an NVMe dataset management deallocate command on the client side. Lightbits receives this deallocate command and manages the information internally within its software, behaving as if the data was deallocated. This approach enables more efficient capacity utilization and garbage collection - and improves write performance and endurance.

High-Level Workflow

Configuring the TRIM Feature on the Lightbits Cluster

The TRIM feature must be enabled for 3.16 installations. For existing deployments, upgrade to 3.16 and perform the following steps to enable the TRIM feature.

Once enabled, this feature cannot be disabled.

Enable the feature with this command:

Bash
    
 
> lbcli update cluster-config --parameter=EnableTrim --value=true
Copy

Enabling TRIM does not automatically reclaim space from previously deleted data on existing volumes. To initiate space reclamation on existing volumes after enabling TRIM, refer to the instructions in the Best Practices section below.
It is not necessary to restart Node Manager. The feature can be enabled by invoking an API (after the upgrade).
In version 3.16.1, lbcli list feature-flags will always show trim as false. Use lbcli list cluster-config to get its status.

Viewing Reduced Storage Using TRIM

This example details the complete process of enabling TRIM, creating and deleting files, and viewing its effects on storage usage.

Create a Lightbits volume and set up the filesystem:

Bash
    
> lbcli create volume --project-name=default --name="test" --acl="test" --replica-count=3 --size=1G#On the client machine> discovery-client add-hostnqn -a 10.20.59.4:8009 -n nqn.2016-01.com.lightbitslabs:uuid:4f47dd3a-4dc8-439a-ad46-bed23e1bbdba:suffix -q test --name dummy.txt> nvme list… /dev/nvme0n1          6842f8294caf4216     Lightbits LightOS                        1           1.00  GB /   1.00  GB      4 KiB +  0 B   3.10 …> mkfs.xfs /dev/nvme0n1> mkdir /tmp/my_test_fs> mount /dev/nvme0n1 /tmp/my_test_fs
Copy

Check the initial storage usage:

Bash
    
 
> lbcli list nodes
Copy

Use the dd command to create a ~0.5GB file (or 0.5GB per replica/node):

Bash
    
 
# On the client machine> dd if=/dev/random of=/tmp/my_test_fs/my_file.bin bs=1M count=500
Copy

Wait approximately one minute, and then check the storage usage:

Bash
    
 
> lbcli list nodes
Copy

Note the increase in used storage.

Delete the file:

Bash
    
 
# On the client machine> rm /tmp/my_test_fs/my_file.bin
Copy

Check the storage usage again:

Bash
    
 
> lbcli list nodes
Copy

Note that the used storage has not decreased yet.

Enable the TRIM feature on the Lightbits cluster:

Bash
    
 
> lbcli update cluster-config --parameter=EnableTrim --value=true
Copy

Verify that the TRIM feature is set to “true”:

Bash
    
 
> lbcli list cluster-configCluster Config Parameter ValueAllowedNumRevives 2ClockDriftIntervalForRaisingEvent 24h0m0sClusterName 41bcae54-1906-54a4-99bc-28a3e70fcaf9DefaultQoSName no-limit-policyDeviceHealthIntervalForRaisingEvent 24h0m0sDisableVolumeStatisticsExport falseDurationToTurnIntoPermanentFailure 4h0m0sEnableTrim trueEvictionNoProgressTimeout 1h0m0sRevivesWindowDuration 2h0m0sVolumeDeletionFromNodeDelay 30s
Copy

Set up and run fstrim.timer on the client machine:

Bash
    
​x
 
> sudo systemctl enable fstrim.timer> sudo systemctl start fstrim.timer​# Verify the timer status> sudo systemctl status fstrim.timer
Copy

Manually trigger a TRIM operation to see its immediate effects:

Bash
    
 
> sudo systemctl start fstrim.service
Copy

Wait approximately one minute, and then check the updated storage usage:

Bash
    
 
> lbcli list nodes
Copy

You should now see a reduction in the "used" storage, indicating that the TRIM operations have successfully reclaimed space.

Example Output:

Bash
    
> lbcli list nodesName         UUID                                   State     NVMe endpoint     Failure domains                 Capacity   Used      Local rebuild progressserver00-0   36469d85-0001-5a22-80b0-72f1d339f250   Active    10.20.59.4:4420   [rack06-server70-vm02 ser...]   7.0 GiB    536 MiB   Noneserver01-0   2fb1c01b-b4b6-573c-a320-4295372e55d8   Active    10.20.59.5:4420   [rack06-server64-vm05 ser...]   7.0 GiB    536 MiB   Noneserver02-0   a0625c8f-eccb-56f4-80d3-b95a836085a0   Active    10.20.59.7:4420   [rack05-server74-vm05 ser...]   7.0 GiB    536 MiB   None
Copy

To verify that TRIM is working on an ongoing basis:

Create another test file:

Bash
    
 
# On the client machine> dd if=/dev/random of=/tmp/my_test_fs/another_file.bin bs=1M count=300
Copy

Check the storage usage:

Bash
    
 
> lbcli list nodes
Copy

Delete the file:

Bash
    
 
# On the client machine> rm /tmp/my_test_fs/another_file.bin
Copy

Wait for the next scheduled TRIM operation (or trigger it manually), and then check the storage usage again:

Bash
    
> lbcli list nodesName         UUID                                   State     NVMe endpoint     Failure domains                 Capacity   Used      Local rebuild progressserver00-0   36469d85-0001-5a22-80b0-72f1d339f250   Active    10.20.59.4:4420   [rack06-server70-vm02 ser...]   7.0 GiB    32 MiB    Noneserver01-0   2fb1c01b-b4b6-573c-a320-4295372e55d8   Active    10.20.59.5:4420   [rack06-server64-vm05 ser...]   7.0 GiB    32 MiB    Noneserver02-0   a0625c8f-eccb-56f4-80d3-b95a836085a0   Active    10.20.59.7:4420   [rack05-server74-vm05 ser...]   7.0 GiB    32 MiB    None
Copy

TRIM Operations - Best Practices

Initial Setup After Enabling TRIM

Enable TRIM on the Lightbits cluster:

Bash
    
 
> lbcli update cluster-config --parameter=EnableTrim --value=true
Copy

On each client machine, set up fstrim.timer:

Bash
    
 
> sudo systemctl enable fstrim.timer> sudo systemctl start fstrim.timer
Copy

Verify the timer status:

Bash
    
 
> sudo systemctl status fstrim.timer
Copy

Run an initial manual TRIM to reclaim space from previously-deleted data:

Bash
    
 
> sudo systemctl start fstrim.service
Copy

Regular Maintenance

Periodically check the status of fstrim.timer:

Bash
    
 
> sudo systemctl status fstrim.timer
Copy

Monitor the storage usage after TRIM operations:

Bash
    
 
> lbcli list nodes
Copy

If the storage is not being reclaimed as expected, manually trigger TRIM:

Bash
    
 
> sudo systemctl start fstrim.service
Copy

Customizing the TRIM Schedule

If a weekly schedule is not suitable, customize the TRIM schedule as follows on every client machine.

Create a custom timer file:

Bash
    
 
> sudo systemctl edit fstrim.timer
Copy

Add your custom schedule. For example, to run it daily:

Bash
    
 
[Timer]OnCalendar=daily
Copy

Save the file and reload the systemd daemon:

Bash
    
 
> sudo systemctl daemon-reload
Copy

Restart the timer:

Bash
    
 
> sudo systemctl restart fstrim.timer
Copy

Avoiding Continuous TRIM

Do not use the -o discard mount option for filesystems on Linux, as it can significantly impact filesystem performance.

Red Hat does not recommend using Continuous TRIM. See the relevant Red Hat documentation for additional information.

TRIM Troubleshooting

If TRIM is not working:

Verify that it is enabled on the Lightbits cluster:

Bash
    
 
> lbcli get cluster-config
Copy

Check the fstrim.timer status on the client:

Bash
    
 
> sudo systemctl status fstrim.timer
Copy

Review the system logs for any TRIM-related errors:

Bash
    
 
> sudo journalctl | grep fstrim
Copy

If these or other issues persist, contact Lightbits Customer Success for assistance.

Last updated on

Was this page helpful?