Release 3.21.1

New Feature (GA)

Added Rebuild and Rebalance Throttle Controls, a cluster-wide setting that caps the bandwidth used by rebuild and rebalance traffic on each node and sets whether this background I/O or user I/O takes priority, so operators can protect application performance during recovery or let recovery run faster when needed.

LBM1-49133

New Feature (GA)

Added Spare Capacity Reservation, a cluster-level setting (SpareCapacityReservationPercentage, 0-50%) that keeps part of each node's capacity free so surviving nodes always have room to rebuil after a node failure.

LBM1-49135

New Feature (GA)

Server eviction (evict data), introduced as a Technology Preview, is now generally available and officially supported for production use. Evicting a server migrates all of its volumes and snapshots to other servers in the cluster before planned maintenance or removal, keeping volumes fully protected throughout, and can also be used to move data off a server whose nodes have become unavailable without waiting for them to reach a permanently failed state.

LBM1-49135

New Feature (Tech Preview)

Added Tech Preview support for installing the Lightbits storage cluster on Red Hat Enterprise Linux (RHEL) 10, extending the server OS options beyond the existing RHEL family and Ubuntu releases (RHEL 10 has been supported on the client side since earlier releases). This Tech Preview is intended for evaluation only: customers are encouraged to validate and test their workloads with Lightbits deployed on RHEL 10 in non-production environments. It is not supported for production use and is not generally available in this release.

LBM1-49136

Enhancement

Clarified the event raised when a container upgrade partially fails to refresh configuration files: the description no longer truncates mid-path, and a genuine error reading the target image is now surfaced as a warning instead of being silently treated as ‘nothing to refresh’.

LBM1-45797

Enhancement

Fixed a packaging issue where an upgrade could overwrite a cluster's customized /etc/profile-generator/user-config.yaml; it is now correctly preserved across upgrades in both RPM and container mode.

LBM1-45583

Enhancement

Fixed a rare race condition that could cause a volume replica to serve outdated data. It required a volume with two or more replicas whose primary server had become read-only after running out of storage capacity or metadata space, while another replica finished rebuilding and took over as primary. If the first server's read-only state cleared within the same fraction of a second as that takeover, it could return as primary without the rebuild it needed.

LBM1-39499

Enhancement

Fixed an issue where a Protection Group could remain stuck as read-only for an extended period after a node migration, even once the affected node had become writable again. A node whose rebuild had actually completed was still counted as 'not yet synced' until it reached the up-to-date state, which kept the count of synced nodes below the threshold needed to leave read-only mode.

LBM1-34456

Enhancement

Hardened the OpenStack Cinder driver: a process-global TLS-warning suppression that could silence certificate-verification warnings for every other backend sharing the same cinder-volume process has been removed, the management-channel help text no longer contradicts the driver's HTTPS-only behavior, and the API JWT can no longer appear in logs at DEBUG level or in a generated sample config.

LBM1-47383

Enhancement

Improved cluster rebalance performance on large clusters: the per-node selection pass used to do far more work than its answer needed, which could make a drain or rebalance take too long to finish on clusters with many nodes and protection groups. (Measured: roughly 3.4x fewer allocations for a 32-node, 480-group rebalance pass.)

LBM1-47484

Enhancement

K8S CSI driver IP-ACL enforcement support: the new opt-in ip-acl StorageClass parameter makes the plugin manage each volume's IP-ACL - the node plugin enrolls the node's data-path addresses at stage time and the IP-ACL is reset when the last node detaches. Statically provisioned volumes opt in with the |ipacl:enabled volumeHandle suffix, and the workload-examples charts expose the parameter end to end.

LBM1-40422

Enhancement

Lightbits CSI plugin: Added support for NVMe in-band authentication (DH-HMAC-CHAP). When the new inBandAuth Helm value is enabled and the Lightbits cluster has In-Band Authentication turned on, each node registers itself as a trusted host, its secrets are generated by the cluster and stored in the discovery-client configuration, and every NVMe/TCP connection is authenticated. On clusters without In-Band Authentication, the plugin connects as before. This requires the discovery-client to run in a container; otherwise the Helm chart does not render.

LBM1-48471

Enhancement

List snapshots now allows filtering by a volume_uuid; e.g., lbcli list snapshots --project-name X --volume-uuid Y.

LBM1-47642

Enhancement

Log streaming: when an rsyslog target's endpoint includes a port or a scheme (for example "host.example.net:24712" or "tcp://host.example.net"), the API now returns a clear error saying the endpoint must be a hostname or IP address only and the port must be set in the separate port field. Other invalid endpoint values now report the expected format (IP address or RFC 1123 hostname). FQDNs are supported. The API reference now documents the rsyslog endpoint, port, timeout and debug fields. lbcli is not affected: its --endpoint flag still accepts host:port.

LBM1-48175

Enhancement

Protection groups left behind when nodes become permanently failed are now removed automatically, every 12 hours by default. On long-running clusters these leftover protection groups can build up and slow down switchovers. Upgrade note: after upgrading from 3.20.1, the cleanup is enabled on every cluster, including clusters where it had been turned off. To keep it off, set the cleanup interval back to 0 after the upgrade.

LBM1-47138

Enhancement

The installation TUI can now deploy Lightbits in container mode on Ubuntu and Debian. It detects the target operating system, signs in to the container registry and pulls the images, checks before deployment that the selected Lightbits version supports the target servers, and verifies the installation with lbcli when it finishes.

LBM1-45442

Enhancement

The installation TUI's Remote Targets step now supports bulk server entry: paste a list of addresses, including ranges and CIDR blocks, instead of adding hosts one at a time.

LBM1-46424

Enhancement

The KATO grace period, previously hardcoded to 10 seconds, is now configurable via nvme_kato_grace_secs; setting it to 0 disables the KATO grace counter.

LBM1-47388

Enhancement

The Node dashboard now includes a panel showing metadata utilization as a percentage.

LBM1-46712

Enhancement

When a node's journal device fails, the node now automatically creates and uses a new journal device on restart, provided it can shut down gracefully and enough devices remain. This reduces node downtime after a journal device failure.

LBM1-42323

Enhancement

When installing on Ubuntu, the installer now automatically opens the ports that Lightbits requires in the ufw firewall.

LBM1-47495

Major

Fixed a bug where deleting a snapshot during a rebuild could cause a damaged object to be read on the primary, triggering data-integrity alerts and preventing the affected data from being correctly persisted on the secondary.

LBM1-47343

Major

Fixed an edge case where the frontend could hang for up to 120 seconds due to a missing timeout on the replicator connection.

LBM1-46486

Major

Fixed an issue where a trim (deallocate) request that arrived during a replica configuration change, such as a node failure, could be reported to the host as successful even though a replica had not applied it, leaving the replicas inconsistent. Such requests now fail back to the host and are retried on the new primary through an ANA path change.

LBM1-48806

Major

Fixed an issue where aborted client-kernel NVMe commands triggered unexpected node down/up state transitions in VMware environments.

LBM1-45998

Major

Fixed an issue where, after a node was restarted or replaced, volume recovery on that node could stall indefinitely on clusters with many clone volumes created from a small number of shared snapshots. Clone volumes could be processed before the snapshots they depend on and use up all available recovery workers, so the source snapshots were never processed and the affected protection groups could not finish rebuilding. Source snapshots are now processed before their clones.

LBM1-48205

Major

Fixed failing node-manager start when the number of healthy SSDs is lower than the InitialDeviceCount.

LBM1-48532

Major

PEERMON (connectivity report) now fails a node if any of the replication connections to it has more than 15s of down time with no live link (configurable in duroslight/conf.yaml as replicator/peer_dead_timeout_ms).

LBM1-45912

Major

Proactive rebalance algorithm priority tweak. No difference in behavior should be perceivable externally.

LBM1-47330

Minor

An edge case during migration would allow migration to continue even if the target node was permanently failed. Migration will now exit when its target node is permanently failed.

LBM1-44464

Minor

Fixed a failure in cluster root key rotation on TPM-backed clusters, where the rotation could time out and report an internal error. TPM operations are now given a longer default timeout and are bounded so that a stalled helper process cannot block the rotation.

LBM1-46100

Minor

Fixed a false UnrecoverableDataIntegrityError event, with no volume ID, and a "Rebuild read detected a damaged object" message in the frontend log. This could happen during a rebuild when a volume with previously deleted snapshots was deleted.

LBM1-48646

Minor

Fixed a volume leak when retyping an OpenStack Cinder volume between projects.

LBM1-44442

Minor

Fixed an issue in the management services where an operation that kept failing was eventually retried with almost no delay. After enough consecutive failures (about 1.5 minutes of retries on 3.18.x, or about 16 hours on 3.19.x and 3.20.x), the delay between retries dropped to nearly zero. The continuous retries could keep a CPU core busy and flood the logs, pushing out the older log entries needed to investigate the original failure.

LBM1-48041

Minor

Fixed an issue where a node that had just restarted could briefly report its power-up as already 100% complete, based on a leftover value from before the restart; this could let a recovery step proceed before the node was actually ready. A node that cannot currently be queried now correctly reports unknown progress instead of a stale value.

LBM1-47398

Minor

Fixed an issue where a snapshot or volume created while a node was inactive, and deleted while that node was still powering up, could stay behind on the node. The leftover could block deletion of the snapshot it was created from, or prevent volumes on that node from rebuilding correctly.

LBM1-48012

Minor

Fixed an issue where a volume rebuild could fail if a snapshot was deleted during the rebuild.

LBM1-46869

Minor

Fixed an issue where the node manager completed its shutdown before the data-path service had stopped, which delayed the server's shutdown. The data-path service now has dedicated time to stop.

LBM1-44334

Minor

Fixed an RPM upgrade issue where the Lightbits package's etcd version requirement (>= v3.5.30) could be satisfied by version-string comparison even against an older installed etcd package (such as 3.5.18), so an upgrade from v3.18.3 or v3.20.1 could leave the cluster on an outdated etcd. Upgrades now pull in the correct etcd version.

LBM1-48605

Minor

Fixed an upgrade preflight check that could intermittently pin its etcd snapshot to an already-down etcd member and fail with a timeout even though cluster quorum was intact; the preflight now selects a healthy member so cluster upgrades start deterministically.

LBM1-46187

Minor

Fixed incorrect file permissions on container service unit files (e.g., a .service file marked executable when it should not be), which could trigger incorrect systemd warnings during installation.

LBM1-46294

Minor

Fixed node cleanup (Ansible) not stopping and removing leftover journaling RAID (mdadm) devices, which could cause a subsequent install to fail due to insufficient available NVMe drives.

LBM1-45625

Minor

Fixed the Chrony and Cluster dashboards in the monitoring stack, which showed no data because of an incorrect data-source reference.

LBM1-45884

Minor

Fixed the TCP segments re-transmitted panel in the monitoring dashboard showing negative values after a counter reset.

LBM1-43755

Minor

Fixed the volume-state panel on the Cluster dashboard, which reported an incorrect number of volumes in the migrating state.

LBM1-46295

Minor

Fixed volumes configured with a specific IP-ACL (not allow-any) being invisible to an allowed host when the node is set to listen on all interfaces; an allowed IPv4 host address arriving as an IPv4-mapped IPv6 address no longer fails the ACL match.

LBM1-46207

Minor

OpenStack Cinder: retyping a volume to change only its QoS policy now updates the policy in place, without migrating the volume's data.

LBM1-47082

Minor

Reduced logging noise from lightbox-exporter: a repeated SMART or NVMe error-log collection failure is now logged as a warning once, and again only when the failure changes or clears.

LBM1-44531

Minor

Removed an incorrect profile-generator startup failure seen on some installs when node-manager already held the shared local database lock; profile-generator no longer depends on that lock and will not fail to start for this reason.

LBM1-46174

Minor

SSD SMART alert reliability. Aligned the for: durations of the SSD SMART gauge-threshold alerts (SmartTemperatureCritical, SmartTemperatureExtreme, SmartCriticalWarning, SmartDeviceUnhealthy, SmartAvailableSpareLow, SmartCriticalWear) to 10 minutes so a single missed Prometheus scrape (short network blip, exporter restart, target-response jitter) no longer resets the alert's evaluation streak. Real incidents that briefly race a scrape now fire reliably instead of being suppressed until the next continuous window; the observable alert-firing latency for a persistent overheating or health event stays under 15 minutes end to end.

LBM1-44667

Minor

The API now returns a conflict (HTTP 409) instead of an internal server error (HTTP 500) when a request is rejected because another task is already running on the same resource.

LBM1-46814