Known Issues in Lightbits 3.19.1

46633

Cluster eviction might be delayed and time out before it starts, due to a long effort of pre-processing of cluster objects. The issue is relevant for clusters with a large number of nodes, and known large number of PGs (more than a thousand protection groups).

Workaround: Increase cluster-config EvictionNoProgressTimeout timeout. Consult Lightbits for the best value for a specific cluster.

46209

The bundled Grafana Alloy log-streaming agent ships with Grafana's anonymous usage statistics reporting enabled by default. The report is sent out on a fixed four-hour schedule to https://stats.grafana.org/. In air-gapped or firewalled deployments, these attempts fail and produce recurring error entries in the journal. In connected deployments, they may represent an unsanctioned outbound connection to a third party. The exact list of fields collected is documented at https://grafana.com/docs/alloy/latest/data-collection/.

46108

In rare cases, adding an NVMe device may fail when the add operation coincides with lightbox-exporter collecting NVMe error metrics for the same device. If this occurs, re-issue the add device command.

45729

When SSD journaling is configured with multiple NVMe devices, cluster cleanup does not remove the RAID (mdadm) arrays created by Lightbits. Before re-deploying on the same servers, manually remove these arrays using mdadm.

44875

lbcli and REST documents describe that you can evict a server with 1x volumes with the force flag, but this is not possible. It also indicates that you can evict a server even if it could lead to risk of service loss using the force flag. However, force will only allow you to evict a server if its nodes are unavailable but still requires other nodes to be active.

44819

Lightbits release 3.19.1 introduced stringent validation of any REST requests sent to the Lightbits api-service. While previous faulty requests that had some non-supported fields or corrupted fields would have been accepted, and those fields would just be ignored, the 3.19.1 rejects such malformed requests.

44762

Log stream to a rsyslog target only supports the secured mode of operation. Unsecured mode is not currently supported.

44673

In rare cases, performing a KEK rotation while simultaneously executing a high volume of volume/snapshot control plane operations could result in increased etcd contention. This affects only customers utilizing cluster-level encryption with the KEK rotation API..

44642

Update volume commands issued from VCP V1.4.0 (or older) to a Lightbits cluster running Lightbits 3.19.1 release or newer will fail due to invalid arguments.

44435

pmem_init and lbe executables are linked using staticx and therefore require /tmp to be mounted with the exec option enabled; otherwise, these executables will fail to run with "permission denied".

44386

Grafana dashboards may display a short spike of negative BW/Throughput rate, following the stoppage of a Lightbits service.

44383

When encryption is enabled, deleting a server that hosts the Active CM could prevent subsequent server additions from completing successfully. To avoid this, stop the CM service on the removed server (or shut down the server) after it has been removed from the cluster.

44382

When a node manager is offline, if the sole journal NVMe device is removed or fails and becomes invisible at the OS level, the failed device might not be detected upon node restart. The node will resume normal operation, but without journaling protection. This affects only nodes configured with a single journal NVMe device.

44159

In a rare case, a rebuild could fail after the following sequence: two or more snapshots are created while a node is inactive, and then all are deleted with at least one deletion occurring after the node's recovery had started. This has been resolved.

44124

Lightbits 3.19.1 enforces consistent formatting for Protocol Buffer (protobuf) duration fields: values must now be expressed in seconds (e.g., 3600s). Alternative unit formats such as 60m or 1h are no longer accepted.

43471

[Log Streaming] Log lines longer than the destination rsyslog server's maxMessageSize (default 8 KB) are truncated by the receiving rsyslog. To stream full-length lines (Lightbits logs can reach ~48 KB), increase maxMessageSize on the rsyslog server — e.g. global(maxMessageSize="64k") before any inputs — and restart rsyslog. This setting is on the rsyslog server and cannot be configured from Lightbits.

42963

When SSD Journaling is enabled, if Duroslight fails due to a non-journal-related issue, the Node Manager (NM) might incorrectly classify the failure as a journal device failure, causing the NM to remain inactive and enter a Permanently Failed state. When SSD Journaling is disabled, a Duroslight failure does not impact the failure scenario. However, users might receive a spurious "journal device failed" event even when journaling is not in use. This is cosmetic only and does not reflect an actual journal issue.

40883

In the specific case of using VCP to upgrade a cluster, the upgrade to Lightbits version 3.17.1 or higher will fail because VCP cannot parse the new version format. To successfully upgrade the cluster, use the Lightbits core CLI or REST API directly.

37830

In a very rare case, a node could fail to recover and return to an active state if an I/O error or bad block is encountered on an underlying SSD during its startup sequence. This issue prevents a key service (gftl) from initializing correctly and could require manual intervention - such as the removal of the failed SSD from the system - to allow the node to successfully complete its recovery.

37505

In a rare combination of events, the 'physicalOwnedCapacity' volume statistic could report an incorrect value if data at a specific LBA is overwritten with content that has a different compression ratio. In this scenario, the updated length of the overwritten data is not correctly accounted for in the statistic.

28027

A server upgrade status will not update in the following sequence: 1. A server is upgraded to release x.y.z. 2. The operation fails (i.e., times out); however, binaries on the server are updated to version x.y.z. 3. At a later time, the upgrade is attempted again to version x.y.z (this operation is skipped internally, as binaries have already been updated). 4. The upgrade status will continue to show the failed upgrade operation, even though the last upgrade returned with no error.