Known Issues in Lightbits 3.21.1

49019

In rare cases, after a node becomes active, the frontend service on this or another node could transiently fail a replication connection (with connection error: seastar::metrics::double_registration, automatically reestablished), thus leaving some stale stats under /mnt/stats. Subsequently, an access to the stats (manual or via lb_support) could cause a frontend segfault.

48993

When a cluster upgrade is started with a proxy password or a registry token, these credentials are written in clear text to service logs and the system journal on the cluster servers, and are therefore included in support bundles. They are also returned in the error message when a second upgrade is requested while one is already in progress. Workaround: Use dedicated, short-lived proxy credentials and registry tokens for upgrades, rotate them after the upgrade, and treat logs and support bundles from upgraded servers as sensitive.

48991

In the lb-csi Helm chart, any non-empty value of inBandAuth other than the boolean false enables NVMe In-Band Authentication in the node plugin, without a warning. For example, the quoted string "false" or a misspelled value turns In-Band Authentication on. Workaround: Set inBandAuth only to an unquoted boolean, true or false.

48949

In container-based deployments, if the Lightbits server container is terminated abruptly (for example, with SIGKILL), the node may fail to restart. The node-manager service and its container report as running, but the node stays Inactive in the cluster, and a service restart can hang for up to 30 minutes. Workaround: Reboot the affected server.

48930

In a rare situation where during a rebuild at least two snapshots are deleted, one containing data and a higher one containing a trim to the same blocks, the trim might be applied to the next existing version/snapshot - thus losing the valid data in it.

48695

In v3.21.1, the cluster manager can stop placing new volumes on a node that still has free capacity. This can happen when volumes are created and deleted while protection groups are migrating - for example, after a node fails or is disabled. Existing volumes on the node are not affected, but all new volumes are placed on other nodes, which can lead to uneven capacity use and to volume creation failing for lack of space even though free capacity exists. Workaround: Restart the cluster-manager service on the server that runs the active cluster manager; this rebuilds the capacity accounting and restores normal placement.

48673

In Lightbits v3.19 and later, a server can fail to return to Active after it reconnects to the cluster. This can happen when, while the server was disconnected, a volume was created, a snapshot of that volume was taken, and a clone of that snapshot was created, each with a replica on that server. After connectivity is restored, the server could fail to rebuild the clone's replica, and it remains Inactive. Impact: No data is lost, and volumes stay available through their replicas on other servers. Until the server becomes Active again, every volume with a replica on that server has reduced redundancy, and the cluster cannot restore full protection on that server. Workaround: Contact Lightbits Support.

48647

In Lightbits v3.21.1, adding an NVMe device can fail with a "no empty slot" error if the only free slot on the node is still being reconstructed by a local RAID rebuild; the device then remains unmanaged. Workaround: Wait for the local rebuild to finish, and then add the device again.

48621

In Lightbits v3.19.2 and later, when the cluster reaches its maximum number of stored events, automatic event cleanup can delete the wrong events: instead of the oldest events, it removes events by identifier order, which can silently erase whole periods of recent event history. Deleted events cannot be recovered.

48620

In container-based Lightbits deployments (v3.19.2 and later), support bundles collected on the host with the Ansible logs-collector role are incomplete: lbcli command output files are empty, the event and snapshot listings and the etcd JSON dump are missing, and no package list is collected. Workaround: Collect support logs with lbcli fetch logs (or the equivalent API call), which is not affected.

48572

In rare cases, if the node manager service on one server is restarted shortly after data has been moved between servers (for example, during a server eviction), the node manager service on other servers may crash and restart automatically. This can happen whether the first restart is unexpected or part of planned maintenance. During this short period, the availability of some volumes may be impacted. The cluster recovers without any action from the operator.

48546

node-manager can fail to start after losing a data drive when journaling is enabled and the journal devices are selected from a device pool that overlaps with the data devices. This is a related but distinct trigger from the general "missing drive" restart failure: it can still occur even once that issue is fixed. No workaround is currently available.

48538

If the rebuild on the destination node of a protection group migration fails - for example, a migration started by proactive rebalance after a node is added - the migration can remain stuck indefinitely. The migration task stays in the Running state, the protection group keeps an extra, unsynchronized replica, and the rebuild is not retried automatically, including after a cluster-manager restart. Affected versions: v3.18.2 through v3.21.1. Workaround: Contact Lightbits Support to restart the rebuild. Do not restart services on the destination node to recover, because this can cancel the migration and discard the rebuild progress.

48498

In rare cases, the storage backend service on a node can stop unexpectedly while the node is rebuilding data locally after a drive failure, particularly when data is also being rebalanced onto that node and the node's storage is close to full. Shortly before it stops, reads served by that node may slow down or be retried. The node then becomes inactive and the system tries to restart the service automatically; until the node is back, volumes with a replica on it may be degraded, read-only or unavailable. Affected versions: found in v3.21.1; earlier versions may also be affected.

48452

The installer's etcd disk performance check does not work as intended; it can pass disks whose write IOPS are well below the required 10,000. Workaround: Before installation, verify the etcd disk's random-write performance manually (for example, with fio), and make sure it meets the 10,000 IOPS requirement.

48435

On some AMD EPYC servers, NVMe SSDs could report I/O timeouts. This can cause a storage node to be marked as failed and - while it is out of service - volumes with reduced protection could become read-only. Workaround: Disable PCIe Relaxed Ordering in the server BIOS.

48171

When a server becomes read-only after running out of metadata capacity, the proactive rebalance feature does not move any volumes or snapshots off that server, even when other servers have free space. The server cannot recover on its own, and volumes with a replica on it remain in Degraded or ReadOnly protection state until capacity is freed manually. This affects all currently released versions. Workaround: Free capacity on the affected server - for example, by deleting unneeded volumes or snapshots that have a replica on it - or contact Lightbits Support.

48062

In v3.21.1, when NVMe devices are added to the same node in quick succession, adding a device may fail with an AddNVMeDeviceOperationFailed event (cause AddNVMeDeviceTaskFailure), and the device is not added to the node.

48003

In v3.21.1, if a node is down while the cluster moves one of its volumes' replicas to other nodes, and a different node later fails, the cluster may not rebuild the affected volume back onto the recovered node, leaving the volume in Degraded protection state indefinitely. Workaround: Restart the node-manager service on the recovered node; the cluster then completes the rebalance and the volume returns to FullyProtected.

47894

On clusters that use volume IP access control lists, if a client IP address is removed from a volume's IP ACL, or the volume is deleted, while the node-manager service on a server is stopped or restarting (for example, during a rolling upgrade), the client keeps network access to that server's NVMe and discovery ports after node-manager starts again. The access remains until the server is rebooted. Workaround: Avoid changing volume IP ACLs or deleting volumes while node-manager is stopped or an upgrade is in progress; if this has happened, reboot the affected server to remove the stale access rules.

47887

In the Lightbits vSphere plugin, the QoS Policy list in the Edit Volume dialog includes an entry named "default" that does not correspond to any QoS policy on the cluster. If "default" is selected, the dialog cannot be submitted, and any other changes made in the same dialog (such as ACL, size or compression) are silently discarded without an error message. Workaround: To remove a rate limit from a volume, select the cluster's "no-limit-policy" QoS policy instead of "default".

47753

Changing a node's Permanent Failure (PF) threshold can behave unexpectedly if the node is already Inactive; lowering the threshold to a value that has not yet elapsed does not take effect until the original threshold time passes, and raising the threshold on an already-inactive node can prevent it from reaching Permanent Failure at all. Workaround: Change a node's PF threshold before performing maintenance that will make it inactive. If you need to shorten the threshold after a node is already inactive, use a value that has already elapsed.

47318

When updating a volume with 'lbcli update volume', --acl and --ip-acl entries separated by a comma followed by a space (for example, "nqn.a, nqn.b") are not trimmed. A host NQN with a leading space is stored as-is and matches no host, so that host loses access to the volume; an IP address with a leading space is rejected. Workaround: Separate --acl and --ip-acl entries with commas and no spaces (for example, "nqn.a,nqn.b"), or repeat the flag once per entry.

47300

If the cluster manager's connection to the cluster's internal coordination service is interrupted (for example, due to a leader change, network disruption, or a brief service restart), the cluster manager's view of cluster servers can stop updating entirely, with no automatic recovery. Workaround: Restart the cluster-manager service to restore normal operation.

47035

In rare cases, after a rolling upgrade or another change of the active paths, a client host may lose access to a volume: I/O to the volume hangs and the client kernel logs 'no usable path - requeuing I/O', although the volume is healthy on the cluster. The cause is under investigation. If this occurs, contact Lightbits Support.

46633

On large clusters (for example, clusters with more than 1,000 protection groups), evicting a server can fail with a timeout before any data has been moved, because the cluster manager takes a long time to prepare the eviction. Workaround: Increase the EvictionNoProgressTimeout cluster configuration parameter. Contact Lightbits Support for the recommended value for your cluster.

45712

In rare cases, if the internal connection used to track volume or snapshot state changes is interrupted (for example, due to a leader change, network interruption, or a brief service restart), a volume or snapshot update can remain stuck showing as still in progress even though the underlying change has completed. Workaround: Restart node-manager on the affected server to resolve the stuck state.

45202

When a volume is created with lbcli, any part of a label key before a hyphen is dropped. For example, --labels="pve-vmid=100" is saved with the key "vmid" instead of "pve-vmid". Workaround: Avoid hyphens in label keys, or create the volume through the REST API, which is not affected.

45157

During a node power-up after an abrupt shutdown, a node's reported power-up progress could reach 100%, drop back to 50-60%, and then climb to 100% again. This is because the last of three internal recovery phases is not included in the calculation. This affects the reported percentage only. Recovery completes correctly and the node returns to service only when it is actually done.