Lightbits Supported Events
Lightbits services will generate events for various operations, activities, statuses, errors, or warnings that can occur in a Lightbits cluster. These events can be collected via official Lightbits API, or - from Release 3.0.3 - by scraping them from the Lightbits services log files.
The format and information of events will be identical in either format they collected through (API/logs). Events in logs will be preceded by a dedicated "LBEVENT" tag, to simplify filtering them out of logs.
The default location for all logs of Lightbits services is /var/log/<service-name>.log; however, this may be customized in each cluster install by updating the service role template:
logging:
filename: /var/log/node-manager.log
Event Name | Event Cause | Event Code | Cause Code | Severity | Event Type | Reporting Service | Description |
|---|---|---|---|---|---|---|---|
Node Inactive | Local RAID failure | 201 | 102 | High | Node | Node Manager | Node state has changed to inactive due to multiple NVMe device disk failures (EC enabled)/single disk failure (no EC). Note: This event was renamed. |
Node Inactive | OS health check failed | 201 | 110 | High | Node | Node Manager | Node state has changed to inactive due to a kernel panic being detected in the OS. |
Node Inactive | Duroslight health check failed | 201 | 105 | Medium | Node | Node Manager | Node state has changed to inactive due to a failure with Duroslight. |
Node Inactive | Heartbeat to node failed | 201 | 107 | Medium | Node | Cluster Manager | Node state has changed to inactive due to a node being unreachable (network failure/server down). |
Node Inactive | Connectivity issue | 201 | 108 | Medium | Node | Cluster Manager | Node state has changed to inactive due to connectivity errors in dataplane. |
Node Inactive | FailedToEnableEncryption | 201 | 1304 | High | Node | Cluster Manager | Node state has changed to inactive due to failure in encryption. |
Node Inactive | GftlReplaceDiskAfterPowerUp | 201 | 118 | High | Node | Cluster Manager | Node state has changed to inactive: GFTL reports that a disk needs to be replaced after power-up. |
Node Inactive | GftlLFError | 201 | 120 | High | Node | Cluster Manager | Node state has changed to inactive due to a GFTL logical-format (LF) error. |
Node Inactive | GftlStorageSchemeDataIntegrityError | 201 | 121 | High | Node | Cluster Manager | Node state has changed to inactive due to a GFTL storage-scheme data integrity error. |
Node Inactive | GftlDataIntegrityError | 201 | 122 | High | Node | Cluster Manager | Node state has changed to inactive due to a a GFTL data integrity error. |
Node Inactive | GftlUnknownStatus | 201 | 123 | High | Node | Cluster Manager | Node state has changed to inactive: GFTL reported an unknown/unrecognized status. |
Node Inactive | GftlSystemCapacityReadStatusError | 201 | 124 | High | Node | Cluster Manager | Node state has changed to inactive due to a GFTL error reading system capacity status. |
Node Inactive | GFTLInstanceFileAccessError | 201 | 125 | High | Node | Cluster Manager | Node state has changed to inactive due to a GFTL instance file access error. |
Node Active | N/A | 200 | 1 | Info | Node | Cluster Manager | Node is active. |
Node in Permanent Failure State Inactive | Node Inactive | 206 | 112 | High | Node | Cluster Manager | Node entered permanent failure state (duration of node in inactive state exceeded configured time window of inactivity). |
Node Powerup | Started | 208 | 2 | Info | Node | Node Manager | Node power-up started. |
Node Powerup | Completed | 208 | 3 | Info | Node | Node Manager | Node power-up completed. |
Node Powerup | Failed | 208 | 4 | High | Node | Node Manager | Node power-up failed. |
Node High Storage Usage |
| 204 | 500 | Info | Node | Node Manager | Node entered degraded performance state due to high storage utilization (> 70%). |
Node High Storage Usage |
| 204 | 501 | Medium | Node | Node Manager | Node entered degraded performance state due to high metadata utilization. |
Node Read-Only State | Node entered read-only state | 209 | 504 | High | Node | Node Manager | Node entered read-only state. |
Node Read-Only State | Node exited read-only state | 209 | 506 | Info | Node | Node Manager | Node exited the read-only state. |
|
| 204 | 505 | High | Node | Node Manager | Node metadata is at high utilization (90%), and nearing read-only state. |
| Initiated | 205 | 2 | Medium | Node | Node Manager | Local Raid rebuild has started. |
| Completed | 205 | 3 | Info | Node | Node Manager | Local Raid rebuild has completed. |
|
| 205 | 504 | High | Node | Node Manager | Local Raid rebuild was halted (as the node entered read-only state, no further free storage to complete the rebuild). |
|
| 205 | 506 | Medium | Node | Node Manager | Local Raid rebuild resumed (continued with local Raid rebuild after it was halted). |
Node Unattached | N/A | 210 | 1 | Info | Node | Cluster Manager | Node entered Unattached state. All volume and snapshot resources were migrated off of this node. |
Server Clock Drift | Clock drift detected | 1000 | 1000 | High | Server | Node Manager | Detected a clock drift between the reporting server and the rest of the servers in the cluster. |
Server Linux VM Write Cache Configuration Error | Server Linux VM Write Cache Configuration Error | 1200 | 1200 | High | Server | Node Manager | For Internal use |
Server Upgrade | Started | 301 | 2 | Info | Server | Upgrade Manager | Upgrade of the server started. |
Server Upgrade | Finished | 301 | 3 | Info | Server | Upgrade Manager | Upgrade of the server completed. |
Server Upgrade | Failed | 301 | 4 | High | Server | Upgrade Manager | Upgrade of server has failed. Note: Additional failure-specific information will be returned here. |
Server Upgrade Skipped | Server non-upgradeable | 302 | 1100 | Medium | Server | Upgrade Manager | Skip upgrade operation for server, as the server is non-upgradeable. Note: Additional specific information to skip the cause will be returned here. |
Server Upgrade Incomplete | FailedToComplete | 304 | 4 | Medium | Server | Upgrade Manager | One or more product-managed (replace-class) config files could not be refreshed from the new images during a container upgrade. The upgrade still completed; the affected files are listed in the event description. |
| Failed | 700 | 4 | High | Server | Upgrade Manager | Failed to complete (upgrade manager failed to start up). |
| Started | 401 | 2 | Info | Server | Upgrade Manager | Started an upgrade of the cluster. |
| Finished | 401 | 3 | Info | Server | Upgrade Manager | Completed an upgrade of the cluster. |
| Failed | 401 | 4 | High | Server | Upgrade Manager | Upgrade of the cluster has failed. Note: Additional failure-specific information will be returned here. |
|
| 1001 | 1003 | Medium | NVMe SSD | Node Manager | An NVMe SSD device is unhealthy. |
|
| 1001 | 1002 | Medium | NVMe SSD | Node Manager | An NVMe SSD device is unhealthy. |
|
| 1001 | 1004 | Medium | NVMe SSD | Node Manager | An NVMe SSD device is unhealthy. |
|
| 1001 | 1005 | Medium | NVMe SSD | Node Manager | An NVMe SSD device is unhealthy. |
|
| 1001 | 1007 | Medium | NVMe SSD | Node Manager | An NVMe SSD device is unhealthy. |
|
| 1001 | 1006 | Medium | NVMe SSD | Node Manager | An NVMe SSD device is unhealthy. |
| N/A | 501 | 1 | High | NVMe SSD | Node Manager | An NVMe SSD device has failed. |
|
| 501 | 114 | Critical | NVMe SSD | Node Manager | An NVMe SSD used for journaling has failed. |
|
| 211 | 114 | High | Node | Node Manager | Node journaling device failed, but no failed NVMe device was detected. |
| N/A | 502 | 1 | Info | NVMe SSD | Node Manager | Added a new NVMe SSD to a node. |
|
| 503 | 901 | High | NVMe SSD | Node Manager | Failed to add a new NVMe SSD to a node. |
| N/A | 600 | 1 | Info | Volume | Cluster Manager | Volumes are in fully protected protection state (the event is issued on entering a new state; the event will return a list of volumes affected by this change of state). |
| N/A | 601 | 1 | Medium | Volume | Cluster Manager | Volumes are in degraded protection state (the event is issued on entering a new state; the event will return a list of volumes affected by this change of state). |
| N/A | 602 | 1 | High | Volume | Cluster Manager | Volumes are in read-only protection state (the event is issued on entering a new state; the event will return a list of volumes affected by this change of state). |
| N/A | 603 | 1 | High | Volume | Cluster Manager | Volumes are in unavailable protection state (the event is issued on entering a new state; the event will return a list of volumes affected by this change of state). |
|
| 402 | 1001 | High | Cluster | Cluster Manager | Cluster utilization is high. |
|
| 403 | 802 | Critical | Cluster | Cluster Manager | A cross-member etcd keyspace-hash mismatch was detected; a detailed summary is returned in the event description. |
|
| 405 | 803 | Info | Cluster | Cluster Manager | A previously detected cross-member etcd data corruption is no longer present (a clean scan cleared the alert). |
|
| 1100 | 1113 | Critical | Volume/NVMeSSD | Node Manager | Unrecoverable data integrity error. |
|
| 1100 | 1112 | Critical | Volume/NVMeSSD | Node Manager | Unrecoverable data integrity error. |
|
| 1100 | 1111 | Critical | Volume/NVMeSSD | Node Manager | Unrecoverable data integrity error. |
|
| 1100 | 1110 | Critical | Volume/NVMeSSD | Node Manager | Unrecoverable data integrity error. |
|
| 1101 | 1113 | High | Volume/NVMeSSD | Node Manager | Recoverable data integrity error. |
|
| 1101 | 1112 | High | Volume/NVMeSSD | Node Manager | Recoverable data integrity error. |
|
| 1101 | 1111 | High | Volume/NVMeSSD | Node Manager | Recoverable data integrity error. |
|
| 1101 | 1110 | High | Volume/NVMeSSD | Node Manager | Recoverable data integrity error. |
|
| 1100 | 1116 | Critical | Node/NVMeSSD | Node Manager | Detected data integrity error during node rebuild. |
|
| 1101 | 1115 | High | Node/NVMeSSD | Node Manager | Detected data integrity error during node rebuild, during recovery from a graceful shutdown. |
|
| 1102 | 1118 | High | Node/NVMeSSD | Node Manager | Detected data integrity error during garbage collection processing. |
| FailedToGetKEK | 1400 | 1500 | Medium | Server Encryption | Node Manager | Failed to get the encryption key from the cluster manager. |
| FailedToInitializeTPM | 1400 | 1501 | Medium | Server Encryption | Node Manager | Failed to initialize the TPM key. |
| FailedToReadKEK | 1400 | 1502 | Medium | Server Encryption | Node Manager | Failed to set the encryption key in the cache. |
| FailedToWriteKEK | 1400 | 1503 | Medium | Server Encryption | Node Manager | Failed to save the encryption key. |
| MissingDEK | 1301 | 1300 | Critical | Cluster Encryption | Node Manager | Failed to encrypt or update a volume on a node, due to a missing DEK. |
| CorruptedDEK | 1301 | 1301 | Critical | Cluster Encryption | Node Manager | Failed to encrypt or update a volume on a node, due to a corrupted DEK. |
| NotEnoughActiveNodes | 1302 | 1302 | Medium | Cluster Encryption | Cluster Manager | There are not enough servers with active nodes up to enable encryption. |
| FailedToDistributeKekToNodes | 1302 | 1303 | Medium | Cluster Encryption | Cluster Manager | Cluster Manager failed to distribute the Cluster Encryption Key to all servers with active nodes. |
| FailedToEnableEncryption | 1302 | 1304 | Medium | Cluster Encryption | Cluster Manager | Failed to enable encryption on the cluster level. |
| Completed | 321 | 3 | Info | Server | Cluster Manager | Server disable operation completed successfully (move server into maintenance mode). |
| Failed to Complete | 321 | 4 | Medium | Server | Cluster Manager | Server disable operation failed to complete (a more specific error cause is returned in the event description).. |
| Completed | 322 | 3 | Info | Server | Cluster Manager | Server enable operation completed successfully (move server out of maintenance mode). |
| Completed | 311 | 3 | Info | Server | Cluster Manager/Node Manager | New server created in the cluster (server added to the cluster). |
| Failed to Complete | 312 | 4 | High | Server | Cluster Manager | Failed to add a new server to the cluster. |
| Completed | 313 | 3 | Info | Server | Cluster Manager | Deleted a server from the cluster. |
| Initiated | 351 | 2 | High | Server | Cluster Manager | Started an eviction of a server's resources. |
| Completed | 351 | 3 | Info | Server | Cluster Manager | Completed an eviction of a server's resources. |
| Failed to Complete | 351 | 4 | High | Server | Cluster Manager | Failed to complete an eviction of a server's resources. |
| FailedEvictNotEnoughSpace | 351 | 1700 | High | Server | Cluster Manager | Failed to evict server: cluster does not have enough capacity to migrate all of the server's volumes. |
| FailedEvictOngoingRebuilds | 351 | 1701 | High | Server | Cluster Manager | Failed to evict server: one or more volumes could not be evicted due to ongoing rebuilds/migrations. |
| FailedEvictMigrationFailed | 351 | 1702 | High | Server | Cluster Manager | Failed to evict server: could not determine destinations for all of the server's resources. |
| FailedEvictNoProgress | 351 | 1703 | High | Server | Cluster Manager | Failed to evict server: no eviction progress detected within the configured timeout. |
| Initiated | 352 | 2 | Info | Server | Cluster Manager | Initiated an abortion of an ongoing server eviction. |
| Completed | 352 | 3 | Info | Server | Cluster Manager | Completed issuing an abort of an ongoing eviction. |
| Initiated | 1304 | 2 | Info | Cluster Encryption | Cluster Manager | Started the process to rotate the cluster root key. |
| Completed | 1304 | 3 | Info | Cluster Encryption | Cluster Manager | Completed the process of rotating the cluster root key. |
| Complete | 1304 | 4 | Medium | Cluster Encryption | Cluster Manager | Failed to complete cluster root key rotation. Either due to a fatal error, failing the rotation complexly, or only causing it to take longer or be stuck without failing it. |
© 2026 Lightbits Labs™