Cluster I/O throttling puts a bandwidth ceiling on the replication traffic that Lightbits generates on its own, so that rebuilding or moving data does not starve an application’s I/O. Cluster I/O represents both types of traffic - client I/O and recovery I/O - while throttling is applied only to recovery (rebuild/rebalance).
Lightbits moves data between nodes without being asked to in two situations: when a copy of your data is missing and has to be re-created (rebuild), and when data is deliberately relocated between nodes (rebalance). Both are healthy behaviors, and both compete with client I/O for the same NVMe devices and the same network. On a busy cluster, the result is a latency spike on volumes that were never part of the rebuild or rebalance operations.
Throttling lets you cap that background traffic in MiB/s. You set one cluster-wide policy; every node applies it to the recovery traffic it receives. The default is unlimited, which is how every earlier Lightbits release behaves; nothing changes until you set a cap.
The trade-off is explicit and permanent: a lower cap protects client latency and lengthens the window in which your data is under-protected. Choosing a value below gives the arithmetic.
For CLI-related information on throttling, see lbcli get throttling, lbcli get throttling io, lbcli set throttling, and lbcli set throttling io.
How it Works
The policy is one object and can be applied to any node in the cluster, stored in etcd once for the whole cluster, and applied independently by every node.
It carries four bandwidth caps and one priority setting:
Field | Caps | Enforced Today |
|---|---|---|
| Rebuild the data that a node receives. | Yes |
| Rebalance the data that a node receives. | Yes |
| Rebuild the data that a node sends. | No - accepted and stored, not applied. |
| Rebalance the data that a node sends. | No - accepted and stored, not applied. |
| Relative weighting of background vs client I/O. | No - accepted and stored, not applied. |
All four limits are in MiB/s, and 0 means unlimited. Unlimited is the default and is what an unconfigured cluster runs with.
Each node enforces the policy against the traffic it is receiving. A cap of 100 MiB/s on a six-node cluster does not mean 100 MiB/s of recovery traffic cluster-wide; it means up to 100 MiB/s arriving at each node, or up to 600 MiB/s in aggregate if every node is a destination at once. Size the number against what a single node's hardware devices and NIC can spare, not against the cluster total.
Within one node the cap is a shared budget. If three volumes are rebuilding onto the same node at the same time, they together get rebuildRxLimit, not rebuildRxLimit each.
How a Change Reaches the Nodes
The API service validates the request and writes the policy to the cluster store. Each node's manager watches that key and pushes the new limits into its own data path. A change applies to traffic that is already in flight; a rebuild running at 500 MiB/s slows to the new cap within seconds, with no restart and no interruption. Each node also re-reads the policy when it starts, so a node that was down during the change comes back with the current limits.
Enforcement is a token bucket per class. A class that has been idle may burst briefly above its cap when it wakes - by design, roughly 10 ms of the configured rate, with a floor of 1 MiB. Over any interval longer than a second or so, the observed rate tracks the cap.
How Limits are Applied to Operations
To determine which limit applies to which operation, the rule is whether the data movement was planned. Data moved by an administrative decision is rebalance; everything else is rebuild.
Cluster Operation | Charged To |
|---|---|
Node fails or is powered off, and its data is re-created elsewhere. | Rebuild |
Node returns after an outage and catches up on writes it missed. | Rebuild |
A device fails and its data is re-created. | Rebuild |
A permanently failed node is replaced. | Rebuild |
You evict or drain a node for maintenance. | Rebalance |
The cluster proactively relocates data to even out capacity. | Rebalance |
This split is deliberate. A rebuild is the cluster racing to get your data back to full protection, and you usually will want it to happen fast. A rebalance is housekeeping that you scheduled, and you can afford to make it slow and invisible. Setting rebalanceRxLimit low and rebuildRxLimit high or unlimited is the common configuration.
Both limits are independent budgets. If a node fails during a planned drain, rebuild traffic and rebalance traffic arrive at the same node against two separate caps, and the node can receive up to the sum of the two.
Limitations
Read this section before you set a cap in production.
rebuildTxLimit, rebalanceTxLimit and priority are accepted by the API, stored, and returned when you read the policy back - but the data path does not act on them currently. Setting them does not change anything. Use rebuildRxLimit and rebalanceRxLimit.
Under-protected data stays under-protected for longer. If a second failure lands inside that longer window, you can lose data that you would have otherwise kept. Treat the rebuild cap as a risk decision, not a tuning knob.
A cluster-wide budget is not something you can express. See the How it Works section above for additional information.
This feature caps background traffic; it does not reserve bandwidth for your applications or give them scheduling priority. If your workload alone saturates a node, a background cap will not help it.
The cap limits what a node writes as it receives a copy. The source node's read and send effort is not limited, so a sending node that also serves clients sees no relief from these settings.
A class that has been idle can release up to one bucket at once - about 10 ms of the configured rate, minimum 1 MiB. This is measured over seconds, not milliseconds.
The final reconciliation phase of a rebuild runs outside the throttle, so a rebuild's total traffic is slightly above what the cap alone would predict. It is a short phase and small relative to the bulk copy.
There is no per-volume, per-node, per-pool, or time-of-day scope, and no scheduling. If you want a cap only during business hours, you set it and unset it yourself.