Auto Scaling Overview
In the Lightbits SDS, when a node reaches 90% capacity, it becomes read-only and data consistency is at risk. The auto scaling process increases the cluster size (adds an additional server) when the storage cluster is over the scale-out capacity threshold (the default is 80%) This is in order to increase the capacity capabilities of the cluster.
Configuration
- ScaleOutRatio: When the cluster capacity passes this ScaleOutRatio threshold, the auto scaling starts and adds an additional instance to the cluster. As this can take several minutes, it is recommended that the configuration should not be too close to the node’s read-only threshold (which is by default 90% - 0.9). On the other hand, the Lightbits cluster is intended to work under high capacity, and decreasing the threshold too much means that the cluster is under-utilized and might create unwanted AWS costs. The recommendation is to leave it at the 80% default (0.8), or something close to that.
- MaxInstancesCount: In order to plan your AWS costs properly, you can configure the maximum number of instances in a cluster. So for example the minimum cluster size of Lightbits is three instances and the maximum cluster size is currently 16. The default configuration is set to 8. You can configure the maximum instances to be 6, so that the cluster automatically increases the cluster size to six but won’t go beyond that. This configuration can be changed at any time. Configuration can’t be lower than 3 (this is the minimum Lightbits cluster size), or higher than 16 (currently the maximum Lightbits cluster size). Lowering this number could cause Lightbits to go into read-only mode if the 90% threshold is exceeded (the hard limit on the cluster level), without the possibility to scale-out.
- In the AWS console, go to Lambda Dashboard, and from the left menu, select Function.
- In the filter, enter Maintain. This will show you the maintenance Lambda that is responsible for the auto scaling execution. Click the Lambda Function and then click on the Environment Variable on the side and switch to the Configuration tab. You will see ScaleOutRatio and MaxInstancesCount. It is recommended not to change any other parameters as it may impact the stability of the cluster.
The table below describes the environment variables that can be configured by the user.
| Parameter Name | Type | Required | Default | Comments |
|---|---|---|---|---|
| ScaleOutRatio | Float | No | 0.8 | Decimal percent defines the threshold that - if passed - the cluster will scale out (the value should be between 0.5 to 0.9). 0.8 reflects 80% full capacity. |
| MaxInstancesCount | Int | No | 8 | The maximum number of instances in the cluster. Beyond this number, auto scaling will not add additional servers (the value should be between 3 and 16). |
The following screenshot illustrates the Lambda environment variables.
Process Overview
- Scale-out threshold is exceeded.
- The cluster identifies the need to scale out.
- The cluster is scaled out to provide an additional instance (by increasing the number of instances in the ASG).
- The new instance is configured to extend the cluster (with the same Lightbits version and configuration as the other cluster instances).
- Lightbits Proactive Rebalance will start balancing out the volumes on the new number of instances, and moving some of them to the new instance.
Limitations
- Scale-out will only scale out one node at a time (n+1), during each scale-out operation. So if for example there are three instances and the threshold is exceeded, it will add one additional server until the next time the threshold is exceeded.
- Rapid scaling (n+x) is currently not supported.
- Scale-in is currently not supported (it will be supported in one of the upcoming versions).
- The cluster will only scale to the Max Instances count (environment variable), or maximum supported instances - which is 16.
Was this page helpful?