Upgrading the Lightbits Storage Cluster in AWS
Every few months Lightbits publishes a new Lightbits version, and in AWS a new AMI is published. This enables our customers to trigger an upgrade process and make sure that their cluster is up to date with all the latest new features, with no downtime.
The Lightbits upgrade in AWS ensures that only instances with the latest AMI version are in the cluster. The upgrade process is triggered by the customer by updating CF and the AMI provided by it.
Please note that the upgrade process is only triggered once you update CF, and is not triggered automatically by an update of the AMI.
Although the process is automatic and with no downtime to the cluster, it is still recommended to trigger the upgrade during your defined maintenance window.
Lightbits SDS currently does not support skipping versions and therefore it is recommended to install the upgrades as they come out. For example, if there are versions 1.5, 1.6, and 1.7, you will not be able to upgrade from v1.5 to v1.7 before first upgrading to v1.6.
Upgrade Process
Lightbits provides you with the S3 URL where the updated CF files are located. Note that new Lightbits AMIs are generated by Lightbits and placed in AWS.
These new AMIs are referred from new CF files. Once obtaining S3 links to new CF templates, you can perform an upgrade either via the AWS console or via AWS CLI.
You can always find the latest version of Lightbits AMI in this public S3 folder.
The upgrade is triggered by stack update (CF), with a new CF template. This will update the new AMI ID of the new Lightbits storage version, in the auto scaling launch template in the Lightbits Auto Scaling Group (LOSASG). This will also update any other stack resources required for the upgrade.
Once the AMI is updated in the template, the maintenance Lambda with ASG capabilities will replace all the existing storage instances with new ones based on the upgraded AMI. This also includes migration of all the data from the old instances to the new ones. The process is done as a rolling upgrade, so that at any given moment all the data is available and no downtime is required.
Note that if during the upgrade there are other maintenance activities to be performed (auto scaling/healing), these will take precedence and will be performed before the upgrade.
Updating the stack will update the Lambdas as well as all other resources that have changed or have been added according to the changes that are specified for the stack update. A stack update is conducted by updating CF with a new root stack template (the Marketplace root template) provided by Lightbits. The default configurations in the update stack are as they were during the initial deployment. If you have changed any of the documented environment variables these will automatically be backed up before the upgrade process and post-upgrade will be restored; this way any of the documented environment variables will be maintained as they were before the upgrade. You should not change any parameters in the upgrade process. If for example you deployed MVP 1.0 with the backup service disabled - and then post-deployment you configure to enable the backup - during the upgrade to the next version MVP 1.x, you will see in the configuration that the backup is disabled. This is ok and should not be touched. The Lightbits cluster will maintain the configuration automatically.
How to Trigger an Upgrade in CloudFormation
The following details the upgrade procedure using the AWS console.
- In the AWS console, go to CloudFormation.
- Filter by lightbitslabs. This will give you the storage cluster CF.
- Select the root stack (if you have a few clusters, make sure to select the one you want to upgrade). The root stack has the name that you provided when first initializing the stack, typically named ..-root-marketplace.yml).
- Toggle view nested in the header bar to show only the root stack.
- Click the Update button.
- Step 1: Prerequisite - Prepare template, and select Replace current template. For Specify template, select Amazon S3 URL and fill it in with (Stack Template URL provided by Lightbits).
- In most cases this can be found here.
- Click Next.
- Under Specify stack details, ignore the parameter fields. We will make sure to keep your configuration up to date after the upgrade.
- Skip the Configure stack options page without modification by clicking Next.
- Select I acknowledge that this template may create IAM resources.
- Click Update Stack.
- The stack will start updating the resources and its nested contained stacks according to the current CF templates in the release. See here to find additional information regarding the monitoring of the progress.
Updated Lambdas might fail or error out due to a lack of resources or permissions within their first runs post-update. This could happen if resources they depend on have not updated yet (e.g., IAM roles and permissions). As soon as the update is complete, this should be resolved.
Logs and Troubleshooting
After stack creation is complete, check the Lambda's CloudWatch log group (logs depend on the run/schedule of the Lambda, as well as the fact that there is a delay in between the run and the output in CloudWatch). It is recommended to view logs by Log Group > search all log streams in order to view the entire log output.
- In the AWS console, go to CloudWatch > Log groups.
- Filter by 'maintain'.
- You can see all of the logs of the maintained Lambda.
- LogDebug level: In order to retrieve a more detailed log, the Debug environment variable of the Lambda can be set to true (see the Auto Scaling in this guide, for more on how to update Lambda environment variables).