Backup and Restore

The Lightbits storage cluster uses replication for data availability and durability, and backup can be used for longer-term durability.

The Lightbits cluster in AWS backs up the data on a cluster to AWS S3 (Amazon’s Simple Storage Service). This provides yet another layer of protection for the data, so if the whole cluster fails and all data is lost - or if there is a need to shut down the cluster - the backed-up data still remains in S3.

This is also a method to move data between clusters in the same region or in different regions. For example, users have a regulatory obligation to save a backup of the data in a different region. The application is in AWS us-east1. The backup bucket can be set up in us-west1 and the backup process will back up the data between the regions. Users can then restore the data to a new cluster in us-west1 or to any other AWS region.

Prerequisites

• S3 Bucket: By default the CF deployment will create a new bucket for the backup/restore process. If you want to use your own bucket or if you want to restore from a previously created bucket, this can be done post-deployment.
• BucketPolicy: If you are using an existing bucket, you will need to make sure that the Lambda and Instance IAM roles created by the CF for execution of the backup and restore have read/write access to the bucket.

The below should be added to the Bucket Policy:

{

  "Version": "2012-10-17",

  "Statement": [

{

    "Effect": "Allow",

    "Principal": {

      "AWS": "<ARN of the instance Role>"

    },

    "Action": [

      "s3:GetObject",

      "s3:PutObject"

    ],

    "Resource": "arn:aws:s3:::bucket-name/*"

  },

{

    "Effect": "Allow",

    "Principal": {

      "AWS": "<ARN of the Lambda Role>"

    },

    "Action": [

      "s3:GetObject",

      "s3:PutObject"

    ],

    "Resource": "arn:aws:s3:::bucket-name/*"

  }

  ]

}

In order to configure an existing bucket, you will need to find the IAM roles created by the CF stack for the Exporter Lambda and Instances.

All the resources created by the CF template can be found in the CF stack:

1. In the AWS console, go to the CloudFormation dashboard.
2. From the left menu, select Stacks.
3. In the filter, enter the given name of the stack. In the example below, the name is demo.

4. For the instance IAM Role ARN, click on the lbinfra nested stack.
5. From the top menu, click Resources.

6. Copy the InstanceRole physical ID.
7. For the instance IAM Role ARN, click the lbapp nested stack.
8. From the top menu, click Resources.

9. Copy the LambdaRole physical ID.
10. Go to the existing S3 bucket in the S3 Dashboard.

11. From the top menu, select the Permissions tab.
12. Click Edit under the bucket policy.
13. Copy the bucket ARN and paste it in the above policy example.

14. You should have no errors on the last line.

15. Click Save Changes.
16. You should get a green message on the screen that says Successfully updated bucket policy.
17. Once this is done, the existing IAM role of the Backup Lambda and Exporter instances will have the required permissions to the S3 bucket for the Backup and Restore to work properly.