Adding a New Server

The following are instructions for adding a new server in order to expand cluster size/capacity.

1. First, ensure that you have all of Ansible’s original settings (including the generated files). The master jwt will also need to be common between all nodes. Note that you should "cd" into the directory where you extracted light-app.
2. Edit the Ansible inventories files (hosts, /host_var/server##.yaml). Create a new server##.yaml for a new target. Append the new server to the three sections in the hosts file: in the top section, in [duros_nodes], and in [etcd]. The hosts file must contain all of the other nodes of the cluster under [etcd] that already exist, in order to recognize which cluster it should be added to.
3. Run the ansible-playbook with the same parameters as before, and add the following parameter: --limit:<new_server name>.

Example if Deployed via Ansible

ansible-playbook -i ansible/inventory/cluster_example/hosts \

            playbooks/deploy-lightos.yml -vvv \

            -e new_etcd_member=true \

            -e extend_etcd_cluster=true \

            --limit server03

Run the above command from the directory where light-app was extracted, such that relative paths provided for the hosts and deploy-lightos.yml files are accessible.

This command relies on the jwt files and the lightos-certificate directory that were created during deployment, to be located in ~ ($HOME).

Example if Deployed via Docker and Ansible

Note that the following command must be run within Docker.

cd light-app    

docker run -it --rm --net=host \

    -e UID=`id -u` \

    -e GID=`id -g` \

    -e UNAME=$USER \

    -v `pwd`:/ansible \

    -w /ansible \

    -e ANSIBLE_LOG_PATH=/ansible/ansible.log \

    docker.lightbitslabs.com/lightos-3-(Minor Ver)-(Rev)-rhl-(8/9)/lb-ansible:v9.13.0 \

    sh -c 'ansible-playbook \

            -i ansible/inventories/cluster_example/hosts \

            playbooks/deploy-lightos.yml -vvv \

            -e system_jwt_path=/ansible/lightos_jwt \

            -e certificates_directory=/ansible/lightos-certificates \

            -e new_etcd_member=true \

            -e extend_etcd_cluster=true \

            --limit <new_server>'

4. Once the installation is finished, the new server will be rebooted. The new etcd will sync with the existing cluster, and the cluster should be extended automatically.

Cleaning Up a Failed Server Add

If the server addition failed or you have a corrupted server, and you want to remove it and re-add it to the cluster, follow these steps:

1. First, run cleanup playbook on this server and add the following parameter: --limit <new_server>, to clean all Lightbits metadata and software from this server.

Example

# If using Ansible Method, run this:

ansible-playbook -i ansible/inventories/cluster_example/hosts playbooks/cleanup-lightos-playbook.yml -t cleanup --limit <new_server>

​

# If using Docker Method, run this:

cd light-app    

docker run -it --rm --net=host \

    -e UID=`id -u` \

    -e GID=`id -g` \

    -e UNAME=$USER \

    -v `pwd`:/ansible \

    -w /ansible \

    -e ANSIBLE_LOG_PATH=/ansible/ansible.log \

    docker.lightbitslabs.com/lightos-3-(Minor Ver)-(Rev)-rhl-(8/9)/lb-ansible:v9.13.0 \

    sh -c 'ansible-playbook -i ansible/inventories/cluster_example/hosts \

           playbooks/cleanup-lightos-playbook.yml -t cleanup --limit <new_server>'

2. Enter etcdctl member list to get the ID of the server that failed to add, and then enter etcdctl member remove <new_server_id> to remove it from the etcd database.
3. After addressing any remaining issues, repeat the steps above to add a new server.