DMS Installation

Several services and configurations compose and enable the DMS. DMS support therefore includes an Ansible playbook and roles that help automate the deployment process.

The playbook invokes the following roles:

1. ansible-docker-role: Install Docker, Docker Compose, configure Docker log-plugin, optionally configure docker-image registries, and login to these registries.
2. ansible-dms-role: Deploy all DMS services by uploading all relevant configurations and docker-compose files, and run these services.

Prerequisites

System Requirements

• Ubuntu Server 24.04 / AlmaLinux + RockyLinux 9.x
• Minimum 10GB RAM
• 60GB available disk space

Required Permissions

• sudo access on the target server
• Permission to install system packages
• Docker daemon access

Required Credentials

• Lightbits container registry credentials
• CloudSmith API key

Ansible Controller Provisioning

In order to run Ansible, Lightbits will provide an Ansible container image that has been validated to correctly run the DMS playbook. This container has a specific Ansible version (v9.13.0), as well as roles and collections needed to deploy the service.

For optimal compatibility, Lightbits recommends using the provided image to deploy and run deploy-dms-playbook.

Requirements For Ansible-Controller:

• Docker Engine:

  • Ubuntu installation guide
  • RHEL installation guide
  • Generic Linux installation guide
  • MacOS

• Docker Compose Plugin:

  • All platforms installation guide

Define the APP_VERSION variable

Here we will define some variables that will be referenced across this guide. The first is the APP_VERSION variable, which defines the version of the Lightbits DMS service,

This will be used later to fetch to correct containers and binaries from the CloudSmith repository.

export APP_VERSION=v1.3.0

Login to CloudSmith

Lightbits uses CloudSmith as a hosting service for artifacts. This is where you can retrieve Lightbits software from similar containers, tarballs, and other packages. For ease of use. you should define the following environment variables:

export CLOUDSMITH_USERNAME=dms 

export CLOUDSMITH_API_KEY=<YOUR_API_KEY>

You can now log in to the Lightbits image registry:

echo $CLOUDSMITH_API_KEY | docker login docker.lightbitslabs.com -u $CLOUDSMITH_USERNAME --password-stdin

Download the dms-ansible tarball from the Lightbits repository, and extract it:

curl -1sLf "https://dl.lightbitslabs.com/$CLOUDSMITH_API_KEY/dms/raw/versions/$APP_VERSION/dms-ansible-$APP_VERSION.tar.gz?accept_eula=1" \ -o dms-ansible-$APP_VERSION.tar.gz tar xzvf dms-ansible-$APP_VERSION.tar.gz cd dms-ansible-$APP_VERSION

DMS Services Deployment

Before running the playbook, Lightbits needs to create an inventory that contains instructions on how to connect to a DMS host, as well as group_vars - which contain deployment-specific information.

This folder will hold hosts and group_varsinformation.

mkdir -p inventory

Ansible connects to a remote host using SSH. Lightbits supports either of the two modes. Choose one of them for the installation:

• SSH access to the DMS server using a username/password. This is the default mode of operation.
• SSH access to the DMS server using SSH key. This mode requires uncommenting the following line in docker-compose.yml.

# - ${SSH_KEY_PATH}:${SSH_KEY_PATH_IN_CONT} # (6) contains ssh key file path, uncomment when SSH-key file is the preferred mode of operation.

SSH Username/Password Option

The following steps detail the requirements for using SSH username/password. Skip to the SSH key section below if that is the desired mode of operation.

First, create the inventory/hosts file using an SSH username/password.

See this link for an example inventory file.

The file should then be placed at inventory/hosts.

cat << EOF > inventory/hosts

dms  ansible_host=$DMS_HOSTNAME \

     ansible_connection=ssh \

     ansible_ssh_user=$ANSIBLE_SSH_USER \

     ansible_ssh_pass=$ANSIBLE_SSH_PASS \

     ansible_become_user=root \

     ansible_become_pass=$ANSIBLE_ROOT_PASS

​

[dms_servers]

dms

EOF

Next, create an .env file to set an lb-ansible user using an SSH username/password.

These environment variables specify that the lb-ansible container should run with the correct user ID and username.

cat << EOF > .env

UID=`id -u`

GID=`id -g`

UNAME=`id -un`

EOF

SSH Key Option

The following steps detail the requirements for using SSH Key. Skip this section if SSH username/password is the desired mode of operation.

Define the SSH key file to use.

If you do not have a ssh-key, you will need to generate a new one:

ssh-keygen -t rsa -b 4096

Specify the SSH-key to use (adjust it to match your key path):

export SSH_KEY_PATH=~/.ssh/id_rsa

Uncomment the following line in docker-compose.yml:

# - ${SSH_KEY_PATH}:${SSH_KEY_PATH_IN_CONT} # (6) contains ssh key file path, uncomment when SSH-key file is the preferred mode of operation.

Create the inventory/hosts file using SSH_Key.

See this link for an example inventory file.

The file should be placed at inventory/hosts.

cat << EOF > inventory/hosts

dms  ansible_host=$DMS_HOSTNAME \

     ansible_connection=ssh \

     ansible_ssh_user=$ANSIBLE_SSH_USER \

     ansible_become_user=root \

     ansible_ssh_private_key_file=$SSH_KEY_PATH

​

[dms_servers]

dms

EOF

Next, create an .env file to set an lb-ansible user using SSH_Key.

These environment variables specify that the lb-ansible container should run with the correct user ID and username.

cat << EOF > .env

UID=`id -u`

GID=`id -g`

UNAME=`id -un`

SSH_KEY_PATH_IN_CONT=/opt/id_rsa

EOF

Next, create the group_vars file.

The group_vars/all.yml file enables customizing the deployment by injecting variables. These parameters define the image repository that Lightbits should pull the images from.

The following should be specified:

mkdir -p inventory/group_vars

​

cat << EOF > inventory/group_vars/all.yml

run_observability_services: true

image_registry: docker.lightbitslabs.com/dms

image_registries:

- name: docker.lightbitslabs.com

  url: https://docker.lightbitslabs.com

  security:

    insecure: false

  login:

    username: $CLOUDSMITH_USERNAME

    password: $CLOUDSMITH_API_KEY

EOF

Adding Linux Users to docker Group

Because Docker interaction requires being part of docker group, you may want to add more than one user to the dockergroup when deploying DMS.

For example, with ansible-user and dms-user, the first is used for deploying DMS using Ansible, and the second (dms-user) is used for day-to-day operations running dmscli commands.

By default, Ansible playbook will add only ansible-user to the docker group, and if logging in as dms-user, the sudo docker command will need to be run, or you can manually add the user to the docker group using the following command:

sudo usermod -aG docker $USER

In order to make this more streamlined, group_vars/all.yml can be specified in the following section:

docker_users:

- dms-user

This will make sure the dms-user user is added to the docker group. Note that the installation will add ansible-user as well (implicitly), so that it can also run Docker commands.

Skip Installing Observability Services on the DMS Server

By default, the playbook will install Prometheus and Grafana and configure it to run on the DMS server.

It is better practice to not run those, in case you have these services running externally and configure Prometheus to scrape the DMS service from outside.

To skip installation, you should set the following value to false:

run_observability_services: false

Usage of Persistent Storage

In order for the DMS service to be durable and retain information following restarts, power cycles, migration and upgrades, the persistent_storage_path variable must be set to specify a durable and persistent volume where DMS will store its data.

Prerequisites for Persistent Storage Before Configuring Persistent Storage

1. Ensure that sufficient storage space is available on the persistent volume.
2. Verify that the volume meets performance requirements for DMS operations.
3. Ensure that proper filesystem permissions are set.
4. Verify that the volume is properly mounted and accessible.
5. Consider a backup strategy for the persistent data.

Overriding Workflow Specific Variables

There are many variables that can be overridden during the installation/upgrade phase.

Since these steps use Ansible, all of the common Ansible ways to override variables such as host_vars/group_vars/cmd-line/env-vars apply.

You can look at the list of variables and their defaults in the roles themselves. Each role has a README.md with the relevant variables.

Variable definition and default values can be found here:

• roles/ansible-dms-role/README.md
• roles/ansible-observability-role/README.md

Using the group_vars/all.yml File

In order to override some of the variables at deployment/upgrade, specify the variables you want to override in your group_vars/all.yml file.

In order to override a nested variable - for example workflows.thickClone.maxConcurrentWorkflowTasks - replicate the structure in your group_vars/all.yml file.

Ansible will automatically merge this with the default dictionary.

In your group_vars/all.yml:

workflows:

  thickClone:

    # These values will override the defaults

    maxConcurrentActivities: 3

    maxConcurrentWorkflowTasks: 4

    maxGoroutines: 5

    # The 'activities' dictionary and its keys will be inherited from defaults/main.yml

    # because they are not redefined here.

The Command-Line Option: --extra-vars

The best way to pass nested data is to use a JSON string, as it is unambiguous for the shell:

ansible-playbook your-playbook.yml --extra-vars '{

  "workflows": {

    "thickClone": {

      "maxConcurrentActivities": 3,

      "maxConcurrentWorkflowTasks": 4,

      "maxGoroutines": 5

    }

  }

}'

This method is highly effective for CI/CD pipelines or one-off overrides.

Now that the Ansible controller is ready, all that is needed to deploy the DMS service is to execute the playbook and wait for DMS to be deployed:

docker compose run --rm -v `pwd`/inventory:/inventory deploy-dms

After the installation, follow these steps to use the DMS CLI service (dmscli):

1. If DMS was just deployed, re-login to the DMS server or run exec $SHELL to reread .bashrc - which has the dmscli alias defined.
2. Do the following to generate bash_completion for dmscli:

dmscli completion bash | sudo tee /etc/bash_completion.d/dmscli.sh