Evicting Data (Technology Preview)

User-triggered eviction of data will migrate all of a server's resources (e.g., volumes, snapshots) to other servers within a cluster. Volumes that are associated with this server are migrated prior to any maintenance operation on the server, so the volumes are fully protected when disabling the server.

Additionally, the timing of evicting data is user-triggered - so in a scenario of failed servers, there is no need to wait for a permanent threshold (or any other time to expire) before the server can be deleted. You can initiate the evict at your convenience.

Enabling the Evict Data Feature

To evict data from a server, the feature must first be enabled. Once enabled, the data can be evicted from the server by invoking disable server with the evict flag. See "Disabling and Evicting a Server" below for additional information.

To enable evicting data, use the following command:

lbcli enable feature-flag evict-data

See the lbcli enable feature-flag evict-data (Technology Preview) page in the Lightbits CLI Guide, for additional information.

Disabling and Evicting a Server

The evict-data flag is optional (the default operation is to not evict on disable). To evict a server when disabling it, you will have to specify this flag.

lbcli disable server —name={name} —-evict-data

Disabling a Server - Eviction Process

1. Once the server is triggered to be disabled, the server's state will change to 'Disabling'.

2. The eviction will occur while 'Disabling', This is done by utilizing Rebalance. If a previous migration is still in progress (e.g., a previously triggered rebalance), incoming migrations to the server being disabled will be stopped.

3. Failed Evictions

   1. If the migration fails, this means that server disablement failed, and the server will change back to the status 'Enabled'. The last eviction will be in status 'Failed', with an error indicating that the migration failed. An event will be raised for both the failed eviction and the failed disablement.
   2. When the rebalance algorithm could not find a way to evict all the volumes from the evicted server, the eviction process will fail. The cause of this problem can be a lack of free space on the servers.
   3. Reason for failures while rebalancing resources include:
      • The destination node becomes full during eviction (because of new data from the user).
      • The source server becomes inactive during the eviction.
      • The destination server becomes inactive during the eviction.

4. Once the eviction is completed (i.e., all of the server's nodes are unattached), the server’s state will change to ‘Disabled’.

Monitoring the Eviction

Once a server is prompted to be disabled, its status will change to "Disabling". The LastEviction of a server displays the status and progress of the last eviction that was conducted for the server. This also includes the current eviction.

lbcli list servers --output-format=yaml

List Servers Detailed View

The below is a sample output of LastEviction(output format: yaml):

Note that this is for either the last eviction issued, or the current ongoing one.

LastEviction:

  endTime: null

  errorMessage: ""

  progress: 0

  startTime: "2024-02-06T11:09:56.250098246Z"

  status: Evicting

Additional Considerations

1. To successfully evict a server, ensure that there are enough servers/resources available to evict the data to. A cluster should have at least four servers.
2. Evicting a server that has volumes that are placed with placement affinity is not supported.
3. Aborting/cancelling an eviction (re-enable) means that when the server is re-enabled while being evicted, the server will be available for volume placement again. It also means that the migrations set into motion by the eviction are being stopped.