Disabling and Evicting a Server

By default, when you disable a server, it will not evict the data and nodes will become inactive - impacting data availability. Once nodes are inactive, data will be migrated only after the DurationToTurnIntoPermanentFailure passes (by default 4 hours). When adding the evict-data flag, the server will move to disabling state and migration will start immediately. During the eviction the volumes will remain fully protected, and only once all volumes have been migrated successfully to other servers in the cluster will the nodes of the server become inactive and the server be disabled.

If the nodes on the evicted server are already unavailable and the server (or any other server in the cluster) are in RiskOfServiceLoss or SourceOfRiskOfServiceLoss, the evict server API will fail. If there is is still a need to evict the server taking into consideration the risk, you can run the disable server with the --force flag.

1. Once the server is triggered to be disabled, the server's state will change to 'Disabling'.

2. The eviction will occur while 'Disabling', This is done by utilizing Rebalance. If a previous migration is still in progress (e.g., a previously triggered rebalance), incoming migrations to the server being disabled will be stopped.

3. Events:

   1. Disable server:

      1. ServerDisableOperation - Initiated / completed / FailedToComplete

   2. Evict process:

      1. ServerEviction - Initiated / completed / FailedToComplete

4. Failed Evictions

   1. If the migration fails, this means that server eviction failed, and the server will change back to the status 'Enabled'. The last eviction will be in status 'Failed', with an error indicating that the migration failed. An event will be raised for both the failed eviction and the failed to disable server. You should run the get server API to get the Evict status, progress and failure message. Note that even if eviction failed, it could be that some of the data was already migrated to another server successfully. There is no rollback process and the dynamic rebalance algorithm will rebalance the data if required.

   2. When the rebalance algorithm could not find a way to evict all the volumes from the evicted server, the eviction process will fail. The cause of this problem can be a lack of free space on the servers.

   3. Reason for failures while rebalancing resources include:

      • The destination node becomes full during eviction (because of new data from the user).

      • A node on the source server becomes inactive during the eviction.

      • A node on the destination server becomes inactive during the eviction (or any node in the cluster).

      • If the dynamic rebalance process is running in parallel to the evict process, this might cause the evict to fail in certain cases.

   4. If evict fails due to one of the above, once the issue is remediated (i.e., all nodes are back to being active and the rebalance process is finished), you can try and rerun the evict process.

5. Once the eviction is completed (i.e., all of the server's nodes are unattached), the server’s state will change to ‘Disabled’.