adding-and-removing-hosts.rst

=========================
Adding and removing hosts
=========================

This page discusses how to add and remove nodes from an existing cluster. The
procedure differs depending on the type of nodes being added or removed, which
services are running, and how they are configured. Here we will consider two
types of nodes - controllers and compute nodes. Other types of nodes will need
consideration.

Any procedure being used should be tested before being applied in a production
environment.

Adding new hosts
================

.. _adding-new-controllers:

Adding new controllers
----------------------

The :doc:`bootstrap-servers command
</reference/deployment-and-bootstrapping/bootstrap-servers>` can be used to
prepare the new hosts that are being added to the system.  It adds an entry to
``/etc/hosts`` for the new hosts, and some services, such as RabbitMQ, require
entries to exist for all controllers on every controller. If using a
``--limit`` argument, ensure that all controllers are included, e.g. via
``--limit control``. Be aware of the :ref:`potential issues <rebootstrapping>`
with running ``bootstrap-servers`` on an existing system.

.. code-block:: console

   kolla-ansible -i <inventory> bootstrap-servers [ --limit <limit> ]

Pull down container images to the new hosts. The ``--limit`` argument may be
used and only needs to include the new hosts.

.. code-block:: console

   kolla-ansible -i <inventory> pull [ --limit <limit> ]

Deploy containers to the new hosts. If using a ``--limit`` argument, ensure
that all controllers are included, e.g. via ``--limit control``.

.. code-block:: console

   kolla-ansible -i <inventory> deploy [ --limit <limit> ]

The new controllers are now deployed. It is recommended to perform testing
of the control plane at this point to verify that the new controllers are
functioning correctly.

Some resources may not be automatically balanced onto the new controllers. It
may be helpful to manually rebalance these resources onto the new controllers.
Examples include networks hosted by Neutron DHCP agent, and routers hosted by
Neutron L3 agent. The `removing-existing-controllers`_ section provides an
example of how to do this.

.. _adding-new-compute-nodes:

Adding new compute nodes
------------------------

The :doc:`bootstrap-servers command
</reference/deployment-and-bootstrapping/bootstrap-servers>`, can be used to
prepare the new hosts that are being added to the system.  Be aware of the
:ref:`potential issues <rebootstrapping>` with running ``bootstrap-servers`` on
an existing system.

.. code-block:: console

   kolla-ansible -i <inventory> bootstrap-servers [ --limit <limit> ]

Pull down container images to the new hosts. The ``--limit`` argument may be
used and only needs to include the new hosts.

.. code-block:: console

   kolla-ansible -i <inventory> pull [ --limit <limit> ]

Deploy containers on the new hosts. The ``--limit`` argument may be used and
only needs to include the new hosts.

.. code-block:: console

   kolla-ansible -i <inventory> deploy [ --limit <limit> ]

The new compute nodes are now deployed. It is recommended to perform
testing of the compute nodes at this point to verify that they are functioning
correctly.

Server instances are not automatically balanced onto the new compute nodes. It
may be helpful to live migrate some server instances onto the new hosts.

.. code-block:: console

   openstack server migrate <server> --live-migration --host <target host> --os-compute-api-version 2.30

Alternatively, a service such as :watcher-doc:`Watcher </>` may be used to do
this automatically.

Removing existing hosts
=======================

.. _removing-existing-controllers:

Removing existing controllers
-----------------------------

When removing controllers or other hosts running clustered services, consider
whether enough hosts remain in the cluster to form a quorum. For example, in a
system with 3 controllers, only one should be removed at a time. Consider also
the effect this will have on redundancy.

Before removing existing controllers from a cluster, it is recommended to move
resources they are hosting. Here we will cover networks hosted by Neutron DHCP
agent and routers hosted by Neutron L3 agent. Other actions may be necessary,
depending on your environment and configuration.

For each host being removed, find Neutron routers on that host and move them.
Disable the L3 agent. For example:

.. code-block:: console

   l3_id=$(openstack network agent list --host <host> --agent-type l3 -f value -c ID)
   target_l3_id=$(openstack network agent list --host <target host> --agent-type l3 -f value -c ID)
   openstack router list --agent $l3_id -f value -c ID | while read router; do
     openstack network agent remove router $l3_id $router --l3
     openstack network agent add router $target_l3_id $router --l3
   done
   openstack network agent set $l3_id --disable

Repeat for DHCP agents:

.. code-block:: console

   dhcp_id=$(openstack network agent list --host <host> --agent-type dhcp -f value -c ID)
   target_dhcp_id=$(openstack network agent list --host <target host> --agent-type dhcp -f value -c ID)
   openstack network list --agent $dhcp_id -f value -c ID | while read network; do
     openstack network agent remove network $dhcp_id $network --dhcp
     openstack network agent add network $target_dhcp_id $network --dhcp
   done

Stop all services running on the hosts being removed:

.. code-block:: console

   kolla-ansible -i <inventory> stop --yes-i-really-really-mean-it [ --limit <limit> ]

Remove the hosts from the Ansible inventory.

Reconfigure the remaining controllers to update the membership of clusters such
as MariaDB and RabbitMQ. Use a suitable limit, such as ``--limit control``.

.. code-block:: console

   kolla-ansible -i <inventory> deploy [ --limit <limit> ]

Perform testing to verify that the remaining cluster hosts are operating
correctly.

For each host, clean up its services:

.. code-block:: console

   openstack network agent list --host <host> -f value -c ID | while read id; do
     openstack network agent delete $id
   done

   openstack compute service list --os-compute-api-version 2.53 --host <host> -f value -c ID | while read id; do
     openstack compute service delete --os-compute-api-version 2.53 $id
   done

.. _removing-existing-compute-nodes:

Removing existing compute nodes
-------------------------------

When removing compute nodes from a system, consider whether there is capacity
to host the running workload on the remaining compute nodes. Include overhead
for failures that may occur.

Before removing compute nodes from a system, it is recommended to migrate or
destroy any instances that they are hosting.

For each host, disable the compute service to ensure that no new instances are
scheduled to it.

.. code-block:: console

   openstack compute service set <host> nova-compute --disable

If possible, live migrate instances to another host.

.. code-block:: console

   openstack server list --host <host> -f value -c ID | while read server; do
     openstack server migrate --live-migration $server
   done

Verify that the migrations were successful.

Stop all services running on the hosts being removed:

.. code-block:: console

   kolla-ansible -i <inventory> stop --yes-i-really-really-mean-it [ --limit <limit> ]

Remove the hosts from the Ansible inventory.

Perform testing to verify that the remaining cluster hosts are operating
correctly.

For each host, clean up its services:

.. code-block:: console

   openstack network agent list --host <host> -f value -c ID | while read id; do
     openstack network agent delete $id
   done

   openstack compute service list --os-compute-api-version 2.53 --host <host> -f value -c ID | while read id; do
     openstack compute service delete --os-compute-api-version 2.53 $id
   done