If any nova compute service fails to register itself, Kolla Ansible will
fail the host that queries the Nova API. This is the first compute host
in the inventory, and fails in the task:

    Waiting for nova-compute services to register themselves

Other hosts continue, often leading to further errors later on. Clearly
this is not idea.

This change modifies the behaviour to query the compute service list
until all expected hosts are present, but does not fail the querying
host if they are not. A new task is added that executes for all hosts,
and fails only those hosts that have not registered successfully.

Alternatively, to fail all hosts in a cell when any compute service
fails to register, set nova_compute_registration_fatal to true.

Change-Id: I12c1928cf1f1fb9e28f1741e7fe4968004ea1816
Closes-Bug: #1940119

Designate sink is an optional service that consumes notifications,
users should have an option to disable it when they don't use them.

Change-Id: I1d5465d9845aea94cff39ff5158cd8b1dccc4834

Change Ia1239069ccee39416b20959cbabad962c56693cf added support for
running a libvirt daemon on the host, rather than using the nova_libvirt
container. It did not cover migration of existing hosts from using a
container to using a host daemon.

This change adds a kolla-ansible nova-libvirt-cleanup command which may
be used to clean up the nova_libvirt container, volumes and related
items on hosts, once it has been disabled.

The playbook assumes that compute hosts have been emptied of VMs before
it runs. A future extension could support migration of existing VMs, but
this is currently out of scope.

Change-Id: I46854ed7eaf1d5b5e3ccd8531c963427848bdc99

In some cases it may be desirable to run the libvirt daemon on the host.
For example, when mixing host and container OS distributions or
versions.

This change makes it possible to disable the nova_libvirt container, by
setting enable_nova_libvirt_container to false. The default values of
some Docker mounts and other paths have been updated to point to default
host directories rather than Docker volumes when using a host libvirt
daemon.

This change does not handle migration of existing systems from using
a nova_libvirt container to libvirt on the host.

Depends-On: https://review.opendev.org/c/openstack/ansible-collection-kolla/+/830504

Change-Id: Ia1239069ccee39416b20959cbabad962c56693cf

In Kolla Ansible OpenStack deployments, by default, libvirt is
configured to allow read-write access via an unauthenticated,
unencrypted TCP connection, using the internal API network.  This is to
facilitate migration between hosts.

By default, Kolla Ansible does not use encryption for services on the
internal network (and did not support it until Ussuri). However, most
other services on the internal network are at least authenticated
(usually via passwords), ensuring that they cannot be used by anyone
with access to the network, unless they have credentials.

The main issue here is the lack of authentication. Any client with
access to the internal network is able to connect to the libvirt TCP
port and make arbitrary changes to the hypervisor. This could include
starting a VM, modifying an existing VM, etc. Given the flexibility of
the domain options, it could be seen as equivalent to having root access
to the hypervisor.

Kolla Ansible supports libvirt TLS [1] since the Train release, using
client and server certificates for mutual authentication and encryption.
However, this feature is not enabled by default, and requires
certificates to be generated for each compute host.

This change adds support for libvirt SASL authentication, and enables it
by default. This provides base level of security. Deployments requiring
further security should use libvirt TLS.

[1] https://docs.openstack.org/kolla-ansible/latest/reference/compute/libvirt-guide.html#libvirt-tls

Depends-On: https://review.opendev.org/c/openstack/kolla/+/833021
Closes-Bug: #1964013
Change-Id: Ia91ceeb609e4cdb144433122b443028c0278b71e

Change-Id: I547ab4b05aa14ed3bbee8be2dc77a6840d4816f6

Role vars have a higher precedence than role defaults. This allows to
import default vars from another role via vars_files without overriding
project_name (see related bug for details).

Change-Id: I3d919736e53d6f3e1a70d1267cf42c8d2c0ad221
Related-Bug: #1951785

The documentation for novncproxy_base_url says:

    If using noVNC >= 1.0.0, you should use ``vnc_lite.html`` instead of
    ``vnc_auto.html``.

While novnc packages in CentOS, Debian, and Ubuntu still provide
vnc_auto.html for compatibility, this could be dropped in the future.

Change-Id: I04883c877015c1835c8b6b2c8be1fb7156ceb340

This is required for libvirtd with cgroupsv2 (Debian Bullseye and
soon others).
Otherwise, device attachments simply fail.
The warning message suggests filtering will be disabled but it
actually just fails the action entirely.

Change-Id: Id1fbd49a31a6e6e51b667f646278b93897c05b21
Closes-Bug: #1941940

It was removed in [1] as part of cgroupsv2 cleanup.
However, the testing did not catch the fact that the legacy
cgroups behaviour was actually still breaking despite latest
Docker and setting to use host's cgroups namespace.

[1] 286a03ba

Closes-Bug: #1941706
Change-Id: I629bb9e70a3fd6bd1e26b2ca22ffcff5e9e8c731

This change enables the use of Docker healthchecks for
nova-spicehtml5proxy service.

Implements: blueprint container-health-check
Change-Id: I584c588c20781e6c6567429811aecf97967baea3

We get a nice optimisation by using a filtered loop instead
of task skipping per service with 'when'.

Partially-Implements: blueprint performance-improvements
Change-Id: I8f68100870ab90cb2d6b68a66a4c97df9ea4ff52

This trivial patch is setting "timeout tunnel" in haproxy's
configuration for spicehtml5proxy. This option extends time
when spice's websocket connection is closed, so spice will
not be freezed. Default value is set to 1h as it is in novnc.

Closes-Bug: #1938549
Change-Id: I3a5cd98ecf4916ebd0748e7c08111ad0e4dca0b2

By default, Ansible injects a variable for every fact, prefixed with
ansible_. This can result in a large number of variables for each host,
which at scale can incur a performance penalty. Ansible provides a
configuration option [0] that can be set to False to prevent this
injection of facts. In this case, facts should be referenced via
ansible_facts.<fact>.

This change updates all references to Ansible facts within Kolla Ansible
from using individual fact variables to using the items in the
ansible_facts dictionary. This allows users to disable fact variable
injection in their Ansible configuration, which may provide some
performance improvement.

This change disables fact variable injection in the ansible
configuration used in CI, to catch any attempts to use the injected
variables.

[0] https://docs.ansible.com/ansible/latest/reference_appendices/config.html#inject-facts-as-vars

Change-Id: I7e9d5c9b8b9164d4aee3abb4e37c8f28d98ff5d1
Partially-Implements: blueprint performance-improvements

They are handled by Docker since at least 18.09 (tested).
Backport to Wallaby at most to not introduce needless restarts in
already stable branches.

Depends-On: https://review.opendev.org/c/openstack/kolla-ansible/+/792583
Change-Id: Ia95355c529f1b0222dc1de06632984b6d130b9ec

Makes nova-libvirt container always run in 'host' CgroupnsMode
to ensure it works.

Change-Id: I75105baf434977c68bc5c8ca1f5213e602c52c8c

We don't do the best job with it and it's better to rely on users'
and distros' default policies than try to water those down.

Closes-Bug: #1837551
Change-Id: I72b13adef60900fc31f1293c516030026f004216

In order to disable libvirt debug in CI (which takes vast amount of storage)
this change introduces nova_libvirt_logging_debug and disables that in CI.

Change-Id: I90bfd1b300ad3202ea4d139fda6d6beb44c5820f

Libvirt may reasonably expect that its secrets directory
(/etc/libvirt/secrets) is persistent. However, the nova_libvirt
container does not map the secrets directory to a volume, so it will not
survive a recreation of the container. Furthermore, if Cinder or Nova
Ceph RBD integration is enabled, nova_libvirt's config.json includes an
entry for /etc/libvirt/secrets which will wipe out the directory on a
restart of the container.

Previously, this appeared to cause an issue with encrypted volumes,
which could fail to attach in certain situations as described in bug
1821696. Nova has since made a related change, and the issue can no
longer be reproduced. However, making the secret store persistent seems
like a sensible thing to do, and may prevent hitting other corner cases.

This change maps /etc/libvirt/secrets to a Docker volume in the
nova_libvirt container.  We also modify config.json for the nova_libvirt
container to merge the /etc/libvirt/secrets directory, to ensure that
secrets added in the container during runtime are not overwritten when
the container restarts.

Change-Id: Ia7e923dddb77ff6db3c9160af931354a2b305e8d
Related-Bug: #1821696

Searchlight project is retiring in Wallaby cycle[1].
This commit removes the ansible roles of Searchlight project
before its code is removed.

Needed-By: https://review.opendev.org/c/openstack/searchlight/+/764526

[1] http://lists.openstack.org/pipermail/openstack-discuss/2020-November/018637.html

Change-Id: I85aab66376ea4f1376c2705066ba3c7e5645644f

This change enables the use of Docker healthchecks for core OpenStack
services.
Also check-failures.sh has been updated to treat containers with
unhealthy status as failed.

Implements: blueprint container-health-check
Change-Id: I79c6b11511ce8af70f77e2f6a490b59b477fefbb

via KOLLA_SKIP and KOLLA_UNSET

Change-Id: I7d9af21c2dd8c303066eb1ee4dff7a72bca24283
Related-Bug: #1837551

Change-Id: Ifcedcc72307732393a92a702a7567addc043b5b2

Previously we mounted /etc/timezone if the kolla_base_distro is debian
or ubuntu. This would fail prechecks if debian or ubuntu images were
deployed on CentOS. While this is not a supported combination, for
correctness we should fix the condition to reference the host OS rather
than the container OS, since that is where the /etc/timezone file is
located.

Change-Id: Ifc252ae793e6974356fcdca810b373f362d24ba5
Closes-Bug: #1882553

This makes use of udev rules to make it smarter and override
host-level packages settings.
Additionally, this masks Ubuntu-only service that is another
pain point in terms of /dev/kvm permissions.
Fingers crossed for no further surprises.

Change-Id: I61235b51e2e1325b8a9b4f85bf634f663c7ec3cc
Closes-bug: #1681461

when enable kolla_dev_mod, nova-cell role clones code failed,
because we use nova-cell repository which is not exists.
in fact, nova-cell role should use nova repository too

Change-Id: I7fa62726d0d5b0aeb3bd5fa06dc0e59667f94fa0

The RabbitMQ 'openstack' user has the 'administrator' tag assigned via
the RabbitMQ definitions.json file.

Since the Train release, the nova-cell role also configures the RabbitMQ
user, but omits the tag. This causes the tag to be removed from the
user, which prevents it from accessing the management UI and API.

This change adds support for configuring user tags to the
service-rabbitmq role, and sets the administrator tag by default.

Change-Id: I7a5d6fe324dd133e0929804d431583e5b5c1853d
Closes-Bug: #1875786

Some services look for /etc/timezone on Debian/Ubuntu, so we should
introduce it to the containers.

In addition, added prechecks for /etc/localtime and /etc/timezone.

Closes-Bug: #1821592
Change-Id: I9fef14643d1bcc7eee9547eb87fa1fb436d8a6b3

Kolla-Ansible Ceph deployment mechanism has been deprecated in Train [1].

This change removes the Ansible code and associated CI jobs.

[1]: https://review.opendev.org/669214

Change-Id: Ie2167f02ad2f525d3b0f553e2c047516acf55bc2

In dev mode currently the python source is mounted under python2.7
site-packages. This change fixes this to use the distro_python_version
variable to ensure dev mode works with Python 3 images.

Change-Id: Ieae3778a02f1b79023b4f1c20eff27b37f481077
Partially-Implements: blueprint python-3

For the CentOS 7 to 8 transition, we will have a period where both
CentOS 7 and 8 images are available. We differentiate these images via a
tag - the CentOS 8 images will have a tag of train-centos8 (or
master-centos8 temporarily).

To achieve this, and maintain backwards compatibility for the
openstack_release variable, we introduce a new 'openstack_tag' variable.
This variable is based on openstack_release, but has a suffix of
'openstack_tag_suffix', which is empty except on CentOS 8 where it has a
value of '-centos8'.

Change-Id: I12ce4661afb3c255136cdc1aabe7cbd25560d625
Partially-Implements: blueprint centos-rhel-8

This patch adds initial support for deploying multiple Nova cells.

Splitting a nova-cell role out from the Nova role allows a more granular
approach to deploying and configuring Nova services.

A new enable_cells flag has been added that enables the support of
multiple cells via the introduction of a super conductor in addition to
cell-specific conductors. When this flag is not set (the default), nova
is configured in the same manner as before - with a single conductor.

The nova role now deploys the global services:

* nova-api
* nova-scheduler
* nova-super-conductor (if enable_cells is true)

The nova-cell role handles services specific to a cell:

* nova-compute
* nova-compute-ironic
* nova-conductor
* nova-libvirt
* nova-novncproxy
* nova-serialproxy
* nova-spicehtml5proxy
* nova-ssh

This patch does not support using a single cell controller for managing
more than one cell. Support for sharing a cell controller will be added
in a future patch.

This patch should be backwards compatible and is tested by existing CI
jobs. A new CI job has been added that tests a multi-cell environment.

ceph-mon has been removed from the play hosts list as it is not
necessary - delegate_to does not require the host to be in the play.

Documentation will be added in a separate patch.

Partially Implements: blueprint support-nova-cells
Co-Authored-By: Mark Goddard <mark@stackhpc.com>
Change-Id: I810aad7d49db3f5a7fd9a2f0f746fd912fe03917

The idea is to factor out a role for deploying Nova related services
to cells. Since all deployments use cells, this role can be used
in both regular deployments which have just cell0 and cell1,
and deployments with many cells.

Partially Implements: blueprint support-nova-cells
Change-Id: Ib1f36ec0a773c384f2c1eac1843782a3e766045a

The rolling upgrade has been the default since Stein. The legacy
upgrade has been removed because it doesn't follow the upgrade
guide [1].

[1] https://docs.openstack.org/nova/latest/user/upgrade.html

Change-Id: I2aa879699cb4e9955bf5c38053eada5a53fb6211

To securely support live migration between computenodes we should enable
tls, with cert auth, instead of TCP with no auth support.

Implements: blueprint libvirt-tls

Change-Id: I22ea6233933c840b853fdcc8e03400b2bf577271

Use upstream Ansible modules for registration of services, endpoints,
users, projects, roles, and role grants.

Change-Id: I7c9138d422cc91c177fd8992347176bb54156b5a

Nova-consoleauth support was removed in
I099080979f5497537e390f531005a517ab12aa7a, but these variables were
left.

Change-Id: I1ce1631119bba991225835e8e409f11d53276550

During an upgrade, nova pins the version of RPC calls to the minimum
seen across all services. This ensures that old services do not receive
data they cannot handle. After the upgrade is complete, all nova
services are supposed to be reloaded via SIGHUP to cause them to check
again the RPC versions of services and use the new latest version which
should now be supported by all running services.

Due to a bug [1] in oslo.service, sending services SIGHUP is currently
broken. We replaced the HUP with a restart for the nova_compute
container for bug 1821362, but not other nova services. It seems we need
to restart all nova services to allow the RPC version pin to be removed.

Testing in a Queens to Rocky upgrade, we find the following in the logs:

Automatically selected compute RPC version 5.0 from minimum service
version 30

However, the service version in Rocky is 35.

There is a second issue in that it takes some time for the upgraded
services to update the nova services database table with their new
version. We need to wait until all nova-compute services have done this
before the restart is performed, otherwise the RPC version cap will
remain in place. There is currently no interface in nova available for
checking these versions [2], so as a workaround we use a configurable
delay with a default duration of 30 seconds. Testing showed it takes
about 10 seconds for the version to be updated, so this gives us some
headroom.

This change restarts all nova services after an upgrade, after a 30
second delay.

[1] https://bugs.launchpad.net/oslo.service/+bug/1715374
[2] https://bugs.launchpad.net/nova/+bug/1833542

Change-Id: Ia6fc9011ee6f5461f40a1307b72709d769814a79
Closes-Bug: #1833069
Related-Bug: #1833542

The nova-consoleauth service was deprecated during the Rocky release [1]
and has not been necessary since unless you're using cells v1. As Kolla
has never supported cells v1, which is finally being removed during
Train [2], we can get ahead of the curve and stop deploying
nova-consoleauth immediately.

[1] https://specs.openstack.org/openstack/nova-specs/specs/rocky/implemented/convert-consoles-to-objects.html
[2] https://blueprints.launchpad.net/nova/+spec/remove-cells-v1/

Change-Id: I099080979f5497537e390f531005a517ab12aa7a

The installation guide [1] uses endpoints ending at the /v2.1 suffix.

[1] https://docs.openstack.org/nova/stein/install/controller-install.html

Change-Id: I92af045da67f9e746fd6e4b94e56bb8aa72850c4