Future proof against silent loss of tests on CentOS Stream 9

Change-Id: I01b10d25ee0482c4f5b9bfb304e15e55cb158cc6

These were inadvertently skipped following the move to CentOS Stream,
due to matching the distro name on CentOS Linux.

Backport to Victoria.

Change-Id: I478d6304c3f8d207bc7062a0909147a169003027

Libvirt debug logs can grow very large, particularly on Ubuntu (we have
seen up to 1.1GB). Disable libvirt debug logging using variable
nova_libvirt_logging_debug which was added to Kolla Ansible during the
Wallaby cycle (see change I90bfd1b300ad3202ea4d139fda6d6beb44c5820f) and
backported to Victoria and Ussuri.

Change-Id: Ic047e80aa382148b009b1c03581b417d17ec2598

By default, Ansible injects a variable for every fact, prefixed with
ansible_. This can result in a large number of variables for each host,
which at scale can incur a performance penalty. Ansible provides a
configuration option [0] that can be set to False to prevent this
injection of facts. In this case, facts should be referenced via
ansible_facts.<fact>.

This change updates all references to Ansible facts within Kayobe
from using individual fact variables to using the items in the
ansible_facts dictionary. This allows users to disable fact variable
injection in their Ansible configuration, which may provide some
performance improvement.

This change disables fact variable injection in the ansible
configuration used in CI, to catch any attempts to use the injected
variables.

[0] https://docs.ansible.com/ansible/latest/reference_appendices/config.html#inject-facts-as-vars

Story: 2007993
Task: 42464
Depends-On: https://review.opendev.org/c/openstack/kolla-ansible/+/791276

Change-Id: I14db53ed6e57d37bbd28dd5819e432e3fe6628b2

Change-Id: I68332411e53c1d50ed56ddba3f46ee37db22f976

The iscsi deploy interface was removed from Ironic [1]. Switch to the
direct deploy interface instead, which is the default.

[1] https://review.opendev.org/c/openstack/ironic/+/789382

Depends-On: https://review.opendev.org/c/openstack/kolla-ansible/+/801267
Change-Id: Ia0dcd3d302e0f04adbda4f3abf75a0ca7dd20dee

libvirt-6.0.0-36.el8 was built [0] and released for CentOS Stream 8. It
includes a backport of a commit [1] from libvirt 7.4.0 to avoid erroring
out on unknown firmware features, which fixes the issue seen with the
newer edk2-ovmf package.

This reverts commit 1e14fa3a.

[0] https://koji.mbox.centos.org/koji/buildinfo?buildID=17918
[1] https://github.com/libvirt/libvirt/commit/61d95a1073833ec4323c1ef28e71e913c55aa7b9

Change-Id: Ibe177fe078769204c7e89c04e40870890fe501ba

Currently we do not have any coverage of building container images. This
change adds a basic test of building the base image to the seed jobs.

A potential future enhancement could involve pushing to a local registry
running on the seed.

Change-Id: Id7378dd844aca7cb5634535308eba0f60342c81d

Some CentOS Stream CI jobs started to fail when defining VMs with:

    libvirt.libvirtError: internal error: unknown feature amd-sev-es

This was introduced by edk2-ovmf-20200602gitca407c7246bf-5.el8.noarch.
This change excludes this package version in Zuul jobs, which fixes the
kayobe-seed-vm-centos8s job.

Overcloud centos8s jobs were also broken by the nova-libvirt Kolla image
containing this package, which is used by Tenks. Updated images using
the previous version of edk2-ovmf [1][2] are now available from quay.io.

This also needs to be fixed for Kayobe and Tenks in general, unless a
new package is released soon.

[1] https://review.opendev.org/c/openstack/kolla/+/792999
[2] https://review.opendev.org/c/openstack/kolla/+/793174

Change-Id: I5bc8197143ce763cdaf8a9ac0b782b524d5df258

CentOS repositories were renamed following the separation of CentOS
Linux and Stream. This change updates their names when using a local
mirror. It also removes the old files.

Change-Id: Ifcc3781d771e150b5771f2bf20cd8bcbc5faa443
Story: 2008795
Task: 42435

Overcloud upgrade jobs often fail when running on nodes with a 38G root
disk. The overcloud upgrade itself generally works, but post-upgrade
testing fails, for example when RabbitMQ gets in alarm state due to lack
of free disk space, which prevents Nova from functioning correctly and
makes it return "Unknown Error (HTTP 504)".

Change-Id: I536a526340f6d7dea7a69372904b777292774eba

Some hosts in the kayobe inventory might not be in the kolla-ansible
inventory so it makes sense for kayobe to manage NTP.

Change-Id: Iacb579a46b0e9769a4c404a858d17968f74dd7e0
Depends-On: https://review.opendev.org/c/openstack/kayobe-config-dev/+/786040
Story: 2007872
Task: 40240

We can now rely on the os_distribution variable to set sensible defaults
for Ubuntu. It is set automatically in kayobe-config-dev.

The interfaces pause workaround is no longer required since switching to
systemd-networkd.

Depends-On: https://review.opendev.org/c/openstack/kayobe-config-dev/+/788475
Change-Id: I76d658b93b9195c760c3a91fe5aee96bad8d6b42

The centos binary images now include nova 23.0.0 packages.

Change-Id: I0c7d57231a63e1f9ff3fdcdef1be7fff57160e08

Based on the discussions from recent Kolla meetings [1][2][3],
we established that, to avoid the DockerHub pull limit issues
in our CI, we will try using quay.io.

The images have already been published via Kolla CI
for master/Wallaby [4] and Victoria [5].

This change applies only to CI.
The user-facing move is yet to be discussed.

[1] http://eavesdrop.openstack.org/meetings/kolla/2021/kolla.2021-03-17-15.00.log.html#l-102
[2] http://eavesdrop.openstack.org/meetings/kolla/2021/kolla.2021-03-24-15.00.log.html#l-137
[3] http://eavesdrop.openstack.org/meetings/kolla/2021/kolla.2021-03-31-15.01.log.html#l-50
[4] https://review.opendev.org/c/openstack/kolla/+/781130
[5] https://review.opendev.org/c/openstack/kolla/+/781899

See related Kolla Ansible change
I22e11a994801447526ac26fc2815a1679876225e

Change-Id: Id6690dc9a486af0ed7d42555d84cf77d6de0b6af

This change adds support for policy-based routing via systemd-networkd.
Due to differences in the configuration mechanism, routing policy
rules are configured via dicts for Ubuntu, while remaining as strings
on CentOS. Ideally we would support both formats.

Story: 2004960
Task: 42217

Change-Id: I77aec0160eb7e4dd763326bfe6e3d9a44b248108

This change adds support for network configuration via systemd-networkd
on Ubuntu systems.

This is implemented via an Ansible Galaxy role,
stackhpc.systemd_networkd which was forked from
aruhier.systemd_networkd. Several improvements were made in
https://github.com/stackhpc/ansible-role-systemd-networkd/pull/1,
including:

* Add support for removing unexpected config files
* Use become where necessary
* Refactor config generation into a single task to improve performance

The systemd_networkd role does not add much abstraction on top of the
systemd-networkd configuration file format, which provides a lot of
flexibility at the expense of additional code in Kayobe. This code is
implemented as filter plugins, similarly to the existing
MichaelRigart.interfaces role.

This patch includes support for:

* Ethernet interfaces
* bridges
* bonds
* VLANs
* virtual Ethernet pairs (to connect Linux bridges and OVS bridges)
* static IP addresses
* static routes
* MTU

Some network attributes are currently not supported for
systemd-networkd:

* rules
* route options
* ethtool_opts
* zone
* allowed addresses

Story: 2004960
Task: 41881

Change-Id: I248b5bb9ce5a80a07a2a311cb3aca6daca920720

Change-Id: Idf5419d9020d491c310d2c09c495b0a420578214

Change-Id: Iff105f9a491dc6277586c1d53b84747c38feb585

The previous release has not been Rocky for a while.

Change-Id: Ic082ed08614c0f90d82ff211ac3dd19ed78d2bf5

The nodepool images include CirrOS images.

Also update to newly released 0.5.2 image.

Change-Id: I95fc2c3873c8cd17edf928f613a06d2154196786

This commit adds Zuul job for deploying seed services against
Ubuntu 20.04 LTS (Focal Fossa).

Co-Authored-By: Mark Goddard <mark@stackhpc.com>

Story: 2004960
Task: 41807
Depends-On: https://review.opendev.org/c/openstack/kayobe/+/772929
Change-Id: I839cd1b9d63c9bbf3e0c75def76b8bff68637c75

* Need to specify bash for &> syntax

Story: 2004960
Task: 41806

Change-Id: Ic33d3e31096f7b0cec2e4e9284f5c1078a4ff04c

Issues:

* bond configuration not working on Ubuntu - disabled in CI test

Story: 2004960
Task: 41560

Change-Id: Ib124dc6f313a2a13fa509b10702c6141f102f2f9

* Use source images
* Need to specify bash for &> syntax

Issues worked around:

* Manually configuring bridge via ip commands makes ifup fail to bring
  up the link. Adds a kayobe-network-bootstrap Zuul CI role that adds
  persistent configuration for the all-in-one network.

* bridge not active after interfaces role bounce. Added a pause, similar
  to https://github.com/michaelrigart/ansible-role-interfaces/pull/31

* fails installing docker python module for kolla user. WARNING: The
  repository located at mirror-int.ord.rax.opendev.org is not a trusted
  or secure host and is being ignored ERROR: No matching distribution
  found for docker===4.4.0 Adding trusted host for PyPI mirror.

* Tenks fails to create block devices - missing qemu-img (in qemu-utils)

* Tenks qemu emulator is different on Ubuntu

Remaining issues:

* Bare metal testing is unreliable on Ubuntu - some jobs see IPMI
  failures such as the following:

    ipmitool chassis bootdev pxe

    Error setting Chassis Boot Parameter 5\nError setting Chassis Boot
    Parameter 0\n

  Bare metal testing is disabled on Ubuntu for now.

Depends-On: https://review.opendev.org/766984
Depends-On: https://review.opendev.org/766958

Story: 2004960
Task: 29393

Change-Id: I1985efae7c18f55c3ff7c27c17d6242523904f3e

This partially reverts commit 47bbb96b
which triggered a retry on vexxhost clouds.

The issue was introduced in Ie8fd965165e8d347d27528a2c16d0647e412ccdc,
which applied some fixes for CentOS 8.3, and inadvertently removed
the Tenks variable that forces the use of qemu for 'bare metal' VMs.
This lead to autodetection of KVM, which does not work well when nested
in all CI cloud providers.

This change fixes the issue by forcing the use of qemu for the overcloud
once more. It also adds a similar option for the seed VM job.

Change-Id: I6bc8da2b8da903e09b97df8cd95c68a562c11db9

This is now merged into kayobe-config-dev [1].

[1] https://review.opendev.org/c/openstack/kayobe-config-dev/+/777076

Change-Id: I87b63966b55f04f80fd0255fbf2732d527732187

Jobs that test baremetal deployment are currently failing when run on
Vexxhost infrastructure. The seed VM job also has trouble on Vexxhost,
although does sometimes pass. It's not clear why yet, and we are still
investigating. Until we have a solution, fail in the pre phase when we
land on vexxhost clouds, in order to trigger the zuul retry mechanism.

Also use the normal number (5) of attempts for upgrade jobs, since these
no longer deploy the previous release in the pre phase.

Finally, incorporate a fix from
https://review.opendev.org/c/openstack/kayobe-config-dev/+/777076 to
allow this to merge. This part can be reverted once the
kayobe-config-dev patch merges.

[1] http://lists.openstack.org/pipermail/openstack-discuss/2021-February/020685.html

Change-Id: Ie5ca04428855cbaf505adc57de76ea0b2d404080

Change-Id: Id5ed0b594d6270c4a03606caa776bbb291be2694

This workaround is no longer needed for current CentOS 8 images as
resolv.conf file comes now empty.

Moreover, the workaround has been preventing seed host configure from
running successfully on Ubuntu hosts.

Change-Id: I30e34e3c58db3f3a93a07f3d6d53671286e4260e

This requires stackhpc.os-images v1.10.0 or newer, for compatibility
with CentOS 8 when SELinux is enabled: we disable SELinux, but without
rebooting it stays enabled.

This Ansible role was updated to v1.10.2 in master and stable/victoria
by I5efdbd52556721914fe69d7c6ba454b2c721b643, for another reason.
Remember to bump the requirement when backporting to earlier releases.

It also needs changes in the way we interact with Bifrost to avoid using
the env-vars file which has been removed. This is implemented by change
I25078e69acdf41a4ef9957f99fe5047de54b778d.

Finally, it requires building seed deployment images only after
deploying Bifrost, because the task copying images onto the seed expects
/etc/kolla/bifrost to exist.

We also copy log files to identify issues when the job fails.

Change-Id: I4719b4d397c01b35c78cb84c6d686dd27742d1c0

* Bump stackhpc.libvirt-host to v1.7.1. On seed-hypervisors installed
  using CentOS 8.2 or earlier, interaction with libvirt may fail due to
  libgcrypt being incompatible. See
  https://github.com/stackhpc/ansible-role-libvirt-host/issues/42

* Bump MichaelRigart.interfaces to v1.9.2. The CentOS 8.3 cloud image
  includes an ifcfg-ens3-1 file. See
  https://github.com/michaelrigart/ansible-role-interfaces/pull/93

* Previously a second libvirt daemon was installed by Tenks on the host,
  however changes in libvirt 6.0.0 to separate libvirtd into multiple
  daemons do not allow for customisation of the PID files used by the
  new daemons. This leads to a conflict between the container and host
  daemons. Update the Tenks config to use the containerised Nova libvirt
  daemon. This depends on a change to the stackhpc.libvirt-host role:
  https://github.com/stackhpc/ansible-role-libvirt-host/pull/44

* Not CentOS 8.3 related, but tox jobs are now failing on python
  dependencies. Remove upper limits from docker and paramiko.

* Not CentOS 8.3 related, but Bifrost has enabled authentication by
  default. We are not ready to support this, so override it.

Story: 2008429
Task: 41378

Change-Id: Ie8fd965165e8d347d27528a2c16d0647e412ccdc

For some reason this configuration was omitted or incomplete in these
jobs. Without this configuration, Kayobe development configuration adds
eth1 to the breth1 bridge if it exists, or leaves the bridge without a
port otherwise. This can lead to network connectivity issues, if the IP
address of eth1 becomes inaccessible when it is added to breth1.

This change fixes the issue by ensuring that the dummy1 interface
exists, and configuring the aio network to add it to breth1 as a port,
instead of eth1.

Change-Id: I47115e5d8d1fb448cb9e46dc43b56c64987e428b

Ironic is now disabled by default in the overcloud. This brings Kayobe's
default set of services into line with Kolla Ansible. For environments
using Ironic in the overcloud, set kolla_enable_ironic to true in
kolla.yml.

Story: 2008207
Task: 40991

Change-Id: I33eb4fa534847e199a599350b525d4762a2beaac

The flag dnf_install_epel was being used to predicate installation
of the EPEL RPM repo package, but not preventing the installation
of configuration for access to EPEL.  This small patch completes
the job.

Updates the overcloud host configure CI job to install EPEL mirrors,
since it was previously relying on this buggy behaviour.

Change-Id: Ib417837d0772338b16ea9f7f2540549f277d5de8

Change-Id: I5fc49fb734d0fe94f5f75c66eb4c1a935774ef30

This reverts commit bb9a595e.

Change-Id: Icd7b0f3b02bf3f22468daa8893c043ac9f3be4d6

Ironic has an issue where nodes will be switched to use the direct
deploy interface during upgrade, if it is enabled. This should be fixed
via https://review.opendev.org/755295, after which this patch should be
reverted.

Change-Id: I55af5d979bdbee9dd963845204c2f082fa7b27d7

Adds the kayobe-seed-vm-centos8 CI job to configure the Zuul VM as a
seed hypervisor, and use nested virt to provision a seed VM.  This
ensures that the seed hypervisor code paths are tested.

The job uses a Cirros image for the seed VM rather than the usual CentOS
cloud image. This is to reduce bandwidth required to download the image.
It does mean that the resulting seed VM cannot be used as a seed, but
nested virt would make this slow and unreliable anyway. Cirros does not
load cdrom drivers by default, so we add the configdrive as a disk
rather than a cdrom device.

Depends-On: https://review.opendev.org/617161

Change-Id: I2268a1ddf9a2870c713f32a40689e1686365aabd
Story: 2001655
Task: 6683

Switches to use the IPA builder project for building IPA images.

Switches the IPA images used by default to CentOS 8 based image.

Changes the file extension of the IPA kernel image from vmlinuz to
kernel.

Story: 2007070
Task: 37953

Change-Id: I82fc455f41f48dacb453e135870dd776895d7c99
Story: 2006574
Task: 39485