We build IPA images and a deployment image in the seed jobs, so we don't
need to download Cirros or IPA images. Also, these downloads depend on
external resources which may make jobs less reliable.

For seed upgrade jobs, disable IPA and deployment image downloads.

Change-Id: Ib59c8bc2d8938eca18c943bb2e66ed185152a739

The kayobe-seed-ubuntu-focal job is currently fairly unreliable, often
failing to build the base container image.

We are not using the mirrors provided by OpenDev infra, which may be
making these builds less reliable.

This change disables container image builds in CI on Ubuntu. It should
be reverted if they are made more reliable.

Change-Id: I648fa6423ad9ff43120c7808f080b0359ad8621c

When TLS is enabled, extra RAM usage is causing the OOM killer to
terminate Tenks VMs, which are using large RSS amounts (around 1.5 GB).

Disable Heat and Horizon to free up enough memory to make the job pass.

Change-Id: If483a6a6fb6d5b2c9b6b7dbd22939b0b46599538

Change-Id: I615707976454a91c8f6aecc5eda1852def7197d4

nova-libvirt images now include qemu-utils on master [1] and xena [2].

[1] https://review.opendev.org/c/openstack/kolla/+/830401
[2] https://review.opendev.org/c/openstack/kolla/+/831411

Change-Id: I8f5f93340642d055cce7ef306d942e75b10c86a9

Previously we were using the zuul user in the TLS jobs. This was due to
a permissions issue when accessing the CA certificate in kayobe-config
in the zuul user's home directory.

This change reverts to the default of using the stack user for the TLS
jobs. In order to make this work, the generated CA cert chain is added
to the trust store.

Change-Id: I875f8976df75dee68ba00842fe624c29cc1b123c

Also synchronise Ansible settings between Kayobe and Kolla Ansible.

Change-Id: Idaea4a984391a8cd05a5b0eee254ac6bad531a3e

This change uses the new Galaxy requirements file in Kolla Ansible to
install the openstack.kolla collection.

Cross-project dependencies on ansible-collection-kolla are supported.

Story: 2009854
Task: 44504

Depends-On: https://review.opendev.org/c/openstack/kolla-ansible/+/819430
Change-Id: Iac185dd2bbbca128c6cf71b2734e94b3e1c6133b

This patch adds the openstack.kolla collection to the Galaxy
requirements. It is installed from the OpenDev git repository. The
collection is not yet used by Kayobe.

Zuul cross-project dependencies on the ansible-collection-kolla
repository are supported (and used in this commit).

Story: 2009854
Task: 44503

Depends-On: https://review.opendev.org/c/openstack/ansible-collection-kolla/+/820165/
Change-Id: I91cbac839f816a00ac54bc4a350f44b5ae457cc3

It is now set globally in kayobe-config-dev.

Depends-On: https://review.opendev.org/c/openstack/kayobe-config-dev/+/829225
Change-Id: Ib46834fd226c3f59146bbaada53c8d079481d315

Previously we were seeing issues with ipmitool and virtualbmc on Ubuntu:

  Error setting Chassis Boot Parameter 5\nError setting Chassis Boot Parameter 0

The dependent change fixes these issues, and this change enables bare
metal testing in Ubuntu CI.

Depends-On: https://review.opendev.org/c/openstack/kayobe-config-dev/+/829006

Change-Id: I96827fc32c1594ca9a0535e259929c49d3f0e704

Depends-On: https://review.opendev.org/c/openstack/kolla-ansible/+/826723
Change-Id: I52ffa0b23ce2ae54fbfe5879a25b128ca1864195

Set the Ironic boot mode to legacy BIOS explicitly in Tenks config in
anticipation of an upcoming change to the default boot mode.

Override the boot mode to UEFI in the overcloud TLS job to improve
coverage. This requires enabling iPXE booting.

Depends-On: https://review.opendev.org/c/openstack/tenks/+/827479/

Change-Id: Id1b4e9775c834b8b97e086241ee8b247977225a2

EPEL is no longer required for a default installation. Let's disable it.

Also clean up the install_epel variable from Kolla Ansible globals.yml
template, since it never existed.

Story: 2009757
Task: 44227

Change-Id: I96eb4685f997e85ad2ee5318640d58d0287a016d

Change-Id: Ic8c212111301d6b144e3aa0fe124b022b0296ec6

As a first step towards supporting multiple overcloud disk images, this
change introduces a new command to build a disk image directly with DIB:
`kayobe overcloud host image build`.

It also disables building a root disk image during Bifrost bootstrap if
overcloud_dib_build_host_images is set to true.

Change-Id: I93d242889e225b4e60254f6b9cc5eeb457294ac8
Story: 2002098
Task: 41693

Also remove some Wallaby upgrade compatibility code in CI playbooks

Change-Id: I84e336bec37eac12c659c7a9d9fc4ddd191a379d

Depends-On: https://review.opendev.org/c/openstack/kayobe/+/812670
Depends-On: https://review.opendev.org/c/openstack/kayobe-config-dev/+/805239
Depends-On: https://review.opendev.org/c/openstack/kayobe/+/813212

Change-Id: I7dbd643fec72223bfda20cbbc37809ae78073804

We see frequent failures of upgrade jobs on stable branches due to lack
of disk space. Disable heat in an attempt to free up extra space, since
we do not need it for our testing.

Change-Id: I20c99ca9dd6e78b041e4662c9b1b54eac904e3c9

Follow up adding tests for Id60e25e129e323f3c07e702bb81a11efc530fb3e.

Change-Id: Ieb6d6e4d491b4ceb44d5fe364f0da215ac303fe1

Currently, the overcloud host configure test results are only returned
on success, when they are least useful. This is because the zuul_return
task is not reached when the test task fails. This change reorders the
tasks to fix the issue.

Change-Id: Ie572eda567782cc377767bd8d925145e2ad4bd61

This is only supported on CentOS for now due to limitations of the
Ansible role used to configure tuned.

Change-Id: Ie07c5f467975f8da2f720e70c94cea6285981d72
Co-Authored-By: Pierre Riteau <pierre@stackhpc.com>
Story: 2007853
Task: 40155

Future proof against silent loss of tests on CentOS Stream 9

Change-Id: I01b10d25ee0482c4f5b9bfb304e15e55cb158cc6

These were inadvertently skipped following the move to CentOS Stream,
due to matching the distro name on CentOS Linux.

Backport to Victoria.

Change-Id: I478d6304c3f8d207bc7062a0909147a169003027

Now that the previous release is Wallaby, we can add overcloud and seed
upgrades on Ubuntu.

Change-Id: I69868dca6e1f3e4a1bf123c9e7f1f3412162b264

Source images get the most test coverage, so it makes sense to build and
deploy these by default.

Change-Id: I297b83985b09e888c5ee64c1a39f8a1dfcacc5c1

Libvirt debug logs can grow very large, particularly on Ubuntu (we have
seen up to 1.1GB). Disable libvirt debug logging using variable
nova_libvirt_logging_debug which was added to Kolla Ansible during the
Wallaby cycle (see change I90bfd1b300ad3202ea4d139fda6d6beb44c5820f) and
backported to Victoria and Ussuri.

Change-Id: Ic047e80aa382148b009b1c03581b417d17ec2598

By default, Ansible injects a variable for every fact, prefixed with
ansible_. This can result in a large number of variables for each host,
which at scale can incur a performance penalty. Ansible provides a
configuration option [0] that can be set to False to prevent this
injection of facts. In this case, facts should be referenced via
ansible_facts.<fact>.

This change updates all references to Ansible facts within Kayobe
from using individual fact variables to using the items in the
ansible_facts dictionary. This allows users to disable fact variable
injection in their Ansible configuration, which may provide some
performance improvement.

This change disables fact variable injection in the ansible
configuration used in CI, to catch any attempts to use the injected
variables.

[0] https://docs.ansible.com/ansible/latest/reference_appendices/config.html#inject-facts-as-vars

Story: 2007993
Task: 42464
Depends-On: https://review.opendev.org/c/openstack/kolla-ansible/+/791276

Change-Id: I14db53ed6e57d37bbd28dd5819e432e3fe6628b2

Change-Id: I68332411e53c1d50ed56ddba3f46ee37db22f976

The iscsi deploy interface was removed from Ironic [1]. Switch to the
direct deploy interface instead, which is the default.

[1] https://review.opendev.org/c/openstack/ironic/+/789382

Depends-On: https://review.opendev.org/c/openstack/kolla-ansible/+/801267
Change-Id: Ia0dcd3d302e0f04adbda4f3abf75a0ca7dd20dee

libvirt-6.0.0-36.el8 was built [0] and released for CentOS Stream 8. It
includes a backport of a commit [1] from libvirt 7.4.0 to avoid erroring
out on unknown firmware features, which fixes the issue seen with the
newer edk2-ovmf package.

This reverts commit 1e14fa3a.

[0] https://koji.mbox.centos.org/koji/buildinfo?buildID=17918
[1] https://github.com/libvirt/libvirt/commit/61d95a1073833ec4323c1ef28e71e913c55aa7b9

Change-Id: Ibe177fe078769204c7e89c04e40870890fe501ba

Currently we do not have any coverage of building container images. This
change adds a basic test of building the base image to the seed jobs.

A potential future enhancement could involve pushing to a local registry
running on the seed.

Change-Id: Id7378dd844aca7cb5634535308eba0f60342c81d

Some CentOS Stream CI jobs started to fail when defining VMs with:

    libvirt.libvirtError: internal error: unknown feature amd-sev-es

This was introduced by edk2-ovmf-20200602gitca407c7246bf-5.el8.noarch.
This change excludes this package version in Zuul jobs, which fixes the
kayobe-seed-vm-centos8s job.

Overcloud centos8s jobs were also broken by the nova-libvirt Kolla image
containing this package, which is used by Tenks. Updated images using
the previous version of edk2-ovmf [1][2] are now available from quay.io.

This also needs to be fixed for Kayobe and Tenks in general, unless a
new package is released soon.

[1] https://review.opendev.org/c/openstack/kolla/+/792999
[2] https://review.opendev.org/c/openstack/kolla/+/793174

Change-Id: I5bc8197143ce763cdaf8a9ac0b782b524d5df258

CentOS repositories were renamed following the separation of CentOS
Linux and Stream. This change updates their names when using a local
mirror. It also removes the old files.

Change-Id: Ifcc3781d771e150b5771f2bf20cd8bcbc5faa443
Story: 2008795
Task: 42435

Overcloud upgrade jobs often fail when running on nodes with a 38G root
disk. The overcloud upgrade itself generally works, but post-upgrade
testing fails, for example when RabbitMQ gets in alarm state due to lack
of free disk space, which prevents Nova from functioning correctly and
makes it return "Unknown Error (HTTP 504)".

Change-Id: I536a526340f6d7dea7a69372904b777292774eba

Some hosts in the kayobe inventory might not be in the kolla-ansible
inventory so it makes sense for kayobe to manage NTP.

Change-Id: Iacb579a46b0e9769a4c404a858d17968f74dd7e0
Depends-On: https://review.opendev.org/c/openstack/kayobe-config-dev/+/786040
Story: 2007872
Task: 40240

We can now rely on the os_distribution variable to set sensible defaults
for Ubuntu. It is set automatically in kayobe-config-dev.

The interfaces pause workaround is no longer required since switching to
systemd-networkd.

Depends-On: https://review.opendev.org/c/openstack/kayobe-config-dev/+/788475
Change-Id: I76d658b93b9195c760c3a91fe5aee96bad8d6b42

The centos binary images now include nova 23.0.0 packages.

Change-Id: I0c7d57231a63e1f9ff3fdcdef1be7fff57160e08

Based on the discussions from recent Kolla meetings [1][2][3],
we established that, to avoid the DockerHub pull limit issues
in our CI, we will try using quay.io.

The images have already been published via Kolla CI
for master/Wallaby [4] and Victoria [5].

This change applies only to CI.
The user-facing move is yet to be discussed.

[1] http://eavesdrop.openstack.org/meetings/kolla/2021/kolla.2021-03-17-15.00.log.html#l-102
[2] http://eavesdrop.openstack.org/meetings/kolla/2021/kolla.2021-03-24-15.00.log.html#l-137
[3] http://eavesdrop.openstack.org/meetings/kolla/2021/kolla.2021-03-31-15.01.log.html#l-50
[4] https://review.opendev.org/c/openstack/kolla/+/781130
[5] https://review.opendev.org/c/openstack/kolla/+/781899

See related Kolla Ansible change
I22e11a994801447526ac26fc2815a1679876225e

Change-Id: Id6690dc9a486af0ed7d42555d84cf77d6de0b6af

This change adds support for policy-based routing via systemd-networkd.
Due to differences in the configuration mechanism, routing policy
rules are configured via dicts for Ubuntu, while remaining as strings
on CentOS. Ideally we would support both formats.

Story: 2004960
Task: 42217

Change-Id: I77aec0160eb7e4dd763326bfe6e3d9a44b248108