Follows the same pattern as apt config.yml

Change-Id: Ifb347372032120ec676747bad6201e0d82725976

This commit changes seed-images jobs to build seed and overcloud
container images, each with a customisable regex.

The rocky9 job continues to build only the base image, although for both
overcloud and seed instead of seed only.

The other seed-images jobs (centos9s and ubuntu-jammy) now build all
default overcloud images. This takes longer and can fail for various
reasons unrelated to Kayobe, but this is fine since the jobs are marked
as experimental.

This is to attempt to catch bugs in the image build code, such as the
ones fixed by [1].

Note that the ubuntu-jammy job does not build the bifrost-deploy seed
image because the build is currently timing out. The centos9s job builds
it successfully.

Also update release documentation to remind developers to run
experimental jobs.

[1] https://review.opendev.org/c/openstack/kayobe/+/921012

Change-Id: Idf705d9cf41766a897444898ac31c4635b70ec16

Change-Id: Ib12ac0e8dac8804085ea30bac7b4b0e659309cb0

Change-Id: Iaef2420c1f12fa4296c0e1c3d5afc4687d4b3c4a

This support is now available in the MichaelRigart.interfaces role.

The host configuration CI test has been updated to test policy-based
routing routes and rules on CentOS Stream and Rocky Linux. It also now
tests both the string and dict rule formats on CentOS and Rocky.

Change-Id: Ie77530c38ab426dcbaa442776bcf048d7bbc0f01

It was mentioned on openstack-discuss [1] that 0.5.2 has a known kernel
bug fixed in 0.5.3.

[1] https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/thread/7L6DSFN6GB65FBVHQATTLSQ7HWGBZJHM/

Change-Id: Ib637471d67002734084736f47d574815f13f1711

This allows use of repositories and proxies protected with HTTP basic
authentication.

Change-Id: I0ec4ec3e9d60bb1431b44dd6718415214ad80025

This has now been removed from upstream docker-ce packages.

```
the devicemapper storage-driver has been deprecated and removed
```

and has shown deprecation warnings for some time.

This change also includes some fixes necessary for CI to pass:

* update the Zuul previous_release variable to 2023.2

Closes-Bug: #2051233
Depends-On: https://review.opendev.org/c/openstack/kolla-ansible/+/906858
Change-Id: I263f59ea77e39accffe4febe0d47e56b35d9800e

Change-Id: I06b0bd4634ebb8b78baa23656c891333fd96b88b

Change-Id: I42c4a44afe8b0b19a2d7d14d9b748bf1e2ca5c10

This requires disabling libvirt_vm_trust_guest_rx_filters, which when
enabled triggers the following errors when booting baremetal instances
with Tenks on Libvirt 9 (and most likely since 8.9.0):

    Cannot set interface flags on 'macvtap1': Value too large for defined data type

This is apparently triggered by a Libvirt commit refreshing rx-filters
more often [1].

As explained in I71a2051d8acd63379bd70bc1287a059d4a7f6387, this setting
was added to allow traffic destined for other MAC addresses to reach VMs
when using a macvtap interface.

This will prevent multicast from working, but we don't need it for
baremetal tests in CI.

This setting will be enabled again once the issue is resolved in either
Libvirt or Tenks.

This reverts commit 21c68bbf.

Also increase timeout of upgrade jobs which is too short now due to the
added delay added by bare metal testing.

[1] https://gitlab.com/libvirt/libvirt/-/commit/060d4c83ef436cf56abfad51a4d64c39448e199d

Change-Id: I2cfd2667abb1ae8988b7a7fd9761b75c20a0eaa4

Kolla Ansible enabled RabbitMQ HA queues by default, which require a
manual migration step [1]. Adds these to the Kayobe upgrade CI.

[1] https://review.opendev.org/c/openstack/kolla-ansible/+/882825

Change-Id: I82c286fd17e3a1d7f31952442fa281302cda7ee4

Kolla Ansible enabled RabbitMQ HA queues by default, which require a
manual migration step [1]. Work around the failing precheck by enabling
HA queues in the previous release until we implement migration code in
Kayobe CI jobs.

[1] https://review.opendev.org/c/openstack/kolla-ansible/+/882825

Change-Id: Idbbe0dd57acc9b7a9440a67c2b677e526a6be917

For Rocky Linux 9, Kayobe will now disable STP on a bridge by default,
to preserve compatibility with network scripts, as Network Manager
enables STP on all bridges by default.
Enabling STP can lead to port down event if BPDU guard is enabled
on the switch.

Closes-Bug: #2028775

Change-Id: I35eaa92f4243af00697306aa801e5a733885ce4f

Enable that by default in CI

Change-Id: I654f4dfa73afee478fb2c39594dc6f36081fb7c0

Rocky Linux 9.2 shipped with Libvirt 9.0.0 which breaks our bare metal
testing. Temporarily run bare metal testing only on Ubuntu.

This allows us to make rocky9 jobs voting again.

Change-Id: I8866cbc07fc28897648f3dc6f2a163323184e8a9

More than one year ago, change I96827fc32c1594ca9a0535e259929c49d3f0e704
enabled bare metal testing on Ubuntu, but only for non-upgrade jobs. It
should be safe to test during upgrade jobs too.

Change-Id: I9c698916999b30bf3fd8f7dfe5add7d332a84b6c

This is not needed anymore because the flag got renamed in Zed.

Change-Id: I0187f9a3f23dc59582059d2c7eb4ca1b283002b4

Change ``ipa_build_dib_elements_default`` and
``ipa_build_dib_env_default`` to use ``os_distribution`` and
``os_release`` by default. This allows for Ubuntu images to be built
when running on Ubuntu.

Rocky will still build CentOS images, as Rocky IPA images have not been
tested yet.

Change-Id: Iefd2d0b7a3a3e07f5c112d58e2ec0b3da0a747d3

The 'kayobe * host configure' commands no longer use the 'kolla-ansible
bootstrap-servers' command, and associated 'baremetal' role in Kolla
Ansible. The functionality provided by the 'baremetal' role has been
extracted into the openstack.kolla Ansible collection, and split
into separate roles. This allows Kayobe to use it directly, and only the
necessary parts.

This change improves failure handling in these Kayobe commands, and aims
to reduce confusion over which '--limit' and '--tags' arguments to
provide.  This ensures that if a host fails during a host configuration
command, other hosts are able to continue to completion. Previously, if
any host failed during the Kayobe playbooks, the 'kolla-ansible
bootstrap-servers' command would not run. This is useful at scale, where
host failures occur more frequently.

This change has implications for configuration of Kayobe, since some
variables that were previously in Kolla Ansible are now in Kayobe.

Several parts of the baremetal role have been split out and used here:

* apparmor-libvirt: disable AppArmor rules for libvirt on Ubuntu.
* docker: Docker installation & configuration. The docker role in
  openstack.kolla combines functionality from kolla-ansible and kayobe.
* etc-hosts: it proved difficult to generalise this, so we have some
  almost duplicated the code from kolla-ansible here. Requires delegated
  fact gathering for the case when --limit is used.
* firewall: support to disable UFW, for feature parity.
* kolla-packages: miscellaneous package installs & removals.

The addition of the stack user to the docker group has been moved to the
user bootstrapping playbook, and the docker SDK installation has been
moved to the virtualenv setup playbook.

Depends-On: https://review.opendev.org/c/openstack/ansible-collection-kolla/+/829587

Story: 2009854
Task: 44505

Change-Id: I61a61ca59652b13687c2247d5881012b51f666a7

This build takes time and can fail due to lack of disk space. It got
enabled when we changed overcloud_dib_build_host_images to true.

Also fix bifrost overrides which was wrongly applied, we need to use
dib.yml instead of bifrost.yml, like in kayobe-seed-base.

Change-Id: I1edafbb41a26587a5ef794b3b9886fdf189a0a1a

Not only TLS jobs need that treatment, Rocky9/CentOS
Stream 9 jobs have the same issue - let's disable
Heat and Horizon in all overcloud jobs.

Change-Id: Iecab44969cea015b363ec6884ef6a7c9960a6b3f

Change-Id: Ie88ca550d4ed619209c08719328ea69e10c274ad

Upgrade CI job needs clouds.yaml to be used from Zed

Newer version of ansible-collections-openstack uses different return
value. [1]

[1] https://review.opendev.org/c/openstack/ansible-collections-openstack/+/841224

Change-Id: Ic0608bc6033025cb47655d601ffaf3744637832f

Yoga upper constraints were used to keep compatibility with Python 3.6.
This is not needed with all supported OS using Python 3.9 or newer.

This reverts commits d2e0d64e and
d190e9e3.

Change-Id: I35a07bcc2b7c9cbb49fa60e6802cc6288a34fbd8

CentOS Stream 8 support has been dropped. Migration path will be present
in Yoga release - as a followup change.

MichaelRigart.interfaces does not support custom routes for
NetworkManager yet. It has been disabled in CI for Rocky Linux 9
temporarily.

Non-voting CentOS Stream 9 CI overcloud job is using RL9 container
images (as kolla CI is not building CS9 images anymore).

Change-Id: Idf5ee822b03ba40179803c981500a6bad37594bf

Supports creating and using swap files, or using pre-existing swap
devices.

Story: 2004958
Task: 29390

Change-Id: Iadb540f42036a4a63cdd5b695b82f1504b3a4a28

Change-Id: I7c863d1875908d2b885918ec7caed747ae6e345b

Followups after I295e8f5f1cc9b7af1cd45ac788db473510220170

Change-Id: I798a59ffeff060352e73ae755314a83222c92260

Depends-On: https://review.opendev.org/c/openstack/kolla-ansible/+/859828

Change-Id: I295e8f5f1cc9b7af1cd45ac788db473510220170

This allows operators to configure arbitrarily named VLAN interfaces
using systemd-networkd.

Story: 2010266
Task: 46178

Change-Id: I666d7011bde0050ebc509b427c1d4f5a66b6231a

Co-Authored-By: Bartosz Bezak <bartosz@stackhpc.com>

Change-Id: I06a3e9922cf95979f3bca120cd82633046270fa3

Enables the installation and configuration of firewalld on Ubuntu
systems.

Change-Id: I4a97a2aeed277be672e15e5c7727b810e11d3c42
Story: 2010160
Task: 45818

Change-Id: Iec0b9cd24eda4fc0fc38003dea66c50ece7425b6

The disable-selinux role has been renamed to selinux and now supports
setting desired state.

Previously Kayobe was defaulting to disabling and rebooted the host - to
avoid audit logs filling up. This change allows operators to define
desired SELinux state and defaults to permissive - to adhere to those
site policies that require SELinux to be at least in permissive state.

Change-Id: I42933b0b7d55c69c9f6992e331fafb2e6c42d4d1

Change-Id: Ibf4b928222713bedb7e856307f5ad91e60953795

Requirements upper constraints bumped python-novaclient to version
18.0.0 [1], which requires Python 3.8 [2]. This results in failures when
installing python-openstackclient on CentOS and Rocky with Python 3.6.

    ERROR: Cannot install python-openstackclient==5.8.0 because these package versions have conflicting dependencies.

    The conflict is caused by:
        python-openstackclient 5.8.0 depends on python-novaclient>=17.0.0
        The user requested (constraint) python-novaclient===18.0.0

Work around this issue by using yoga upper constraints until we upgrade
to CentOS Stream 9 and Rocky Linux 9.

This also fixes another issue seen on Ubuntu where image uploads to
Glance through Ansible fail with a 400 Bad Request error. This is caused
by the bump of openstacksdk to version 0.99.0 and will be fixed by a new
release of ansible-collections-openstack.

[1] https://review.opendev.org/c/openstack/requirements/+/842808
[2] https://review.opendev.org/c/openstack/python-novaclient/+/838944

Change-Id: I40c6b898963c2218d41d37bd73d40ce8dcf22b87

Enable the Ironic ipxe boot interface by default, following a similar
change in Ironic [1].

Drop the kolla_enable_ironic_ipxe flag, following a similar change in
Kolla Ansible [2]. Both PXE and iPXE are now enabled by default. Users
may revert to using PXE for ironic inspector's dnsmasq, by setting
ironic_dnsmasq_serve_ipxe to false in etc/kayobe/kolla/globals.yml.

[1] https://review.opendev.org/c/openstack/ironic/+/816824
[2] https://review.opendev.org/c/openstack/kolla-ansible/+/834512/

Depends-On: https://review.opendev.org/c/openstack/kolla-ansible/+/832159
Depends-On: https://review.opendev.org/c/openstack/kolla-ansible/+/834511
Depends-On: https://review.opendev.org/c/openstack/kolla-ansible/+/837069

Change-Id: Ifb80bd15a20c9cfb8fbc6e0f6ac23baae631a18e

Disk and container image builds tend to be fairly unreliable.
With 3 voting seed jobs all building images, this can introduce
instability into the CI jobs.

This change adds a non-voting kayobe-seed-images-centos8s job, which
does the following:

* Builds IPA images
* Builds an overcloud host image
* Builds a base container image

Similar Rocky and Ubuntu jobs are added to the experimental pipeline,
and may be run by commenting 'check experimental' in gerrit.

The existing kayobe-seed-* jobs no longer build images.

Change-Id: Idecda342f3ab86733e8d59061458d44af834dbb0

The contextfilter decorator was deprecated in jinja2 3.0.0, and has been
dropped in 3.1.0. This results in the following warning, and failed
attempts to use filters:

    [WARNING]: Skipping plugin (networks.py) as it seems to be invalid:
    module 'jinja2' has no attribute 'contextfilter'

This change switches to use the pass_context decorator. The minimum
version of Jinja2 is raised to 3 to ensure pass_context is present.

This change also includes some changes to address issues with image
builds in CI, caused by CentOS Scream.

1. disable IPA image builds in seed deploy jobs

IPA image builds will be split out into a separate job. For now, disable
them.

2. disable overcloud host image builds in seed deploy jobs

Overcloud host image builds will be split out into a separate job. For
now, disable them.

Depends-On: https://review.opendev.org/c/openstack/kayobe/+/835279
Change-Id: If657bf5b0117812d3c53942464cc41cf86cc8ad5