This support is now available in the MichaelRigart.interfaces role.

The host configuration CI test has been updated to test policy-based
routing routes and rules on CentOS Stream and Rocky Linux. It also now
tests both the string and dict rule formats on CentOS and Rocky.

Change-Id: Ie77530c38ab426dcbaa442776bcf048d7bbc0f01

This allows use of repositories and proxies protected with HTTP basic
authentication.

Change-Id: I0ec4ec3e9d60bb1431b44dd6718415214ad80025

This has now been removed from upstream docker-ce packages.

```
the devicemapper storage-driver has been deprecated and removed
```

and has shown deprecation warnings for some time.

This change also includes some fixes necessary for CI to pass:

* update the Zuul previous_release variable to 2023.2

Closes-Bug: #2051233
Depends-On: https://review.opendev.org/c/openstack/kolla-ansible/+/906858
Change-Id: I263f59ea77e39accffe4febe0d47e56b35d9800e

Change-Id: I06b0bd4634ebb8b78baa23656c891333fd96b88b

Change-Id: I42c4a44afe8b0b19a2d7d14d9b748bf1e2ca5c10

This requires disabling libvirt_vm_trust_guest_rx_filters, which when
enabled triggers the following errors when booting baremetal instances
with Tenks on Libvirt 9 (and most likely since 8.9.0):

    Cannot set interface flags on 'macvtap1': Value too large for defined data type

This is apparently triggered by a Libvirt commit refreshing rx-filters
more often [1].

As explained in I71a2051d8acd63379bd70bc1287a059d4a7f6387, this setting
was added to allow traffic destined for other MAC addresses to reach VMs
when using a macvtap interface.

This will prevent multicast from working, but we don't need it for
baremetal tests in CI.

This setting will be enabled again once the issue is resolved in either
Libvirt or Tenks.

This reverts commit 21c68bbf.

Also increase timeout of upgrade jobs which is too short now due to the
added delay added by bare metal testing.

[1] https://gitlab.com/libvirt/libvirt/-/commit/060d4c83ef436cf56abfad51a4d64c39448e199d

Change-Id: I2cfd2667abb1ae8988b7a7fd9761b75c20a0eaa4

Kolla Ansible enabled RabbitMQ HA queues by default, which require a
manual migration step [1]. Adds these to the Kayobe upgrade CI.

[1] https://review.opendev.org/c/openstack/kolla-ansible/+/882825

Change-Id: I82c286fd17e3a1d7f31952442fa281302cda7ee4

Kolla Ansible enabled RabbitMQ HA queues by default, which require a
manual migration step [1]. Work around the failing precheck by enabling
HA queues in the previous release until we implement migration code in
Kayobe CI jobs.

[1] https://review.opendev.org/c/openstack/kolla-ansible/+/882825

Change-Id: Idbbe0dd57acc9b7a9440a67c2b677e526a6be917

For Rocky Linux 9, Kayobe will now disable STP on a bridge by default,
to preserve compatibility with network scripts, as Network Manager
enables STP on all bridges by default.
Enabling STP can lead to port down event if BPDU guard is enabled
on the switch.

Closes-Bug: #2028775

Change-Id: I35eaa92f4243af00697306aa801e5a733885ce4f

Enable that by default in CI

Change-Id: I654f4dfa73afee478fb2c39594dc6f36081fb7c0

Rocky Linux 9.2 shipped with Libvirt 9.0.0 which breaks our bare metal
testing. Temporarily run bare metal testing only on Ubuntu.

This allows us to make rocky9 jobs voting again.

Change-Id: I8866cbc07fc28897648f3dc6f2a163323184e8a9

More than one year ago, change I96827fc32c1594ca9a0535e259929c49d3f0e704
enabled bare metal testing on Ubuntu, but only for non-upgrade jobs. It
should be safe to test during upgrade jobs too.

Change-Id: I9c698916999b30bf3fd8f7dfe5add7d332a84b6c

This is not needed anymore because the flag got renamed in Zed.

Change-Id: I0187f9a3f23dc59582059d2c7eb4ca1b283002b4

Change ``ipa_build_dib_elements_default`` and
``ipa_build_dib_env_default`` to use ``os_distribution`` and
``os_release`` by default. This allows for Ubuntu images to be built
when running on Ubuntu.

Rocky will still build CentOS images, as Rocky IPA images have not been
tested yet.

Change-Id: Iefd2d0b7a3a3e07f5c112d58e2ec0b3da0a747d3

The 'kayobe * host configure' commands no longer use the 'kolla-ansible
bootstrap-servers' command, and associated 'baremetal' role in Kolla
Ansible. The functionality provided by the 'baremetal' role has been
extracted into the openstack.kolla Ansible collection, and split
into separate roles. This allows Kayobe to use it directly, and only the
necessary parts.

This change improves failure handling in these Kayobe commands, and aims
to reduce confusion over which '--limit' and '--tags' arguments to
provide.  This ensures that if a host fails during a host configuration
command, other hosts are able to continue to completion. Previously, if
any host failed during the Kayobe playbooks, the 'kolla-ansible
bootstrap-servers' command would not run. This is useful at scale, where
host failures occur more frequently.

This change has implications for configuration of Kayobe, since some
variables that were previously in Kolla Ansible are now in Kayobe.

Several parts of the baremetal role have been split out and used here:

* apparmor-libvirt: disable AppArmor rules for libvirt on Ubuntu.
* docker: Docker installation & configuration. The docker role in
  openstack.kolla combines functionality from kolla-ansible and kayobe.
* etc-hosts: it proved difficult to generalise this, so we have some
  almost duplicated the code from kolla-ansible here. Requires delegated
  fact gathering for the case when --limit is used.
* firewall: support to disable UFW, for feature parity.
* kolla-packages: miscellaneous package installs & removals.

The addition of the stack user to the docker group has been moved to the
user bootstrapping playbook, and the docker SDK installation has been
moved to the virtualenv setup playbook.

Depends-On: https://review.opendev.org/c/openstack/ansible-collection-kolla/+/829587

Story: 2009854
Task: 44505

Change-Id: I61a61ca59652b13687c2247d5881012b51f666a7

This build takes time and can fail due to lack of disk space. It got
enabled when we changed overcloud_dib_build_host_images to true.

Also fix bifrost overrides which was wrongly applied, we need to use
dib.yml instead of bifrost.yml, like in kayobe-seed-base.

Change-Id: I1edafbb41a26587a5ef794b3b9886fdf189a0a1a

Not only TLS jobs need that treatment, Rocky9/CentOS
Stream 9 jobs have the same issue - let's disable
Heat and Horizon in all overcloud jobs.

Change-Id: Iecab44969cea015b363ec6884ef6a7c9960a6b3f

Change-Id: Ie88ca550d4ed619209c08719328ea69e10c274ad

Upgrade CI job needs clouds.yaml to be used from Zed

Newer version of ansible-collections-openstack uses different return
value. [1]

[1] https://review.opendev.org/c/openstack/ansible-collections-openstack/+/841224

Change-Id: Ic0608bc6033025cb47655d601ffaf3744637832f

Yoga upper constraints were used to keep compatibility with Python 3.6.
This is not needed with all supported OS using Python 3.9 or newer.

This reverts commits d2e0d64e and
d190e9e3.

Change-Id: I35a07bcc2b7c9cbb49fa60e6802cc6288a34fbd8

CentOS Stream 8 support has been dropped. Migration path will be present
in Yoga release - as a followup change.

MichaelRigart.interfaces does not support custom routes for
NetworkManager yet. It has been disabled in CI for Rocky Linux 9
temporarily.

Non-voting CentOS Stream 9 CI overcloud job is using RL9 container
images (as kolla CI is not building CS9 images anymore).

Change-Id: Idf5ee822b03ba40179803c981500a6bad37594bf

Supports creating and using swap files, or using pre-existing swap
devices.

Story: 2004958
Task: 29390

Change-Id: Iadb540f42036a4a63cdd5b695b82f1504b3a4a28

Change-Id: I7c863d1875908d2b885918ec7caed747ae6e345b

Followups after I295e8f5f1cc9b7af1cd45ac788db473510220170

Change-Id: I798a59ffeff060352e73ae755314a83222c92260

Depends-On: https://review.opendev.org/c/openstack/kolla-ansible/+/859828

Change-Id: I295e8f5f1cc9b7af1cd45ac788db473510220170

This allows operators to configure arbitrarily named VLAN interfaces
using systemd-networkd.

Story: 2010266
Task: 46178

Change-Id: I666d7011bde0050ebc509b427c1d4f5a66b6231a

Co-Authored-By: Bartosz Bezak <bartosz@stackhpc.com>

Change-Id: I06a3e9922cf95979f3bca120cd82633046270fa3

Enables the installation and configuration of firewalld on Ubuntu
systems.

Change-Id: I4a97a2aeed277be672e15e5c7727b810e11d3c42
Story: 2010160
Task: 45818

Change-Id: Iec0b9cd24eda4fc0fc38003dea66c50ece7425b6

The disable-selinux role has been renamed to selinux and now supports
setting desired state.

Previously Kayobe was defaulting to disabling and rebooted the host - to
avoid audit logs filling up. This change allows operators to define
desired SELinux state and defaults to permissive - to adhere to those
site policies that require SELinux to be at least in permissive state.

Change-Id: I42933b0b7d55c69c9f6992e331fafb2e6c42d4d1

Change-Id: Ibf4b928222713bedb7e856307f5ad91e60953795

Requirements upper constraints bumped python-novaclient to version
18.0.0 [1], which requires Python 3.8 [2]. This results in failures when
installing python-openstackclient on CentOS and Rocky with Python 3.6.

    ERROR: Cannot install python-openstackclient==5.8.0 because these package versions have conflicting dependencies.

    The conflict is caused by:
        python-openstackclient 5.8.0 depends on python-novaclient>=17.0.0
        The user requested (constraint) python-novaclient===18.0.0

Work around this issue by using yoga upper constraints until we upgrade
to CentOS Stream 9 and Rocky Linux 9.

This also fixes another issue seen on Ubuntu where image uploads to
Glance through Ansible fail with a 400 Bad Request error. This is caused
by the bump of openstacksdk to version 0.99.0 and will be fixed by a new
release of ansible-collections-openstack.

[1] https://review.opendev.org/c/openstack/requirements/+/842808
[2] https://review.opendev.org/c/openstack/python-novaclient/+/838944

Change-Id: I40c6b898963c2218d41d37bd73d40ce8dcf22b87

Enable the Ironic ipxe boot interface by default, following a similar
change in Ironic [1].

Drop the kolla_enable_ironic_ipxe flag, following a similar change in
Kolla Ansible [2]. Both PXE and iPXE are now enabled by default. Users
may revert to using PXE for ironic inspector's dnsmasq, by setting
ironic_dnsmasq_serve_ipxe to false in etc/kayobe/kolla/globals.yml.

[1] https://review.opendev.org/c/openstack/ironic/+/816824
[2] https://review.opendev.org/c/openstack/kolla-ansible/+/834512/

Depends-On: https://review.opendev.org/c/openstack/kolla-ansible/+/832159
Depends-On: https://review.opendev.org/c/openstack/kolla-ansible/+/834511
Depends-On: https://review.opendev.org/c/openstack/kolla-ansible/+/837069

Change-Id: Ifb80bd15a20c9cfb8fbc6e0f6ac23baae631a18e

Disk and container image builds tend to be fairly unreliable.
With 3 voting seed jobs all building images, this can introduce
instability into the CI jobs.

This change adds a non-voting kayobe-seed-images-centos8s job, which
does the following:

* Builds IPA images
* Builds an overcloud host image
* Builds a base container image

Similar Rocky and Ubuntu jobs are added to the experimental pipeline,
and may be run by commenting 'check experimental' in gerrit.

The existing kayobe-seed-* jobs no longer build images.

Change-Id: Idecda342f3ab86733e8d59061458d44af834dbb0

The contextfilter decorator was deprecated in jinja2 3.0.0, and has been
dropped in 3.1.0. This results in the following warning, and failed
attempts to use filters:

    [WARNING]: Skipping plugin (networks.py) as it seems to be invalid:
    module 'jinja2' has no attribute 'contextfilter'

This change switches to use the pass_context decorator. The minimum
version of Jinja2 is raised to 3 to ensure pass_context is present.

This change also includes some changes to address issues with image
builds in CI, caused by CentOS Scream.

1. disable IPA image builds in seed deploy jobs

IPA image builds will be split out into a separate job. For now, disable
them.

2. disable overcloud host image builds in seed deploy jobs

Overcloud host image builds will be split out into a separate job. For
now, disable them.

Depends-On: https://review.opendev.org/c/openstack/kayobe/+/835279
Change-Id: If657bf5b0117812d3c53942464cc41cf86cc8ad5

Adds support for SASL authentication of libvirt TCP and TLS connections
when using a compute host libvirt daemon.

In line with the dependent Kolla Ansible patch, we enable SASL by
default, and use DIGEST-MD5 with TCP and SCRAM-SHA-256 with TLS.

Depends-On: https://review.opendev.org/c/openstack/kolla-ansible/+/833022
Depends-On: https://github.com/stackhpc/ansible-role-libvirt-host/pull/52

Story: 2009858
Task: 44735

Change-Id: Id3972c24022aeb6421494c3cccdc8e7cbce802e6

In some cases it may be desirable to run libvirt daemon on the host. For
example, when mixing host and container OS distributions.

This change makes it possible to disable the nova_libvirt container, by
setting kolla_enable_nova_libvirt_container to false.

The stackhpc.libvirt-host role is used in order to install and configure
a libvirt daemon on compute hosts when
kolla_enable_nova_libvirt_container is false.

Depends-On: https://review.opendev.org/c/openstack/kolla-ansible/+/825357
Depends-On: https://review.opendev.org/c/openstack/kayobe-config-dev/+/829225
Depends-On: https://github.com/stackhpc/ansible-role-libvirt-host/pull/51

Story: 2009858
Task: 44495

Change-Id: I73fef63fb886a9d543d2f4231fb009523495edb3

This change adds support for configuration of Apt package manager in
/etc/apt/apt.conf.d/. This allows adding arbitrary global configuration
options for Apt. Options can be added in different files, allowing for
different filename-based priorities.

CI tests and documentation are provided.

Story: 2009655
Task: 43987

Change-Id: I9d7d18851359e97cd01b4c2287bf79110796b25a

This change adds support for configuring Apt repositories on Ubuntu
hosts during host configuration.

Repositories are configured in a single file
(/etc/apt/sources.list.d/kayobe.sources), using the modern deb822
format [1]. This format is more flexible and readable than the original
single-line format, particularly if multiple options are used.

Using a single file allows us to more easily keep the set of
repositories in sync, since Ansible doesn't make it easy to clean things
up.

Support is added for marking repositories as signed by a particular GPG
key. This approach is now preferred over the deprecated [2] apt-key
tool, which resulted in a set of globally trusted keys.

It is also possible to disable the repositories in
/etc/apt/sources.list via apt_disable_sources_list. This allows for
replacing the standard repositories with a local mirror.

CI tests and documentation are provided.

[1] https://manpages.ubuntu.com/manpages/focal/en/man5/sources.list.5.html
[2] https://manpages.ubuntu.com/manpages/groovy/man8/apt-key.8.html

Story: 2009655
Task: 43818

Change-Id: I3f821937b0930a0ac9341178de7ae5123d82b957

Change-Id: If7d6e58b19f98ccb7cc4c209e458cb6f4f4765ad