Change-Id: Ib12ac0e8dac8804085ea30bac7b4b0e659309cb0

This requires disabling libvirt_vm_trust_guest_rx_filters, which when
enabled triggers the following errors when booting baremetal instances
with Tenks on Libvirt 9 (and most likely since 8.9.0):

    Cannot set interface flags on 'macvtap1': Value too large for defined data type

This is apparently triggered by a Libvirt commit refreshing rx-filters
more often [1].

As explained in I71a2051d8acd63379bd70bc1287a059d4a7f6387, this setting
was added to allow traffic destined for other MAC addresses to reach VMs
when using a macvtap interface.

This will prevent multicast from working, but we don't need it for
baremetal tests in CI.

This setting will be enabled again once the issue is resolved in either
Libvirt or Tenks.

This reverts commit 21c68bbf.

Also increase timeout of upgrade jobs which is too short now due to the
added delay added by bare metal testing.

[1] https://gitlab.com/libvirt/libvirt/-/commit/060d4c83ef436cf56abfad51a4d64c39448e199d

Change-Id: I2cfd2667abb1ae8988b7a7fd9761b75c20a0eaa4

Kolla Ansible enabled RabbitMQ HA queues by default, which require a
manual migration step [1]. Adds these to the Kayobe upgrade CI.

[1] https://review.opendev.org/c/openstack/kolla-ansible/+/882825

Change-Id: I82c286fd17e3a1d7f31952442fa281302cda7ee4

Kolla Ansible enabled RabbitMQ HA queues by default, which require a
manual migration step [1]. Work around the failing precheck by enabling
HA queues in the previous release until we implement migration code in
Kayobe CI jobs.

[1] https://review.opendev.org/c/openstack/kolla-ansible/+/882825

Change-Id: Idbbe0dd57acc9b7a9440a67c2b677e526a6be917

Enable that by default in CI

Change-Id: I654f4dfa73afee478fb2c39594dc6f36081fb7c0

Rocky Linux 9.2 shipped with Libvirt 9.0.0 which breaks our bare metal
testing. Temporarily run bare metal testing only on Ubuntu.

This allows us to make rocky9 jobs voting again.

Change-Id: I8866cbc07fc28897648f3dc6f2a163323184e8a9

More than one year ago, change I96827fc32c1594ca9a0535e259929c49d3f0e704
enabled bare metal testing on Ubuntu, but only for non-upgrade jobs. It
should be safe to test during upgrade jobs too.

Change-Id: I9c698916999b30bf3fd8f7dfe5add7d332a84b6c

This is not needed anymore because the flag got renamed in Zed.

Change-Id: I0187f9a3f23dc59582059d2c7eb4ca1b283002b4

Upgrade CI job needs clouds.yaml to be used from Zed

Newer version of ansible-collections-openstack uses different return
value. [1]

[1] https://review.opendev.org/c/openstack/ansible-collections-openstack/+/841224

Change-Id: Ic0608bc6033025cb47655d601ffaf3744637832f

Yoga upper constraints were used to keep compatibility with Python 3.6.
This is not needed with all supported OS using Python 3.9 or newer.

This reverts commits d2e0d64e and
d190e9e3.

Change-Id: I35a07bcc2b7c9cbb49fa60e6802cc6288a34fbd8

CentOS Stream 8 support has been dropped. Migration path will be present
in Yoga release - as a followup change.

MichaelRigart.interfaces does not support custom routes for
NetworkManager yet. It has been disabled in CI for Rocky Linux 9
temporarily.

Non-voting CentOS Stream 9 CI overcloud job is using RL9 container
images (as kolla CI is not building CS9 images anymore).

Change-Id: Idf5ee822b03ba40179803c981500a6bad37594bf

The disable-selinux role has been renamed to selinux and now supports
setting desired state.

Previously Kayobe was defaulting to disabling and rebooted the host - to
avoid audit logs filling up. This change allows operators to define
desired SELinux state and defaults to permissive - to adhere to those
site policies that require SELinux to be at least in permissive state.

Change-Id: I42933b0b7d55c69c9f6992e331fafb2e6c42d4d1

Requirements upper constraints bumped python-novaclient to version
18.0.0 [1], which requires Python 3.8 [2]. This results in failures when
installing python-openstackclient on CentOS and Rocky with Python 3.6.

    ERROR: Cannot install python-openstackclient==5.8.0 because these package versions have conflicting dependencies.

    The conflict is caused by:
        python-openstackclient 5.8.0 depends on python-novaclient>=17.0.0
        The user requested (constraint) python-novaclient===18.0.0

Work around this issue by using yoga upper constraints until we upgrade
to CentOS Stream 9 and Rocky Linux 9.

This also fixes another issue seen on Ubuntu where image uploads to
Glance through Ansible fail with a 400 Bad Request error. This is caused
by the bump of openstacksdk to version 0.99.0 and will be fixed by a new
release of ansible-collections-openstack.

[1] https://review.opendev.org/c/openstack/requirements/+/842808
[2] https://review.opendev.org/c/openstack/python-novaclient/+/838944

Change-Id: I40c6b898963c2218d41d37bd73d40ce8dcf22b87

Adds support for SASL authentication of libvirt TCP and TLS connections
when using a compute host libvirt daemon.

In line with the dependent Kolla Ansible patch, we enable SASL by
default, and use DIGEST-MD5 with TCP and SCRAM-SHA-256 with TLS.

Depends-On: https://review.opendev.org/c/openstack/kolla-ansible/+/833022
Depends-On: https://github.com/stackhpc/ansible-role-libvirt-host/pull/52

Story: 2009858
Task: 44735

Change-Id: Id3972c24022aeb6421494c3cccdc8e7cbce802e6

nova-libvirt images now include qemu-utils on master [1] and xena [2].

[1] https://review.opendev.org/c/openstack/kolla/+/830401
[2] https://review.opendev.org/c/openstack/kolla/+/831411

Change-Id: I8f5f93340642d055cce7ef306d942e75b10c86a9

This change uses the new Galaxy requirements file in Kolla Ansible to
install the openstack.kolla collection.

Cross-project dependencies on ansible-collection-kolla are supported.

Story: 2009854
Task: 44504

Depends-On: https://review.opendev.org/c/openstack/kolla-ansible/+/819430
Change-Id: Iac185dd2bbbca128c6cf71b2734e94b3e1c6133b

Also remove some Wallaby upgrade compatibility code in CI playbooks

Change-Id: I84e336bec37eac12c659c7a9d9fc4ddd191a379d

We see frequent failures of upgrade jobs on stable branches due to lack
of disk space. Disable heat in an attempt to free up extra space, since
we do not need it for our testing.

Change-Id: I20c99ca9dd6e78b041e4662c9b1b54eac904e3c9

Now that the previous release is Wallaby, we can add overcloud and seed
upgrades on Ubuntu.

Change-Id: I69868dca6e1f3e4a1bf123c9e7f1f3412162b264

Libvirt debug logs can grow very large, particularly on Ubuntu (we have
seen up to 1.1GB). Disable libvirt debug logging using variable
nova_libvirt_logging_debug which was added to Kolla Ansible during the
Wallaby cycle (see change I90bfd1b300ad3202ea4d139fda6d6beb44c5820f) and
backported to Victoria and Ussuri.

Change-Id: Ic047e80aa382148b009b1c03581b417d17ec2598

The iscsi deploy interface was removed from Ironic [1]. Switch to the
direct deploy interface instead, which is the default.

[1] https://review.opendev.org/c/openstack/ironic/+/789382

Depends-On: https://review.opendev.org/c/openstack/kolla-ansible/+/801267
Change-Id: Ia0dcd3d302e0f04adbda4f3abf75a0ca7dd20dee

Overcloud upgrade jobs often fail when running on nodes with a 38G root
disk. The overcloud upgrade itself generally works, but post-upgrade
testing fails, for example when RabbitMQ gets in alarm state due to lack
of free disk space, which prevents Nova from functioning correctly and
makes it return "Unknown Error (HTTP 504)".

Change-Id: I536a526340f6d7dea7a69372904b777292774eba

The centos binary images now include nova 23.0.0 packages.

Change-Id: I0c7d57231a63e1f9ff3fdcdef1be7fff57160e08

Based on the discussions from recent Kolla meetings [1][2][3],
we established that, to avoid the DockerHub pull limit issues
in our CI, we will try using quay.io.

The images have already been published via Kolla CI
for master/Wallaby [4] and Victoria [5].

This change applies only to CI.
The user-facing move is yet to be discussed.

[1] http://eavesdrop.openstack.org/meetings/kolla/2021/kolla.2021-03-17-15.00.log.html#l-102
[2] http://eavesdrop.openstack.org/meetings/kolla/2021/kolla.2021-03-24-15.00.log.html#l-137
[3] http://eavesdrop.openstack.org/meetings/kolla/2021/kolla.2021-03-31-15.01.log.html#l-50
[4] https://review.opendev.org/c/openstack/kolla/+/781130
[5] https://review.opendev.org/c/openstack/kolla/+/781899

See related Kolla Ansible change
I22e11a994801447526ac26fc2815a1679876225e

Change-Id: Id6690dc9a486af0ed7d42555d84cf77d6de0b6af

The previous release has not been Rocky for a while.

Change-Id: Ic082ed08614c0f90d82ff211ac3dd19ed78d2bf5

* Use source images
* Need to specify bash for &> syntax

Issues worked around:

* Manually configuring bridge via ip commands makes ifup fail to bring
  up the link. Adds a kayobe-network-bootstrap Zuul CI role that adds
  persistent configuration for the all-in-one network.

* bridge not active after interfaces role bounce. Added a pause, similar
  to https://github.com/michaelrigart/ansible-role-interfaces/pull/31

* fails installing docker python module for kolla user. WARNING: The
  repository located at mirror-int.ord.rax.opendev.org is not a trusted
  or secure host and is being ignored ERROR: No matching distribution
  found for docker===4.4.0 Adding trusted host for PyPI mirror.

* Tenks fails to create block devices - missing qemu-img (in qemu-utils)

* Tenks qemu emulator is different on Ubuntu

Remaining issues:

* Bare metal testing is unreliable on Ubuntu - some jobs see IPMI
  failures such as the following:

    ipmitool chassis bootdev pxe

    Error setting Chassis Boot Parameter 5\nError setting Chassis Boot
    Parameter 0\n

  Bare metal testing is disabled on Ubuntu for now.

Depends-On: https://review.opendev.org/766984
Depends-On: https://review.opendev.org/766958

Story: 2004960
Task: 29393

Change-Id: I1985efae7c18f55c3ff7c27c17d6242523904f3e

This partially reverts commit 47bbb96b
which triggered a retry on vexxhost clouds.

The issue was introduced in Ie8fd965165e8d347d27528a2c16d0647e412ccdc,
which applied some fixes for CentOS 8.3, and inadvertently removed
the Tenks variable that forces the use of qemu for 'bare metal' VMs.
This lead to autodetection of KVM, which does not work well when nested
in all CI cloud providers.

This change fixes the issue by forcing the use of qemu for the overcloud
once more. It also adds a similar option for the seed VM job.

Change-Id: I6bc8da2b8da903e09b97df8cd95c68a562c11db9

Jobs that test baremetal deployment are currently failing when run on
Vexxhost infrastructure. The seed VM job also has trouble on Vexxhost,
although does sometimes pass. It's not clear why yet, and we are still
investigating. Until we have a solution, fail in the pre phase when we
land on vexxhost clouds, in order to trigger the zuul retry mechanism.

Also use the normal number (5) of attempts for upgrade jobs, since these
no longer deploy the previous release in the pre phase.

Finally, incorporate a fix from
https://review.opendev.org/c/openstack/kayobe-config-dev/+/777076 to
allow this to merge. This part can be reverted once the
kayobe-config-dev patch merges.

[1] http://lists.openstack.org/pipermail/openstack-discuss/2021-February/020685.html

Change-Id: Ie5ca04428855cbaf505adc57de76ea0b2d404080

* Bump stackhpc.libvirt-host to v1.7.1. On seed-hypervisors installed
  using CentOS 8.2 or earlier, interaction with libvirt may fail due to
  libgcrypt being incompatible. See
  https://github.com/stackhpc/ansible-role-libvirt-host/issues/42

* Bump MichaelRigart.interfaces to v1.9.2. The CentOS 8.3 cloud image
  includes an ifcfg-ens3-1 file. See
  https://github.com/michaelrigart/ansible-role-interfaces/pull/93

* Previously a second libvirt daemon was installed by Tenks on the host,
  however changes in libvirt 6.0.0 to separate libvirtd into multiple
  daemons do not allow for customisation of the PID files used by the
  new daemons. This leads to a conflict between the container and host
  daemons. Update the Tenks config to use the containerised Nova libvirt
  daemon. This depends on a change to the stackhpc.libvirt-host role:
  https://github.com/stackhpc/ansible-role-libvirt-host/pull/44

* Not CentOS 8.3 related, but tox jobs are now failing on python
  dependencies. Remove upper limits from docker and paramiko.

* Not CentOS 8.3 related, but Bifrost has enabled authentication by
  default. We are not ready to support this, so override it.

Story: 2008429
Task: 41378

Change-Id: Ie8fd965165e8d347d27528a2c16d0647e412ccdc

Ironic is now disabled by default in the overcloud. This brings Kayobe's
default set of services into line with Kolla Ansible. For environments
using Ironic in the overcloud, set kolla_enable_ironic to true in
kolla.yml.

Story: 2008207
Task: 40991

Change-Id: I33eb4fa534847e199a599350b525d4762a2beaac

This reverts commit bb9a595e.

Change-Id: Icd7b0f3b02bf3f22468daa8893c043ac9f3be4d6

Ironic has an issue where nodes will be switched to use the direct
deploy interface during upgrade, if it is enabled. This should be fixed
via https://review.opendev.org/755295, after which this patch should be
reverted.

Change-Id: I55af5d979bdbee9dd963845204c2f082fa7b27d7

Switches to use the IPA builder project for building IPA images.

Switches the IPA images used by default to CentOS 8 based image.

Changes the file extension of the IPA kernel image from vmlinuz to
kernel.

Story: 2007070
Task: 37953

Change-Id: I82fc455f41f48dacb453e135870dd776895d7c99
Story: 2006574
Task: 39485

Missing bifrost image for Train, so no seed upgrade job yet.

Depends-On: https://review.opendev.org/711067

Change-Id: I0e88683f775769c1a80879685b0e7a2983599b08
Story: 2006574
Task: 39047

Currently Kayobe stderr is logged to the Zuul playbook output. This
change redirects stderr to the ansible log files.

Change-Id: Ia2f25a2e41859159275d418b57ab59bb97f58d6a

Also refactors the CI job playbooks to create the previous git repo
clones for upgrade jobs in the kayobe-base job.

Change-Id: I10c2b872887a47fab1a373fc0525dcd6fc78f7e2

Kayobe uses a number of virtual environments on the remote hosts for
python dependencies such as shade, python-openstackclient, docker, etc.
By default these are stored in /opt/kayobe/venvs/. Typically we do not
provide version restrictions when installing these packages, so over the
course of time they may become stale and incompatible.

This change installs the latest version of packages allowed by OpenStack
upper constraints.

It also adds a new variable, 'pip_upper_constraints_file', to set the
upper constraints file. The existing variable
'kolla_upper_constraints_file' now defaults to the value of
'pip_upper_constraints_file'.

Change-Id: I8d2956f95bbc44b5a9e88e7569372048a62f12f5
Story: 2005923
Task: 34193

Using Kayobe with cliff 2.15.0 fails with the following error:

kayobe playbook run playbook.yml
invalid conflict_resolution value: 'ignore'

This change restricts cliff to <2.15.0.

Also, fix git cloning previous release repos in CI to allow this patch
to merge.

Zuul provides repositories with all branches containing all prospective
changes, not just the current branch. We can use this in the upgrade
jobs to include changes to the previous release branch when a patch
depends on them.

Change-Id: I7966cf33f6133090d83232504b8c1da4c7817e0c
Depends-On: https://review.opendev.org/665703/
Story: 2005891
Task: 33748

Allows users to explicitly specify which type of tenks
deployment they wish to create / destroy.
Preserves existing behaviour with defaults.
Modifies Zuul tests to use new tenks-deploy entrypoints.

Change-Id: I9aafed8481fd7564c0fc0abe5f6b21eceb824d75

Adds a new abstract parent job for the following kayobe deploy jobs:

kayobe-overcloud-centos
kayobe-overcloud-upgrade-centos
kayobe-seed-centos
kayobe-seed-upgrade-centos

This parent includes commonly used variables, and pre-run and post-run
playbooks for preparing and executing diagnostics, plus other
initialisation.

Change-Id: I9ab89fbc49bb32d86af56d50ec3914740e65d0f8