The newer version sets the connection.autoconnect-slaves=1 config
option[1]. This will ensure that slave interfaces will also reconnect
after bouncing the parent.

[1] https://github.com/michaelrigart/ansible-role-interfaces/releases/tag/v1.15.1

Closes-Bug: #2072340
Change-Id: I2d7a8f1368d4f0e02d199029c15c453a56582b41
(cherry picked from commit 447635ac)

This change bumps up the maximum supported Ansible version to 10.x
(ansible-core 2.17.x) and minimum to 9.x. This synchronises Kayobe with
Kolla Ansible.

Change-Id: Idfbf0eed0b2333d550f822e40a0d96ae227f6159

Change-Id: I546d62490d66a8829fc906e7d021dc056001b4bb

Change-Id: Ib12ac0e8dac8804085ea30bac7b4b0e659309cb0

Change-Id: Ica6c0b204094ed61aef8242c934eb3b7715ce944

Supports calling custom Kolla Ansible commands directly after a
``kayobe control host bootstrap``.

Change-Id: I19f188cc002f8578618003e90c0a4a154b806e49

When python3 is missing from the host operating system, the install.sh
and install-dev.sh scripts fail with the following error:

    Creating virtual environment in /home/centos/kayobe-venv
    /home/centos/kayobe/dev/functions: line 169: python3: command not found

Change-Id: I50d47ceeeae803545d78a40bbeec1aaa2f5112b1

Since using the to_bool function in more places in
I3a5a43fef9c3d68d0db02be12b9f892c437e513d, we are now more strict about
the result of the variable dump. If there are no controllers in the
inventory, the result will not be a valid boolean and the to_bool
function will exit non-zero.

This change fixes the issue by running against localhost, which should
always be in the inventory.

Change-Id: Idcfd9d335f11f6c4d676033128d207f62b363ee9

This requires disabling libvirt_vm_trust_guest_rx_filters, which when
enabled triggers the following errors when booting baremetal instances
with Tenks on Libvirt 9 (and most likely since 8.9.0):

    Cannot set interface flags on 'macvtap1': Value too large for defined data type

This is apparently triggered by a Libvirt commit refreshing rx-filters
more often [1].

As explained in I71a2051d8acd63379bd70bc1287a059d4a7f6387, this setting
was added to allow traffic destined for other MAC addresses to reach VMs
when using a macvtap interface.

This will prevent multicast from working, but we don't need it for
baremetal tests in CI.

This setting will be enabled again once the issue is resolved in either
Libvirt or Tenks.

This reverts commit 21c68bbf.

Also increase timeout of upgrade jobs which is too short now due to the
added delay added by bare metal testing.

[1] https://gitlab.com/libvirt/libvirt/-/commit/060d4c83ef436cf56abfad51a4d64c39448e199d

Change-Id: I2cfd2667abb1ae8988b7a7fd9761b75c20a0eaa4

Kolla Ansible enabled RabbitMQ HA queues by default, which require a
manual migration step [1]. Adds these to the Kayobe upgrade CI.

[1] https://review.opendev.org/c/openstack/kolla-ansible/+/882825

Change-Id: I82c286fd17e3a1d7f31952442fa281302cda7ee4

Various functions in the development/testing scripts rely on 'kayobe
configuration dump' to extract the value of flags. If this command fails
for any reason, we should exit the script. Currently, some places we
continue and return 1, since we check the output against the string
'true'.

The to_bool helper function handles failure by checking for a valid
boolean output, so let's use that everywhere.

Change-Id: I3a5a43fef9c3d68d0db02be12b9f892c437e513d

Yoga upper constraints were used to keep compatibility with Python 3.6.
This is not needed with all supported OS using Python 3.9 or newer.

This reverts commits d2e0d64e and
d190e9e3.

Change-Id: I35a07bcc2b7c9cbb49fa60e6802cc6288a34fbd8

CentOS Stream 8 support has been dropped. Migration path will be present
in Yoga release - as a followup change.

MichaelRigart.interfaces does not support custom routes for
NetworkManager yet. It has been disabled in CI for Rocky Linux 9
temporarily.

Non-voting CentOS Stream 9 CI overcloud job is using RL9 container
images (as kolla CI is not building CS9 images anymore).

Change-Id: Idf5ee822b03ba40179803c981500a6bad37594bf

Fixed sed replacement line.

Change-Id: I2e26a08ea37e5f558af533b38d096f011ed2a5b8

This is changed in order to facilitate running Ironic checks before
OpenStack is deployed.

Change-Id: Ibc4c9c93f52414c23cd7d8b4f1b32c85cb233681

This is required to be able to install tenks. Otherwise, we try to
install Jinja2 3.1.2 which requires Python 3.7 or newer.

Change-Id: Ie497b191b6de8bc818dc4a2a12f7129a02d0fd00

Requirements upper constraints bumped python-novaclient to version
18.0.0 [1], which requires Python 3.8 [2]. This results in failures when
installing python-openstackclient on CentOS and Rocky with Python 3.6.

    ERROR: Cannot install python-openstackclient==5.8.0 because these package versions have conflicting dependencies.

    The conflict is caused by:
        python-openstackclient 5.8.0 depends on python-novaclient>=17.0.0
        The user requested (constraint) python-novaclient===18.0.0

Work around this issue by using yoga upper constraints until we upgrade
to CentOS Stream 9 and Rocky Linux 9.

This also fixes another issue seen on Ubuntu where image uploads to
Glance through Ansible fail with a 400 Bad Request error. This is caused
by the bump of openstacksdk to version 0.99.0 and will be fixed by a new
release of ansible-collections-openstack.

[1] https://review.opendev.org/c/openstack/requirements/+/842808
[2] https://review.opendev.org/c/openstack/python-novaclient/+/838944

Change-Id: I40c6b898963c2218d41d37bd73d40ce8dcf22b87

Enable the Ironic ipxe boot interface by default, following a similar
change in Ironic [1].

Drop the kolla_enable_ironic_ipxe flag, following a similar change in
Kolla Ansible [2]. Both PXE and iPXE are now enabled by default. Users
may revert to using PXE for ironic inspector's dnsmasq, by setting
ironic_dnsmasq_serve_ipxe to false in etc/kayobe/kolla/globals.yml.

[1] https://review.opendev.org/c/openstack/ironic/+/816824
[2] https://review.opendev.org/c/openstack/kolla-ansible/+/834512/

Depends-On: https://review.opendev.org/c/openstack/kolla-ansible/+/832159
Depends-On: https://review.opendev.org/c/openstack/kolla-ansible/+/834511
Depends-On: https://review.opendev.org/c/openstack/kolla-ansible/+/837069

Change-Id: Ifb80bd15a20c9cfb8fbc6e0f6ac23baae631a18e

In some cases it may be desirable to run libvirt daemon on the host. For
example, when mixing host and container OS distributions.

This change makes it possible to disable the nova_libvirt container, by
setting kolla_enable_nova_libvirt_container to false.

The stackhpc.libvirt-host role is used in order to install and configure
a libvirt daemon on compute hosts when
kolla_enable_nova_libvirt_container is false.

Depends-On: https://review.opendev.org/c/openstack/kolla-ansible/+/825357
Depends-On: https://review.opendev.org/c/openstack/kayobe-config-dev/+/829225
Depends-On: https://github.com/stackhpc/ansible-role-libvirt-host/pull/51

Story: 2009858
Task: 44495

Change-Id: I73fef63fb886a9d543d2f4231fb009523495edb3

Change-Id: If7d6e58b19f98ccb7cc4c209e458cb6f4f4765ad

Previously we were using the zuul user in the TLS jobs. This was due to
a permissions issue when accessing the CA certificate in kayobe-config
in the zuul user's home directory.

This change reverts to the default of using the stack user for the TLS
jobs. In order to make this work, the generated CA cert chain is added
to the trust store.

Change-Id: I875f8976df75dee68ba00842fe624c29cc1b123c

Set the Ironic boot mode to legacy BIOS explicitly in Tenks config in
anticipation of an upcoming change to the default boot mode.

Override the boot mode to UEFI in the overcloud TLS job to improve
coverage. This requires enabling iPXE booting.

Depends-On: https://review.opendev.org/c/openstack/tenks/+/827479/

Change-Id: Id1b4e9775c834b8b97e086241ee8b247977225a2

As a first step towards supporting multiple overcloud disk images, this
change introduces a new command to build a disk image directly with DIB:
`kayobe overcloud host image build`.

It also disables building a root disk image during Bifrost bootstrap if
overcloud_dib_build_host_images is set to true.

Change-Id: I93d242889e225b4e60254f6b9cc5eeb457294ac8
Story: 2002098
Task: 41693

Depends-On: https://review.opendev.org/c/openstack/kayobe/+/812670
Depends-On: https://review.opendev.org/c/openstack/kayobe-config-dev/+/805239
Depends-On: https://review.opendev.org/c/openstack/kayobe/+/813212

Change-Id: I7dbd643fec72223bfda20cbbc37809ae78073804

Arguments are passed through to kayobe-env in kayobe-config, which
allows to set the Kayobe environment.

Change-Id: I4c72e32e5379237340284a09874b0c500e41ad0f

By default, Ansible injects a variable for every fact, prefixed with
ansible_. This can result in a large number of variables for each host,
which at scale can incur a performance penalty. Ansible provides a
configuration option [0] that can be set to False to prevent this
injection of facts. In this case, facts should be referenced via
ansible_facts.<fact>.

This change updates all references to Ansible facts within Kayobe
from using individual fact variables to using the items in the
ansible_facts dictionary. This allows users to disable fact variable
injection in their Ansible configuration, which may provide some
performance improvement.

This change disables fact variable injection in the ansible
configuration used in CI, to catch any attempts to use the injected
variables.

[0] https://docs.ansible.com/ansible/latest/reference_appendices/config.html#inject-facts-as-vars

Story: 2007993
Task: 42464
Depends-On: https://review.opendev.org/c/openstack/kolla-ansible/+/791276

Change-Id: I14db53ed6e57d37bbd28dd5819e432e3fe6628b2

The iscsi deploy interface was removed from Ironic [1]. Switch to the
direct deploy interface instead, which is the default.

[1] https://review.opendev.org/c/openstack/ironic/+/789382

Depends-On: https://review.opendev.org/c/openstack/kolla-ansible/+/801267
Change-Id: Ia0dcd3d302e0f04adbda4f3abf75a0ca7dd20dee

/usr/bin/python may be python 2 on Focal, which causes problems with
Ansible on the control host. By installing the python-is-python3 package
we ensure that the correct interpreter is used.

This change updates the installation documentation and development
environment scripts.

Story: 2004960
Task: 42579

Change-Id: Ie94099075bae3c491f9cf830c38e6cfc8af605a6

* add 'bridge_type: linuxbridge' to tenks configuration to avoid
  dependency on OVS
* extend seed development environment testing to include overcloud
  provisioning and deployment
* remove seed hypervisor and seed VM environments. These are very
  stale, and largely replaced by a-universe-from-nothing. Add a link to
  that workshop on the same page

Change-Id: I9928e5912e6770bdcc1d5d0884d2f101c16ee6a9

All instances of 'kayobe control host bootstrap' in the development
scripts use a helper function, except for during seed_hypervisor_deploy.
The helper adds a retry mechanism to combat flakiness often seen during
Ansible Galaxy installs.

This change fixes the issue.

TrivialFix

Change-Id: I954cb604a18874744b3673ebf2e2c29caa18ce8f

A bug has been introduced to the which package in CentOS Stream 8 which
causes it to fail when used with the following bash options:

set -u
set -o pipefail

Then, when running which we see the following output:

environment: line 1: _declare: unbound variable

As found by Pierre, this seems to be caused by the implementation of
which as a bash function which references an unbound variable
(_declare). It's tracked in Fedora by
https://bugzilla.redhat.com/show_bug.cgi?id=1944877#

.

This change works around the issue by using the /usr/bin/which binary.

Co-Authored-By: Pierre Riteau <pierre@stackhpc.com>

Change-Id: I468d4e0460c13791b9f01d5854ef45472528c6fe
Story: 2008795
Task: 42215

We still see flakiness when downloading content from Ansible Galaxy,
often HTTP 520. This change increases the retries from 3 to 10, and adds
a 5 second delay between attempts.

Change-Id: I0c46e5fcc6979027dc6f1bc5cc49e923a205f654
Related: https://github.com/ansible/galaxy/issues/2429

The 'openstack server show <server> -f value -c addresses' command
previously had output like this:

    <network name>=<IP>

Now it shows a Python output like this:

    {'<network name>': ['IP']}

This broke the parsing of the command output when determining which IP
address to use to access a bare metal instance via SSH.

This change fixes the issue by querying the server's port in Neutron,
and using the fixed IP address.

Change-Id: I55b5f185fb7136d3c6fa565aa46598f21c94eb43

* Use source images
* Need to specify bash for &> syntax

Issues worked around:

* Manually configuring bridge via ip commands makes ifup fail to bring
  up the link. Adds a kayobe-network-bootstrap Zuul CI role that adds
  persistent configuration for the all-in-one network.

* bridge not active after interfaces role bounce. Added a pause, similar
  to https://github.com/michaelrigart/ansible-role-interfaces/pull/31

* fails installing docker python module for kolla user. WARNING: The
  repository located at mirror-int.ord.rax.opendev.org is not a trusted
  or secure host and is being ignored ERROR: No matching distribution
  found for docker===4.4.0 Adding trusted host for PyPI mirror.

* Tenks fails to create block devices - missing qemu-img (in qemu-utils)

* Tenks qemu emulator is different on Ubuntu

Remaining issues:

* Bare metal testing is unreliable on Ubuntu - some jobs see IPMI
  failures such as the following:

    ipmitool chassis bootdev pxe

    Error setting Chassis Boot Parameter 5\nError setting Chassis Boot
    Parameter 0\n

  Bare metal testing is disabled on Ubuntu for now.

Depends-On: https://review.opendev.org/766984
Depends-On: https://review.opendev.org/766958

Story: 2004960
Task: 29393

Change-Id: I1985efae7c18f55c3ff7c27c17d6242523904f3e

This partially reverts commit 47bbb96b
which triggered a retry on vexxhost clouds.

The issue was introduced in Ie8fd965165e8d347d27528a2c16d0647e412ccdc,
which applied some fixes for CentOS 8.3, and inadvertently removed
the Tenks variable that forces the use of qemu for 'bare metal' VMs.
This lead to autodetection of KVM, which does not work well when nested
in all CI cloud providers.

This change fixes the issue by forcing the use of qemu for the overcloud
once more. It also adds a similar option for the seed VM job.

Change-Id: I6bc8da2b8da903e09b97df8cd95c68a562c11db9

Also increase attempts to 12, in line with Kolla Ansible CI.

Change-Id: I81cabf27f44af3c8135efe8e427db1ffee5f0091

This requires stackhpc.os-images v1.10.0 or newer, for compatibility
with CentOS 8 when SELinux is enabled: we disable SELinux, but without
rebooting it stays enabled.

This Ansible role was updated to v1.10.2 in master and stable/victoria
by I5efdbd52556721914fe69d7c6ba454b2c721b643, for another reason.
Remember to bump the requirement when backporting to earlier releases.

It also needs changes in the way we interact with Bifrost to avoid using
the env-vars file which has been removed. This is implemented by change
I25078e69acdf41a4ef9957f99fe5047de54b778d.

Finally, it requires building seed deployment images only after
deploying Bifrost, because the task copying images onto the seed expects
/etc/kolla/bifrost to exist.

We also copy log files to identify issues when the job fails.

Change-Id: I4719b4d397c01b35c78cb84c6d686dd27742d1c0

* Bump stackhpc.libvirt-host to v1.7.1. On seed-hypervisors installed
  using CentOS 8.2 or earlier, interaction with libvirt may fail due to
  libgcrypt being incompatible. See
  https://github.com/stackhpc/ansible-role-libvirt-host/issues/42

* Bump MichaelRigart.interfaces to v1.9.2. The CentOS 8.3 cloud image
  includes an ifcfg-ens3-1 file. See
  https://github.com/michaelrigart/ansible-role-interfaces/pull/93

* Previously a second libvirt daemon was installed by Tenks on the host,
  however changes in libvirt 6.0.0 to separate libvirtd into multiple
  daemons do not allow for customisation of the PID files used by the
  new daemons. This leads to a conflict between the container and host
  daemons. Update the Tenks config to use the containerised Nova libvirt
  daemon. This depends on a change to the stackhpc.libvirt-host role:
  https://github.com/stackhpc/ansible-role-libvirt-host/pull/44

* Not CentOS 8.3 related, but tox jobs are now failing on python
  dependencies. Remove upper limits from docker and paramiko.

* Not CentOS 8.3 related, but Bifrost has enabled authentication by
  default. We are not ready to support this, so override it.

Story: 2008429
Task: 41378

Change-Id: Ie8fd965165e8d347d27528a2c16d0647e412ccdc

While we always test baremetal compute in CI, development environments
may not. Given that Ironic is now disabled by default, we should make
this work out of the box.

Story: 2008207
Task: 41003

Change-Id: Id3128380f5ff74d24265f6b2132c6d7992bf00ba

Change-Id: I5fc49fb734d0fe94f5f75c66eb4c1a935774ef30