This is required to be able to install tenks. Otherwise, we try to
install Jinja2 3.1.2 which requires Python 3.7 or newer.

Change-Id: Ie497b191b6de8bc818dc4a2a12f7129a02d0fd00

Requirements upper constraints bumped python-novaclient to version
18.0.0 [1], which requires Python 3.8 [2]. This results in failures when
installing python-openstackclient on CentOS and Rocky with Python 3.6.

    ERROR: Cannot install python-openstackclient==5.8.0 because these package versions have conflicting dependencies.

    The conflict is caused by:
        python-openstackclient 5.8.0 depends on python-novaclient>=17.0.0
        The user requested (constraint) python-novaclient===18.0.0

Work around this issue by using yoga upper constraints until we upgrade
to CentOS Stream 9 and Rocky Linux 9.

This also fixes another issue seen on Ubuntu where image uploads to
Glance through Ansible fail with a 400 Bad Request error. This is caused
by the bump of openstacksdk to version 0.99.0 and will be fixed by a new
release of ansible-collections-openstack.

[1] https://review.opendev.org/c/openstack/requirements/+/842808
[2] https://review.opendev.org/c/openstack/python-novaclient/+/838944

Change-Id: I40c6b898963c2218d41d37bd73d40ce8dcf22b87

Enable the Ironic ipxe boot interface by default, following a similar
change in Ironic [1].

Drop the kolla_enable_ironic_ipxe flag, following a similar change in
Kolla Ansible [2]. Both PXE and iPXE are now enabled by default. Users
may revert to using PXE for ironic inspector's dnsmasq, by setting
ironic_dnsmasq_serve_ipxe to false in etc/kayobe/kolla/globals.yml.

[1] https://review.opendev.org/c/openstack/ironic/+/816824
[2] https://review.opendev.org/c/openstack/kolla-ansible/+/834512/

Depends-On: https://review.opendev.org/c/openstack/kolla-ansible/+/832159
Depends-On: https://review.opendev.org/c/openstack/kolla-ansible/+/834511
Depends-On: https://review.opendev.org/c/openstack/kolla-ansible/+/837069

Change-Id: Ifb80bd15a20c9cfb8fbc6e0f6ac23baae631a18e

In some cases it may be desirable to run libvirt daemon on the host. For
example, when mixing host and container OS distributions.

This change makes it possible to disable the nova_libvirt container, by
setting kolla_enable_nova_libvirt_container to false.

The stackhpc.libvirt-host role is used in order to install and configure
a libvirt daemon on compute hosts when
kolla_enable_nova_libvirt_container is false.

Depends-On: https://review.opendev.org/c/openstack/kolla-ansible/+/825357
Depends-On: https://review.opendev.org/c/openstack/kayobe-config-dev/+/829225
Depends-On: https://github.com/stackhpc/ansible-role-libvirt-host/pull/51

Story: 2009858
Task: 44495

Change-Id: I73fef63fb886a9d543d2f4231fb009523495edb3

Change-Id: If7d6e58b19f98ccb7cc4c209e458cb6f4f4765ad

Previously we were using the zuul user in the TLS jobs. This was due to
a permissions issue when accessing the CA certificate in kayobe-config
in the zuul user's home directory.

This change reverts to the default of using the stack user for the TLS
jobs. In order to make this work, the generated CA cert chain is added
to the trust store.

Change-Id: I875f8976df75dee68ba00842fe624c29cc1b123c

Set the Ironic boot mode to legacy BIOS explicitly in Tenks config in
anticipation of an upcoming change to the default boot mode.

Override the boot mode to UEFI in the overcloud TLS job to improve
coverage. This requires enabling iPXE booting.

Depends-On: https://review.opendev.org/c/openstack/tenks/+/827479/

Change-Id: Id1b4e9775c834b8b97e086241ee8b247977225a2

As a first step towards supporting multiple overcloud disk images, this
change introduces a new command to build a disk image directly with DIB:
`kayobe overcloud host image build`.

It also disables building a root disk image during Bifrost bootstrap if
overcloud_dib_build_host_images is set to true.

Change-Id: I93d242889e225b4e60254f6b9cc5eeb457294ac8
Story: 2002098
Task: 41693

Depends-On: https://review.opendev.org/c/openstack/kayobe/+/812670
Depends-On: https://review.opendev.org/c/openstack/kayobe-config-dev/+/805239
Depends-On: https://review.opendev.org/c/openstack/kayobe/+/813212

Change-Id: I7dbd643fec72223bfda20cbbc37809ae78073804

Arguments are passed through to kayobe-env in kayobe-config, which
allows to set the Kayobe environment.

Change-Id: I4c72e32e5379237340284a09874b0c500e41ad0f

By default, Ansible injects a variable for every fact, prefixed with
ansible_. This can result in a large number of variables for each host,
which at scale can incur a performance penalty. Ansible provides a
configuration option [0] that can be set to False to prevent this
injection of facts. In this case, facts should be referenced via
ansible_facts.<fact>.

This change updates all references to Ansible facts within Kayobe
from using individual fact variables to using the items in the
ansible_facts dictionary. This allows users to disable fact variable
injection in their Ansible configuration, which may provide some
performance improvement.

This change disables fact variable injection in the ansible
configuration used in CI, to catch any attempts to use the injected
variables.

[0] https://docs.ansible.com/ansible/latest/reference_appendices/config.html#inject-facts-as-vars

Story: 2007993
Task: 42464
Depends-On: https://review.opendev.org/c/openstack/kolla-ansible/+/791276

Change-Id: I14db53ed6e57d37bbd28dd5819e432e3fe6628b2

The iscsi deploy interface was removed from Ironic [1]. Switch to the
direct deploy interface instead, which is the default.

[1] https://review.opendev.org/c/openstack/ironic/+/789382

Depends-On: https://review.opendev.org/c/openstack/kolla-ansible/+/801267
Change-Id: Ia0dcd3d302e0f04adbda4f3abf75a0ca7dd20dee

/usr/bin/python may be python 2 on Focal, which causes problems with
Ansible on the control host. By installing the python-is-python3 package
we ensure that the correct interpreter is used.

This change updates the installation documentation and development
environment scripts.

Story: 2004960
Task: 42579

Change-Id: Ie94099075bae3c491f9cf830c38e6cfc8af605a6

* add 'bridge_type: linuxbridge' to tenks configuration to avoid
  dependency on OVS
* extend seed development environment testing to include overcloud
  provisioning and deployment
* remove seed hypervisor and seed VM environments. These are very
  stale, and largely replaced by a-universe-from-nothing. Add a link to
  that workshop on the same page

Change-Id: I9928e5912e6770bdcc1d5d0884d2f101c16ee6a9

All instances of 'kayobe control host bootstrap' in the development
scripts use a helper function, except for during seed_hypervisor_deploy.
The helper adds a retry mechanism to combat flakiness often seen during
Ansible Galaxy installs.

This change fixes the issue.

TrivialFix

Change-Id: I954cb604a18874744b3673ebf2e2c29caa18ce8f

A bug has been introduced to the which package in CentOS Stream 8 which
causes it to fail when used with the following bash options:

set -u
set -o pipefail

Then, when running which we see the following output:

environment: line 1: _declare: unbound variable

As found by Pierre, this seems to be caused by the implementation of
which as a bash function which references an unbound variable
(_declare). It's tracked in Fedora by
https://bugzilla.redhat.com/show_bug.cgi?id=1944877#

.

This change works around the issue by using the /usr/bin/which binary.

Co-Authored-By: Pierre Riteau <pierre@stackhpc.com>

Change-Id: I468d4e0460c13791b9f01d5854ef45472528c6fe
Story: 2008795
Task: 42215

We still see flakiness when downloading content from Ansible Galaxy,
often HTTP 520. This change increases the retries from 3 to 10, and adds
a 5 second delay between attempts.

Change-Id: I0c46e5fcc6979027dc6f1bc5cc49e923a205f654
Related: https://github.com/ansible/galaxy/issues/2429

The 'openstack server show <server> -f value -c addresses' command
previously had output like this:

    <network name>=<IP>

Now it shows a Python output like this:

    {'<network name>': ['IP']}

This broke the parsing of the command output when determining which IP
address to use to access a bare metal instance via SSH.

This change fixes the issue by querying the server's port in Neutron,
and using the fixed IP address.

Change-Id: I55b5f185fb7136d3c6fa565aa46598f21c94eb43

* Use source images
* Need to specify bash for &> syntax

Issues worked around:

* Manually configuring bridge via ip commands makes ifup fail to bring
  up the link. Adds a kayobe-network-bootstrap Zuul CI role that adds
  persistent configuration for the all-in-one network.

* bridge not active after interfaces role bounce. Added a pause, similar
  to https://github.com/michaelrigart/ansible-role-interfaces/pull/31

* fails installing docker python module for kolla user. WARNING: The
  repository located at mirror-int.ord.rax.opendev.org is not a trusted
  or secure host and is being ignored ERROR: No matching distribution
  found for docker===4.4.0 Adding trusted host for PyPI mirror.

* Tenks fails to create block devices - missing qemu-img (in qemu-utils)

* Tenks qemu emulator is different on Ubuntu

Remaining issues:

* Bare metal testing is unreliable on Ubuntu - some jobs see IPMI
  failures such as the following:

    ipmitool chassis bootdev pxe

    Error setting Chassis Boot Parameter 5\nError setting Chassis Boot
    Parameter 0\n

  Bare metal testing is disabled on Ubuntu for now.

Depends-On: https://review.opendev.org/766984
Depends-On: https://review.opendev.org/766958

Story: 2004960
Task: 29393

Change-Id: I1985efae7c18f55c3ff7c27c17d6242523904f3e

This partially reverts commit 47bbb96b
which triggered a retry on vexxhost clouds.

The issue was introduced in Ie8fd965165e8d347d27528a2c16d0647e412ccdc,
which applied some fixes for CentOS 8.3, and inadvertently removed
the Tenks variable that forces the use of qemu for 'bare metal' VMs.
This lead to autodetection of KVM, which does not work well when nested
in all CI cloud providers.

This change fixes the issue by forcing the use of qemu for the overcloud
once more. It also adds a similar option for the seed VM job.

Change-Id: I6bc8da2b8da903e09b97df8cd95c68a562c11db9

Also increase attempts to 12, in line with Kolla Ansible CI.

Change-Id: I81cabf27f44af3c8135efe8e427db1ffee5f0091

This requires stackhpc.os-images v1.10.0 or newer, for compatibility
with CentOS 8 when SELinux is enabled: we disable SELinux, but without
rebooting it stays enabled.

This Ansible role was updated to v1.10.2 in master and stable/victoria
by I5efdbd52556721914fe69d7c6ba454b2c721b643, for another reason.
Remember to bump the requirement when backporting to earlier releases.

It also needs changes in the way we interact with Bifrost to avoid using
the env-vars file which has been removed. This is implemented by change
I25078e69acdf41a4ef9957f99fe5047de54b778d.

Finally, it requires building seed deployment images only after
deploying Bifrost, because the task copying images onto the seed expects
/etc/kolla/bifrost to exist.

We also copy log files to identify issues when the job fails.

Change-Id: I4719b4d397c01b35c78cb84c6d686dd27742d1c0

* Bump stackhpc.libvirt-host to v1.7.1. On seed-hypervisors installed
  using CentOS 8.2 or earlier, interaction with libvirt may fail due to
  libgcrypt being incompatible. See
  https://github.com/stackhpc/ansible-role-libvirt-host/issues/42

* Bump MichaelRigart.interfaces to v1.9.2. The CentOS 8.3 cloud image
  includes an ifcfg-ens3-1 file. See
  https://github.com/michaelrigart/ansible-role-interfaces/pull/93

* Previously a second libvirt daemon was installed by Tenks on the host,
  however changes in libvirt 6.0.0 to separate libvirtd into multiple
  daemons do not allow for customisation of the PID files used by the
  new daemons. This leads to a conflict between the container and host
  daemons. Update the Tenks config to use the containerised Nova libvirt
  daemon. This depends on a change to the stackhpc.libvirt-host role:
  https://github.com/stackhpc/ansible-role-libvirt-host/pull/44

* Not CentOS 8.3 related, but tox jobs are now failing on python
  dependencies. Remove upper limits from docker and paramiko.

* Not CentOS 8.3 related, but Bifrost has enabled authentication by
  default. We are not ready to support this, so override it.

Story: 2008429
Task: 41378

Change-Id: Ie8fd965165e8d347d27528a2c16d0647e412ccdc

While we always test baremetal compute in CI, development environments
may not. Given that Ironic is now disabled by default, we should make
this work out of the box.

Story: 2008207
Task: 41003

Change-Id: Id3128380f5ff74d24265f6b2132c6d7992bf00ba

Change-Id: I5fc49fb734d0fe94f5f75c66eb4c1a935774ef30

Adds the kayobe-seed-vm-centos8 CI job to configure the Zuul VM as a
seed hypervisor, and use nested virt to provision a seed VM.  This
ensures that the seed hypervisor code paths are tested.

The job uses a Cirros image for the seed VM rather than the usual CentOS
cloud image. This is to reduce bandwidth required to download the image.
It does mean that the resulting seed VM cannot be used as a seed, but
nested virt would make this slow and unreliable anyway. Cirros does not
load cdrom drivers by default, so we add the configdrive as a disk
rather than a cdrom device.

Depends-On: https://review.opendev.org/617161

Change-Id: I2268a1ddf9a2870c713f32a40689e1686365aabd
Story: 2001655
Task: 6683

This ensures we are using the appropriate IPA images in the upgraded
environment.

Change-Id: I4a72d9ae49ad41716522c3074c16d8ca23c3ff94

Switches to use the IPA builder project for building IPA images.

Switches the IPA images used by default to CentOS 8 based image.

Changes the file extension of the IPA kernel image from vmlinuz to
kernel.

Story: 2007070
Task: 37953

Change-Id: I82fc455f41f48dacb453e135870dd776895d7c99
Story: 2006574
Task: 39485

* Always use Python 3
* Drop code paths for CentOS 7
* Drop support for Yum
* Remove support for host NTP daemon, always use chrony
* Switch references from 'yum_install_epel' to 'dnf_install_epel'
* Remove overcloud host image workaround for tagged VLAN admin network
* Remove the kayobe.utils.yum_install function, which is unused

Change-Id: I368f6edafed9779658798fc342116b4c1b3ffd48
Story: 2006574
Task: 39481

Backport: train, stein, rocky

This fixes issues seen with a-universe-from-nothing using stable/train.

Change-Id: Ib477de5f3af2e4c182d0c2999c274dbb5553531c
Story: 2007572
Task: 39469

Tests various non-default configuration:

* Custom users
* Network interfaces, VLANs, bridges, bonds
* Software RAID
* LVM & docker devicemapper
* timezone
* Package mirrors
* yum-cron / DNF automatic

This improved test coverage allows us to be more confident about these
features working on CentOS 8.

Change-Id: I36148e4356deb7d5ec00d8d3ebeb2d3932ff4f94
Story: 2006574
Task: 38938

Detect current branch from .gitreview and use upper constraints to
install python-openstackclient, to guarantee compatibility with the
Python version in use.

Change-Id: Ie44508fe3d3b08190afa5a43748e43548a63ff82

Co-Authored-By: Mark Goddard <mark@stackhpc.com>

Change-Id: I2a7a82d7f576739c5516a0072f953712ffa5c233
Story: 2004959
Task: 29392

Add testing of the dataplane to the overcloud jobs. To support both
baremetal provisioning and VMs with VXLAN tenant networks, we use the
provision-net as our external network to which floating IPs are
attached.

We depend on a backport of parts of this patch to allow testing of SSH
connectivity after upgrades.

Depends-On: https://review.opendev.org/708915/
Depends-On: https://review.opendev.org/709145/

Change-Id: I6316e8959cff987e4e97280889e1038a9537ed32

In case of failures in kayobe-overcloud-centos, the error message fails
with:
kayobe/dev/functions: line 569: LOGDIR: unbound variable

Example:
https://zuul.opendev.org/t/openstack/build/ce1fadc3ee6d4842a599da57a670cc18

This can be reproduced with:
set -eu

if [[ -n ${LOGDIR} ]]; then
    echo "LOGDIR set"
else
    echo "else"
fi

Fix the error reporting with assigning an empty string to LOGDIR by
default.

Change-Id: Ieef73950f89e4dfb727ddc59ef2750d9b81f3c58

We no longer need to cd to the kayobe repo path, since all tested
versions of kayobe support pip installation.

Change-Id: Id7b574cbc6bb5195eee16cfc8443075712ce3254

Not setting cpu-mode will reduce the number of TCG emulated features.
Hopefully this will make CI more reliable.

Change-Id: I24a8832c02db6ba019ab8f5c2b9d7216a9b7d213

This is to resolve the following issue in CI:

    c/_cffi_backend.c:15:17: fatal error: ffi.h: No such file or directory
     #include <ffi.h>
                     ^
    compilation terminated.
    error: command 'gcc' failed with exit status 1
    ----------------------------------------
ERROR: Command errored out with exit status 1: /home/zuul/kayobe-venv/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-PV3WhJ/cffi/setup.py'"'"'; __file__='"'"'/tmp/pip-install-PV3WhJ/cffi/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-ZvlZVY/install-record.txt --single-version-externally-managed --compile --install-headers /home/zuul/kayobe-venv/include/site/python2.7/cffi Check the logs for full command output.

Change-Id: I1bd724be7dc77058870f37cb1c9404472fa466ca
see: https://zuul.opendev.org/t/openstack/build/c20a316a699b4073abf75960634ebfcd

Docker CE has added a default DROP policy to the FORWARD chain.  When
nova-compute runs on the controller, kolla ansible sets the
net.bridge.bridge-nf-call-iptables sysctl to 1, which causes iptables to
process frames forwarded across bridges.

Currently, the kayobe-overcloud-centos job is failing quite frequently
with timeouts when deploying bare metal compute. Experimentation with
iptables hasn't revealed why this only happens sometimes, or exactly
what traffic is being blocked, but opening up the firewall does seem to
fix the issue. We won't see this in production since control and compute
services are on separate hosts.

This change updates the iptables configuration used in CI to forward all
frames on the bridge, breth1.

Change-Id: If96437b73b9b5c58600ba1b004f53ee0c1f14398
Story: 2006534
Task: 36590

This network conflicts with the ironic provision-net since both are flat
networks on physnet1. We don't need the external network in our CI, so
don't create it.

Change-Id: Id17d4ea00ceb684b34bf12e911758d504c520de0
Depends-On: https://review.opendev.org/#/c/670129/