The iscsi deploy interface was removed from Ironic [1]. Switch to the
direct deploy interface instead, which is the default.

[1] https://review.opendev.org/c/openstack/ironic/+/789382

Depends-On: https://review.opendev.org/c/openstack/kolla-ansible/+/801267
Change-Id: Ia0dcd3d302e0f04adbda4f3abf75a0ca7dd20dee

/usr/bin/python may be python 2 on Focal, which causes problems with
Ansible on the control host. By installing the python-is-python3 package
we ensure that the correct interpreter is used.

This change updates the installation documentation and development
environment scripts.

Story: 2004960
Task: 42579

Change-Id: Ie94099075bae3c491f9cf830c38e6cfc8af605a6

All instances of 'kayobe control host bootstrap' in the development
scripts use a helper function, except for during seed_hypervisor_deploy.
The helper adds a retry mechanism to combat flakiness often seen during
Ansible Galaxy installs.

This change fixes the issue.

TrivialFix

Change-Id: I954cb604a18874744b3673ebf2e2c29caa18ce8f

A bug has been introduced to the which package in CentOS Stream 8 which
causes it to fail when used with the following bash options:

set -u
set -o pipefail

Then, when running which we see the following output:

environment: line 1: _declare: unbound variable

As found by Pierre, this seems to be caused by the implementation of
which as a bash function which references an unbound variable
(_declare). It's tracked in Fedora by
https://bugzilla.redhat.com/show_bug.cgi?id=1944877#

.

This change works around the issue by using the /usr/bin/which binary.

Co-Authored-By: Pierre Riteau <pierre@stackhpc.com>

Change-Id: I468d4e0460c13791b9f01d5854ef45472528c6fe
Story: 2008795
Task: 42215

We still see flakiness when downloading content from Ansible Galaxy,
often HTTP 520. This change increases the retries from 3 to 10, and adds
a 5 second delay between attempts.

Change-Id: I0c46e5fcc6979027dc6f1bc5cc49e923a205f654
Related: https://github.com/ansible/galaxy/issues/2429

The 'openstack server show <server> -f value -c addresses' command
previously had output like this:

    <network name>=<IP>

Now it shows a Python output like this:

    {'<network name>': ['IP']}

This broke the parsing of the command output when determining which IP
address to use to access a bare metal instance via SSH.

This change fixes the issue by querying the server's port in Neutron,
and using the fixed IP address.

Change-Id: I55b5f185fb7136d3c6fa565aa46598f21c94eb43

* Use source images
* Need to specify bash for &> syntax

Issues worked around:

* Manually configuring bridge via ip commands makes ifup fail to bring
  up the link. Adds a kayobe-network-bootstrap Zuul CI role that adds
  persistent configuration for the all-in-one network.

* bridge not active after interfaces role bounce. Added a pause, similar
  to https://github.com/michaelrigart/ansible-role-interfaces/pull/31

* fails installing docker python module for kolla user. WARNING: The
  repository located at mirror-int.ord.rax.opendev.org is not a trusted
  or secure host and is being ignored ERROR: No matching distribution
  found for docker===4.4.0 Adding trusted host for PyPI mirror.

* Tenks fails to create block devices - missing qemu-img (in qemu-utils)

* Tenks qemu emulator is different on Ubuntu

Remaining issues:

* Bare metal testing is unreliable on Ubuntu - some jobs see IPMI
  failures such as the following:

    ipmitool chassis bootdev pxe

    Error setting Chassis Boot Parameter 5\nError setting Chassis Boot
    Parameter 0\n

  Bare metal testing is disabled on Ubuntu for now.

Depends-On: https://review.opendev.org/766984
Depends-On: https://review.opendev.org/766958

Story: 2004960
Task: 29393

Change-Id: I1985efae7c18f55c3ff7c27c17d6242523904f3e

Also increase attempts to 12, in line with Kolla Ansible CI.

Change-Id: I81cabf27f44af3c8135efe8e427db1ffee5f0091

This requires stackhpc.os-images v1.10.0 or newer, for compatibility
with CentOS 8 when SELinux is enabled: we disable SELinux, but without
rebooting it stays enabled.

This Ansible role was updated to v1.10.2 in master and stable/victoria
by I5efdbd52556721914fe69d7c6ba454b2c721b643, for another reason.
Remember to bump the requirement when backporting to earlier releases.

It also needs changes in the way we interact with Bifrost to avoid using
the env-vars file which has been removed. This is implemented by change
I25078e69acdf41a4ef9957f99fe5047de54b778d.

Finally, it requires building seed deployment images only after
deploying Bifrost, because the task copying images onto the seed expects
/etc/kolla/bifrost to exist.

We also copy log files to identify issues when the job fails.

Change-Id: I4719b4d397c01b35c78cb84c6d686dd27742d1c0

While we always test baremetal compute in CI, development environments
may not. Given that Ironic is now disabled by default, we should make
this work out of the box.

Story: 2008207
Task: 41003

Change-Id: Id3128380f5ff74d24265f6b2132c6d7992bf00ba

Change-Id: I5fc49fb734d0fe94f5f75c66eb4c1a935774ef30

Adds the kayobe-seed-vm-centos8 CI job to configure the Zuul VM as a
seed hypervisor, and use nested virt to provision a seed VM.  This
ensures that the seed hypervisor code paths are tested.

The job uses a Cirros image for the seed VM rather than the usual CentOS
cloud image. This is to reduce bandwidth required to download the image.
It does mean that the resulting seed VM cannot be used as a seed, but
nested virt would make this slow and unreliable anyway. Cirros does not
load cdrom drivers by default, so we add the configdrive as a disk
rather than a cdrom device.

Depends-On: https://review.opendev.org/617161

Change-Id: I2268a1ddf9a2870c713f32a40689e1686365aabd
Story: 2001655
Task: 6683

This ensures we are using the appropriate IPA images in the upgraded
environment.

Change-Id: I4a72d9ae49ad41716522c3074c16d8ca23c3ff94

* Always use Python 3
* Drop code paths for CentOS 7
* Drop support for Yum
* Remove support for host NTP daemon, always use chrony
* Switch references from 'yum_install_epel' to 'dnf_install_epel'
* Remove overcloud host image workaround for tagged VLAN admin network
* Remove the kayobe.utils.yum_install function, which is unused

Change-Id: I368f6edafed9779658798fc342116b4c1b3ffd48
Story: 2006574
Task: 39481

Backport: train, stein, rocky

This fixes issues seen with a-universe-from-nothing using stable/train.

Change-Id: Ib477de5f3af2e4c182d0c2999c274dbb5553531c
Story: 2007572
Task: 39469

Tests various non-default configuration:

* Custom users
* Network interfaces, VLANs, bridges, bonds
* Software RAID
* LVM & docker devicemapper
* timezone
* Package mirrors
* yum-cron / DNF automatic

This improved test coverage allows us to be more confident about these
features working on CentOS 8.

Change-Id: I36148e4356deb7d5ec00d8d3ebeb2d3932ff4f94
Story: 2006574
Task: 38938

Detect current branch from .gitreview and use upper constraints to
install python-openstackclient, to guarantee compatibility with the
Python version in use.

Change-Id: Ie44508fe3d3b08190afa5a43748e43548a63ff82

Co-Authored-By: Mark Goddard <mark@stackhpc.com>

Change-Id: I2a7a82d7f576739c5516a0072f953712ffa5c233
Story: 2004959
Task: 29392

Add testing of the dataplane to the overcloud jobs. To support both
baremetal provisioning and VMs with VXLAN tenant networks, we use the
provision-net as our external network to which floating IPs are
attached.

We depend on a backport of parts of this patch to allow testing of SSH
connectivity after upgrades.

Depends-On: https://review.opendev.org/708915/
Depends-On: https://review.opendev.org/709145/

Change-Id: I6316e8959cff987e4e97280889e1038a9537ed32

In case of failures in kayobe-overcloud-centos, the error message fails
with:
kayobe/dev/functions: line 569: LOGDIR: unbound variable

Example:
https://zuul.opendev.org/t/openstack/build/ce1fadc3ee6d4842a599da57a670cc18

This can be reproduced with:
set -eu

if [[ -n ${LOGDIR} ]]; then
    echo "LOGDIR set"
else
    echo "else"
fi

Fix the error reporting with assigning an empty string to LOGDIR by
default.

Change-Id: Ieef73950f89e4dfb727ddc59ef2750d9b81f3c58

We no longer need to cd to the kayobe repo path, since all tested
versions of kayobe support pip installation.

Change-Id: Id7b574cbc6bb5195eee16cfc8443075712ce3254

This is to resolve the following issue in CI:

    c/_cffi_backend.c:15:17: fatal error: ffi.h: No such file or directory
     #include <ffi.h>
                     ^
    compilation terminated.
    error: command 'gcc' failed with exit status 1
    ----------------------------------------
ERROR: Command errored out with exit status 1: /home/zuul/kayobe-venv/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-PV3WhJ/cffi/setup.py'"'"'; __file__='"'"'/tmp/pip-install-PV3WhJ/cffi/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-ZvlZVY/install-record.txt --single-version-externally-managed --compile --install-headers /home/zuul/kayobe-venv/include/site/python2.7/cffi Check the logs for full command output.

Change-Id: I1bd724be7dc77058870f37cb1c9404472fa466ca
see: https://zuul.opendev.org/t/openstack/build/c20a316a699b4073abf75960634ebfcd

Docker CE has added a default DROP policy to the FORWARD chain.  When
nova-compute runs on the controller, kolla ansible sets the
net.bridge.bridge-nf-call-iptables sysctl to 1, which causes iptables to
process frames forwarded across bridges.

Currently, the kayobe-overcloud-centos job is failing quite frequently
with timeouts when deploying bare metal compute. Experimentation with
iptables hasn't revealed why this only happens sometimes, or exactly
what traffic is being blocked, but opening up the firewall does seem to
fix the issue. We won't see this in production since control and compute
services are on separate hosts.

This change updates the iptables configuration used in CI to forward all
frames on the bridge, breth1.

Change-Id: If96437b73b9b5c58600ba1b004f53ee0c1f14398
Story: 2006534
Task: 36590

This network conflicts with the ironic provision-net since both are flat
networks on physnet1. We don't need the external network in our CI, so
don't create it.

Change-Id: Id17d4ea00ceb684b34bf12e911758d504c520de0
Depends-On: https://review.opendev.org/#/c/670129/

Galaxy install often fails in CI, scuppering an otherwise good run.

Change-Id: I3d02afe33cdf32b1d285bd4bdc4a8074883113ca

Allows users to explicitly specify which type of tenks
deployment they wish to create / destroy.
Preserves existing behaviour with defaults.
Modifies Zuul tests to use new tenks-deploy entrypoints.

Change-Id: I9aafed8481fd7564c0fc0abe5f6b21eceb824d75

This is supported in kolla-ansible via the ironic_inspector_pxe_filter
variable, which can be added to globals.yml. The default value for that
variable changed in the Stein release from 'iptables' to 'dnsmasq',
since the iptables filter does not work with Docker CE [1].

This change removes the inspector_manage_firewall variable.

This change also adds an iptables rule in CI tests to allow DHCP packets
to be forwarded, to ensure bare metal servers can be deployed.

[1] https://bugs.launchpad.net/kolla-ansible/+bug/1823044

Depends-On: https://review.openstack.org/649673
Change-Id: Idac6777b4d97fbd17698fc2086ceb068d7b2e326
Related-Bug: #1823044

This script is used by developers to activate the kayobe virtual
environment and source the configuration's kayobe-env file. A cd to /tmp
is an unexpected outcome of running the script.

To test the location-independent installation, remove the chdir from the
zuul job tasks that execute kayobe commands.

Change-Id: I59194952901fa648382489f48dc7aafb03d3a682
Story: 2004252
Task: 29347

This adds support for deploying a virtualised control plane via Tenks, using
the Kayobe development scripts tenks-deploy.sh and tenks-teardown.sh.

Change-Id: I752455af9eb44cdb0f9921fd0c876fc2dfb50a5c

Currently in the upgrade job we are seeing the OOM killer kick in during
the 'overcloud service configuration save' command. Ansible is quite
inefficient when copying large files around, so excluding the large IPA
images should relieve some memory pressure.

Change-Id: I3a230b0a699154606ca8faa00a85d45ae815c599
Story: 2004704
Task: 28733

This adds the ansible playbooks required by kayobe to the manifest by
using the data_files option in setuptools. When using pip to install
kayobe into a virtualenv, these files will be placed in
<venv>/kayobe/share/.

In an editable install, e.g using `pip install -e .`, data_files are not
installed into the virtualenv. Instead, we must follow the egg-link file
to find out the actual location.

Story: 2004252
Task: 27787
Change-Id: Ibef040eceb547476007f83c0d5dcdb2bc6986d1e

Adds the kayobe-seed-upgrade-centos job, which performs an upgrade of
the seed services from the previous release to the current release.

Change-Id: Ia3eb39cf81cb3618fd94c4456bd576b52098c946
Story: 2004308
Task: 27873

tenks-deploy.sh deploys a minimal virtualized baremetal test cluster
locally. It also adds it to the overcloud-base CI job. To make the new
CI job work, we need to configure the firewall on the test machine to
allow the baremetal machines to communicate with the openstack services.

Change-Id: I7487a2606cf0bac71c5c63db41b2b518a6f6398b
Depends-On: https://review.openstack.org/#/c/615939
Depends-On: https://review.openstack.org/#/c/618003
Story: 2004297
Task: 27850

Use stable/rocky branch of:

* Kolla
* Kolla ansible
* Bifrost
* IPA
* OpenStack services
* Requirements

Also updates Kolla Ansible inventory template.

The seed deploy job has been made non-voting and non-gating, because we
are waiting for bifrost change https://review.openstack.org/#/c/618740
to merge, be released, and for the kolla bifrost image to use the new
package.

Change-Id: Id5e7fdbd196f96e1e75ffc68bc93aab18fa38aa7
Story: 2001864
Task: 27798
Depends-On: I58e4f951d4a3dd89e0784fd82d8a62dbba374f79

There is currently no coverage of upgrades in CI, which leaves us open
to regression in this infrequently tested but crucial area. This change
adds the required scripts and Zuul configuration.

A control plane is first deployed using the Kayobe stable/pike branch
and associated default configuration. The control plane is tested by
deploying then deleting a server instance. An upgrade to Queens is
performed, using the Kayobe master branch, or code in review if
applicable. The upgraded control plane is tested by deploying then
deleting a second server instance.

A workaround was required to restart the nova_compute service after the
upgrade, since the SIGHUP sent to it by Kolla Ansible during upgrade
appeared to be putting it into a degraded state.

A future improvement to this test could be to leave a server instance
running during the upgrade.

Change-Id: I0e595524e39d1131fe3ec6aaf2aeec3ff3d6a536
Story: 2003472
Task: 24732

Previously we were not checking the status of created VMs in CI, meaning
that VM creation failures would be silently ignored. This change checks
the status, requiring it to be ACTIVE for a pass.

Change-Id: Ia559c81b4944c2c6c7dedfca0c6c570db390704b

Ensure all references to the Ansible control host are worded as such, to
ensure consistency and avoid potential confusion with the OpenStack
controllers.

Change-Id: Id92e537ccbfdd55287b8eae296f649640c70ce17

We use the demo script from kolla ansible, init-runonce, to create
resources in the control plane to make it ready for booting a VM. We
then create a nova server, and wait for it to become active. We do not
currently test that the VM boots successfully by accessing it via SSH.

Change-Id: I61be554366565decd9f4ff7805a3969aa37da4b9

Adds a new CI job, kayobe-seed-centos, that deploys a single VM as a
seed host. The VM provided by Zuul is configured as the seed, rather
than a seed hypervisor that provisions a seed VM using nested virt.

This approach should be more reliable, faster, and easier to get
working, at the expense of not testing the seed hypervisor code paths.

Currently the seed services are deployed, but not tested. A future
change could test discovery and provisioning of an overcloud host using
the seed.

Change-Id: Iefe16215af6f1a9fb8ec78094cd6bd76a82a040a
Story: 2001655
Task: 6683

Due to how bash expands tildes, the `~` is treated literally here and
results in a folder in the Vagrant user's home directory called `~`.

This commit switches to using the `$HOME` environment variable instead.