Change-Id: Ie42ab7a0dc9dd1ed1925b3a17134b3770ae8ba98

When customising kolla_bifrost_inspector_extra_kernel_options, for
example to define which console to use, operators should note that
Kayobe defines extra kernel parameters that are important for
inspection, such as ipa-collect-lldp=1.

Change-Id: Id789dede2d1886ef5ec66ebc86968d6cf62fa2de

Kolla Ansible has recently updated the default Docker configuration to
stop using an insecure registry [1]. To avoid breaking existing Kayobe
deployments, automatically set docker_registry_insecure to true if we
deploy a registry without TLS.

[1] https://review.opendev.org/c/openstack/kolla-ansible/+/805449

Change-Id: Ifec7102812b5503cb02f207098192e99e7193d49

* Improve docs
* Fix up some comments

Change-Id: Iee05721bbe084f5580805cd82b12d065a2c61a1e

This is only supported on CentOS for now due to limitations of the
Ansible role used to configure tuned.

Change-Id: Ie07c5f467975f8da2f720e70c94cea6285981d72
Co-Authored-By: Pierre Riteau <pierre@stackhpc.com>
Story: 2007853
Task: 40155

Change-Id: Ic49de8d27da6604429e09fb0122eb64239cf58a8

Follow up to Id60e25e129e323f3c07e702bb81a11efc530fb3e, adds support for
firewalld configuration on Infra VMs.

Change-Id: Idd1ab982d4bca1cbdb0c4c6041cf3b6c17eae6cb

CentOS cloud images come with net.ifnames=0 on the kernel command line,
which disables consistent network device naming. This does not provide a
good experience on bare metal because NIC ordering can vary. This is
specific to cloud images: an ISO installation would use consistent
network device naming.

We now set net.ifnames=1 in the DIB default environment to use
consistent network device naming. The parameters `nofb nomodeset
gfxpayload=text` are also set to preserve defaults from DIB.

To restore the existing behaviour, set DIB_BOOTLOADER_DEFAULT_CMDLINE to
`nofb nomodeset gfxpayload=text net.ifnames=0` in the
kolla_bifrost_dib_env_vars_extra dictionary.

Change-Id: I20465eab4e0aec6620578a92d3bdbddcec0954df

This change allows you to define additional VMs to deploy
on the seed-hypervisor.

Co-authored-by: Piotr Parczewski <piotr@stackhpc.com>
Co-authored-by: Will Szumski <will@stackhpc.com>
Co-authored-by: Mark Goddard <mark@stackhpc.com>
Story: 2008741
Task: 42095
Change-Id: I8055fc5eb0a9edadcb35767303c659922f2d07ca

Adds support for configuring firewalld for CentOS hosts managed by
Kayobe.

* create zones
* set default zone
* set zone for interfaces
* define rules

Change-Id: Id60e25e129e323f3c07e702bb81a11efc530fb3e
Story: 2008991
Task: 42644

Source images get the most test coverage, so it makes sense to build and
deploy these by default.

Change-Id: I297b83985b09e888c5ee64c1a39f8a1dfcacc5c1

Ansible facts can have a large impact on the performance of the Ansible
control host. This patch introduces some control over which facts are
gathered (kayobe_ansible_setup_gather_subset) and which facts are stored
(kayobe_ansible_setup_filter). By default we do not change the default
values of these arguments to the setup module. The flexibility of these
arguments is limited, but they do provide enough for a large performance
improvement in a typical moderate to large OpenStack cloud.

In particular, the large complex dict fact for each interface has a
large effect, and on an OpenStack controller or hypervisor there may be
many virtual interfaces. We can use the kayobe_ansible_setup_filter
variable to help:

    kayobe_ansible_setup_filter: 'ansible_[!qt]*'

This causes Ansible to collect but not store facts matching that
pattern, which includes the virtual interface facts. Currently we are
not referencing other facts matching the pattern within Kayobe.
Note that including the 'ansible_' prefix causes meta facts module_setup
and gather_subset to be filtered, but this seems to be the only way to
get a good match on the interface facts. To work around this, we use
ansible_facts rather than module_setup to detect whether facts exist in
the cache.

The exact improvement will vary, but has been reported to be as large as
18x on systems with many virtual interfaces.

This change also introduces a new command to gather facts for Kayobe &
Kolla Ansible on demand, 'kayobe overcloud facts gather'. This can be
used to populate a fact cache.

Depends-On: https://review.opendev.org/c/openstack/kolla-ansible/+/794610
Story: 2007993
Task: 42586

Change-Id: I5ce3c734433e1682ee942867505468c57440e689

Add support for configuring apt's proxy setting on Ubuntu hosts.

Change-Id: Iea1daff70fca5cf49f4e7f44af71a900678bb5c9
Story: 2009035
Task: 42782

Change-Id: I5bdad8a2f379c11e6b55df8fd2481f5ae01ed6b9

Change-Id: I670593ba8cbcd7c523b9bd6c42b4d0ca29e856ea

This may be used to deploy different images on different hosts.

Change-Id: I941cc28a914f3a56f50abdde70f13d6616ff52e7
Story: 2002098
Task: 41694

By default, Ansible injects a variable for every fact, prefixed with
ansible_. This can result in a large number of variables for each host,
which at scale can incur a performance penalty. Ansible provides a
configuration option [0] that can be set to False to prevent this
injection of facts. In this case, facts should be referenced via
ansible_facts.<fact>.

This change updates all references to Ansible facts within Kayobe
from using individual fact variables to using the items in the
ansible_facts dictionary. This allows users to disable fact variable
injection in their Ansible configuration, which may provide some
performance improvement.

This change disables fact variable injection in the ansible
configuration used in CI, to catch any attempts to use the injected
variables.

[0] https://docs.ansible.com/ansible/latest/reference_appendices/config.html#inject-facts-as-vars

Story: 2007993
Task: 42464
Depends-On: https://review.opendev.org/c/openstack/kolla-ansible/+/791276

Change-Id: I14db53ed6e57d37bbd28dd5819e432e3fe6628b2

Change-Id: I097b826cb477577fc9482bbc3b8875b71a505a99

Documentation about DIB_DISABLE_KERNEL_CLEANUP was left in place after
the variable was removed in Ie25e88bb96f186399ae69837e93185127628ce86.

Backport: wallaby, victoria, ussuri

Change-Id: Iddb699aa98ddee8446d532ce4bd23ba9174fb38e

Story: 2008961
Task: 42597
Change-Id: I2637085a1df22e39d64c33b823fc948063d3e06f

Adds a new flag, 'docker_registry_network_mode', which defaults to
'host'. This may be used to set the network mode of the Docker registry
container.

This is a follow up to I404dd52701426a10c2e92727bd52b7fd7112abf6, which
changed the network mode from the default of bridge to host. It allows
that change to be backported to stable branches, without modifying the
default value.

Change-Id: Ic8ec3bb98f8f016e1d089bf10bd0538264394241

Some hosts in the kayobe inventory might not be in the kolla-ansible
inventory so it makes sense for kayobe to manage NTP.

Change-Id: Iacb579a46b0e9769a4c404a858d17968f74dd7e0
Depends-On: https://review.opendev.org/c/openstack/kayobe-config-dev/+/786040
Story: 2007872
Task: 40240

Change-Id: I3c4d81f59bb61fdd23234777b8d839adecfe86f2
Story: 2004960
Task: 42324

This change adds support for policy-based routing via systemd-networkd.
Due to differences in the configuration mechanism, routing policy
rules are configured via dicts for Ubuntu, while remaining as strings
on CentOS. Ideally we would support both formats.

Story: 2004960
Task: 42217

Change-Id: I77aec0160eb7e4dd763326bfe6e3d9a44b248108

This change adds support for network configuration via systemd-networkd
on Ubuntu systems.

This is implemented via an Ansible Galaxy role,
stackhpc.systemd_networkd which was forked from
aruhier.systemd_networkd. Several improvements were made in
https://github.com/stackhpc/ansible-role-systemd-networkd/pull/1,
including:

* Add support for removing unexpected config files
* Use become where necessary
* Refactor config generation into a single task to improve performance

The systemd_networkd role does not add much abstraction on top of the
systemd-networkd configuration file format, which provides a lot of
flexibility at the expense of additional code in Kayobe. This code is
implemented as filter plugins, similarly to the existing
MichaelRigart.interfaces role.

This patch includes support for:

* Ethernet interfaces
* bridges
* bonds
* VLANs
* virtual Ethernet pairs (to connect Linux bridges and OVS bridges)
* static IP addresses
* static routes
* MTU

Some network attributes are currently not supported for
systemd-networkd:

* rules
* route options
* ethtool_opts
* zone
* allowed addresses

Story: 2004960
Task: 41881

Change-Id: I248b5bb9ce5a80a07a2a311cb3aca6daca920720

Allow passing through additional host variables from Kayobe to Kolla
Ansible without overriding the entire list.

* kolla_seed_inventory_pass_through_host_vars_extra
* kolla_seed_inventory_pass_through_host_vars_map_extra
* kolla_overcloud_inventory_pass_through_host_vars_extra
* kolla_overcloud_inventory_pass_through_host_vars_map_extra

Story: 2008797
Task: 42201

Change-Id: I41d9169f0312108e42bb12c52c6c7ee2509cf2ab

Change-Id: I848d834aa36943027c126e26e93e4a4680521144
Story: 2002009
Task: 40037

It's now possible to change Docker's default 64M SHM size
for custom containers running on the seed.

Change-Id: Ic6c2ec38e8a22b8acc90e17a552e471aa8313f7d

This variable allows to customise the upper constraints file used to
install packages inside the ipa-build-dib virtual environment. This can
be used when we need a newer version of diskimage-builder than the one
available in upper constraints for the current release.

Change-Id: Idbe57e7edc3fae25153f5e24ad6b7847b1c4660c

Change-Id: Id5ed0b594d6270c4a03606caa776bbb291be2694

Kayobe currently supports definition of various different networks -
public, internal, tunnel, etc. These typically map to a VLAN or flat
network, with an IP subnet. When a cloud exceeds the size of a single
VLAN/subnet, this approach no longer works.

One way to resolve this is to have multiple subnets that map to a single
logical network, and provide routing between them. This is a similar
concept to neutron's routed networks, but for the control plane.

This change provides documentation for the currently tested parts of
this feature.

Change-Id: Ic06c6d4fff0fa568eb9ed3a9c30ce21c7699d965
Story: 2008180
Task: 40938

Change-Id: Id21616b5c03922002cd7c99d6df7976a502b4e3c
Story: 2008369
Task: 41279

Kayobe currently supports definition of various different networks -
public, internal, tunnel, etc. These typically map to a VLAN or flat
network, with an IP subnet. When a cloud exceeds the size of a single
VLAN/subnet, this approach no longer works.

One way to resolve this is to have multiple subnets that map to a single
logical network, and provide routing between them. This is a similar
concept to neutron's routed networks, but for the control plane.

An issue arising from this is that if different hosts can have different
network definitions for the internal and public networks, it is no
longer trivial to use a network attribute [1] to specify the VIP address
and FQDN. Furthermore, the play that generates Kolla Ansible's
globals.yml containing the VIP and FQDN variables runs as localhost,
which does not necessarily have the internal and public networks
defined.

To resolve this, we add global variables for the VIPs and FQDNs. The
default values are as before, except in the case where HAProxy is
disabled, which we no longer provide a useful default for. That
configuration is very rarely used in practice, and the need to reference
the IP address of a host in the network group makes it difficult to
define safely.

[1] https://docs.openstack.org/kayobe/latest/configuration/reference/network.html#global-network-configuration

Story: 2008180
Task: 40937

Change-Id: I2c428ffc2b285aee03d8f59ae7cd3fb7230ce4ae

To avoid switching existing deployments from devicemapper to overlay2,
we check the existing storage driver configuration directly with the
Docker daemon, or if unreachable by reading the /etc/docker/daemon.json
configuration file.

Co-Authored-By: Pierre Riteau <pierre@stackhpc.com>
Story: 2005667
Task: 30972

Change-Id: Iaf2ee8c9f302f4684ae039bb00b2e2e5969cf1fc

Story: 2008170
Task: 40925

Change-Id: I3014983f481a5dca7c93e140b3e10caa5d537669

Kayobe generates a host_vars file for each host in the Kolla Ansible
inventory. These contain network interfaces and other host-specific
things. Currently this is done by iterating over all hosts, which does
not scale well with a large number of hosts.

This change extracts the host vars generation into a separate role, and
executes it in a play targeted at all hosts, with delegate_to:
localhost. This ensures that host variable files are generated in
parallel.

Story: 2007993
Task: 40629

Change-Id: Iae75e17024adee9c2874c14d3ed36f4c87ba48d7

Adds support for HTTP basic authentication with the Docker registry.

The kolla docker registry password is now written to passwords.yml.

Change-Id: Ie6e854a66a6660d4e02771fe2b5dd97af814194d
Story: 2007952
Task: 40429

The variables in yum.yml and yum-cron.yml were deprecated in Ussuri.
This patch removes them, and updates the defaults in dnf.yml.

Story: 2008160
Task: 40906

Change-Id: I97cc98dd2ff726e5885fefcab17f17796d9fd453

This allows us to add configuration scenarios.

Change-Id: Id636f78c61237fb27fa65fa3d4b3fc1a4cf0ba6a
Story: 2004360
Task: 40777