CentOS 8 support is now fairly complete - time to drop CentOS 7.

Partially-Implements: blueprint centos-rhel-8

Change-Id: I940b1d3eceb98e16fa366c243672f588b1412d70

The repo is Python 3 now, so update hacking to version 3.0 which
supports Python 3.

Fix problems found by updated hacking version.

Remove hacking and friends from lower-constraints, they are not needed
during installation.

Change-Id: I7ef5ac8a89e94f5da97780198619b6facc86ecfe

We don't need tox nor tell our users to use it when doing kolla
builds.
Tox is going away from base infra images.
It's already gone from aarch64 (arm64) ones.

Change-Id: I2eb5203ad93c011b8806f4b6fb56db081c14a2cb

Not everyone wants Kafka data stored on a Docker volume. This
change allows a user to flexibly control where the data is stored.

Change-Id: I2ba8c7a85c7bf2564f954a43c6e6dbb3257fe902

We released CirrOS 0.5.1. Time to move then.

Change-Id: Ibca24836f19b3cbf6166fa39a3702883938feda8

The affected command was meant to create a tenant network, so let
us really test it this way.

Not marking CI, because someone may be using this script.

Change-Id: I5abe46948992121a11a36f941d4f8fac1caa92b1

Since fluentd is disabled in MariaDB jobs - haproxy logs are not getting
populated.

Change-Id: I56b3fc1be6940d97905cdb2c4452b846f106c071
Depends-on: https://review.opendev.org/713704

While supporting both CentOS 7 and 8, we used the tag 'master-centos8'
for CentOS 8 images. We are now ready to drop CentOS 7 support, and
Kolla is switching to publish CentOS 8 images using the master tag on
the master branch, so we should use this.

Depends-On: https://review.opendev.org/713265

Partially-Implements: blueprint centos-rhel-8

Change-Id: I07d2c285e3214a6dc827a8e8eacf263048ee099b

This avoids issues with IPv6 not enabled in docker.

Change-Id: I0faa1df8a04acd1746fa8a5f0d8455f819c254b5

Also adds gawk for timestamping.

This helps to correlate init events with failures elsewhere.

Change-Id: I22fdb683ecf9870b2d66fedd6b40b7004317130a

Clients are starting to release versions that don't support Python 2.
The ironic scenario is currently failing on stable branches for this
reason.

Use upper constraints to avoid installing these new versions on stable
branches.

Change-Id: I4f91b53cbf2297d70da4b54d6c402c1427aacdd9

I'm not marking it CI-only as ppl seem to be using it.
OTOH, not adding a release note as we are not promoting its usage.

This is to allow us to customize for CI.

Change-Id: I8100a6cb63b1e54078629bd6ca8475dc5896784a

This allows you to tune the performance of InfluxDB by locating the
volume on a drive that is separate to the default docker storage.

Change-Id: Iea555a2702b225b30f5d7035b8a703d4f3376ee7

The bash part of version checking fails when ANSIBLE_DEBUG is set
while running kolla-ansible, so need to skip it.

Change-Id: I4adc986365c4b0c11eb69e128eb7d037a24a2baf

Also better organise the 'tips and tricks' section.

Change-Id: If20a89ae93b64b5a1b5895bb9dc73c4e70adaba7
Related-Bug: #1862649

Kolla-Ansible Ceph deployment mechanism has been deprecated in Train [1].

This change removes the Ansible code and associated CI jobs.

[1]: https://review.opendev.org/669214

Change-Id: Ie2167f02ad2f525d3b0f553e2c047516acf55bc2

Adds new CI job definitions for CentOS 8:

- kolla-ansible-centos8-source
- kolla-ansible-centos8-binary
- kolla-ansible-centos8-source-ceph-ansible
- kolla-ansible-centos8-source-cinder-lvm
- kolla-ansible-centos8-source-mariadb
- kolla-ansible-centos8-source-bifrost
- kolla-ansible-centos8-source-zun
- kolla-ansible-centos8-source-swift
- kolla-ansible-centos8-source-scenario-nfv
- kolla-ansible-centos8-source-ironic
- kolla-ansible-centos8-binary-ironic
- kolla-ansible-centos8-source-masakari
- kolla-ansible-centos8-source-cells

The following jobs are added to the check pipeline:

- kolla-ansible-centos8-source
- kolla-ansible-centos8-binary
- kolla-ansible-centos8-source-cinder-lvm
- kolla-ansible-centos8-source-mariadb
- kolla-ansible-centos8-source-zun
- kolla-ansible-centos8-source-swift
- kolla-ansible-centos8-source-scenario-nfv
- kolla-ansible-centos8-source-ironic
- kolla-ansible-centos8-binary-ironic
- kolla-ansible-centos8-source-cells

The following jobs are not yet passing so are not added to the check
pipeline:

- kolla-ansible-centos8-source-ceph-ansible
- kolla-ansible-centos8-source-bifrost
- kolla-ansible-centos8-source-masakari

The kolla-ansible-centos8-source job is added to the gate.

Upgrade jobs will be added when CentOS 8 support exists in Train.

Depends-On: https://review.opendev.org/704337
Depends-On: https://review.opendev.org/704848
Depends-On: https://review.opendev.org/704965



Co-Authored-By: Mark Goddard <mark@stackhpc.com>

Change-Id: Ibd806feee71721b122b77d7eff33228ca1cc2853
Partially-Implements: blueprint centos-rhel-8

As discussed during the meeting. [1]

[1] http://eavesdrop.openstack.org/meetings/kolla/2020/kolla.2020-02-05-15.00.html

Change-Id: I339fc7f01f640a32ff35d19ded54b6f8e2fdec15

Change-Id: I957fcea4ab180dd82d1faaa9a012f83d3647f043

Due to regex matching of images, we are currently building quite a few
unnecessary images in CI. Images that can unexpectedly match:

* barbican-keystone-listener
* ironic-neutron-agent
* prometheus-haproxy-exporter
* prometheus-memcached-exporter

This change improves the situation by anchoring matches to the start of
the name. A full solution would be to specify every image required for
each job. This would be quite verbose and require maintenance.

Change-Id: I7387ccd0cb394c54c79bfba8b1c09fce7ee2973e

This change introduces prune-images command.

Uses docker_prune module of Ansible that comes with version 2.8.

Depends-On: https://review.opendev.org/#/c/699333/

Implements: blueprint docker-image-pruning

Change-Id: Icbf374dd50e1cc1f1604bb4fa779b34279efd50c

* Adding zuul centos-source/ubuntu-source ceph-ansible jobs
* Jobs will deploy all Ceph integrated OpenStack components, i.e.
  cinder, glance, nova
* Will utilize core openstack testing script

Depends-On: https://review.opendev.org/685032
Depends-On: https://review.opendev.org/698301

Implements: blueprint ceph-ansible
Change-Id: I233082b46785f74014177f579aeac887a25b2ae2

Maximum supported version is set to 2.9

Updated the minimum supported version to 2.8

Implements: blueprint ansible-max-version

Change-Id: I97cc95e37f49886e6d74f2d5a789b923b14b5a2d

In CentOS/RHEL 8 there is no scsi-target-utils package, nor is it
available in EPEL. It is removed from kolla in [1]. In RHEL 7 and beyond
the LIO kernel subsystem can be used instead of the tgtd daemon.

This change removes support for the SCSI target daemon on CentOS/RHEL 8.
The 'tgtd' image is no longer available for CentOS/RHEL 8.

[1] https://review.openstack.org/#/c/613815/5

Change-Id: I718fc16cde2dd177b2a1c2f79b932426034897fe
Related: blueprint centos-rhel-8

This switches our test jobs to use python 3 for the installation of
OpenStack clients used to test the control plane.

Change-Id: I6bf0a315cef174c42a69db3a7d57f602b27ac92c
Related: blueprint python 3

Co-authored-by: Radosław Piliszek <radoslaw.piliszek@gmail.com>
Depends-on: https://review.opendev.org/694476
Change-Id: I6e7f2f4229c8b579dcc17dacffeb74160875ae29

Continue to reduce the scope of setup_gate.sh. Allows us to more easily
select python 2 or 3.

Change-Id: If2eeeacbbbdf58afb765b4a39772b5a1af7b952b
Partially Implements: blueprint python-3

The kolla-ansible script uses a python interpreter to detect the
location of playbooks and to check that the kolla_ansible python module
is importable. This change updates the script to support a python
interpreter named python or python3.

Partially Implements: blueprint python-3

Change-Id: Id5dcc53cc5dd9780632c04b6b73c56ea5da484a2

The kolla-ansible script now (Train+) checks whether the kolla_ansible
python module is installed, and emits the following message if not:

ERROR: kolla_ansible has to be available in the PYTHONPATH (e.g.
installed)

It is supposed to exit 1 if this check fails, but does not.

This change fixes this by removing a subshell from the kolla-ansible
script.

Change-Id: I2c1a7398f2b8f876abcf2508874835154cb4ce57
Closes-Bug: #1854049

Separate upgrade logic to is_upgrade job var and rename
scenarios to match.

Rename "ACTION" to "SCENARIO" (as it is a scenario).

Separate testing of dashboard (aka Horizon) and increase
its timeout to 5 minutes (CentOS 7 slow as always).

Separate initialization of core OpenStack.

Use gate setup script from ./tests/

Remove useless tox setupenv.

Do not deploy Heat when not really necessary.

Change-Id: I4fca319ccc3de7188f8b7b44c9c71321e3899467

This fixes Debian job failures during image building.

See https://github.com/docker/for-linux/issues/711
for upstream details.

Change-Id: Icf3ffb261605ffe5d8f2618c2ed4cb97db97dd49

As part of the effort to implement Ansible code linting in CI
(using ansible-lint) - we need to implement recommendations from
ansible-lint output [1].

One of them is to stop using local_action in favor of delegate_to -
to increase readability and and match the style of typical ansible
tasks.

[1]: https://review.opendev.org/694779/

Partially implements: blueprint ansible-lint

Change-Id: I46c259ddad5a6aaf9c7301e6c44cd8a1d5c457d3

Change-Id: Ie7ee70e19536c2d8f4300df55c9c6ca38abb7ae4

Tests the following operations for MariaDB:

* Stop
* Recovery

Backup and restore will be added in a separate change.

Depends-On: https://review.opendev.org/693329
Change-Id: I836d91554715cce0e82c1bbebb7430c457418b2d

This playbook is neither kolla-ansible nor Zuul. Much of what it does is
stale and no longer required. Remove it.

Change-Id: I6fed9d8f2439a53c6ac35c670b56b43579236ee9

Starting with the Train release, kolla_ansible (the package) must be
available in the PYTHONPATH to run most playbooks.
It is documented.
This change makes the error message friendlier.

Change-Id: Icf0399d21b3fde8d530d73e6e7ee4a57665da276
Closes-bug: #1849943

Introduce kolla_address filter.
Introduce put_address_in_context filter.

Add AF config to vars.

Address contexts:
- raw (default): <ADDR>
- memcache: inet6:[<ADDR>]
- url: [<ADDR>]

Other changes:

globals.yml - mention just IP in comment

prechecks/port_checks (api_intf) - kolla_address handles validation

3x interface conditional (swift configs: replication/storage)

2x interface variable definition with hostname
(haproxy listens; api intf)

1x interface variable definition with hostname with bifrost exclusion
(baremetal pre-install /etc/hosts; api intf)

neutron's ml2 'overlay_ip_version' set to 6 for IPv6 on tunnel network

basic multinode source CI job for IPv6

prechecks for rabbitmq and qdrouterd use proper NSS database now

MariaDB Galera Cluster WSREP SST mariabackup workaround
(socat and IPv6)

Ceph naming workaround in CI
TODO: probably needs documenting

RabbitMQ IPv6-only proto_dist

Ceph ms switch to IPv6 mode

Remove neutron-server ml2_type_vxlan/vxlan_group setting
as it is not used (let's avoid any confusion)
and could break setups without proper multicast routing
if it started working (also IPv4-only)

haproxy upgrade checks for slaves based on ipv6 addresses

TODO:

ovs-dpdk grabs ipv4 network address (w/ prefix len / submask)
not supported, invalid by default because neutron_external has no address
No idea whether ovs-dpdk works at all atm.

ml2 for xenapi
Xen is not supported too well.
This would require working with XenAPI facts.

rp_filter setting
This would require meddling with ip6tables (there is no sysctl param).
By default nothing is dropped.
Unlikely we really need it.

ironic dnsmasq is configured IPv4-only
dnsmasq needs DHCPv6 options and testing in vivo.

KNOWN ISSUES (beyond us):

One cannot use IPv6 address to reference the image for docker like we
currently do, see: https://github.com/moby/moby/issues/39033
(docker_registry; docker API 400 - invalid reference format)
workaround: use hostname/FQDN

RabbitMQ may fail to bind to IPv6 if hostname resolves also to IPv4.
This is due to old RabbitMQ versions available in images.
IPv4 is preferred by default and may fail in the IPv6-only scenario.
This should be no problem in real life as IPv6-only is indeed IPv6-only.
Also, when new RabbitMQ (3.7.16/3.8+) makes it into images, this will
no longer be relevant as we supply all the necessary config.
See: https://github.com/rabbitmq/rabbitmq-server/pull/1982

For reliable runs, at least Ansible 2.8 is required (2.8.5 confirmed
to work well). Older Ansible versions are known to miss IPv6 addresses
in interface facts. This may affect redeploys, reconfigures and
upgrades which run after VIP address is assigned.
See: https://github.com/ansible/ansible/issues/63227

Bifrost Train does not support IPv6 deployments.
See: https://storyboard.openstack.org/#!/story/2006689



Change-Id: Ia34e6916ea4f99e9522cd2ddde03a0a4776f7e2c
Implements: blueprint ipv6-control-plane
Signed-off-by: Radosław Piliszek <radoslaw.piliszek@gmail.com>

Change-Id: I0628b16e3ebdb3fa8196acdc1bd9c63e75bcfb09

Sometimes as cloud admins, we want to only update code that is running
in a cloud.  But we dont need to do anything else.  Make an action in
kolla-ansible that allows us to do that.

Change-Id: I904f595c69f7276e71692696471e32fd1f88e6e8
Implements: blueprint deploy-containers-action

This is required for the dict2items filter.

Change-Id: I60a04e839bf06506ff36c2631a286130d5fde972