Change-Id: If354fbdeb2fd8d026faa0210b84da0a688c3ae0a

Kolla-Ansible Ceph deployment mechanism has been deprecated in Train [1].

This change removes the Ansible code and associated CI jobs.

[1]: https://review.opendev.org/669214

Change-Id: Ie2167f02ad2f525d3b0f553e2c047516acf55bc2

As discussed during the meeting. [1]

[1] http://eavesdrop.openstack.org/meetings/kolla/2020/kolla.2020-02-05-15.00.html

Change-Id: I339fc7f01f640a32ff35d19ded54b6f8e2fdec15

Change-Id: Ifc87adbac6bef0aaab2ad0e80739654b3e3123ad

Add a TLS scenario in zuul to generate self signed certificates and
to configure TLS to be enabled in the open stack deployment.

Change-Id: If10a23dfa67212e843ef26486c9523074cc920e7
Partially-Implements: blueprint custom-cacerts

* Adding zuul centos-source/ubuntu-source ceph-ansible jobs
* Jobs will deploy all Ceph integrated OpenStack components, i.e.
  cinder, glance, nova
* Will utilize core openstack testing script

Depends-On: https://review.opendev.org/685032
Depends-On: https://review.opendev.org/698301

Implements: blueprint ceph-ansible
Change-Id: I233082b46785f74014177f579aeac887a25b2ae2

Maximum supported version is set to 2.9

Updated the minimum supported version to 2.8

Implements: blueprint ansible-max-version

Change-Id: I97cc95e37f49886e6d74f2d5a789b923b14b5a2d

In CentOS/RHEL 8 there is no scsi-target-utils package, nor is it
available in EPEL. It is removed from kolla in [1]. In RHEL 7 and beyond
the LIO kernel subsystem can be used instead of the tgtd daemon.

This change removes support for the SCSI target daemon on CentOS/RHEL 8.
The 'tgtd' image is no longer available for CentOS/RHEL 8.

[1] https://review.openstack.org/#/c/613815/5

Change-Id: I718fc16cde2dd177b2a1c2f79b932426034897fe
Related: blueprint centos-rhel-8

Debian/source to match Kolla CI test.

Change-Id: Ica45d5673db8ba1ebd4bfeb6e3d37b5ad5dd412b

Partially-implements: blueprint kolla-deployment-scenario-testing
Change-Id: I82266829dba47eac81f440a53706c2e9064b8e62

Co-authored-by: Radosław Piliszek <radoslaw.piliszek@gmail.com>
Depends-on: https://review.opendev.org/694476
Change-Id: I6e7f2f4229c8b579dcc17dacffeb74160875ae29

This change switches the CI jobs to use python 3 for local execution of
the kolla-ansible commands.

For upgrades, we use python 2 for the previous (Train) deploy, then
reinstall using python 3 for the (Ussuri) upgrade.

NOTE: This is separate from the python interpreter used on remote hosts,
which is configured via ansible_python_interpreter.

Partially Implements: blueprint python-3
Related: blueprint drop-py2-support

Change-Id: I5bdc165f68b7bde1f9ef30fe8216f2a44e6d4706

Continue to reduce the scope of setup_gate.sh. Allows us to more easily
select python 2 or 3.

Change-Id: If2eeeacbbbdf58afb765b4a39772b5a1af7b952b
Partially Implements: blueprint python-3

Separate upgrade logic to is_upgrade job var and rename
scenarios to match.

Rename "ACTION" to "SCENARIO" (as it is a scenario).

Separate testing of dashboard (aka Horizon) and increase
its timeout to 5 minutes (CentOS 7 slow as always).

Separate initialization of core OpenStack.

Use gate setup script from ./tests/

Remove useless tox setupenv.

Do not deploy Heat when not really necessary.

Change-Id: I4fca319ccc3de7188f8b7b44c9c71321e3899467

Change-Id: I27507816e3fe62df2a043dad96e7d1bb4b439869
Closes-bug: #1847331

Resolves a number of TODOs in the CI configuration that provide support
for upgrading from the Stein release.

Change-Id: I9bac5c230b82ac7c097fe6ca2556e428abda31a1
Depends-On: https://review.opendev.org/694254

Tests the following operations for MariaDB:

* Stop
* Recovery

Backup and restore will be added in a separate change.

Depends-On: https://review.opendev.org/693329
Change-Id: I836d91554715cce0e82c1bbebb7430c457418b2d

Commit 73b6a66f added installation of
Python 3 package. But without root permissions it fails.

Change-Id: I65ca794955a1b1419853bf63be36cb0d1f2d2345

Debian source images are published now.

Change-Id: Id611fd2fa71eb54ca08d1f68de9505d28ad4ea40

Needed for https://review.opendev.org/#/c/691316

Change-Id: I64e250eb15882f50ecbbc57b87e036f5772f7e3a

Introduce kolla_address filter.
Introduce put_address_in_context filter.

Add AF config to vars.

Address contexts:
- raw (default): <ADDR>
- memcache: inet6:[<ADDR>]
- url: [<ADDR>]

Other changes:

globals.yml - mention just IP in comment

prechecks/port_checks (api_intf) - kolla_address handles validation

3x interface conditional (swift configs: replication/storage)

2x interface variable definition with hostname
(haproxy listens; api intf)

1x interface variable definition with hostname with bifrost exclusion
(baremetal pre-install /etc/hosts; api intf)

neutron's ml2 'overlay_ip_version' set to 6 for IPv6 on tunnel network

basic multinode source CI job for IPv6

prechecks for rabbitmq and qdrouterd use proper NSS database now

MariaDB Galera Cluster WSREP SST mariabackup workaround
(socat and IPv6)

Ceph naming workaround in CI
TODO: probably needs documenting

RabbitMQ IPv6-only proto_dist

Ceph ms switch to IPv6 mode

Remove neutron-server ml2_type_vxlan/vxlan_group setting
as it is not used (let's avoid any confusion)
and could break setups without proper multicast routing
if it started working (also IPv4-only)

haproxy upgrade checks for slaves based on ipv6 addresses

TODO:

ovs-dpdk grabs ipv4 network address (w/ prefix len / submask)
not supported, invalid by default because neutron_external has no address
No idea whether ovs-dpdk works at all atm.

ml2 for xenapi
Xen is not supported too well.
This would require working with XenAPI facts.

rp_filter setting
This would require meddling with ip6tables (there is no sysctl param).
By default nothing is dropped.
Unlikely we really need it.

ironic dnsmasq is configured IPv4-only
dnsmasq needs DHCPv6 options and testing in vivo.

KNOWN ISSUES (beyond us):

One cannot use IPv6 address to reference the image for docker like we
currently do, see: https://github.com/moby/moby/issues/39033
(docker_registry; docker API 400 - invalid reference format)
workaround: use hostname/FQDN

RabbitMQ may fail to bind to IPv6 if hostname resolves also to IPv4.
This is due to old RabbitMQ versions available in images.
IPv4 is preferred by default and may fail in the IPv6-only scenario.
This should be no problem in real life as IPv6-only is indeed IPv6-only.
Also, when new RabbitMQ (3.7.16/3.8+) makes it into images, this will
no longer be relevant as we supply all the necessary config.
See: https://github.com/rabbitmq/rabbitmq-server/pull/1982

For reliable runs, at least Ansible 2.8 is required (2.8.5 confirmed
to work well). Older Ansible versions are known to miss IPv6 addresses
in interface facts. This may affect redeploys, reconfigures and
upgrades which run after VIP address is assigned.
See: https://github.com/ansible/ansible/issues/63227

Bifrost Train does not support IPv6 deployments.
See: https://storyboard.openstack.org/#!/story/2006689



Change-Id: Ia34e6916ea4f99e9522cd2ddde03a0a4776f7e2c
Implements: blueprint ipv6-control-plane
Signed-off-by: Radosław Piliszek <radoslaw.piliszek@gmail.com>

This ensures that failure of a single host fails the whole play at that
task. This can avoid confusing errors such as when the task
"Assert that the nodepool private IPv4 address is assigned" fails on one
host, causing subsequent errors on other hosts.

Note that this only affects the Zuul playbooks, not Kolla Ansible's
playbooks.

Change-Id: I77a6534dd2ddd188f795e17d17a44be249d01f31

Change-Id: I0628b16e3ebdb3fa8196acdc1bd9c63e75bcfb09

After modernising docker configuration
(I1215e04ec15b01c0b43bac8c0e81293f6724f278), we lost our
registry-mirrors configuration in CI that lets us use a mirror of
Dockerhub.

This change uses the new docker_custom_config variable to configure the
registry mirror.

Change-Id: I1430413c12e9d0b59e4f216ff66372de0f3a4f21

VXLAN is necessary to run HA in CI (due to floating VIP
address handled by keepalived).
It also turned out to be required to have private
IPv6 address assignments.
This patch is based on linux bridge rather than OVS
to avoid problems with OVS deployed in containers.

This patch enables haproxy in multinode jobs.

Includes saving of linux networking details.

Makes DASHBOARD_URL agree with OS_AUTH_URL - properly uses the
pre-upgrade value for testing.

Co-authored-by: Radosław Piliszek <radoslaw.piliszek@gmail.com>
Depends-on: https://review.opendev.org/683068
Depends-on: https://review.opendev.org/682957
Change-Id: I66888712da80c3d6f84ee4949762961664d3adea

This lets us control the upgrade process entirely from the
current branch.

Change-Id: Ic8c39e415846596c23dae93c2839375a24e8b888
Signed-off-by: Radosław Piliszek <radoslaw.piliszek@gmail.com>

These filters can be used to capture a lot of the logic that we
currently have in 'when' statements, about which services are enabled
for a particular host.

In order to use these filters, it is necessary to install the
kolla_ansible python module, and not just the dependencies listed in
requirements.txt. The CI test and quickstart install from source
documentation has been updated accordingly.

Ansible is not currently in OpenStack global requirements, so for unit
tests we avoid a direct dependency on Ansible and provide fakes where
necessary.

Change-Id: Ib91cac3c28e2b5a834c9746b1d2236a309529556

Also slightly refactor test-openstack.sh script.

Change-Id: I7f10f073e89d2b66367bbb700201b3cd412fc433
Depends-On: https://review.opendev.org/#/c/674241
Depends-On: https://review.opendev.org/#/c/668410
Depends-On: https://review.opendev.org/#/c/668409

At the end of a CI run, check all log files.

Change-Id: I99afc1c5207757e35beabf7daebd86c56151c96d

- Test Zun on CentOS too
- Make etcd change also trigger Zun jobs (like kuryr and zun)
- Test multinode Zun deployments instead of AIO
  (more likely to break)
- In Zun scenario, stop configuring docker for legacy swarm mode
  (Zun is no swarm)
- Separate test-zun.sh testing script
- Show appcontainer to see which node it has been started on

Change-Id: I289b1009fe00aedb9b78cbd83298b14da5fd9670
Depends-On: https://review.opendev.org/676736


Signed-off-by: Radosław Piliszek <radoslaw.piliszek@gmail.com>

Add Masakari testing into the Gate.

Change-Id: I52df33f963e7d2ae4059887df3d24d9e6642134e
Depends-On: https://review.opendev.org/#/c/615469/
Depends-On: https://review.opendev.org/#/c/615715


Implements: blueprint ansible-masakari
Co-Authored-By: Gaëtan Trellu <gaetan.trellu@incloudus.com>

Multinode jobs did not run sanity checks for all the hosts,
only primary. Now they check all.

Additionally upgrades are now checked using the proper
(pre-upgrade) scripts (not that it matters too much as they
are the same atm) and both checks are done, not only failures,
but also config.

Change-Id: I10552e256edbddd5b1f8a8a7f8805262e72ce8d8
Signed-off-by: Radosław Piliszek <radoslaw.piliszek@gmail.com>

Some kolla-ansible jobs failed due to using external mirrors
instead of local ones.
This was due to not using the template override provided by kolla.
This patch fixes that.

Depends-On: https://review.opendev.org/668226


Change-Id: I27f714fdf05e521aa8ce25c5683a452ceb35eeb8
Signed-off-by: Radosław Piliszek <radoslaw.piliszek@gmail.com>

These were not templated, so always evaluated to true. This shouldn't be
causing any issues.

Change-Id: I7b8e407e688ba201c4f7d1a94bbd41af0918e7df

Adds four new CI jobs for testing centos/ubuntu binary/source deploys
with ironic enabled. These are run only when there are changes to the
ironic role.

Performs some simple testing by creating a node using the fake-hardware
hardware type and creating a server.

Change-Id: Ie669e57ce2af53257b4ca05f45193cb73f48827a
Depends-On: https://review.opendev.org/664011

Add CI jobs for testing an upgrade of a multinode system with Ceph
enabled. As for the existing upgrade job, we upgrade from the previous
release to the current release.

Change-Id: I931772ca4c63757769467a57c80dc0726a11167a
Depends-On: https://review.opendev.org/658163

This commit was bulk generated and pushed by the OpenDev sysadmins
as a part of the Git hosting and code review systems migration
detailed in these mailing list posts:

http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003603.html
http://lists.openstack.org/pipermail/openstack-discuss/2019-April/004920.html

Attempts have been made to correct repository namespaces and
hostnames based on simple pattern matching, but it's possible some
were updated incorrectly or missed entirely. Please reach out to us
via the contact information listed at https://opendev.org/ with any
questions you may have.

Periodic jobs don't have zuul.change defined, since there is no change
being tested. This causes an early failure when referencing zuul.change
to set the image tag for built images. In periodic jobs we'll never need
to build images because there is no dependent kolla change under test.

Change-Id: I6d9d81cf17b7d0d7aaf87cd96418c904c46681f2

Typically, non-executable files should have 660 or 600 and executable
files and directories should have 770. All should be owned by the
'config_owner_user' and 'config_owner_group' variables.

This change adds a script to check the owner and permissions of config
files under /etc/kolla, and runs it at the end of CI jobs.

Change-Id: Icdbabf36e284b9030017a0dc07b9dc81a37758ab
Related-Bug: #1821579

This patch adds two new jobs:

* kolla-ansible-centos-source-upgrade
* kolla-ansible-ubuntu-source-upgrade

These jobs first deploy a control plane using the previous release of
Kolla Ansible, then upgrade to the current release.

Because we can't change the branch of the git repository on the Zuul
executor, we change the branch of the kolla-ansible repository on the
primary node to the branch of the previous release, in this case
stable/rocky. A new remote-template role has been added that supports
generating templates using a remote template source, to generate config
files using the previous kolla-ansible branch.

If the change being tested depends on a kolla change for the current
branch, then we build images. Rather than using the current
kolla-ansible version to tag the images, we now tag them with
change_<gerrit change ID>. This is because the version of kolla-ansible
will change from the previous release to the current one as we upgrade
the system.

Finally, it should be noted that the 'previous_release' variable in the
Zuul config needs to be updated with each release, since this sets the
release of kolla-ansible that is installed initially.

Depends-On: https://review.openstack.org/645089/
Depends-On: https://review.openstack.org/644250/
Depends-On: https://review.openstack.org/645816/
Depends-On: https://review.openstack.org/645840/
Change-Id: If301e0affcd55360fefe3b105f023ae5c47b0853