Change-Id: I5befc72a4894d625ca352b27df9d3aa84a2f5b2c

Adds a new Zuul job, kolla-ansible-centos8-source-magnum, for testing
deployment of Magnum, Octavia and associated services.

Change-Id: I61b293ba6bb52064ea98a73e2dff0023fa01a2a2

If we don't set it, then Zun chooses one randomly (the first one
from Neutron).
This may break if it is a network that is not available on
target hosts, e.g. external via L3 agent router.

Since capsules do not support nets yet [1], this patch ensures
desired network creation order in init-runonce instead.

[1] https://bugs.launchpad.net/zun/+bug/1895263

Change-Id: Iaa113dcfb826164a2772d2c91d34ec0236be0817

Per the recent Kayobe brekage due to TLS support in Ironic [1],
let's test Ironic Inspector API as well.

[1] https://review.opendev.org/750804

Change-Id: I7ccf0c4286f8907bc2fa2eabc41ec2876c9815a9

Change-Id: Ia4626479e092be8b033bcd4e75e78a33167423d3

The Kolla-Ansible part.

This switches Kolla-Ansible to use the kolla-build-config role
instead of generating config locally.

Depends-On: https://review.opendev.org/607159
Change-Id: I859acbe4f84ccbdc53764574a58e6f0fab4094a3

This is confusing as it is not meant to be used by users.
Also, various tools show duplicated matches due to both locations
containing the exact same content.

Change-Id: I2debe121f64954e57788270d3258775f29f1cbb0

This should improve performance of CI jobs.

Change-Id: I729862b89b4fe65cbb7f852fac06741b636e4939

There is a time once every 2 years when ubuntu team releases new LTS
release. And then UCA joins with binary packages for current OpenStack
development cycle.

It is this time for Ubuntu 20.04 'focal'.

Includes CI fix to pass:

[CI] Temporarily block new Ansible

The proper fix [1] needs fixing older branches before newer.
This one allows to fix CI first, in the usual order.

To revert after [1] gets merged in all relevant branches.

[1] https://review.opendev.org/745648

Old-Change-Id: Ifbd37d8addd4322773118e2e9d46494741a8ae66
Related-Bug: #1891145

Depends-on: https://review.opendev.org/#/c/738994/


Change-Id: Ib8b70ee40ec2d19509cc84c0f530612f81907721
Co-Authored-By: Radosław Piliszek <radoslaw.piliszek@gmail.com>

Change-Id: Ic697729bda80bfb83171bf68223a2703b8318aad

Change-Id: I2e22ec47f644de2f1509a0111c9e1fffe8da0a1a

Change-Id: I2e736920f5b2cb4a78c2e6c216665394faf83001

Change-Id: I27188936f516318bfc7e35cfe7276198ed8f9ac9

Tests prometheus, grafana, and centralised logging.

The tests could be improved in future by querying logs in elasticsearch,
and metrics in prometheus.

Change-Id: Iabad035d583d291169f23be3d71931cb260e87ae

The common role was previously added as a dependency to all other roles.
It would set a fact after running on a host to avoid running twice. This
had the nice effect that deploying any service would automatically pull
in the common services for that host. When using tags, any services with
matching tags would also run the common role. This could be both
surprising and sometimes useful.

When using Ansible at large scale, there is a penalty associated with
executing a task against a large number of hosts, even if it is skipped.
The common role introduces some overhead, just in determining that it
has already run.

This change extracts the common role into a separate play, and removes
the dependency on it from all other roles. New groups have been added
for cron, fluentd, and kolla-toolbox, similar to other services. This
changes the behaviour in the following ways:

* The common role is now run for all hosts at the beginning, rather than
  prior to their first enabled service
* Hosts must be in the necessary group for each of the common services
  in order to have that service deployed. This is mostly to avoid
  deploying on localhost or the deployment host
* If tags are specified for another service e.g. nova, the common role
  will *not* automatically run for matching hosts. The common tag must
  be specified explicitly

The last of these is probably the largest behaviour change. While it
would be possible to determine which hosts should automatically run the
common role, it would be quite complex, and would introduce some
overhead that would probably negate the benefit of splitting out the
common role.

Partially-Implements: blueprint performance-improvements

Change-Id: I6a4676bf6efeebc61383ec7a406db07c7a868b2a

more info: https://review.opendev.org/#/c/721733/



Depends-On: I561ead226f714d98c8e06e6027715a64c3a8e47e
Depends-On: I21c9ab9820f78cf76adf11c5f0591c60f76372a8
Change-Id: Ic740d090211ee331b374a6dac69dfde466df7200
Co-Authored-By: jacky06 <zhang.min@99cloud.net>

more info: https://opendev.org/openstack/kolla-ansible/commit/a6c97d7284c7de437ebfc9f8ee289244f29e65d7



Change-Id: I778d472cc7f6ca19852482a3e309d793973d75a6
Co-Authored-By: jacky06 <zhang.min@99cloud.net>

It seems we used virtualenv for ceph-ansible only.

Change-Id: I7f9002283462dbe4bae3c1d7ff1dedcc4e7d01f2

Change-Id: I4d6e4f43bcf3dc31cad0675f94724dbc4fcb466f

Replaced "kolla_external_fqdn_cacert" and "kolla_internal_fqdn_cacert" with
"kolla_admin_openrc_cacert". OS_CACERT is now set to the value of
"kolla_admin_openrc_cacert" in the generated admin-openrc.sh file.

Change-Id: If195d5402579cee9a14b91f63f5fde84eb84cccf
Partially-Implements: blueprint add-ssl-internal-network
Depends-On: https://review.opendev.org/#/c/731344/

Update the certificate generation task to create a root CA for the
self-signed certificates. The internal and external facing certificates
are then generated using the root CA.

Updated openstack_cacert to use system CA trust store in CI tests
certificate by default.

Change-Id: I6c2adff7d0128146cf086103ff6060b0dcefa37b
Partially-Implements: blueprint add-ssl-internal-network

This also uses the recommended machinery to set qemu instead
of relying on config file override so that we test what we
really want to test.

Change-Id: I560e4f9d0a69c347e6aaf3b970331157c1a56f18

This ought to fix #1864238 for the stable branches.

Change-Id: I218905247a4a4003ecfc2c9ab3e47767bb5ab33e
Related-bug: #1864238

Ussuri is Py3 everywhere. We are free to drop any py2 leftovers.

Depends-On: https://review.opendev.org/731805
Change-Id: I1a9c9a14af351cd3e8b01a40f323a82ffa673d35

The Monasca Log API has been removed and in this change we switch
to using the unified API. If dedicated log APIs are required then
this can be supported through configuration. Out of the box the
Monasca API is used for both logs and metrics which is envisaged to
work for most use cases.

In order to use the unified API for logs, we need to disable the
legacy Kafka client. We also rename the Monasca API config file
to remove a warning about using the old style name.

Depends-On: https://review.opendev.org/#/c/728638
Change-Id: I9b6bf5b6690f4b4b3445e7d15a40e45dd42d2e84

Ansible 2.9.8 includes a regression on the fileglob plugin [1] that
causes the HAProxy role to fail.

This change blacklists Ansible 2.9.8 to work around the issue.

[1] https://github.com/ansible/ansible/issues/69450

Change-Id: I12ca3b154fc7fed6a221880596e0acb5f6278bb7
Related-Bug: #1878192

This allows you to extend lists in yaml config. This is useful, for
example, in prometheus.yml, where it would be nice to be able to
extend the scrape_configs to include exporters that aren't packaged
with kolla-ansible. This would provide a mechanism to do so.

Change-Id: I7a10e363f42e8ffaae3c0d2c2a758853e2cab7e1
Related: blueprint custom-prometheus-targets

This causes non-local side effects that are hard to track down. E.g:

--- import errors ---
Failed to import test module: tests.test_merge_yaml
Traceback (most recent call last):
  File "/home/will/.pyenv/versions/3.7.7/lib/python3.7/unittest/loader.py", line 436, in _find_test_path
    module = self._get_module_from_name(name)
  File "/home/will/.pyenv/versions/3.7.7/lib/python3.7/unittest/loader.py", line 377, in _get_module_from_name
    __import__(name)
  File "/home/will/code/kolla-ansible/tests/test_merge_yaml.py", line 19, in <module>
    from ansible.errors import AnsibleModuleError
ModuleNotFoundError: No module named 'ansible.errors'; 'ansible' is not a package

This `'ansible' is not a package` message occurs because ansible is a Mock.

Depends-On: https://review.opendev.org/#/c/726768/
Change-Id: Iddbdd3d855daadbf12536cc990559e6b8e123051

Fixes:
- SB/NB DB address format (single host) for SB/NB DB daemon
- SB/NB DB address format (all hosts) for Neutron / northd /
  ovn-ovs bootstrap
- OVN tests

Change-Id: I539773c48f89b731d068280c228ce11782bf5788
Closes-Bug: #1875222

Zun has a new component "zun-cni-daemon" which should be
deployed in every compute nodes. It is basically an implementation
of CNI (Container Network Interface) that performs the neutron
port binding.

If users is using the capsule (pod) API, the recommended deployment
option is using "cri" as capsule driver. This is basically to use
a CRI runtime (i.e. CRI plugin for containerd) for supporting
capsules (pods). A CRI runtime needs a CNI plugin which is what
the "zun-cni-daemon" provides.

The configuration is based on the Zun installation guide [1].
It consits of the following steps:
* Configure the containerd daemon in the host. The "zun-compute"
  container will use grpc to communicate with this service.
* Install the "zun-cni" binary at host. The containerd process
  will invoke this binary to call the CNI plugin.
* Run a "zun-cni-daemon" container. The "zun-cni" binary will
  communicate with this container via HTTP.

Relevant patches:
Blueprint: https://blueprints.launchpad.net/zun/+spec/add-support-cri-runtime
Install guide: https://review.opendev.org/#/c/707948/
Devstack plugin: https://review.opendev.org/#/c/705338/
Kolla image: https://review.opendev.org/#/c/708273/

[1] https://docs.openstack.org/zun/latest/install/index.html

Depends-On: https://review.opendev.org/#/c/721044/
Change-Id: I9c361a99b355af27907cf80f5c88d97191193495

Debian defaults to Python2 which is not complete in aarch64 images.
This patch changes CI to always use Python3.

We need to install several Python modules to have working ussuri jobs.

"Failed to import the required Python library (setuptools) on primary's Python /usr/bin/python3."

And then several Python2 ones for train->ussuri upgrade jobs:

"Unable to find any of pip2 to use. pip needs to be installed."

Change-Id: Ia0d3ff15d97d1cabbb0b8e7f32e8712ca3f94732

Pipe hid an issue that Ironic client was not available in CI,
making it look as if fake-hardware driver was not usable.

This commit applies separation of concerns to easily discern
between the two failures and provide better debugging experience.

Change-Id: Iac3d84b7d707b84a8426c8d787b29a368ff7ae85
Related-Bug: #1872481

Removes and/or replaces all mentions of py27.

Cleans up obsolete requirements and their lower-constraints.

Separates test-requirements.

Makes lower-constraints pass outside of CI (MarkupSafe).

Adds FIXMEs about some hacky Mocks that may misbehave.

Change-Id: Ifc090bf3c1db17d8542ee591c91e8225a597bfe2

Bashate warned on init-swift.sh because of E043 -- arithmetic
compound has inconsistent return semantics: (( next_port++ ))
New Zuul reported that on every change as a warning.
This is fixed here.

This change makes Bashate always produce errors so that we do not
introduce such warnings again.

Change-Id: I40166b377ec2580e17901375b636183bca492d3a

Change-Id: I150c8952aad7b6fdb1260b343c26a96fb2360bce

Drops support for creating Python 2 virtualenvs in bootstrap-servers,
and looking for a python2 interpreter in the kolla-ansible script.

Also forces the use of Python 3 as the remote interpreter in CI on
Debian and Ubuntu hosts, since they typically symlink the unversioned
interpreter to python2.7.

Change-Id: Id0e977de381e7faafed738674a140ba36184727e
Partially-Implements: blueprint drop-py2-support

Followup of last reviews from [1].

[1]: https://review.opendev.org/#/c/696841

Change-Id: I7085093b20e8848e09dc521ae9fbf120e909470d

We publish those images for a while.

Change-Id: Ifc157b43e87e4f77a1c70d98343bd0ef9cc0de79

AArch64 does not have a way to get cpu features from libvirt.

Change-Id: Ieed404e17e8a9829f38c03f7ee7fdb3caa3919e8

AArch64 needs a bit more time to boot testing instance. So give it more
time by doubling amount of connect attempts.

Change-Id: I87ca65691dfbac84349e8af24d2f36f1db1c8be1