Change-Id: If5b4ba975a65e07d2704eb6bdb9d841d6a9c3d42

Nova services may reasonably expect cell databases to exist when they
start. The current cell setup tasks in kolla run after the nova
containers have started, meaning that cells may or may not exist in the
database when they start, depending on timing. In particular, we are
seeing issues in kolla CI currently with jobs timing out waiting for
nova compute services to start. The following error is seen in the nova
logs of these jobs, which may or may not be relevant:

No cells are configured, unable to continue

This change creates the cell0 and cell1 databases prior to starting nova
services.

In order to do this, we must create new containers in which to run the
nova-manage commands, because the nova-api container may not yet exist.
This required adding support to the kolla_docker module for specifying a
command for the container to run that overrides the image's command.

We also add the standard output and error to the module's result when a
non-detached container is run. A secondary benefit of this is that the
output of bootstrap containers is now displayed in the Ansible output if
the bootstrapping command fails, which will help with debugging.

Change-Id: I2c1e991064f9f588f398ccbabda94f69dc285e61
Closes-Bug: #1808575

This change adds support to comfigure tty,
it was enabled by default but a recent patch
removed it. Some services such as Karaf in opendaylight
requires a TTY during startup.

Closes-Bug: #1806662
Change-Id: Ia4335523b727d0e45505cbb1efb40ccf04c27db7

the bug[0] has beed fix, restart container is unncessary, remove it
is safe.

[0]: https://bugs.launchpad.net/oslo.cache/+bug/1590779

Change-Id: Iadfb3666bad8160b5c071fa1a512e2dcdda84043

Refer to neutron docs [1][2]
[1]: https://docs.openstack.org/neutron/pike/admin/archives/adv-config.html#l3-metering-service-driver
[2]: https://wiki.openstack.org/wiki/Neutron/Metering/Bandwidth


Co-Authored-By: ZhijunWei <wzj334965317@outlook.com>

Change-Id: I4a676d041bc2a86497cb139d9347365738b156df

Change-Id: I01e58d3548d6adc4a2d6f1088773df7941da3865

Switch the CI test script to use the openstack command rather than nova
etc.

Change-Id: Id1c6d35e1e50987d230adb7942da6a7ae5c0f050

A spec to Deprecate the Glance Registry Service[0] was accepted in Newton,
but it contained the ambiguous statement, "Mark the service as deprecated
and ready for removal in the Q release." kolla-ansible disable the
glance-registry in Q release[1], and since we are in S now,
remove glance-registry is safe.

[0]: http://specs.openstack.org/openstack/glance-specs/specs/newton/approved/glance/deprecate-registry.html
[1]: https://review.openstack.org/#/c/566804/

Change-Id: I48f794029e97aa6f76bbd500e33f28f51a3f2ac4

Improve where openstack services are deployed
to a variable, so we will avoid many conditions
when more CI jobs are added with different configs.
Heat is enabled by default in globals.yml, lets deploy it
at least.

Change-Id: If0f009cf434c3ed5874b3171d8c75911ef8e4be1

This PS adds deployment of mistral-event-engine service
within mistral role.

Change-Id: If187e1a49a321d1e85fb3cfff2ea037e567b30dc

This change is intended to ease the addition of new test scenarios, by
splitting the existing setup_gate.sh script up into several smaller
scripts. We may wish to further break these scripts up in future,
depending on the requirements of new test jobs.

Change-Id: Ifa442ac79a54f749c56d20f5f7383c3852d95268

Closes-Bug: 1786741
Change-Id: Ic217f3f8194cdcb5e28045e771a0160c1f12280e

With a pseudo terminal, service is not treated as a daemon
and signals would not work as expected.

Change-Id: I16aa29a7924df51659d973a81d8005ae3d86f57b
Related-Bug: #1799642

zun-wsproxy group is used in multinode[1], but it missing from
site.yml this PS to fix this.

[1]: https://github.com/openstack/kolla-ansible/blob/master/ansible/inventory/multinode#L676

Change-Id: I3b06f15dba5b6569202b038a28fb1a2728cffd25
Closes-Bug: #1799234

Introduce a job 'kolla-ansible-ubuntu-source-zun' to test kolla
with Zun enabled. To reduce CI resource, this job will be triggered
only if there are changes on the Zun's ansible roles.

Change-Id: I0ba207e1d3761da2d6992c5834d4f59e7e1d6628

Kolla Ansible now claims [1] to support executing as a user other than
root.  We should ensure that this is tested in CI.

This change removes the 'become' from hosts in the inventory, and sets
the remote user to 'kolla', as configured via the bootstrap-servers
command. The bootstrap-servers command and other ansible commands
executed before it still need to execute as the zuul user and not as
kolla, since kolla does not exist yet.

The autogenerated SSH private key in passwords.yml is now added to the
zuul user's SSH config, such that it can SSH as the kolla user, which
has authorised this key.

[1]
https://blueprints.launchpad.net/kolla-ansible/+spec/ansible-specific-task-become

Change-Id: I8f3587e2908bc5b8889cd6fbc01981a6186d33e6

If the main CI job fails before generating an ARA report, the SQLite
database file will not exist. This cases the job to fail with
POST_FAILURE, rather than FAILURE, and the following is seen in the
logs:

rsync: change_dir "/home/zuul/.ara" failed: No such file or directory
rsync error: some files/attrs were not transferred (see previous errors)
(code 23) at main.c(1655) [Receiver=3.1.1]
rsync: [Receiver] write error: Broken pipe (32)

This change fixes this by checking for an SQLite database file, and only
intiating the rsync transfer if it exists.

Change-Id: I370e5bc9f137abe552918a3215a025fa61e3a0ca

Depends-On: I75e00312b36e1678b90a42cf58d24652323eff27
Change-Id: Ia716fabffca41eff816e59bbf9f4cab79ee8b72f

Change-Id: I05960661b8efb32a3d57686dc3780d4138309ef5
Closes-Bug: #1793262

Add support to remove images from kolla_docker
ansible module.

Change-Id: Ib6bcb98b5b295a2d590df3013188913d1f7f3584

Since I701d495675178c3ed8ec1f00b31d09f198b38a6f merged, etcd only runs
on the control hosts, not the compute hosts. We therefore no longer
require the etcd group to include the compute hosts.

Since the group mapping is now static, we can remove the use of
host_in_groups from the etcd service, in favour of the simpler method of
specifying the group.

Change-Id: Id8f888d7321a30a85ff95e742f7e6c8e2b9c696f
Related-Bug: #1790415

The current bluestore disk label naming is inconsistent with the
filestore. The filestore naming format is that the disk prefixes
belonging to the same osd are the same and the suffixes are
different.

This patch keeps the bluestore's disk naming as well.

Change-Id: I71dda29fc4a6765300ce7bb173d2c448c24f6eca

In order to migrate to the latest release of rabbitmq (3.7), we need to
first remove this deprecated plugin which is no longer supported (the
problems it solved are now addressed in rabbitmq itself).

This avoids a circular dependency in CI where the new images depend on
the new clustering and the new clustering depends on the new images.

Change-Id: I921459f3e40b9e0d4af9497384e49aabf0abe79b

This commit is to apply resource-constraints only to few OpenStack services.
Commit to apply constraints to other services will be made in coming commits.

Partially-Implements: blueprint resource-constraints

Change-Id: Icafa54baca24d2de64238222a5677b9d8b90e2aa

Partially-Implements: blueprint networking-baremetal

Change-Id: I92b9505843f12692aef96764a314e5db49001a9b

This commit will constrain the dimensions of service `Nova`
and sub-containers deployed along with it.

A user can give the dimension values in `/etc/kolla/globals.yml`
the data-types just like stated in this commit.

Reference-Docs:
https://docs.docker.com/config/containers/resource_constraints/

Added Test-cases for the same.

Partially-Implements: blueprint resource-constraints
Change-Id: I6458d8fb7b26a6e7c3a9fd0d674d9cf129b0bf5d

Support Kolla Ceph to deploy blustore OSDs with Kolla-ansible.

Please refer to [1] for bluestore OSD configuration

The patch includes:
1. Set Ceph OSD store type group_vars/all.yml. The default value
is "bluestore" in Rocky.

2. Make Kolla Ceph to deploy bluestore OSDs with Kolla-ansible

3. Update gate test configuration for Ceph bluestore OSD test

[1]: specs/kolla-ceph-bluestore.rst

Partially-Implements: blueprint kolla-ceph-bluestore
Depends-On: I00eaa600a5e9ad4c1ebca2eeb523bca3d7a25128
Change-Id: I14f20a00654dff32c36d078ebb9005d91a3e60b2
Signed-off-by: Tone Zhang <tone.zhang@arm.com>

Deploys a bifrost container using kolla-ansible bifrost-deploy.

IPA and disk images are downloaded rather than built to improve
reliability.

Currently only minimal testing of the deployment is performed, creating
and deleting an ironic node. Ideally we would perform a bare metal node
deployment.

The job is based on CentOS, as Ubuntu bifrost deployment is currently
failing with a python-MySQLdb error.

Change-Id: Ic45094594c21116b5b0d6a606f568fc7954175e3

cAdvisor (Container Advisor) provides metrics on resource usage and
performance characteristics of running containers.  This change
deploys a cadvisor container and configures prometheus to scrape
data from it.

Change-Id: I55dd4fee954f9be68efda397746861ddaaa0a565
Partially-Implements: blueprint prometheus

When enable_ironic_ipxe is set in /etc/kolla/globals.yml,
the following happens:

- a new docker container, ironic_ipxe, is created. This contains
  an apache webserver used to serve up the boot images
- ironic is configured to use ipxe

Change-Id: I08fca1864a00afb768494406c49e968920c83ae7
Implements: blueprint ironic-ipxe

Apply Glance database upgrade procedure:

https://docs.openstack.org/glance/latest/admin/zero-downtime-db-upgrade.html

Rolling upgrade in Glance requires that glance-registry is not deployed and run

Depends-On: https://review.openstack.org/532128
Depends-On: https://review.openstack.org/566804

Change-Id: Ia5d0cee2385facefda9d423051b74ab51d075f13
Implements: blueprint apply-service-upgrade-procedure

refer to https://review.openstack.org/#/c/564169/

Change-Id: I093b0b8fd3b34c9b35149c9e4ae945a8e0e67f53

This patch adds the ansible role to deploy the prometheus service which
can be used to collect performance metrics accross the environment

Partially-Implements: blueprint prometheus
Change-Id: I908b9c9dad63ab5c9b80be1e3a80a4fc8191cb9e

This patch allows configuration of the Infoblox
pluggable IPAM driver in neutron [0].

When 'infoblox' is chosen as the driver, an Infoblox
IPAM agent can be started as well.  The agent
allows for enhanced DNS capabilities by listening
for neutron and nova notifications.

[0] https://github.com/openstack/networking-infoblox/blob/master/README.rst

Change-Id: I4f863750a7806a7b6eaf13900d44e5f063afe3de
Depends-On: Ia44f0e0d7a0d60cebf0857ad51700e02eba5099b
Partially-Implements: blueprint neutron-ipam-driver-infoblox

In old docker, if you do not specify ipc_mode, the default value is empty,
but in the latest docker, such as 17.09.0, if not specified, the default
is "IpcMode": "shareable", which will cause all containers to be deleted
and re-create when to redeploy or upgrade. This commit solves the
problem.

Change-Id: Ia8269b9c8066880e4aee23d6fdea8d9c04c41e44
Closes-Bug: #1747586

This feature replaces splitting of compute hosts into inner
and external with a new variable 'neutron_compute_dvr_mode'
that controls whether computes will have full-blown DVR or
internal only (tenant) networking.

Change-Id: I6720ccfcfcec89f9996d4cb5ae60f31eb3113667
Implements: blueprint dvr-mode-property

More info please check[0][1].

[0] http://lists.openstack.org/pipermail/openstack-dev/2018-March/128902.html
[1] https://ara.readthedocs.io/en/latest/advanced.html

Change-Id: I5cf68c0bb54878e328387143712dec9582be04a1

This patchset implements yamllint test to all *.yml
files.

Also fixes syntax errors to make jobs to pass.

Change-Id: I3186adf9835b4d0cada272d156b17d1bc9c2b799

Services such as Zookeeper and Kafka use ini files which do not
explicitly specify sections. This change supports merging ini
files with no sections, so that the configuration for these
services follows the example configuration provided for them
as closely as possible.

Closes-Bug: #1756101
Partially-Implements: blueprint monasca-roles
Change-Id: I1061729875e5545c7af7d80779f9c2124b6c7134

Till now we've been flusing iptables in the gates to allow cross node
communication in the multi node ceph jobs. This raised security
concerns, in particular it exposed memcached to the external net.

This patch uses the infra provided role 'multi-node-firewall' in order
to correctly configure iptables. Thanks to Jeremy Stanley and Jeffrey
for help with this.

Closes-Bug: #1749326
Change-Id: Iafaf1cf1d9b0227b0f869969d0bd52fbde3791a0