Add support to remove images from kolla_docker
ansible module.

Change-Id: Ib6bcb98b5b295a2d590df3013188913d1f7f3584

This commit is to apply resource-constraints only to few OpenStack services.
Commit to apply constraints to other services will be made in coming commits.

Partially-Implements: blueprint resource-constraints

Change-Id: Icafa54baca24d2de64238222a5677b9d8b90e2aa

Partially-Implements: blueprint networking-baremetal

Change-Id: I92b9505843f12692aef96764a314e5db49001a9b

This commit will constrain the dimensions of service `Nova`
and sub-containers deployed along with it.

A user can give the dimension values in `/etc/kolla/globals.yml`
the data-types just like stated in this commit.

Reference-Docs:
https://docs.docker.com/config/containers/resource_constraints/

Added Test-cases for the same.

Partially-Implements: blueprint resource-constraints
Change-Id: I6458d8fb7b26a6e7c3a9fd0d674d9cf129b0bf5d

Support Kolla Ceph to deploy blustore OSDs with Kolla-ansible.

Please refer to [1] for bluestore OSD configuration

The patch includes:
1. Set Ceph OSD store type group_vars/all.yml. The default value
is "bluestore" in Rocky.

2. Make Kolla Ceph to deploy bluestore OSDs with Kolla-ansible

3. Update gate test configuration for Ceph bluestore OSD test

[1]: specs/kolla-ceph-bluestore.rst

Partially-Implements: blueprint kolla-ceph-bluestore
Depends-On: I00eaa600a5e9ad4c1ebca2eeb523bca3d7a25128
Change-Id: I14f20a00654dff32c36d078ebb9005d91a3e60b2
Signed-off-by: Tone Zhang <tone.zhang@arm.com>

Deploys a bifrost container using kolla-ansible bifrost-deploy.

IPA and disk images are downloaded rather than built to improve
reliability.

Currently only minimal testing of the deployment is performed, creating
and deleting an ironic node. Ideally we would perform a bare metal node
deployment.

The job is based on CentOS, as Ubuntu bifrost deployment is currently
failing with a python-MySQLdb error.

Change-Id: Ic45094594c21116b5b0d6a606f568fc7954175e3

cAdvisor (Container Advisor) provides metrics on resource usage and
performance characteristics of running containers.  This change
deploys a cadvisor container and configures prometheus to scrape
data from it.

Change-Id: I55dd4fee954f9be68efda397746861ddaaa0a565
Partially-Implements: blueprint prometheus

When enable_ironic_ipxe is set in /etc/kolla/globals.yml,
the following happens:

- a new docker container, ironic_ipxe, is created. This contains
  an apache webserver used to serve up the boot images
- ironic is configured to use ipxe

Change-Id: I08fca1864a00afb768494406c49e968920c83ae7
Implements: blueprint ironic-ipxe

Apply Glance database upgrade procedure:

https://docs.openstack.org/glance/latest/admin/zero-downtime-db-upgrade.html

Rolling upgrade in Glance requires that glance-registry is not deployed and run

Depends-On: https://review.openstack.org/532128
Depends-On: https://review.openstack.org/566804

Change-Id: Ia5d0cee2385facefda9d423051b74ab51d075f13
Implements: blueprint apply-service-upgrade-procedure

refer to https://review.openstack.org/#/c/564169/

Change-Id: I093b0b8fd3b34c9b35149c9e4ae945a8e0e67f53

This patch adds the ansible role to deploy the prometheus service which
can be used to collect performance metrics accross the environment

Partially-Implements: blueprint prometheus
Change-Id: I908b9c9dad63ab5c9b80be1e3a80a4fc8191cb9e

This patch allows configuration of the Infoblox
pluggable IPAM driver in neutron [0].

When 'infoblox' is chosen as the driver, an Infoblox
IPAM agent can be started as well.  The agent
allows for enhanced DNS capabilities by listening
for neutron and nova notifications.

[0] https://github.com/openstack/networking-infoblox/blob/master/README.rst

Change-Id: I4f863750a7806a7b6eaf13900d44e5f063afe3de
Depends-On: Ia44f0e0d7a0d60cebf0857ad51700e02eba5099b
Partially-Implements: blueprint neutron-ipam-driver-infoblox

In old docker, if you do not specify ipc_mode, the default value is empty,
but in the latest docker, such as 17.09.0, if not specified, the default
is "IpcMode": "shareable", which will cause all containers to be deleted
and re-create when to redeploy or upgrade. This commit solves the
problem.

Change-Id: Ia8269b9c8066880e4aee23d6fdea8d9c04c41e44
Closes-Bug: #1747586

This feature replaces splitting of compute hosts into inner
and external with a new variable 'neutron_compute_dvr_mode'
that controls whether computes will have full-blown DVR or
internal only (tenant) networking.

Change-Id: I6720ccfcfcec89f9996d4cb5ae60f31eb3113667
Implements: blueprint dvr-mode-property

More info please check[0][1].

[0] http://lists.openstack.org/pipermail/openstack-dev/2018-March/128902.html
[1] https://ara.readthedocs.io/en/latest/advanced.html

Change-Id: I5cf68c0bb54878e328387143712dec9582be04a1

This patchset implements yamllint test to all *.yml
files.

Also fixes syntax errors to make jobs to pass.

Change-Id: I3186adf9835b4d0cada272d156b17d1bc9c2b799

Services such as Zookeeper and Kafka use ini files which do not
explicitly specify sections. This change supports merging ini
files with no sections, so that the configuration for these
services follows the example configuration provided for them
as closely as possible.

Closes-Bug: #1756101
Partially-Implements: blueprint monasca-roles
Change-Id: I1061729875e5545c7af7d80779f9c2124b6c7134

Till now we've been flusing iptables in the gates to allow cross node
communication in the multi node ceph jobs. This raised security
concerns, in particular it exposed memcached to the external net.

This patch uses the infra provided role 'multi-node-firewall' in order
to correctly configure iptables. Thanks to Jeremy Stanley and Jeffrey
for help with this.

Closes-Bug: #1749326
Change-Id: Iafaf1cf1d9b0227b0f869969d0bd52fbde3791a0

This reverts commit 2a4ce0b8.

Change-Id: I52efd29791a230df1ce7e3acde4b7184aec34150

We have pin keyston to queens release which supports UUID token through
https://review.openstack.org/546475, let us use UUID in queens and
migrate to fernet in rocky cycle.

This reverts commit df0bf190.

Change-Id: Ifb0112315b5047461ce0bf02c754cc0beac52d9a

we have gates currently broken due using rocky keystone code.
we have no way to test our release until we branch out queens.
this change doesnt change globals default, only our gates tests.

Change-Id: I189aa8cafc73a365668d79e4b186aac21da63a6f

Missing container status check in recreate_or_restart_container,
this causes if the container is not running (kolla-ansible stop),
to not be started with deploy/reconfigure/upgrade if any other param
changes.

Change-Id: I5cff5f367e963ba8b1807ec46469da817e40e468
Closes-Bug: #1714015

- Ceph images are not being built when using depends-on a kolla build
  job.
- Sync inventory files with current ones which causes ceph to fail due
  missing groups.
- Small corrections in ceph config.yml syntax.
- fix preparse the disk only once
- enable ceph nfs only when enable_ceph_nfs is true

Co-Authored-By: Jeffrey Zhang <zhang.lei.fly@gmail.com>
Change-Id: Id0c7963bf59e2af4944834dcd16589a638e78ba5

Use the same router will result in conflict in the CI environment.

Change-Id: Id1ba69ebbf67b4cead855adbe440343c05ea3d1e
Closes-Bug: #1744913

By default, the ansible task will gather facts, it can be configured by
set "father_facts: true", it's unncessary to run setup in tasks, it will
waste much time when hosts is large.

Change-Id: I9462a538c76b6271a34531d8752b72cc695f693e
Closes-Bug: #1743485

Change-Id: Ifdf2ede2cb5f057d1d1f02bfa00ee05872e94b9f

multinode gate change introduced bug that breaks kolla deploy gates.
This is fix.

Change-Id: Ibc5fe3adb3555e177d3a6a64250cbc9bf50430d1

Another step to migrate to new CI

Change-Id: Id9bf9b4b1114f5be66f6c028a0db16d869efeceb
Depends-On: Iaacdca31b27b7033ec08499bcea7061b9d3ec279

Change-Id: Id71c1ecda9f8647a0eb290fe02a12afb3df09988

- move check container failure from post.yml to run.yml
- add binary related jobs
- use static kolla-ansible src dir, which is helpful for kolla project
  to use.
- generate correct /etc/hosts by using private ip address and hostname
- fix the wrong api interface in global.yml file

Change-Id: Idfdee6dfe18f0fa2d4f984df59b57553122ce298

Partial-Bug: #1720601
Change-Id: Ibc20a6ae8c645ff82f3c14a6286073dffd4cfae2

Fluentd send logs to stdout,
this changes creates a file with fluentd logs
output to easy discover missing patterns during fluentd
changes.

Change-Id: I131f95089eac60ccb4c48cf5071c3b44c5ea42ca

Check status of ceph cluster

Change-Id: I4919a32794cc75bd3e8f411a219f778238a334ee

Ceph gates notoriously run out of memory

Change-Id: I5cf5cc5adf5379095ed9e3094494e65b76ff62d8

Docker has an issue[0] when restart container. But stop then start
works.

[0] https://github.com/moby/moby/issues/29704

Change-Id: If0a9c0c257cd72209be8e138a1f0b8871500e089
Closes-Bug: #1707097

Log retrieval was out of sync since repo split
and from multinode gates.

Many useful information retrieved before like
ps, df, docker info, etc is not in kolla-ansible
gates.

Also, his change fix logs visualization to have
colored view, allowing to easily identify errors.

Change-Id: I948233e26ceb6efc58b962bcb4b710b3f006232b

Ansible check if modules parameters are named like
%password% and allow to hidden log param in this case.
This requires adding "no_log" parameters.
This patch just add "no_log" param in order to avoid
this warning.

Change-Id: I9c1df1093e0fd101090292d6e8bf3527f99aeb17
Closes-Bug: #1702244

The pypi package 'docker-py' [1] has been renamed to 'docker' [2].
It is better to move to the new 'docker' package because the old
package will be deprecated and all the new features will go into
the new package only.

Package 'docker' has been added to requirements [3]. The old
package 'docker-py' is still allowed to be in the global requirements
during the transition period but it should be removed after all or
most of the projects finsih the migration.

[1] https://pypi.python.org/pypi/docker-py
[2] https://pypi.python.org/pypi/docker
[3] https://review.openstack.org/#/c/423715/

Change-Id: Ibcd5a57a1fbf55dcc5a690e41f20917f95b63da0

First non-trivial scenario we'll be testing in gates will be cinder +
ceph. This patchset is supposed to create framework to easily add new
testing scenarios.

Change-Id: I6790eaf5cf3297af04a64e4169a39b6e2cd1890f

This reverts commit 94a882ba.

Upstream has fix this issue by Ice89ad9dd486ad5fcac534ef5f7d8aae3b6b0f97

Change-Id: Ib17f93e4030c6f7da4e32ee84fcc5f7139e57895
Closes-Bug: #1694420