Second part of patchset:
https://review.opendev.org/c/openstack/kolla-ansible/+/799229/


in which was suggested to split patch into smaller ones.

THis change adds container_engine to module parameters
so when we introduce podman, kolla_toolbox can be used
for both engines.

Signed-off-by: Ivan Halomi <i.halomi@partner.samsung.com>
Co-authored-by: Martin Hiner <m.hiner@partner.samsung.com>
Change-Id: Ic2093aa9341a0cb36df8f340cf290d62437504ad

This patch adds initial support for deploying multiple Nova cells.

Splitting a nova-cell role out from the Nova role allows a more granular
approach to deploying and configuring Nova services.

A new enable_cells flag has been added that enables the support of
multiple cells via the introduction of a super conductor in addition to
cell-specific conductors. When this flag is not set (the default), nova
is configured in the same manner as before - with a single conductor.

The nova role now deploys the global services:

* nova-api
* nova-scheduler
* nova-super-conductor (if enable_cells is true)

The nova-cell role handles services specific to a cell:

* nova-compute
* nova-compute-ironic
* nova-conductor
* nova-libvirt
* nova-novncproxy
* nova-serialproxy
* nova-spicehtml5proxy
* nova-ssh

This patch does not support using a single cell controller for managing
more than one cell. Support for sharing a cell controller will be added
in a future patch.

This patch should be backwards compatible and is tested by existing CI
jobs. A new CI job has been added that tests a multi-cell environment.

ceph-mon has been removed from the play hosts list as it is not
necessary - delegate_to does not require the host to be in the play.

Documentation will be added in a separate patch.

Partially Implements: blueprint support-nova-cells
Co-Authored-By: Mark Goddard <mark@stackhpc.com>
Change-Id: I810aad7d49db3f5a7fd9a2f0f746fd912fe03917

The idea is to factor out a role for deploying Nova related services
to cells. Since all deployments use cells, this role can be used
in both regular deployments which have just cell0 and cell1,
and deployments with many cells.

Partially Implements: blueprint support-nova-cells
Change-Id: Ib1f36ec0a773c384f2c1eac1843782a3e766045a

A common class of problems goes like this:

* kolla-ansible deploy
* Hit a problem, often in ansible/roles/*/tasks/bootstrap.yml
* Re-run kolla-ansible deploy
* Service fails to start

This happens because the DB is created during the first run, but for some
reason we fail before performing the DB sync. This means that on the second run
we don't include ansible/roles/*/tasks/bootstrap_service.yml because the DB
already exists, and therefore still don't perform the DB sync. However this
time, the command may complete without apparent error.

We should be less careful about when we perform the DB sync, and do it whenever
it is necessary. There is an argument for not doing the sync during a
'reconfigure' command, although we will not change that here.

This change only always performs the DB sync during 'deploy' and
'reconfigure' commands.

Change-Id: I82d30f3fcf325a3fdff3c59f19a1f88055b566cc
Closes-Bug: #1823766
Closes-Bug: #1797814

Many tasks that use Docker have become specified already, but
not all. This change ensures all tasks that use the following
modules have become:

* kolla_docker
* kolla_ceph_keyring
* kolla_toolbox
* kolla_container_facts

It also adds become for 'command' tasks that use docker CLI.

Change-Id: I4a5ebcedaccb9261dbc958ec67e8077d7980e496

When ansible goes in to a loop, by default it prints all the keys for
the item it is looping over. Some roles, when setting up the databases,
iterate over an object that includes the database password.

Override the loop label to hide everything but the database name.

Change-Id: I336a81a5ecd824ace7d40e9a35942a1c853554cd

At the moment the "databases user and setting permissions" task for
designate and nova leaks the database_password because of the use
of with_items:

---snip---
TASK [nova : Creating Nova databases user and setting permissions] *********************************************************
ok: [x -> y] => (item={u'database_password': u'password', u'database_name': u'nova', u'database_username': u'nova'})
ok: [x -> y] => (item={u'database_password': u'password', u'database_name': u'nova_cell0', u'database_username': u'nova'})
ok: [x -> y] => (item={u'database_password': u'password', u'database_name': u'nova_api', u'database_username': u'nova_api'})
---snap---

Change-Id: I141e4153223c8772c82a31d81e58057ce266c0b9
Co-authored-by: Bernd Müller <mueller@b1-systems.de>

when creating Nova databases user and setting permissions,
no need to register the database_user_create, and it used
nowhere, remove it is safe

Change-Id: If456b7c2ed25aa729be7d98ef875230c66581d65

include is marked as deprecated since ansible 2.4[0]

[0] https://docs.ansible.com/ansible/2.4/include_module.html#deprecated



Co-Authored-By: confi-surya <singh.surya64mnnit@gmail.com>
Change-Id: Ic9d71e1865d1c728890625aeddf424a5734c0a8a

This change prevents the task running if the database
has not been changed when a preconfigured external
database is not in use. This was the behaviour before
the preconfigured database feature was introduced.

Close-Bug: 1763375
Change-Id: I4f68905732dac75bc0deddab88047c9c0d7d65b5

When the virt dirver is xenapi, we should explicitly specify host
with dom0's hostname for nova-compute. The host field should be same
as neutron-openvswitch-agent-xenapi.

In order to use dom0's hostname, we need pull the bootstrap_xenapi
to before running nova config tasks. The dom0's hostname will be
included in the XenAPI facts.

Change-Id: Ibfc964bbd6c569062ad33c1399f9a4612baf5b92
blueprint: xenserver-support

This change allows the following use cases:

1. Using an already-configured MariaDB / MySQL server / Cluster
2. Using already-created DB users, without requiring root DB access.

Update: added external mariadb precheck

Change-Id: I78b0d178306d7c5293b0bf53e445f19f18b4b824
Implements: blueprint external-mariadb-support.
Closes-Bug: #1603121

This is to add a bootstrap task for XenAPI to bootstrap the compute
node. One compute node is composed with a XenServer and a DomU VM
running on the XenServer. Before it's ready for deploying OpenStack
sevices on it, we need bootstrap it firstly. The package of
python-os-xenapi contains a tool and some utils which were developed
to finish all of the needed bootstrap tasks
e.g.:
1. configure himn(host internal management network)
2. configure ipatables to allow traffic
3. install xapi plugins to dom0
4. gather XenAPI facts and save them into a file
...

The task added in this commit is to invoke that tool so that the
bootstrap tasks can be completed automatically during kolla deployment.

Depends-On: Ie2d7d40f2755580aac4a10f3d302190a8bd4fe6f
Change-Id: I1500535a1c9a085bcbfec5db5fbddcf040e6892d
blueprint: xenserver-support

when create database user, it should use
database_user, not database_name.

Change-Id: I4dfa01d1a5a46c5c58f1fc47b0be71b186462764
Closes-Bug: #1698762

Change-Id: I64dd52d909a884b2b9aea6f016d789d99caa3f17
Closes-Bug: #1652580

Change-Id: I00d2dcb0895548ba169ab85764cf546c2214cbf5

This change makes each step of the kolla deployment aware
of the port database was configured to listen on.
It defaults mariadb_port to database_port.

Change-Id: I8e85d5732015afc0a5481cb33e0b629fdfa84a1b
Closes-Bug: #1576151
DocImpact

This creates tree and playbook for nova upgrade. Also other service
upgrades will follow standard setup here.

Change-Id: Ic31759efaee4986eb87b9ff0968f13189d130d48
Partially-Implements: blueprint upgrade-kolla
Implements: blueprint upgrade-nova

Two patches merged that did not have teh correct name for
kolla_toolbox

TrivialFix

Change-Id: I09e230e29cff496247c32bf227715bfb7caa0a95

This is a required patch.

Kilo introduced a nova-api database but didn't use it. [1]
Mitaka now uses it and has broken booting instances. [2]

[1] https://wiki.openstack.org/wiki/ReleaseNotes/Kilo#OpenStack_Compute_.28Nova.29
[2] https://github.com/openstack/nova/blob/df0fca62cf5324f5b6eae0fed1f88c6c9ed61c71/releasenotes/notes/request-spec-api-db-b9cc6e0624d563c5.yaml

Change-Id: If0a7f4ba4937fab8d60f5236ca00db06faa768c7
Closes-Bug: #1540579

This change is needed for clarity. We have a kolla-ansible script.
We have a kolla-mesos repo. We plan to have a kolla-ansible repo.
Already we have had far too much confusion about whether we are
talking about the container or the project. Naming this kolla-toolbox
eliminates all of that confusion and its probably a bit more accurate
of a name too.

Closes-Bug: #1541053
Change-Id: I8fd1f49d5a22b36ede5b10f46b9fe02ddda9007e

Add bootstrap label to all bootstrap containers to ensure that when
the a new container is launched a difference is seen between it and
the bootstrap container since we cannot rely on ENV variables for
this. This only affects mariadb at this stage, but it is needed to
ensure rabbitmq works when we switch to named volumes.

Change-Id: Ia022af26212d2e5445c06149848831037a508407
Closes-Bug: #1538136

Due to changes with the drop-root work, we lost the ability to write
to /var/lib/nova/*. This fixes those permissions and ensures cross
container talk works properly between nova_libvirt and nova_compute

Additionally, this fixes another issue introduced which saw that
nova-compute could not run sudo commands as it did not have a proper
sudoers entry

Testing from previous deploys means you need a fresh environment. You
have to remove all of the named volumes that kolla created in docker.
Check these with `docker volume ls`

Signed-off-by: Hui Kang <kangh@us.ibm.com>
Signed-off-by: Artur Zarzycki <azarzycki@mirantis.com>
Co-Authored-By: Sam Yaple <sam@yaple.net>
Co-Authored-By: Hui Kang <kangh@us.ibm.com>
Closes-Bug: #1533350
Change-Id: I7f864c448a2414e0b5d89f48337be411b891df35

Change-Id: I0c6df8e7b732a37ff3cabff8d588f8b7d15df192
Partially-Implements: blueprint kolla-docker-module

Additionally remove tty from the container cleanup docker section. It
was added in a sed in a previous patchset by serves no purpose.

Change-Id: Ib617870616bca687f72ffaa44b2e9a3a11ef1011
Partially-Implements: blueprint cleanup-playbooks

The bootstrap tasks could be started on any node listed in the
site.yml

The issue is that all the tasks must run on the same node, and the
only node that all the tasks can run on is the 'api' node due to
needing to start the bootstrap container which binds in the configs.

delegate_to is required to ensure that the proper node gets the task.

Co-Authored-By: Sam Yaple <sam@yaple.net>
Change-Id: Ica04ab366777a571a92212ac22c482f1881d4ecd
Closes-Bug: #1513598
Related-Bug: #1513439

The original reason for having the data container in the bootstrap.yml
was to keep consistency with rabbitmq and mariadb, but in light of
this bug we need to move the data container out of the bootstrap.yml
role since we will be limiting the hosts that those tasks can run on.

Change-Id: I7f2f5979b01807275908699d1243756cb97d3588
Partial-Bug: #1513598

database_user_create was not correctly referenced when parsing the
variable names. This could never actually lead to a situation that
reported a false change, but it could break an operation if you were
using the --step option with ansible and skipped the database create
task.

TrivialFix
Backport: Liberty

Change-Id: Idf69fffcc3814f509448ccea11b7d175f074ccf1

The bootstrap must occur on the nova-api node due to binding in the
nova-api directory (same goes for all other services)

Closes-Bug: #1513439
Backport: Liberty
Change-Id: Iab88b49712828085e4d7e7f85e6d8f0b7999a9bf

sudo requires a tty to function by default on centos. Instead of
tweaking the sudo conf we can just add a tty. This has the added
advantage of making the containers more friendly if you have to
`docker exec -i <container> bash` into them.

Change-Id: If97a02ca1d37c243a787d98ade54bde8d641aecd
backport: liberty
Partially-Implements: blueprint functional-testing-gate

Ansible 1.9.2 contains the fix needed for docker-py >1.2.0

This is needed for some gate reasons, but it is also a good version
bump because it fixes a few issues with delegate_to.

Change-Id: Iafbabb3b0232620849d0548c5cd9d8d316c2b0f3
backport: liberty
Partially-Implements: blueprint functional-testing-gate

This brings Kolla images inline with FHS and should make finding
locations of things more consistent and reliable with the linux world
at large.

Change-Id: Iece5b4da4bace0fb8b1f41a65ab2c852ec73e6f8
Closes-Bug: #1485742

Change-Id: I38c545484a7b1f3906d844eb3a7486e11c4b276a
Partially-Implements: blueprint replace-config-external

Libvirt shuts down existing virtual machines.  To solve this problem,
the /var/lib/libvirt directory needs to be persisted.  By persisting
this directory in the nova data container, libvirt can communicate
with the sockets setup by the previous data container.

Change-Id: I362c533f8493984123686fe6e3f8e43d97434ca6
Closes-Bug: #1494060

Make nova-libvirt and nova-compute use a data container called
nova_data.  This data container contains /var/lib/nova/instances
data so upgrades to nova-compute work properly.

Change-Id: I89362eb0857f7b07503c35178475bc91d0d19ca8
Closes-Bug: #1486333

No need for a new task.

Change-Id: I958f87ac9a1434a2f6052e0271f37e5f3109775e
Closes-Bug: #1492337

Currently bootstrap containers are waited to exit but are not
checked for exit status and ansible runs further tasks. If
bootstrapping fails we notice it at much later time.

Change-Id: I137fc11b0f9d1f03d2ded08a213e8dbd62741f92
Closes-Bug: #1492337

Not quite halloween yet...

Change-Id: Ib1973eea0b2dddc0e0845731d1212c4d4447cca5
Closes-Bug: #1490502

Change-Id: Id6a82e6622f712c6fca8205357dec4e5161e46d2
Closes-Bug: #1487387

The register operation was registering user_database which is a global
variable in group_vars/all.yml.  This results in glance not being deployable
because user_database is overwrriten at keystone task registration time.

The result of this is the playbooks fail to deploy.

Change-Id: Ic69725fb09aa0a368ec5b0d8f47bfc768c29c70c
Closes-Bug: #1486876