Change from file to template

Closes-Bug: #2089229
Change-Id: I7cbc6a94608baf4f04ef231cc88397fc5dcf0a9b
(cherry picked from commit f30dd3e5)

Most roles are not leveraging the jinja filters available.
According to [1] filtering the list of services makes the execution
faster than skipping the tasks.

This patchset also includes some cosmetic changes to genconfig.
Individual services are now also using a jinja filter. This has
no impact on performance, just makes the tasks look cleaner.

Naming of some vars in genconfig was changed to "service" to make
the tasks more uniform as some were previously using
the service name and some were using "service".

Three metrics from the deployment were taken and those were
- overall deployment time [s]
- time spent on the specific role [s]
- CPU usage (measured with perf) [-]
Overall genconfig time went down on avg. from 209s to 195s
Time spent on the loadbalancer role went down on avg. from 27s to 23s
Time spent on the neutron role went down on avg from 102s to 95s
Time spent on the nova-cell role went down on avg. from 54s to 52s
Also the average CPUs utilized reported by perf went down
from 3.31 to 3.15.
For details of how this was measured see the comments in gerrit.

[1] - https://github.com/stackhpc/ansible-scaling/blob/master/doc/skip.md



Change-Id: Ib0f00aadb6c7022de6e8b455ac4b9b8cd6be5b1b
Signed-off-by: Roman Krček <roman.krcek@tietoevry.com>

This patch is adding a feature for an option to copy different
ceph configuration files and corresponding keyrings for cinder,
glance, manila, gnocchi and nova services.

This is especially useful when the deployment uses availability
zones as below example.

  - Individual compute can read/write to individual ceph
    cluster in same AZ.
  - Cinder can write to several ceph clusters in several AZs.
  - Glance can use multistore and upload images to
    several ceph clusters in several AZs at once.

Change-Id: Ie4d8ab5a3df748137835cae1c943b9180cd10eb1

mainly jinja spacing and jinja[invalid] related

Change-Id: I6f52f2b0c1ef76de626657d79486d31e0f47f384

In some cases it may be desirable to run the libvirt daemon on the host.
For example, when mixing host and container OS distributions or
versions.

This change makes it possible to disable the nova_libvirt container, by
setting enable_nova_libvirt_container to false. The default values of
some Docker mounts and other paths have been updated to point to default
host directories rather than Docker volumes when using a host libvirt
daemon.

This change does not handle migration of existing systems from using
a nova_libvirt container to libvirt on the host.

Depends-On: https://review.opendev.org/c/openstack/ansible-collection-kolla/+/830504

Change-Id: Ia1239069ccee39416b20959cbabad962c56693cf

Consistently use template instead of copy. This has the added
advantage of allowing variables inside ceph conf files and keyrings.

Closes-Bug: 1959565

Signed-off-by: Imran Hussain <ih@imranh.co.uk>
Change-Id: Ibd0ff2641a54267ff06d3c89a26915a455dff1c1

Change-Id: Ib6719a033b37be3e248b682795b7243c60b22b84

Change-Id: I500cc8800c412bc0e95edb15babad5c1189e6ee4

Introduce user modifiable variables instead of fixed-names
of Ceph keyring files for external Ceph functionality.

Change-Id: I1a33b3f9d6eca5babf53b91187461e43aef865ce

Since [1] nova-compute uses rbd python library instead of libvirt to cleanup
volumes and get pool info - so it requires cinder keyring on filesystem.

In external ceph case it is often that nova key does not exist (is simply a copied
cinder key) and the rbd user is set to cinder - therefore the earlier mentioned
operations will fail due to a missing keyring on the filesystem.

[1]: https://review.opendev.org/#/c/668564/

Change-Id: Idef21dc5f7e9ff512bc8920630a3de61a1e69eee
Backport: train
Closes-Bug: #1859408

When using external ceph without nova integration, kolla-ansible fails
because 'nova_cephx_raw_key' is undefined.
This patch fixes the issue by applying the 'default' filter to
prevent failure on undefined.
The change in behavior was introduced by [1].

[1] https://review.opendev.org/689753


commit 44709f41

Change-Id: I2fdca1a6a78e78623733a387a2d8c7e29d449083
Closes-Bug: #1853862
Co-Authored-By: Mark Goddard <mark@stackhpc.com>

As part of the effort to implement Ansible code linting in CI
(using ansible-lint) - we need to implement recommendations from
ansible-lint output [1].

One of them is to stop using local_action in favor of delegate_to -
to increase readability and and match the style of typical ansible
tasks.

[1]: https://review.opendev.org/694779/

Partially implements: blueprint ansible-lint

Change-Id: I46c259ddad5a6aaf9c7301e6c44cd8a1d5c457d3

Cephx keys are not picked up by "local_action: shell cat [...]"
when using ansible-vault encrypted keyrings.
This commit changes the logic to use the file lookup plugin and
extracts the key using jinja2 regex filters. The raw keys are then
set as ansible facts.

Closes-Bug: 1849127

Change-Id: Iacb1e42307c4de6a7a379e8cf279e073995fd5d3

This patch adds initial support for deploying multiple Nova cells.

Splitting a nova-cell role out from the Nova role allows a more granular
approach to deploying and configuring Nova services.

A new enable_cells flag has been added that enables the support of
multiple cells via the introduction of a super conductor in addition to
cell-specific conductors. When this flag is not set (the default), nova
is configured in the same manner as before - with a single conductor.

The nova role now deploys the global services:

* nova-api
* nova-scheduler
* nova-super-conductor (if enable_cells is true)

The nova-cell role handles services specific to a cell:

* nova-compute
* nova-compute-ironic
* nova-conductor
* nova-libvirt
* nova-novncproxy
* nova-serialproxy
* nova-spicehtml5proxy
* nova-ssh

This patch does not support using a single cell controller for managing
more than one cell. Support for sharing a cell controller will be added
in a future patch.

This patch should be backwards compatible and is tested by existing CI
jobs. A new CI job has been added that tests a multi-cell environment.

ceph-mon has been removed from the play hosts list as it is not
necessary - delegate_to does not require the host to be in the play.

Documentation will be added in a separate patch.

Partially Implements: blueprint support-nova-cells
Co-Authored-By: Mark Goddard <mark@stackhpc.com>
Change-Id: I810aad7d49db3f5a7fd9a2f0f746fd912fe03917

The idea is to factor out a role for deploying Nova related services
to cells. Since all deployments use cells, this role can be used
in both regular deployments which have just cell0 and cell1,
and deployments with many cells.

Partially Implements: blueprint support-nova-cells
Change-Id: Ib1f36ec0a773c384f2c1eac1843782a3e766045a

Currently, we have a lot of logic for checking if a handler should run,
depending on whether config files have changed and whether the
container configuration has changed. As rm_work pointed out during
the recent haproxy refactor, these conditionals are typically
unnecessary - we can rely on Ansible's handler notification system
to only trigger handlers when they need to run. This removes a lot
of error prone code.

This patch removes conditional handler logic for all services. It is
important to ensure that we no longer trigger handlers when unnecessary,
because without these checks in place it will trigger a restart of the
containers.

Implements: blueprint simplify-handlers

Change-Id: I4f1aa03e9a9faaf8aecd556dfeafdb834042e4cd

They are used only to obtain keys for the next task.

Change-Id: I2fac22af4710b70e4df8e3a272bcfb6cc8b8532e
Signed-off-by: Radosław Piliszek <radoslaw.piliszek@gmail.com>

When Nova, Glance, or Cinder are deployed alongside an external Ceph deployment
handlers will fail to trigger if keyring files are updated, which results in the
containers not being restarted.

This change adds the missing 'when' conditions for nova-libvirt, nova-compute,
cinder-volume, cinder-backup, and glance-api containers.

Change-Id: I8e183aac9a72e7a7210f7edc7cdcbaedd4fbcaa9

When ceph.conf changed, we need restart some containers.

Change-Id: Iddeaf9dd4f288165fcef288e5384d79b61a0910b
Closes-Bug: #1810010

If upgrading the nova, cinder or manila services via 'kolla-ansible
upgrade', the Ceph config files are not generated. Users will expect
that these files are generated, to pull in any changes from their
configuration or the base kolla configuration.

This change moves Ceph tasks inside config.yml to ensure that they are
performed during deploy, reconfigure and upgrade. This has been done for
nova, cinder, gnocchi and manila - glance already does this.

Change-Id: Ic75692c2bcba9b81dee922ff6fbbccd160e7fa19
Closes-Bug: #1794275

Various ceph-related tasks were missing a 'become' that would allow them
to work as a non-root user. This seems to only cause a problem after an
initial deployment, perhaps due to the recursive ownership & permissions
changes at the end of the ceph.yml and external_ceph.yml files.

This change adds the necessary becomes.

Change-Id: I887c7b3bdef49db1dd1bf9e5bdbf5dc47b7f41af
Closes-Bug: #1795125

It is not necessary to enable the nova cephfs backend to make use
of cinder cephfs volumes.

Change-Id: I35c3d2e49769962e5c47f585d91d1efd492a53d6
Closes-Bug: #1778107

Change-Id: Ice5b058fe17ad7d648f21d1ddacaa339c6f7102f

Change-Id: I6b6114e85a4df82458c441f471929513c2526c23

Add become to only neccesary tasks in roles:
- glance
- heat
- horizon
- keystone
- neutron
- nova
- openvswitch

Gate is also updated to use 'become' feature

Change-Id: I2f3f27306e9f384148e1ad4d54d8da2ebef34d00
Partial-Implements: blueprint ansible-specific-task-become

It is external_ceph.yml for cinder/glance/gnocchi.

Change-Id: I534bffe5fc914e2850fb5cb5719fd99afa06c6c4

In order to speed up deployment time some "local" actions should be run
only once using 'run_once: True'.
This will decrease deployment time in case of multihost configuration.

Change-Id: I6015d772d35c15e96c52f577013b6e41197cb41a

If used external ceph for nova,the ceph storage not enable cephx.
So ceph keyring file not does not exist.
Task throw error of check ceph keyring files.

Change-Id: I6257c107b94abf4d363e854229aaab8301d1d694
Closes-Bug: #1684522

Closes-Bug: #1671862
Change-Id: I6b77773238cd65a3cc009d8d9a38bdbea0035212

Nova external ceph task have a type which break deployment.
State module not present, the module used should be stat.

Change-Id: Ie8a0b30f44fc35a597334383a85353d324e765cd
Closes-Bug: #1671526

Booting from volume require cinder's ceph client secret now. Move cinder
before nova in site.yml, because nova depends on cinder ceph client key
now.

Change-Id: I01c9ed80843d98305b8963894c4917c21a35d3ac
Closes-Bug: #1670676

Change-Id: I63197f8c5646e44a9a7287e644c904a1e227af23

Introduced nova backend selection flag for Ceph and priority if
multiple backends are configured

Add mechanism to deploy arbitrary ceph.conf and keyring files into
nova-compute and nova-libvirt containers

Added documentation

Change-Id: Id010ca9cc2d914e5358ef79edeb600a28220dd4b
Implements: blueprint external-ceph