This patch introduces an optional backend encryption for Keystone
service. When used in conjunction with enabling TLS for service API
endpoints, network communcation will be encrypted end to end, from
client through HAProxy to the Keystone service.

Change-Id: I6351147ddaff8b2ae629179a9bc3bae2ebac9519
Partially-Implements: blueprint add-ssl-internal-network

Kolla-Ansible Ceph deployment mechanism has been deprecated in Train [1].

This change removes the Ansible code and associated CI jobs.

[1]: https://review.opendev.org/669214

Change-Id: Ie2167f02ad2f525d3b0f553e2c047516acf55bc2

This allows users to supply an Elasticsearch Curator actions file
to manage log retention [1]. Curator then runs on a cron job, which
defaults to every day. A default curator actions file is provided,
which can be customised by the end user if required.

[1] https://www.elastic.co/guide/en/elasticsearch/client/curator/current/actionfile.html

Change-Id: Ide9baea9190ae849e61b9d8b6cff3305bdcdd534

Depends-On: https://review.opendev.org/692948/
Depends-On: https://review.opendev.org/692691/
Change-Id: I07827b896d36c3723697540fcff164224f6729af

Opendaylight support has been deprecated in Train - time to remove it.

Change-Id: I3a61bfbcbf366c327ea3e25d2424bc3fedca29f0

This patch adds initial support for deploying multiple Nova cells.

Splitting a nova-cell role out from the Nova role allows a more granular
approach to deploying and configuring Nova services.

A new enable_cells flag has been added that enables the support of
multiple cells via the introduction of a super conductor in addition to
cell-specific conductors. When this flag is not set (the default), nova
is configured in the same manner as before - with a single conductor.

The nova role now deploys the global services:

* nova-api
* nova-scheduler
* nova-super-conductor (if enable_cells is true)

The nova-cell role handles services specific to a cell:

* nova-compute
* nova-compute-ironic
* nova-conductor
* nova-libvirt
* nova-novncproxy
* nova-serialproxy
* nova-spicehtml5proxy
* nova-ssh

This patch does not support using a single cell controller for managing
more than one cell. Support for sharing a cell controller will be added
in a future patch.

This patch should be backwards compatible and is tested by existing CI
jobs. A new CI job has been added that tests a multi-cell environment.

ceph-mon has been removed from the play hosts list as it is not
necessary - delegate_to does not require the host to be in the play.

Documentation will be added in a separate patch.

Partially Implements: blueprint support-nova-cells
Co-Authored-By: Mark Goddard <mark@stackhpc.com>
Change-Id: I810aad7d49db3f5a7fd9a2f0f746fd912fe03917

This commit follows up the work in Kolla to provide deploy and configure the
Prometheus blackbox exporter.

An example blackbox-exporter module has been added (disabled by default)
called os_endpoint. This allows for the probing of endpoints over HTTP
and HTTPS. This can be used to monitor that OpenStack endpoints return a status
code of either 200 or 300, and the word 'versions' in the payload.

This change introduces a new variable `prometheus_blackbox_exporter_endpoints`.
Currently no defaults are specified because the configuration is heavily
dependent on the deployment.

Co-authored-by: Jack Heskett <Jack.Heskett@gresearch.co.uk>
Change-Id: I36ad4961078d90e2fd70c9a3368f5157d6fd89cd

Masakari provides Instances High Availability Service for
OpenStack clouds by automatically recovering failed Instances.

Depends-On: https://review.openstack.org/#/c/615469/


Change-Id: I0b3457232ee86576022cff64eb2e227ff9bbf0aa
Implements: blueprint ansible-masakari
Co-Authored-By: Gaëtan Trellu <gaetan.trellu@incloudus.com>

The nova-consoleauth service was deprecated during the Rocky release [1]
and has not been necessary since unless you're using cells v1. As Kolla
has never supported cells v1, which is finally being removed during
Train [2], we can get ahead of the curve and stop deploying
nova-consoleauth immediately.

[1] https://specs.openstack.org/openstack/nova-specs/specs/rocky/implemented/convert-consoles-to-objects.html
[2] https://blueprints.launchpad.net/nova/+spec/remove-cells-v1/

Change-Id: I099080979f5497537e390f531005a517ab12aa7a

The project has been retired and there will be no Train release [1].
This patch removes Neutron LBaaS support in Kolla.

[1] https://review.opendev.org/#/c/658494/

Change-Id: Ic0d3da02b9556a34d8c27ca21a1ebb3af1f5d34c

Qinling is an OpenStack project to provide "Function as a Service".
This project aims to provide a platform to support serverless functions.

Change-Id: I239a0130f8c8b061b531dab530d65172b0914d7c
Implements: blueprint ansible-qinling-support
Story: 2005760
Task: 33468

refer to [0]

[0]: https://docs.openstack.org/ceilometer/latest/install/install-compute-rdo.html


Co-Authored-By: zhulingjie <easyzlj@gmail.com>

Change-Id: I4cda336dedb3d807b80d13bcc219268a8d667b4d

This patch implements the support for the elasticsearch-exporter in
kolla-ansible

The configuration and prechecks are reused from the other exporters

Depends-On: Id138f12e10102a6dd2cd8d84f2cc47aa29af3972
Change-Id: Iae0eac0179089f159804490bf71f1cf2c38dde54

Because kolla-ansible not have cyborg so should add it.

Implements: blueprint add-cyborg-to-kolla-ansible

Depend-On: I497e67e3a754fccfd2ef5a82f13ccfaf890a6fcd

Change-Id: I6f7ae86f855c5c64697607356d0ff3161f91b239

Vitrage Collector service has been removed from Vitrage in change:
Ie713456b2df96e24d0b15d2362a666162bfb4300.

Change-Id: I45023940c1d2573bfed49d4ce3fac16ed2d559e4
Signed-off-by: Bartosz Zurkowski <b.zurkowski@samsung.com>
Co-Authored-By: Kien Nguyen <kiennt65@viettel.com.vn>

This patch implements the initial support for the
openstack-exporter[0] in the kolla-ansible
prometheus monitoring system.

The configuration and prechecks are reused from the other
exporters and a new template is provided for generating
a os-client-config file required by the exporter.

The default scrape interval is 60 seconds, but it can
be extended via a configuration option.

[0] https://github.com/Linaro/openstack-exporter

Change-Id: I4a34c4bb56e74b5cd544972cbd6540d9acb6e4a1

A spec to Deprecate the Glance Registry Service[0] was accepted in Newton,
but it contained the ambiguous statement, "Mark the service as deprecated
and ready for removal in the Q release." kolla-ansible disable the
glance-registry in Q release[1], and since we are in S now,
remove glance-registry is safe.

[0]: http://specs.openstack.org/openstack/glance-specs/specs/newton/approved/glance/deprecate-registry.html
[1]: https://review.openstack.org/#/c/566804/

Change-Id: I48f794029e97aa6f76bbd500e33f28f51a3f2ac4

Refer to neutron docs [1][2]
[1]: https://docs.openstack.org/neutron/pike/admin/archives/adv-config.html#l3-metering-service-driver
[2]: https://wiki.openstack.org/wiki/Neutron/Metering/Bandwidth


Co-Authored-By: ZhijunWei <wzj334965317@outlook.com>

Change-Id: I4a676d041bc2a86497cb139d9347365738b156df

This PS adds deployment of mistral-event-engine service
within mistral role.

Change-Id: If187e1a49a321d1e85fb3cfff2ea037e567b30dc

The concept of splitting the compute group into external/internal just
to specify agent_mode for Neutron DVR was deemed to be heavy handed, and
depreacated in the Pike cycle.

Now that Rocky has been released we can remove these completely for Stein.

Change-Id: I28a1eba7f40fee55a7ec41c27451e39e4d7fd8f0

The Monasca Grafana fork allows users to log into Grafana with their
OpenStack user credentials and see metrics associated with their
OpenStack project. The long term goal is to enable Keystone support
in upstream Grafana, but this work seems to have stalled.

Partially-Implements: blueprint monasca-grafana
Change-Id: Icc04613b2571c094ae23b66d0bcc38b58c0ee4e1

The Monasca Agent collects metrics and in this change is deployed
across the control plane. These metrics are collected into an OpenStack
project. It supports configuring a small number of plugins, which can
be extended in later commits. It also makes the Monasca Agent credentials
available to other roles, such as the common role to allow forwarding
logs to Monasca.

Partially-Implements: blueprint monasca-roles
Change-Id: I76b34fc5e1c76407a45fcf272268d5798b473ca2

The Monasca Persister reads metrics from Kafka and stores them
in a configurable time series database.

Change-Id: I8166b32bfb1583098ab8318a5f38d25bddb81e89
Partially-Implements: blueprint monasca-roles

The Monasca Notification engine generates alerts such as Slack
notifications from alerts.

Change-Id: I84861d5feefe6b6f38acc4dd71e94c386d40b562
Partially-Implements: blueprint monasca-roles

Monasca Thresh is a Storm topology which generates alerts from
metric streams according to alarms defined via the Monasca API.

This change runs the thresholder in local mode, which means that
the log output for the topology is directed to stdout and the
topology is restarted if the container is restarted. A future
change will improve the log collection and introduce a better
way of the checking the topology is running for multi-node
clusters.

Change-Id: I063dca5eead15f3cec009df62f0fc5d857dd4bb0
Partially-Implements: blueprint monasca-roles

Storm is required for running the Monasca thresholder component for
generating alerts.

Change-Id: I5e1ef74dc55a787293abbb3e629b5ab1ce5f4bbb
Partially-Implements: blueprint monasca-roles

Having all services in one giant haproxy file makes altering
configuration for a service both painful and dangerous. Each service
should be configured with a simple set of variables and rendered with a
single unified template.

Available are two new templates:

* haproxy_single_service_listen.cfg.j2: close to the original style, but
only one service per file
* haproxy_single_service_split.cfg.j2: using the newer haproxy syntax
for separated frontend and backend

For now the default will be the single listen block, for ease of
transition.

Change-Id: I6e237438fbc0aa3c89a3c8bd706a53b74e71904b

The log metrics service generates metrics from log messages
which allows further analysis and alerting to be performed
on them. Basic configuration is provided so that metrics
are generated for high level warning logs such as error, or
warning.

Change-Id: I45cc17817c716296451f620f304c0b1108162a56
Partially-Implements: blueprint monasca-roles

Since I701d495675178c3ed8ec1f00b31d09f198b38a6f merged, etcd only runs
on the control hosts, not the compute hosts. We therefore no longer
require the etcd group to include the compute hosts.

Since the group mapping is now static, we can remove the use of
host_in_groups from the etcd service, in favour of the simpler method of
specifying the group.

Change-Id: Id8f888d7321a30a85ff95e742f7e6c8e2b9c696f
Related-Bug: #1790415

Partially-Implements: blueprint networking-baremetal

Change-Id: I92b9505843f12692aef96764a314e5db49001a9b

This is a Logstash component which reads processed logs from Kafka
and writes them to Elasticsearch (or some other backend supported by
Logstash).

Ingesting the logs from this service with Fluentd will be covered under
a different commit.

Change-Id: I2d722991ab2072c54c4715507b19a4c9279f921b
Partially-Implements: blueprint monasca-roles

This patch extends the prometheus role for being able
to deploy the prometheus-alertmanager[0] container.

The variable enable_prometheus_alertmanager
decides if the container should be deployed and enabled.

If enabled, the following configuration and actions are performed:

- The alerting section on the prometheus-server configuration
is added pointing the prometheus-alertmanager host group as targets.

- HAProxy is configured to load-balance over the prometheus-alertmanager
host group. (external/internal).

Please note that a default (dummy) configuration is provided, that
allows the service to start, the operator should extend it via a node custom config

[0] https://github.com/openstack/kolla/tree/master/docker/prometheus/prometheus-alertmanager



Change-Id: I3a13342c67744a278cc8d52900a913c3ccc452ae
Closes-Bug: 1774725
Signed-off-by: Jorge Niedbalski <jorge.niedbalski@linaro.org>

The Monasca Log Transformer takes raw, unstandardised logs from one
Kafka topic, standardises them with whatever rules the operator wants
to use, and then writes them to a standardised logs topic in Kafka. It
is currently implemented as a Logstash config file.

Since Kolla does a fairly good job of standardising logs, this service
does very little processing. However, when other sources of logs
are used, it may be useful to add rules to the Transformer, particularly
if it's not possible to standardise the logs at source.

Ingesting the logs from this service with Fluentd will be covered under
a different commit.

Change-Id: I31cbb7e9a40a848391f517a56a67e3fd5bc12529
Partially-Implements: blueprint monasca-roles

the zun-wsproxy image is exists in kolla[0], but kolla-ansible
missing, this ps to add it.

[0]: https://github.com/openstack/kolla/tree/master/docker/zun/zun-wsproxy


Co-Authored-By: ZhijunWei <wzj334965317@outlook.com>

Change-Id: I89ef3463dfa5df8cf2d963ff0f0c7ddc382fc79b
Closes-Bug: #1765728

Deploy multipathd with cinder-volume when multipathd is enabled

Change-Id: Ibc6cab29498508769d6c42a870cf34f587ec7cc7
Closes-Bug:1757379

cAdvisor (Container Advisor) provides metrics on resource usage and
performance characteristics of running containers.  This change
deploys a cadvisor container and configures prometheus to scrape
data from it.

Change-Id: I55dd4fee954f9be68efda397746861ddaaa0a565
Partially-Implements: blueprint prometheus

When enable_ironic_ipxe is set in /etc/kolla/globals.yml,
the following happens:

- a new docker container, ironic_ipxe, is created. This contains
  an apache webserver used to serve up the boot images
- ironic is configured to use ipxe

Change-Id: I08fca1864a00afb768494406c49e968920c83ae7
Implements: blueprint ironic-ipxe

This patch adds the prometheus_memcached_exporter[0] to the
list of  available exporters, following the conventions
used by the previously integrated exporters.

[0] https://github.com/openstack/kolla/tree/master/docker/prometheus-memcached-exporter



Change-Id: I103b0ee19ef2fd17ce19a27d60773675ad234c1c
Closes-Bug: #1773303
Signed-off-by: Jorge Niedbalski <jorge.niedbalski@linaro.org>

Deploys the Monasca Log API with mod_wsgi + Apache.

Change-Id: I28f0aa31c59b0b6917be2b125b5f8a0d7a7035af
Partially-Implements: blueprint monasca-roles

Deploys the Monasca API with mod_wsgi + Apache.

Co-Authored-By: Mark Goddard <mark@stackhpc.com>

Partially-Implements: blueprint monasca-roles
Change-Id: I3e03762217fbef1fb0cbff6239abb109cbec226b