The Monasca alerting pipeline provides multi-tenancy alerts and
notifications. It runs as an Apache Storm topology and generally
places a significant memory and CPU burden on monitoring hosts,
particularly when there are lot of metrics. This is fine if the
alerting service is in use, but sometimes it is not. For example
you may use Prometheus for monitoring the control plane, and
wish to offer tenants a monitoring service via Monasca without
alerting and notification functionality. In this case it makes
sense to disable this part of the Monasca pipeline and this patch
adds support for that.

If the service is ever re-enabled, all alerts and notifications
should spawn back automatically since they are persisted in the
central mysql database cluster.

Change-Id: I84aa04125c621712f805f41c8efbc92c8e156db9

The Log Metrics service is an admin only service. We now have
support in Fluentd via the Prometheus plugin to create metrics
from logs. These metrics can be scraped into Monasca or Prometheus.
It therefore makes sense to deprecate this service, starting by
disabling it by default, and then removing it in the Xena release.
This should improve the stability of the Monasca metrics pipeline
by ensuring that all metrics pass via the Monasca API for
validation, and ensure that metrics generated from logs are
available to both Prometheus and Monasca users by default.

Change-Id: I704feb4434c1eece3eb00c19dc5f934fd4bc27b4

Historically Monasca Log Transformer has been for log
standardisation and processing. For example, logs from different
sources may use slightly different error levels such as WARN, 5,
or WARNING. Monasca Log Transformer is a place where these could
be 'squashed' into a single error level to simplify log searches
based on labels such as these.

However, in Kolla Ansible, we do this processing in Fluentd so
that the simpler Fluentd -> Elastic -> Kibana pipeline also
benefits. This helps to avoid spreading out log parsing
configuration over many services, with the Fluentd Monasca output
plugin being yet another potential place for processing (which
should be avoided). It therefore makes sense to remove this
service entirely, and squash any existing configuration which
can't be moved to Fluentd into the Log Perister service. I.e.
by removing this pipeline, we don't loose any functionality,
we encourage log processing to take place in Fluentd, or at least
outside of Monasca, and we make significant gains in efficiency
by removing a topic from Kafka which contains a copy of all logs
in transit.

Finally, users forwarding logs from outside the control plane,
eg. from tenant instances, should be encouraged to process the
logs at the point of sending using whichever framework they are
forwarding them with. This makes sense, because all Logstash
configuration in Monasca is only accessible by control plane
admins. A user can't typically do any processing inside Monasca,
with or without this change.

Change-Id: I65c76d0d1cd488725e4233b7e75a11d03866095c

trivial fix
see: https://review.opendev.org/c/openstack/kolla-ansible/+/763191

Change-Id: I7f5a5ba5e9a6b3866fc4d2c72b7b4884c85020bd

This change enables the use of Docker healthchecks for magnum
services.
Implements: blueprint container-health-check

Change-Id: I14d862aa599915c781d02b71a0e57d2124de9abc

it was confused to customize opts in trove-conductor.conf
or trove-taskmanager.conf now.
if we want to customize a opts,The operator needs
to know which service is using the configuration opts.
actually trove uses trove.conf is enough for all services
this change combines  all trove config files.

Change-Id: I5a630109e3c4b59bff216146a3ed64c6d47e247f

Update the Monasca docs to improve security considerations.

Trivial-Fix
Change-Id: I97eb8441466f8c6abdbd66068257765bdbe32d4d

This pull request adds support for the OpenID Connect authentication
flow in Keystone and enables both ID and access token authentication
flows. The ID token configuration is designed to allow users to
authenticate via Horizon using an identity federation; whereas the
Access token is used to allow users to authenticate in the OpenStack CLI
using a federated user.

Without this PR, if one wants to configure OpenStack to use identity
federation, he/she needs to do a lot of configurations in the keystone,
Horizon, and register quite a good number of different parameters using
the CLI such as mappings, identity providers, federated protocols, and
so on. Therefore, with this PR, we propose a method for operators to
introduce/present the IdP's metadata to Kolla-ansible, and based on the
presented metadata, Kolla-ansible takes care of all of the
configurations to prepare OpenStack to work in a federated environment.

Implements: blueprint add-openid-support
Co-Authored-By: Jason Anderson <jasonanderson@uchicago.edu>
Change-Id: I0203a3470d7f8f2a54d5e126d947f540d93b8210

If kolla-ansible is installed via pip install --user, currently the
kolla-ansible script is unable to locate the installed playbooks.
This leads to a failure when running commands.

This change fixes the issue by checking for the user's .local directory
as a possible installation path.

This fixes some of the scenario tests which were failing after switching
to a user installation in Ifaf1948ed5d42eebaa62d7bad375bbfc12b134d5.
Most tests did not fail since the kolla-ansible script in the source
checkout was used.

Closes-Bug: #1915527

Change-Id: I5b47a146627d06bb3fe4a747c5f20290c726b0f9

One of the renos was causing issues due to a duplicated id.
This change makes tox doc8 env lint renos and fixes
the offending reno.

Change-Id: Id3ae6e144b4261c97726cdec172ea9bef093de9e

This change enables the use of Docker healthchecks for manila services.
Implements: blueprint container-health-check

Change-Id: I3a2239764b7e3d6db51e535404388a512aba7629

There are a few issues fixed here:

- The Barbican API service doesn't set a log file, so all the Barbican API
  service logs go to loadwsgi.py.log by default.
- The logs in loadwsgi.py.log are not ingested properly by Fluentd.
- uWSGI logs go to barbican-api.log. This would normally be used as the log
  file for the Barbican API service logs.

This patch makes the following changes to address the above issues:

- All uWSGI logs (from the Emperor and Vassals) go to barbican_api_uwsgi_access.log
  Although these logs aren't strictly all access logs, this follows the existing
  pattern for WSGI logs.
- The Barbican API service logs are written to barbican-api.log instead of
  loadwsgi.py.log. This follows the pattern used by other OpenStack services.
- Fluentd is configured to parse the Barbican API service logs as it would with
  other OpenStack Python services.

Change-Id: I6d03fa8c81c52b6f061514a836bbd15bb6639aaf
Closes-Bug: #1891343

--db-sock JSON-RPC socket name
--db-nb-sock OVN_Northbound db socket
--db-sb-sock OVN_Southbound db socket [1]
so should use db-nb-sock and db-sb-sock

Closes-bug: #1913031

[1] https://github.com/ovn-org/ovn/blob/master/utilities/ovn-ctl

Change-Id: Ife38237a308c87465d5ac3faf7d8de93fd49de4e

Change-Id: Id9110a1f536377cea0386dda6814035d73de13b1
Implements: blueprint remove-unicode

Kolla Ansible's release notes page [1] has the same heading as
Kolla project's [2] which is confusing. This commit aims to fix
that, as well as to do some minor cleanup of docs configuration
by removing part that is never used.

[1] - https://docs.openstack.org/releasenotes/kolla-ansible/
[2] - https://docs.openstack.org/releasenotes/kolla/

Change-Id: I0da97d5a5b0a58d5c5e0e52b0687e2249d3fd222

With tips and clarifications.

Change-Id: Ic744e13805c4a158d1156a230f8c57d7a980d55f

It is now possible to deploy either 1.x or 2.x version of Prometheus.
The new 2.x version introduces breaking changes in terms of storage
format and command line options.

Change-Id: I80cc6f1947f3740ef04b29839bfa655b14fae146
Co-Authored-By: Radosław Piliszek <radoslaw.piliszek@gmail.com>

With this patch, Monasca no longer relies on automatic topic creation
in Kafka, and instead pre-creates all topics before bringing up the
containers. If the topic already exists then it will not be
changed, therefore existing users are not affected.

This patch allows per topic customisations, such as increasing the
number of partitions on particular topics and also works around
a race condition in automatic topic creation where multiple instances
of the same service could race to create a topic causing some of the
services to restart and throw an error before resuming normal
operation.

Change-Id: Ib15c95bb72cf79e9e55945d757b248e06f5f4065

The bootstrap process tries to removes existing apparmor profiles but
doesn't consider the case where those are disabled. This change fixes
the scenario where the libvirt profile exists but is disabled.

Closes-Bug: 1909874
Change-Id: Ied0f2acc420bd5cf1e092c8aee358cba35bd8d5d

This change enables the use of Docker healthchecks for cloudkitty
services.
Implements: blueprint container-health-check

Change-Id: I19892035382ffff5200e88da53408a19e72c9d68

This change fix ansible deploy ovs-dpdk failed and
neutron_openvswitch_agent container can't start..

dpdk_tunnel is a role variable, but kolla_address gets vaule
from hostvars. so we need remove this variable and it's friends
to group/all.yaml

neutron_openvswitch_agent connects to ovs-db with 127.0.0.1,
but ovs-db listen on management interface.

Closes-Bug: 1908850

Change-Id: I86a13d2476644bfa2545a6737752cda1ade34d23

As announced on the openstack-discuss ML[1], Karbor is retiring
this cycle (Wallaby).

Needed-By: https://review.opendev.org/c/openstack/karbor/+/767032

[1] http://lists.openstack.org/pipermail/openstack-discuss/2020-November/018643.html

Change-Id: I222cf302e507f6a9de0347c79ec536aa7be22bb6

This can improve performance of image format conversion and encryption, if
sufficient memory is available on the cinder-volume host.

Closes-Bug: #1897276

Change-Id: I4ca1c4db7b66fdfc6bb873aad2570234f3882d81

Partial-Bug: #1897276

Change-Id: Ia06da456a7f26f0f2ceebc35eb88c0da0767e1c6

Searchlight project is retiring in Wallaby cycle[1].
This commit removes the ansible roles of Searchlight project
before its code is removed.

Needed-By: https://review.opendev.org/c/openstack/searchlight/+/764526

[1] http://lists.openstack.org/pipermail/openstack-discuss/2020-November/018637.html

Change-Id: I85aab66376ea4f1376c2705066ba3c7e5645644f

Qinling project is retiring in Wallaby cycle[1].
This commit removes the ansible roles of Qinling project
before its code is removed.

Needed-By: https://review.opendev.org/c/openstack/qinling/+/764521

[1] http://lists.openstack.org/pipermail/openstack-discuss/2020-November/018638.html

Change-Id: I6543bacff638b1649511f7e779807954c34ef570

Mariadb recovery fails if a cluster has previously been deployed, but any of
the mariadb containers do not exist.

Steps to reproduce
==================

* Deploy a mariadb galera cluster
* Remove the mariadb container from at least one host (docker rm -f mariadb)
* Run kolla-ansible mariadb_recovery

Expected results
================

The cluster is recovered, and a new container deployed where necessary.

Actual results
==============

The task 'Stop MariaDB containers' fails on any host where the container does
not exist.

Solution
========

This change fixes the issue by using the 'ignore_missing' flag for kolla_docker
with the stop_container action. This means the task does not fail when the
container does not exist. It is also necessary to swap some 'docker cp'
commands for 'cp' on the host, using the path to the volume.

Closes-Bug: #1907658

Change-Id: Ibd4a6adeb8443e12c45cbab65f501392ffb16fc7

The 'prechecks : Checking Docker version' task previously failed with
Docker 20.10.0. The regex used to parse the version was returning
0.10.0, which is not above the minimum. The previous version of 19.x
would have been parsed as 9.x, which is above the minimum.

This change fixes the issue by matching the beginning and end of the
version using \b.

Depends-On: https://review.opendev.org/766183

Change-Id: I2a23eea7effb5b9a5e73361bcd48bd2e16d1569c
Closes-Bug: 1907436

Change-Id: I7970c5b02f178fd8fb35c984117f6bc848353a5b
Closes-Bug: #1906944

Those loglevels can build up over time and create unnecessary high metrics cardinality.

Change-Id: Ib1a03772d0bd58758430b37b4f2f67126cf86fa3
Closes-bug: #1906796

Change-Id: I1ff4cbdf3f60cb7fd5fe5d3c5d498e05fe2df79a
Closes-Bug: #1904702

Add scrape_timeout option in
prometheus_openstack_exporter job in order
to avoid timeout for large Openstack environment.

Change-Id: If96034e602bee3b3eea34a2656047355e1d17eec
Closes-Bug: #1903547

Change-Id: If6c50606fa3da353728ae4d916df20fcc95b8927

Currently we set enable-chassis-as-gw on compute nodes when distributed FIP
is enabled - that is not required for FIP functionality.

Change-Id: Ic880a9479fa0cdbb1d1cae3dbe9523ef2e1132ce
Closes-Bug: #1901960

Add file to the reno documentation build to show release notes for
stable/victoria.

Use pbr instruction to increment the minor version number
automatically so that master versions are higher than the versions on
stable/victoria.

Change-Id: Iad61fa88f8afa7d5f39154b9466338b417bbf40a
Sem-Ver: feature

Follows existing backends patterns to add support for the GlusterFS
NFS driver.
NFS server type used by the GlusterFS backend, Gluster or Ganesha,
currently supports Gluster.
The GlusterFS NFS driver needs to install the glusterfs-fuse package
in the kolla images manila share container in advance, which has been merged
in https://review.opendev.org/747510

Change-Id: I7fdb121b5bf9850d62246a24f9b17d226028c2ca

During a deploy, if keystone Fernet key rotation happens before the
keystone container starts, the rotation may fail with 'permission
denied'. This happens because config.json for Keystone sets the
permissions for /etc/keystone/fernet-keys.

This change fixes the issue by also setting the permissions for
/etc/keystone/fernet-keys in config.json for keystone-fernet and
keystone-ssh.

Change-Id: I561e4171d14dcaad8a2a9a36ccab84a670daa904
Closes-Bug: #1888512

Currently we check the age of the primary Fernet key on Keystone
startup, and fail if it is older than the rotation interval. While this
may seem sensible, there are various reasons why the key may be older
than this:

* if the rotation interval is not a factor of the number of seconds in a
  week, the rotation schedule will be lumpy, with the last rotation
  being up to twice the nominal rotation interval
* if a keystone host is unavailable at its scheduled rotation time,
  rotation will not happen. This may happen multiple times

We could do several things to avoid this issue:

1. remove the check on the age of the key
2. multiply the rotation interval by some factor to determine the
   allowed key age

This change goes for the more simple option 1. It also cleans up some
terminology in the keystone-startup.sh script.

Closes-Bug: #1895723

Change-Id: I2c35f59ae9449cb1646e402e0a9f28ad61f918a8

The correct path according to Ubuntu cron manpage [1] is
/var/spool/cron/crontabs/$USER

[1]: http://manpages.ubuntu.com/manpages/trusty/man8/cron.8.html

Closes-Bug: #1898765
Change-Id: Id5fc354e3e32cae2468cd2557a2967859e3b4e16

Nova has reversed their deprecation of the VMware driver, and the Kolla
community has shown an interest in it.

Change-Id: I82f1074da56ed16c08317d1f92ed7f0a6f4a149a