Change-Id: I0d7c7f47e6653cf2903589a9c86798a8c6404af5

Nova always tries to create the rabbitmq user regardless of
whether RabbitMQ is enabled or not.
This ps also adds an external rabbitmq doc.

Change-Id: Iec517226e4c82ea351889b55689a3efceaadcc76

In the Xena release, Ironic removed the iSCSI driver [1]. The
recommended driver is direct, which uses HTTP to transfer the disk
image. This requires an HTTP server, and the simplest option is to use
the one currently deployed when enable_ironic_ipxe is set to true. For
this reason, this patch always enables the HTTP server running on the
conductor.

iPXE is still enabled separately, since it cannot currently be used at
the same time as PXE.

[1] https://review.opendev.org/c/openstack/ironic/+/789382

Change-Id: I30c2ad2bf2957ac544942aefae8898cdc8a61ec6

The variable octavia_amphora_flavor should be octavia_amp_flavor.

The variable for customising network and subnet was only mentioned in
the example.

Change-Id: I3ba5a7ccc2c810fea12bc48584c064738e5aa35e

Ansible facts can have a large impact on the performance of the Ansible
control host. This patch introduces some control over which facts are
gathered (kolla_ansible_setup_gather_subset) and which facts are stored
(kolla_ansible_setup_filter). By default we do not change the default
values of these arguments to the setup module. The flexibility of these
arguments is limited, but they do provide enough for a large performance
improvement in a typical moderate to large OpenStack cloud.

In particular, the large complex dict fact for each interface has a
large effect, and on an OpenStack controller or hypervisor there may be
many virtual interfaces. We can use the kolla_ansible_setup_filter
variable to help:

    kolla_ansible_setup_filter: 'ansible_[!qt]*'

This causes Ansible to collect but not store facts matching that
pattern, which includes the virtual interface facts. Currently we are
not referencing other facts matching the pattern within Kolla Ansible.
Note that including the 'ansible_' prefix causes meta facts module_setup
and gather_subset to be filtered, but this seems to be the only way to
get a good match on the interface facts. To work around this, we use
ansible_facts rather than module_setup to detect whether facts exist in
the cache.

The exact improvement will vary, but has been reported to be as large as
18x on systems with many virtual interfaces.

For reference, here are some other tunings tried:

* Increased the number of forks (great speedup depending of the size of
  the deployment)
* Use `strategy = mitogen_linear` (cut processing time in half)
* Ansible caching (little speed up)
* SSH tunning (little speed up)

Co-Authored-By: Mark Goddard <mark@stackhpc.com>
Closes-Bug: #1921538
Change-Id: Iae8ca4aae945892f1dc65e1b10381d2e26e88805

This commit adds two new cli commands to allow an operator
to read and write passwords into a configured Hashicorp Vault
KV.

Change-Id: Icf0eaf7544fcbdf7b83f697cc711446f47118a4d

This patch is adding configuration option to
manipulate with kernel option sysctl_net_ipv4_tcp_retries2.

More informations about kernel option in [1][2]
and RedHat suggestion [3] to set for DBs and HA.

[1]: https://pracucci.com/linux-tcp-rto-min-max-and-tcp-retries2.html
[2]: https://blog.cloudflare.com/when-tcp-sockets-refuse-to-die/
[3]: https://access.redhat.com/solutions/726753

Closes-Bug: #1917068
Change-Id: Ia0decbbfa4e33b1889b635f8bb1c9094567a2ce6

By default, Ansible injects a variable for every fact, prefixed with
ansible_. This can result in a large number of variables for each host,
which at scale can incur a performance penalty. Ansible provides a
configuration option [0] that can be set to False to prevent this
injection of facts. In this case, facts should be referenced via
ansible_facts.<fact>.

This change updates all references to Ansible facts within Kolla Ansible
from using individual fact variables to using the items in the
ansible_facts dictionary. This allows users to disable fact variable
injection in their Ansible configuration, which may provide some
performance improvement.

This change disables fact variable injection in the ansible
configuration used in CI, to catch any attempts to use the injected
variables.

[0] https://docs.ansible.com/ansible/latest/reference_appendices/config.html#inject-facts-as-vars

Change-Id: I7e9d5c9b8b9164d4aee3abb4e37c8f28d98ff5d1
Partially-Implements: blueprint performance-improvements

On machines with many cores, we were seeing excessive CPU load on systems
that were not very busy. With the following Erlang VM argument we saw
RabbitMQ CPU usage drop from about 150% to around 20%, on a system with
40 hyperthreads.

    +S 2:2

By default RabbitMQ starts N schedulers where N is the number of CPU
cores, including hyper-threaded cores. This is fine when you assume all
your CPUs are dedicated to RabbitMQ. Its not a good idea in a typical
Kolla Ansible setup. Here we go for two scheduler threads.
More details can be found here:
https://www.rabbitmq.com/runtime.html#scheduling
and here:
https://erlang.org/doc/man/erl.html#emulator-flags

    +sbwt none

This stops busy waiting of the scheduler, for more details see:
https://www.rabbitmq.com/runtime.html#busy-waiting
Newer versions of rabbit may need additional flags:
"+sbwt none +sbwtdcpu none +sbwtdio none"
But this patch should be back portable to older versions of RabbitMQ
used in Train and Stein.

Note that information on this tuning was found by looking at data from:
rabbitmq-diagnostics runtime_thread_stats
More details on that can be found here:
https://www.rabbitmq.com/runtime.html#thread-stats

Related-Bug: #1846467

Change-Id: Iced014acee7e590c10848e73feca166f48b622dc

Change-Id: Idaae03612dd7feabd2cdc57e510947328524e98b

Change-Id: Id9a24c54db9a70c777e51ea10504159d5090f9a7

* add CentOS Stream 8 info & reno
* mark RHEL as deprecated in docs

Change-Id: Ifa3b8c7365e4092202bf9e89b67293cf8d983e0a

* Enables the Neutron packet logging framework for OVS
(https://docs.openstack.org/neutron/latest/admin/config-logging.html).
* Adds a toggle variable "enable_neutron_packet_logging"

Change-Id: Ica3594cdac634b496949a06ed813dccd18090af4
Implements: blueprint neutron-log-service-plugin

As agreed during IRC meeting. [1]

[1] http://eavesdrop.openstack.org/meetings/kolla/2021/kolla.2021-04-07-15.00.html

Change-Id: Ibffa019e2106784e97f5c482106bb5082f788752

In the Xena cycle it was decided to remove the Monasca
Grafana fork due to lack of maintenance. This commit removes
the service and provides a limited workaround using the
Monasca Grafana datasource with vanilla Grafana.

Depends-On: I9db7ec2df050fa20317d84f6cea40d1f5fd42e60
Change-Id: I4917ece1951084f6665722ba9a91d47764d3709a

Trivial Fix

Change-Id: Ie08877e339455bed45ee467a87de9648678e88c5

Change-Id: I713f6fafe328e060a71dbb584e61603e547deaf6

The current behaviour is to support supplying a single
folder of Grafana dashboards which can then be populated
into a single folder in Grafana. Some users may wish
to have sub-folders of Dashboards, and load these into
separate dashboard folders in Grafana via a custom
provisioning file. For example, a user may have a
sub-folder of Ceph dashboards that they wish to keep
separate from OpenStack dashboards. This patch supports
sub-folders whilst not affecting the original mechanism.

Trivial-Fix

Change-Id: I9cd289a1ea79f00cee4d2ef30cbb508ac73f9767

Change-Id: Iede747ceaafa54a00186761943fe2f4ac13f9559

Minor corrections to doc and release note.

Change-Id: I8a90cbac0b9a1eaa5f6c02271515f2357547f908

Per [1].

[1] http://lists.openstack.org/pipermail/openstack-discuss/2021-February/020707.html

Change-Id: Id6f3cd158bf5d01750971249b11364b6a8631789
Closes-Bug: #1885689

Change-Id: I947c2940518c0f4872acaa977edeaca370dc9a96

Change-Id: I08030ac88911d3594c75cb2184767067ad177139

Change-Id: Ie888e84a3b6e27afc23f89f643fdaa58880aae6d

The docker configuration should be a URL, not a host:port.

Closes-Bug: #1919932
Change-Id: I5025fdb7e48c79a107b45f1454f5d5e81367a2f9

Allow users to import custom grafana dashboards.
Dashboards as JSON files should be placed into
"{{ node_custom_config }}/grafana/dashboards/" folder.

Change-Id: Id0f83b8d08541b3b74649f097b10c9450201b426

Min version stays as 2.9

Change-Id: I7ec8c5eb36757248c9aa016dc7d4e495ec5bb635

Change-Id: Ief84e093829677c97c8df9a08aefca43b1e51aac

This change allows a user to forward control plane logs
directly to Elasticsearch from Fluentd, rather than via
the Monasca Log API when Monasca is enabled. The Monasca
Log API can continue to handle tenant logs.

For many use cases this is simpler, reduces resource
consumption and helps to decouple control plane logging
services from tenant logging services.

It may not always be desired, so is optional and off by
default.

Change-Id: I195e8e4b73ca8f573737355908eb30a3ef13b0d6

The Monasca alerting pipeline provides multi-tenancy alerts and
notifications. It runs as an Apache Storm topology and generally
places a significant memory and CPU burden on monitoring hosts,
particularly when there are lot of metrics. This is fine if the
alerting service is in use, but sometimes it is not. For example
you may use Prometheus for monitoring the control plane, and
wish to offer tenants a monitoring service via Monasca without
alerting and notification functionality. In this case it makes
sense to disable this part of the Monasca pipeline and this patch
adds support for that.

If the service is ever re-enabled, all alerts and notifications
should spawn back automatically since they are persisted in the
central mysql database cluster.

Change-Id: I84aa04125c621712f805f41c8efbc92c8e156db9

The Log Metrics service is an admin only service. We now have
support in Fluentd via the Prometheus plugin to create metrics
from logs. These metrics can be scraped into Monasca or Prometheus.
It therefore makes sense to deprecate this service, starting by
disabling it by default, and then removing it in the Xena release.
This should improve the stability of the Monasca metrics pipeline
by ensuring that all metrics pass via the Monasca API for
validation, and ensure that metrics generated from logs are
available to both Prometheus and Monasca users by default.

Change-Id: I704feb4434c1eece3eb00c19dc5f934fd4bc27b4

Historically Monasca Log Transformer has been for log
standardisation and processing. For example, logs from different
sources may use slightly different error levels such as WARN, 5,
or WARNING. Monasca Log Transformer is a place where these could
be 'squashed' into a single error level to simplify log searches
based on labels such as these.

However, in Kolla Ansible, we do this processing in Fluentd so
that the simpler Fluentd -> Elastic -> Kibana pipeline also
benefits. This helps to avoid spreading out log parsing
configuration over many services, with the Fluentd Monasca output
plugin being yet another potential place for processing (which
should be avoided). It therefore makes sense to remove this
service entirely, and squash any existing configuration which
can't be moved to Fluentd into the Log Perister service. I.e.
by removing this pipeline, we don't loose any functionality,
we encourage log processing to take place in Fluentd, or at least
outside of Monasca, and we make significant gains in efficiency
by removing a topic from Kafka which contains a copy of all logs
in transit.

Finally, users forwarding logs from outside the control plane,
eg. from tenant instances, should be encouraged to process the
logs at the point of sending using whichever framework they are
forwarding them with. This makes sense, because all Logstash
configuration in Monasca is only accessible by control plane
admins. A user can't typically do any processing inside Monasca,
with or without this change.

Change-Id: I65c76d0d1cd488725e4233b7e75a11d03866095c

This option disables copy of certificates from the operator host to
kolla-ansible managed hosts.

This is especially useful if you already have some mechanisms to handle
your certificates directly on your hosts.

Co-Authored-By: Marc 'risson' Schmitt <marc.schmitt@risson.space>
Change-Id: Ie18b2464cb5a65a88c4ac191a921b8074a14f504

Deprecates support for Prometheus v1.x.
In Xena support for it will be removed from Kolla Ansible.

Change-Id: I027b19621196c698e09f79af294ba1b5dbfc0516

(cherry picked from kolla commit abc4f7c124265f9e7b72d448b2ef18d6d4e84888)

Change-Id: I7e51a76287cd0f5a22d11dc3af9284b556d06eca

This commit adds possibility to deploy Neutron's DHCP agents in OVN
networking scenario.

Co-Authored-By: Michał Nasiadka <mnasiadka@gmail.com>

Change-Id: I073d04319b510182f5c1478e728c0c5bcc8799f1

Update the Monasca docs to improve security considerations.

Trivial-Fix
Change-Id: I97eb8441466f8c6abdbd66068257765bdbe32d4d

This pull request adds support for the OpenID Connect authentication
flow in Keystone and enables both ID and access token authentication
flows. The ID token configuration is designed to allow users to
authenticate via Horizon using an identity federation; whereas the
Access token is used to allow users to authenticate in the OpenStack CLI
using a federated user.

Without this PR, if one wants to configure OpenStack to use identity
federation, he/she needs to do a lot of configurations in the keystone,
Horizon, and register quite a good number of different parameters using
the CLI such as mappings, identity providers, federated protocols, and
so on. Therefore, with this PR, we propose a method for operators to
introduce/present the IdP's metadata to Kolla-ansible, and based on the
presented metadata, Kolla-ansible takes care of all of the
configurations to prepare OpenStack to work in a federated environment.

Implements: blueprint add-openid-support
Co-Authored-By: Jason Anderson <jasonanderson@uchicago.edu>
Change-Id: I0203a3470d7f8f2a54d5e126d947f540d93b8210

Stolen from Kayobe.

Change-Id: I42a2ef84e5fe570e8001908614337e64d035ec99

If the Octavia/Amphora management network is created by Kolla, support
setting the IP address family and IPv6 address/RA mode.

Closes-Bug: 1913409

Change-Id: I9f2ef2196654c91596cb5c4b3c157bcee267226a