Changes name of ansible module kolla_docker to
kolla_container.

Change-Id: I13c676ed0378aa721a21a1300f6054658ad12bc7
Signed-off-by: Martin Hiner <m.hiner@partner.samsung.com>

docker_restart_policy: no causes systemd units to not get created
and we use it in CI to disable restarts on services.

Introducing oneshot policy to not create systemd unit for oneshot
containers (those that are running bootstrap tasks, like db
bootstrap and don't need a systemd unit), but still create systemd
units for long lived containers but with Restart=No.

Change-Id: I9e0d656f19143ec2fcad7d6d345b2c9387551604

This command can be invoked with ``kolla-ansible rabbitmq-reset-state``.
This is primarily designed to be used when enabling HA queues[1].

As such, this also updates the RabbitMQ documentation to use this
command.

[1] https://docs.openstack.org/kolla-ansible/latest/reference/message-queues/rabbitmq.html#high-availability

Change-Id: I6ad95a3618fc1a34af56657ef99ef14dc979f17a

Use case: exposing single external https frontend and
load balancing services using FQDNs.

Support different ports for internal and external endpoints.

Introduced kolla_url filter to normalize urls like:
- https://magnum.external:443/v1
- http://magnum.external:80/v1



Change-Id: I9fb03fe1cebce5c7198d523e015280c69f139cd0
Co-Authored-By: Jakub Darmach <jakub@stackhpc.com>

Hardcoded docker value in commands is not supported anymore
and kolla_container_engine is used instead.

Change-Id: I25d9563c82842ac51d41467ff7b4144b306fdb12
Signed-off-by: Ivan Halomi <i.halomi@partner.samsung.com>

Ansible 2.14.3 introduced a change that broke the method used for
restarting MariaDB and RabbitMQ serially [1][2]. In
I57425680a4cdbf0daeb9b2cc35920f1b933aa4a8 we limited to 2.14.2 to work
around this. Ansible upstream claim this behaviour was unintentional,
and will not fix it.

This change moves to a different approach where we use separate plays
with a 'serial' keyword to execute the restart.

This change also removes the restriction on the maximum supported
version of 2.14.2 on ansible-core - any 2.14 release is now supported.

[1] https://github.com/ansible/ansible/commit/65366f663de7d044f42ae6dd53368fd4c1f88b35
[2] https://github.com/ansible/ansible/issues/80848

Depends-On: https://review.opendev.org/c/openstack/kolla/+/884208

Change-Id: I5a12670d07077d24047aaff57ce8d33ccf7156ff

When using externally managed certificates, according to [1],
one should set `kolla_externally_managed_cert: yes` and ensure
that the certificates are in the correct place.

However, RabbitMQ precheck still expects the certificates to be
available on the controller node. This is incorrect.

Fix by not running the tasks in question when `kolla_externally_managed_cert: yes`

[1] https://docs.openstack.org/kolla-ansible/latest/admin/tls.html



Closes-Bug: 1999081
Related-Bug: 1940286
Signed-off-by: Magnus Lööf <magnus.loof@basalt.se>
Change-Id: I9f845a7bdf5055165e199ab1887ed3ccbfb9d808

Change-Id: I1d8021a1bc780449e3ca96183c6f4abaed17b382

Currently, the process of enabling RabbitMQ HA with the variable
``om_enable_rabbitmq_high_availbility`` requires some manual steps to
migrate from transient to mirrored queues. In preparation for setting
this variable to ``True`` by default, this adds a precheck that will
fail if a system is currently running non-mirrored queues and
``om_enable_rabbitmq_high_availbility`` is set to ``True``.

Includes a helpful message informing the operator of their choice.
Either follow the manual procedure to migrate the queues described in
the docs, or set ``om_enable_rabbitmq_high_availbility`` to ``False``.

The RabbitMQ HA section of the reference docs is updated to include
these instructions.

Change-Id: Ic5e64998bd01923162204f7bb289cc110187feec

With the addition of the variable
`om_enable_rabbitmq_high_availability`, this feature in the upgrade
task should be brought back. It is also now used in the deploy task. The
`ha-all` policy is cleared only when
`om_enable_rabbitmq_high_availability` is set to `false`.

Change-Id: Ia056aa40e996b1f0fed43c0f672466c7e4a2f547

Puts the RabbitMQ node into maintenance mode before restarting the
container. This will make the node shutdown less disruptive. For details
on what maintenance mode does, see:
https://www.rabbitmq.com/upgrade.html#maintenance-mode

Change-Id: Ia61573f3fb95fe8fcde6b789ca77ef5b45fe0a65

Since RMQ 3.8 we can use rolling upgrade [1].

Depends-On: https://review.opendev.org/c/openstack/kolla/+/872393

[1]: https://www.rabbitmq.com/upgrade.html#rolling-upgrades

Change-Id: If6a7c6c12d9226a2406728108b3c87b3485ac55f

Following ideas here:
https://wiki.openstack.org/wiki/Large_Scale_Configuration_Rabbit

Make sure old messages with no consumer are dropped after the message
TTL of 10 mins, longer than the 1 min RPC timeout.
Also ensure queues expire after an hour of inactivity, so queues from
removed nodes or renamed nodes don't grow over time.

Change-Id: Ifb28ac68b6328adb604a7474d01e5f7a47b2e788

Adds two new flags to alter behaviour in RabbitMQ:
    * `rabbitmq_message_ttl_ms`, which lets you set a TTL on messages.
    * `rabbitmq_queue_expiry_ms`, which lets you set an expiry time on queues.
See https://www.rabbitmq.com/ttl.html for more information on both.

Change-Id: I51ca37ffbb1bb5c07f2d39873f0f33ca20263f2a

Changes the default value of `rabbitmq-ha-promote-on-shutdown` to
`"always"`.

We are seeing issues with RabbitMQ automatically recovering when nodes
are restarted. https://www.rabbitmq.com/ha.html#cluster-shutdown

Rather than waiting for operator interventions, it is better we allow
recovery to happen, even if that means we may loose some messages.
A few failed and timed out operations is better than a totaly broken
cloud. This is achieved using ha-promote-on-shutdown=always.

Note, when a node failure is detected, this is already the default
behaviour from 3.7.5 onwards:
https://www.rabbitmq.com/ha.html#promoting-unsynchronised-mirrors

Related-Bug: #1954925
Change-Id: I484a81163f703fa27112df22473d657e2a9ab964

With the parameter rabbitmq_datadir_volume it is possible
to use a directory as volume for the rabbitmq service. By default,
a volume named rabbitmq is used (the previous default).

Change-Id: I99d6bd71ca79cba81062dedfb767c5ed341bb182

Currently we do not follow the RabbitMQ advice on replicas here:
https://www.rabbitmq.com/ha.html#replication-factor

Here we reduce the number of replicas to n // 2 + 1 as advised
above. The hope it this helps speed up recovery from rabbit
issues.

Related-Bug: #1954925
Change-Id: Ib6bcb26c499c9884faa4a0cd51abaec00cacb096

Adds the flag `rabbitmq_ha_replica_count` to change how many different
nodes a queue should be mirrored across. If the value is not set, then
it defaults to "ha-mode":"all". This value is unset by default to avoid
any unexpected changes to the RabbitMQ definitions.json file, as that
would trigger an unexpected restart of RabbitMQ during the next deploy.

Change-Id: Iee98cd937197a73a3b04aa8501fa325e8ecfff24

By default ha-promote-on-shutdown=when-synced. However we are seeing
issues with RabbitMQ automatically recovering when nodes are restarted.
https://www.rabbitmq.com/ha.html#cluster-shutdown

Rather than waiting for operator interventions, it is better we allow
recovery to happen, even if that means we may loose some messages.
A few failed and timed out operations is better than a totaly broken
cloud. This is achieved using ha-promote-on-shutdown=always.

Note, when a node failure is detected, this is already the default
behaviour from 3.7.5 onwards:
https://www.rabbitmq.com/ha.html#promoting-unsynchronised-mirrors

This patch adds the option to change the ha-promote-on-shutdown
definition, using the flag `rabbitmq_ha_promote_on_shutdown`. This
value is unset by default to avoid any unexpected changes to the
RabbitMQ definitions.json file, as that would trigger an unexpected
restart of RabbitMQ during the next deploy.

Related-Bug: #1954925

Change-Id: I2146bda2c72ddac2c9923c6941b0596395fd9ab5

As rabbitmq's configuration file is not ini or yaml file,
there is no option to extend configuration by new config
options via merge_configs or merge_yaml.

This patch moves config options to dictionary
so it can be overriden in /etc/kolla/globals.yml.

Change-Id: I5cd772f4fb80a0e200fb24d67be735ca81e3fdeb

A combination of durable queues and classic queue mirroring can be used
to provide high availability of RabbitMQ. However, these options should
only be used together, otherwise the system will become unstable. Using
the flag ``om_enable_rabbitmq_high_availability`` will either enable
both options at once, or neither of them.

There are some queues that should not be mirrored:
* ``reply`` queues (these have a single consumer and TTL policy)
* ``fanout`` queues (these have a TTL policy)
* ``amq`` queues (these are auto-delete queues, with a single consumer)
An exclusionary pattern is used in the classic mirroring policy. This
pattern is ``^(?!(amq\\.)|(.*_fanout_)|(reply_)).*``

Change-Id: I51c8023b260eb40b2eaa91bd276b46890c215c25

When running in check mode, some prechecks previously failed because
they use the command module which is silently not run in check mode.
Other prechecks were not running correctly in check mode due to e.g.
looking for a string in empty command output or not querying which
containers are running.

This change fixes these issues.

Closes-Bug: #2002657
Change-Id: I5219cb42c48d5444943a2d48106dc338aa08fa7c

assert will also fail when we're not meeting the conditions, makes
clear what we're actually testing, and isn't listed as a skipped task
when the condition is ok.

Change-Id: I4c919b523dde2602c81179ab3d28b913650b4c9f

Regularly, we experience issues in Kolla Ansible deployments because we
use wrong options in OpenStack configuration files. This is because
OpenStack services ignore unknown options. We also need to keep on top
of deprecated options that may be removed in the future. Integrating
oslo-config-validator into Kolla Ansible will greatly help.

Adds a shared role to run oslo-config-validator on each service. Takes
into account that services have multiple containers, and these may also
use multiple config files. Service roles are extended to use this shared
role. Executed with the new command ``kolla-ansible validate-config``.

Change-Id: Ic10b410fc115646d96d2ce39d9618e7c46cb3fbc

Second part of patchset:
https://review.opendev.org/c/openstack/kolla-ansible/+/799229/


in which was suggested to split patch into smaller ones.

This change adds container_engine variable to kolla_container_facts
module, this prepares module to be used with docker and podman as well
without further changes in roles.

Signed-off-by: Ivan Halomi <i.halomi@partner.samsung.com>
Co-authored-by: Martin Hiner <m.hiner@partner.samsung.com>
Change-Id: I9e8fa30646844ab4a288555f3aafdda345b3a118

First part of patchset:
 https://review.opendev.org/c/openstack/kolla-ansible/+/799229/


in which was suggested to split patch into smaller ones.

This implements kolla_container_engine variable
in command calls of docker,so later on it can be
also used for podman without further change.

Signed-off-by: Ivan Halomi <i.halomi@partner.samsung.com>
Change-Id: Ic30b67daa2e215524096ad1f4385c569e3d41b95

This patch adds loadbalancer-config role
which is "wrapper" around haproxy-config
and proxysql-config role which will be added
in follow-up patches.

Change-Id: I64d41507317081e1860a94b9481a85c8d400797d

It is no longer needed per the removed comment.

Change-Id: I8d88c21c7e115b842a56f0ba5c780c3bde593964

ansible-lint introduced var-spacing - let's fix our code.

Change-Id: I0d8aaf3c522a5a6a5495032f6dbed8a2be0251f0

Change-Id: Ib4b15ed4feac82d8492b1c0f0238a752eac668e6

We have only one value for install_type now and it gets removed from
image names.

Change-Id: I8bf95fd7aa9dd26b80d618ca0fcb097003b4cb0a

this adds back the ability to configure
the rabbitmq/erlang kernel network interface
which was removed in https://review.opendev.org/#/c/584427/
seemingly by accident.

Closes-Bug: 1900160

Change-Id: I6f00396495853e117429c17fadfafe809e322a31

Follow up to I91d0e23b22319cf3fdb7603f5401d24e3b76a56e, which fixes a
conditional corner case when removing the ha-all policy.

Change-Id: Iea75551bc6d0da7dd10515dd8bd28c014eed7a5e

When OpenStack is deployed with Kolla-Ansible, by default there
are no durable queues or exchanges created by the OpenStack
services in RabbitMQ. In Rabbit terminology, not being durable
is referred to as `transient`, and this means that the queue
is generally held in memory.

Whether OpenStack services create durable or transient queues is
traditionally controlled by the Oslo Notification config option:
`amqp_durable_queues`. In Kolla-Ansible, this remains set to
the default of `False` in all services. The only `durable`
objects are the `amq*` exchanges which are internal to RabbitMQ.

More recently, Oslo Notification has introduced support for
Quorum queues [7]. These are a successor to durable classic
queues, however it isn't yet clear if they are a good fit for
OpenStack in general [8].

For clustered RabbitMQ deployments, Kolla-Ansible configures all
queues as `replicated` [1]. Replication occurs over all nodes
in the cluster. RabbitMQ refers to this as 'mirroring of classic
queues'.

In summary, this means that a multi-node Kolla-Ansible deployment
will end up with a large number of transient, mirrored queues
and exchanges. However, the RabbitMQ documentation warns against
this, stating that 'For replicated queues, the only reasonable
option is to use durable queues: [2]`. This is discussed
further in the following bug report: [3].

Whilst we could try enabling the `amqp_durable_queues` option
for each service (this is suggested in [4]), there are
a number of complexities with this approach, not limited to:

1) RabbitMQ is planning to remove classic queue mirroring in
   favor of 'Quorum queues' in a forthcoming release [5].
2) Durable queues will be written to disk, which may cause
   performance problems at scale. Note that this includes
   Quorum queues which are always durable.
3) Potential for race conditions and other complexity
   discussed recently on the mailing list under:
   `[ops] [kolla] RabbitMQ High Availability`

The remaining option, proposed here, is to use classic
non-mirrored queues everywhere, and rely on services to recover
if the node hosting a queue or exchange they are using fails.
There is some discussion of this approach in [6]. The downside
of potential message loss needs to be weighed against the real
upsides of increasing the performance of RabbitMQ, and moving
to a configuration which is officially supported and hopefully
more stable. In the future, we can then consider promoting
specific queues to quorum queues, in cases where message loss
can result in failure states which are hard to recover from.

[1] https://www.rabbitmq.com/ha.html
[2] https://www.rabbitmq.com/queues.html
[3] https://github.com/rabbitmq/rabbitmq-server/issues/2045
[4] https://wiki.openstack.org/wiki/Large_Scale_Configuration_Rabbit
[5] https://blog.rabbitmq.com/posts/2021/08/4.0-deprecation-announcements/
[6] https://fuel-ccp.readthedocs.io/en/latest/design/ref_arch_1000_nodes.html#replication
[7] https://bugs.launchpad.net/oslo.messaging/+bug/1942933
[8] https://www.rabbitmq.com/quorum-queues.html#use-cases

Partial-Bug: #1954925
Change-Id: I91d0e23b22319cf3fdb7603f5401d24e3b76a56e

Change-Id: I547ab4b05aa14ed3bbee8be2dc77a6840d4816f6

Move new variables added in I4d694d6224c813285d228d6bc7eece5731db1078 to
role defaults.

Change-Id: Ie09a2dbae2701cb18fd1eb5bfab76e82f9920fb3

rabbitmq starting from 3.8.0, built-in Prometheus support,
prometheus plugins are enabled by default, when the environment is
"enable_prometheus is no", rabbitmq role will disable prometheus plugins

Closes-Bug: #1885106

Change-Id: I4d694d6224c813285d228d6bc7eece5731db1078

Role vars have a higher precedence than role defaults. This allows to
import default vars from another role via vars_files without overriding
project_name (see related bug for details).

Change-Id: I3d919736e53d6f3e1a70d1267cf42c8d2c0ad221
Related-Bug: #1951785

We get a nice optimisation by using a filtered loop instead
of task skipping per service with 'when'.

Partially-Implements: blueprint performance-improvements
Change-Id: I8f68100870ab90cb2d6b68a66a4c97df9ea4ff52

As mentioned in the Iced014acee7e590c10848e73feca166f48b622dc
commit message, in Ussuri+ we can use ``+sbwtdcpu none
+sbwtdio none`` as well. This is due to relying on RMQ-provided
erlang in version 23.x.

This change adds the extra arguments by default.
It should be backported down to Ussuri before we do a release with
Iced014acee7e590c10848e73feca166f48b622dc.

Change-Id: I32e247a6cb34d7f6763b544f247fd408dce2b3a2