Closes-Bug: #2086171
Change-Id: I4001ddad198dcf40f71e9196e126cc78a19d1437
(cherry picked from commit fa54e69e)

As version-check.yml is added to deploy.yml, we must make sure the
tasks are only run when the rabbitmq container exists.

Change-Id: Iaa31bae739110094affb5e402ed9ac40b153ac3d
(cherry picked from commit f5ad7829)

Given we bump the RabbitMQ version each release, there is a manual
upgrade to an intermediary RabbitMQ version needed before a skip-level
upgrade can be performed.

Change-Id: Id8a5ebe19a50ebdc59d12667889472c803b8d7c8

Required before a SLURP upgrade

Change-Id: I09a45d26a6075554b204e007f64122f23de5f53c

Refactor that prepares kolla_container_facts
module for introducing more actions that will be moved
from kolla_container module and kolla_container_volume_facts.

This change is based on a discussion about adding a new action
to kolla_container module that retrieves all names of the running
containers. It was agreed that kolla-ansible should follow Ansible's
direction of splitting modules between action modules and facts
modules. Because of this, kolla_container_facts needs to be able
to handle different requests for data about containers or volumes.

Change-Id: Ieaec8f64922e4e5a2199db2d6983518b124cb4aa
Signed-off-by: Ivan Halomi <ivan.halomi@tietoevry.com>

The Kolla project supports building images with
user-defined prefixes. However, Kolla-ansible is unable
to use those images for installation.

This patch fixes that issue.

Closes-Bug: #2073541
Change-Id: Ia8140b289aa76fcd584e0e72686e3786215c5a99

Most roles are not leveraging the jinja filters available.
According to [1] filtering the list of services makes the execution
faster than skipping the tasks.

This patchset also includes some cosmetic changes to genconfig.
Individual services are now also using a jinja filter. This has
no impact on performance, just makes the tasks look cleaner.

Naming of some vars in genconfig was changed to "service" to make
the tasks more uniform as some were previously using
the service name and some were using "service".

Three metrics from the deployment were taken and those were
- overall deployment time [s]
- time spent on the specific role [s]
- CPU usage (measured with perf) [-]
Overall genconfig time went down on avg. from 209s to 195s
Time spent on the loadbalancer role went down on avg. from 27s to 23s
Time spent on the neutron role went down on avg from 102s to 95s
Time spent on the nova-cell role went down on avg. from 54s to 52s
Also the average CPUs utilized reported by perf went down
from 3.31 to 3.15.
For details of how this was measured see the comments in gerrit.

[1] - https://github.com/stackhpc/ansible-scaling/blob/master/doc/skip.md



Change-Id: Ib0f00aadb6c7022de6e8b455ac4b9b8cd6be5b1b
Signed-off-by: Roman Krček <roman.krcek@tietoevry.com>

Also enable these after an upgrade.

Partial-Bug: #2058512
Change-Id: Ib9bdae2e25c2b6cce30e4c8024015ab5875bc1ff

It's inactive and hasn't produced a 2024.1 release [1].
In addition to that, there's a CVE that hasn't really been patched [2].

Also drop outward_rabbitmq that was used only with Murano.

[1]: https://governance.openstack.org/tc/reference/emerging-technology-and-inactive-projects.html#current-inactive-projects
[2]: https://lists.openstack.org/archives/list/openstack-announce@lists.openstack.org/thread/4FYM6GSIM5WZSJQIG4TT5Q3UBKQIHLWX/

Change-Id: I691205730b0e10a42ce61f3340cc39ee51bd1010

Fix existing spelling errors

Change-Id: Ie689cf5a344aaa630a4860448b09242333a8e119

Closes-Bug: #2057676

Change-Id: I9e0287a4e80b1ebcecf9e3b66c11d4233970a30b

As per [1].

[1]: https://rabbitmq-website.pages.dev/docs/feature-flags

Depends-On: https://review.opendev.org/c/openstack/kolla/+/911093

Change-Id: Ib5bfc99a5023e4b949c1ea38eca9bfd1ea9cd633

Change-Id: Ic9bd25a09b860838910dbe3d55f94421a0461c57

Adds a precheck to fail if non-quorum queues are found in RabbitMQ.

Currently excludes fanout and reply queues, pending support in
oslo.messaging [1].

[1]: https://review.opendev.org/c/openstack/oslo.messaging/+/888479

Closes-Bug: #2045887
Change-Id: Ibafdcd58618d97251a3405ef9332022d4d930e2b

Changes name of ansible module kolla_docker to
kolla_container.

Change-Id: I13c676ed0378aa721a21a1300f6054658ad12bc7
Signed-off-by: Martin Hiner <m.hiner@partner.samsung.com>

docker_restart_policy: no causes systemd units to not get created
and we use it in CI to disable restarts on services.

Introducing oneshot policy to not create systemd unit for oneshot
containers (those that are running bootstrap tasks, like db
bootstrap and don't need a systemd unit), but still create systemd
units for long lived containers but with Restart=No.

Change-Id: I9e0d656f19143ec2fcad7d6d345b2c9387551604

Kolla have updated the RabbitMQ image version from 3.11 to 3.12. While
in 3.12, there are a few feature flags that is marked as required [1].
If those are not enable, RabbitMQ will fail to start after upgrade.

This commit added additional required feature flags to the
rabbitmq_feature_flags list.

[1] https://www.rabbitmq.com/feature-flags.html#core-feature-flags

Change-Id: I401bbaae059bca5c656269f65ac8b7bff0aefae2

This command can be invoked with ``kolla-ansible rabbitmq-reset-state``.
This is primarily designed to be used when enabling HA queues[1].

As such, this also updates the RabbitMQ documentation to use this
command.

[1] https://docs.openstack.org/kolla-ansible/latest/reference/message-queues/rabbitmq.html#high-availability

Change-Id: I6ad95a3618fc1a34af56657ef99ef14dc979f17a

Use case: exposing single external https frontend and
load balancing services using FQDNs.

Support different ports for internal and external endpoints.

Introduced kolla_url filter to normalize urls like:
- https://magnum.external:443/v1
- http://magnum.external:80/v1



Change-Id: I9fb03fe1cebce5c7198d523e015280c69f139cd0
Co-Authored-By: Jakub Darmach <jakub@stackhpc.com>

Hardcoded docker value in commands is not supported anymore
and kolla_container_engine is used instead.

Change-Id: I25d9563c82842ac51d41467ff7b4144b306fdb12
Signed-off-by: Ivan Halomi <i.halomi@partner.samsung.com>

Ansible 2.14.3 introduced a change that broke the method used for
restarting MariaDB and RabbitMQ serially [1][2]. In
I57425680a4cdbf0daeb9b2cc35920f1b933aa4a8 we limited to 2.14.2 to work
around this. Ansible upstream claim this behaviour was unintentional,
and will not fix it.

This change moves to a different approach where we use separate plays
with a 'serial' keyword to execute the restart.

This change also removes the restriction on the maximum supported
version of 2.14.2 on ansible-core - any 2.14 release is now supported.

[1] https://github.com/ansible/ansible/commit/65366f663de7d044f42ae6dd53368fd4c1f88b35
[2] https://github.com/ansible/ansible/issues/80848

Depends-On: https://review.opendev.org/c/openstack/kolla/+/884208

Change-Id: I5a12670d07077d24047aaff57ce8d33ccf7156ff

When using externally managed certificates, according to [1],
one should set `kolla_externally_managed_cert: yes` and ensure
that the certificates are in the correct place.

However, RabbitMQ precheck still expects the certificates to be
available on the controller node. This is incorrect.

Fix by not running the tasks in question when `kolla_externally_managed_cert: yes`

[1] https://docs.openstack.org/kolla-ansible/latest/admin/tls.html



Closes-Bug: 1999081
Related-Bug: 1940286
Signed-off-by: Magnus Lööf <magnus.loof@basalt.se>
Change-Id: I9f845a7bdf5055165e199ab1887ed3ccbfb9d808

Change-Id: I1d8021a1bc780449e3ca96183c6f4abaed17b382

Currently, the process of enabling RabbitMQ HA with the variable
``om_enable_rabbitmq_high_availbility`` requires some manual steps to
migrate from transient to mirrored queues. In preparation for setting
this variable to ``True`` by default, this adds a precheck that will
fail if a system is currently running non-mirrored queues and
``om_enable_rabbitmq_high_availbility`` is set to ``True``.

Includes a helpful message informing the operator of their choice.
Either follow the manual procedure to migrate the queues described in
the docs, or set ``om_enable_rabbitmq_high_availbility`` to ``False``.

The RabbitMQ HA section of the reference docs is updated to include
these instructions.

Change-Id: Ic5e64998bd01923162204f7bb289cc110187feec

With the addition of the variable
`om_enable_rabbitmq_high_availability`, this feature in the upgrade
task should be brought back. It is also now used in the deploy task. The
`ha-all` policy is cleared only when
`om_enable_rabbitmq_high_availability` is set to `false`.

Change-Id: Ia056aa40e996b1f0fed43c0f672466c7e4a2f547

Puts the RabbitMQ node into maintenance mode before restarting the
container. This will make the node shutdown less disruptive. For details
on what maintenance mode does, see:
https://www.rabbitmq.com/upgrade.html#maintenance-mode

Change-Id: Ia61573f3fb95fe8fcde6b789ca77ef5b45fe0a65

Since RMQ 3.8 we can use rolling upgrade [1].

Depends-On: https://review.opendev.org/c/openstack/kolla/+/872393

[1]: https://www.rabbitmq.com/upgrade.html#rolling-upgrades

Change-Id: If6a7c6c12d9226a2406728108b3c87b3485ac55f

Following ideas here:
https://wiki.openstack.org/wiki/Large_Scale_Configuration_Rabbit

Make sure old messages with no consumer are dropped after the message
TTL of 10 mins, longer than the 1 min RPC timeout.
Also ensure queues expire after an hour of inactivity, so queues from
removed nodes or renamed nodes don't grow over time.

Change-Id: Ifb28ac68b6328adb604a7474d01e5f7a47b2e788

Adds two new flags to alter behaviour in RabbitMQ:
    * `rabbitmq_message_ttl_ms`, which lets you set a TTL on messages.
    * `rabbitmq_queue_expiry_ms`, which lets you set an expiry time on queues.
See https://www.rabbitmq.com/ttl.html for more information on both.

Change-Id: I51ca37ffbb1bb5c07f2d39873f0f33ca20263f2a

Changes the default value of `rabbitmq-ha-promote-on-shutdown` to
`"always"`.

We are seeing issues with RabbitMQ automatically recovering when nodes
are restarted. https://www.rabbitmq.com/ha.html#cluster-shutdown

Rather than waiting for operator interventions, it is better we allow
recovery to happen, even if that means we may loose some messages.
A few failed and timed out operations is better than a totaly broken
cloud. This is achieved using ha-promote-on-shutdown=always.

Note, when a node failure is detected, this is already the default
behaviour from 3.7.5 onwards:
https://www.rabbitmq.com/ha.html#promoting-unsynchronised-mirrors

Related-Bug: #1954925
Change-Id: I484a81163f703fa27112df22473d657e2a9ab964

With the parameter rabbitmq_datadir_volume it is possible
to use a directory as volume for the rabbitmq service. By default,
a volume named rabbitmq is used (the previous default).

Change-Id: I99d6bd71ca79cba81062dedfb767c5ed341bb182

Currently we do not follow the RabbitMQ advice on replicas here:
https://www.rabbitmq.com/ha.html#replication-factor

Here we reduce the number of replicas to n // 2 + 1 as advised
above. The hope it this helps speed up recovery from rabbit
issues.

Related-Bug: #1954925
Change-Id: Ib6bcb26c499c9884faa4a0cd51abaec00cacb096

Adds the flag `rabbitmq_ha_replica_count` to change how many different
nodes a queue should be mirrored across. If the value is not set, then
it defaults to "ha-mode":"all". This value is unset by default to avoid
any unexpected changes to the RabbitMQ definitions.json file, as that
would trigger an unexpected restart of RabbitMQ during the next deploy.

Change-Id: Iee98cd937197a73a3b04aa8501fa325e8ecfff24

By default ha-promote-on-shutdown=when-synced. However we are seeing
issues with RabbitMQ automatically recovering when nodes are restarted.
https://www.rabbitmq.com/ha.html#cluster-shutdown

Rather than waiting for operator interventions, it is better we allow
recovery to happen, even if that means we may loose some messages.
A few failed and timed out operations is better than a totaly broken
cloud. This is achieved using ha-promote-on-shutdown=always.

Note, when a node failure is detected, this is already the default
behaviour from 3.7.5 onwards:
https://www.rabbitmq.com/ha.html#promoting-unsynchronised-mirrors

This patch adds the option to change the ha-promote-on-shutdown
definition, using the flag `rabbitmq_ha_promote_on_shutdown`. This
value is unset by default to avoid any unexpected changes to the
RabbitMQ definitions.json file, as that would trigger an unexpected
restart of RabbitMQ during the next deploy.

Related-Bug: #1954925

Change-Id: I2146bda2c72ddac2c9923c6941b0596395fd9ab5

As rabbitmq's configuration file is not ini or yaml file,
there is no option to extend configuration by new config
options via merge_configs or merge_yaml.

This patch moves config options to dictionary
so it can be overriden in /etc/kolla/globals.yml.

Change-Id: I5cd772f4fb80a0e200fb24d67be735ca81e3fdeb

A combination of durable queues and classic queue mirroring can be used
to provide high availability of RabbitMQ. However, these options should
only be used together, otherwise the system will become unstable. Using
the flag ``om_enable_rabbitmq_high_availability`` will either enable
both options at once, or neither of them.

There are some queues that should not be mirrored:
* ``reply`` queues (these have a single consumer and TTL policy)
* ``fanout`` queues (these have a TTL policy)
* ``amq`` queues (these are auto-delete queues, with a single consumer)
An exclusionary pattern is used in the classic mirroring policy. This
pattern is ``^(?!(amq\\.)|(.*_fanout_)|(reply_)).*``

Change-Id: I51c8023b260eb40b2eaa91bd276b46890c215c25

When running in check mode, some prechecks previously failed because
they use the command module which is silently not run in check mode.
Other prechecks were not running correctly in check mode due to e.g.
looking for a string in empty command output or not querying which
containers are running.

This change fixes these issues.

Closes-Bug: #2002657
Change-Id: I5219cb42c48d5444943a2d48106dc338aa08fa7c

assert will also fail when we're not meeting the conditions, makes
clear what we're actually testing, and isn't listed as a skipped task
when the condition is ok.

Change-Id: I4c919b523dde2602c81179ab3d28b913650b4c9f

Regularly, we experience issues in Kolla Ansible deployments because we
use wrong options in OpenStack configuration files. This is because
OpenStack services ignore unknown options. We also need to keep on top
of deprecated options that may be removed in the future. Integrating
oslo-config-validator into Kolla Ansible will greatly help.

Adds a shared role to run oslo-config-validator on each service. Takes
into account that services have multiple containers, and these may also
use multiple config files. Service roles are extended to use this shared
role. Executed with the new command ``kolla-ansible validate-config``.

Change-Id: Ic10b410fc115646d96d2ce39d9618e7c46cb3fbc

Second part of patchset:
https://review.opendev.org/c/openstack/kolla-ansible/+/799229/


in which was suggested to split patch into smaller ones.

This change adds container_engine variable to kolla_container_facts
module, this prepares module to be used with docker and podman as well
without further changes in roles.

Signed-off-by: Ivan Halomi <i.halomi@partner.samsung.com>
Co-authored-by: Martin Hiner <m.hiner@partner.samsung.com>
Change-Id: I9e8fa30646844ab4a288555f3aafdda345b3a118