Fixed Magnum role failed to create trustee user role when running
with Ansible check mode enabled.

Closes-Bug: 2095519
Change-Id: I6627e12e53e7223340b26cfbe6e6e1f2c6a3d4fe
(cherry picked from commit 72ffcfb6)

It has been added in magnum-cluster-api driver in v0.15.0

[1]: https://github.com/vexxhost/magnum-cluster-api/commit/2a53f3e340524deee3ddbf08b41071fba070d7d3#diff-50c86b7ed8ac2cf95bd48334961bf0530cdc77b5a56f852c5c61b89d735fd711R53

Closes-Bug: #2047360

Change-Id: Ib02389c03ab8f61fdd3827cb30fcc18f3dc952a9
(cherry picked from commit 99e61073)

Build upon changes in kolla which change strategy of installing projects
in containers when in dev mode. This fixes problems where when package
file manifest changes, the changes were not reflected in to
devmode-enabled container.

It changes the strategy of installing projects in dev mode in containers.
Instead of bind mounting the project's git repository to the venv
of the container, the repository is bind mounted to
/dev-mode/<project_name> from which the it is installed using pip
on every startup of the container using kolla_install_projects script.

Also updates docs to reflect the changes.

Depends-On: https://review.opendev.org/c/openstack/kolla/+/925712


Closes-Bug: #1814515
Singed-off-by: Roman Krček <roman.krcek@tietoevry.com>
Change-Id: If191cd0e3fcf362ee058549a1b6c244d109b6d9a

For possible config options see docs
https://docs.openstack.org/keystonemiddleware/latest/middlewarearchitecture.html#memcache-protection



Closes-bug: #1850733
Signed-off-by: Roman Krček <roman.krcek@tietoevry.com>
Change-Id: I169e27899f7350f5eb8adb1f81a062c51e6cbdfc

Refactor that prepares kolla_container_facts
module for introducing more actions that will be moved
from kolla_container module and kolla_container_volume_facts.

This change is based on a discussion about adding a new action
to kolla_container module that retrieves all names of the running
containers. It was agreed that kolla-ansible should follow Ansible's
direction of splitting modules between action modules and facts
modules. Because of this, kolla_container_facts needs to be able
to handle different requests for data about containers or volumes.

Change-Id: Ieaec8f64922e4e5a2199db2d6983518b124cb4aa
Signed-off-by: Ivan Halomi <ivan.halomi@tietoevry.com>

The Kolla project supports building images with
user-defined prefixes. However, Kolla-ansible is unable
to use those images for installation.

This patch fixes that issue.

Closes-Bug: #2073541
Change-Id: Ia8140b289aa76fcd584e0e72686e3786215c5a99

Most roles are not leveraging the jinja filters available.
According to [1] filtering the list of services makes the execution
faster than skipping the tasks.

This patchset also includes some cosmetic changes to genconfig.
Individual services are now also using a jinja filter. This has
no impact on performance, just makes the tasks look cleaner.

Naming of some vars in genconfig was changed to "service" to make
the tasks more uniform as some were previously using
the service name and some were using "service".

Three metrics from the deployment were taken and those were
- overall deployment time [s]
- time spent on the specific role [s]
- CPU usage (measured with perf) [-]
Overall genconfig time went down on avg. from 209s to 195s
Time spent on the loadbalancer role went down on avg. from 27s to 23s
Time spent on the neutron role went down on avg from 102s to 95s
Time spent on the nova-cell role went down on avg. from 54s to 52s
Also the average CPUs utilized reported by perf went down
from 3.31 to 3.15.
For details of how this was measured see the comments in gerrit.

[1] - https://github.com/stackhpc/ansible-scaling/blob/master/doc/skip.md



Change-Id: Ib0f00aadb6c7022de6e8b455ac4b9b8cd6be5b1b
Signed-off-by: Roman Krček <roman.krcek@tietoevry.com>

This change automates the prometheus blackbox monitoring configuration
for common endpoints. Custom endpoints can be added to
prometheus_blackbox_exporter_endpoints_custom.

Change-Id: Id6f51a2bebee3ab63b84ca7032aad17c2933838c

Service user passwords will now be updated in keystone if services are
reconfigured with new passwords set in config. This behaviour can be
overridden.

Closes-Bug: #2045990
Change-Id: I91671dda2242255e789b521d19348b0cccec266f

Closes-Bug: #2047360
Change-Id: I73490d84da39a74ea7ac493c7dd41fe7bfe2f578

If a file {{ node_custom_config }}/magnum/kubeconfig exists, it is
copied to /var/lib/magnum/.kube/config in all Magnum Service Containers.
At this location, the vexxhost/magnum-cluster-api will loo for the Kubeconfig
configuration file to control the Cluster API Control Plane. If the
vexxhost/magnum-cluster-api is installed in the Magnum container images,
control of a cluster API control plane can then take place via the Magnum API.

Depends-On: https://review.opendev.org/c/openstack/kolla/+/902101
Change-Id: I986c5192fe96b9c480a2d8fa87d719a50ce78186

This implements a global toggle `om_enable_rabbitmq_quorum_queues`
to enable quorum queues for each service in RabbitMQ, similar to
what was done for HA[0].

Quorum Queues are enabled by default.

Quorum queues are more reliable, safer, simpler and faster than
replicated mirrored classic queues[1].

Mirrored classic queues are deprecated and scheduled for removal
in RabbitMQ 4.0[2].

Notice, that we do not need a new policy in the RabbitMQ definitions
template, because their usage is enabled on the client side and can't
be set using a policy[3].

Notice also, that quorum queues are not yet enabled in oslo.messaging
for the usage of reply_ and fanout_ queues (transient queues).
This will change once[4] is merged.

[0]: https://review.opendev.org/c/openstack/kolla-ansible/+/867771
[1]: https://www.rabbitmq.com/quorum-queues.html
[2]: https://blog.rabbitmq.com/posts/2021/08/4.0-deprecation-announcements/
[3]: https://www.rabbitmq.com/quorum-queues.html#declaring
[4]: https://review.opendev.org/c/openstack/oslo.messaging/+/888479



Signed-off-by: Sven Kieske <kieske@osism.tech>
Change-Id: I6c033d460a5c9b93c346e9e47e93b159d3c27830

Changes name of ansible module kolla_docker to
kolla_container.

Change-Id: I13c676ed0378aa721a21a1300f6054658ad12bc7
Signed-off-by: Martin Hiner <m.hiner@partner.samsung.com>

docker_restart_policy: no causes systemd units to not get created
and we use it in CI to disable restarts on services.

Introducing oneshot policy to not create systemd unit for oneshot
containers (those that are running bootstrap tasks, like db
bootstrap and don't need a systemd unit), but still create systemd
units for long lived containers but with Restart=No.

Change-Id: I9e0d656f19143ec2fcad7d6d345b2c9387551604

Change-Id: I901c0a57efcb6cbaaac43f64f2243fff7d7980c8

Use case: exposing single external https frontend and
load balancing services using FQDNs.

Support different ports for internal and external endpoints.

Introduced kolla_url filter to normalize urls like:
- https://magnum.external:443/v1
- http://magnum.external:80/v1



Change-Id: I9fb03fe1cebce5c7198d523e015280c69f139cd0
Co-Authored-By: Jakub Darmach <jakub@stackhpc.com>

Change-Id: I54e68a3002d69f7b1be2704259c6a072f81aa586

A combination of durable queues and classic queue mirroring can be used
to provide high availability of RabbitMQ. However, these options should
only be used together, otherwise the system will become unstable. Using
the flag ``om_enable_rabbitmq_high_availability`` will either enable
both options at once, or neither of them.

There are some queues that should not be mirrored:
* ``reply`` queues (these have a single consumer and TTL policy)
* ``fanout`` queues (these have a TTL policy)
* ``amq`` queues (these are auto-delete queues, with a single consumer)
An exclusionary pattern is used in the classic mirroring policy. This
pattern is ``^(?!(amq\\.)|(.*_fanout_)|(reply_)).*``

Change-Id: I51c8023b260eb40b2eaa91bd276b46890c215c25

When running in check mode, some prechecks previously failed because
they use the command module which is silently not run in check mode.
Other prechecks were not running correctly in check mode due to e.g.
looking for a string in empty command output or not querying which
containers are running.

This change fixes these issues.

Closes-Bug: #2002657
Change-Id: I5219cb42c48d5444943a2d48106dc338aa08fa7c

The ``[oslo_messaging_rabbit] heartbeat_in_pthread`` config option
is set to ``true`` for wsgi applications to allow the RabbitMQ
heartbeats to function. For non-wsgi applications it is set to ``false``
as it may otherwise break the service [1].

[1] https://docs.openstack.org/releasenotes/oslo.messaging/zed.html#upgrade-notes

Change-Id: Id89bd6158aff42d59040674308a8672c358ccb3c

Regularly, we experience issues in Kolla Ansible deployments because we
use wrong options in OpenStack configuration files. This is because
OpenStack services ignore unknown options. We also need to keep on top
of deprecated options that may be removed in the future. Integrating
oslo-config-validator into Kolla Ansible will greatly help.

Adds a shared role to run oslo-config-validator on each service. Takes
into account that services have multiple containers, and these may also
use multiple config files. Service roles are extended to use this shared
role. Executed with the new command ``kolla-ansible validate-config``.

Change-Id: Ic10b410fc115646d96d2ce39d9618e7c46cb3fbc

Second part of patchset:
https://review.opendev.org/c/openstack/kolla-ansible/+/799229/


in which was suggested to split patch into smaller ones.

THis change adds container_engine to module parameters
so when we introduce podman, kolla_toolbox can be used
for both engines.

Signed-off-by: Ivan Halomi <i.halomi@partner.samsung.com>
Co-authored-by: Martin Hiner <m.hiner@partner.samsung.com>
Change-Id: Ic2093aa9341a0cb36df8f340cf290d62437504ad

Second part of patchset:
https://review.opendev.org/c/openstack/kolla-ansible/+/799229/


in which was suggested to split patch into smaller ones.

This change adds container_engine variable to kolla_container_facts
module, this prepares module to be used with docker and podman as well
without further changes in roles.

Signed-off-by: Ivan Halomi <i.halomi@partner.samsung.com>
Co-authored-by: Martin Hiner <m.hiner@partner.samsung.com>
Change-Id: I9e8fa30646844ab4a288555f3aafdda345b3a118

mainly jinja spacing and jinja[invalid] related

Change-Id: I6f52f2b0c1ef76de626657d79486d31e0f47f384

This patch adds loadbalancer-config role
which is "wrapper" around haproxy-config
and proxysql-config role which will be added
in follow-up patches.

Change-Id: I64d41507317081e1860a94b9481a85c8d400797d

Depends-On: https://review.opendev.org/c/openstack/kolla/+/769385
Depends-On: https://review.opendev.org/c/openstack/kolla/+/765781

Change-Id: I3c4182a6556dafd2c936eaab109a068674058fca

ansible-lint introduced var-spacing - let's fix our code.

Change-Id: I0d8aaf3c522a5a6a5495032f6dbed8a2be0251f0

Render {{ openstack_service_workers }} for workers
of each openstack service is not enough. There are
several services which has to have more workers because
there are more requests sent to them.

This patch is just adding default value for workers for
each service and sets {{ openstack_service_workers }} as
default, so value can be overrided in hostvars per server.
Nothing changed for normal user.

Change-Id: Ifa5863f8ec865bbf8e39c9b2add42c92abe40616

Per comments on [1].

[1] https://review.opendev.org/c/openstack/kolla-ansible/+/843727

Change-Id: I60162b54bc06e158534d29311d4474b34750c64d

Fixes an issue where access rules failed to validate:

    Cannot validate request with restricted access rules. Set
    service_type in [keystone_authtoken] to allow access rule validation

I've used the values from the endpoint. This was mostly a straight
forward copy and paste, except:

- versioned endpoints e.g cinderv3 where I stripped the version
- monasca has multiple endpoints associated with a single service. For
  this, I concatenated logging and monitoring to be logging-monitoring.

Closes-Bug: #1965111
Change-Id: Ic4b3ab60abad8c3dd96cd4923a67f2a8f9d195d7

Following up on [1].
The 3 variables are only introducing noise after we removed
the reliance on Keystone's admin port.

[1] I5099b08953789b280c915a6b7a22bdd4e3404076

Change-Id: I3f9dab93042799eda9174257e604fd1844684c1c

Change-Id: Ib4b15ed4feac82d8492b1c0f0238a752eac668e6

We have only one value for install_type now and it gets removed from
image names.

Change-Id: I8bf95fd7aa9dd26b80d618ca0fcb097003b4cb0a

Role vars have a higher precedence than role defaults. This allows to
import default vars from another role via vars_files without overriding
project_name (see related bug for details).

Change-Id: I3d919736e53d6f3e1a70d1267cf42c8d2c0ad221
Related-Bug: #1951785

The admin interface for endpoints never had any real use, the
functionality was the same as for the public or internal endpoints,
except for Keystone. Even for Keystone with API v3 it would no longer
really be needed, but it is still being required by some libraries that
cannot be changed in order to stay backwards compatible.

Signed-off-by: Dr. Jens Harbott <harbott@osism.tech>
Change-Id: Icf3bf08deab2c445361f0a0124d87ad8b0e4e9d9

We get a nice optimisation by using a filtered loop instead
of task skipping per service with 'when'.

Partially-Implements: blueprint performance-improvements
Change-Id: I8f68100870ab90cb2d6b68a66a4c97df9ea4ff52

By default, Ansible injects a variable for every fact, prefixed with
ansible_. This can result in a large number of variables for each host,
which at scale can incur a performance penalty. Ansible provides a
configuration option [0] that can be set to False to prevent this
injection of facts. In this case, facts should be referenced via
ansible_facts.<fact>.

This change updates all references to Ansible facts within Kolla Ansible
from using individual fact variables to using the items in the
ansible_facts dictionary. This allows users to disable fact variable
injection in their Ansible configuration, which may provide some
performance improvement.

This change disables fact variable injection in the ansible
configuration used in CI, to catch any attempts to use the injected
variables.

[0] https://docs.ansible.com/ansible/latest/reference_appendices/config.html#inject-facts-as-vars

Change-Id: I7e9d5c9b8b9164d4aee3abb4e37c8f28d98ff5d1
Partially-Implements: blueprint performance-improvements

Magnum has various sections in its configuration file for OpenStack
clients. When internal TLS is enabled, these may need a CA certificate
to be specified.

This change adds a CA certificate configuration, based on
openstack_cacert, for all clients using internal endpoints.

Note: we are explicitly not adding the configuration for the
[magnum_client] ca_file and [drivers] openstack_ca_file options, since
these use the public endpoint by default. These options may be
provided via custom configuration if necessary.

Change-Id: Ie59b3777c0a2c142b580addd67e279bc4b2f2c90
Co-Authored-By: Kyle Dean
Closes-Bug: #1919389

follow: https://review.opendev.org/c/openstack/kolla-ansible/+/791980
Change-Id: I7231ae0b2702d56879092a2c34b7f8bb3b07f50b

Closes-Bug: #1933025

Change-Id: Ib67d715ddfa986a5b70a55fdda39e6d0e3333162