Change from file to template

Closes-Bug: #2089229
Change-Id: I7cbc6a94608baf4f04ef231cc88397fc5dcf0a9b
(cherry picked from commit f30dd3e5)

Build upon changes in kolla which change strategy of installing projects
in containers when in dev mode. This fixes problems where when package
file manifest changes, the changes were not reflected in to
devmode-enabled container.

It changes the strategy of installing projects in dev mode in containers.
Instead of bind mounting the project's git repository to the venv
of the container, the repository is bind mounted to
/dev-mode/<project_name> from which the it is installed using pip
on every startup of the container using kolla_install_projects script.

Also updates docs to reflect the changes.

Depends-On: https://review.opendev.org/c/openstack/kolla/+/925712


Closes-Bug: #1814515
Singed-off-by: Roman Krček <roman.krcek@tietoevry.com>
Change-Id: If191cd0e3fcf362ee058549a1b6c244d109b6d9a

This patch removes the nova_libvirt_secret container volume because
it is a complete antipattern, and during testing, I found that
it causes problems. When it was necessary to copy libvirt secrets
from /etc/kolla/nova-libvirt/secrets, the container logs reported that
the resource is busy - precisely because it was a mounted container
volume. This, of course, is unnecessary because the secrets are copied
to the kolla host in /etc/kolla/nova-libvirt/secrets.

Closes-Bug: #2073678
Change-Id: I715a6a95f9d32d62a8199727ddbaddd0dd7baa2d

Refactor that prepares kolla_container_facts
module for introducing more actions that will be moved
from kolla_container module and kolla_container_volume_facts.

This change is based on a discussion about adding a new action
to kolla_container module that retrieves all names of the running
containers. It was agreed that kolla-ansible should follow Ansible's
direction of splitting modules between action modules and facts
modules. Because of this, kolla_container_facts needs to be able
to handle different requests for data about containers or volumes.

Change-Id: Ieaec8f64922e4e5a2199db2d6983518b124cb4aa
Signed-off-by: Ivan Halomi <ivan.halomi@tietoevry.com>

This fixes an issue where it is not possible to customise the `host`
config option in the Nova Compute Ironic config file without breaking
detection of the service.

This is a backwards compatible fix, which allows a user to set the
`host` config option using Ansible host or group vars.

Other reasons for not using the default host setting of
`{{ ansible_hostname }}-ironic` are covered in [1].

[1] https://specs.openstack.org/openstack/nova-specs/specs/2024.1/approved/ironic-shards.html#migrate-from-peer-list-to-shard-key.

Closes-Bug: #2056571
Change-Id: I9b562f6a5722f21b7dbec2a4d53a46a57c829155

The Kolla project supports building images with
user-defined prefixes. However, Kolla-ansible is unable
to use those images for installation.

This patch fixes that issue.

Closes-Bug: #2073541
Change-Id: Ia8140b289aa76fcd584e0e72686e3786215c5a99

Most roles are not leveraging the jinja filters available.
According to [1] filtering the list of services makes the execution
faster than skipping the tasks.

This patchset also includes some cosmetic changes to genconfig.
Individual services are now also using a jinja filter. This has
no impact on performance, just makes the tasks look cleaner.

Naming of some vars in genconfig was changed to "service" to make
the tasks more uniform as some were previously using
the service name and some were using "service".

Three metrics from the deployment were taken and those were
- overall deployment time [s]
- time spent on the specific role [s]
- CPU usage (measured with perf) [-]
Overall genconfig time went down on avg. from 209s to 195s
Time spent on the loadbalancer role went down on avg. from 27s to 23s
Time spent on the neutron role went down on avg from 102s to 95s
Time spent on the nova-cell role went down on avg. from 54s to 52s
Also the average CPUs utilized reported by perf went down
from 3.31 to 3.15.
For details of how this was measured see the comments in gerrit.

[1] - https://github.com/stackhpc/ansible-scaling/blob/master/doc/skip.md



Change-Id: Ib0f00aadb6c7022de6e8b455ac4b9b8cd6be5b1b
Signed-off-by: Roman Krček <roman.krcek@tietoevry.com>

It was deprecated in Antelope cycle.

Change-Id: I499e69ec6db63e4067e49376e2a1f3e01e48fe62

Fix existing spelling errors

Change-Id: Ie689cf5a344aaa630a4860448b09242333a8e119

When kolla VIP address is changed the cell0 database connection is
now updated to the new address.

Closes-bug: #1915302
Change-Id: I35be54efb5aaa230702d0cebaae04f1e64c3bca3
Signed-off-by: Roman Krček <roman.krcek@tietoevry.com>

Closes-Bug: #2049607

Change-Id: I14ae2be2e19ad06e3190e2e948bac7ce77e80d4b

This patch disables periodic compute.instance.exists
notifications when designate is enabled.

Related-Bug: #2049503
Change-Id: I39fe2db9182de23c1df814d911eec15e86317702

The addition of an instance resize operation [1] to CI testing is
triggering a failure in kolla-ansible-debian-ovn jobs, which are using a
nodeset with multiple nodes:

    oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
    Command: scp -r /var/lib/nova/instances/8ca2c7e8-acae-404c-af7d-6cac38e354b8_resize/disk 192.0.2.2:/var/lib/nova/instances/8ca2c7e8-acae-404c-af7d-6cac38e354b8/disk
    Exit code: 255
    Stdout: ''
    Stderr: "Warning: Permanently added '[192.0.2.2]:8022' (ED25519) to the list of known hosts.\r\nsubsystem request failed on channel 0\r\nscp: Connection closed\r\n"

This is not seen on Ubuntu Jammy, which uses OpenSSH 8.9, while Debian
Bookworm uses OpenSSH 9.2. This is likely related to this change in
OpenSSH 9.0 [2]:

    This release switches scp(1) from using the legacy scp/rcp protocol
    to using the SFTP protocol by default.

Configure sftp subsystem like on RHEL9 derivatives. Even though it is
not yet required for Ubuntu, we also configure it so we are ready for
the Noble release.

[1] https://review.opendev.org/c/openstack/kolla-ansible/+/904249
[2] https://www.openssh.com/txt/release-9.0

Closes-Bug: #2048700
Change-Id: I9f1129136d7664d5cc3b57ae5f7e8d05c499a2a5

This implements a global toggle `om_enable_rabbitmq_quorum_queues`
to enable quorum queues for each service in RabbitMQ, similar to
what was done for HA[0].

Quorum Queues are enabled by default.

Quorum queues are more reliable, safer, simpler and faster than
replicated mirrored classic queues[1].

Mirrored classic queues are deprecated and scheduled for removal
in RabbitMQ 4.0[2].

Notice, that we do not need a new policy in the RabbitMQ definitions
template, because their usage is enabled on the client side and can't
be set using a policy[3].

Notice also, that quorum queues are not yet enabled in oslo.messaging
for the usage of reply_ and fanout_ queues (transient queues).
This will change once[4] is merged.

[0]: https://review.opendev.org/c/openstack/kolla-ansible/+/867771
[1]: https://www.rabbitmq.com/quorum-queues.html
[2]: https://blog.rabbitmq.com/posts/2021/08/4.0-deprecation-announcements/
[3]: https://www.rabbitmq.com/quorum-queues.html#declaring
[4]: https://review.opendev.org/c/openstack/oslo.messaging/+/888479



Signed-off-by: Sven Kieske <kieske@osism.tech>
Change-Id: I6c033d460a5c9b93c346e9e47e93b159d3c27830

Changes name of ansible module kolla_docker to
kolla_container.

Change-Id: I13c676ed0378aa721a21a1300f6054658ad12bc7
Signed-off-by: Martin Hiner <m.hiner@partner.samsung.com>

docker_restart_policy: no causes systemd units to not get created
and we use it in CI to disable restarts on services.

Introducing oneshot policy to not create systemd unit for oneshot
containers (those that are running bootstrap tasks, like db
bootstrap and don't need a systemd unit), but still create systemd
units for long lived containers but with Restart=No.

Change-Id: I9e0d656f19143ec2fcad7d6d345b2c9387551604

Change-Id: I7df734cf15eba1af4da92f9c4ef12cc761442ca1

This change adds basic deployment based on Podman
container manager as an alternative to Docker.

Signed-off-by: Ivan Halomi <i.halomi@partner.samsung.com>
Signed-off-by: Martin Hiner <m.hiner@partner.samsung.com>
Signed-off-by: Petr Tuma <p.tuma@partner.samsung.com>
Change-Id: I2b52964906ba8b19b8b1098717b9423ab954fa3d
Depends-On: Ie4b4c1cf8fe6e7ce41eaa703b423dedcb41e3afc

Related to ansible-core bug [1].

[1]: https://github.com/ansible/ansible/issues/81945

Change-Id: Idede07a5f0fd1a3ee63e1c54a263e358e95b3a86

hostnqn is generated using to_uuid filter

Usually "nvme gen-hostnqn" command is used to generate
hostnqn, and it has the format of:
nqn.2014-08.org.nvmexpress:uuid:67dc8c8e-0262-4d81-ac51-ace7c25e4daa

"nqn.2014-08.org.nvmexpress:uuid:" is always static

Closes-Bug: #2035975

Change-Id: I6ece4fe8c18c0167a2707c24693fbe39ed15cdba

When using the 'q35' machine type, by default, it allows only
a single PCIe device to be hotplugged. Nova currently sets
'num_pcie_ports' to "0" (defaults to libvirt's "1"), which is
not sufficient for hotplug use.

The max value is 28, but 16 is enough in most cases.

More info please check

* [0] https://libvirt.org/pci-hotplug.html
* [1] https://marcin.juszkiewicz.com.pl/2018/02/19/hotplug-in-vm-easy-to-say/
* [2] https://review.opendev.org/c/openstack/nova/+/545034



Co-Authored-By: Bartosz Bezak <bartosz@stackhpc.com>

Change-Id: I7b8063566837aeb947927b7d6858417982b63deb
Closes-Bug: 1909063

For example, an operator may wish to customise the
nova-compute-ironic service configuration without affecting
other Nova services.

Closes-Bug: #2034949

Change-Id: If8648d8e85ab3dbcbb4ecba674b2e34b06898327

With the libvirt driver, during live migration,skip comparing guest CPU
with the destination host. When using QEMU >= 2.9 and libvirt >= 4.4.0,
libvirt will do the correct thing with respect to checking CPU
compatibility on the destination host during live migration.[1]

[1] https://opendev.org/openstack/nova/commit/267a40663cd8d0b94bbc5ebda4ece55a45753b64

Change-Id: I947c94b59368c7a2740583bf57e407296473d75e

Use case: exposing single external https frontend and
load balancing services using FQDNs.

Support different ports for internal and external endpoints.

Introduced kolla_url filter to normalize urls like:
- https://magnum.external:443/v1
- http://magnum.external:80/v1



Change-Id: I9fb03fe1cebce5c7198d523e015280c69f139cd0
Co-Authored-By: Jakub Darmach <jakub@stackhpc.com>

This patch is adding a feature for an option to copy different
ceph configuration files and corresponding keyrings for cinder,
glance, manila, gnocchi and nova services.

This is especially useful when the deployment uses availability
zones as below example.

  - Individual compute can read/write to individual ceph
    cluster in same AZ.
  - Cinder can write to several ceph clusters in several AZs.
  - Glance can use multistore and upload images to
    several ceph clusters in several AZs at once.

Change-Id: Ie4d8ab5a3df748137835cae1c943b9180cd10eb1

As of I3629b84d3255a8fe9d8a7cea8c6131d7c40899e8 nova
now requires the service_user section to be configured
to address CVE-2023-2088. This change adds
the service user section to the nova.conf template in
the nova and nova-cell roles.

Related-Bug: #2004555
Signed-off-by: Sven Kieske <kieske@osism.tech>
Change-Id: I2189dafca070accfd8efcd4b8cc4221c6decdc9f
(cherry picked from commit a77ea13ef1991543df29b7eea14b1f91ef26f858)
(cherry picked from commit 03c12abbcc107bfec451f4558bc97d14facae01c)
(cherry picked from commit cb105dc293ff1cdb11ab63fa3e3bf39fd17e0ee0)
(cherry picked from commit efe6650d09441b02cf93738a94a59723d84c5b19)

Add checking for container readiness before create sasl user

Closes-Bug: #2015589
Change-Id: Ic650ba6be1f192e3cbeaa94de3d00507636c1c92

Closes-Bug: #2005119
Change-Id: I542f7ae19b4400355b04854f42a1d1802a6efeea

When generating config, we do not want to restart any services.

TrivialFix

Change-Id: Ice13759c406ff7cec91c9d740b0e0c43eda99510

Previously, when running one of the following commands:

  kolla-ansible deploy --check
  kolla-ansible genconfig --check

deployment or configuration generation fails for various reasons.

MariaDB fails to lookup the existing cluster.

Keystone fails to generate cron config.

Nova-cell fails to get the cell settings.

Closes-Bug: #2002661
Change-Id: I5e765f498ae86d213d0a4379ca5d473db1499962

As per the RBAC new direction in Zed cycle, we have dropped the
system scope from API policies and all the policies are hardcoded
to project scoped so that any user accessing APIs using system scope
will get 403 error. It is dropped from all the OpenStack services
except for the Ironic service which will have system scope and to
support ironic only deployment, we are keeping system as well as project
scope in Keystone.

Complete discussion and direction can be found in the below gerrit
change and TC goal direction:

- https://review.opendev.org/c/openstack/governance/+/847418
- https://governance.openstack.org/tc/goals/selected/consistent-and-secure-rbac.html#the-issues-we-are-facing-with-scope-concept

As phase-2 of RBAC goal, services will start enabling the new
defaults and project scope by default. For example: Nova did in
- https://review.opendev.org/c/openstack/nova/+/866218

Kolla who start accessing the services using system scope token
- https://review.opendev.org/c/openstack/kolla-ansible/+/692179

This commit partially revert the above change except keeping
system scope usage for Keystone and Ironic. Rest all services are changed
to use the project scope token.

And enable the scope and new defaults for Nova which was disabled
by https://review.opendev.org/c/openstack/kolla-ansible/+/870804

Change-Id: I0adbe0a6c39e11d7c9542569085fc5d580f26c9d

In order to honour configured max number of attempts
it has to be presented in nova.conf inside of
nova_conductor container, otherwise the default value
of 3 will be used

Closes-Bug: #2003587
Change-Id: I928af332b8658223444594f96417830233057284

A combination of durable queues and classic queue mirroring can be used
to provide high availability of RabbitMQ. However, these options should
only be used together, otherwise the system will become unstable. Using
the flag ``om_enable_rabbitmq_high_availability`` will either enable
both options at once, or neither of them.

There are some queues that should not be mirrored:
* ``reply`` queues (these have a single consumer and TTL policy)
* ``fanout`` queues (these have a TTL policy)
* ``amq`` queues (these are auto-delete queues, with a single consumer)
An exclusionary pattern is used in the classic mirroring policy. This
pattern is ``^(?!(amq\\.)|(.*_fanout_)|(reply_)).*``

Change-Id: I51c8023b260eb40b2eaa91bd276b46890c215c25

When running in check mode, some prechecks previously failed because
they use the command module which is silently not run in check mode.
Other prechecks were not running correctly in check mode due to e.g.
looking for a string in empty command output or not querying which
containers are running.

This change fixes these issues.

Closes-Bug: #2002657
Change-Id: I5219cb42c48d5444943a2d48106dc338aa08fa7c

The ``[oslo_messaging_rabbit] heartbeat_in_pthread`` config option
is set to ``true`` for wsgi applications to allow the RabbitMQ
heartbeats to function. For non-wsgi applications it is set to ``false``
as it may otherwise break the service [1].

[1] https://docs.openstack.org/releasenotes/oslo.messaging/zed.html#upgrade-notes

Change-Id: Id89bd6158aff42d59040674308a8672c358ccb3c

Regularly, we experience issues in Kolla Ansible deployments because we
use wrong options in OpenStack configuration files. This is because
OpenStack services ignore unknown options. We also need to keep on top
of deprecated options that may be removed in the future. Integrating
oslo-config-validator into Kolla Ansible will greatly help.

Adds a shared role to run oslo-config-validator on each service. Takes
into account that services have multiple containers, and these may also
use multiple config files. Service roles are extended to use this shared
role. Executed with the new command ``kolla-ansible validate-config``.

Change-Id: Ic10b410fc115646d96d2ce39d9618e7c46cb3fbc

See https://github.com/ansible/ansible/pull/76677

Change-Id: If822dcfc4c1abf7a22be35ffd90fd05ee46cb0d9

Second part of patchset:
https://review.opendev.org/c/openstack/kolla-ansible/+/799229/


in which was suggested to split patch into smaller ones.

THis change adds container_engine to module parameters
so when we introduce podman, kolla_toolbox can be used
for both engines.

Signed-off-by: Ivan Halomi <i.halomi@partner.samsung.com>
Co-authored-by: Martin Hiner <m.hiner@partner.samsung.com>
Change-Id: Ic2093aa9341a0cb36df8f340cf290d62437504ad

Second part of patchset:
https://review.opendev.org/c/openstack/kolla-ansible/+/799229/


in which was suggested to split patch into smaller ones.

This change adds container_engine variable to kolla_container_facts
module, this prepares module to be used with docker and podman as well
without further changes in roles.

Signed-off-by: Ivan Halomi <i.halomi@partner.samsung.com>
Co-authored-by: Martin Hiner <m.hiner@partner.samsung.com>
Change-Id: I9e8fa30646844ab4a288555f3aafdda345b3a118

First part of patchset:
 https://review.opendev.org/c/openstack/kolla-ansible/+/799229/


in which was suggested to split patch into smaller ones.

This implements kolla_container_engine variable
in command calls of docker,so later on it can be
also used for podman without further change.

Signed-off-by: Ivan Halomi <i.halomi@partner.samsung.com>
Change-Id: Ic30b67daa2e215524096ad1f4385c569e3d41b95