Closes-Bug: #2062401
Change-Id: I2f2bdbc9e1c6ad6da4ac7098ddd36143123c3062
(cherry picked from commit 904fae2a)

Followup on Ib69fc0017b3bfbc8da4dfd4301710fbf88be661a. This change
adds the ability to provide the NTP (time source) server for multiple
DHCP ranges in the Ironic Inspector DHCP server.

Change-Id: I4bbfef3a391b8582ae73cbe06138715b43584dec
Signed-off-by: Maksim Malchuk <maksim.malchuk@gmail.com>

This patch adds missing ironic_database_shard_id
which is used by proxysql-config role to generate
proxysql configuration.

Change-Id: I11ddfe73c79cc59a97d119091c8b9a3f5eda001d

When using dnsmasq as a DHCP server, unless you use the noping option
(and that is not recommended), the NET_RAW capabilty is required so
that dnsmasq can send ICMP packets. These are used to check an address
is not currently in use[1].  Docker enables this capability by
default. Podman runs containers with a minimal set of capabilities[3].

[1] https://lists.thekelleys.org.uk/pipermail/dnsmasq-discuss/2019q1/012840.html
[2] https://docs.docker.com/engine/containers/run/#runtime-privilege-and-linux-capabilities
[3] https://github.com/rhatdan/common/blob/f39f2a3f8c7680b9e456b9d235570e511807d6c6/docs/containers.conf.5.md?plain=1#L84-L101

Closes-Bug: #2055282
Change-Id: Ib3a1313df680d91c7f008063937ca7d37e82f690

Build upon changes in kolla which change strategy of installing projects
in containers when in dev mode. This fixes problems where when package
file manifest changes, the changes were not reflected in to
devmode-enabled container.

It changes the strategy of installing projects in dev mode in containers.
Instead of bind mounting the project's git repository to the venv
of the container, the repository is bind mounted to
/dev-mode/<project_name> from which the it is installed using pip
on every startup of the container using kolla_install_projects script.

Also updates docs to reflect the changes.

Depends-On: https://review.opendev.org/c/openstack/kolla/+/925712


Closes-Bug: #1814515
Singed-off-by: Roman Krček <roman.krcek@tietoevry.com>
Change-Id: If191cd0e3fcf362ee058549a1b6c244d109b6d9a

See [1].

[1]: https://opendev.org/openstack/ironic-inspector/commit/0b9b1756660b4ea63b44c0f01bbf3c1aa71c1f1a

Change-Id: I8866cdab396b805ec75bc4ccccdc5c1909e63bcf

For possible config options see docs
https://docs.openstack.org/keystonemiddleware/latest/middlewarearchitecture.html#memcache-protection



Closes-bug: #1850733
Signed-off-by: Roman Krček <roman.krcek@tietoevry.com>
Change-Id: I169e27899f7350f5eb8adb1f81a062c51e6cbdfc

Refactor that prepares kolla_container_facts
module for introducing more actions that will be moved
from kolla_container module and kolla_container_volume_facts.

This change is based on a discussion about adding a new action
to kolla_container module that retrieves all names of the running
containers. It was agreed that kolla-ansible should follow Ansible's
direction of splitting modules between action modules and facts
modules. Because of this, kolla_container_facts needs to be able
to handle different requests for data about containers or volumes.

Change-Id: Ieaec8f64922e4e5a2199db2d6983518b124cb4aa
Signed-off-by: Ivan Halomi <ivan.halomi@tietoevry.com>

The Kolla project supports building images with
user-defined prefixes. However, Kolla-ansible is unable
to use those images for installation.

This patch fixes that issue.

Closes-Bug: #2073541
Change-Id: Ia8140b289aa76fcd584e0e72686e3786215c5a99

Most roles are not leveraging the jinja filters available.
According to [1] filtering the list of services makes the execution
faster than skipping the tasks.

This patchset also includes some cosmetic changes to genconfig.
Individual services are now also using a jinja filter. This has
no impact on performance, just makes the tasks look cleaner.

Naming of some vars in genconfig was changed to "service" to make
the tasks more uniform as some were previously using
the service name and some were using "service".

Three metrics from the deployment were taken and those were
- overall deployment time [s]
- time spent on the specific role [s]
- CPU usage (measured with perf) [-]
Overall genconfig time went down on avg. from 209s to 195s
Time spent on the loadbalancer role went down on avg. from 27s to 23s
Time spent on the neutron role went down on avg from 102s to 95s
Time spent on the nova-cell role went down on avg. from 54s to 52s
Also the average CPUs utilized reported by perf went down
from 3.31 to 3.15.
For details of how this was measured see the comments in gerrit.

[1] - https://github.com/stackhpc/ansible-scaling/blob/master/doc/skip.md



Change-Id: Ib0f00aadb6c7022de6e8b455ac4b9b8cd6be5b1b
Signed-off-by: Roman Krček <roman.krcek@tietoevry.com>

This change automates the prometheus blackbox monitoring configuration
for common endpoints. Custom endpoints can be added to
prometheus_blackbox_exporter_endpoints_custom.

Change-Id: Id6f51a2bebee3ab63b84ca7032aad17c2933838c

Related-Bug: #2062401

Change-Id: I2c058e0fe56b32fce36e1c9f637c12fe5950ff21

inspector is not running as a WSGI

Related-Bug: #2054705
Change-Id: I20dbaef29b2ef2d6ceffc21c156c6fa4b5e8d205

This reverts commit d77372e8.

Reason for revert: service role support has been fixed in Ironic [1]
and added to Kolla-Ansible.

[1] https://review.opendev.org/c/openstack/ironic/+/907148

Closes-Bug: #2051837

Change-Id: I49664e3a353f54e0d51f454c552a78846ba64101

Add the service role to ironic service users. Ironic recently enforced
new policy validation as part of the RBAC efforts. [1][2]
Service user support was also added to Ironic. [3]
Admin role needs to stay as not all services added service role support. [4][5]

[1] https://review.opendev.org/c/openstack/ironic/+/902009
[2] https://opendev.org/openstack/governance/src/commit/e2a47de10a689a78c31765fd1b020f17c0d3109c/goals/selected/consistent-and-secure-rbac.rst#phase-2
[3] https://review.opendev.org/c/openstack/ironic/+/907148
[4] https://review.opendev.org/q/topic:bp%252Fpolicy-service-role-default
[5] https://review.opendev.org/q/topic:%22New-Location-Apis%22

Related-Bug: #2051837
Change-Id: I048402c2247188cf57f35437f557f84ac25d4ff2

Ironic recently started to enforce new policies and scope [1].
And Ironic is one of the sole openstack project which need
system scope for some admin related api calls [2].
However Ironic also started to allow project-scope behaviour
for service role with setting
``rbac_service_role_elevated_access``[3] [4]. This change enables
this setting to get similar behaviour of service role as other
openstack projects.

[1] https://review.opendev.org/c/openstack/ironic/+/902009
[2] https://opendev.org/openstack/governance/src/commit/e2a47de10a689a78c31765fd1b020f17c0d3109c/goals/selected/consistent-and-secure-rbac.rst?display=source#L261
[3] https://review.opendev.org/c/openstack/ironic/+/907148
[4] https://opendev.org/openstack/ironic/src/commit/8ec56066223301230ac0ed0f0c471a10d366b474/releasenotes/notes/service-project-service-role-fix-e4d1a8c23856926a.yaml

Related-Bug: #2051837

Change-Id: If8d7cf1663145d0398a2e936486e2b316d4df5e0

Ironic started enforcing new RBAC policies [1]. Kolla/Kayobe
CI jobs are failing, as K-A doesn't have service role support.
Moreover Ironic RBAC is not yet stable enough [2].
Disable enforcing new policies until fix merges and Kolla
Ansible service role support is added.

[1] https://review.opendev.org/c/openstack/ironic/+/902009
[2] https://review.opendev.org/c/openstack/ironic/+/907148

Related-Bug: #2051837

Change-Id: I424cff6ac96dfe0dd5dc58afca2b785f494c9f02

These were missed in I081aa1345603fa27c390e4e09231a5ff226bcb39

Change-Id: I2884bca3c06ff98004e318757a20b60c12375924

This implements a global toggle `om_enable_rabbitmq_quorum_queues`
to enable quorum queues for each service in RabbitMQ, similar to
what was done for HA[0].

Quorum Queues are enabled by default.

Quorum queues are more reliable, safer, simpler and faster than
replicated mirrored classic queues[1].

Mirrored classic queues are deprecated and scheduled for removal
in RabbitMQ 4.0[2].

Notice, that we do not need a new policy in the RabbitMQ definitions
template, because their usage is enabled on the client side and can't
be set using a policy[3].

Notice also, that quorum queues are not yet enabled in oslo.messaging
for the usage of reply_ and fanout_ queues (transient queues).
This will change once[4] is merged.

[0]: https://review.opendev.org/c/openstack/kolla-ansible/+/867771
[1]: https://www.rabbitmq.com/quorum-queues.html
[2]: https://blog.rabbitmq.com/posts/2021/08/4.0-deprecation-announcements/
[3]: https://www.rabbitmq.com/quorum-queues.html#declaring
[4]: https://review.opendev.org/c/openstack/oslo.messaging/+/888479



Signed-off-by: Sven Kieske <kieske@osism.tech>
Change-Id: I6c033d460a5c9b93c346e9e47e93b159d3c27830

* Updates etcd to v3.4
* Updated the config to use v3.4's logging mechanism
* Deprecated etcd CA parameters aren't used, so we are not affected
  by their removal.
* Note that we are not currently guarding against skip-version updates for
  etcd.

Notable non-voting jobs exercising some of this:
* kolla-ansible-ubuntu-upgrade-cephadm (cinder->tooz->etcd3gw->etcd)
* kolla-ansible-ubuntu-zun (see
  https://review.opendev.org/c/openstack/openstack-ansible/+/883194 )

Depends-On: https://review.opendev.org/c/openstack/kolla/+/890464
Change-Id: I086e7bbc7db64421445731a533265e7056fbdb43

Changes name of ansible module kolla_docker to
kolla_container.

Change-Id: I13c676ed0378aa721a21a1300f6054658ad12bc7
Signed-off-by: Martin Hiner <m.hiner@partner.samsung.com>

docker_restart_policy: no causes systemd units to not get created
and we use it in CI to disable restarts on services.

Introducing oneshot policy to not create systemd unit for oneshot
containers (those that are running bootstrap tasks, like db
bootstrap and don't need a systemd unit), but still create systemd
units for long lived containers but with Restart=No.

Change-Id: I9e0d656f19143ec2fcad7d6d345b2c9387551604

This change adds basic deployment based on Podman
container manager as an alternative to Docker.

Signed-off-by: Ivan Halomi <i.halomi@partner.samsung.com>
Signed-off-by: Martin Hiner <m.hiner@partner.samsung.com>
Signed-off-by: Petr Tuma <p.tuma@partner.samsung.com>
Change-Id: I2b52964906ba8b19b8b1098717b9423ab954fa3d
Depends-On: Ie4b4c1cf8fe6e7ce41eaa703b423dedcb41e3afc

Change-Id: Iafc4f184da727e40fb4952d96aa2e0be8feaca75

This commit adds the ironic-prometheus-exporter, following the
conventions used by the previously integrated exporters. '[The] Ironic
Prometheus Exporter is a Tool to expose hardware sensor data in the
Prometheus format through an HTTP endpoint.'[0]

Prometheus has been enabled in CI jobs to ensure test coverage.

[0] https://opendev.org/openstack/ironic-prometheus-exporter

Depends-On: https://review.opendev.org/c/openstack/kolla/+/874415

Change-Id: I6d421effd833d2e0524dd0b81736445c9a730ea9

With the parameter ironic_agent_files_directory it is possible to provide
the directory for the ironic-agent.kernel and ironic-agent.initramfs
files. By default the parameter is set to the value of node_custom_config.
This corresponds to the existing behaviour.

Change-Id: I53bb0eddc5380713a967356c85897d8df8ce505f

Use case: exposing single external https frontend and
load balancing services using FQDNs.

Support different ports for internal and external endpoints.

Introduced kolla_url filter to normalize urls like:
- https://magnum.external:443/v1
- http://magnum.external:80/v1



Change-Id: I9fb03fe1cebce5c7198d523e015280c69f139cd0
Co-Authored-By: Jakub Darmach <jakub@stackhpc.com>

ironic tftp service binds on 0.0.0.0. This may be
an issue in some setup. This patch propose a better
default, such as using the same listen address as
the dnsmasq service

Closes-Bug: #2024664

Change-Id: I0401bfc03cd31d72c5a2ae0a111889d5c29a8aa2

Change-Id: I1649a389bdc3977b936402c3ce3e55056d74ba08

deployments

This allows services to work with etcd when coordination is enabled
for TLS internal deployments. Without this fix, we fail to connect to
etcd with the coordination backend and the service itself crashes.

Change-Id: I0c1d6b87e663e48c15a846a2774b0a4531a3ca68

Hardcoding the first etcd host creates a single point of failure.

Change-Id: I0f83030fcd84ddcdc4bf2226e76605c7cab84cbb

As per the RBAC new direction in Zed cycle, we have dropped the
system scope from API policies and all the policies are hardcoded
to project scoped so that any user accessing APIs using system scope
will get 403 error. It is dropped from all the OpenStack services
except for the Ironic service which will have system scope and to
support ironic only deployment, we are keeping system as well as project
scope in Keystone.

Complete discussion and direction can be found in the below gerrit
change and TC goal direction:

- https://review.opendev.org/c/openstack/governance/+/847418
- https://governance.openstack.org/tc/goals/selected/consistent-and-secure-rbac.html#the-issues-we-are-facing-with-scope-concept

As phase-2 of RBAC goal, services will start enabling the new
defaults and project scope by default. For example: Nova did in
- https://review.opendev.org/c/openstack/nova/+/866218

Kolla who start accessing the services using system scope token
- https://review.opendev.org/c/openstack/kolla-ansible/+/692179

This commit partially revert the above change except keeping
system scope usage for Keystone and Ironic. Rest all services are changed
to use the project scope token.

And enable the scope and new defaults for Nova which was disabled
by https://review.opendev.org/c/openstack/kolla-ansible/+/870804

Change-Id: I0adbe0a6c39e11d7c9542569085fc5d580f26c9d

A combination of durable queues and classic queue mirroring can be used
to provide high availability of RabbitMQ. However, these options should
only be used together, otherwise the system will become unstable. Using
the flag ``om_enable_rabbitmq_high_availability`` will either enable
both options at once, or neither of them.

There are some queues that should not be mirrored:
* ``reply`` queues (these have a single consumer and TTL policy)
* ``fanout`` queues (these have a TTL policy)
* ``amq`` queues (these are auto-delete queues, with a single consumer)
An exclusionary pattern is used in the classic mirroring policy. This
pattern is ``^(?!(amq\\.)|(.*_fanout_)|(reply_)).*``

Change-Id: I51c8023b260eb40b2eaa91bd276b46890c215c25

When running in check mode, some prechecks previously failed because
they use the command module which is silently not run in check mode.
Other prechecks were not running correctly in check mode due to e.g.
looking for a string in empty command output or not querying which
containers are running.

This change fixes these issues.

Closes-Bug: #2002657
Change-Id: I5219cb42c48d5444943a2d48106dc338aa08fa7c

The ``[oslo_messaging_rabbit] heartbeat_in_pthread`` config option
is set to ``true`` for wsgi applications to allow the RabbitMQ
heartbeats to function. For non-wsgi applications it is set to ``false``
as it may otherwise break the service [1].

[1] https://docs.openstack.org/releasenotes/oslo.messaging/zed.html#upgrade-notes

Change-Id: Id89bd6158aff42d59040674308a8672c358ccb3c

Regularly, we experience issues in Kolla Ansible deployments because we
use wrong options in OpenStack configuration files. This is because
OpenStack services ignore unknown options. We also need to keep on top
of deprecated options that may be removed in the future. Integrating
oslo-config-validator into Kolla Ansible will greatly help.

Adds a shared role to run oslo-config-validator on each service. Takes
into account that services have multiple containers, and these may also
use multiple config files. Service roles are extended to use this shared
role. Executed with the new command ``kolla-ansible validate-config``.

Change-Id: Ic10b410fc115646d96d2ce39d9618e7c46cb3fbc

Second part of patchset:
https://review.opendev.org/c/openstack/kolla-ansible/+/799229/


in which was suggested to split patch into smaller ones.

THis change adds container_engine to module parameters
so when we introduce podman, kolla_toolbox can be used
for both engines.

Signed-off-by: Ivan Halomi <i.halomi@partner.samsung.com>
Co-authored-by: Martin Hiner <m.hiner@partner.samsung.com>
Change-Id: Ic2093aa9341a0cb36df8f340cf290d62437504ad

Second part of patchset:
https://review.opendev.org/c/openstack/kolla-ansible/+/799229/


in which was suggested to split patch into smaller ones.

This change adds container_engine variable to kolla_container_facts
module, this prepares module to be used with docker and podman as well
without further changes in roles.

Signed-off-by: Ivan Halomi <i.halomi@partner.samsung.com>
Co-authored-by: Martin Hiner <m.hiner@partner.samsung.com>
Change-Id: I9e8fa30646844ab4a288555f3aafdda345b3a118

The correct option to use is valid_interfaces [1], not os_endpoint_type.
The os_endpoint_type option was removed in Train.

[1] https://docs.openstack.org/ironic-inspector/wallaby/configuration/sample-config.html

Change-Id: I3906d7b9a2bebfe5c323cba5f80add3e932468c8
Closes-Bug: #1995246
Related-Bug: #1990675

First part of patchset:
 https://review.opendev.org/c/openstack/kolla-ansible/+/799229/


in which was suggested to split patch into smaller ones.

This implements kolla_container_engine variable
in command calls of docker,so later on it can be
also used for podman without further change.

Signed-off-by: Ivan Halomi <i.halomi@partner.samsung.com>
Change-Id: Ic30b67daa2e215524096ad1f4385c569e3d41b95