By default, Ansible injects a variable for every fact, prefixed with
ansible_. This can result in a large number of variables for each host,
which at scale can incur a performance penalty. Ansible provides a
configuration option [0] that can be set to False to prevent this
injection of facts. In this case, facts should be referenced via
ansible_facts.<fact>.

This change updates all references to Ansible facts within Kolla Ansible
from using individual fact variables to using the items in the
ansible_facts dictionary. This allows users to disable fact variable
injection in their Ansible configuration, which may provide some
performance improvement.

This change disables fact variable injection in the ansible
configuration used in CI, to catch any attempts to use the injected
variables.

[0] https://docs.ansible.com/ansible/latest/reference_appendices/config.html#inject-facts-as-vars

Change-Id: I7e9d5c9b8b9164d4aee3abb4e37c8f28d98ff5d1
Partially-Implements: blueprint performance-improvements

This reverts commit c6259158.

Reason for revert: cAdvisor fails with:

invalid value "percpu,referenced_memory,cpu_topology,resctrl,udp,advtcp,sched,hugetlb,memory_numa,tcp,process" for flag -disable_metrics: unsupported metric "referenced_memory" specified in disable_metrics

Change-Id: I1a0eea5c20f95f38c707401b56b7d2454484377d

Adds support for passing extra runtime options to cAdvisor.
By default new options disable exporting rarely useful metrics
and labels by cAdvisor. This helps reducing the load on Prometheus
and cAdvisor itself.

Change-Id: Id0144e8fa518e3236cb94ba2e3961fb455d36443

Reference:

https://github.com/prometheus/alertmanager#turn-off-high-availability

Closes-Bug: #1926463
Change-Id: I60e1dedeac25fa8fe9538a3a8e582bd8cc9324d7

The rabbitmq_prometheus plugin is available in RabbitMQ 3.8.

https://www.rabbitmq.com/prometheus.html



Implements: blueprint rabbitmq-prometheus
Co-Authored-By: Mark Goddard <mark@stackhpc.com>
Change-Id: I4d69a93a6c70db8d40626042cdbe773747b238ae

Deprecates support for Prometheus v1.x.
In Xena support for it will be removed from Kolla Ansible.

Change-Id: I027b19621196c698e09f79af294ba1b5dbfc0516

It is now possible to deploy either 1.x or 2.x version of Prometheus.
The new 2.x version introduces breaking changes in terms of storage
format and command line options.

Change-Id: I80cc6f1947f3740ef04b29839bfa655b14fae146
Co-Authored-By: Radosław Piliszek <radoslaw.piliszek@gmail.com>

This reverts commit 9cae59be.

Reason for revert: This patch was found to introduce issues with fluentd customisation. The underlying issue is not currently fully understood, but could be a sign of other obscure issues.

Change-Id: Ia4859c23d85699621a3b734d6cedb70225576dfc
Closes-Bug: #1906288

Add scrape_timeout option in
prometheus_openstack_exporter job in order
to avoid timeout for large Openstack environment.

Change-Id: If96034e602bee3b3eea34a2656047355e1d17eec
Closes-Bug: #1903547

Main plays are action-redirect-stubs, ideal for import_tasks.

This avoids 'include' penalty and makes logs/ara look nicer.

Fixes haproxy and rabbitmq not to check the host group as well.

Change-Id: I46136fc40b815e341befff80b54a91ef431eabc0
Partially-Implements: blueprint performance-improvements

Config plays do not need to check containers. This avoids skipping
tasks during the genconfig action.

Ironic and Glance rolling upgrades are handled specially.

Swift and Bifrost do not use the handlers at all.

Partially-Implements: blueprint performance-improvements
Change-Id: I140bf71d62e8f0932c96270d1f08940a5ba4542a

Change-Id: I639145a709f1d3b9882bbdfb20a754646d1f5270

The Prometheus OpenStack exporter was needlessly configured to use the
prometheus Docker volume and change permissions of /data, which does
not exist in the container image.

This must have been copy-pasted from existing Prometheus code.

Change-Id: I96017c17e68ca7a00a2d5ac41f2f43ef87694514

Including tasks has a performance penalty when compared with importing
tasks. If the include has a condition associated with it, then the
overhead of the include may be lower than the overhead of skipping all
imported tasks. For unconditionally included tasks, switching to
import_tasks provides a clear benefit.

Benchmarking of include vs. import is available at [1].

This change switches from include_tasks to import_tasks where there is
no condition applied to the include.

[1] https://github.com/stackhpc/ansible-scaling/blob/master/doc/include-and-import.md#task-include-and-import

Partially-Implements: blueprint performance-improvements

Change-Id: Ia45af4a198e422773d9f009c7f7b2e32ce9e3b97

The goal for this push request is to normalize the construction and use
 of internal, external, and admin URLs. While extending Kolla-ansible
 to enable a more flexible method to manage external URLs, we noticed
 that the same URL was constructed multiple times in different parts
 of the code. This can make it difficult for people that want to work
 with these URLs and create inconsistencies in a large code base with
 time. Therefore, we are proposing here the use of
 "single Kolla-ansible variable" per endpoint URL, which facilitates
 for people that are interested in overriding/extending these URLs.

As an example, we extended Kolla-ansible to facilitate the "override"
of public (external) URLs with the following standard
"<component/serviceName>.<companyBaseUrl>".
Therefore, the "NAT/redirect" in the SSL termination system (HAproxy,
HTTPD or some other) is done via the service name, and not by the port.
This allows operators to easily and automatically create more friendly
 URL names. To develop this feature, we first applied this patch that
 we are sending now to the community. We did that to reduce the surface
  of changes in Kolla-ansible.

Another example is the integration of Kolla-ansible and Consul, which
we also implemented internally, and also requires URLs changes.
Therefore, this PR is essential to reduce code duplicity, and to
facility users/developers to work/customize the services URLs.

Change-Id: I73d483e01476e779a5155b2e18dd5ea25f514e93
Signed-off-by: Rafael Weingärtner <rafael@apache.org>

Previously we mounted /etc/timezone if the kolla_base_distro is debian
or ubuntu. This would fail prechecks if debian or ubuntu images were
deployed on CentOS. While this is not a supported combination, for
correctness we should fix the condition to reference the host OS rather
than the container OS, since that is where the /etc/timezone file is
located.

Change-Id: Ifc252ae793e6974356fcdca810b373f362d24ba5
Closes-Bug: #1882553

Including tasks has a performance penalty when compared with importing
tasks. If the include has a condition associated with it, then the
overhead of the include may be lower than the overhead of skipping all
imported tasks. In the case of the check-containers.yml include, the
included file only has a single task, so the overhead of skipping this
task will not be greater than the overhead of the task import. It
therefore makes sense to switch to use import_tasks there.

Partially-Implements: blueprint performance-improvements

Change-Id: I65d911670649960708b9f6a4c110d1a7df1ad8f7

This change disables services in the Prometheus openstack-exporter
if they are not enabled in the deployment. Such behaviour allows
to avoid warnings and errors in the log files and keep the
log file contents clean and informative.

Change-Id: I4dcac976620a5f451e3d273183199aefe400994a

Change-Id: Ia134a518b63bb59cfad631cc488181f5245160e6

The common role was previously added as a dependency to all other roles.
It would set a fact after running on a host to avoid running twice. This
had the nice effect that deploying any service would automatically pull
in the common services for that host. When using tags, any services with
matching tags would also run the common role. This could be both
surprising and sometimes useful.

When using Ansible at large scale, there is a penalty associated with
executing a task against a large number of hosts, even if it is skipped.
The common role introduces some overhead, just in determining that it
has already run.

This change extracts the common role into a separate play, and removes
the dependency on it from all other roles. New groups have been added
for cron, fluentd, and kolla-toolbox, similar to other services. This
changes the behaviour in the following ways:

* The common role is now run for all hosts at the beginning, rather than
  prior to their first enabled service
* Hosts must be in the necessary group for each of the common services
  in order to have that service deployed. This is mostly to avoid
  deploying on localhost or the deployment host
* If tags are specified for another service e.g. nova, the common role
  will *not* automatically run for matching hosts. The common tag must
  be specified explicitly

The last of these is probably the largest behaviour change. While it
would be possible to determine which hosts should automatically run the
common role, it would be quite complex, and would introduce some
overhead that would probably negate the benefit of splitting out the
common role.

Partially-Implements: blueprint performance-improvements

Change-Id: I6a4676bf6efeebc61383ec7a406db07c7a868b2a

The flag -es.uri is no longer accepted - it should be --es.uri.
Similarly with -web.listen-address. The following error is seen:

    elasticsearch_exporter: error: unknown short flag '-e', try --help

This change switches to double dashed long options.

Change-Id: I039f4cad970352146462450742056f5990a81b06
Closes-Bug: #1880242

related to newly introduced merge mechanism.
1) Per-host overrides cannot be run_once.
2) Since merge_yaml is silent about missing files, it ignored
   the fact that no proper file was given due to wrong variable
   being referenced (see the closed bug).

Change-Id: I6db4af4c6e3364838bdae510f300038b0c1560b0
Closes-Bug: #1882460

you might refer to:
https://github.com/gophercloud/utils/blob/b0167b94122ca14ce50258a760b4e9b22788f0d7/openstack/clientconfig/results.go#L41

Change-Id: Ia326360c412aad9ca4d1735cc6486aa2fce22c1a
Closes-Bug: #1850812

* Reworked tox pep8 into linters job, that runs:
  - pep8
  - bandit
  - bashate
  - doc8
  - yamllint
  - ansible-lint (validate-all-files.py + ansible-lint)

* Skip E701 - missing galaxy_info in meta and E602 see [1].
* Skip E301 and E503 - followup later in a separate change
* Added ansible-role-jobs to zuul.d/project.yaml which will run
  openstack-tox-linters job in check queue
* Fixed remaining style issue
* Made tox and docs reference the new env for linters
* Dropped pype environment (not supported)

[1]: https://github.com/ansible/ansible-lint/issues/457

Change-Id: I494b4b151804aac8173120e6c6e42bc2fdb00234

This provides a generic mechanism to include extra files
that you can reference in prometheus.yml, for example:

scrape_targets:
  - job_name: ipmi
    params:
      module: default
    scrape_interval: 1m
    scrape_timeout: 30s
    metrics_path: /ipmi
    scheme: http
    file_sd_configs:
    - files:
      - /etc/prometheus/extras/file_sd/ipmi-exporter-targets.yml
      refresh_interval: 5m

Change-Id: Ie2f085204b71725b901a179ee51541f1f383c6fa
Related: blueprint custom-prometheus-targets

This provides a mechanism to scrape targets defined outside of kolla-ansible.

Depends-On: https://review.opendev.org/#/c/685671/
Change-Id: I0950341b147bb374b4128f09f807ef5a756f5dfa
Related: blueprint custom-prometheus-targets

Refactor service configuration to use the copy certificates task. This
reduces code duplication and simplifies implementing encrypting backend
HAProxy traffic for individual services.

Change-Id: I0474324b60a5f792ef5210ab336639edf7a8cd9e

Some services look for /etc/timezone on Debian/Ubuntu, so we should
introduce it to the containers.

In addition, added prechecks for /etc/localtime and /etc/timezone.

Closes-Bug: #1821592
Change-Id: I9fef14643d1bcc7eee9547eb87fa1fb436d8a6b3

This is useful to people who manage their Prometheus Server
externally to Kolla Ansible, or want to use the exporters with
another framework such as Monasca.

Change-Id: Ie3f61e2e186c8e77e21a7b53d2bd7d2a27eee18e

When change the cert file in /etc/kolla/certificate/.
The certificate in the container has not changed.
So I think can use kolla-ansible deploy when certificate is
changed. restart <container>

Partially-Implements: blueprint custom-cacerts

Change-Id: Iaac6f37e85ffdc0352e8062ae5049cc9a6b3db26
Signed-off-by: yj.bai <bai.yongjun@99cloud.net>

Both include_role and import_role expect role's name to be given
via "name" param instead of "role".
This worked but caused errors with ansible-lint.
See: https://review.opendev.org/694779

Change-Id: I388d4ae27111e430d38df1abcb6c6127d90a06e0

We assume that all groups are present in the inventory, and quite obtuse
errors can result if any are not.

This change adds a precheck that checks for the presence of all expected
groups in the inventory for each service. It also introduces a common
service-precheck role that we can use for other common prechecks.

Change-Id: Ia0af1e7df4fff7f07cd6530e5b017db8fba530b3
Partially-Implements: blueprint improve-prechecks

Service REST API urls should be constructed using the
{{ internal_protocol }} and {{ external_protocol }} configuration
parameters.

Change-Id: Id1e8098cf59f66aa35b371149fdb4b517fa4c908
Closes-Bug: 1862817

Service configuration urls should be constructed using
kolla_internal_fqdn instead of kolla_internal_vip_address. Otherwise SSL
validation will fail when certificates are issued using domain names.

Change-Id: I21689e22870c2f6206e37c60a3c33e19140f77ff
Closes-Bug: 1862419

Change-Id: I4f553bd0888e200ddf744604c5029e67a95ee2cd
Closes-bug: #1863041

Kolla-Ansible Ceph deployment mechanism has been deprecated in Train [1].

This change removes the Ansible code and associated CI jobs.

[1]: https://review.opendev.org/669214

Change-Id: Ie2167f02ad2f525d3b0f553e2c047516acf55bc2

When kolla_copy_ca_into_containers is set to "yes", the Certificate
Authority in /etc/kolla/certificates will be copied into service
containers to enable trust for that CA. This is especially useful when
the CA is self signed, and would not be trusted by default.

Partially-Implements: blueprint custom-cacerts

Change-Id: I4368f8994147580460ebe7533850cf63a419d0b4

Co-Authored-By: Marcin Juszkiewicz <marcin.juszkiewicz@linaro.org>

Change-Id: I65d9604d8522f0a60fbfeea718a63866410768b6

For the CentOS 7 to 8 transition, we will have a period where both
CentOS 7 and 8 images are available. We differentiate these images via a
tag - the CentOS 8 images will have a tag of train-centos8 (or
master-centos8 temporarily).

To achieve this, and maintain backwards compatibility for the
openstack_release variable, we introduce a new 'openstack_tag' variable.
This variable is based on openstack_release, but has a suffix of
'openstack_tag_suffix', which is empty except on CentOS 8 where it has a
value of '-centos8'.

Change-Id: I12ce4661afb3c255136cdc1aabe7cbd25560d625
Partially-Implements: blueprint centos-rhel-8

In a deployment where Prometheus is enabled and
Alertmanager is disabled the task "Copying over
prometheus config file" in
'ansible/roles/prometheus/tasks/config.yml' will
fail to template the Prometheus configuration file
'ansible/roles/prometheus/templates/prometheus.yml.j2'
as the variable 'prometheus_alert_rules' does not
contain the key 'files'. This commit fixes this bug.

Change-Id: Idbe1e52dd3693a6f168d475f9230a253dae64480
Closes-Bug: #1854540