Co-Authored-By: Mark Goddard <mark@stackhpc.com>

Change-Id: I75ca59d981bcd2dd51faa296ab0b4223a891f5cb

Overrides default fluentd buffer config to stop log files from using
datestamped filenames, allowing logrotate to manage them.

Closes-Bug: #1940118
Change-Id: I40c4e209470d21e0a02fd447fb628acfdae9fa9d

Closes-Bug: #1954723
Change-Id: I6ffc8caae1aef757d37c629fbc05bb129f80147b

Role vars have a higher precedence than role defaults. This allows to
import default vars from another role via vars_files without overriding
project_name (see related bug for details).

Change-Id: I3d919736e53d6f3e1a70d1267cf42c8d2c0ad221
Related-Bug: #1951785

After we remove the problematic, legacy "parser" plugin, the
config only needs the new, modern syntax.
This removes warnings that were logged.

Depends-On: https://review.opendev.org/c/openstack/kolla/+/823071
Change-Id: I33aa06de6ce88288769a8291ebd59e78e07cdbaf

To satisfy the needs of the modern parser plugin.

Needed-By: https://review.opendev.org/c/openstack/kolla/+/823071


Co-Authored-By: Dr. Jens Harbott <harbott@osism.tech>
Change-Id: I2b748d0544c14bebefe8c62aa5aafaaa5371ce53

We stopped using this file in Queens
(https://review.opendev.org/494635), but the file was not removed at
that time.

Change-Id: Ibe5fb291e7c39965f5c4ff5ee4ea0bb1f8e6e9c2
Closes-Bug: #1840158

We no longer have to support multiple fluentd versions and rely
on the workaround.

Change-Id: Id4083a9f07317fc04467e52176850f3909703169

Change-Id: I67995073b92d6ea9ccc208ae96e8a6045369b801

Change-Id: I5d6af3707c3de4466699f1a0e6ff2f2e2fe4e123

This patch adding way to change schedule
for logrotate.

Depends-On: https://review.opendev.org/c/openstack/kolla/+/813008
Change-Id: I033c51704bfb806f6515db26b8159fae12469ba5

This change adds the dnsmasq.log for the ironic-dnsmasq container and
also enables more verbose logging when debug logging enabled.
This can be triggered globbaly via 'openstack_logging_debug' or per
service via 'ironic_logging_debug' or 'neutron_logging_debug'.

Change-Id: I0e6b089beb88827effbcc365625eb2df902f5470
Signed-off-by: Maksim Malchuk <maksim.malchuk@gmail.com>

This is unused since we switched to a single config file for logrotate.

Change-Id: I79a5fd84b071b2a127c09ac41c3de9074289fca0

chrony is not supported in Xena cycle, remove it from kolla

Moved tasks from chrony role to chrony-cleanup.yml playbook to avoid a
vestigial chrony role.

Co-Authored-By: Mark Goddard <mark@stackhpc.com>

Change-Id: I5a730d55afb49d517c85aeb9208188c81e2c84cf

Closes-Bug: #1945070
Change-Id: I1b2a82b57cb9884b6c3c3ad07f6449ae29042a3d

Currently only operations done with default kolla_toolbox user are logged
to /var/log/kolla/ansible.log.

In order to fix logging, permissions to ansible.log must allow writing
for other users in kolla group - and then a separate patch will follow
to make custom ansible.cfg file usable by other toolbox users.

Partial-Bug: #1942846
Change-Id: I1be60ac7647b1a838e97f05f15ba5f0e39e8ae3c

It seems to have been mistakenly introduced by
de00bf49
"Simplify handler conditionals"

Change-Id: I65b6e322fa11a870f32099bbfd62150cbea4feb5

Certain overrides for rabbitmq may need to be set for `rabbitmqctl` in
kolla-toolbox aswell.
This commit allows to override `rabbitmq-env.conf` and `erl_inetrc` in
kolla-toolbox.

Change-Id: Idef6adcf9700f75a2db503444a8de093ee21a9c5

Closes-Bug: #1939883
Change-Id: Ica311acba445cccac1d20757ced6f15a064ebcaa

We get a nice optimisation by using a filtered loop instead
of task skipping per service with 'when'.

Partially-Implements: blueprint performance-improvements
Change-Id: I8f68100870ab90cb2d6b68a66a4c97df9ea4ff52

This patch adds support for integrating Prometheus with Fluentd.
This can be used to extract useful information about the status
of Fluentd, such as output buffer capacity and logging rate,
and also to extract metrics from logs via custom Fluentd
configuration. More information can be found here in [1].

[1] https://docs.fluentd.org/monitoring-fluentd/monitoring-prometheus

Change-Id: I233d6dd744848ef1f1589a462dbf272ed0f3aaae

Remove tempest role as planned

Change-Id: If3cf073e88c83f670c867a49afe48845f9e81008

By default, Ansible injects a variable for every fact, prefixed with
ansible_. This can result in a large number of variables for each host,
which at scale can incur a performance penalty. Ansible provides a
configuration option [0] that can be set to False to prevent this
injection of facts. In this case, facts should be referenced via
ansible_facts.<fact>.

This change updates all references to Ansible facts within Kolla Ansible
from using individual fact variables to using the items in the
ansible_facts dictionary. This allows users to disable fact variable
injection in their Ansible configuration, which may provide some
performance improvement.

This change disables fact variable injection in the ansible
configuration used in CI, to catch any attempts to use the injected
variables.

[0] https://docs.ansible.com/ansible/latest/reference_appendices/config.html#inject-facts-as-vars

Change-Id: I7e9d5c9b8b9164d4aee3abb4e37c8f28d98ff5d1
Partially-Implements: blueprint performance-improvements

Remove rally role as planned

Change-Id: Ic898efe42b21b01c45d4621af2cf90ecd7afc398

the project is deprecated and in the process of being removed
from OpenStack upstream.

Change-Id: I9d5ebed293a5fb25f4cd7daa473df152440e8b50

We really want elasticsearch or monasca to catch all logs,
to providd the required centrailsed logging.

While these appears to make little material difference,
it should make it harder for logs to not get caught by
any of the outputs we have configured.

TrivialFix

Change-Id: I3bb74dcdc3cbe78cd1e1657f44e2a0af9d6508ef

When using elasticsearch 7 with fluentd, you seem to get a lot
of warnings in the docker logs output that look like:

    [types removal] Specifying types in bulk requests is deprecated.

The docs suggest adding suppress_type_name to stop these warnings,
and that seems to work without affecting any functionality.

Further info here:
https://github.com/uken/fluent-plugin-elasticsearch/issues/785

Closes-Bug: #1930856
Change-Id: I45be67df3717f78d78bcdc7df69600ab8681922f

Currently the logs tagged with infra.mariadb.xinetd flow into
elasticsearch with no hostname or programname attach, thus making
navigating the logs very hard.

The quick fix is renaming the tag to infra.mariadb-xinetd, which is just
enought to ensure the logs are processed correctly with the existing
filters.

TrivialFix

Change-Id: Icd72206de7c1f701bdf35c8fb3b128ef2dbe29a8

Currently when elasticsearch log output is enabled there are lots
of warnings going into elasticsearch about type being deprecated
and needing to move to @type. This change stops those warnings.

TrivialFix

Change-Id: Ideac1925cb764ad0d7d8416f56d5e4a993c6d8b6

This change inform us that the file is 'Ansible managed'

TrivialFix

Change-Id: I99ebeb493ff9c3c7af0010ce1efea45c7f9a2559
Signed-off-by: Maksim Malchuk <maksim.malchuk@gmail.com>

When enable_elasticsearch is False, and kolla_internal_fqdn is set
and elasticsearch_address is the default, fluentd is still enabling
output to elasticsearch.

elasticsearch_address defaults to kolla_internal_fqdn, but the
check to enable logging to an "external" elasticsearch host
compares with kolla_internal_vip_address (fqdn != vip address).

This patch fixes the comparison to use the kolla_internal_fqdn.

Closes-Bug: #1927880
Change-Id: If23a6b1273c2639d1296becc9d222546d52f63ac

This is a follow up on the change with the following ID:

I337f42e174393f68b43e876ef075a74c887a5314

TrivialFix

Change-Id: Ibb67811d7b086ef9ef4c695ae589171af0c4d657

Adds HAcluster Ansible role. This role contains High Availability
clustering solution composed of Corosync, Pacemaker and Pacemaker Remote.

HAcluster is added as a helper role for Masakari which requires it for
its host monitoring, allowing to provide HA to instances on a failed
compute host.

Kolla hacluster images merged in [1].

[1] https://review.opendev.org/#/c/668765/



Change-Id: I91e5c1840ace8f567daf462c4eb3ec1f0c503823
Implements: blueprint ansible-pacemaker-support
Co-Authored-By: Radosław Piliszek <radoslaw.piliszek@gmail.com>
Co-Authored-By: Mark Goddard <mark@stackhpc.com>

Co-Authored-By: Bartosz Bezak <bartosz@stackhpc.com>

Depends-On: https://review.opendev.org/c/openstack/kolla/+/782906

Change-Id: I73cd776772a45c9ffcb045f4d0eff8f4bdc997f1

This change also adds support for Octavia backend TLS.

Closes-Bug: #1874228
Depends-On: https://review.opendev.org/c/openstack/kolla/+/779892
Change-Id: I5ff84aec4cdbc15f6a797391815243821dbdbd67

One use case for this is so that you can generate config in a CI job
without access to the container repository. It also removes the
dependency of having docker configured for config generation.

TrivialFix

Change-Id: I0d388851c8b953af0494e44ae569e7eb9e15c326

Change-Id: Ief84e093829677c97c8df9a08aefca43b1e51aac

This change allows a user to forward control plane logs
directly to Elasticsearch from Fluentd, rather than via
the Monasca Log API when Monasca is enabled. The Monasca
Log API can continue to handle tenant logs.

For many use cases this is simpler, reduces resource
consumption and helps to decouple control plane logging
services from tenant logging services.

It may not always be desired, so is optional and off by
default.

Change-Id: I195e8e4b73ca8f573737355908eb30a3ef13b0d6

There are a few issues fixed here:

- The Barbican API service doesn't set a log file, so all the Barbican API
  service logs go to loadwsgi.py.log by default.
- The logs in loadwsgi.py.log are not ingested properly by Fluentd.
- uWSGI logs go to barbican-api.log. This would normally be used as the log
  file for the Barbican API service logs.

This patch makes the following changes to address the above issues:

- All uWSGI logs (from the Emperor and Vassals) go to barbican_api_uwsgi_access.log
  Although these logs aren't strictly all access logs, this follows the existing
  pattern for WSGI logs.
- The Barbican API service logs are written to barbican-api.log instead of
  loadwsgi.py.log. This follows the pattern used by other OpenStack services.
- Fluentd is configured to parse the Barbican API service logs as it would with
  other OpenStack Python services.

Change-Id: I6d03fa8c81c52b6f061514a836bbd15bb6639aaf
Closes-Bug: #1891343

As announced on the openstack-discuss ML[1], Karbor is retiring
this cycle (Wallaby).

Needed-By: https://review.opendev.org/c/openstack/karbor/+/767032

[1] http://lists.openstack.org/pipermail/openstack-discuss/2020-November/018643.html

Change-Id: I222cf302e507f6a9de0347c79ec536aa7be22bb6