Add global_request_id according to oslo.log defaults [1],
also support lines like:

2024-12-23 20:48:29.905 2 ERROR oslo_service.periodic_task ['Traceback (most recent call last):\\n'

[1]: https://docs.openstack.org/oslo.log/latest/configuration/#DEFAULT.logging_context_format_string

Closes-Bug: #2044370

Change-Id: I80c3e20de0b7503de6331b55879e85892323b3d6
(cherry picked from commit 5a3fb2b4)

When merging change [1], fluentd_enable_watch_timer was
unintentionally missed in Let’s Encrypt, as change [2] had been
merged earlier.

[1] https://review.opendev.org/c/openstack/kolla-ansible/+/785309
[2] https://review.opendev.org/c/openstack/kolla-ansible/+/899895

Change-Id: I7c72faecbdb66c7fd196acd3e7b2351851983490

By default, the watch timer in Fluentd is set to True.
To save CPU and I/O consumption this can be set to False, which
kolla-ansible has been hardcoding so far.

When the watch timer is disabled, in_tail relies entirely
on inotify. In certain constellations, this may not work
reliably. In these cases, the watch timer needs to be activated, so this
change adds a variable to make the setting configurable.

Change-Id: Ic8ce6fbc3ed8f31d5d090e114b35703532679729

Also stop using the old config name (td-agent.conf)

Change-Id: Ied2736b891cd8c6dfcc509a8fd6b1fc8bfe21cb4

It was deprecated in Antelope cycle.

Change-Id: I499e69ec6db63e4067e49376e2a1f3e01e48fe62

It's inactive and hasn't produced a 2024.1 release [1].
In addition to that, there's a CVE that hasn't really been patched [2].

Also drop outward_rabbitmq that was used only with Murano.

[1]: https://governance.openstack.org/tc/reference/emerging-technology-and-inactive-projects.html#current-inactive-projects
[2]: https://lists.openstack.org/archives/list/openstack-announce@lists.openstack.org/thread/4FYM6GSIM5WZSJQIG4TT5Q3UBKQIHLWX/

Change-Id: I691205730b0e10a42ce61f3340cc39ee51bd1010

It's inactive and hasn't produced a 2024.1 release [1].

[1]: https://governance.openstack.org/tc/reference/emerging-technology-and-inactive-projects.html#current-inactive-projects

Change-Id: I217b3633f07e5b2c657e20b19aaa4fbb46535a97

It's inactive and hasn't produced a 2024.1 release [1].

[1]: https://governance.openstack.org/tc/reference/emerging-technology-and-inactive-projects.html#current-inactive-projects

Change-Id: I888963751b6e1ed080588297c2889e700431516c

It's inactive and hasn't produced a 2024.1 release [1].
There are some efforts to restore Freezer, but let's remove it for now.

[1]: https://governance.openstack.org/tc/reference/emerging-technology-and-inactive-projects.html#current-inactive-projects

Change-Id: Ie42012af9e5c64bca23a6e6826bfc4651fd194bd

Change-Id: Ia6db7d6a41ddbda8fcbf563dc55a0c65ef8db9be

Depends-On: https://review.opendev.org/c/openstack/kolla/+/901508
Change-Id: I8c7d3de95d0f1f8e57a993b8c3417d90459e19be

Adds configurations and changes tasks to enable the
systemd plugin. Additionaly, the plugin is set to
read logs from the /var/log/journal directory

Implements: enable-fluent-plugin-systemd
Signed-off-by: Juan Pablo Suazo <jsuazo@whitestack.com>
Change-Id: Ic714a341befa5f906d9c0f78fa86f4c934df87cd

Depends-On: https://review.opendev.org/c/openstack/kolla/+/899773
Change-Id: I016c7a54ee2c43b6bbad0cb3eb3fe216f3d66238

Apache grok pattern updated to match the size of response in bytes,
the time taken to serve the request and the user-agent that made
the request.

Change-Id: I8ce6e6e10bbf55ef8aa21f983f4a6c4b6dc24402

Related-Bug: #2015184

Closes-Bug: #2015184

Change-Id: I6ea0949d58e38e2cda41c9702001e7bd500f23f1

The default value of logging_context_format_string and
logging_user_identity_format has changed [1][2][3].
This is to correct fluentd to use the same format.

[1]https://opendev.org/openstack/oslo.log/commit/ebdee7f39920ad5b4268ee296952432b0d41a375
[2]https://opendev.org/openstack/oslo.log/commit/7ed065f38df0595881d07eb91cec9e5c07cb38be

[3]https://review.opendev.org/c/openstack/oslo.log/+/838185

Closes-Bug: #1872220

Change-Id: Ide0079ef4d28e3a3fa60288a74b936531c52ce60

This change also adds support for Trove backend TLS.

Depends-On: https://review.opendev.org/c/openstack/kolla/+/854744
Change-Id: I2acf7820b24b112b57b0c00a01f5c4b8cb85ce25

Support to deploy skyline by kolla-ansible.

Implements: blueprint skyline
Depends-On: https://review.opendev.org/c/openstack/kolla/+/826948

Change-Id: Ice5621491a432ba32138abd6f62d1f815cc219e0

Their cleanup has been added to monasca cleanup command.

Change-Id: I19a846e2683ae70b33ca64d2aba7ac71eb724588

This change replaces ElasticSearch with OpenSearch, and Kibana
with OpenSearch Dashboards. It migrates the data from ElasticSearch
to OpenSearch upon upgrade.

No TLS support is in this patch (will be a followup).

A replacement for ElasticSearch Curator will be added as a followup.

Depends-On: https://review.opendev.org/c/openstack/kolla/+/830373



Co-authored-by: Doug Szumski <doug@stackhpc.com>
Co-authored-by: Kyle Dean <kyle@stackhpc.com>
Change-Id: Iab10ce7ea5d5f21a40b1f99b28e3290b7e9ce895

A few minor fixes were noted in this review [1], and they
are addressed here.

TrivialFix

[1]: https://review.opendev.org/c/openstack/kolla-ansible/+/861392/

Change-Id: If30d9c2b48615dfb54edcb8d782c4c24b968ac4b

Kolla Ansible is switching to OpenSearch and is dropping support for
deploying ElasticSearch. This is because the final OSS release of
ElasticSearch has exceeded its end of life.

Monasca is affected because it uses both Logstash and ElasticSearch.
Whilst it may continue to work with OpenSearch, Logstash remains an
issue.

In the absence of any renewed interest in the project, we remove
support for deploying it. This helps to reduce the complexity
of log processing configuration in Kolla Ansible, freeing up
development time.

Change-Id: I6fc7842bcda18e417a3fd21c11e28979a470f1cf

Fluentd has a default timeout of 5s for flushing data to ElasticSearch.
If there is a significant backlog of unsent log messages, this timeout
can be exceeded, resulting in Fluentd failing to make further progress.

Raise the default timeout to 60s.

This patch adopts the configuration parameters previously proposed by
Krzysztof Klimonda.

Closes-Bug: #1983031
Closes-Bug: #1896611
Change-Id: I1aaab654a5a0752fccef2cfb8cc0bde4a0ee2562

In fluentd v1.0, "format" became a deprecated parameter,
replaced by "<parse>"

Change-Id: I0825e81fcd90fbc7f64c8df3ea9ae199ad79294a

Adds a new configuration file that provides fluentd with an appropiate regex to match with OpenvSwitch logs in both default files.

The regex is segmented with variable as to isolate the relevant parts of each log message.

Closes-Bug: #1965815
Signed-off-by: Juan Pablo Suazo <jsuazo@whitestack.com>
Change-Id: Ife83c50c048d517a5c8a5dee588f8f7846fcee00

In Xena [1] we removed Monasca Grafana service, but some components were left
to support cleanup operations.

[1]: https://review.opendev.org/c/openstack/kolla-ansible/+/788228

Change-Id: Iccc7bc3628bb7cbab1ac28f41c7b7dc7695894c6

This project [1] can provide a one-stop solution to log collection,
cleaning, indexing, analysis, alarm, visualization, report generation
and other needs, which involves helping operator or maintainer to
quickly solve retrieve problems, grasp the operational health of the
platform, and improve the level of platform management.

[1] https://wiki.openstack.org/wiki/Venus

Change-Id: If3562bbed6181002b76831bab54f863041c5a885

Change-Id: I5b3ab3ab8153cda283dec772bf1393af0caf4137
Closes-Bug: 1919179

Co-Authored-By: Mark Goddard <mark@stackhpc.com>

Change-Id: I75ca59d981bcd2dd51faa296ab0b4223a891f5cb

Overrides default fluentd buffer config to stop log files from using
datestamped filenames, allowing logrotate to manage them.

Closes-Bug: #1940118
Change-Id: I40c4e209470d21e0a02fd447fb628acfdae9fa9d

After we remove the problematic, legacy "parser" plugin, the
config only needs the new, modern syntax.
This removes warnings that were logged.

Depends-On: https://review.opendev.org/c/openstack/kolla/+/823071
Change-Id: I33aa06de6ce88288769a8291ebd59e78e07cdbaf

To satisfy the needs of the modern parser plugin.

Needed-By: https://review.opendev.org/c/openstack/kolla/+/823071


Co-Authored-By: Dr. Jens Harbott <harbott@osism.tech>
Change-Id: I2b748d0544c14bebefe8c62aa5aafaaa5371ce53

We stopped using this file in Queens
(https://review.opendev.org/494635), but the file was not removed at
that time.

Change-Id: Ibe5fb291e7c39965f5c4ff5ee4ea0bb1f8e6e9c2
Closes-Bug: #1840158

We no longer have to support multiple fluentd versions and rely
on the workaround.

Change-Id: Id4083a9f07317fc04467e52176850f3909703169

This change adds the dnsmasq.log for the ironic-dnsmasq container and
also enables more verbose logging when debug logging enabled.
This can be triggered globbaly via 'openstack_logging_debug' or per
service via 'ironic_logging_debug' or 'neutron_logging_debug'.

Change-Id: I0e6b089beb88827effbcc365625eb2df902f5470
Signed-off-by: Maksim Malchuk <maksim.malchuk@gmail.com>

Closes-Bug: #1945070
Change-Id: I1b2a82b57cb9884b6c3c3ad07f6449ae29042a3d

This patch adds support for integrating Prometheus with Fluentd.
This can be used to extract useful information about the status
of Fluentd, such as output buffer capacity and logging rate,
and also to extract metrics from logs via custom Fluentd
configuration. More information can be found here in [1].

[1] https://docs.fluentd.org/monitoring-fluentd/monitoring-prometheus

Change-Id: I233d6dd744848ef1f1589a462dbf272ed0f3aaae

By default, Ansible injects a variable for every fact, prefixed with
ansible_. This can result in a large number of variables for each host,
which at scale can incur a performance penalty. Ansible provides a
configuration option [0] that can be set to False to prevent this
injection of facts. In this case, facts should be referenced via
ansible_facts.<fact>.

This change updates all references to Ansible facts within Kolla Ansible
from using individual fact variables to using the items in the
ansible_facts dictionary. This allows users to disable fact variable
injection in their Ansible configuration, which may provide some
performance improvement.

This change disables fact variable injection in the ansible
configuration used in CI, to catch any attempts to use the injected
variables.

[0] https://docs.ansible.com/ansible/latest/reference_appendices/config.html#inject-facts-as-vars

Change-Id: I7e9d5c9b8b9164d4aee3abb4e37c8f28d98ff5d1
Partially-Implements: blueprint performance-improvements

the project is deprecated and in the process of being removed
from OpenStack upstream.

Change-Id: I9d5ebed293a5fb25f4cd7daa473df152440e8b50

We really want elasticsearch or monasca to catch all logs,
to providd the required centrailsed logging.

While these appears to make little material difference,
it should make it harder for logs to not get caught by
any of the outputs we have configured.

TrivialFix

Change-Id: I3bb74dcdc3cbe78cd1e1657f44e2a0af9d6508ef