Skip to content
Snippets Groups Projects
Select Git revision
  • deployment default
1 result
Search by author
  • Any Author
  • KANAPA SYLVERE p1919734 KANAPA SYLVERE p1919734 p1919734
1 author
  1. Apr 08, 2021
  3. Mar 25, 2021
  5. Mar 04, 2021
    • Doug Szumski's avatar
      Support bypassing Monasca Log API for control plane logs · ca1a80ab
      Doug Szumski authored 
      This change allows a user to forward control plane logs
directly to Elasticsearch from Fluentd, rather than via
the Monasca Log API when Monasca is enabled. The Monasca
Log API can continue to handle tenant logs.

For many use cases this is simpler, reduces resource
consumption and helps to decouple control plane logging
services from tenant logging services.

It may not always be desired, so is optional and off by
default.

Change-Id: I195e8e4b73ca8f573737355908eb30a3ef13b0d6
      ca1a80ab
  7. Jan 27, 2021
    • Doug Szumski's avatar
      Fix Barbican API log config · 1ff4e58d
      Doug Szumski authored 
      There are a few issues fixed here:

- The Barbican API service doesn't set a log file, so all the Barbican API
  service logs go to loadwsgi.py.log by default.
- The logs in loadwsgi.py.log are not ingested properly by Fluentd.
- uWSGI logs go to barbican-api.log. This would normally be used as the log
  file for the Barbican API service logs.

This patch makes the following changes to address the above issues:

- All uWSGI logs (from the Emperor and Vassals) go to barbican_api_uwsgi_access.log
  Although these logs aren't strictly all access logs, this follows the existing
  pattern for WSGI logs.
- The Barbican API service logs are written to barbican-api.log instead of
  loadwsgi.py.log. This follows the pattern used by other OpenStack services.
- Fluentd is configured to parse the Barbican API service logs as it would with
  other OpenStack Python services.

Change-Id: I6d03fa8c81c52b6f061514a836bbd15bb6639aaf
Closes-Bug: #1891343
      1ff4e58d
  9. Dec 22, 2020
  11. Dec 16, 2020
  13. Oct 12, 2020
    • James Kirsch's avatar
      Add support for encrypting backend Neutron API Server · 93ad57f4
      James Kirsch authored 
      Add TLS support for backend Neutron API Server communication using
HAProxy to perform TLS termination. When used in conjunction with
enabling TLS for service API endpoints, network communication will be
encrypted end to end, from client through HAProxy to the Neutron
service.

Change-Id: Ib333a1f1bd12491df72a9e52d961161210e2d330
Partially-Implements: blueprint add-ssl-internal-network
      93ad57f4
  15. Sep 18, 2020
  17. Aug 19, 2020
    • Rafael Weingärtner's avatar
      Standardize use and construction of endpoint URLs · f425c067
      Rafael Weingärtner authored 
      
The goal for this push request is to normalize the construction and use
 of internal, external, and admin URLs. While extending Kolla-ansible
 to enable a more flexible method to manage external URLs, we noticed
 that the same URL was constructed multiple times in different parts
 of the code. This can make it difficult for people that want to work
 with these URLs and create inconsistencies in a large code base with
 time. Therefore, we are proposing here the use of
 "single Kolla-ansible variable" per endpoint URL, which facilitates
 for people that are interested in overriding/extending these URLs.

As an example, we extended Kolla-ansible to facilitate the "override"
of public (external) URLs with the following standard
"<component/serviceName>.<companyBaseUrl>".
Therefore, the "NAT/redirect" in the SSL termination system (HAproxy,
HTTPD or some other) is done via the service name, and not by the port.
This allows operators to easily and automatically create more friendly
 URL names. To develop this feature, we first applied this patch that
 we are sending now to the community. We did that to reduce the surface
  of changes in Kolla-ansible.

Another example is the integration of Kolla-ansible and Consul, which
we also implemented internally, and also requires URLs changes.
Therefore, this PR is essential to reduce code duplicity, and to
facility users/developers to work/customize the services URLs.

Change-Id: I73d483e01476e779a5155b2e18dd5ea25f514e93
Signed-off-by: default avatarRafael Weingärtner <rafael@apache.org>
      f425c067
  19. Aug 06, 2020
  21. Jul 28, 2020
    • Mark Goddard's avatar
      Performance: use a single config file for fluentd · 56a07702
      Mark Goddard authored 
      Currently we generate multiple fluentd configuration files for inputs,
filters, formatters and outputs.
These are then included from the main td-agent.conf configuration file.
With a large number of hosts, this can take a long time to template.

Benchmarking of templating is available at [1].

This change switches to a single fluentd configuration file, with the
include done locally. For the default template files included with Kolla
Ansible we use Jinja includes, but this does not work with templates in
a different directory. We therefore use the Ansible template lookup
plugin, which has a slightly higher overhead than a jinja include, but
far lower than generating multiple templates. This should drastically
improve the performance of this task.

[1] https://github.com/stackhpc/ansible-scaling/blob/master/doc/template.md

Partially-Implements: blueprint performance-improvements

Change-Id: Ia8623be0aa861fea3e54d2c9e1c971dfd8e3afa9
      56a07702
    • Mark Goddard's avatar
      Performance: use a single config file for logrotate · f329af7d
      Mark Goddard authored 
      Currently we generate a logrotate configuration file for each enabled
service. These are then included from a logrotate.d directory. With a
large number of hosts, this can take a long time to template.

Benchmarking of templating is available at [1].

This change switches to a single logrotate configuration file for all
services, with the include done locally using jinja. This should
drastically improve the performance of this task.

[1] https://github.com/stackhpc/ansible-scaling/blob/master/doc/template.md

Partially-Implements: blueprint performance-improvements

Change-Id: I39cfa70bef6560f615cad516c43aaef6a523b964
      f329af7d
  23. Jul 27, 2020
    • Michal Nasiadka's avatar
      fluentd: log to a file instead of stdout · 696533f2
      Michal Nasiadka authored 
      fluentd logs currently to stdout, which is known to produce big docker logs
in /var/lib/docker. This change makes fluentd to log to /var/log/kolla/fluentd.

Closes-Bug: #1888852
Change-Id: I8fe0e54cb764a26d26c6196cef68aadc6fd57b90
      696533f2
  25. Jul 21, 2020
    • Pierre Riteau's avatar
      Fix fluentd warnings caused by "type copy" · 5b057812
      Pierre Riteau authored 
      A "@type copy" statement is already present at the beginning of each
match element, so extra "type copy" are not needed. They are causing the
following warnings in fluentd logs:

[warn]: parameter 'type' in <match syslog.local0.**>
[warn]: parameter 'type' in <match syslog.local1.**>

This commit also harmonizes indentation of the Monasca config block.

Change-Id: I779c2b942d007acbdd43d999f2fc0cdc131d431f
Related-Bug: #1885873
      5b057812
  27. Jul 02, 2020
  29. Jul 01, 2020
  31. Jun 25, 2020
    • Mark Goddard's avatar
      Support CA certificate for fluentd & Elasticsearch · 31f3f848
      Mark Goddard authored 
      Currently there is no way to configure a CA certificate bundle file for
fluentd to Elasticsearch communication. This change adds a new variable,
'fluentd_elasticsearch_cacert' with a default value set to the value of
'openstack_cacert.

Closes-Bug: #1885109

Change-Id: I5bbf55a4dd4ccce9fa2635cee720139c088268e3
      31f3f848
  33. Jun 20, 2020
  35. Jun 19, 2020
  37. Jun 16, 2020
  39. May 23, 2020
    • Doug Szumski's avatar
      Switch to Monasca API for logs · b39a0f80
      Doug Szumski authored 
      The Monasca Log API has been removed and in this change we switch
to using the unified API. If dedicated log APIs are required then
this can be supported through configuration. Out of the box the
Monasca API is used for both logs and metrics which is envisaged to
work for most use cases.

In order to use the unified API for logs, we need to disable the
legacy Kafka client. We also rename the Monasca API config file
to remove a warning about using the old style name.

Depends-On: https://review.opendev.org/#/c/728638
Change-Id: I9b6bf5b6690f4b4b3445e7d15a40e45dd42d2e84
      b39a0f80
  41. May 20, 2020
  43. May 15, 2020
  45. Apr 30, 2020
  47. Apr 17, 2020
    • Q.hongtao's avatar
      Add vitrage-persistor service · d0b06f67
      Q.hongtao authored 
      Kolla Ansible was missing vitrage-persistor service
required by Vitrage for data storage.

Depends on fixing availability of Kolla image.

Change-Id: I8158ba66b8b624f6bcb89da9c990a30a68b7187b
Depends-On: Id5e143636f9a81e7294b775f3d8b9134bee58054
Closes-Bug: #1869319
      d0b06f67
  49. Apr 10, 2020
  51. Mar 18, 2020
  53. Mar 17, 2020
    • Doug Szumski's avatar
      Make Fluentd config folders readable · c92378d7
      Doug Szumski authored 
      Currently, config folders lack the execute bit so Fluentd
cannot read the config and just does nothing when it starts up. This
change explicitly sets the execute bit on folders which need it,
rather than doing it in a more generic way which is more risky from
a security perspective.

Change-Id: Ia840f4b67043df4eaa654f47673dcdc973f13d9c
Closes-Bug: #1867754
      c92378d7
  55. Mar 10, 2020
    • Will Szumski's avatar
      Use macro to avoid repetition · a1c51b73
      Will Szumski authored 
      I didn't use a for loop as the logic for omitting the
comma for the final element dirties the logic.

Change-Id: Id29d5deebcc5126d69a1bd8395e0df989f2081f0
      a1c51b73
    • Will Szumski's avatar
      Delete stale fluent config on restart · e7870e9d
      Will Szumski authored 
      We already only include .conf files in fluent.conf:

(fluentd)[fluentd@cpu-e-1041 /etc/fluentd]$ cat fluent.conf
@include input/*.conf
@include filter/*.conf
@include format/*.conf
@include output/*.conf

so this change should not cause ill effect. This works because of the
merge option in config files:

merge: merges the source directory into the target directory instead of
replacing it. Boolean, defaults to false.

see https://docs.openstack.org/kolla/latest/admin/kolla_api.html#kolla-api-external-config

Change-Id: I28f63ec81f1ea5bc4a213d053bfb2c04388d5925
Closes-Bug: #1862211
      e7870e9d
  57. Mar 02, 2020
  59. Feb 11, 2020
  61. Feb 04, 2020
    • Doug Szumski's avatar
      Disable Fluentd Monasca plugin retry limit · f19eb7de
      Doug Szumski authored 
      By default a retry limit of 17 exists. When the limit is reached buffered
logs are discarded. To avoid this, we disable the retry limit. The risk of
bringing down the host by filling the Fluent data docker volume is managed
by the maximum buffer size which is 2GB by default.

In summary, after this change, the net behaviour is that Fluentd should
buffer up to a maximum of 2GB of logs locally, and attept to post them to
the Monasca Log API at intervals not exceeding 30 minutes.

Closes-Bug: #1855702
Change-Id: I0d5a3dab29635c00411f4f51e5a0721726df2abd
      f19eb7de
    • Doug Szumski's avatar
      Enable buffering to file for Monasca logs · 5293b129
      Doug Szumski authored 
      This enables buffering to file, rather than memory for Monasca logs.
A dedicated docker volume is used for the file buffer. If a post
to the Monasca Log API fails, retries will be made using an exponential
backoff algorithm with a maximum retry interval of 30mins. The maximum
interval is set relatively low to try and reduce the risk of large
buffers accumulating, and therefore the risk of overloading the Monasca
Log API.

Closes-Bug: #1855700
Change-Id: Ib5286e9dbaf2bc92d2f4960b2131223ab5dbdbec
      5293b129
  63. Dec 18, 2019
  65. Dec 09, 2019