Fluentd has a default timeout of 5s for flushing data to ElasticSearch.
If there is a significant backlog of unsent log messages, this timeout
can be exceeded, resulting in Fluentd failing to make further progress.

Raise the default timeout to 60s.

This patch adopts the configuration parameters previously proposed by
Krzysztof Klimonda.

Closes-Bug: #1983031
Closes-Bug: #1896611
Change-Id: I1aaab654a5a0752fccef2cfb8cc0bde4a0ee2562

In fluentd v1.0, "format" became a deprecated parameter,
replaced by "<parse>"

Change-Id: I0825e81fcd90fbc7f64c8df3ea9ae199ad79294a

Change-Id: I5b3ab3ab8153cda283dec772bf1393af0caf4137
Closes-Bug: 1919179

Co-Authored-By: Mark Goddard <mark@stackhpc.com>

Change-Id: I75ca59d981bcd2dd51faa296ab0b4223a891f5cb

Overrides default fluentd buffer config to stop log files from using
datestamped filenames, allowing logrotate to manage them.

Closes-Bug: #1940118
Change-Id: I40c4e209470d21e0a02fd447fb628acfdae9fa9d

Closes-Bug: #1945070
Change-Id: I1b2a82b57cb9884b6c3c3ad07f6449ae29042a3d

When using elasticsearch 7 with fluentd, you seem to get a lot
of warnings in the docker logs output that look like:

    [types removal] Specifying types in bulk requests is deprecated.

The docs suggest adding suppress_type_name to stop these warnings,
and that seems to work without affecting any functionality.

Further info here:
https://github.com/uken/fluent-plugin-elasticsearch/issues/785

Closes-Bug: #1930856
Change-Id: I45be67df3717f78d78bcdc7df69600ab8681922f

Currently when elasticsearch log output is enabled there are lots
of warnings going into elasticsearch about type being deprecated
and needing to move to @type. This change stops those warnings.

TrivialFix

Change-Id: Ideac1925cb764ad0d7d8416f56d5e4a993c6d8b6

This is a follow up on the change with the following ID:

I337f42e174393f68b43e876ef075a74c887a5314

TrivialFix

Change-Id: Ibb67811d7b086ef9ef4c695ae589171af0c4d657

This change allows a user to forward control plane logs
directly to Elasticsearch from Fluentd, rather than via
the Monasca Log API when Monasca is enabled. The Monasca
Log API can continue to handle tenant logs.

For many use cases this is simpler, reduces resource
consumption and helps to decouple control plane logging
services from tenant logging services.

It may not always be desired, so is optional and off by
default.

Change-Id: I195e8e4b73ca8f573737355908eb30a3ef13b0d6

Add TLS support for backend Neutron API Server communication using
HAProxy to perform TLS termination. When used in conjunction with
enabling TLS for service API endpoints, network communication will be
encrypted end to end, from client through HAProxy to the Neutron
service.

Change-Id: Ib333a1f1bd12491df72a9e52d961161210e2d330
Partially-Implements: blueprint add-ssl-internal-network

Change-Id: Iabc0115d3476a626df134cc70cb473bf6e72487e
Closes-Bug: #1890439

The goal for this push request is to normalize the construction and use
 of internal, external, and admin URLs. While extending Kolla-ansible
 to enable a more flexible method to manage external URLs, we noticed
 that the same URL was constructed multiple times in different parts
 of the code. This can make it difficult for people that want to work
 with these URLs and create inconsistencies in a large code base with
 time. Therefore, we are proposing here the use of
 "single Kolla-ansible variable" per endpoint URL, which facilitates
 for people that are interested in overriding/extending these URLs.

As an example, we extended Kolla-ansible to facilitate the "override"
of public (external) URLs with the following standard
"<component/serviceName>.<companyBaseUrl>".
Therefore, the "NAT/redirect" in the SSL termination system (HAproxy,
HTTPD or some other) is done via the service name, and not by the port.
This allows operators to easily and automatically create more friendly
 URL names. To develop this feature, we first applied this patch that
 we are sending now to the community. We did that to reduce the surface
  of changes in Kolla-ansible.

Another example is the integration of Kolla-ansible and Consul, which
we also implemented internally, and also requires URLs changes.
Therefore, this PR is essential to reduce code duplicity, and to
facility users/developers to work/customize the services URLs.

Change-Id: I73d483e01476e779a5155b2e18dd5ea25f514e93
Signed-off-by: Rafael Weingärtner <rafael@apache.org>

A "@type copy" statement is already present at the beginning of each
match element, so extra "type copy" are not needed. They are causing the
following warnings in fluentd logs:

[warn]: parameter 'type' in <match syslog.local0.**>
[warn]: parameter 'type' in <match syslog.local1.**>

This commit also harmonizes indentation of the Monasca config block.

Change-Id: I779c2b942d007acbdd43d999f2fc0cdc131d431f
Related-Bug: #1885873

In Fluentd v0.12, both the in memory and file buffer chunk size default
to 8MB. In v1.0 the file buffer defaults to 256MB. This can exceed the
Monasca Log or Unified API maximum chunk size which is set to 10MB.
This can result in logs being rejected and filling the local buffer
on disk.

Change-Id: I9c495773db726a3c5cd94b819dff4141737a1d6e
Closes-Bug: #1885885
Co-Authored-By: Sebastian Luna Valero <sebastian.luna.valero@gmail.com>

Currently there is no way to configure a CA certificate bundle file for
fluentd to Elasticsearch communication. This change adds a new variable,
'fluentd_elasticsearch_cacert' with a default value set to the value of
'openstack_cacert.

Closes-Bug: #1885109

Change-Id: I5bbf55a4dd4ccce9fa2635cee720139c088268e3

The Monasca Log API has been removed and in this change we switch
to using the unified API. If dedicated log APIs are required then
this can be supported through configuration. Out of the box the
Monasca API is used for both logs and metrics which is envisaged to
work for most use cases.

In order to use the unified API for logs, we need to disable the
legacy Kafka client. We also rename the Monasca API config file
to remove a warning about using the old style name.

Depends-On: https://review.opendev.org/#/c/728638
Change-Id: I9b6bf5b6690f4b4b3445e7d15a40e45dd42d2e84

Change-Id: I812665059783617d581d748e619b29426f89b353

Add TLS support for Glance api using HAProxy to perform TLS termination.

Change-Id: I77051baaeb5d3f7dd9002262534e7d35f3926809
Partially-Implements: blueprint add-ssl-internal-network

By default a retry limit of 17 exists. When the limit is reached buffered
logs are discarded. To avoid this, we disable the retry limit. The risk of
bringing down the host by filling the Fluent data docker volume is managed
by the maximum buffer size which is 2GB by default.

In summary, after this change, the net behaviour is that Fluentd should
buffer up to a maximum of 2GB of logs locally, and attept to post them to
the Monasca Log API at intervals not exceeding 30 minutes.

Closes-Bug: #1855702
Change-Id: I0d5a3dab29635c00411f4f51e5a0721726df2abd

This enables buffering to file, rather than memory for Monasca logs.
A dedicated docker volume is used for the file buffer. If a post
to the Monasca Log API fails, retries will be made using an exponential
backoff algorithm with a maximum retry interval of 30mins. The maximum
interval is set relatively low to try and reduce the risk of large
buffers accumulating, and therefore the risk of overloading the Monasca
Log API.

Closes-Bug: #1855700
Change-Id: Ib5286e9dbaf2bc92d2f4960b2131223ab5dbdbec

Enable reconnect_on_error option so that ES plugin re-establishes
a new session to the ES cluster on errors. Also, enable buffering
to the file, so that the buffer survives container restarts.

Co-Authored-By: Michal Nasiadka <mnasiadka@gmail.com>
Co-Authored-By: Radosław Piliszek <radoslaw.piliszek@gmail.com>
Co-Authored-By: Doug Szumski <doug@stackhpc.com>
Closes-Bug: #1830724
Change-Id: Ia40685b9d4fc02194e03c8791ddeb3d29d7f07f6

Introduce kolla_address filter.
Introduce put_address_in_context filter.

Add AF config to vars.

Address contexts:
- raw (default): <ADDR>
- memcache: inet6:[<ADDR>]
- url: [<ADDR>]

Other changes:

globals.yml - mention just IP in comment

prechecks/port_checks (api_intf) - kolla_address handles validation

3x interface conditional (swift configs: replication/storage)

2x interface variable definition with hostname
(haproxy listens; api intf)

1x interface variable definition with hostname with bifrost exclusion
(baremetal pre-install /etc/hosts; api intf)

neutron's ml2 'overlay_ip_version' set to 6 for IPv6 on tunnel network

basic multinode source CI job for IPv6

prechecks for rabbitmq and qdrouterd use proper NSS database now

MariaDB Galera Cluster WSREP SST mariabackup workaround
(socat and IPv6)

Ceph naming workaround in CI
TODO: probably needs documenting

RabbitMQ IPv6-only proto_dist

Ceph ms switch to IPv6 mode

Remove neutron-server ml2_type_vxlan/vxlan_group setting
as it is not used (let's avoid any confusion)
and could break setups without proper multicast routing
if it started working (also IPv4-only)

haproxy upgrade checks for slaves based on ipv6 addresses

TODO:

ovs-dpdk grabs ipv4 network address (w/ prefix len / submask)
not supported, invalid by default because neutron_external has no address
No idea whether ovs-dpdk works at all atm.

ml2 for xenapi
Xen is not supported too well.
This would require working with XenAPI facts.

rp_filter setting
This would require meddling with ip6tables (there is no sysctl param).
By default nothing is dropped.
Unlikely we really need it.

ironic dnsmasq is configured IPv4-only
dnsmasq needs DHCPv6 options and testing in vivo.

KNOWN ISSUES (beyond us):

One cannot use IPv6 address to reference the image for docker like we
currently do, see: https://github.com/moby/moby/issues/39033
(docker_registry; docker API 400 - invalid reference format)
workaround: use hostname/FQDN

RabbitMQ may fail to bind to IPv6 if hostname resolves also to IPv4.
This is due to old RabbitMQ versions available in images.
IPv4 is preferred by default and may fail in the IPv6-only scenario.
This should be no problem in real life as IPv6-only is indeed IPv6-only.
Also, when new RabbitMQ (3.7.16/3.8+) makes it into images, this will
no longer be relevant as we supply all the necessary config.
See: https://github.com/rabbitmq/rabbitmq-server/pull/1982

For reliable runs, at least Ansible 2.8 is required (2.8.5 confirmed
to work well). Older Ansible versions are known to miss IPv6 addresses
in interface facts. This may affect redeploys, reconfigures and
upgrades which run after VIP address is assigned.
See: https://github.com/ansible/ansible/issues/63227

Bifrost Train does not support IPv6 deployments.
See: https://storyboard.openstack.org/#!/story/2006689



Change-Id: Ia34e6916ea4f99e9522cd2ddde03a0a4776f7e2c
Implements: blueprint ipv6-control-plane
Signed-off-by: Radosław Piliszek <radoslaw.piliszek@gmail.com>

Add options for configuring TLS and authentication for elasticsearch
connections in in fluentd.

Change-Id: I936adc2aeaa3c87081be1c44aa0221caf2124e23
Closes-Bug: #1831078

Let the Monasca Fluentd output plugin handle mapping of the log
message.

Change-Id: I4a74a91b9b38d5c172397a7e7204e626bcedcfac
Closes-Bug: #1830184
Depends-On: https://review.opendev.org/#/c/660988/

When Monasca is enabled disable direct logging to ElasticSearch and
send all logs harvested by Fluentd to the Monasca Log API.

This change also cleans up output files which may be left behind when
the various log forwarding options are enabled / disabled.

Partially-Implements: blueprint monasca-roles
Change-Id: I7197966c5117176407d60c86c08d3bcea5e8131a

This change adds enable_fluentd option and enables some other log shippers
to be integrated. When enable_fluentd is "no", syslog server is also disabled.
Then, this change also adds syslog parameters to use a syslog server
prepared by users.

Change-Id: I7c83ef7fe30a6b9ab7385bcee953ad07e96b0a83
Implements: blueprint fluentd-enable-option

As the logrotate will create new filename with timestamp, fluentd
with symlink will staled when restart fluentd and can not set
permission on the old file.

Remove symlink from the global conf.

Change-Id: I404868d5b5077b9f2135e37605421bc84717f9d3
Closes-Bug: #1685170

When Kibana interface is opened for the first time, it requires creating
a default index pattern. To view, analyse and search logs, at least one
index pattern has to be created.
This patch automate "flog-*" default index pattern creation and set it
has default config.

Partially-implements: blueprint default-kibana-dashboard
Change-Id: Ie36696f9ad38ba7e49e65e0793a3b98d9f03ee8d

Haproxy binds the elasticsearch service to kolla_internal_vip_address but
the output templates for fluentd (td-agent) point to a non-existent
kolla_external_vip_address.

Output should also be able to be sent to an external elasticsearch
instance (as per the documentation regarding overriding
elasticsearch_address)

Change these settings so that fluentd outputs to either
the default elasticsearch_address (i.e. kolla_internal_vip_address) or to
the external elasticsearch instance.

Closes-Bug: #1673990
Change-Id: I081533ae8ea9aad186e9c44e1dee069729931453

Change-Id: I93fe8141a88d6e0600a1f44fe49d96d7816fae87
Closes-Bug: #1664868

* add fluentd role
* remove heka configure

Co-Authored-By: yangzb09 <yangzb09@qq.com>

Partially-Implements: blueprint add-fluentd-role

Change-Id: Ica804a99f5bb8b157f406299c5982b7b6283b3e3