fluentd logs currently to stdout, which is known to produce big docker logs
in /var/lib/docker. This change makes fluentd to log to /var/log/kolla/fluentd.

Closes-Bug: #1888852
Change-Id: I8fe0e54cb764a26d26c6196cef68aadc6fd57b90

A "@type copy" statement is already present at the beginning of each
match element, so extra "type copy" are not needed. They are causing the
following warnings in fluentd logs:

[warn]: parameter 'type' in <match syslog.local0.**>
[warn]: parameter 'type' in <match syslog.local1.**>

This commit also harmonizes indentation of the Monasca config block.

Change-Id: I779c2b942d007acbdd43d999f2fc0cdc131d431f
Related-Bug: #1885873

Change-Id: Ia134a518b63bb59cfad631cc488181f5245160e6

we should clone freezer code before run bootstray,
otherwise, the directory /opt/stack/freezer which is empty will
mount into freezer_api container.

Closes-Bug: #1888242

Change-Id: I7c22dd380fd5b1dff7b421109c4ae37bab11834a

Option "trove_auth_url/os_region_name" from group "DEFAULT" is deprecated.
Use option "auth_url/region_name" from group service_credentials

Change-Id: I15d6891582c92c7fc813f280a2b47ebaaca77eba

This makes use of udev rules to make it smarter and override
host-level packages settings.
Additionally, this masks Ubuntu-only service that is another
pain point in terms of /dev/kvm permissions.
Fingers crossed for no further surprises.

Change-Id: I61235b51e2e1325b8a9b4f85bf634f663c7ec3cc
Closes-bug: #1681461

Switch to the Confluent Kafka client in all remaining Python based
Monasca services. This should allow us to later un-pin the Kafka
messaging version for Monasca.

Change-Id: I42bc78ffe304ba21c448c2e08b025e93a70ddb44

Co-Authored-By: Doug Szumski <doug@stackhpc.com>
Closes-Bug: #1884090
Depends-On: https://review.opendev.org/#/c/736768

Change-Id: If2d0dd1739e484b14e3c15a185a236918737b0ab

The Castellan (Barbican client) has different parameters to control
the used CA file.
This patch uses them.
Moreover, this aligns Barbican with other services by defaulting
its client config to the internal endpoint.

See also [1].

[1] https://bugs.launchpad.net/castellan/+bug/1876102

Closes-Bug: #1886615

Change-Id: I6a174468bd91d214c08477b93c88032a45c137be

planned removal

Change-Id: Ib37ea4d42f82096a682cebc724c45c9dd39c8b47

The bug is fixed[1], releated task is unncessary.

[1]: https://storyboard.openstack.org/#!/story/2006393

Depends-On: Ib62ca3ee4626084e5e9b90e93e4fa97938023457
Change-Id: I2553c3c4a6d3c82405c68c52db2e7585477b1dff

The nova-cell role sets the following sysctls on compute hosts, which
require the br_netfilter kernel module to be loaded:

    net.bridge.bridge-nf-call-iptables
    net.bridge.bridge-nf-call-ip6tables

If it is not loaded, then we see the following errors:

    Failed to reload sysctl:
    sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-iptables: No such file or directory
    sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-ip6tables: No such file or directory

Loading the br_netfilter module resolves this issue.

Typically we do not see this since installing Docker and configuring it
to manage iptables rules causes the br_netfilter module to be loaded.
There are good reasons [1] to disable Docker's iptables management
however, in which case we are likely to hit this issue.

This change loads the br_netfilter module in the nova-cell role for
compute hosts.

[1] https://bugs.launchpad.net/kolla-ansible/+bug/1849275



Co-Authored-By: Dincer Celik <hello@dincercelik.com>

Change-Id: Id52668ba8dab460ad4c33fad430fc8611e70825e

Change-Id: Idfa7feb9157cdf12af9641be10c8eb8478930f15
Closes-Bug: #1886747

The value should be the full path to the keyring file, not just the
name. Without this fix Gnocchi fails to connect to Ceph.

Change-Id: Iaa69b2096b09a448345de50911e21436875d48d6
Closes-Bug: #1886711

The common role was previously added as a dependency to all other roles.
It would set a fact after running on a host to avoid running twice. This
had the nice effect that deploying any service would automatically pull
in the common services for that host. When using tags, any services with
matching tags would also run the common role. This could be both
surprising and sometimes useful.

When using Ansible at large scale, there is a penalty associated with
executing a task against a large number of hosts, even if it is skipped.
The common role introduces some overhead, just in determining that it
has already run.

This change extracts the common role into a separate play, and removes
the dependency on it from all other roles. New groups have been added
for cron, fluentd, and kolla-toolbox, similar to other services. This
changes the behaviour in the following ways:

* The common role is now run for all hosts at the beginning, rather than
  prior to their first enabled service
* Hosts must be in the necessary group for each of the common services
  in order to have that service deployed. This is mostly to avoid
  deploying on localhost or the deployment host
* If tags are specified for another service e.g. nova, the common role
  will *not* automatically run for matching hosts. The common tag must
  be specified explicitly

The last of these is probably the largest behaviour change. While it
would be possible to determine which hosts should automatically run the
common role, it would be quite complex, and would introduce some
overhead that would probably negate the benefit of splitting out the
common role.

Partially-Implements: blueprint performance-improvements

Change-Id: I6a4676bf6efeebc61383ec7a406db07c7a868b2a

Change-Id: I51054561af93f03e09fa86eeec7c579aca103cdf

1. Use the children group for site.yml
2. Add some missing groups

Change-Id: I01d686368b11a105a8965cf987d23772ecbf97de

Change I810aad7d49db3f5a7fd9a2f0f746fd912fe03917 for supporting multiple
Nova cells updated the list of containers that require a policy file to
only include nova-api, nova-compute, and nova-compute-ironic.

The nova-conductor config.json template was left unchanged and fails to
copy the nova policy file into its container. This can be seen on a
fresh deployment, but might be missed on an upgrade if an older policy
file is still available in /etc/kolla/nova-conductor.

This commit removes the nova_policy_file block from the nova-conductor
config.json template, as it shouldn't be required.

Backport: ussuri, train
Change-Id: I17256b182d207aeba3f92c65a6d7cf3611180558
Closes-Bug: #1886170

Work was done to selectively enable Open vSwitch deployment for Manila
services as bug #1884939.  However this did not appear to catch all
cases.  This patch adds a couple more.

Change-Id: I6187997a00f908e87ceace6f79f5f7262ea78738
Closes-Bug: #1886166
Co-Authored-By: Sebastian Luna Valero <sebastian.luna.valero@gmail.com>

barbican alway use default notification driver (defalt '')
so we should change this value according to whether enable
notification

Change-Id: Ia17a64fe9bf31042369dec19f1f76b1ab8592288

Time format in Ruby Time.strptime is not accepting padding flags,
therefore we need to remove them for the Fluentd to be able
to parse MariaDB xinetd logs properly.

Change-Id: Iabfa9afdcad505106a5580eb2d058273ee5f7c1f
Closes-Bug: #1886002

In Fluentd v0.12, both the in memory and file buffer chunk size default
to 8MB. In v1.0 the file buffer defaults to 256MB. This can exceed the
Monasca Log or Unified API maximum chunk size which is set to 10MB.
This can result in logs being rejected and filling the local buffer
on disk.

Change-Id: I9c495773db726a3c5cd94b819dff4141737a1d6e
Closes-Bug: #1885885
Co-Authored-By: Sebastian Luna Valero <sebastian.luna.valero@gmail.com>

In the spirit of Kolla-Ansible, we generally try to provide
workable defaults.
The default for Elasticsearch curator schedule was fine except for
multinode deploys where it would cause all nodes to run at the
same time producing broken runs (race condition in the get-delete
cycle).
It is easy to improve this situation by embracing poor-man's
reimplementation of keystone's fernet key rotation schedule.
ES Curator does not need all the complexity of the former so it
can be handled very well by shifting by as many hours as the
instance's index dictates. It should rarely if ever need more time
(most likely still in minutes range rather than hours).

Change-Id: I9d6758c8550308d13d936de1a14afbe4124e593b

Resolve trivial syntax error in Fluentd output config for Monasca.

Change-Id: I20b37bb83a76bfabb1126925a1b4f1f59767b7a3
Co-Authored-By: Sebastian Luna Valero <sebastian.luna.valero@gmail.com>
Closes-Bug: #1885873

While all other clients should use internalURL, the Magnum client itself
and Keystone interface for trustee credentials should be publicly
accessible (upstream default when no config is specified) since
instances need to be able to reach them.

Closes-Bug: #1885420
Change-Id: I74359cec7147a80db24eb4aa4156c35d31a026bf

There were two issues with it. Lack of /usr/local/bin in PATH
for CentOS and wrong crontab path for Ubuntu/Debian.
This patch mirrors how it is handled in keystone.

Change-Id: Ib54b261e12c409d66b792648807646015826e83c
Closes-Bug: #1885732

The Zun configuration file does not set the CA for the clients the Zun
service uses: zun_client, glance_client, neutron_client, cinder_client,
and placement_client. This will cause the Zun service to fail when
TLS is enabled in the OpenStack deployment.

Depends-On: https://review.opendev.org/#/c/736809
Change-Id: Ieed843c890210608699c1a63deed66c9bb63986c

Change-Id: Ia22f286e85be90983ca79291b3a54596bba30d6c

The etcd service protocol is currently configured with internal_protocol.
The etcd service is not load balanced by a HAProxy container, so
there is no proxy layer to do TLS termination when internal_protocol
is configured to be "https".

Until the etcd service is configured to deploy with native TLS
termination, the etcd uses should be independent of
internal_protocol, and "http" by default.

Change-Id: I730c02331514244e44004aa06e9399c01264c65d
Closes-Bug: 1884137

Minor scalability improvement, not currently applied to storm.

Change-Id: I928d362067c52c3113bc0fbd3ae4b9be1810b7e5
TrivialFix

Currently openvswitch sets system-id based on inventory_hostname, but when
Ansible inventory contains ip addresses - then it will only take first ip
octet - resulting in multiple OVN chassis being named i.e. "10".
Then Neutron and OVN have problems functioning, because a chassis named "10"
will be created and deleted multiple times per second - this ends up in
ovsdb and neutron-server processes using up to 100% CPU.

Adding openvswitch role to ovn CI job triggers.

Change-Id: Id22eb3e74867230da02543abd93234a5fb12b31d
Closes-Bug: #1884734

Currently, if internal TLS communication is enabled, Kibana to
Elasticsearch communication is unverified. This is because we set
elasticsearch.ssl.verificationMode to 'none' by default (via
kibana_elasticsearch_ssl_verify). This is poor a security
posture.

This change changes the default value of
'kibana_elasticsearch_ssl_verify' to 'true'.

Change-Id: Ie4fa8e3a60d69cf5c4bdd975030c92be8113ffb1
Closes-Bug: #1885110

Currently there is no way to configure a CA certificate bundle file for
fluentd to Elasticsearch communication. This change adds a new variable,
'fluentd_elasticsearch_cacert' with a default value set to the value of
'openstack_cacert.

Closes-Bug: #1885109

Change-Id: I5bbf55a4dd4ccce9fa2635cee720139c088268e3

Change openvswitch & neutron-openvswitch-agent to deploy only
with manila generic backend - which uses ovs-vsctl functionality
when configuring share servers.

Change-Id: I124108cda62b38ea498612ff9ddb07d6122a330c
Closes-Bug: #1884939

Magnum, Cinder and Octavia clients in Magnum now use endpoint_type of
internalURL by default consistent with other clients also used by the
conductor. Additionally, they also use the globally defined
`openstack_region_name` for region_name.

Closes-Bug: #1885096

Change-Id: Ibec511013760cc4f681a2ec1b769b532be3daf2d

Change-Id: I7214ef38ea529f7585d7a0c75b8b0498ea4c58a2
Closes-Bug: #1885078

ZooKeeper is a dependency of Apache Storm.

TrivialFix

Change-Id: Icf952be2e0b53f2e82e8ce18a48bcfa100b41cd9

when enable kolla_dev_mod, nova-cell role clones code failed,
because we use nova-cell repository which is not exists.
in fact, nova-cell role should use nova repository too

Change-Id: I7fa62726d0d5b0aeb3bd5fa06dc0e59667f94fa0

Depends-On: I561504160e5548c54d1af31821c3366ab34cf0ec
Change-Id: I15e5c0e0a956ee181873cf002229532a15ff959d
Co-Authored-By: jacky06 <zhang.min@99cloud.net>

more info: https://review.opendev.org/#/c/721733/



Depends-On: I561ead226f714d98c8e06e6027715a64c3a8e47e
Depends-On: I21c9ab9820f78cf76adf11c5f0591c60f76372a8
Change-Id: Ic740d090211ee331b374a6dac69dfde466df7200
Co-Authored-By: jacky06 <zhang.min@99cloud.net>