Replaced "kolla_external_fqdn_cacert" and "kolla_internal_fqdn_cacert" with
"kolla_admin_openrc_cacert". OS_CACERT is now set to the value of
"kolla_admin_openrc_cacert" in the generated admin-openrc.sh file.

Change-Id: If195d5402579cee9a14b91f63f5fde84eb84cccf
Partially-Implements: blueprint add-ssl-internal-network
Depends-On: https://review.opendev.org/#/c/731344/

The Monasca Log API has been removed and in this change we switch
to using the unified API. If dedicated log APIs are required then
this can be supported through configuration. Out of the box the
Monasca API is used for both logs and metrics which is envisaged to
work for most use cases.

In order to use the unified API for logs, we need to disable the
legacy Kafka client. We also rename the Monasca API config file
to remove a warning about using the old style name.

Depends-On: https://review.opendev.org/#/c/728638
Change-Id: I9b6bf5b6690f4b4b3445e7d15a40e45dd42d2e84

Depends-On: https://review.opendev.org/710217/

Change-Id: I85652f23e487c40192106d23f2cdd45a3077deca

Change-Id: I812665059783617d581d748e619b29426f89b353

Add TLS support for Glance api using HAProxy to perform TLS termination.

Change-Id: I77051baaeb5d3f7dd9002262534e7d35f3926809
Partially-Implements: blueprint add-ssl-internal-network

Zun has a new component "zun-cni-daemon" which should be
deployed in every compute nodes. It is basically an implementation
of CNI (Container Network Interface) that performs the neutron
port binding.

If users is using the capsule (pod) API, the recommended deployment
option is using "cri" as capsule driver. This is basically to use
a CRI runtime (i.e. CRI plugin for containerd) for supporting
capsules (pods). A CRI runtime needs a CNI plugin which is what
the "zun-cni-daemon" provides.

The configuration is based on the Zun installation guide [1].
It consits of the following steps:
* Configure the containerd daemon in the host. The "zun-compute"
  container will use grpc to communicate with this service.
* Install the "zun-cni" binary at host. The containerd process
  will invoke this binary to call the CNI plugin.
* Run a "zun-cni-daemon" container. The "zun-cni" binary will
  communicate with this container via HTTP.

Relevant patches:
Blueprint: https://blueprints.launchpad.net/zun/+spec/add-support-cri-runtime
Install guide: https://review.opendev.org/#/c/707948/
Devstack plugin: https://review.opendev.org/#/c/705338/
Kolla image: https://review.opendev.org/#/c/708273/

[1] https://docs.openstack.org/zun/latest/install/index.html

Depends-On: https://review.opendev.org/#/c/721044/
Change-Id: I9c361a99b355af27907cf80f5c88d97191193495

Kolla Ansible was missing vitrage-persistor service
required by Vitrage for data storage.

Depends on fixing availability of Kolla image.

Change-Id: I8158ba66b8b624f6bcb89da9c990a30a68b7187b
Depends-On: Id5e143636f9a81e7294b775f3d8b9134bee58054
Closes-Bug: #1869319

Refactor service configuration to use the copy certificates task. This
reduces code duplication and simplifies implementing encrypting backend
HAProxy traffic for individual services.

Change-Id: I0474324b60a5f792ef5210ab336639edf7a8cd9e

mistralclient osc plugin does not support cacert and insecure [1]
mistralclient interface support fixed in [2]

[1] https://bugs.launchpad.net/python-mistralclient/+bug/1715091
[2] https://review.opendev.org/#/q/topic:bug/1854339

Change-Id: I44726b12358bc3c5898ba952371fb838693aca2c

Some services look for /etc/timezone on Debian/Ubuntu, so we should
introduce it to the containers.

In addition, added prechecks for /etc/localtime and /etc/timezone.

Closes-Bug: #1821592
Change-Id: I9fef14643d1bcc7eee9547eb87fa1fb436d8a6b3

kolla-toolbox container name variable is fixed in some places,
but in the defaults directory, kolla-toolbox container_name variable
is variable. If the kolla-toolbox container_name variable is changed
during deployment, it will not be assigned to kolla-toolbox,
but a fixed 'kolla-toolbox' name.

Change-Id: I9579017761ff47477dba597282be9ae6fab4242a

The 'kolla-ansible stop' command can be used to stop the services
running on hosts. However, if you run this command in an environment
with heterogeneous nodes (most real world scenarios have at least
control/compute), then it fails. This is because it only checks
whether a container is enabled, and not whether the host is in the
correct group. For example, it fails with nova-libvirt:

    No such container: nova_libvirt to stop.

This change fixes the issue by only attempting to stop containers on
hosts to which they are mapped.

Change-Id: Ibecac60d1417269bbe25a280996ca9de6e6d018f
Closes-Bug: #1868596

Fluentd cannot accept empty 'path' parameter.

I refactored the service list following the general pattern
we have.

Change-Id: I83d820efcc7e86bac9f8bda26a8f8bece72159e6
Closes-bug: #1867953

Currently, config folders lack the execute bit so Fluentd
cannot read the config and just does nothing when it starts up. This
change explicitly sets the execute bit on folders which need it,
rather than doing it in a more generic way which is more risky from
a security perspective.

Change-Id: Ia840f4b67043df4eaa654f47673dcdc973f13d9c
Closes-Bug: #1867754

Fix elasticsearch schema in fluentd when kolla_enable_tls_internal is
true.

Change-Id: I51286d2def7a762d569740c1abc5b924b682ad9d
Closes-Bug: #1867481

I didn't use a for loop as the logic for omitting the
comma for the final element dirties the logic.

Change-Id: Id29d5deebcc5126d69a1bd8395e0df989f2081f0

When change the cert file in /etc/kolla/certificate/.
The certificate in the container has not changed.
So I think can use kolla-ansible deploy when certificate is
changed. restart <container>

Partially-Implements: blueprint custom-cacerts

Change-Id: Iaac6f37e85ffdc0352e8062ae5049cc9a6b3db26
Signed-off-by: yj.bai <bai.yongjun@99cloud.net>

We already only include .conf files in fluent.conf:

(fluentd)[fluentd@cpu-e-1041 /etc/fluentd]$ cat fluent.conf
@include input/*.conf
@include filter/*.conf
@include format/*.conf
@include output/*.conf

so this change should not cause ill effect. This works because of the
merge option in config files:

merge: merges the source directory into the target directory instead of
replacing it. Boolean, defaults to false.

see https://docs.openstack.org/kolla/latest/admin/kolla_api.html#kolla-api-external-config

Change-Id: I28f63ec81f1ea5bc4a213d053bfb2c04388d5925
Closes-Bug: #1862211

The logrotate rotation interval and count are not configurable.
Currently, the configuration is a "default" that keeps 6 weeks of logs.

Change-Id: I4f55ee2a98f7861cb8de2724f5edc32da6d2f9ee

Both include_role and import_role expect role's name to be given
via "name" param instead of "role".
This worked but caused errors with ansible-lint.
See: https://review.opendev.org/694779

Change-Id: I388d4ae27111e430d38df1abcb6c6127d90a06e0

We assume that all groups are present in the inventory, and quite obtuse
errors can result if any are not.

This change adds a precheck that checks for the presence of all expected
groups in the inventory for each service. It also introduces a common
service-precheck role that we can use for other common prechecks.

Change-Id: Ia0af1e7df4fff7f07cd6530e5b017db8fba530b3
Partially-Implements: blueprint improve-prechecks

Kolla-Ansible Ceph deployment mechanism has been deprecated in Train [1].

This change removes the Ansible code and associated CI jobs.

[1]: https://review.opendev.org/669214

Change-Id: Ie2167f02ad2f525d3b0f553e2c047516acf55bc2

By default a retry limit of 17 exists. When the limit is reached buffered
logs are discarded. To avoid this, we disable the retry limit. The risk of
bringing down the host by filling the Fluent data docker volume is managed
by the maximum buffer size which is 2GB by default.

In summary, after this change, the net behaviour is that Fluentd should
buffer up to a maximum of 2GB of logs locally, and attept to post them to
the Monasca Log API at intervals not exceeding 30 minutes.

Closes-Bug: #1855702
Change-Id: I0d5a3dab29635c00411f4f51e5a0721726df2abd

This enables buffering to file, rather than memory for Monasca logs.
A dedicated docker volume is used for the file buffer. If a post
to the Monasca Log API fails, retries will be made using an exponential
backoff algorithm with a maximum retry interval of 30mins. The maximum
interval is set relatively low to try and reduce the risk of large
buffers accumulating, and therefore the risk of overloading the Monasca
Log API.

Closes-Bug: #1855700
Change-Id: Ib5286e9dbaf2bc92d2f4960b2131223ab5dbdbec

When kolla_copy_ca_into_containers is set to "yes", the Certificate
Authority in /etc/kolla/certificates will be copied into service
containers to enable trust for that CA. This is especially useful when
the CA is self signed, and would not be trusted by default.

Partially-Implements: blueprint custom-cacerts

Change-Id: I4368f8994147580460ebe7533850cf63a419d0b4

docker_image_facts has been deprecated.

[1] https://docs.ansible.com/ansible/2.8/modules/docker_image_facts_module.html

Depends-On: https://review.opendev.org/#/c/699333/
Change-Id: I4556e603ec501f9bdaab5f20cee754467129c4e9

For the CentOS 7 to 8 transition, we will have a period where both
CentOS 7 and 8 images are available. We differentiate these images via a
tag - the CentOS 8 images will have a tag of train-centos8 (or
master-centos8 temporarily).

To achieve this, and maintain backwards compatibility for the
openstack_release variable, we introduce a new 'openstack_tag' variable.
This variable is based on openstack_release, but has a suffix of
'openstack_tag_suffix', which is empty except on CentOS 8 where it has a
value of '-centos8'.

Change-Id: I12ce4661afb3c255136cdc1aabe7cbd25560d625
Partially-Implements: blueprint centos-rhel-8

Change-Id: Ifa8fb271ee2d5642785097755f7347e3be00f8e9
Signed-off-by: yj.bai <bai.yongjun@99cloud.net>

deploy rabbitmq cluster by train with ipv6 report:
unable to connect to epmd (port 4369) on control-1: address (cannot connect to host/port)

Closes-Bug: #1856725
Change-Id: I36ebb4e196ece8a304269e8c85e39dda72faae50
Signed-off-by: yj.bai <bai.yongjun@99cloud.net>

WSGI log files use a different input configuration than OpenStack log
files. Currently this depends on log files matching either *-access.log
or *-error.log. Some services use *_access.log or *_error.log, so are
not parsed correctly.

This change modifies the fluentd configuration to accept an underscore
or hyphen for WSGI log file names.

Change-Id: I566d6cac0b6749054fd5422ec8f36f99dacb1db7
Closes-Bug: #1720371

Enable reconnect_on_error option so that ES plugin re-establishes
a new session to the ES cluster on errors. Also, enable buffering
to the file, so that the buffer survives container restarts.

Co-Authored-By: Michal Nasiadka <mnasiadka@gmail.com>
Co-Authored-By: Radosław Piliszek <radoslaw.piliszek@gmail.com>
Co-Authored-By: Doug Szumski <doug@stackhpc.com>
Closes-Bug: #1830724
Change-Id: Ia40685b9d4fc02194e03c8791ddeb3d29d7f07f6

Depends-On: https://review.opendev.org/692948/
Depends-On: https://review.opendev.org/692691/
Change-Id: I07827b896d36c3723697540fcff164224f6729af

- transitional handling of fluentd_binary var is no longer required

Change-Id: Ic2978252fb981fe15e600aa486e8af585d05c402

Opendaylight support has been deprecated in Train - time to remove it.

Change-Id: I3a61bfbcbf366c327ea3e25d2424bc3fedca29f0

Change-Id: I49b24545501085d5a44f4de73f0c6dd21e06e2a0
Closes-Bug: #1835501

MariaDB logs contain two different log message formats, one output
from mysqld and one from mysqld_safe. This patch splits the message
formats by tag and parses them separately.

Change-Id: I58857be67ae387eeda7487811a6af85b0f95970c
Closes-Bug: #1845629

Introduce kolla_address filter.
Introduce put_address_in_context filter.

Add AF config to vars.

Address contexts:
- raw (default): <ADDR>
- memcache: inet6:[<ADDR>]
- url: [<ADDR>]

Other changes:

globals.yml - mention just IP in comment

prechecks/port_checks (api_intf) - kolla_address handles validation

3x interface conditional (swift configs: replication/storage)

2x interface variable definition with hostname
(haproxy listens; api intf)

1x interface variable definition with hostname with bifrost exclusion
(baremetal pre-install /etc/hosts; api intf)

neutron's ml2 'overlay_ip_version' set to 6 for IPv6 on tunnel network

basic multinode source CI job for IPv6

prechecks for rabbitmq and qdrouterd use proper NSS database now

MariaDB Galera Cluster WSREP SST mariabackup workaround
(socat and IPv6)

Ceph naming workaround in CI
TODO: probably needs documenting

RabbitMQ IPv6-only proto_dist

Ceph ms switch to IPv6 mode

Remove neutron-server ml2_type_vxlan/vxlan_group setting
as it is not used (let's avoid any confusion)
and could break setups without proper multicast routing
if it started working (also IPv4-only)

haproxy upgrade checks for slaves based on ipv6 addresses

TODO:

ovs-dpdk grabs ipv4 network address (w/ prefix len / submask)
not supported, invalid by default because neutron_external has no address
No idea whether ovs-dpdk works at all atm.

ml2 for xenapi
Xen is not supported too well.
This would require working with XenAPI facts.

rp_filter setting
This would require meddling with ip6tables (there is no sysctl param).
By default nothing is dropped.
Unlikely we really need it.

ironic dnsmasq is configured IPv4-only
dnsmasq needs DHCPv6 options and testing in vivo.

KNOWN ISSUES (beyond us):

One cannot use IPv6 address to reference the image for docker like we
currently do, see: https://github.com/moby/moby/issues/39033
(docker_registry; docker API 400 - invalid reference format)
workaround: use hostname/FQDN

RabbitMQ may fail to bind to IPv6 if hostname resolves also to IPv4.
This is due to old RabbitMQ versions available in images.
IPv4 is preferred by default and may fail in the IPv6-only scenario.
This should be no problem in real life as IPv6-only is indeed IPv6-only.
Also, when new RabbitMQ (3.7.16/3.8+) makes it into images, this will
no longer be relevant as we supply all the necessary config.
See: https://github.com/rabbitmq/rabbitmq-server/pull/1982

For reliable runs, at least Ansible 2.8 is required (2.8.5 confirmed
to work well). Older Ansible versions are known to miss IPv6 addresses
in interface facts. This may affect redeploys, reconfigures and
upgrades which run after VIP address is assigned.
See: https://github.com/ansible/ansible/issues/63227

Bifrost Train does not support IPv6 deployments.
See: https://storyboard.openstack.org/#!/story/2006689



Change-Id: Ia34e6916ea4f99e9522cd2ddde03a0a4776f7e2c
Implements: blueprint ipv6-control-plane
Signed-off-by: Radosław Piliszek <radoslaw.piliszek@gmail.com>

This role can be used by other roles to register RabbitMQ resources.
Currently support is provided for creating virtual hosts and users.

Change-Id: Ie1774a10b4d629508584af679b8aa9e372847804
Partially Implements: blueprint support-nova-cells
Depends-On: https://review.opendev.org/684742

Sometimes as cloud admins, we want to only update code that is running
in a cloud.  But we dont need to do anything else.  Make an action in
kolla-ansible that allows us to do that.

Change-Id: I904f595c69f7276e71692696471e32fd1f88e6e8
Implements: blueprint deploy-containers-action

add clear old environment
set openstack client to use internalURL
set manila client to use internalURL

Change-Id: I263fa11ff5439b28d63a6a9ce7ba460cb56fb8e2