Add privileged capability to cyborg agent.

Change-Id: Id237df1acb1b44c4e6442b39838058be1a95fcc6
Closes-bug: #1873715

Kolla Ansible was missing vitrage-persistor service
required by Vitrage for data storage.

Depends on fixing availability of Kolla image.

Change-Id: I8158ba66b8b624f6bcb89da9c990a30a68b7187b
Depends-On: Id5e143636f9a81e7294b775f3d8b9134bee58054
Closes-Bug: #1869319

It was missed in kolla-ceph removal in [1].

[1] 4e6fe7a6

Change-Id: Iacfce48f23dc5887c5f549a70250eee3e4e3f026

It is unnecessary to add octavia user into admin project.
Octavia project does not require this action. Like other projects,
octavia user in service project with admin role is enough.

[1] https://docs.openstack.org/octavia/latest/install/install-ubuntu.html
[2] https://docs.openstack.org/octavia/latest/contributor/guides/dev-quick-start.html#production-deployment-walkthrough
[3] https://github.com/openstack/octavia/blob/master/devstack/plugin.sh

Closes-Bug: #1873176
Change-Id: I35d35177aaabfc6f0abc533a1f756b363bd02308

Followup of last reviews from [1].

[1]: https://review.opendev.org/#/c/696841

Change-Id: I7085093b20e8848e09dc521ae9fbf120e909470d

Change-Id: I500cc8800c412bc0e95edb15babad5c1189e6ee4

Change-Id: I4673f436d8943e6fce7e579446c27ec8215b7346

The "balance" keyword is not valid in a frontend section. From the
HAProxy documentation[1]:

balance <algorithm> [ <arguments> ]
balance url_param <param> [check_post [<max_wait>]]
  Define the load balancing algorithm to be used in a backend.
  May be used in sections :   defaults | frontend | listen | backend
                                 yes   |    no    |   yes  |   yes

When running HAProxy using the "split" template style, where a
frontend/backend pair are used instead of one listen section, HAProxy
will emit warnings for the Horizon config due to this.

[1]: https://www.haproxy.org/download/1.5/doc/configuration.txt

Closes-Bug: #1872540
Change-Id: I91cee275d91a51944298618493f4ea0cd80282cc

This fixes Octavia in scenarios requiring providing
CA cert (self-signed, internally-signed).

Change-Id: I60b7ec85f4fd8bbacf5df0ab7ed9a00658c91871
Closes-Bug: #1872404

Change-Id: I22a995195a1d12bb759cba9777527c23475124f2

Deploy a small cloud. Add one host to the compute group in the
inventory, and scale out:

$ kolla-ansible deploy --limit <new compute host>

The command succeeds, but creating an instance fails with the following:

    Host 'compute0' is not mapped to any cell

This happens because we only discover computes on the first host in the
cell's nova conductor group. If that host is not in the specified limit,
the discovery will not happen.

This change fixes the issue by running compute discovery when any ironic
or virtualised compute hosts are in the play batch, and delegating it to
a conductor.

Change-Id: Ie984806240d147add825ffa8446ae6ff55ca4814
Closes-Bug: #1869371

When using the split config style, all backends would be empty, which
meant that HAProxy was unable to serve any traffic. This turned out to
be due to a bad default in the split config template.

Closes-Bug: #1872545
Change-Id: I952e526e735e1d31445963f04d41d66bbdbfdee4

Refactor service configuration to use the copy certificates task. This
reduces code duplication and simplifies implementing encrypting backend
HAProxy traffic for individual services.

Change-Id: I0474324b60a5f792ef5210ab336639edf7a8cd9e

etcd via tooz does not support group membership required by
Designate coordination.
The best k-a can do is not to configure etcd in Designate.

Change-Id: I2f64f928e730355142ac369d8868cf9f65ca357e
Closes-bug: #1872205
Related-bug: #1840070

Allow operators to use custom parameters with the ceilometer-upgrade
command. This is quite useful when using the dynamic pollster subsystem;
that sub-system provides flexibility to create and edit pollsters configs,
which affects gnocchi resource-type configurations. However, Ceilometer
uses default and hard-coded resource-type configurations; if one customizes
some of its default resource-types, he/she can get into trouble during
upgrades. Therefore, the only way to work around it is to use the
"--skip-gnocchi-resource-types" flag. This PR introduces a method for
operators to execute such customization, and many others if needed.

Depends-On: https://review.opendev.org/#/c/718190/
Change-Id: I92f0edba92c9e3707d89b3ff4033ac886b29cf6d

mistralclient osc plugin does not support cacert and insecure [1]
mistralclient interface support fixed in [2]

[1] https://bugs.launchpad.net/python-mistralclient/+bug/1715091
[2] https://review.opendev.org/#/q/topic:bug/1854339

Change-Id: I44726b12358bc3c5898ba952371fb838693aca2c

Change-Id: I137b4f6b229d9ef34ed5be0bc4c4828ed655269e
Closes-Bug: #1872008

Some services look for /etc/timezone on Debian/Ubuntu, so we should
introduce it to the containers.

In addition, added prechecks for /etc/localtime and /etc/timezone.

Closes-Bug: #1821592
Change-Id: I9fef14643d1bcc7eee9547eb87fa1fb436d8a6b3

In kolla ansible we typically configure services to communicate via IP
addresses rather than hostnames. One accidental exception to this was
live migration, which used the hostname of the destination even when
not required (i.e. TLS not being used for libvirt).

To make such hostnames work, k-a adds entries to /etc/hosts in the
bootstrap-servers command. Alternatively users may provide DNS.

One problem with using /etc/hosts is that, if a new compute host is
added to the cloud, or an IP address is changed, that will not be
reflected in the /etc/hosts file of other hosts. This would cause live
migration to the new host from an old host to fail, as the name cannot
be resolved.

The workaround for this was to update the /etc/hosts file (perhaps via
bootstrap-servers) on all hosts after adding new compute hosts. Then the
nova_libvirt container had to be restarted to pick up the change.

Similarly, if user has overridden the migration_interface, the used
hostname could point to a wrong address on which libvirt would not
listen.

This change adds the live_migration_inbound_addr option to nova.conf. If
TLS is not in use for libvirt, this will be set to the IP address of the
host on the migration network. If TLS is enabled for libvirt,
live_migration_inbound_addr will be set to migration_hostname, since
certificates will typically reference the hostname rather than the
host's IP. With libvirt TLS enabled, DNS is recommended to avoid the
/etc/hosts issue which is likely the case in production deployments.

Change-Id: I0201b46a9fbab21433a9f53685131aeb461543a8
Closes-Bug: #1729566

This patch introduces an optional backend encryption for Keystone
service. When used in conjunction with enabling TLS for service API
endpoints, network communcation will be encrypted end to end, from
client through HAProxy to the Keystone service.

Change-Id: I6351147ddaff8b2ae629179a9bc3bae2ebac9519
Partially-Implements: blueprint add-ssl-internal-network

Implement OVN Ansible role.

Implements: blueprint ovn-controller-neutron-ansible

Depends-On: https://review.opendev.org/713422
Change-Id: Icd425dea85d58db49c838839d8f0b864b4a89a78

This is a follow up to I001defc75d1f1e6caa9b1e11246abc6ce17c775b. To
maintain previous behaviour, and ensure we catch any host configuration
changes, we should perform host configuration during upgrade.

Change-Id: I79fcbf1efb02b7187406d3c3fccea6f200bcea69
Related-Bug: #1860161

Elasticsearch 6.x dropped support for mapping types[1], which by default
the Kibana index used. This means that when deploying ELK 6.x, the
Kibana index must be migrated to the new schema to preserve dashboards
and visualizations. There is a process defined[2], which involves
creating a new index with the specified schema, then reindexing the old
index's data into the new index, then doing a rename/delete.

This adds support for that workflow via Ansible. It takes place after
the ES container is restarted after an upgrade, so there will be a
(short) period of time where the Kibana index is not migrated. During
this time, Kibana still loads, but presents the user with a status
screen informing that the index needs migration.

[1]:
https://www.elastic.co/guide/en/elasticsearch/reference/6.x/removal-of-types.html
[2]: https://www.elastic.co/guide/en/kibana/6.x/migrating-6.0-index.html

Implements: blueprint elasticsearch-kibana-version-upgrade
Depends-On: https://review.opendev.org/709624
Change-Id: I4550629e2113f3da7f1cecfeab0d5fe0d899dae8

This updates the elasticsearch configuration file (and loading
mechanism) for ELK 6.x.

The default location for the configuration for all package
distributions is /etc/elasticsearch[1], so now that is where we
overwrite the elasticsearch.yml.

The path.conf and path.scripts paths are no longer supported and will
raise exceptions if utilized in 6.x.

[1]:
https://www.elastic.co/guide/en/elasticsearch/reference/6.x/settings.html#config-files-location

Implements: blueprint elasticsearch-kibana-version-upgrade
Depends-On: https://review.opendev.org/#/c/647748/
Change-Id: I4f74bfe07d4b7ca18953b11e767cf0bb94dfd67e

manila share container name variable is fixed in some places,
but in the defaults directory, manila share container_name variable
is variable. If the manila share container_name variable is changed
during deployment, it will not be assigned to container name,
but a fixed 'manila_share' name.

Change-Id: Iea23c62518add8d6820b76b16edd3221906b0ffb

The repo is Python 3 now, so update hacking to version 3.0 which
supports Python 3.

Fix problems found by updated hacking version.

Remove hacking and friends from lower-constraints, they are not needed
during installation.

Change-Id: I7ef5ac8a89e94f5da97780198619b6facc86ecfe

The use of default(omit) is for module parameters, not templates. We
define a default value for openstack_cacert, so it should never be
undefined anyway.

Change-Id: Idfa73097ca168c76559dc4f3aa8bb30b7113ab28

Currently there are a few services that perform host configuration
tasks. This is done in config.yml. This means that these changes are
performed during 'kolla-ansible genconfig', when we might expect not to
be making any changes to the remote system.

This change separates out these host configuration tasks into a
config-host.yml file, which is included directly from deploy.yml.

One change in behaviour is that this prevents these tasks from running
during an upgrade or genconfig. This is probably what we want, but we
should be careful when any of these host configuration tasks are
changed, to ensure they are applied during an upgrade if necessary.

Change-Id: I001defc75d1f1e6caa9b1e11246abc6ce17c775b
Closes-Bug: #1860161

One way to improve the performance of Ansible is through fact caching.
Rather than gather facts in every play, we can configure Ansible to
cache them in a persistent store. An example Ansible configuration for
doing this is as follows:

[defaults]
gathering = smart
fact_caching = jsonfile
fact_caching_connection = ./facts
fact_caching_timeout = 86400

This does not affect Kolla Ansible however, since we use the setup
module which unconditionally gathers facts regardless of the state of
the cache. This gets worse with large inventories limited to a small
batch of hosts via --limit or serial, since the limited hosts must
gather facts for all others.

One way to detect whether facts exist for a host is via the
'module_setup' variable, which exists only when facts exist. This change
uses the 'module_setup' fact to determine whether facts need to be
gathered for hosts outside of the batch. For hosts in the batch, we
switch from using the setup module to gather_facts on the play, which
can use the 'smart' gathering logic.

Change-Id: I04841fb62b2e1d9e97ce4b75ce3a7349b9c74036
Partially-Implements: blueprint performance-improvements

In [1] only neutron-openvswitch-agent was fixed and not xenapi.
That merged in Ussuri and went cleanly into Train.
In Stein and Rocky, the backport was not clean and
accidentally fixed xenapi instead of the regular one.

Neither the original bug nor its incomplete fix were released,
except for Rocky. :-(
Hence this patch also removes the confusing reno instead of
adding a new one.

[1] https://review.opendev.org/713129

Change-Id: I331417c8d61ba6f180bcafa943be697418326645
Closes-bug: #1869832
Related-bug: #1867506

Not everyone wants Kafka data stored on a Docker volume. This
change allows a user to flexibly control where the data is stored.

Change-Id: I2ba8c7a85c7bf2564f954a43c6e6dbb3257fe902

keystone and keystone_fernet container name variable is fixed
in some places, but in the defaults directory, keystone
and keystone_fernet container_name variable is variable.
If the keystone and keystone_fernet container_name variable is
changed during deployment, it will not be assigned to keystone
and keystone_fernet, but a fixed 'keystone' and 'keystone_fernet' name.

Change-Id: Ifc8ac69e6abc4586f0e4fd820b9022aea9f76396

kolla-toolbox container name variable is fixed in some places,
but in the defaults directory, kolla-toolbox container_name variable
is variable. If the kolla-toolbox container_name variable is changed
during deployment, it will not be assigned to kolla-toolbox,
but a fixed 'kolla-toolbox' name.

Change-Id: I9579017761ff47477dba597282be9ae6fab4242a

This patch fix creating statck resource failure in heat.

Change-Id: I00c23f8b89765e266d045cc463ce4d863d0d6089
Closes-Bug: #1869137

Change-Id: I9395ae32378f4ff1fd57be78d7daec7745579e04
Closes-Bug: #1869133

Deploy HAProxy on one or more servers. Add another server to the
inventory in the haproxy group, and run the following:

kolla-ansible prechecks --limit <new host>

The following task will fail:

    TASK [haproxy : Checking if kolla_internal_vip_address and
    kolla_external_vip_address are not pingable from any node]

This happens because ansible does not execute on hosts where
haproxy/keepalived is running, and therefore does not know that the VIP
should be active.

This change skips VIP prechecks when not all HAProxy hosts are in the
play.

Closes-Bug: #1868986

Change-Id: Ifbc73806b768f76f803ab01c115a9e5c2e2492ac

mariadb container name variable is fixed in some places,
but in the defaults directory, mariadb container_name variable
is variable. If the mariadb container_name variable is changed
during deployment, it will not be assigned to container_name,
but a fixed 'mariadb' name.

Change-Id: Ie8efa509953d5efa5c3073c9b550be051a7f4f9b

The 'kolla-ansible stop' command can be used to stop the services
running on hosts. However, if you run this command in an environment
with heterogeneous nodes (most real world scenarios have at least
control/compute), then it fails. This is because it only checks
whether a container is enabled, and not whether the host is in the
correct group. For example, it fails with nova-libvirt:

    No such container: nova_libvirt to stop.

This change fixes the issue by only attempting to stop containers on
hosts to which they are mapped.

Change-Id: Ibecac60d1417269bbe25a280996ca9de6e6d018f
Closes-Bug: #1868596

We released CirrOS 0.5.1. Time to move then.

Change-Id: Ibca24836f19b3cbf6166fa39a3702883938feda8

This is useful to people who manage their Prometheus Server
externally to Kolla Ansible, or want to use the exporters with
another framework such as Monasca.

Change-Id: Ie3f61e2e186c8e77e21a7b53d2bd7d2a27eee18e