this patchset has implemented:
  - network (lb-mgmt-net)
  - security groups and rules (used by amphora and health manager)
  - amphora flavor (used by amphora)
  - nova keypair (used by amphora at the time of debugging)

Add a octavia_amp_listen_port variable which used by amphora
Add amp_image_owner_id in octavia.conf

Implements: blueprint implement-automatic-deploy-of-octavia
Co-Authored-By: zhangchun <zhangchun@yovole.com>

Depends-On: https://review.opendev.org/652030

Change-Id: I67009d046925cfc02c1e0073c80085c1471975f6

Currently we generate multiple fluentd configuration files for inputs,
filters, formatters and outputs.
These are then included from the main td-agent.conf configuration file.
With a large number of hosts, this can take a long time to template.

Benchmarking of templating is available at [1].

This change switches to a single fluentd configuration file, with the
include done locally. For the default template files included with Kolla
Ansible we use Jinja includes, but this does not work with templates in
a different directory. We therefore use the Ansible template lookup
plugin, which has a slightly higher overhead than a jinja include, but
far lower than generating multiple templates. This should drastically
improve the performance of this task.

[1] https://github.com/stackhpc/ansible-scaling/blob/master/doc/template.md

Partially-Implements: blueprint performance-improvements

Change-Id: Ia8623be0aa861fea3e54d2c9e1c971dfd8e3afa9

The kolla-genpwd and kolla-mergepwd commands can be used to manipulate
the kolla passwords.yml file. The format is a YAML encoded dict of
password variable names to their values. If the format is not a dict,
the error messages are unhelpful.  In particular, this can happen if the
file is encrypted e.g. via Ansible Vault.

For kolla-genpwd:

    AttributeError: 'NoneType' object has no attribute 'items'

For kolla-mergepwd:

    AttributeError: 'NoneType' object has no attribute 'update'

This change adds a more friendly message.

Change-Id: I27f0835b904e05006ae401adf383090322e1b891
Closes-Bug: #1880220

This includes some lightweight refactoring to avoid code
duplication.

This patch is made to be backportable to Train.
We now include Ansible in testing since Ussuri so the comments
about the bool filter are wrong.

Change-Id: Ia2e0f7f24988763bacfeafefb7977021f5949f4e
Closes-bug: #1848941

W503 and W504 are incompatible and we need to choose one of them.
Existing codes follows W503, so we disable W504.

Change-Id: Ic745e956dd332eb0fa49b93c1e6acb12f8a7f26c

In Ibecac60d1417269bbe25a280996ca9de6e6d018f, the services in the common
role were marked as being mapped to the 'all' group, since the
'service_mapped_to_host' filter expects every service definition to have
either a 'group' or 'host_in_groups' field. While this allows the filter
to pass the common services without error, it will not actually show
them as being mapped to any hosts. This is because the filter uses the
'group_names' variable, which contains all of the groups that a host
belongs to, except the default 'all' group.

This change fixes the issue by returning True from
service_mapped_to_host when the service's group is 'all'.

Change-Id: I39c8416f5d30a535c1743f9c43434b7d2a382196
Related-Bug: #1868596

The repo is Python 3 now, so update hacking to version 3.0 which
supports Python 3.

Fix problems found by updated hacking version.

Remove hacking and friends from lower-constraints, they are not needed
during installation.

Change-Id: I7ef5ac8a89e94f5da97780198619b6facc86ecfe

Now that py2 is gone, oslotest dropped dependency on mock and will
soon affect Ussuri CI [1], let's use unittest.mock built in py3.

This also fixes py38 jobs and proactively prevents py36 and py37
failing due to [1]. This is because we never included mock in
test-requirements (but in lower-constraints where it does not
really belong at all) and instead relied on oslotest to bring
it in.

[1] https://review.opendev.org/716322

Change-Id: I30e82e2d87418272a71c7ee089a8acdaf8872158

The service_mapped_to_host filter is used to check if a service is
mapped to a host, based on the group for the service or its
host_in_groups attribute if one exists. We check if the service's group
is in the 'groups' list. However, to get the list of groups to which a
host belongs, we should use the 'group_names' list.

This filter is currently only used in neutron IPv6 module loading, so
the effects are minimal.

Change-Id: I37409ca8d273b0426df0a648db222dc5432e738a
Closes-Bug: #1868285

Kolla-Ansible Ceph deployment mechanism has been deprecated in Train [1].

This change removes the Ansible code and associated CI jobs.

[1]: https://review.opendev.org/669214

Change-Id: Ie2167f02ad2f525d3b0f553e2c047516acf55bc2

to clean old keys on merge.

Change-Id: Ifcc99e7c737707eea9e951db066dc94fd85bd9f7

This moves the Nova Cells filters alongside the service filters
for ease of testing.

Partially Implements: blueprint support-nova-cells
Change-Id: I32d35c065812c6b46c64bacdf283a0bdad0f8a0f

Introduce kolla_address filter.
Introduce put_address_in_context filter.

Add AF config to vars.

Address contexts:
- raw (default): <ADDR>
- memcache: inet6:[<ADDR>]
- url: [<ADDR>]

Other changes:

globals.yml - mention just IP in comment

prechecks/port_checks (api_intf) - kolla_address handles validation

3x interface conditional (swift configs: replication/storage)

2x interface variable definition with hostname
(haproxy listens; api intf)

1x interface variable definition with hostname with bifrost exclusion
(baremetal pre-install /etc/hosts; api intf)

neutron's ml2 'overlay_ip_version' set to 6 for IPv6 on tunnel network

basic multinode source CI job for IPv6

prechecks for rabbitmq and qdrouterd use proper NSS database now

MariaDB Galera Cluster WSREP SST mariabackup workaround
(socat and IPv6)

Ceph naming workaround in CI
TODO: probably needs documenting

RabbitMQ IPv6-only proto_dist

Ceph ms switch to IPv6 mode

Remove neutron-server ml2_type_vxlan/vxlan_group setting
as it is not used (let's avoid any confusion)
and could break setups without proper multicast routing
if it started working (also IPv4-only)

haproxy upgrade checks for slaves based on ipv6 addresses

TODO:

ovs-dpdk grabs ipv4 network address (w/ prefix len / submask)
not supported, invalid by default because neutron_external has no address
No idea whether ovs-dpdk works at all atm.

ml2 for xenapi
Xen is not supported too well.
This would require working with XenAPI facts.

rp_filter setting
This would require meddling with ip6tables (there is no sysctl param).
By default nothing is dropped.
Unlikely we really need it.

ironic dnsmasq is configured IPv4-only
dnsmasq needs DHCPv6 options and testing in vivo.

KNOWN ISSUES (beyond us):

One cannot use IPv6 address to reference the image for docker like we
currently do, see: https://github.com/moby/moby/issues/39033
(docker_registry; docker API 400 - invalid reference format)
workaround: use hostname/FQDN

RabbitMQ may fail to bind to IPv6 if hostname resolves also to IPv4.
This is due to old RabbitMQ versions available in images.
IPv4 is preferred by default and may fail in the IPv6-only scenario.
This should be no problem in real life as IPv6-only is indeed IPv6-only.
Also, when new RabbitMQ (3.7.16/3.8+) makes it into images, this will
no longer be relevant as we supply all the necessary config.
See: https://github.com/rabbitmq/rabbitmq-server/pull/1982

For reliable runs, at least Ansible 2.8 is required (2.8.5 confirmed
to work well). Older Ansible versions are known to miss IPv6 addresses
in interface facts. This may affect redeploys, reconfigures and
upgrades which run after VIP address is assigned.
See: https://github.com/ansible/ansible/issues/63227

Bifrost Train does not support IPv6 deployments.
See: https://storyboard.openstack.org/#!/story/2006689



Change-Id: Ia34e6916ea4f99e9522cd2ddde03a0a4776f7e2c
Implements: blueprint ipv6-control-plane
Signed-off-by: Radosław Piliszek <radoslaw.piliszek@gmail.com>

The method `Fernet.generate_key()` generates a binary string in Python 3:
```
>>> Fernet.generate_key()
b'qSMZlOK23pZUw_Uyy-ZRPUfPskMXKGCGmhG6AHCFiV8='
```

Unless properly written as a string to the Kolla `passwords.yml` file,
the Fernet key will end up in the final Barbican config like this:
```
[simple_crypto_plugin]
kek = b'qSMZlOK23pZUw_Uyy-ZRPUfPskMXKGCGmhG6AHCFiV8='
```

Due to the fact that the key is incorrectly written to the barbican
config file (it should be written as a string), every barbican secret
store fails with:

```
barbican.api.controllers   File "/var/lib/kolla/venv/lib/python3.6/site-packages/barbican/plugin/store_crypto.py", line 83, in store_secret
barbican.api.controllers     encrypting_plugin, context.project_model)
barbican.api.controllers   File "/var/lib/kolla/venv/lib/python3.6/site-packages/barbican/plugin/store_crypto.py", line 290, in _find_or_create_kek_objects
barbican.api.controllers     kek_meta_dto = plugin_inst.bind_kek_metadata(kek_meta_dto)
barbican.api.controllers   File "/var/lib/kolla/venv/lib/python3.6/site-packages/barbican/plugin/crypto/simple_crypto.py", line 104, in bind_kek_metadata
barbican.api.controllers     encryptor = fernet.Fernet(self.master_kek)
barbican.api.controllers   File "/var/lib/kolla/venv/lib/python3.6/site-packages/cryptography/fernet.py", line 38, in __init__
barbican.api.controllers     "Fernet key must be 32 url-safe base64-encoded bytes."
barbican.api.controllers ValueError: Fernet key must be 32 url-safe base64-encoded bytes.
```

This commit fixes the issue described above by properly writing
the Fernet key as a string to the Kolla `passwords.yml` file.

Closes-Bug: #1848191
Change-Id: I27fc0159c889bc2e1576fdd69b7d02a320b620f8

These filters can be used to capture a lot of the logic that we
currently have in 'when' statements, about which services are enabled
for a particular host.

In order to use these filters, it is necessary to install the
kolla_ansible python module, and not just the dependencies listed in
requirements.txt. The CI test and quickstart install from source
documentation has been updated accordingly.

Ansible is not currently in OpenStack global requirements, so for unit
tests we avoid a direct dependency on Ansible and provide fakes where
necessary.

Change-Id: Ib91cac3c28e2b5a834c9746b1d2236a309529556

When methods for passwords generation and merge are
extracted then external apps and scripts can use
those methods without resolving to subprocess execution
or injecting sys.argv.

Change-Id: I99aff7852180534129fa36859075306eea776ba9
Signed-off-by: Maciej Kucia <maciej@kucia.net>

Sha password is not always valid for barbican cripto key.
Use a fernet key so it always gets valid.

Not need release note for upgrade, users with a working
barbican not regenerate passwords, only new passwords will
get new type.

Change-Id: Ic8c4ca63219295d697062cff9cbf30fadbe49bf3

Due to the changes in hmac.new and how binary strings
are dumped in yaml.safe_dump some changes were needed to
make sure that we dumped only strings, not binary strings.

Change-Id: Ic2fbcf2347023c1e9e666203dfe40dbeaf24ce5f

Change-Id: I87df49939f600cfa1041193808ce6bdcf4620ffc

OSprofile allows user/devs trace OpenStack requests.

Implements: blueprint enable-osprofiler
Co-Authored-By: Bertrand Lallau <bertrand.lallau@gmail.com>
Change-Id: I82ea85d726011ef6cbf99380f395452d6d7f8053

pycrypto is no longer maintained [1]. This patch rewrites functions
using pycrypto and replaces them with the cryptography equivalent

[1] http://lists.openstack.org/pipermail/openstack-dev/2017-March/113568.html

Change-Id: I375b5876ec2f4c4f32b9f6b3f41d209a59a0f615

Remove Python specific types from YAML output
Produce only basic YAML tags

Change-Id: Ib6a4c18663897efb7243ed1ff84df1c9f2abf8bf

Link https://docs.openstack.org/project-install-guide/key-manager/newton/barbican-backend.html#simple-crypto-plugin

Change-Id: I351738c2a98090c56ac69e477fbe5ddec4cc5b26
Closes-Bug: #1672001

Booting from volume require cinder's ceph client secret now. Move cinder
before nova in site.yml, because nova depends on cinder ceph client key
now.

Change-Id: I01c9ed80843d98305b8963894c4917c21a35d3ac
Closes-Bug: #1670676

* Rename kolla namespace to kolla_ansible
* remove oslo.config.opts entry points which is uesless
* delete useless tools/version-check.py script

Change-Id: I005dd7223ff23afbb2ce8cbfd0ebec0969102798