Default user group should be set much earlier in deployment
and should be used consistently accross all projects.

Change-Id: Id399f9ddebc903bb9c3eeb5a0ff6f33ca6d6828c
Closes-Bug: #1650501

The task for keystone is missing a 'per service' entry for it's config
augments. This means for example that users could not add:

/etc/kolla/config/keystone/keystone-fernet.conf

or

/etc/kolla/config/keystone/keystone.conf

and have keystone.conf augmented for those services only.

Change-Id: I8d2570b4a52dc6c3552397b0a6fa7866133dc2f1
Closes-Bug: #1646898

Include custom policy.json files in service-api.json.j2 files

Change-Id: Ic55bfc6f61131aa72c3497ce8b2282056bcc7f92
Partially-Implements: blueprint custom-policies

Currently, policy.json is put in
"{{ node_config_directory }}/{{ service_name }}"
in target nodes.

Relocation policy.json to "{{ node_config_directory }}/{{ item }}"
with item is corresponding service compoment config directory.

Currently, the policy.json is copied to all services, but it
should be reviewed and left only in neccesary service
(at many cases, only API service needs that).

Redundant files will be removed in follow up patchset.

Change-Id: I0e997dccf4ec438c9c0436db71ec2fd06650f50d
Closes-Bug: #1639686

Keystone uses fernet as default provider in its code now. This patch
adds provider=token in keystone.conf file explicitly.

TrivialFix

Change-Id: Id7142ff4f00ee99579ad420573eafefea0f4dcb7

* Merge prechecks.yml and site.yml playbook
* Create empty precheck.yml into all roles.

Change-Id: I8a138558a26c0a2a66c5fd48ed37be657c99c1dd
Implements: blueprint condition-pre-check

Allow operators to use their custom policy files.
Avoid maintain policy files in kolla repos, only copying
the files when an operator add their custom config.

Implements: blueprint custom-policies
Change-Id: Icf3c961b87cbc7a1f1dd2ffbfffcf271d151d862

Previous version of keystone reconfigure fails at adding fernet
components to fact variable "keystone_item".
Ansible can not create fact variable using newly created variable
in a same task.

This patch set split this task into two tasks, first create variable
containing fernet components, then merge it to "keystone_item".

Change-Id: I15571ab20d6104d30350e8b922401b462336fca0
Closes-Bug: #1636047

TrivialFix

Change-Id: I817aa52caf56b7d54b266e553db6db1ceb38b773
Signed-off-by: Jeffrey Zhang <zhang.lei.fly@gmail.com>

Change-Id: I211d9f2dbdd9647ec99e8b43be6332c8ad64114e
Partial-Bug: #1631503

* install openssh client in keystone-fernet container
* install rsync in keystone-ssh container
* fix syntax issue in ssh configuration
* copy ssh configuration into keystone-fernet container
* copy id_rsa.pub into keystone-ssh container
* copy id_rsa into keystone-fernet container
* use full path to ssh binary in used scripts
* add missing newlines at EOF
* when using type source set /var/lib/keystone as home
  directory for the user keystone

Co-Authored-By: Jeffrey Zhang <jeffrey.zhang@99cloud.net>
Change-Id: Id6b41030056a69f6516a054beb2fc0e08226e876
Closes-bug: #1623013

TrivialFix

Change-Id: I2f0945c444016227a09486d62cec53db51ac1ec9

By default CADF events added even if they are disbaled in all.yml.
Boolean check is missing is added so that CADF configurations will
be added only if it is enabled.

Change-Id: I757ae176228cc4e74d06ce85b27200bdcdd5dd5c
Closes-Bug: #1607904

TrivialFix

Change-Id: I23d74821c7f65cdf20c214f7622f4df0d3c0e172

do_reconfigure.yml is introduced to use serial directive. But we use
it in wrong. Now serial has moved to playbook file. So it is time to
remove the do_reconfigure.yml file

Closes-Bug: #1628152
Change-Id: I8d42d27e6bc302a0e575b0353956eaef9b2ca9fd

The use of the admin_token_auth middleware presents a security risk
and was removed from [pipeline:api_v3], [pipeline:admin_api],
and [pipeline:public_api].

Change-Id: I3a3ca2e74c0ae341105d3481f97956c6da473046
Closes-bug: #1587747

* use template for transport_url that is working for kolla-kubernetes
* use transport_url parameter in DEFAULT section

Co-authored-by: Christian Berendt <berendt@betacloud-solutions.de>

Change-Id: I800b5bc4c178b5df4f208ee50944e2806fd6a7c1
Closes-bug: #1625352

keystone-fernet container need keystone.conf file

TrivialFix

Change-Id: If4a75a4d9e88350963c31190bc09cee4862aa34d

Closes-Bug: #1624598
Change-Id: If66d82c782e6c8255b94cb00cdf0893630db596c

Change-Id: I0a40867ae65011fbfd53308b9d7f42d10435f5fa
Closes-bug: #1607905

Useful for upgrade etc., which is preferablly done serially.

Example usage: tools/kolla-ansible deploy OR tools/kolla-ansible upgrade

Closes-Bug: #1576708

DocImpact

Change-Id: I34b2e16f8ce53e472a4682a4738c4ac0f5abf00c

This addresses the ansible aspects of fernet key bootstrapping as
well as distributed key rotation.

- Bootstrapping is handled in the same way as keystone bootstrap.
- A new keystone-fernet and keystone-ssh container is created to allow
  the nodes to communicate with each other (taken from nova-ssh).
- The keystone-fernet is a keystone container with crontab installed.
  This will handle key rotations through keystone-manage and trigger
  an rsync to push new tokens to other nodes.
- Key rotation is setup to be balanced across the keystone nodes using
  a round-robbin style. This ensures that any node failures will not
  stop the keys from rotating. This is configured by a desired token
  expiration time which then determines the cron scheduling for each
  node as well as the number of fernet tokens in rotation.
- Ability for recovered node to resync with the cluster. When a node
  starts it will run sanity checks to ensure that its fernet tokens
  are not stale. If they are it will rsync with other nodes to ensure
  its tokens are up to date.

The Docker component is implemented in:
  https://review.openstack.org/#/c/349366

Change-Id: I15052c25a1d1149d364236f10ced2e2346119738
Implements: blueprint keystone-fernet-token

Note: This should not result in any behavior changes in regular Kolla,
just Kolla-Kubernetes and only when you've overridden stuff in globals.yml

Allows override of interface address and memcached pools, so that
Kubernetes can do the right thing.

There are some significant architectural issues involved in
memcached pooling in the Kolla-kubernetes world.  Avoiding them right
now.

Current working Kolla-Kubernetes globals.yml file, assuming that your
memcached servers are available under the DNS alias "memcached":

api_interface_address: "0.0.0.0"

memcached_servers: "memcached"

keystone_database_address: "mariadb"
keystone_admin_url: "{{ admin_protocol }}://keystone-admin:{{ keystone_admin_port }}/v3"
keystone_internal_url: "{{ internal_protocol }}://keystone-public:{{ keystone_public_port }}/v3"
keystone_public_url: "{{ public_protocol }}://keystone-public:{{ keystone_public_port }}/v3"

Co-authored-by: Ryan Hallisey <rhallise@redhat.com>
Change-Id: I5126f81da7b4d48001b87f73d58bbbfad658209c
Partially-implements: blueprint api-interface-bind-address-override

Migrate to full variable syntax in with_ loop
instead of bare variables for:
- ceph
- ceilometer
- glance
- heat
- horizon
- keystone
- mariadb
- memcached
- neutron
- nova
- rabbitmq

Change-Id: Ib925b4ecea47ac758a77739c9c7e140f06933b08
Partial-Implements: blueprint ansible2

Ansible's template action supports replacing keystone's wsgi default
config with custom config, it should only add with_first_found param
to config.yml to support this.

Change-Id: Id66302802db9a57188067982ea697f16faa1d8eb
Closes-Bug: #1609655

Set permission for /etc/keystone/domains/ directory to 0700 instead of
0600 which is required for Keystone to be able to list directory
contents.

Change-Id: I7536cc3f0bcbc8aea298ad76713e45cbb93a4b27
Closes-Bug: #1609319

Note: This should not result in any behavior changes in regular Kolla, just
Kolla-Kubernetes and only when you've overridden stuff in globals.yml

Allows override of interface address and memcached pools, so that Kubernetes
can do the right thing.

There are some significant architectural issues involved in memcached pooling
in the Kolla-kubernetes world.  Avoiding them right now.

Current working with this Kolla-Kubernetes globals.yml file:

api_interface_address: "0.0.0.0"

memcached_servers: "memcached"

keystone_database_address: "mariadb"
keystone_admin_url: "http://keystone-admin:35357/v3"
keystone_internal_url: "http://keystone-public:5000/v3"
keystone_public_url: "http://keystone-public:5000/v3

"

Three tings to note:
 * In Kolla-Kubernetes, the service is not using net=host, so a
   0.0.0.0 interface address is totally OK.  That patch has been merged.
 * In Kolla-Kubernetes, the global.yml file doesn't do var substitution
   so you have to be explicit about the URLs, otherwise Keystone will
   look like it was provisioned but it won't quite be provisioned right.
 * In order to not duplicate tons of code, moved the keystone_admin_url /
   keystone_internal_url / keystone_public_url to the common defaults
   from the keystone defaults.

Co-Authored-By: Ryan Hallisey <rhallise@redhat.com>
Change-Id: I586ce1c6c3300254c4e2a398ff46645df576aeb0
Partially-implements: blueprint api-interface-bind-address-override

Copy custom policy.json into keystone container

Change-Id: I58787c3dd7adbeff47d0898c23db95f5919510d3
Closes-Bug: #1604431

Keystones openstack_auth is undefined for Keystones sanity check
closes-bug: #1601808

Change-Id: I445482ce32b924d9220e8c351363b6793558d3b4

Currently, it is not possible to make use of Keystone Domain
specific settings. Such as different domains using different
LDAP servers or SQL.

To enable for example domain ACME - domain settings would be
put into:
{{ node_custom_config }}keystone/domains/keystone.ACME.conf

Change-Id: I23620978c618dd4a3598d7cb74c3e9cf8c2394ac
Closes-Bug: #1599868
Signed-off-by: Dave Walker (Daviey) <email@daviey.com>

An operator may want to specify the location of custom config
files so that kolla can detect their location and merge
them with the default configs generated.

Partially implements: blueprint multi-project-config

Change-Id: Ibfb38d07a36dfa7fe25381adc34cc1d3cbe7d1e1

This change makes each step of the kolla deployment aware
of the port database was configured to listen on.
It defaults mariadb_port to database_port.

Change-Id: I8e85d5732015afc0a5481cb33e0b629fdfa84a1b
Closes-Bug: #1576151
DocImpact

Closes-Bug: #1577148
Change-Id: I636cefc63cf532434a41af3898b63dffa711e280

TrivialFix

Change-Id: I1f03d428c380dfdbde5ef33e7ea43cbf5e9154ce

There seems to be a bug regarding the interaction between the Red Hat
based images and docker exec, where output is missed when attaching a
tty. This can be replicated using the following:

$ docker run -d --name test centos /bin/sleep infinity
$ docker exec -t test echo hi

Repeat the second command over and over, "hi" will only be printed out
every couple of runs.

This affects the keystone register task as sometimes it will not print
it's final json result to stdout, causing ansible to fall over with a
difficult to diagnose error (despite the register having run
successfully).

Disabling the tty fixes this for me, it should not be needed in this
case regardless.

Change-Id: Ie7eb7c01c34ee3c59bd843651195fbcb7259d2c8
Closes-Bug: #1572082

Make sure that all the sevices will attempt to
connect to the database an infinite about of times.
If the database ever disappears for some reason we
want the services to try and reconnect more than just
10 times.

Closes-bug: #1505636
Change-Id: I77abbf72ce5bfd68faa451bb9a72bd2544963f4b

Proposed patch to change keystone and heka log dir
from /var/log/kolla/apache2/ to /var/log/kolla/keystone/

Closes-Bug: #1560620
Change-Id: I70c65ceba5a301cc56880313ca86f01bd35676cb

The in-process cache for keystone tokens has been deprecated due to
"incosistent results and high memory usage" with the expectation we
switch to memcached_servers if we want to stay performant.

Add memcache_servers [cache] section to the appropriate servers as the
[DEFAULT]\memcache_servers options was deprecated.

TrivialFix
Related-Id: Ied2b88c8cefe5655a88d0c2f334de04e588fa75a

Change-Id: Ic971bdddc0be3338b15924f7cc0f97d4a3ad2440

This type of per node configuration is required to support things like
availability zones for nova. As always, if this file doesnt exist it
doesnt get used so this change is safe.

TrivialFix

Change-Id: Iff8172af522c2c96e5f2c173b24a5dfd4d522ed2

TLS can be used to encrypt and authenticate the connection with
OpenStack endpoints.  This patch provides the necessary
parameters and changes the resulting service configurations to
enable TLS for the Kolla deployed OpenStack cloud.

The new input parameters are:

kolla_enable_tls_external: "yes" or "no" (default is "no")
kolla_external_fqdn_cert: "/etc/kolla/certificates/haproxy.pem"
kolla_external_fqdn_cacert: "/etc/kolla/certificates/haproxy-ca.crt"

Implements: blueprint kolla-ssl

Change-Id: I48ef8a781c3035d58817f9bf6f36d59a488bab41