tls.rst

===
TLS
===

This guide describes how to configure Kolla Ansible to deploy OpenStack with
TLS enabled. Enabling TLS on the provided internal and/or external VIP address
allows OpenStack clients to authenticate and encrypt network communication
with OpenStack services.

When an OpenStack service exposes an API endpoint, Kolla Ansible will
configure HAProxy for that service to listen on the internal and/or
external VIP address. The HAProxy container load-balances requests on
the VIPs to the nodes running the service container.

There are two different layers of TLS configuration for OpenStack APIs:

1. Enabling TLS on the internal and/or external VIP, so communication between
   an OpenStack client and the HAProxy listening on the VIP is secure.
2. Enabling TLS on the backend network, so communication between HAProxy and
   the backend API services is secure.

.. note::

  TLS authentication is based on certificates that have been signed by
  trusted Certificate Authorities. Examples of commercial CAs are Comodo,
  Symantec, GoDaddy, and GlobalSign. Letsencrypt.org is a CA that will
  provide trusted certificates at no charge. If using a trusted CA is not
  possible for your project, you can use a private CA, e.g. Hashicorp Vault,
  to create a certificate for your domain, or see
  :ref:`admin-tls-generating-a-private-ca` to use a Kolla Ansible generated
  private CA.

  For details on ACME-enabled CAs, such as letsencrypt.org, please see
  :doc:`acme`.

Quick Start
~~~~~~~~~~~

.. note::

  The certificates generated by Kolla Ansible use a simple Certificate
  Authority setup and are not suitable for a production deployment. Only
  certificates signed by a trusted Certificate Authority should be used in a
  production deployment.

To deploy OpenStack with TLS enabled for the external, internal and backend
APIs, configure the following in ``globals.yml``:

.. code-block:: yaml

  kolla_enable_tls_internal: "yes"
  kolla_enable_tls_external: "yes"
  kolla_enable_tls_backend: "yes"
  kolla_copy_ca_into_containers: "yes"

If deploying on Debian or Ubuntu:

.. code-block:: yaml

  openstack_cacert: "/etc/ssl/certs/ca-certificates.crt"

If on CentOS or Rocky:

.. code-block:: yaml

  openstack_cacert: "/etc/pki/tls/certs/ca-bundle.crt"

The Kolla Ansible ``certificates`` command generates a private test Certificate
Authority, and then uses the CA to sign the generated certificates for the
enabled VIP(s) to test TLS in your OpenStack deployment. Assuming you are
using the ``multinode`` inventory:

.. code-block:: console

  kolla-ansible -i ~/multinode certificates

TLS Configuration for internal/external VIP
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The configuration variables that control TLS for the internal and/or external
VIP are:

- ``kolla_enable_tls_external``
- ``kolla_enable_tls_internal``
- ``kolla_internal_fqdn_cert``
- ``kolla_external_fqdn_cert``

.. note::

  If TLS is enabled only on the internal or external network, then
  ``kolla_internal_vip_address`` and ``kolla_external_vip_address`` must
  be different.

  If there is only a single network configured in your topology
  (as opposed to separate internal and external networks), TLS
  can only be enabled using the internal network configuration variables.

The default state for TLS networking is disabled. To enable external TLS
encryption:

.. code-block:: yaml

  kolla_enable_tls_external: "yes"

To enable internal TLS encryption:

.. code-block:: yaml

  kolla_enable_tls_internal: "yes"

Two certificate files are required to use TLS securely with authentication,
which will be provided by your Certificate Authority:

* server certificate with private key
* CA certificate with any intermediate certificates

The combined server certificate and private key needs to be provided to Kolla
Ansible, with the path configured via ``kolla_external_fqdn_cert`` or
``kolla_internal_fqdn_cert``.  These paths default to ``{{
kolla_certificates_dir }}/haproxy.pem`` and ``{{ kolla_certificates_dir
}}/haproxy-internal.pem`` respectively, where ``kolla_certificates_dir`` is
``/etc/kolla/certificates`` by default.

If the server certificate provided is not already trusted by clients, then
the CA certificate file will need to be distributed to the clients. This is
discussed in more detail in :ref:`admin-tls-openrc` and
:ref:`admin-tls-ca-in-containers`.

.. _admin-tls-openrc:

Configuring the OpenStack Client for TLS
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The location for the CA certificate for the ``admin-openrc.sh`` file is
configured with the ``kolla_admin_openrc_cacert`` variable, which is not set by
default. This must be a valid path on all hosts where ``admin-openrc.sh`` is
used.

When TLS is enabled on a VIP, and ``kolla_admin_openrc_cacert`` is set to
``/etc/pki/tls/certs/ca-bundle.crt``, an OpenStack client will have settings
similar to this configured by ``admin-openrc.sh``:

.. code-block:: shell

  export OS_PROJECT_DOMAIN_NAME=Default
  export OS_USER_DOMAIN_NAME=Default
  export OS_PROJECT_NAME=admin
  export OS_TENANT_NAME=admin
  export OS_USERNAME=admin
  export OS_PASSWORD=demoPassword
  export OS_AUTH_URL=https://mykolla.example.net:5000
  export OS_INTERFACE=internal
  export OS_ENDPOINT_TYPE=internalURL
  export OS_MISTRAL_ENDPOINT_TYPE=internalURL
  export OS_IDENTITY_API_VERSION=3
  export OS_REGION_NAME=RegionOne
  export OS_AUTH_PLUGIN=password
  # os_cacert is optional for trusted certificates
  export OS_CACERT=/etc/pki/tls/certs/ca-bundle.crt

.. _admin-tls-ca-in-containers:

Adding CA Certificates to the Service Containers
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

To copy CA certificate files to the service containers:

.. code-block:: yaml

  kolla_copy_ca_into_containers: "yes"

When ``kolla_copy_ca_into_containers`` is configured to "yes", the CA
certificate files in ``/etc/kolla/certificates/ca`` will be copied into
service containers to enable trust for those CA certificates. This is required
for any certificates that are either self-signed or signed by a private CA,
and are not already present in the service image trust store. Kolla will
install these certificates in the container system wide trust store when the
container starts.

All certificate file names will have the ``kolla-customca-`` prefix prepended
to them when they are copied into the containers. For example, if a certificate
file is named ``internal.crt``, it will be named
``kolla-customca-internal.crt`` in the containers.

For Debian and Ubuntu containers, the certificate files will be copied to the
``/usr/local/share/ca-certificates/`` directory.

For CentOS and Rocky containers, the certificate files will be copied to the
``/etc/pki/ca-trust/source/anchors/`` directory.

In both cases, valid certificates will be added to the system trust store -
``/etc/ssl/certs/ca-certificates.crt`` on Debian and Ubuntu, and
``/etc/pki/tls/certs/ca-bundle.crt`` on CentOS and Rocky.

Configuring a CA bundle
~~~~~~~~~~~~~~~~~~~~~~~

OpenStack services do not always trust CA certificates from the system trust
store by default. To resolve this, the ``openstack_cacert`` variable should be
configured with the path to the CA Certificate in the container.

To use the system trust store on Debian or Ubuntu:

.. code-block:: yaml

   openstack_cacert: /etc/ssl/certs/ca-certificates.crt

For CentOS or Rocky:

.. code-block:: yaml

   openstack_cacert: /etc/pki/tls/certs/ca-bundle.crt

Back-end TLS Configuration
~~~~~~~~~~~~~~~~~~~~~~~~~~

Enabling TLS on the backend services secures communication between the
HAProxy listing on the internal/external VIP and the OpenStack
services. It also enables secure end-to-end communication between OpenStack
services that support TLS termination. The OpenStack services that support
backend TLS termination in Victoria are: Nova, Ironic, Neutron, Keystone,
Glance, Heat, Placement, Horizon, Barbican, and Cinder.

The configuration variables that control back-end TLS for service endpoints
are:

- ``kolla_enable_tls_backend``
- ``kolla_tls_backend_cert``
- ``kolla_tls_backend_key``
- ``haproxy_backend_cacert``
- ``haproxy_backend_cacert_dir``

The default state for back-end TLS is disabled. To enable TLS for the back-end
communication:

.. code-block:: yaml

  kolla_enable_tls_backend: "yes"

It is also possible to enable back-end TLS on a per-service basis. For example,
to enable back-end TLS for Keystone, set ``keystone_enable_tls_backend`` to
``yes``.

The default values for ``haproxy_backend_cacert`` and
``haproxy_backend_cacert_dir`` should suffice if the certificate is in the
system trust store. Otherwise, they should be configured to a location of the
CA certificate installed in the service containers.

Each backend service requires a certificate and private key. In many cases it
is necessary to use a separate certificate and key for each host, or even
per-service. The following precedence is used for the certificate:

* ``{{ kolla_certificates_dir }}/{{ inventory_hostname }}/{{ project_name
  }}-cert.pem``
* ``{{ kolla_certificates_dir }}/{{ inventory_hostname }}-cert.pem``
* ``{{ kolla_certificates_dir }}/{{ project_name }}-cert.pem``
* ``{{ kolla_tls_backend_cert }}``

And for the private key:

* ``{{ kolla_certificates_dir }}/{{ inventory_hostname }}/{{ project_name
  }}-key.pem``
* ``{{ kolla_certificates_dir }}/{{ inventory_hostname }}-key.pem``
* ``{{ kolla_certificates_dir }}/{{ project_name }}-key.pem``
* ``{{ kolla_tls_backend_key }}``

The default for ``kolla_certificates_dir`` is ``/etc/kolla/certificates``.

``kolla_tls_backend_cert`` and ``kolla_tls_backend_key``, default to ``{{
kolla_certificates_dir }}/backend-cert.pem`` and ``{{ kolla_certificates_dir
}}/backend-key.pem`` respectively.

``project_name`` is the name of the OpenStack service, e.g. ``keystone`` or
``cinder``.

.. note::
  The back-end TLS cert/key can be the same certificate that is used for the
  VIP, as long as those certificates are configured to allow requests from
  both the VIP and internal networks.

By default, the TLS certificate will be verified as trustable by the
OpenStack services. Although not recommended for production, it is possible to
disable verification of the backend certificate:

.. code-block:: yaml

  kolla_verify_tls_backend: "no"

.. _admin-tls-generating-a-private-ca:

Generating a Private Certificate Authority
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. note::

  The certificates generated by Kolla Ansible use a simple Certificate
  Authority setup and are not suitable for a production deployment. Only
  certificates signed by a trusted Certificate Authority should be used in a
  production deployment.

It's not always practical to get a certificate signed by a trusted CA. In a
development or internal test OpenStack deployment, it can be useful to
generate certificates locally to enable TLS.

For convenience, the ``kolla-ansible`` command will generate the necessary
certificate files based on the information in the ``globals.yml``
configuration file and the inventory file:

.. code-block:: console

  kolla-ansible -i multinode certificates

The ``certificates`` role performs the following actions:

1. Generates a test root Certificate Authority
2. Generates the internal/external certificates which are signed by the root
   CA.
3. If back-end TLS is enabled, generate the back-end certificate signed by
   the root CA.

The combined certificate and key file ``haproxy.pem`` (which is the default
value for ``kolla_external_fqdn_cert``) will be generated and stored in the
``/etc/kolla/certificates/`` directory, and a copy of the CA certificate
(``root.crt``) will be stored in the ``/etc/kolla/certificates/ca/`` directory.

Generating your certificates without kolla-ansible
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

If you want to manage your TLS certificates outside kolla-ansible directly on
your hosts, you can do it by setting ``kolla_externally_managed_cert`` to
``true``. This will make kolla-ansible ignore any copy of certificate from
the operator to kolla-ansible managed hosts and will keep other configuration
options for TLS as is.

If using this option, make sure that all certificates are present on the
appropriate hosts in the appropriate location.