CI: validate prometheus configs

check if generated prometheus config is valid
via promtool.

This should help prevent bugs like:
https://bugs.launchpad.net/kolla-ansible/+bug/2076660

prior art: haproxy config validation:
https://review.opendev.org/c/openstack/kolla-ansible/+/922840



also add some basic documentation for the
`kolla-ansible validate-config` command.

Signed-off-by: Sven Kieske <kieske@osism.tech>
Change-Id: Ief90861b2c422e0e6c2dd9cb605c94e86c0f2ba1

ansible/roles/prometheus/tasks/config_validate.yml

 ---
+- name: Validating prometheus config files
+  vars:
+    service: "{{ prometheus_services['prometheus-server'] }}"
+  shell: >-
+    {{ kolla_container_engine }} exec -i {{ service.container_name }} bash -c
+    "/opt/prometheus/promtool check config /etc/prometheus/prometheus.yml;
+    /opt/prometheus/promtool check web-config /etc/prometheus/web.yml"
+  register: "prometheus_config_validation_results"
+  check_mode: false
+  become: true
+  when:
+    - inventory_hostname in groups[service.group]
+    - service.enabled | bool
+
+- name: Assert prometheus config files are valid
+  vars:
+    service: "{{ prometheus_services['prometheus-server'] }}"
+  assert:
+    that: "prometheus_config_validation_results.rc == 0"
+    fail_msg: >-
+      "{{ service.container_name }} config files are invalid, the output was:
+       {{ prometheus_config_validation_results.stdout }}"
+    success_msg: >-
+      "{{ service.container_name }} config files are valid"
+  when:
+    - inventory_hostname in groups[service.group]
+    - service.enabled | bool

doc/source/user/multinode.rst

@@ -177,3 +177,19 @@ Run the deployment:
 
    kolla-ansible deploy -i <path/to/multinode/inventory/file>
 
+Validate generated configuration files of enabled services:
+
+.. code-block:: console
+
+   kolla-ansible validate-config -i <path/to/multinode/inventory/file>
+
+.. note::
+
+   Due to the nature of the configuration generation the validation can
+   currently only be done after the first deployment. For some validations
+   it is necessary to access the running containers.
+   The validation tasks can be found - and altered - in each ansible role under
+   ``kolla-ansible/ansible/roles/$role/tasks/config_validate.yml``.
+   The validation for most openstack services is done by the special role:
+   ``service-config-validate``.
+

releasenotes/notes/validate_prometheus_config-45b4bd5e651e61cd.yaml

+---
+
+features:
+  - |
+    kolla-ansible now validates the Prometheus configuration files when
+    called via ``kolla-ansible -i $inventory validate-config``. This
+    validation is done by running the ``promtool check config`` command.
+    See the `documentation <https://docs.openstack.org/kolla-ansible/latest/user/multinode.html>`__
+    for the ``kolla-ansible validate-config`` command for details.