Newer
Older
.. _prometheus-guide:
=====================================================
Prometheus - Monitoring System & Time Series Database
=====================================================
Overview
~~~~~~~~
Kolla can deploy a full working Prometheus setup in either a **all-in-one** or
**multinode** setup.
Preparation and deployment
~~~~~~~~~~~~~~~~~~~~~~~~~~
To enable Prometheus, modify the configuration file ``/etc/kolla/globals.yml``
and change the following:
.. code-block:: yaml
enable_prometheus: "yes"
This will, by default, deploy Prometheus version 2.x. Since Prometheus 1.x data
is not compatible with Prometheus 2.x and no automatic data migration is
provided, any previous Prometheus 1.x deployment will be replaced and all its
stored metrics will become inacessible (but still available in the old data
volume: ``prometheus``; the new data volume defaults to ``prometheus_v2``).
If you rely on Prometheus only as e.g. a source of alert notifications (in pair
with Alertmanager), it might not be worth migrating old metrics and they could
be discarded. Otherwise, it's either possible to use
`remote storage <https://prometheus.io/docs/prometheus/latest/storage/#remote-storage-integrations>`_
or scrape Kolla's Prometheus ``/federate`` endpoint with an external system.
However, if you want to stay on 1.x series, set the following variable:
.. code-block:: yaml
prometheus_use_v1: yes
In order to remove leftover volume containing Prometheus 1.x data, execute:
.. code-block:: console
docker volume rm prometheus
on all hosts wherever Prometheus was previously deployed.
Extending the default command line options
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
It is possible to extend the default command line options for Prometheus by
using a custom variable. As an example, to set remote timeouts to 30 seconds
and data retention period to 2 days:
.. code-block:: yaml
prometheus_cmdline_extras: "-storage.remote.timeout 30s -storage.local.retention 48h"
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
Extending prometheus.cfg
~~~~~~~~~~~~~~~~~~~~~~~~
If you want to add extra targets to scrape, you can extend the default
``prometheus.yml`` config file by placing additional configs in
``{{ node_custom_config }}/prometheus/prometheus.yml.d``. These should have the
same format as ``prometheus.yml``. These additional configs are merged so
that any list items are extended. For example, if using the default value for
``node_custom_config``, you could add additional targets to scape by defining
``/etc/kolla/config/prometheus/prometheus.yml.d/10-custom.yml`` containing the
following:
.. code-block:: jinja
scrape_configs:
- job_name: custom
static_configs:
- targets:
- '10.0.0.111:1234'
- job_name: custom-template
static_configs:
- targets:
{% for host in groups['prometheus'] %}
- '{{ hostvars[host]['ansible_' + hostvars[host]['api_interface']]['ipv4']['address'] }}:{{ 3456 }}'
{% endfor %}
The jobs, ``custom``, and ``custom_template`` would be appended to the default
list of ``scrape_configs`` in the final ``prometheus.yml``. To customize on a per
host basis, files can also be placed in
``{{ node_custom_config }}/prometheus/<inventory_hostname>/prometheus.yml.d``
where, ``inventory_hostname`` is one of the hosts in your inventory. These
will be merged with any files in ``{{ node_custom_config }}/prometheus/prometheus.yml.d``,
so in order to override a list value instead of extending it, you will need to make
sure that no files in ``{{ node_custom_config }}/prometheus/prometheus.yml.d``
set a key with an equivalent hierarchical path.
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
Extra files
~~~~~~~~~~~
Sometimes it is necessary to reference additional files from within
``prometheus.yml``, for example, when defining file service discovery
configuration. To enable you to do this, kolla-ansible will resursively
discover any files in ``{{ node_custom_config }}/prometheus/extras`` and
template them. The templated output is then copied to
``/etc/prometheus/extras`` within the container on startup. For example to
configure `ipmi_exporter <https://github.com/soundcloud/ipmi_exporter>`_, using
the default value for ``node_custom_config``, you could create the following
files:
- ``/etc/kolla/config/prometheus/prometheus.yml.d/ipmi-exporter.yml``:
.. code-block:: jinja
---
scrape_configs:
- job_name: ipmi
params:
module: ["default"]
scrape_interval: 1m
scrape_timeout: 30s
metrics_path: /ipmi
scheme: http
file_sd_configs:
- files:
- /etc/prometheus/extras/file_sd/ipmi-exporter-targets.yml
refresh_interval: 5m
relabel_configs:
- source_labels: [__address__]
separator: ;
regex: (.*)
target_label: __param_target
replacement: ${1}
action: replace
- source_labels: [__param_target]
separator: ;
regex: (.*)
target_label: instance
replacement: ${1}
action: replace
- separator: ;
regex: .*
target_label: __address__
replacement: "{{ ipmi_exporter_listen_address }}:9290"
action: replace
where ``ipmi_exporter_listen_address`` is a variable containing the IP address of
the node where the exporter is running.
- ``/etc/kolla/config/prometheus/extras/file_sd/ipmi-exporter-targets.yml``:
.. code-block:: yaml
---
- targets:
- 192.168.1.1
labels:
job: ipmi_exporter