Newer
Older
.. _prometheus-guide:
=====================================================
Prometheus - Monitoring System & Time Series Database
=====================================================
Overview
~~~~~~~~
Kolla can deploy a full working Prometheus setup in either a **all-in-one** or
**multinode** setup.
Preparation and deployment
~~~~~~~~~~~~~~~~~~~~~~~~~~
To enable Prometheus, modify the configuration file ``/etc/kolla/globals.yml``
and change the following:
.. code-block:: yaml
enable_prometheus: "yes"
Note: This will deploy Prometheus version 2.x. Any potentially existing
Prometheus 1.x instances deployed by previous Kolla Ansible releases will
conflict with current version and should be manually stopped and/or removed.
If you would like to stay with version 1.x, set the ``enable_prometheus``
variable to ``no``.
In order to remove leftover volume containing Prometheus 1.x data, execute:
.. code-block:: console
docker volume rm prometheus
on all hosts wherever Prometheus was previously deployed.
Extending the default command line options
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
It is possible to extend the default command line options for Prometheus by
using a custom variable. As an example, to set query timeout to 1 minute
and data retention size to 30 gigabytes:
.. code-block:: yaml
prometheus_cmdline_extras: "--query.timeout=1m --storage.tsdb.retention.size=30GB"
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
Extending prometheus.cfg
~~~~~~~~~~~~~~~~~~~~~~~~
If you want to add extra targets to scrape, you can extend the default
``prometheus.yml`` config file by placing additional configs in
``{{ node_custom_config }}/prometheus/prometheus.yml.d``. These should have the
same format as ``prometheus.yml``. These additional configs are merged so
that any list items are extended. For example, if using the default value for
``node_custom_config``, you could add additional targets to scape by defining
``/etc/kolla/config/prometheus/prometheus.yml.d/10-custom.yml`` containing the
following:
.. code-block:: jinja
scrape_configs:
- job_name: custom
static_configs:
- targets:
- '10.0.0.111:1234'
- job_name: custom-template
static_configs:
- targets:
{% for host in groups['prometheus'] %}
- '{{ hostvars[host]['ansible_' + hostvars[host]['api_interface']]['ipv4']['address'] }}:{{ 3456 }}'
{% endfor %}
The jobs, ``custom``, and ``custom_template`` would be appended to the default
list of ``scrape_configs`` in the final ``prometheus.yml``. To customize on a per
host basis, files can also be placed in
``{{ node_custom_config }}/prometheus/<inventory_hostname>/prometheus.yml.d``
where, ``inventory_hostname`` is one of the hosts in your inventory. These
will be merged with any files in ``{{ node_custom_config }}/prometheus/prometheus.yml.d``,
so in order to override a list value instead of extending it, you will need to make
sure that no files in ``{{ node_custom_config }}/prometheus/prometheus.yml.d``
set a key with an equivalent hierarchical path.
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
Extra files
~~~~~~~~~~~
Sometimes it is necessary to reference additional files from within
``prometheus.yml``, for example, when defining file service discovery
configuration. To enable you to do this, kolla-ansible will resursively
discover any files in ``{{ node_custom_config }}/prometheus/extras`` and
template them. The templated output is then copied to
``/etc/prometheus/extras`` within the container on startup. For example to
configure `ipmi_exporter <https://github.com/soundcloud/ipmi_exporter>`_, using
the default value for ``node_custom_config``, you could create the following
files:
- ``/etc/kolla/config/prometheus/prometheus.yml.d/ipmi-exporter.yml``:
.. code-block:: jinja
---
scrape_configs:
- job_name: ipmi
params:
module: ["default"]
scrape_interval: 1m
scrape_timeout: 30s
metrics_path: /ipmi
scheme: http
file_sd_configs:
- files:
- /etc/prometheus/extras/file_sd/ipmi-exporter-targets.yml
refresh_interval: 5m
relabel_configs:
- source_labels: [__address__]
separator: ;
regex: (.*)
target_label: __param_target
replacement: ${1}
action: replace
- source_labels: [__param_target]
separator: ;
regex: (.*)
target_label: instance
replacement: ${1}
action: replace
- separator: ;
regex: .*
target_label: __address__
replacement: "{{ ipmi_exporter_listen_address }}:9290"
action: replace
where ``ipmi_exporter_listen_address`` is a variable containing the IP address of
the node where the exporter is running.
- ``/etc/kolla/config/prometheus/extras/file_sd/ipmi-exporter-targets.yml``:
.. code-block:: yaml
---
- targets:
- 192.168.1.1
labels:
job: ipmi_exporter