Newer
Older
.. _prometheus-guide:
=====================================================
Prometheus - Monitoring System & Time Series Database
=====================================================
Overview
~~~~~~~~
Kolla can deploy a full working Prometheus setup in either a **all-in-one** or
**multinode** setup.
Preparation and deployment
~~~~~~~~~~~~~~~~~~~~~~~~~~
To enable Prometheus, modify the configuration file ``/etc/kolla/globals.yml``
and change the following:
.. code-block:: yaml
enable_prometheus: "yes"
Note: This will deploy Prometheus version 2.x. Any potentially existing
Prometheus 1.x instances deployed by previous Kolla Ansible releases will
conflict with current version and should be manually stopped and/or removed.
If you would like to stay with version 1.x, set the ``enable_prometheus``
variable to ``no``.
In order to remove leftover volume containing Prometheus 1.x data, execute:
.. code-block:: console
docker volume rm prometheus
on all hosts wherever Prometheus was previously deployed.
Extending the default command line options
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
It is possible to extend the default command line options for Prometheus by
using a custom variable. As an example, to set query timeout to 1 minute
and data retention size to 30 gigabytes:
.. code-block:: yaml
prometheus_cmdline_extras: "--query.timeout=1m --storage.tsdb.retention.size=30GB"
Configuration options
~~~~~~~~~~~~~~~~~~~~~
.. list-table:: Configuration options
:widths: 25 25 75
:header-rows: 1
* - Option
- Default
- Description
* - prometheus_scrape_interval
- 60s
- Default scape interval for all jobs
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
Extending prometheus.cfg
~~~~~~~~~~~~~~~~~~~~~~~~
If you want to add extra targets to scrape, you can extend the default
``prometheus.yml`` config file by placing additional configs in
``{{ node_custom_config }}/prometheus/prometheus.yml.d``. These should have the
same format as ``prometheus.yml``. These additional configs are merged so
that any list items are extended. For example, if using the default value for
``node_custom_config``, you could add additional targets to scape by defining
``/etc/kolla/config/prometheus/prometheus.yml.d/10-custom.yml`` containing the
following:
.. code-block:: jinja
scrape_configs:
- job_name: custom
static_configs:
- targets:
- '10.0.0.111:1234'
- job_name: custom-template
static_configs:
- targets:
{% for host in groups['prometheus'] %}
- '{{ hostvars[host]['ansible_' + hostvars[host]['api_interface']]['ipv4']['address'] }}:{{ 3456 }}'
{% endfor %}
The jobs, ``custom``, and ``custom_template`` would be appended to the default
list of ``scrape_configs`` in the final ``prometheus.yml``. To customize on a per
host basis, files can also be placed in
``{{ node_custom_config }}/prometheus/<inventory_hostname>/prometheus.yml.d``
where, ``inventory_hostname`` is one of the hosts in your inventory. These
will be merged with any files in ``{{ node_custom_config }}/prometheus/prometheus.yml.d``,
so in order to override a list value instead of extending it, you will need to make
sure that no files in ``{{ node_custom_config }}/prometheus/prometheus.yml.d``
set a key with an equivalent hierarchical path.
Extra files
~~~~~~~~~~~
Sometimes it is necessary to reference additional files from within
``prometheus.yml``, for example, when defining file service discovery
configuration. To enable you to do this, kolla-ansible will resursively
discover any files in ``{{ node_custom_config }}/prometheus/extras`` and
template them. The templated output is then copied to
``/etc/prometheus/extras`` within the container on startup. For example to
configure `ipmi_exporter <https://github.com/soundcloud/ipmi_exporter>`_, using
the default value for ``node_custom_config``, you could create the following
files:
- ``/etc/kolla/config/prometheus/prometheus.yml.d/ipmi-exporter.yml``:
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
.. code-block:: jinja
---
scrape_configs:
- job_name: ipmi
params:
module: ["default"]
scrape_interval: 1m
scrape_timeout: 30s
metrics_path: /ipmi
scheme: http
file_sd_configs:
- files:
- /etc/prometheus/extras/file_sd/ipmi-exporter-targets.yml
refresh_interval: 5m
relabel_configs:
- source_labels: [__address__]
separator: ;
regex: (.*)
target_label: __param_target
replacement: ${1}
action: replace
- source_labels: [__param_target]
separator: ;
regex: (.*)
target_label: instance
replacement: ${1}
action: replace
- separator: ;
regex: .*
target_label: __address__
replacement: "{{ ipmi_exporter_listen_address }}:9290"
action: replace
where ``ipmi_exporter_listen_address`` is a variable containing the IP address of
the node where the exporter is running.
- ``/etc/kolla/config/prometheus/extras/file_sd/ipmi-exporter-targets.yml``:
.. code-block:: yaml
---
- targets:
- 192.168.1.1
labels:
job: ipmi_exporter