Disable the additional watch timer (enable_watch_timer).
Setting this parameter to false will significantly reduce CPU and I/O
consumption when tailing a large number of files on systems with inotify
support.

Change-Id: Ib432e6cdece01dbb94b3d2f3d7fbdd04b81d25fa

Barbican-api log is a uswsgi process and fluentd expects as
openstack service process.
This change moves log type to uswgi group.

Change-Id: Ifaf5645d0d19e133e41de0c55a55321fccfc391f
Closes-Bug: #1716513

Libvirt logs are not managed by Fluentd. It can be very helpful to
debug 'booting' issue on KVM.

Change-Id: Ie04322adec0482f4d586d2f8564a1a77f2da2605
Closes-Bug: #1713429

'progname' is actually only fullfilled for Openstack services logs.
They are missing for MariaDB and RabbitMQ logs.
When using Kibana, "progname" field appears empty, hence it's difficult
to known where logs comes from.

Change-Id: Ifcbfd981a34a8d64ea69d37934c7f0fae10a90b5
Closes-Bug: #1713412

Actual Fluentd config doesn't manage multiline log pattern like:

2017-08-16 21:41:03.097 7 WARNING oslo_config.cfg [-] Option "firewall_driver" from group "DEFAULT" is deprecated for removal (
nova-network is deprecated, as are any related configuration options.
). Its value may be silently ignored in the future.

This log will generate 1 Fluentd event for each line (in previous
example 3 events).
This is really a wrong behaviour and made Kibana interface not friendly.
This patch uses a multiline parser.

Change-Id: Iee337645d1dc42876ec79be08ee3715f11a53d87
Closes-Bug: #1711399

They dropped the cloudkitty-api command line[0], so we should add wsgi
support for cloudkitty-api.

[0]https://review.openstack.org/#/c/366043/

Change-Id: Ie34d4f2d5c303bbd7ac09a8ab9e8d9bdc763c57b
Closes-Bug: #1713879

Actually a unique 'tail' source is used for all logs files in
kolla/*/*.log.

So log files from services like: chrony, elasticsearch, etcd, grafana,
influxdb, kibana, mongodb... are actually processed by our Fluentd
config, but should not cause all following config will failed to parse
them. It just uses CPU for nothing cause they will
never match the following filters:
* record_transformer => add Hostname, Logger, programname field
* rewrite_tag_filter => rewrite 'programname' TAG if needed

Hence these filters should only be applied to Openstack services logs to
avoid CPU usage.

Furthermore WGI and Apache logs files must be treated differently.

Change-Id: I425979160291e1123e3775e39243bca06ed22c2f
Closes-Bug: #1711390

Tacker has included a new conductor service
to manage mistral workflows for VIM monitoring.

Without conductor, Tacker cannot create VIMs.

This change reworks tacker to include tacker-conductor
service.

Depends-On: I52778e86e4f2c297ead8d4b09983e5e38ca88c70
Closes-Bug: #1710874
Change-Id: I6901e919887551bedc9dba8983ac904e8c48c9ce

Fluentd repo/packages are changed in depends changes below.
Rpm distros need changes in their config for properly
start and deploy fluentd containers.

Change-Id: I474c4d20c407717c6da973383bbbd5539be480f1
Depends-On: I4802618373cfa53d67d1dfe65be7abe1950b3d04

Change-Id: Ic3b994056e6287488a0a9f7dbeb140c007ad86d3

Closes-Bug :#1710842

Change-Id: Ie676fcf4acb0d08f232a8db9313212b819e500de

'@type' must be used everywhere.

Change-Id: I337f42e174393f68b43e876ef075a74c887a5314

placement-api-access.log type is apache_access

Change-Id: Ie29b9f04f2d166d04a4fee53f3749911b44a9bf1
Closes-Bug: #1709054

This patch add "domain" concept to log format.

Change-Id: I5dbe5d3953d757150c7fb7df38e4bb4ddb99e49c

Change-Id: I43fec45a49198daedddbb7c5d477690bbe3864e8

http://paste.openstack.org/show/617484/

Change-Id: I1c771b8012c12300adb611b85898bbf2ef2fe67f
Closes-Bug: #1708449

for example: the placement-api regular match
will override placement-api-access

Change-Id: I80ed896b2477709599be0ce73e00d03da1858349
Closes-Bug: #1708468

kolla-kubernetes is using its own configuration generation[0], so it is
time for kolla-ansible to remove the related code to simplify the
logical.

[0] https://github.com/openstack/kolla-kubernetes/tree/master/ansible

Change-Id: I7bb0b7fe3b8eea906613e936d5e9d19f4f2e80bb
Implements: blueprint clean-k8s-config

* remove ceilometer-api and ceilometer-collector service
* use ceilometer-notification to publish message to proper backend
* remove useless ceilometer_database_type and ceilometer_event_type
  variables
* sync event_definitions.yaml, event_pipeline.yaml and pipeline.yaml
  file with upstream

Change-Id: Ib39053cb5f70bd11ee61d3f26d5b28accecd7190

Zun-api is using app.wsgi as log filename.
This change forzes zun-api.log to ease identification.
Also changes group owner of log dir to kolla group.
Remove glance version as is in zun's defaults
Set state_path, this is where zun will store catched images,
similar as nova cache. No need HA or volume as is only
cached data.

Change-Id: I20f8d311bfde1c0a07d7d35e9f1412c71efa8504

change api.log and registry.log to glance-api.log and glance-registry.log

Closes-bug: #1700718

Change-Id: Ifcde8699fa9537fa06445f79c4bd14b4ee0df32c

Missing config options in kuryr.conf
Missing kuryr logging
Remove useless bootstrap task, justs create a directory
Wrong user permissions in config.json file.
Missing volumes
Missing container capabilities

Depends-On: I4c08a21df263fdefe1fe991cb7ad41cfee65019d
Change-Id: Id4577a78ebf3f1cda5ee36d14e2cc017f38e0f07
Closes-bug: #1697480

Zun rule number causes conflict with rule 24,
it should be updated to the next free number.

Change-Id: Idbe346bda4ca5dc50f6b5e63818b12af33d532a3

Change-Id: I929587720c622833764c61543cac7675360ccdcb

Useful api_interface_address variable has been define here:
https://github.com/openstack/kolla-ansible/blob/master/ansible/group_vars/all.yml#L57
In order to simplify codebase we must use it as much as possible.

Change-Id: I18fec19bf69e05a22a4142a9cd1165eccd022455

This changes moves cinder-api to make use of apache to
launch its processes.
Ubuntu binary removed support in Ocata for cinder-api
system services [1]. At this moment cinder in Ocata branch
is not working at all due this issue.

This changes aims to fix cinder in master only for ubuntu
binary with the purpose of make a clean backport to stable/ocata
without do any other change on the backport process.
A follow up change will migrate other OS/install_type to make use
of Apache but only changing master branch.

[1] http://lists.openstack.org/pipermail/openstack-dev/2017-January/111069.html

Change-Id: I9885d6acadbac1da46844f583e7cd37785505ae0
Closes-Bug: #1676751

As the logrotate will create new filename with timestamp, fluentd
with symlink will staled when restart fluentd and can not set
permission on the old file.

Remove symlink from the global conf.

Change-Id: I404868d5b5077b9f2135e37605421bc84717f9d3
Closes-Bug: #1685170

When Kibana interface is opened for the first time, it requires creating
a default index pattern. To view, analyse and search logs, at least one
index pattern has to be created.
This patch automate "flog-*" default index pattern creation and set it
has default config.

Partially-implements: blueprint default-kibana-dashboard
Change-Id: Ie36696f9ad38ba7e49e65e0793a3b98d9f03ee8d

Issue seems not fixed, it requires more work

This reverts commit 4f6470bb.

Change-Id: I308ea4e0d5ace60cf730944e1883966fc6f8f8ce

Fluentd actually parse all files in /var/log/kolla/*/*.log as Openstack
services files.
Dnsmasq file /var/log/kolla/neutron/dnsmasq.log, must be add to
exclude_path to avoid this warning message:
"/var/log/kolla/neutron/dnsmasq.log unreadable. It is excluded and would
be examined next time."

Change-Id: I8dc5320b9ed299200c4f1e6ee52d34a918e2f110
Closes-Bug: #1678530

Haproxy binds the elasticsearch service to kolla_internal_vip_address but
the output templates for fluentd (td-agent) point to a non-existent
kolla_external_vip_address.

Output should also be able to be sent to an external elasticsearch
instance (as per the documentation regarding overriding
elasticsearch_address)

Change these settings so that fluentd outputs to either
the default elasticsearch_address (i.e. kolla_internal_vip_address) or to
the external elasticsearch instance.

Closes-Bug: #1673990
Change-Id: I081533ae8ea9aad186e9c44e1dee069729931453

Fluent rewrite_tag_filter output plugin must be updated for the
following services:

* Congress
* Designate
* Freezer
* Manila
* Murano
* Octavia
* Senlin
* Trove
* Watcher

Change-Id: I5329002c99f5c940921dff107f2120bafb158168
Closes-Bug: #1673208

Nova placement api, cinder api, volume logs along with mistral logs
are missing from the filter which will allow elastic search to fetch it
from the fluentd

Fixes the issue

Change-Id: I7899ce6e010bfc6534a88f0a2117dc0a9e60c7c7
Closes-Bug: #1673002

Change-Id: Id7be86955b357199fde2ab57a9b24e1f49ecebcd
Closes-Bug: #1667671

Change-Id: I93fe8141a88d6e0600a1f44fe49d96d7816fae87
Closes-Bug: #1664868

Depends-On: 810fcdd28669bbd317c290d498d590ddb2c32f65
Change-Id: I88eff57448cc29ddff0f301f210623b2309e6b4d
Closes-Bug: #1663126

This PS fixes some bugs that causes trunk not to work for
kolla-kubernetes.

Change-Id: I4eaba87c9c75a85ac16823a77ab197e62254e66e

* add fluentd role
* remove heka configure

Co-Authored-By: yangzb09 <yangzb09@qq.com>

Partially-Implements: blueprint add-fluentd-role

Change-Id: Ica804a99f5bb8b157f406299c5982b7b6283b3e3