For other architectures, fluentd is not installed using the
td-agent package.

Change-Id: I508bf16a54dcadac4b5cba03e184b4f620a2b395
Depends-On: https://review.openstack.org/638692

Vitrage Collector service has been removed from Vitrage in change:
Ie713456b2df96e24d0b15d2362a666162bfb4300.

Change-Id: I45023940c1d2573bfed49d4ce3fac16ed2d559e4
Signed-off-by: Bartosz Zurkowski <b.zurkowski@samsung.com>
Co-Authored-By: Kien Nguyen <kiennt65@viettel.com.vn>

OpenDaylight logs have different format than openstack,
is a karaf log with java error traces.
This PS add required config to make fluentd parse properly
ODL logs.

Change-Id: I34fb96c8a424679b3b618f2ff6a840b8dc165bec

This commit adds some filters which format logs so that they
can be correctly sent to the Monasca Log API by the Monasca
Fluentd plugin. In the future the Fluentd plugin could be
extended and this config could be removed.

Partially-Implements: blueprint monasca-roles
Change-Id: I87b6dfb3052d03f87349d30b66078c39d625195d

When Monasca is enabled disable direct logging to ElasticSearch and
send all logs harvested by Fluentd to the Monasca Log API.

This change also cleans up output files which may be left behind when
the various log forwarding options are enabled / disabled.

Partially-Implements: blueprint monasca-roles
Change-Id: I7197966c5117176407d60c86c08d3bcea5e8131a

There are cases when we can lost original timestamp field given from
logs, like when we send our logs to the next fluentd forwarder in chain
of forwarders, it will rewrite our timestamp by default. Save
`Timestamp` field explicitly to avoid such situation and be able to
reconstruct messages date and time.

Closes-Bug: #1781046
Change-Id: I2b4486aedacbe16dc4c0fb2e4e4984bd80e59f2d

Deploys the Monasca Log API with mod_wsgi + Apache.

Change-Id: I28f0aa31c59b0b6917be2b125b5f8a0d7a7035af
Partially-Implements: blueprint monasca-roles

Deploys the Monasca API with mod_wsgi + Apache.

Co-Authored-By: Mark Goddard <mark@stackhpc.com>

Partially-Implements: blueprint monasca-roles
Change-Id: I3e03762217fbef1fb0cbff6239abb109cbec226b

Projects which may use Kafka include Monasca and Oslo
messaging. Initially it will be possible to
deploy a single Kafka cluster which may be shared
among clients. Support for running multiple Kafka
clusters may be added in a future change. This
change also configures logging for Kafka server,
state-change and controller logs.

Partially-Implements: blueprint monasca-roles
Change-Id: Iab8d200c2450238f3c0c708d1f4184490f6e6284

This picks up the abandoned review:
https://review.openstack.org/#/c/412423



Co-Authored-By: Sam Yaple <sam@yaple.net>
Partially-Implements: blueprint monasca-roles
Change-Id: Idc01afcb125271181ee5e15d327d0929f07b49e9

Add ansible role to deploy blazar
Add nova filters to allow use of blazar

Change-Id: I6742ddc9a4736f256491dd0cfd31904fa8eb5652
Implements: blueprint blazar-ansible-role

Previously ironic inspector logs were not being collected, as they are
stored in /var/log/kolla/ironic-inspector/ rather than
/var/log/kolla/ironic/. This change modifies the fluentd input
configuration to use the correct directory.

Also adds the ironic-inspector service to the fluentd programname
rewrite rule.

Change-Id: Ie5d67f395e4c32fef992e674f2d53dbd6a17f238

Co-Authored-By: caoyuan <cao.yuan@99cloud.net>
Co-Authored-By: Eduardo Gonzalez <dabarren@gmail.com>

Depends-on I7ffb991adbd05275e331401cd73fc84362084ebd
Change-Id: I85afeb5dfce0bf6350a444000bdb885c6fe079c9
Partially-implements: blueprint vitrage-roles

- Sets the hostname on rabbitmq and mariadb log messages
  according to the local host.
- Sets the hostname on syslog log messages by extracting it
  from the syslog message. In many cases this will be the
  same as the local host, but it may also be a remote host.

Closes-Bug: #1726596
Change-Id: If7c53916d5a355b3aac33d6a62d89d02474e8b53

The current syntax of Ubuntu fluentd rewrite rules are not supported
anymore, reference to this commit[1].
According to the build of this patch[2], The Centos has no such isuue.
Only Ubuntu needs to be upgrade to use the <rule> section.

* Centos use 01-rewrite-0.12.conf.j2
* Ubuntu use 01-rewrite-0.14.conf.j2

backport: pike

[1] https://github.com/fluent/fluent-plugin-rewrite-tag-filter/commit/248ed8e97def0a60121199c0d0e2fa8785f0d076
[2] https://review.openstack.org/#/c/517907

Depends-on: I47985113fe732569d640a262ca832c6edc8c2bb6
Change-Id: I0b131c2f65652cf3c61b33d8162097b047603923
Closes-Bug: #1730664

By default the syslog input plugin won't extract the
log level from a syslog message. By setting the priority
key, the log level is extracted.

Partial-Bug: #1725240
Change-Id: I9ad5f5e1c875ef62b39fbbba7b271d98cee68dd6

Use the same field name for extracting all log messages to make it
simpler to parse the output from fluentd.

Closes-Bug: #1723459

Change-Id: I55b86061c8f70b25cf88e394fdfc78fa3c85c79f

We want to treat the nova-compute-ironic log in the same way as
the other nova logs so that it doesn't go missing from fluentd
output.

Change-Id: I145389cd0cceeed506e2a9c084d290341047c9aa
Closes-Bug: #1721814

Record transformer is used to add a 'Severity' field in event.
This Field is unused and not generated for other programs, hence
It can be removed safely.

Change-Id: I62e461e5006516dcbd6c64e1770d6936f9a9d55c

This change adds enable_fluentd option and enables some other log shippers
to be integrated. When enable_fluentd is "no", syslog server is also disabled.
Then, this change also adds syslog parameters to use a syslog server
prepared by users.

Change-Id: I7c83ef7fe30a6b9ab7385bcee953ad07e96b0a83
Implements: blueprint fluentd-enable-option

The following warning logs appears in fluentd container:

[warn]: pattern not match: "170908 21:34:12 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql/"

This patch fix it

Change-Id: I9820b8c5fd3456ba9c032fdf326c65632d147e50
Closes-Bug: #1708571

Fluentd config actually used two keywords to named log level:
"log_level" and "severity_label".
This doesn't allow to search and sort ALL logs in Kibana using a unique
field (ex: log_level).
This fix standardize log level keyword naming.

Closes-Bug: #1713410
Change-Id: Ibb2eaf3b6aa21ce6f7945e768e6362c0d40f0507

infra.rabbitmq tag does not match in fluentd

Closes-Bug: #1718874
Change-Id: I92733a4f275baa93b0c3a6ddbdc6801aec534867

progname is actually only fullfilled for Openstack logs and infra logs.
They are missing for program logging in RSYSLOG: HAPROXY and SWIFT.
When using Kibana, "progname" field appears empty, hence it's difficult
to know where logs come from.

Change-Id: Iae98af083cfb79d2a64fa0b70c86b0a01f7a336f
Closes-Bug: #1716051

Disable the additional watch timer (enable_watch_timer).
Setting this parameter to false will significantly reduce CPU and I/O
consumption when tailing a large number of files on systems with inotify
support.

Change-Id: Ib432e6cdece01dbb94b3d2f3d7fbdd04b81d25fa

Barbican-api log is a uswsgi process and fluentd expects as
openstack service process.
This change moves log type to uswgi group.

Change-Id: Ifaf5645d0d19e133e41de0c55a55321fccfc391f
Closes-Bug: #1716513

Libvirt logs are not managed by Fluentd. It can be very helpful to
debug 'booting' issue on KVM.

Change-Id: Ie04322adec0482f4d586d2f8564a1a77f2da2605
Closes-Bug: #1713429

'progname' is actually only fullfilled for Openstack services logs.
They are missing for MariaDB and RabbitMQ logs.
When using Kibana, "progname" field appears empty, hence it's difficult
to known where logs comes from.

Change-Id: Ifcbfd981a34a8d64ea69d37934c7f0fae10a90b5
Closes-Bug: #1713412

Actual Fluentd config doesn't manage multiline log pattern like:

2017-08-16 21:41:03.097 7 WARNING oslo_config.cfg [-] Option "firewall_driver" from group "DEFAULT" is deprecated for removal (
nova-network is deprecated, as are any related configuration options.
). Its value may be silently ignored in the future.

This log will generate 1 Fluentd event for each line (in previous
example 3 events).
This is really a wrong behaviour and made Kibana interface not friendly.
This patch uses a multiline parser.

Change-Id: Iee337645d1dc42876ec79be08ee3715f11a53d87
Closes-Bug: #1711399

They dropped the cloudkitty-api command line[0], so we should add wsgi
support for cloudkitty-api.

[0]https://review.openstack.org/#/c/366043/

Change-Id: Ie34d4f2d5c303bbd7ac09a8ab9e8d9bdc763c57b
Closes-Bug: #1713879

Actually a unique 'tail' source is used for all logs files in
kolla/*/*.log.

So log files from services like: chrony, elasticsearch, etcd, grafana,
influxdb, kibana, mongodb... are actually processed by our Fluentd
config, but should not cause all following config will failed to parse
them. It just uses CPU for nothing cause they will
never match the following filters:
* record_transformer => add Hostname, Logger, programname field
* rewrite_tag_filter => rewrite 'programname' TAG if needed

Hence these filters should only be applied to Openstack services logs to
avoid CPU usage.

Furthermore WGI and Apache logs files must be treated differently.

Change-Id: I425979160291e1123e3775e39243bca06ed22c2f
Closes-Bug: #1711390

Tacker has included a new conductor service
to manage mistral workflows for VIM monitoring.

Without conductor, Tacker cannot create VIMs.

This change reworks tacker to include tacker-conductor
service.

Depends-On: I52778e86e4f2c297ead8d4b09983e5e38ca88c70
Closes-Bug: #1710874
Change-Id: I6901e919887551bedc9dba8983ac904e8c48c9ce

Fluentd repo/packages are changed in depends changes below.
Rpm distros need changes in their config for properly
start and deploy fluentd containers.

Change-Id: I474c4d20c407717c6da973383bbbd5539be480f1
Depends-On: I4802618373cfa53d67d1dfe65be7abe1950b3d04

Change-Id: Ic3b994056e6287488a0a9f7dbeb140c007ad86d3

Closes-Bug :#1710842

Change-Id: Ie676fcf4acb0d08f232a8db9313212b819e500de

'@type' must be used everywhere.

Change-Id: I337f42e174393f68b43e876ef075a74c887a5314

placement-api-access.log type is apache_access

Change-Id: Ie29b9f04f2d166d04a4fee53f3749911b44a9bf1
Closes-Bug: #1709054

This patch add "domain" concept to log format.

Change-Id: I5dbe5d3953d757150c7fb7df38e4bb4ddb99e49c

Change-Id: I43fec45a49198daedddbb7c5d477690bbe3864e8

http://paste.openstack.org/show/617484/

Change-Id: I1c771b8012c12300adb611b85898bbf2ef2fe67f
Closes-Bug: #1708449