Fluentd config actually used two keywords to named log level:
"log_level" and "severity_label".
This doesn't allow to search and sort ALL logs in Kibana using a unique
field (ex: log_level).
This fix standardize log level keyword naming.

Closes-Bug: #1713410
Change-Id: Ibb2eaf3b6aa21ce6f7945e768e6362c0d40f0507

infra.rabbitmq tag does not match in fluentd

Closes-Bug: #1718874
Change-Id: I92733a4f275baa93b0c3a6ddbdc6801aec534867

progname is actually only fullfilled for Openstack logs and infra logs.
They are missing for program logging in RSYSLOG: HAPROXY and SWIFT.
When using Kibana, "progname" field appears empty, hence it's difficult
to know where logs come from.

Change-Id: Iae98af083cfb79d2a64fa0b70c86b0a01f7a336f
Closes-Bug: #1716051

In some scenarios it may be useful to configure custom fluentd outputs
to forward logs to a logging service other than elasticsearch.

This change supports configuration of fluentd outputs by placing
output configuration files in /etc/kolla/config/fluentd/output/*.conf.

Change-Id: I3c0b271d88dbb307ba3a23546e29c72e8baeca55
Implements: blueprint fluentd-custom-outputs

Barbican-api log is a uswsgi process and fluentd expects as
openstack service process.
This change moves log type to uswgi group.

Change-Id: Ifaf5645d0d19e133e41de0c55a55321fccfc391f
Closes-Bug: #1716513

Libvirt logs are not managed by Fluentd. It can be very helpful to
debug 'booting' issue on KVM.

Change-Id: Ie04322adec0482f4d586d2f8564a1a77f2da2605
Closes-Bug: #1713429

'progname' is actually only fullfilled for Openstack services logs.
They are missing for MariaDB and RabbitMQ logs.
When using Kibana, "progname" field appears empty, hence it's difficult
to known where logs comes from.

Change-Id: Ifcbfd981a34a8d64ea69d37934c7f0fae10a90b5
Closes-Bug: #1713412

Actual Fluentd config doesn't manage multiline log pattern like:

2017-08-16 21:41:03.097 7 WARNING oslo_config.cfg [-] Option "firewall_driver" from group "DEFAULT" is deprecated for removal (
nova-network is deprecated, as are any related configuration options.
). Its value may be silently ignored in the future.

This log will generate 1 Fluentd event for each line (in previous
example 3 events).
This is really a wrong behaviour and made Kibana interface not friendly.
This patch uses a multiline parser.

Change-Id: Iee337645d1dc42876ec79be08ee3715f11a53d87
Closes-Bug: #1711399

They dropped the cloudkitty-api command line[0], so we should add wsgi
support for cloudkitty-api.

[0]https://review.openstack.org/#/c/366043/

Change-Id: Ie34d4f2d5c303bbd7ac09a8ab9e8d9bdc763c57b
Closes-Bug: #1713879

Actually a unique 'tail' source is used for all logs files in
kolla/*/*.log.

So log files from services like: chrony, elasticsearch, etcd, grafana,
influxdb, kibana, mongodb... are actually processed by our Fluentd
config, but should not cause all following config will failed to parse
them. It just uses CPU for nothing cause they will
never match the following filters:
* record_transformer => add Hostname, Logger, programname field
* rewrite_tag_filter => rewrite 'programname' TAG if needed

Hence these filters should only be applied to Openstack services logs to
avoid CPU usage.

Furthermore WGI and Apache logs files must be treated differently.

Change-Id: I425979160291e1123e3775e39243bca06ed22c2f
Closes-Bug: #1711390

Tacker has included a new conductor service
to manage mistral workflows for VIM monitoring.

Without conductor, Tacker cannot create VIMs.

This change reworks tacker to include tacker-conductor
service.

Depends-On: I52778e86e4f2c297ead8d4b09983e5e38ca88c70
Closes-Bug: #1710874
Change-Id: I6901e919887551bedc9dba8983ac904e8c48c9ce

Fluentd repo/packages are changed in depends changes below.
Rpm distros need changes in their config for properly
start and deploy fluentd containers.

Change-Id: I474c4d20c407717c6da973383bbbd5539be480f1
Depends-On: I4802618373cfa53d67d1dfe65be7abe1950b3d04

Change-Id: Ic3b994056e6287488a0a9f7dbeb140c007ad86d3

Closes-Bug :#1710842

Change-Id: Ie676fcf4acb0d08f232a8db9313212b819e500de

'@type' must be used everywhere.

Change-Id: I337f42e174393f68b43e876ef075a74c887a5314

placement-api-access.log type is apache_access

Change-Id: Ie29b9f04f2d166d04a4fee53f3749911b44a9bf1
Closes-Bug: #1709054

This patch add "domain" concept to log format.

Change-Id: I5dbe5d3953d757150c7fb7df38e4bb4ddb99e49c

Change-Id: I24f040b686b4dd24b26fc5c5fc5bd5e7ef8a4e70
Depends-on: I464cb4700ca143c1dbac3c20f7402ddb8cda7ce7

Because the default domain name is "Default" in
ansible/group_vars/all.yml (default domain id is "default"),
the domain name in admin-openrc also should be "Default".

Change-Id: I05f905d87ca57c09d4bdf540c4efbe06ee9bff67
Signed-off-by: Alex Yang <yangyang1@zte.com.cn>

Change-Id: I43fec45a49198daedddbb7c5d477690bbe3864e8

http://paste.openstack.org/show/617484/

Change-Id: I1c771b8012c12300adb611b85898bbf2ef2fe67f
Closes-Bug: #1708449

for example: the placement-api regular match
will override placement-api-access

Change-Id: I80ed896b2477709599be0ce73e00d03da1858349
Closes-Bug: #1708468

Change the log mode to 644 ensure that fluent can read the files.

Change-Id: I5b718b82ac4267ab29425b2859b14e4518e75cf1
Closes-Bug: #1708095

kolla-kubernetes is using its own configuration generation[0], so it is
time for kolla-ansible to remove the related code to simplify the
logical.

[0] https://github.com/openstack/kolla-kubernetes/tree/master/ansible

Change-Id: I7bb0b7fe3b8eea906613e936d5e9d19f4f2e80bb
Implements: blueprint clean-k8s-config

* remove ceilometer-api and ceilometer-collector service
* use ceilometer-notification to publish message to proper backend
* remove useless ceilometer_database_type and ceilometer_event_type
  variables
* sync event_definitions.yaml, event_pipeline.yaml and pipeline.yaml
  file with upstream

Change-Id: Ib39053cb5f70bd11ee61d3f26d5b28accecd7190

* "bool" filter is removed when not required
* 'not' is used instead of '== False' check

Change-Id: I85a5bb9a5ea874ac1c397cbf8de416147d2424c3

Zun-api is using app.wsgi as log filename.
This change forzes zun-api.log to ease identification.
Also changes group owner of log dir to kolla group.
Remove glance version as is in zun's defaults
Set state_path, this is where zun will store catched images,
similar as nova cache. No need HA or volume as is only
cached data.

Change-Id: I20f8d311bfde1c0a07d7d35e9f1412c71efa8504

SKIP_LOG_SETUP variable was used by Heka stuff.
Heka has been removed from Kolla and Kolla-ansible.

Change-Id: I4448b26ace899eb888d34a12a85b019597e25435

Logrotate configuration is missing for ironic-inspector. Prior to [1],
ironic-inspector logs were stored in
kolla_logs/ironic/ironic-inspector.log. After [1], the logs
are stored in kolla_logs/ironic-inspector/ironic-inspector.log
but the logrotate config was not updated.

[1]
https://github.com/openstack/kolla-ansible/commit/07453f346094b184a209380e375f6311987675be

Change-Id: I7e78faea361cd67069c1a96aaf0a2ffffc0e8666
Closes-Bug: #1701578

change api.log and registry.log to glance-api.log and glance-registry.log

Closes-bug: #1700718

Change-Id: Ifcde8699fa9537fa06445f79c4bd14b4ee0df32c

Certain services such as Murano and trove require access to a rabbitmq
instance from tenant networks. [0]

Exposing the internal rabbitmq to end users is a security hole, hence
there are two options, 1) use vhosts in the existing rabbitmq, or two a
separate rabbitmq instances. Given the importance of rabbitmq to the
OpenStack deployment, we have decided to go with a separate instance.
Refer to [1] for more detail on the various options.

This change makes the rabbitmq role generic so that it can be reused, in
this case to start 'outward_rabbitmq'. It needs to be exposed via
haproxy both for network isolation and also because this is what Murano
configuration requires.

Follow on patches will be added to add a vhost in this outward instance
for Murano and other services which require access.

Based on the original work by bdaca[2]

[0] http://murano.readthedocs.io/en/stable-liberty/intro/architecture.html
[1] http://lists.openstack.org/pipermail/openstack-dev/2016-December/109091.html
[2] https://review.openstack.org/#/c/374525



Change-Id: Ib2bcc7ed4bf4f883a7cd1dfad3db89201e3cfd8d
Partial-Bug: #1620374
Depends-On: I020eb6219f89a310451becde41f6f1c7f54baadd
Co-Authored-By: Bartłomiej Daca <bartek.daca@gmail.com>

Cron is always restarting due dummy environment
variable is None.
This change adds a useless env value.

Change-Id: Iad435bb0671a25bc12c6a6d0d988d555faba76c3
Closes-Bug: #1697706

Missing config options in kuryr.conf
Missing kuryr logging
Remove useless bootstrap task, justs create a directory
Wrong user permissions in config.json file.
Missing volumes
Missing container capabilities

Depends-On: I4c08a21df263fdefe1fe991cb7ad41cfee65019d
Change-Id: Id4577a78ebf3f1cda5ee36d14e2cc017f38e0f07
Closes-bug: #1697480

Zun rule number causes conflict with rule 24,
it should be updated to the next free number.

Change-Id: Idbe346bda4ca5dc50f6b5e63818b12af33d532a3

Change-Id: I929587720c622833764c61543cac7675360ccdcb

There is a typo in cron.json making cron to fail
when mongodb is enabled.
Only affect master.

```MissingRequiredSource:
/var/lib/kolla/config_files/logrotate/mongdb.conf file is not found```

Change-Id: I79d41e94ee3361f2b4d831d17b1bb34a1964f580

Many of the templates use 600, remove unnecessary permission
on these templates to bring them in line with the others.

Change-Id: I30fe1b3822b9c7bb6ab98729fc519dc1d603db27

Useful api_interface_address variable has been define here:
https://github.com/openstack/kolla-ansible/blob/master/ansible/group_vars/all.yml#L57
In order to simplify codebase we must use it as much as possible.

Change-Id: I18fec19bf69e05a22a4142a9cd1165eccd022455

Change-Id: I57e3b310c65aabeea9f298db78762e9d535e8703
Partially-implements: blueprint better-reconfigure

This changes moves cinder-api to make use of apache to
launch its processes.
Ubuntu binary removed support in Ocata for cinder-api
system services [1]. At this moment cinder in Ocata branch
is not working at all due this issue.

This changes aims to fix cinder in master only for ubuntu
binary with the purpose of make a clean backport to stable/ocata
without do any other change on the backport process.
A follow up change will migrate other OS/install_type to make use
of Apache but only changing master branch.

[1] http://lists.openstack.org/pipermail/openstack-dev/2017-January/111069.html

Change-Id: I9885d6acadbac1da46844f583e7cd37785505ae0
Closes-Bug: #1676751