Some gates are failing to properly pull all logs out resuling in an
UNSTABLE status.

This resets all logs to 777 perms to ensure readability by jenkins.

TrivialFix

Change-Id: Ib92c5bfea5fa8bcc6380ad8aa6001b13888717ac

Test results of this patch set are at
http://paste.openstack.org/show/490450/.

Change-Id: I2957cc70fcca34706ec50a630d785f0630c3f5c0
Closes-Bug: #1553203

This playbook only matters for multinode since AIO can recover from
power outage without additional configuration.

DocImpact
Implements: blueprint mariadb-lights-out
Change-Id: I903c3bcd069af39814bcabcef37684b1f043391f

Currently the binary build for the oraclelinux base is just exceeding
the 10GB limit we have set here. Centos is at just under 9GB so will not
be long before it reaches this limit also.

According to infra -
(http://docs.openstack.org/infra/system-config/contribute-cloud.html)
gate VMs should have a 80GB disk, so 20GB for the docker partition does
not seem excessive here.

TrivialFix

Change-Id: I4d7fb240ea90e1e58f8f8046dd7acd0b9502c20d

We can use the ip command to achieve the same thing.

TrivialFix

Change-Id: I903467c90495388cbc592b56d96bd3a4675df430

This change is a "futureproofing" thing. It has already been discussed
that libvirt should not be a child of nova and should be removed out
to the base docker directory (just like openvswitch isn't a child of
neutron). That is not going to happen this cycle but when it does we
can't change the name of the volume. This updates the volumes to the
proper name of libvirtd. This is in contrast with the libvirtlogd
volume that will be needed in newton due to libvirt 1.3

Of note, the container can remain named nova_libvirt since we can
change that on the fly later without breaking instances.

This wont break liberty as named_volumes are not backported yet.

TrivialFix

Change-Id: I16cf9e1b1dbba9b5a9f5cc883494580e276d4f72

The rabbitmq-server package is upgraded to 3.5.7 in cloud-archive
so we update centos to match

The xen-utils package now needs an explict version, xen-utils-4.6 is
what is provided by cloud-archive mitaka

Libvirt 1.3 is in the ubuntu cloud-archive. This has a new daemon for
logging that needs further implementation in newton. For now, it has
been disabled within the qemu.conf

Co-Authored-By: Jeffrey Zhang <jeffrey.zhang@99cloud.net>
TrivialFix
Change-Id: I84217824817c484b6800a64cbd7767b127a3098e

Add cron image and playbook for logrotate.

The "common" Ansible playbook includes configuration files for
logrotate. At this point the operator cannot customize/override the
logrotate configuration.

Closes-Bug: #1553244
Change-Id: Ic9fdda9a273c9ccd90502f0acc7614d2c7157dca

TrivialFix

Change-Id: I3ad67fd802f7fe3ed7deb88eddebe520ff410a4f

Heka logs almost done. When testing, The collected logs can copy
to the /tmp/logs folder, which will be handle by the gate. This
will be very useful for our gate debug.

TrivialFix

Change-Id: I06ca8ebd13933bfd15cb2fb5e53fc3038b17b8c3

related-bug: 1552433

Change-Id: If89e2a1bc959039fc7088c8295e88a199eabbedc

Relates-to: blueprint kolla-ssl

Change-Id: Id82d22c2195be5d897c781f28087a1ea422b8bea

As we will be switching to pinning versions in Kolla this tool will
help remove the 'manual' part of that equation by looking up the
versions we want.

The idea would be to eventually turn this into an autoproposal bot so
we only ever need to approve changes.

More work can be done here like automatically updating the
kolla-build.conf

Co-Authored-By: Martin André <m.andre@redhat.com>
Partially-Implements: blueprint upgrade-kolla

Change-Id: I052159850b2579ca6db79be71672be382be8a3a3

Relates-to: blueprint kolla-ssl

Change-Id: I6d9531ce4513ed06f1d13081418b167ad264aaff

TrivialFix
Change-Id: I95f2e6d17ba7960b2f87344f30b7884d621f9ecb

This patch includes changes relative to integrating Heka with
Elasticsearch and Kibana.

The main change is the addition of an Heka ElasticSearchOutput plugin
to make Heka send the logs it collects to Elasticsearch.

Since Logstash is not used the enable_elk deploy variable is renamed
to enable_central_logging.

If enable_central_logging is false then Elasticsearch and Kibana are
not started, and Heka won't attempt to send logs to Elasticsearch.

By default enable_central_logging is set to false. If
enable_central_logging is set to true after deployment then the Heka
container needs to be recreated (for Heka to get the new
configuration).

The Kibana configuration used property names that are deprecated in
Kibana 4.2. This is changed to use non-deprecated property names.

Previously logs read from files and from Syslog had a different Type
in Heka. This is changed to always use "log" for the Type. In this
way just one index instead of two is used in Elasticsearch, making
things easier to the user on the visualization side.

The HAProxy configuration is changed to add entries for Kibana.
Kibana server is now accessible via the internal VIP, and also via
the external VIP if there's one configured.

The HAProxy configuration is changed to add an entry for
Elasticsearch. So Elasticsearch is now accessible via the internal
VIP. Heka uses that channel for communicating with Elasticsearch.

Note that currently the Heka logs include "Plugin
elasticsearch_output" errors when Heka starts. This occurs when Heka
starts processing logs while Elasticsearch is not yet started. These
are transient errors that go away when Elasticsearch is ready. And
with buffering enabled on the ElasticSearchOuput plugin logs will be
buffered and then retransmitted when Elasticsearch is ready.

Change-Id: I6ff7a4f0ad04c4c666e174693a35ff49914280bb
Implements: blueprint central-logging-service

We shouldn't be scanning .tox dir anyway.

TrivialFix

Change-Id: If570abdfa69375091751199dcce637682e8de1ef

TrivialFix

Change-Id: Ia47429a38503d4982049c3fa1fc6fca4a8000fbd

Closes-Bug: #1551052
Change-Id: Ic226287bbf1f1e8d5cc2e1d80ce2975448b1f95c

Due to poor planning on our variable names we have a situation where
we have "internal_address" which must be a VIP, but "external_address"
which should be a DNS name. Now with two vips "external_vip_address"
is a new variable.

This corrects that issue by deprecating kolla_internal_address and
replacing it with 4 nicely named variables.

kolla_internal_vip_address
kolla_internal_fqdn
kolla_external_vip_address
kolla_external_fqdn

The default behaviour will remain the same, and the way the variable
inheritance is setup the kolla_internal_address variable can still be
set in globals.yml and propogate out to these 4 new variables like it
normally would, but all reference to kolla_internal_address has been
completely removed.

Change-Id: I4556dcdbf4d91a8d2751981ef9c64bad44a719e5
Partially-Implements: blueprint ssl-kolla

Working towards the blueprint that will add TLS protection
for the external endpoints, kolla needs certificates.

When kolla deploys OpenStack, the external VIP will need
a server side certifcate.  Clients that access those endpoints will
need the public CA certificate that signed that certificate.

This ansible script will create these two certificates to make
it easy to use TLS in a test environment.  The generated
certificate files are:

/etc/kolla/certificates/haproxy.pem  (server side certificate)
/etc/kolla/certificates/haproxy-ca.pem (CA certificate)

The generated certificates are not suitable for use in a
production environment, but will be useful for testing and
verifying operations.

Partially-implements: blueprint ssl-kolla

Change-Id: I208777f9e5eee3bfb06810c7b18a2727beda234d

DocImpact

Partially-implements: bp kolla-reconfig
Change-Id: I9738e80960bcfbef18d1ef1b7942f81c45684e85

Partially implements: blueprint heka

Change-Id: I1322d2dc870e6f8fe052926995d993e8a08a25db

TrivialFix

Change-Id: I0f083654df901fb765e9517308ac94316859d971

TrivialFix
Partially implements: blueprint heka

Change-Id: Ie8f393515c464020f079db41c05d023319920136

TrivialFix
Partially implements: blueprint heka

Change-Id: I5f91056590250f2a477018c1773905b412536f24

Partially implements: blueprint heka
Change-Id: I9ddad09b3f16b323c7ed535243458b0de6620cdb

I accidentally merged a patch with +W that does not work on all gates
that infra uses. Specifically it breaks on RAX for unknown reasons.

This is a quick patch to not fully revert, but allow the gate to stay
green while I work out the issue

Change-Id: I02fc66bbc53103f2d71c648d5e2959f10f0130c0
Partially-Implements: blueprint functional-testing-gate

Make the containers and volumes names of ironic to be
consistent with others.

Partially-implements: blueprint ironic-container
Change-Id: If78b020c81157a948c5c3283f1410bcb6e5c1e61

Change-Id: I18b1015ca62a352e193ef01bd49cd6d6f9372193
Partially-Implements: blueprint functional-testing-gate

TrivialFix

Change-Id: Ia52451d2c0de09ee9dcf79211389a3b1ce4ae738

missing a "_"

TrivialFix

Change-Id: Id7624f4f35c3c53c068d7c6646b363332363c3d5

toos/cleanup-containers script.

Dynamic Ceph OSD bootstrap containers also included.

Change-Id: Ifb5b54c5b78a894ed6e8b9980a6eb9919e52364c
Closes-Bug: #1515544
Signed-off-by: Shaun Smekel <shaun.smekel@theorem.net.au>

Change-Id: Ic6cd45ec708efe4f573e19f3bc910717ac3b4fbb
Partially-Implements: blueprint functional-testing-gate

TrivialFix

Change-Id: I5168191182c249df68fd0b5c70a778f223326974

It was run before ansible was installed and did not properly fail.
This prevented the mirrors for apt from being properly used.

TrivialFix

Change-Id: I5331020993b62c8635a4a97dc3a3d2e01e3d4360

TrivialFix

Change-Id: Id0e7cf9e1c0c4259b8a50c39e95214103256517b

This make sure the check_failure called whenever the script exits
unexpectedly.

TrivialFix

Change-Id: If38e9ad42810deec48afd3f8b57387ae6ffd926c

When breaking neutron-agents into thin containers, we forgot to update
the cleanup-containers script.

TrivialFix

Change-Id: Idad2eb73a4de7c66109ad5bffcce3173970e2af9

The gate currently doesn't provide any context when ansible fails. This
commit adds a new --verbose option to `kolla-ansible` that is passed to
`ansible-playbook`, and sets the default to -vvv for `kolla-ansible
deploy` in the gate.

TrivialFix

Change-Id: If248a4186b140e1ec0d2e787c27481cec7e81eec