Adds option to build.py to only build images matching and its
dependencies

Change-Id: I281bea5bed932b7c948b247392fa6a52b1d1aecf

Additionally, fix an if statement to catch multiple layers of parent
images failing so the grandchildren do not attempt to build either.

Change-Id: Ie9cdf7600e067678e52938910f0d3c072adcdf8a
Partially-Implements: blueprint build-script

Sometimes you need custom code in Docerfiles to solve for example
proxy problems or preconfigure base image in some specific way.
With this commit you can insert content of custom file to the beggining
of base Dockerfile to provide these changes

Partially-implements: bp rhel-based-image-support
Partially-implements: bp add-proxy-to-dockerfiles
Partially-implements: bp build-script
Change-Id: Idbfec764aad627f5c00dc09c4854d5fa1c5ac9cf

Docker cache takes into account things like filesize, hash of file,
and even timestamp and access times when determining whether to use
the cache or not. This modifies the build script to adjust all the
timestamps to 0 epoch. Since we dont care about timestamps this allows
us to make far better use of the cache, including when downloading and
creating tarballs for source builds.

Change-Id: Id4287cf1fcaa2de63feaab9e6f4ebbd4defdb768
Partially-Implements: blueprint build-script

The previous method of using the API image for a data container
worked, but resulted in more data being bindmounted and copied
into the system.  This patch resolves that by essentially using
a distro-type as a FROM in a Dockerfile to generate
distro-type-data.  Further all of the data containers used
throughout ansible are modified to use this new data container.

Change-Id: I8846573d4f2d4b98d4f46c770bfefc6d4c5cd0b5
Partially-Implements: blueprint one-data-container

The purpose for having defaults.yml on the command line was originally
for variable inheritance. Over the past month we have slowly changed
how things are inherited (like ports) allowing us to place these
'defaults' into the group_vars/all.yml location as they should be
values that go unchanged (but can be overridden).

Change-Id: I2becec66bf431bfc9d88fc825b5380c1e173ca70
Partially-Implements: blueprint update-configs

Receive several quickstart related questions every day.  Make a
first pass at cleaning up our documentation to point people
in the right direction.  While we are about it remove compose
related bits.  Anything in the compose directory will remain as
a reference implementation for how to implement the compose
files to work well with our container content.

Change-Id: I9e832e97ac2bacca8eab7e1bfbd82664d2b191b8
Closes-Bug: #1485163
Paritally-Implements: blueprint remove-config-internal

Fix a backtrace.  Also add the default master locations for
the various tarballs.

Change-Id: I9675c2619b95c0ae6a467ffbf964c4b17ae4b791
Paritally-Implements: blueprint gate-source-builds

Accessing the key directly when it doesn't exist was yielding a
KeyError.

Change-Id: Ice877b74d46d59f2e9ecff051b09245e9d3744e1
Paritally-Implements: blueprint gate-source-builds

This implements the source fetching that exists in the current way.
The patch only implements fetching a tarball from a url. Additional
methods needed would be git and local copy.

Partially-Implements: blueprint build-script
Change-Id: I51889a131f050abce9a16d114972b5e329a93862

This adds a new test script for the new python based build script. A new
tox target has been created to execute it for usage in the gate.
Eventually, the other build script stuff can be deleted, but that won't
be done until full functionality exists in the new build.py.

Change-Id: Id9dd19926b1b9d875dcb069fd13d6450318374c8
Paritally-Implements: blueprint build-script

Change-Id: Ic42879451cba2fb7fa35764924ca191949c477c4
Partially-Implements: blueprint dockerfile-template

Switched all print statements to use logger

Added ability to halt operations via Ctrl-C, needs cleanup handler

Added realtime logging when using only one thread (matches bash building
scripts behavior)

main is now returning all the build image statuses, so a quick test
script would look like:

import build
res = build.main()
if len(res[0]):
    LOG.error('failure')

Change-Id: Ic8000a96573b011490dc330a4512c77c602ac3d2
Partially-Implements: blueprint build-script

Added new method generating Dockerfiles from templates as well as the
appropriate code to support this in build.py

Co-Authored-By: Sam Yaple <sam@yaple.net>
Partially-implements: bp dockerfile-template

Change-Id: Id14d10d31cb4aac957e04fbc129de4043d98e033

* Both api and engine start up without error, but can't test properly
until Horizon is fixed (bug #1482196)

* These are source installs are there no RPMs available for Murano
that I can find.

* Also requires a plugin for horizon that will be made as a
separate commit

Co-Authored-By: Mick Thompson <michael.a.thompson@oracle.com>
Implements: blueprint enable-murano-container

Change-Id: Ia46985b21bb5f6d7690501e7199800c8c7fcb51f

We don't use config-internal anymore, so remove validate-min-env-vars
from being gated upon.  This is becuse there is no minimum
environment variables required.  While we are at it we remove
min-env.

Partially-Implements: blueprint remove-config-internal

Change-Id: Ibe2947b35e4dc1549e82acc66c01153495a91882

As much faster than git clone

Change-Id: I173592f1efceac61b49c37f994004cc41c641ae0
Closes-Bug: #1481670

This works as is, but could use some cleaning up and threading needs
to be confirmed broken. If it isn't broken, it should be implemented.

Co-Authored-By: Swapnil Kulkarni <me@coolsvap.net>
Change-Id: I708406dcff8aa9b2f4064f03bda07873ce97d994
Paritally-Implements: blueprint build-script

Got removed as part of a refactor

Change-Id: I7dc47f4af316188803fe2fc654316540b8d23cf8
Closes-Bug: #1480465

Change-Id: Icf845822735babc78ca427c00a2a125c8f0cca32
Closes-Bug: #1481293

Current with due to LP#1478145 the kolla-compose script
fails to create gnocchi containers. Temperory fix to skip
gnocchi till we have gnocchi packges in delorean.

Change-Id: I7dc6aef82908a846a8b632f86a16de47c1a2eef9
Partial-Bug: #1478145

* Remove service_hosts.sh file
* Add all the required variables to tools/genenv

Change-Id: I23954bf148923fbe785f51012fc9c9da909fa380
Closes-Bug: #1447867

Change-Id: Idc5f8aef80a65fea13fecd62b6b40c8ef7458354
Closes-Bug: #1480296

Change-Id: Ie056a8347f6b72ffb647bfb79ee4449ca45f3fef
Closes-Bug: #1476294

Change-Id: I4f6f38b2a202b41a9dec839fa0222585b49fc06c
Partially-Implements: blueprint kolla-ansible-script

Known bugs/TODOs at this time of this patch update:

https://bugs.launchpad.net/kolla/+bug/1477990
https://bugs.launchpad.net/kolla/+bug/1477993



Implements: Blueprint swift-container
Closes-bug: #1477992
Co-Authored-By: Paul Bourke <paul.bourke@oracle.com>
Change-Id: I031f88a7cd1867104fef12df6ab3ffcc06cb65d6

- tools/validate-links.sh has a echo which
  is not necessary. Removed it.

Change-Id: I122c995bc5d99592212505ce23256e27685b4af4
Closes-Bug: #1479647

- Updated the required variables to be sorted so
  that they are better visible one on each line

- Removed variable defaults from each service scripts
  and added them to tools/genenv file which holds
  all default values

Change-Id: Ia7b1b08ba3f963b46b81434ce84f476d56890a93
Closes-Bug: #1476494
Closes-Bug: #1467652

Change-Id: I8d216ca20c97428e3b27d48d0a5249282dcb8c18
Closes-Bug: #1468509

Renamed kolla script to kolla-compose
Update change in the dev guide also

Change-Id: I6cae3d13752ecb4bb3deeb91c5e0f827fde80c2a
Implements: blueprint kolla-compose-script

This makes some changes to init-runonce to allow using Ansible as well as
updating it to use the openstack-client where applicable since the
keystone-client is not usable with keystone v3 api at all.

Change-Id: I6b9d6beb04df7724b1967638e98042ab03221cd1
Closes-Bug: #1477725

Split the content of cleanup-containers into containers cleanup and host
cleanup. This allows to run the cleanup-containers from OSX clients,
while keeping needed code to delete host artifacts left behind by the
neutron agents container.

In addition, cleanup-containers may disappear in the future, replaced by
a docker-compose wrapper.

Co-Authored-By: Martin André <martin.andre@kvhasia.com>
Co-Authored-By: Swapnil Kulkarni <me@coolsvap.net>
Change-Id: I3aa731e7e0f9eb3dad9c363bb5d4a575d84a2fae
Closes-Bug: #1477383

Co-Authored-By: Swapnil Kulkarni <me@coolsvap.net">
Change-Id: I81e5e6bfde3323e6917e3d61e3db760e2fec2019
Implements: blueprint gnocchi-containers

It seems using --git-dir to execute commands on a repo different to
where you currently are also inexplicably requires --work-tree
(https://stackoverflow.com/questions/1386291/git-git-dir-not-working-as-expected)

Change-Id: I42806e7d8a169ad9fc1878a0f4e0c03c57beb88b
Closes-Bug: #1475385

This way it is possible to manually specify MY_IP/MY_DEV when running
the genenv script.

Closes-bug: #1475572

Change-Id: I291815a4ed1ea15adbddf94905706c77c8bb5a09

Added the compose yaml file
Added the genenv
Added fixes to run the containter

Change-Id: Ied3ea92e495fc0fc0274a0da736394d2ab52a754
Partially Implements: blueprint ceilometer-container

The tgtd process was not being started in the cinder-volume container.
This results in iSCSI not being avialable for Cinder to operate
against.  Unfortunately cinder doesn't report the problem because LVM
is responsible for the iSCSI mapping.

Co-Authored-By: Ryan Hallisey <rhallise@redhat.com>

Closes-Bug: 1473090
Closes-Bug: 1469323
Change-Id: Ie3403c5c3f81beb48960e838b4274e0c5796e387

Fixed docker client to use ENV if exists to support boot2docker.

Fixed booleans not working as thought with argParser.

Change-Id: I232ed78443199ce20f4b38e12c861c0f97d55c99
Partially-Implements: blueprint build-script

Added new script for checking user permissions on
docker command execution

Change-Id: I3b727590d9a29f416f5b847912b7250c555c2333
Closes-Bug: #1473554

The following files had requirement for root user
which can be removed as long as as the the user
executing script is part of docker group

Scripts updated
tools/kolla
tools/cleanup-containers

Closes-Bug: #1473551
Change-Id: Ia4aea205f5da20fb1ded4f7a5d11eeebe5f40aa9