As with all tools, this is a first pass at the generation. Perhaps we
even want to move this into kolla/kolla/cmd and be generated with tox
itself in the future.

This tool, when run, will only populate empty fields that have no
values meaning that it is safe to run repeatedly on the same file.

Of note, there is no way to preserve comments in the file after it has
been processed by the yaml parser in python. Comments and sections
will remain in the passwords.yml template for additional documentation
if the user wishes to populate the file themselves.

Use SystemRandom and clean up the docs a bit to not use pronouns.

Co-Authored-By: Steven Dake <stdake@cisco.com>

Closes-Bug: #1559266
Change-Id: I2932d592df8871f1b7811059206d0b4d0553a687

This change allows using a local directory or tarball to build from
sources by specifying the 'local' type in your kolla-build.conf:

    [keystone-base]
    type=local
    location=/home/martin/src/keystone

    [neutron-server-plugin-networking-cisco]
    type=local
    location=/tmp/networking-cisco.tar.gz

Closes-Bug: #1509625
Co-Authored-By: Martin André <m.andre@redhat.com>
Change-Id: I3f6f68b0f44efdac653d186d2625591ad7cb3b2a

kolla-build currently does not provide information
if docker service is not running. Added exception
handling.

Change-Id: I1f167804b8b4d2ad9220281954ff7ea7162a9817
Closes-Bug:#1553912

When kolla-build is running, if there are network issues or if the
source's location (like http://tarballs.openstack.org) fails to
respond due to high number of concurrent requests, kolla-build just
hangs/blocks indefinitely.

This patch set resolves this issue by adding a timeout of 120
seconds for requests.get() in kolla/cmd/build.py, adds a unit test
for it in kolla/tests/test_build.py and also the "timeout" argument
in kolla/common/config.py.

Change-Id: I7c8745a20b9bd1c3f5d6a55c72a794f16fd7e513
Closes-Bug: #1548614

Closes-Bug: #1536272
Change-Id: I6c91e522923eade16ba23711e6014e6b4b8cd3e0

The new oslo.config style parsing for kolla-build.conf stopped plugins
from working. This patch fixes it and adds a unit test.

Co-Authored-By: Jeffrey Zhang <zhang.lei.fly@gmail.com>
Closes-Bug: 1534556
Change-Id: I135797c733ae0cae040a009c364073769b19e4eb

The install type is converted in kolla-build, so it will never
fail in dockerfile, move the check to kolla-build just above
the install type converting.

TrivialFix

Co-Authored-By: Jeffrey Zhang <jeffrey.zhang@99cloud.net>
Change-Id: I1500d3b47e909f94ea9f68c5245297733f63a70b

Change-Id: I3afb8a29c4ae2369f29e0cc94d75ff8a3c42594b
Implements: blueprint custom-repos

Base on the OpenStack Style Guidelines[0]: Do not import objects, only
modules

[0] http://docs.openstack.org/developer/hacking/index.html#imports

TrivialFix

Change-Id: Id1bc5e4b24fd561f1a9f6167af341b00a343a718

We currently support any buildarg via --build-args. This patch picks up
the default supported proxy buildargs if set in the user's environment,
to allow for more transparent proxy support.

The --build-args will take precendence.

DocImpact
Implements blueprint: better-proxy-support

Change-Id: I084e5d1cc8f9993d65167878f9983ad58a68e86e

With this implement, we can add variables at building stage.
For example, add HTTP_PROXY and NO_PROXY when needed like below.

    build.py --build-args \
        HTTP_PROXY:http://127.0.0.1:8080,NO_PROXY:127.0.0.1

More info about build_arg, pls check[0]

[0] https://docs.docker.com/engine/reference/commandline/build/#set-build-time-variables-build-arg

DocImpact

Implements: bp add-buildargs
Change-Id: I29ed7f536670fef59d697603dc562a69d90743c9

Replacing dict.iteritems()/.itervalues() with
six.iteritems(dict)/six.itervalues(dict) was preferred in the past,
but there was a discussion suggesting to avoid six for this.

ref:
http://lists.openstack.org/pipermail/openstack-dev/2015-June/066391.html

Partially implements: blueprint python3
Change-Id: Ie931bc02f558f6db3677b44f645eb3bdf78ccae8

Closes-Bug: #1531848
Change-Id: I3c6ceb9f1f367b8b19ab1b1d06c67817ab93cabb

Two things broke in the related patches. This patch aims to fix that.

1) tools/build.py was no longer usable as it imported kolla from the
global pip install source. If that didnt exist (because kolla wasn't
installed with pip) then it breaks on import.

2) When kolla was installed globally it looked up the path that
matched the installation of docker-py instead of looking up a path
that had the stuff we wanted (kolla images directory)

Change-Id: I94f1856547ce54d506de72926b08c966a36ac608
Related-Id: I05f10d13e7ba1e2b985c2944aec71ce55630442b
Related-Bug: #1530256
Closes-Bug: #1531569

The try-except-finally block in push thread was catching only
ConnectionError and logging information about successful push
even if some other error occured during push.

TrivialFix

Change-Id: I94f5ed67237d291701469a65f76146881cd67a35

By using

    ./tools/build.py --save-dependency a.dot

to gen the Graphviz dot dependency file. Later, you can use the `dot`
to draw the picture.

    dot -Tjpg -o a.jpg a.dot

Implements: blueprint images-dependency-tree
Depends-On: I8e07a1b69fab5f1c587470bfd2104aaba93f0050
Change-Id: If00f4f3fb9d0b10a07ab2abb7ffb1cd9d64902f2

Closes-Bug:#1530256
Change-Id: I05f10d13e7ba1e2b985c2944aec71ce55630442b

Closes-Bug: #1530122
Change-Id: I96039eade2aacf1bcd9d14d958cdddd915fc9a5c

Do not truncate the error messages of build failure images.

TrivialFix
Change-Id: I4f2da78c57c8d919beba84a10357312bfc66e88c
Closes-Bug: #1531152

range maps to xrange() and range() in python2 and python3 respectively.

TrivialFix
Change-Id: I9b2477a17a570c3ce945badacbfa44889c0634a9

To improve consistency, use range() from six.moves

TrivialFix
Change-Id: I3bb67b2bb5bedbc5b46f70d7eeeca7f8b7872250

* Use oslo.config library
* kolla-build section is deprecated in favor of DEFAULT section
* Remove the deprecated `--template` option
* `--no-cache` is deprecated in favor of `--cache` and `--nocache`
  options, which are using oslo BoolOpt inverse feature[0]

[0] https://github.com/openstack/oslo.config/blob/master/oslo_config/cfg.py#L1074

Closes-Bug: #1528542
Change-Id: I62c5ca3c8e057917d2798256e9d9efc8f6578f60

TrivialFix

Change-Id: I72c541c1846e503486b402e88d5b29364920fc83

Closes-Bug: #1529327
Change-Id: Iad6c1454f42d3410c82c722b60d0d305b9eec129

Something was missed during the logging update

TrivialFix

Change-Id: Ic611b7a0e2a53725b1a3a76aba56faeca5f68dfa

- Removed hardcoded yum repository configuration in favor of
  commands dynamically generated based on repo-url and repo-file
  arguments. We maintain a sane default set of repositories.
- Added generic rpm_setup_config parameter to add support for
  installing .rpm or .repo files before building containers.

Co-Authored-By: Ryan Hallisey <rhallise@redhat.com>
Implements: blueprint custom-repos
Change-Id: I1b3a7647a9e7239de3cd162cb6f464f05632bde1

TrivialFix

Change-Id: I4b01402f8feb029563a90d4941c8d829a05794e0

We want to record kolla version of running containers to be able to
detect whether or not we need to perform certain downtime-causing
actions during upgrade.

Change-Id: Ie113029da98303e6809d56edbf6d8de37be128d7
Implements: blueprint record-version

TrivialFix
Change-Id: I39ad214d6a8e1fbdba799076d6a5371cc47824a1

This changes default behaviour of build.py to instad of putting latest
tag on it, it puts current kolla version as found in setup.cfg

Change-Id: I4d6e9a0159c6a5598abd58072594df4204427308
Partially-Implements: blueprint upgrade-nova
Partially-Implements: blueprint record-version

The build script now creates now creates a plugins directory for source
builds to store plugins required for the container, and creates a tar
of all plugins that is extracted at runtime.

However, the creation of the 'plugins' directory is not handled
correctly and generate an exception in the running thread if the
directory exists. This is the case for example if the build script
retries a failing image.

This commit fixes the timeout issues we've experienced with the gate.

Change-Id: Ic5d4fd1ddf71f01c547130518e311fded911c445
Closes-Bug: 1524897

Change-Id: I25d957a2368fa4e52b5fa9253d7ea5ec4046d830
Closes-Bug: #1521949

This change allows additional archives to be fetched as part of the
build and added to images.

An example use case is the installation of plugins for neutron. For
example, to add the networking-arista plugin to neutron-server, one
would do the following in kolla-build.conf:

[neutron-server-plugin-networking-arista]
type = url
location = http://tarballs.openstack.org/networking-arista/networking-arista-master.tar.gz

The format for plugin sections is <image-name>-plugin-<plugin-name>

The required Dockerfile additions will be added in the next patchset.

Background to this change:
https://review.openstack.org/#/c/207545/3/docker_templates/neutron/neutron-server/Dockerfile.j2

Related: blueprint neutron-third-party-plugins
DocImpact
Change-Id: Idec2ca9fd06163c7d820e8c4bc9ebca8d9673856

Added new option in kolla-build.conf

Change-Id: I45fe51966bcb59ea19d112281ba3d5a1ba091a56
Closes-Bug:#1514304

Change-Id: Ia5d9ad1ffa119c2dd3e74272df3fb6d78501ec09
closes-Bug: #1513175

Implements: blueprint python3

Change-Id: I1e02568907aa38dd0efcb428235004ce47d73af8

Off by one error made the --retries option control the number of tries
rather than the number of retries.

Closes-Bug: #1514730
backport: Liberty

Change-Id: I976a8bb9e489d226f44926a6562d4d2af5de099c

pip install default prefix in Ubuntu is /usr/local, and Kolla tools scripts
didnt respect that. So I added few OS checks in this scripts.

I improve config path check in build.py. Added more verbose error if we can't
find config directory.

Change-Id: Ide521ed205b0dc1fc27e237a9a8f4da0168e664f
Closes-Bug: #1512302

build.py -b rhel -t [rdo|rhos|source|binary]

The last patch for this didn't quite fix the problem properly as
it only permitted RHOS builds.

backport: liberty

Change-Id: I27eed202560adce450c07d043cc224e7a6c6bbf6
Closes-Bug: #1513088

Use the absoluate path rather than that with `..`. This will be
helpfull for end-user to see where is the folder/file.

Closes-Bug: #1513726
Change-Id: I7169952d874ddf14469605444044de0163b033d3