diff --git a/README.md b/README.md
index caaf6803c1e3a2861a04347e8fbe9640ecd24036..354656d08ec30ee3d7c849fcc934369fcf3c6d69 100644
--- a/README.md
+++ b/README.md
@@ -1,30 +1,36 @@
-Kolla
-=====
+Kolla Overview
+==============
-The Kolla project is part of the OpenStack TripleO effort, focused on
-deploying OpenStack environments using [Kubernetes][] and [Docker][]
-containers.
+The Kolla project is part of the OpenStack [TripleO][] effort, focused
+on deploying OpenStack services using [Docker][] containers. The initial
+system [spec][] provides additional details of Kolla and the use cases
+it addresses.
-[kubernetes]: https://github.com/GoogleCloudPlatform/kubernetes
-[docker]: http://docker.com/
+[TripleO]: https://wiki.openstack.org/wiki/TripleO
+[Docker]: http://docker.com/
+[spec]: https://github.com/stackforge/kolla/blob/master/specs/containerize-openstack.rst
Getting Started
===============
-Kubernetes deployment on bare metal is a complex topic which is beyond the
-scope of this project at this time. The developers require a development test
-environment. As a result, one of the developers has created a Heat based
-deployment tool that can be
-found [here](https://github.com/larsks/heat-kubernetes).
+Deployment on bare metal is a complex topic which is beyond the scope of
+the project at this time. An environment to simplify the deployment of a
+single or multi-node Kolla cluster is required for development purposes.
+As a result, a [Heat template][] has been created for deploying a Kolla
+cluster to an existing OpenStack cloud.
+[Heat template]: (https://github.com/stackforge/kolla/blob/master/devenv/README.md)
-Build Docker Images
--------------------
+Docker Images
+-------------
-Images are built by the Kolla project maintainers. It is possible to build
-unique images with specific changes, but these would end up in a personal
-namespace. Read the docs directory image workflow documentation for more
-details.
+The [Docker images][] are built by the Kolla project maintainers. A detailed
+process for contributing to the images can be found [here][]. Images reside
+in the Docker Hub [Kollaglue repo][].
+
+[here]: https://github.com/stackforge/kolla/blob/master/docs/image-building.md
+[Docker images]: https://docs.docker.com/userguide/dockerimages/
+[Kollaglue repo]: https://registry.hub.docker.com/repos/kollaglue/
The Kolla developers build images in the kollaglue namespace for the following
services:
@@ -45,161 +51,35 @@ $ sudo docker search kollaglue
```
A list of the upstream built docker images will be shown.
-Use Kubernetes to Deploy OpenStack
-----------------------------------
-
-At this point, we believe the key features for a minimum viable feature set
-are implemented. This includes the capability to launch virtual machines in
-Nova. One key fact is that networking may not entirely work properly yet
-until Neutron is finished, so the virtual machines may not actually behave
-as expected for an end user deployment.
-
-Two options exist for those without an existing Kubernetes environment:
-
-The upstream Kubernetes community provides instructions for running Kubernetes
-using Vagrant, available from:
-https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/getting-started-guides/vagrant.md
-
-The Kolla developers develop Kolla in OpenStack, using Heat to provision the
-necessary servers and other resources. If you are familiar with Heat and
-have a correctly configured environment available, this allows deployment
-of a working Kubernetes cluster automatically. The Heat templates are
-available from https://github.com/larsks/heat-kubernetes/. The templates
-require at least Heat 2014.1.3 (earlier versions have a bug that will prevent
-the templates from working).
-
-Here are some simple steps to get things rolling using the Heat templates:
-
-1. Clone the repository:
- ```
- git clone https://github.com/larsks/heat-kubernetes/
- cd heat-kubernetes
- ```
-
-2. Create an appropriate image by running the get_image.sh script in this
- repository. This will generate an image called `fedora-20-k8s.qcow2`.
- Upload this image to Glance. You can also obtain an appropriate image from
- https://fedorapeople.org/groups/heat/kolla/fedora-20-k8s.qcow2
-
-3. Create a file `local.yaml` with settings appropriate to your OpenStack
- environment. It should look something like:
- ```
- parameters:
- server_image: fedora-20-k8s
- ssh_key_name: sdake
-
-
- dns_nameserver: 8.8.8.8
- external_network_id: 6e7e7701-46a0-49c0-9f06-ac5abc79d6ae
- number_of_minions: 1
- server_flavor: m1.large
- ```
- You *must* provide settings for external_network_id and ssh_key_name; these
- are local to your environment. You will probably also need to provide
- a value for server_image, which should be the name (or UUID) of a Fedora 20
- cloud image or derivative.
-
-4. `heat stack-create -f kubecluster.yaml -e local.yaml my-kube-cluster`
-
-5. Determine the ip addresses of your cluster hosts by running:
- ```
- heat output-show my-kube-cluster kube_minions_external
- ```
-
-6. Connect to the minion node with `ssh fedora@${minion-ip}`
-
-7. On the minion node:
- ```
- minion$ git clone http://github.com/stackforge/kolla
- minion$ cd kolla
- minion$ ./tools/start
- ```
-
-Debugging
-==========
-A few commands for debugging the system.
-
-```
-$ sudo docker images
-```
-Lists all images that have been pulled from the upstream kollaglue repository
-thus far. This can be run on the minion during the `./start` operation to
-check on the download progress.
-
-
-```
-$ sudo docker ps -a
-```
-This will show all processes that docker has started. Removing the `-a` will
-show only active processes. This can be run on the minion during the `./start`
-operation to check that the containers are orchestrated.
-
-
-```
-$ sudo docker logs <containerid>
-```
-This shows the logging output of each service in a container. The containerid
-can be obtained via the `docker ps` operation. This can be run on the minion
-during the `./start` operation to debug the container.
-
-```
-$ kubectl get pods
-```
-This lists all pods of which Kubernetes is aware. This can be run on the
-master or minion.
-
-```
-$ sudo systemctl restart kube-apiserver
-$ sudo systemctl restart kube-scheduler
-```
-This command is needed on the master after heat finishes the creation of the
-Kubernetes system (ie: my-kube-cluster is in CREATE_COMPLETE state). This is
-just a workaround for a bug in kubernetes that should be fixed soon.
-
-
-```
-$ journalctl -f -l -xn -u kube-apiserver -u etcd -u kube-scheduler
-```
-This shows log output on the server for the various daemons and can be filed
-in bug reports in the upstream launchpad tracker.
+Directories
+===========
-```
-$ journalctl -f -l -xn -u kubelet.service -u kube-proxy -u docker
-```
-This shows log output on the minion for the various daemons and can be filed
-in bug reports in the upstream launchpad tracker.
+* docker - contains artifacts for use with docker build to build appropriate
+ images
+* k8s - contains service and pod configuration information for Kubernetes
+* tools - contains different tools for interacting with Kolla
+* devenv - A collection of tools and resources for managing a Kolla
+ development environment.
-```
-$ telnet minion_ip 3306
-```
-This shows that the Mariadb service is running on the minions. Output should
-appear as follows
+Getting Involved
+================
-```
-$ telnet 10.0.0.4 3306
-Trying 10.0.0.4...
-Connected to 10.0.0.4.
-Escape character is '^]'.
+Need a feature? Find a bug? Let us know! Contributions are much appreciated
+and should follow the standard [Gerrit workflow][].
-5.5.39-MariaDB-wsrep
-```
+- We communicate using the #tripleo irc channel.
+- File bugs, blueprints, track releases, etc on [Launchpad][].
+- Attend weekly [meetings][].
+- Contribute [code][]
-If the connection closes before mysql responds then the proxy is not properly
-connecting to the database. This can be seen by using jounalctl and watching
-for a connection error on the node that you can't connect to mysql through.
+[Gerrit workflow]: https://wiki.openstack.org/wiki/Gerrit_Workflow
+[Launchpad]: https://launchpad.net/kolla
+[meetings]: https://wiki.openstack.org/wiki/Meetings/Kolla
+[code]: https://github.com/stackforge/kolla
-```
-$ journalctl -f -l -xn -u kube-proxy
-```
-If you can conect though one and not the other there's probably a problem with
-the overlay network. Double check that you're tunning kernel 3.16+ because
-vxlan support is required. If you kernel version is good try restarting
-openvswitch on both nodes. This has usually fixed the connection issues.
+Contributors
+============
-Directories
-===========
+Check out who's [contributing][].
-* docker - contains artifacts for use with docker build to build appropriate
- images
-* k8s - contains service and pod configuration information for Kubernetes
-* tools - contains different tools for interacting with Kolla
+[contributing]: https://github.com/stackforge/kolla/graphs/contributors
diff --git a/devenv/README.md b/devenv/README.md
new file mode 100644
index 0000000000000000000000000000000000000000..406b60e427882b2b132b64927bb138d06172de25
--- /dev/null
+++ b/devenv/README.md
@@ -0,0 +1,143 @@
+A Kolla Cluster with Heat
+=========================
+
+These [Heat][] templates will deploy an *N*-node [Kolla][] cluster,
+where *N* is the value of the `number_of_nodes` parameter you
+specify when creating the stack.
+
+Kolla has recently undergone a considerable design change. The details
+of the design change is addressed in this [spec][]. As part of the
+design change, containers share pid and networking namespaces with
+the Docker host. Therefore, containers no longer connect to a docker0
+bridge and have separate networking from the host. As a result, Kolla
+networking has a configuration similar to:
+
+
+
+Sharing pid and networking namespaces is detailed in the
+[super privileged containers][] concept.
+
+The Kolla cluster is based on Fedora 21, and makes use of the
+[pkilambi/docker][] [COPR][] repository for Docker packages. This
+is because Kolla requires a newer version of Docker not currently
+packaged in Fedora 21.
+
+These templates are designed to work with the Icehouse or Juno
+versions of Heat. If using Icehouse Heat, this [patch][] is
+required to correct a bug with template validation when using the
+"Fn::Join" function).
+
+[heat]: https://wiki.openstack.org/wiki/Heat
+[kolla]: https://launchpad.net/kolla
+[pkilambi/docker]: https://copr.fedoraproject.org/coprs/pkilambi/docker
+[copr]: https://copr.fedoraproject.org/
+[spec]: https://review.openstack.org/#/c/153798/
+[super privileged containers]: http://sdake.io/2015/01/28/an-atomic-upgrade-process-for-openstack-compute-nodes/
+[patch]: https://review.openstack.org/#/c/121139/
+
+Create the Glance Image
+=======================
+
+After cloning the project, run the get-image.sh script from the project's
+devenv directory:
+
+ $ ./get-image.sh
+
+The script will create a Fedora 21 image with the required modifications.
+
+Copy the image to your Glance image store:
+
+ $ glance image-create --name "fedora-21-x86_64" \
+ --file /var/lib/libvirt/images/fedora-21-x86_64 \
+ --disk-format qcow2 --container-format bare \
+ --is-public True --progress
+
+Create the Stack
+================
+
+Copy local.yaml.example to local.yaml and edit the contents to match
+your deployment environment. Here is an example of a customized
+local.yaml:
+
+ parameters:
+ ssh_key_name: admin-key
+ external_network_id: 028d70dd-67b8-4901-8bdd-0c62b06cce2d
+ dns_nameserver: 192.168.200.1
+
+Review the parameters section of kollacluster.yaml for a full list of
+configuration options. **Note:** You must provide values for:
+
+- `ssh_key_name`
+- `external_network_id`
+
+And then create the stack, referencing that environment file:
+
+ $ heat stack-create -f kollacluster.yaml -e local.yaml kolla-cluster
+
+Access the Kolla Nodes
+======================
+
+You can get the ip address of the Kolla nodes using the `heat
+output-show` command:
+
+ $ heat output-show kolla-cluster kolla_node_external_ip
+ "192.168.200.86"
+
+You can ssh into that server as the `fedora` user:
+
+ $ ssh fedora@192.168.200.86
+
+And once logged in you can run Docker commands, etc:
+
+ $ sudo docker images
+
+Debugging
+==========
+A few commands for debugging the system.
+
+```
+$ sudo docker images
+```
+Lists all images that have been pulled from the upstream kollaglue repository
+thus far. This can be run on the node during the `./start` operation to
+check on the download progress.
+
+```
+$ sudo docker ps -a
+```
+This will show all processes that docker has started. Removing the `-a` will
+show only active processes. This can be run on the node during the `./start`
+operation to check that the containers are orchestrated.
+
+```
+$ sudo docker logs <containerid>
+```
+This shows the logging output of each service in a container. The containerid
+can be obtained via the `docker ps` operation. This can be run on the node
+during the `./start` operation to debug the container.
+
+```
+$ sudo systemctl restart docker
+```
+Restarts the Docker service on the node.
+
+```
+$ journalctl -f -l -xn -u docker
+```
+This shows log output on the server for the docker daemon and can be filed
+in bug reports in the upstream launchpad tracker.
+
+```
+$ telnet <NODE_IP> 3306
+```
+You can use telnet to test connectivity to a container. This example demonstrates
+the Mariadb service is running on the node. Output should appear as follows
+
+```
+$ telnet 10.0.0.4 3306
+Trying 10.0.0.4...
+Connected to 10.0.0.4.
+Escape character is '^]'.
+
+5.5.39-MariaDB-wsrep
+```
diff --git a/devenv/kollacluster.yaml b/devenv/kollacluster.yaml
new file mode 100644
index 0000000000000000000000000000000000000000..0c8c65a95602f7f11b33a1ad7ca42fa93199a63e
--- /dev/null
+++ b/devenv/kollacluster.yaml
@@ -0,0 +1,109 @@
+heat_template_version: 2013-05-23
+
+description: >
+ This template will boot a Kolla cluster with one or more
+ nodes (as specified by the number_of_nodes parameter,
+ which defaults to "1").
+
+parameters:
+ #
+ # REQUIRED PARAMETERS
+ #
+ ssh_key_name:
+ type: string
+ description: name of ssh key to be provisioned on the Nova instances
+
+ external_network_id:
+ type: string
+ description: uuid of a network to use for floating ip addresses
+ #
+ # OPTIONAL PARAMETERS
+ #
+ server_image:
+ type: string
+ default: fedora-21-x86_64
+ description: glance image used to boot the Nova instance
+
+ server_flavor:
+ type: string
+ default: m1.small
+ description: flavor to use when booting the Nova instance
+
+ dns_nameserver:
+ type: string
+ description: address of a dns nameserver reachable in your environment
+ default: 8.8.8.8
+
+ number_of_nodes:
+ type: string
+ description: how many kolla nodes to spawn
+ default: 1
+
+ fixed_network_cidr:
+ type: string
+ description: network range for fixed ip network
+ default: 10.0.0.0/24
+
+resources:
+
+ ######################################################################
+ #
+ # network resources. allocate a network and router for our server.
+ # it would also be possible to take advantage of existing network
+ # resources (and have the deployer provide network and subnet ids,
+ # etc, as parameters), but I wanted to minmize the amount of
+ # configuration necessary to make this go.
+ fixed_network:
+ type: "OS::Neutron::Net"
+
+ # This is the subnet on which we will deploy our server eth0.
+ fixed_subnet:
+ type: "OS::Neutron::Subnet"
+ properties:
+ cidr: {get_param: fixed_network_cidr}
+ network_id:
+ get_resource: fixed_network
+ dns_nameservers:
+ - get_param: dns_nameserver
+
+ # create a router attached to the external network provided as a
+ # parameter to this stack.
+ extrouter:
+ type: "OS::Neutron::Router"
+ properties:
+ external_gateway_info:
+ network:
+ get_param: external_network_id
+
+ # attached fixed_subnet to our extrouter router.
+ extrouter_inside:
+ type: "OS::Neutron::RouterInterface"
+ properties:
+ router_id:
+ get_resource: extrouter
+ subnet_id:
+ get_resource:
+ fixed_subnet
+
+ kolla_nodes:
+ type: "OS::Heat::ResourceGroup"
+ depends_on:
+ - extrouter_inside
+ properties:
+ count: {get_param: number_of_nodes}
+ resource_def:
+ type: kollanode.yaml
+ properties:
+ ssh_key_name: {get_param: ssh_key_name}
+ server_image: {get_param: server_image}
+ server_flavor: {get_param: server_flavor}
+ fixed_network_id: {get_resource: fixed_network}
+ fixed_subnet_id: {get_resource: fixed_subnet}
+ external_network_id: {get_param: external_network_id}
+outputs:
+
+ kolla_node_internal_ip:
+ value: {get_attr: [kolla_nodes, kolla_node_ip_eth0]}
+
+ kolla_node_external_ip:
+ value: {get_attr: [kolla_nodes, kolla_node_external_ip]}
diff --git a/devenv/kollanet.png b/devenv/kollanet.png
new file mode 100644
index 0000000000000000000000000000000000000000..cfb0c60140b25710a2979f2e3c6a98f41449752e
Binary files /dev/null and b/devenv/kollanet.png differ
diff --git a/devenv/kollanode.yaml b/devenv/kollanode.yaml
new file mode 100644
index 0000000000000000000000000000000000000000..915c53a0122059410d6112d8c33d59f7971f107f
--- /dev/null
+++ b/devenv/kollanode.yaml
@@ -0,0 +1,199 @@
+heat_template_version: 2013-05-23
+
+description: >
+ This is a nested stack that defines a single Kolla node,
+ based on a Fedora 21 cloud image. This stack is included by
+ a ResourceGroup resource in the parent template (kollacluster.yaml).
+
+parameters:
+
+ server_image:
+ type: string
+ default: fedora-21-x86_64
+ description: glance image used to boot the server
+
+ server_flavor:
+ type: string
+ default: m1.small
+ description: flavor to use when booting the server
+
+ ssh_key_name:
+ type: string
+ description: name of ssh key to be provisioned on our server
+
+ external_network_id:
+ type: string
+ description: uuid of a network to use for kolla host floating ip addresses
+
+ # The following are all generated in the parent template.
+ fixed_network_id:
+ type: string
+ description: Network from which to allocate fixed addresses.
+ fixed_subnet_id:
+ type: string
+ description: Subnet from which to allocate fixed addresses.
+
+resources:
+
+ node_wait_handle:
+ type: "AWS::CloudFormation::WaitConditionHandle"
+
+ node_wait_condition:
+ type: "AWS::CloudFormation::WaitCondition"
+ depends_on:
+ - kolla_node
+ properties:
+ Handle:
+ get_resource: node_wait_handle
+ Timeout: "6000"
+
+ ######################################################################
+ #
+ # security groups. we need to permit network traffic of various
+ # sorts.
+ #
+ secgroup_base:
+ type: "OS::Neutron::SecurityGroup"
+ properties:
+ rules:
+ - protocol: icmp
+ - protocol: tcp
+ port_range_min: 22
+ port_range_max: 22
+
+ # Use by eth1 to permit all traffic to instances.
+ # Let the Neutron container apply security to this traffic.
+ secgroup_all_open:
+ type: "OS::Neutron::SecurityGroup"
+ properties:
+ rules:
+ - protocol: icmp
+ - protocol: tcp
+ - protocol: udp
+
+ secgroup_kolla:
+ type: "OS::Neutron::SecurityGroup"
+ properties:
+ rules:
+ - protocol: tcp
+ port_range_min: 8773
+ port_range_max: 8776
+ - protocol: tcp
+ port_range_min: 6080
+ port_range_max: 6080
+ - protocol: tcp
+ port_range_min: 6081
+ port_range_max: 6081
+ - protocol: tcp
+ port_range_min: 35357
+ port_range_max: 35357
+ - protocol: tcp
+ port_range_min: 5000
+ port_range_max: 5000
+ - protocol: tcp
+ port_range_min: 9191
+ port_range_max: 9191
+ - protocol: tcp
+ port_range_min: 9292
+ port_range_max: 9292
+ - protocol: tcp
+ port_range_min: 9696
+ port_range_max: 9696
+ - protocol: tcp
+ port_range_min: 80
+ port_range_max: 80
+ - protocol: tcp
+ port_range_min: 443
+ port_range_max: 443
+ - protocol: tcp
+ port_range_min: 8000
+ port_range_max: 8000
+ - protocol: tcp
+ port_range_min: 8004
+ port_range_max: 8004
+ - protocol: tcp
+ port_range_min: 8003
+ port_range_max: 8003
+ - protocol: tcp
+ port_range_min: 8080
+ port_range_max: 8080
+ - protocol: tcp
+ port_range_min: 8777
+ port_range_max: 8777
+
+ kolla_node:
+ type: "OS::Nova::Server"
+ properties:
+ image:
+ get_param: server_image
+ flavor:
+ get_param: server_flavor
+ key_name:
+ get_param: ssh_key_name
+ user_data_format: RAW
+ user_data:
+ str_replace:
+ template: |
+ #!/bin/sh
+
+ yum -y upgrade
+
+ yum -y remove NetworkManager
+ chkconfig network on
+
+ # enable dnf command
+ yum -y install dnf dnf-plugins-core
+
+ # Docker packages
+ dnf -y copr enable pkilambi/docker
+ dnf -y install docker
+
+ # Start Docker
+ systemctl enable docker
+ systemctl start docker
+
+ cfn-signal -e0 --data 'OK' -r 'Setup complete' '$WAIT_HANDLE'
+ params:
+ "$WAIT_HANDLE":
+ get_resource: node_wait_handle
+ networks:
+ - port:
+ get_resource: kolla_node_eth0
+ - port:
+ get_resource: kolla_node_eth1
+
+ kolla_node_eth0:
+ type: "OS::Neutron::Port"
+ properties:
+ network_id:
+ get_param: fixed_network_id
+ security_groups:
+ - get_resource: secgroup_base
+ - get_resource: secgroup_kolla
+ fixed_ips:
+ - subnet_id:
+ get_param: fixed_subnet_id
+
+ kolla_node_eth1:
+ type: "OS::Neutron::Port"
+ properties:
+ network_id:
+ get_param: external_network_id
+ security_groups:
+ - get_resource: secgroup_all_open
+
+ kolla_node_floating:
+ type: "OS::Neutron::FloatingIP"
+ properties:
+ floating_network_id:
+ get_param: external_network_id
+ port_id:
+ get_resource: kolla_node_eth0
+
+outputs:
+
+ kolla_node_ip_eth0:
+ value: {get_attr: [kolla_node_eth0, fixed_ips, 0, ip_address]}
+
+ kolla_node_external_ip:
+ value: {get_attr: [kolla_node_floating, floating_ip_address]}
diff --git a/devenv/local.yaml.example b/devenv/local.yaml.example
new file mode 100644
index 0000000000000000000000000000000000000000..b642526e26f47d000dadc179e30a8f13903722ff
--- /dev/null
+++ b/devenv/local.yaml.example
@@ -0,0 +1,3 @@
+parameters:
+ ssh_key_name: <YOUR_NOVA_KEYPAIR>
+ external_network_id: <NEUTRON_EXTERNAL_NET_ID>