diff --git a/specs/centos8-migration.rst b/specs/centos8-migration.rst new file mode 100644 index 0000000000000000000000000000000000000000..548e8e2986d592c6fa73c059a9a7cac6fbaa1cae --- /dev/null +++ b/specs/centos8-migration.rst @@ -0,0 +1,301 @@ +.. + This work is licensed under a Creative Commons Attribution 3.0 Unported + License. + + http://creativecommons.org/licenses/by/3.0/legalcode + +.. + This template should be in ReSTructured text. The filename in the git + repository should match the launchpad URL, for example a URL of + https://blueprints.launchpad.net/kolla/+spec/awesome-thing should be named + awesome-thing.rst . Please do not delete any of the sections in this + template. If you have nothing to say for a whole section, just write: None + For help with syntax, see http://www.sphinx-doc.org/en/stable/rest.html + To test out your formatting, see http://www.tele3.cz/jbar/rest/rest.html + +================== +CentOS 8 Migration +================== + +https://blueprints.launchpad.net/kolla/+spec/centos-rhel-8 +https://blueprints.launchpad.net/kolla-ansible/+spec/centos-rhel-8 + +CentOS 8 is here, and RDO support for CentOS 7 is going away. + +Problem description +=================== + +This spec is largely focussed on the issue of migrating running systems from +CentOS 7 to CentOS 8. We will not in general look at the details of porting +Kolla to CentOS 8 here. + +Kolla images +------------ + +Train is the last release of OpenStack to support Python 2. Since CentOS 7 +provides only limited support for Python 3, RDO plans to support only CentOS 8 +for the Ussuri cycle. + +While RDO Train initially only provided Python 2 based RPMs for CentOS 7, it is +expected that Python 3 based Train RPMs will be made available for CentOS 8. We +should therefore support building Train images with either CentOS 7 or CentOS 8 +as a base. Both sets of images should be published to Dockerhub. The image +naming scheme will need to take account of the OS version, which it currently +does not. + +Kolla Ansible +------------- + +There is no supported upgrade path between CentOS 7 and 8 hosts - a full +reinstall is necessary. This provides us with one release (Train) that supports +both CentOS 7 and 8, with which we could perform a rolling reinstallation of +hosts. The cloud should continue to operate in this mixed 7/8 mode during the +migration. The following diagram shows an example with 6 hosts h1 to h6 being +migrated in batches of two:: + + h1 h2 h3 h4 h5 h6 + ------------------- + t0 | c7 c7 c7 c7 c7 c7 + t1 | c8 c8 c7 c7 c7 c7 + t2 | c8 c8 c8 c8 c7 c7 + t3 | c8 c8 c8 c8 c8 c8 + +Proposed change +=============== + +Kolla images - Ussuri +--------------------- + +Support for building CentOS 8 based container images will be added to Kolla +during the Ussuri cycle. Initially, this will exist in parallel with CentOS 7 +support, and be controlled via the ``distro-package-manager`` configuration +option, which takes its default value from the base image tag (``7.*`` for +``yum``, ``8.*`` for ``dnf``). + +New build and publishing CI jobs will be added for CentOS 8. We will use an +alternative Docker image tag to tag CentOS 8 images on Dockerhub: +``master-centos8``. An example image name is +``kolla/centos-binary-base:master-centos8``. + +Once this work is complete, support for CentOS 7 will be removed. CentOS 8 +images will then be tagged as ``master`` (or ``ussuri`` after branching). This +two-phase approach allows for a clean backport to Train. A downside is that +``master-centos8`` tags will continue to exist in Dockerhub after the +transition, which could be confusing for users. + +Kolla images - Train +-------------------- + +The CentOS 8 support developed for Ussuri will be backported to the +stable/train branch. Since these changes are significant, we should advertise +this clearly to users who might not be expecting it on a stable branch. +Train images will be built and published for CentOS 7 and 8, with CentOS 8 +images using a tag of ``train-centos8``. + +Kolla Ansible - Ussuri +---------------------- + +Support for deploying CentOS 8 based container images will be added to Kolla +Ansible during the Ussuri cycle. Initially, this will exist in parallel with +CentOS 7 support, and be controlled via Ansible host facts about the host OS +distribution. We will deploy CentOS 7 containers on CentOS 7 hosts, and CentOS +8 containers on CentOS 8 hosts to avoid kernel and userspace incompatibility +issues. This also aligns with the `migration plan +<http://lists.openstack.org/pipermail/openstack-discuss/2019-November/011133.html>`__ +proposed by Tripleo. + +During the migration, we may have a mix of CentOS 7 hosts running CentOS 7 +containers, and CentOS 8 hosts running CentOS 8 containers. + +The image tag used will need to depend on the host OS. We have a hierarchy of +image tag variables, for example: the ``nova-compute`` container uses a tag +defined by ``nova_compute_tag``, which defaults to ``nova_tag``, which defaults +to ``openstack_release``. + +To preserve the usage of ``openstack_release``, a new intermediate variable +will be introduced. In this new scheme, the ``nova-compute`` container uses a +tag defined by ``nova_compute_tag``, which defaults to ``nova_tag``, which +defaults to ``openstack_tag``, which defaults to ``openstack_release ~ +'-centos8'`` on CentOS 8, or ``openstack_release`` otherwise. Providing this +suffix as a variable would help users who are using more fine-grained image +tags (e.g. ``nova_compute_tag``). + +Once this work is complete, support for CentOS 7 will be removed. The +intermediate variable ``openstack_tag`` will remain, but will always default to +``openstack_release``. + +Kolla Ansible - Train +--------------------- + +The CentOS 8 support developed for Ussuri will be backported to the +stable/train branch. Since these changes are significant, we should advertise +this clearly to users who might not be expecting it on a stable branch. + +Kolla Ansible - Rolling Migration +--------------------------------- + +In order to keep the cloud functional during the migration, the changes will be +rolled through in batches. The size and scope of the batches are to be +determined by the operator, but we should provide some guidance on selecting +batches safely to avoid losing quorum of clustered services or reducing service +availability below some minimum threshold. Provisioning of the new OS is out of +the scope of this discussion, but could be handled by Kayobe or another tool. + +We should treat each batch as a removal of a set of hosts followed by an +addition of new set of hosts. This is a good opportunity for us to tidy up +these procedures and fill in gaps where necessary. + +To remove a batch of hosts: + +#. Migrate running OpenStack resources from the hosts to other hosts which are + not in the batch. This includes compute instances (VMs), storage (Swift + disks), network services (DHCP & L3 agents etc.). Ideally this would be + automated, but should at least be discussed in documentation +#. Cleanly disable (if necessary) and shutdown running OpenStack services +#. Shutdown hosts + +The hosts should then be provisioned with CentOS 8 - as mentioned this is out +of scope. Then to add a batch of hosts: + +#. Provision CentOS 8 to hosts (out of scope) +#. Bootstrap hosts +#. Deploy services +#. Validate new hosts + +We may wish to adopt existing data on these hosts, e.g. for Swift a full sync +from empty for each node would take a long time. For systems where Docker +volumes are stored on a separate disk or partition, it may be possible to +preserve these. There may be some corner cases where this does not work well, +and we should check for these and advise in documentation. + +For the majority of these tasks it should be possible to use the ``--limit`` +argument to speed up the process, but this should be investigated and verified. + +Stable upgrades +--------------- + +Where possible, we should aim to use the same versions of packages on CentOS 7 +and 8 for Train. Upgrades should wait until Ussuri. In some cases this may not +be possible, and we should make it clear in release notes where services will +be upgraded in Train. + +Given that a rolled migration may take a significant amount of time to +complete, we should consider that services may be partially upgraded. In cases +where this may cause a problem, it should be called out in the documentation, +with any available mitigation (e.g. disable during migration). + +Feature removal +--------------- + +Due to differences between CentOS 7 and 8, it may not be possible or practical +to support all existing features. We should bear in mind that this will apply +to the Train release, and continue to support these features on CentOS 7. + +Ceph support +^^^^^^^^^^^^ + +Migrating a Ceph cluster deployed by Kolla Ansible to CentOS 8 would represent +a significant challenge. Ceph deployment support has been deprecated since the +Stein release, and this is a good point to remove that support. + +SCSI target daemon +^^^^^^^^^^^^^^^^^^ + +For CentOS 7, the SCSI target daemon (``tgtd``) is provided by EPEL. For CentOS +8, this is no longer available, probably due to a preference for the kernel +SCSI target, `LIO <http://linux-iscsi.org/wiki/LIO>`__. Due to the requirement +to reinstall, this migration provides a good opportunity to stop supporting +``tgtd`` for CentOS. This will affect some Cinder volume drivers, including the +LVM backend. ``tgtd`` will still be supported on Debian/Ubuntu. Note that no +changes are required for the iSCSI initiator. + +Security impact +--------------- + +There should be no security impact due to this change. + +Performance Impact +------------------ + +There should be no performance impact due to this change. + +Alternatives +------------ + +CentOS 7 to 8 upgrade +^^^^^^^^^^^^^^^^^^^^^ + +There are various documented procedures for upgrading from CentOS 7 to 8 +without a reinstall. These typically come with various disclaimers, and are not +recommended for production environments. + +Image naming +^^^^^^^^^^^^ + +Rather than using a different tag to mark CentOS 8 images, it would be possible +to use a base distro of ``centos8``. e.g. ``kolla/centos8-binary-base:master``. +The reason for not doing this is that there is a reasonable amount of logic +both in Kolla and Kolla Ansible that uses the ``base_distro`` which would need +to be updated. + +Don't publish master-centos8 tag +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +During the transition period, we could manage with always building images for +CentOS 8 on master, thus avoiding the need for a temporary ``master-centos8`` +tag. + +Implementation +============== + +Assignee(s) +----------- + +Primary assignees: + Marcin Juszkiewicz (hrw) + Mark Goddard (mgoddard) + Michal Nasiadka (mnasiadka) + Radoslaw Piliszek (yoctozepto) + +Work Items +---------- + +Kolla +^^^^^ + +* Support building CentOS 8 images in Ussuri +* Build and publish both CentOS 7 and 8 images in CI +* Backport CentOS 8 support to Train +* Remove CentOS 7 image support in Ussuri +* Update documentation (image tags etc.) + +Kolla Ansible +^^^^^^^^^^^^^ + +* Support deploying CentOS 8 images in Ussuri +* Test (separately) deployment to both CentOS 7 and 8 in CI +* Backport CentOS 8 support to Train +* Test migration from CentOS 7 to 8 on Train in CI +* Test upgrade from Train to Ussuri on CentOS 8 +* Remove CentOS 7 support in Ussuri +* Update documentation (migration) + +Testing +======= + +This will be tested by parallel build and deploy jobs for CentOS 7 and 8, in +addition to a job testing migration from CentOS 7 to 8. Significant manual +testing will be required to explore edge cases associated with the migration. + +Documentation Impact +==================== + +* Building CentOS 8 images +* Deploying to CentOS 8 +* Image tags for CentOS 8 +* Migration procedure covered in detail + +References +========== + +* `CentOS 8 release notes <https://wiki.centos.org/Manuals/ReleaseNotes/CentOS8.1905>`__