Ironic provides a feature to allow instance images to be served from a
local HTTP server [1]. This is the same server used for PXE images with
iPXE. This does not work currently because the ironic_ipxe container
does not have access to /var/lib/ironic/images (ironic docker volume),
where the images are cached. Note that to make use of this feature, the
following is required in ironic.conf:

[agent]
image_download_source = http

This change fixes the issue by giving ironic_ipxe container access to
the ironic volume.

[1] https://docs.openstack.org/ironic/latest/admin/interfaces/deploy.html#deploy-with-custom-http-servers

Change-Id: I501d02cfd40fbacea32d551c3912640c5661d821
Closes-Bug: #1856194

As part of the effort to implement Ansible code linting in CI
(using ansible-lint) - we need to implement recommendations from
ansible-lint output [1].

One of them is to stop using local_action in favor of delegate_to -
to increase readability and and match the style of typical ansible
tasks.

[1]: https://review.opendev.org/694779/

Partially implements: blueprint ansible-lint

Change-Id: I46c259ddad5a6aaf9c7301e6c44cd8a1d5c457d3

If ironic inspector starts up before ironic-api is accessible, we see
the following error:

AttributeError: StrictVersion instance has no attribute 'version'

This actually prevents startup of ironic inspector.

Example: http://paste.openstack.org/show/756342/

The underlying issue is in python-ironicclient:
https://storyboard.openstack.org/#!/story/2006393.

The workaround employed here is to wait for ironic-api to become
available before starting ironic-inspector.

Change-Id: I674982eba5082fdc951cdcb8247d706278c64c1a
Closes-Bug: #1839866

Introduce kolla_address filter.
Introduce put_address_in_context filter.

Add AF config to vars.

Address contexts:
- raw (default): <ADDR>
- memcache: inet6:[<ADDR>]
- url: [<ADDR>]

Other changes:

globals.yml - mention just IP in comment

prechecks/port_checks (api_intf) - kolla_address handles validation

3x interface conditional (swift configs: replication/storage)

2x interface variable definition with hostname
(haproxy listens; api intf)

1x interface variable definition with hostname with bifrost exclusion
(baremetal pre-install /etc/hosts; api intf)

neutron's ml2 'overlay_ip_version' set to 6 for IPv6 on tunnel network

basic multinode source CI job for IPv6

prechecks for rabbitmq and qdrouterd use proper NSS database now

MariaDB Galera Cluster WSREP SST mariabackup workaround
(socat and IPv6)

Ceph naming workaround in CI
TODO: probably needs documenting

RabbitMQ IPv6-only proto_dist

Ceph ms switch to IPv6 mode

Remove neutron-server ml2_type_vxlan/vxlan_group setting
as it is not used (let's avoid any confusion)
and could break setups without proper multicast routing
if it started working (also IPv4-only)

haproxy upgrade checks for slaves based on ipv6 addresses

TODO:

ovs-dpdk grabs ipv4 network address (w/ prefix len / submask)
not supported, invalid by default because neutron_external has no address
No idea whether ovs-dpdk works at all atm.

ml2 for xenapi
Xen is not supported too well.
This would require working with XenAPI facts.

rp_filter setting
This would require meddling with ip6tables (there is no sysctl param).
By default nothing is dropped.
Unlikely we really need it.

ironic dnsmasq is configured IPv4-only
dnsmasq needs DHCPv6 options and testing in vivo.

KNOWN ISSUES (beyond us):

One cannot use IPv6 address to reference the image for docker like we
currently do, see: https://github.com/moby/moby/issues/39033
(docker_registry; docker API 400 - invalid reference format)
workaround: use hostname/FQDN

RabbitMQ may fail to bind to IPv6 if hostname resolves also to IPv4.
This is due to old RabbitMQ versions available in images.
IPv4 is preferred by default and may fail in the IPv6-only scenario.
This should be no problem in real life as IPv6-only is indeed IPv6-only.
Also, when new RabbitMQ (3.7.16/3.8+) makes it into images, this will
no longer be relevant as we supply all the necessary config.
See: https://github.com/rabbitmq/rabbitmq-server/pull/1982

For reliable runs, at least Ansible 2.8 is required (2.8.5 confirmed
to work well). Older Ansible versions are known to miss IPv6 addresses
in interface facts. This may affect redeploys, reconfigures and
upgrades which run after VIP address is assigned.
See: https://github.com/ansible/ansible/issues/63227

Bifrost Train does not support IPv6 deployments.
See: https://storyboard.openstack.org/#!/story/2006689



Change-Id: Ia34e6916ea4f99e9522cd2ddde03a0a4776f7e2c
Implements: blueprint ipv6-control-plane
Signed-off-by: Radosław Piliszek <radoslaw.piliszek@gmail.com>

Sometimes as cloud admins, we want to only update code that is running
in a cloud.  But we dont need to do anything else.  Make an action in
kolla-ansible that allows us to do that.

Change-Id: I904f595c69f7276e71692696471e32fd1f88e6e8
Implements: blueprint deploy-containers-action

Use upstream Ansible modules for registration of services, endpoints,
users, projects, roles, and role grants.

Change-Id: I7c9138d422cc91c177fd8992347176bb54156b5a

In the Train cycle, ironic added a [nova] section to its configuration.
This is used to configure access to Nova API, for sending power state
callbacks.

This change adds the [nova] section to ironic.conf.

Change-Id: Ib891af1db2a2c838c887e858ea0721f5e6a4fab0
Closes-Bug: #1843070

The ironic configuration in ironic.conf uses several options which have
been removed in the Train cycle:

[glance] glance_api_servers was removed in https://review.opendev.org/#/c/665929.
[neutron] url was removed in https://review.opendev.org/#/c/672971.

We should use the endpoint catalog instead of specifying the endpoint
for both of these, and also ironic inspector. region_name and
valid_interfaces have been added for that purpose.

Other options are deprecated.

[conductor] api_url: Use [service_catalog] section to lookup ironic API
endpoint instead.

[inspector] enabled: No longer used.

Change-Id: If07c4ff9bfea7d780aeff5c3295a0ace7d10ecdc
Closes-Bug: #1843067

Upstream ironic went from $net_default_ip to $net_default_mac in
ironic/drivers/modules/master_grub_cfg.txt with
https://review.opendev.org/#/c/578959/

This commit makes the same change for
ansible/roles/ironic/templates/ironic_pxe_uefi.default.j2

Using $net_default_ip breaks ironic standalone deployments with
[dhcp]dhcp_provider = none

Change-Id: I2ca9a66d2bdb0aab5cd9936c8be8206e6ade3bd5
Closes-Bug: 1842078

Change-Id: Ic80dbe1f4f7289fe2c2143125a381cec4586f7ef
Closes-Bug: #1841908

This commit adds the functionality for an operator to specify
their own trusted CA certificate file for interacting with the
Keystone API.

Implements: blueprint support-trusted-ca-certificate-file
Change-Id: I84f9897cc8e107658701fb309ec318c0f805883b

After all of the discussions we had on
"https://review.opendev.org/#/c/670626/2", I studied all projects that
have an "oslo_messaging" section. Afterwards, I applied the same method
that is already used in "oslo_messaging" section in Nova, Cinder, and
others. This guarantees that we have a consistent method to
enable/disable notifications across projects based on components (e.g.
Ceilometer) being enabled or disabled. Here follows the list of
components, and the respective changes I did.

* Aodh:
The section is declared, but it is not used. Therefore, it will
be removed in an upcomming PR.

* Congress:
The section is declared, but it is not used. Therefore, it will
be removed in an upcomming PR.

* Cinder:
It was already properly configured.

* Octavia:
The section is declared, but it is not used. Therefore, it will
be removed in an upcomming PR.

* Heat:
It was already using a similar scheme; I just modified it a little bit
to be the same as we have in all other components

* Ceilometer:
Ceilometer publishes some messages in the rabbitMQ. However, the
default driver is "messagingv2", and not ''(empty) as defined in Oslo;
these configurations are defined in ceilometer/publisher/messaging.py.
Therefore, we do not need to do anything for the
"oslo_messaging_notifications" section in Ceilometer

* Tacker:
It was already using a similar scheme; I just modified it a little bit
to be the same as we have in all other components

* Neutron:
It was already properly configured.

* Nova
It was already properly configured. However, we found another issue
with its configuration. Kolla-ansible does not configure nova
notifications as it should. If 'searchlight' is not installed (enabled)
the 'notification_format' should be 'unversioned'. The default is
'both'; so nova will send a notification to the queue
versioned_notifications; but that queue has no consumer when
'searchlight' is disabled. In our case, the queue got 511k messages.
The huge amount of "stuck" messages made the Rabbitmq cluster
unstable.

https://bugzilla.redhat.com/show_bug.cgi?id=1478274
https://bugs.launchpad.net/ceilometer/+bug/1665449

* Nova_hyperv:
I added the same configurations as in Nova project.

* Vitrage
It was already using a similar scheme; I just modified it a little bit
to be the same as we have in all other components

* Searchlight
I created a mechanism similar to what we have in AODH, Cinder, Nova,
and others.

* Ironic
I created a mechanism similar to what we have in AODH, Cinder, Nova,
and others.

* Glance
It was already properly configured.

* Trove
It was already using a similar scheme; I just modified it a little bit
to be the same as we have in all other components

* Blazar
It was already using a similar scheme; I just modified it a little bit
to be the same as we have in all other components

* Sahara
It was already using a similar scheme; I just modified it a little bit
to be the same as we have in all other components

* Watcher
I created a mechanism similar to what we have in AODH, Cinder, Nova,
and others.

* Barbican
I created a mechanism similar to what we have in Cinder, Nova,
and others. I also added a configuration to 'keystone_notifications'
section. Barbican needs its own queue to capture events from Keystone.
Otherwise, it has an impact on Ceilometer and other systems that are
connected to the "notifications" default queue.

* Keystone
Keystone is the system that triggered this work with the discussions
that followed on https://review.opendev.org/#/c/670626/2

. After a long
discussion, we agreed to apply the same approach that we have in Nova,
Cinder and other systems in Keystone. That is what we did. Moreover, we
introduce a new topic "barbican_notifications" when barbican is
enabled. We also removed the "variable" enable_cadf_notifications, as
it is obsolete, and the default in Keystone is CADF.

* Mistral:
It was hardcoded "noop" as the driver. However, that does not seem a
good practice. Instead, I applied the same standard of using the driver
and pushing to "notifications" queue if Ceilometer is enabled.

* Cyborg:
I created a mechanism similar to what we have in AODH, Cinder, Nova,
and others.

* Murano
It was already using a similar scheme; I just modified it a little bit
to be the same as we have in all other components

* Senlin
It was already using a similar scheme; I just modified it a little bit
to be the same as we have in all other components

* Manila
It was already using a similar scheme; I just modified it a little bit
to be the same as we have in all other components

* Zun
The section is declared, but it is not used. Therefore, it will
be removed in an upcomming PR.

* Designate
It was already using a similar scheme; I just modified it a little bit
to be the same as we have in all other components

* Magnum
It was already using a similar scheme; I just modified it a little bit
to be the same as we have in all other components

Closes-Bug: #1838985

Change-Id: I88bdb004814f37c81c9a9c4e5e491fac69f6f202
Signed-off-by: Rafael Weingärtner <rafael@apache.org>

Docker has no restart policy named 'never'. It has 'no'.
This has bitten us already (see [1]) and might bite us again whenever
we want to change the restart policy to 'no'.

This patch makes our docker integration honor all valid restart policies
and only valid restart policies.
All relevant docker restart policy usages are patched as well.

I added some FIXMEs around which are relevant to kolla-ansible docker
integration. They are not fixed in here to not alter behavior.

[1] https://review.opendev.org/667363



Change-Id: I1c9764fb9bbda08a71186091aced67433ad4e3d6
Signed-off-by: Radosław Piliszek <radoslaw.piliszek@gmail.com>

The ironic inspector iPXE configuration includes the following kernel
argument:

initrd=agent.ramdisk

However, the ramdisk is actually called ironic-agent.initramfs, so the
argument should be:

initrd=ironic-agent.initramfs

In BIOS boot mode this does not cause a problem, but for compute nodes
with UEFI enabled, it seems to be more strict about this, and fails to
boot.

Change-Id: Ic84f3b79fdd3cd1730ca2fb79c11c7a4e4d824de
Closes-Bug: #1836375

A common class of problems goes like this:

* kolla-ansible deploy
* Hit a problem, often in ansible/roles/*/tasks/bootstrap.yml
* Re-run kolla-ansible deploy
* Service fails to start

This happens because the DB is created during the first run, but for some
reason we fail before performing the DB sync. This means that on the second run
we don't include ansible/roles/*/tasks/bootstrap_service.yml because the DB
already exists, and therefore still don't perform the DB sync. However this
time, the command may complete without apparent error.

We should be less careful about when we perform the DB sync, and do it whenever
it is necessary. There is an argument for not doing the sync during a
'reconfigure' command, although we will not change that here.

This change only always performs the DB sync during 'deploy' and
'reconfigure' commands.

Change-Id: I82d30f3fcf325a3fdff3c59f19a1f88055b566cc
Closes-Bug: #1823766
Closes-Bug: #1797814

Currently, we have a lot of logic for checking if a handler should run,
depending on whether config files have changed and whether the
container configuration has changed. As rm_work pointed out during
the recent haproxy refactor, these conditionals are typically
unnecessary - we can rely on Ansible's handler notification system
to only trigger handlers when they need to run. This removes a lot
of error prone code.

This patch removes conditional handler logic for all services. It is
important to ensure that we no longer trigger handlers when unnecessary,
because without these checks in place it will trigger a restart of the
containers.

Implements: blueprint simplify-handlers

Change-Id: I4f1aa03e9a9faaf8aecd556dfeafdb834042e4cd

Many tasks that use Docker have become specified already, but
not all. This change ensures all tasks that use the following
modules have become:

* kolla_docker
* kolla_ceph_keyring
* kolla_toolbox
* kolla_container_facts

It also adds become for 'command' tasks that use docker CLI.

Change-Id: I4a5ebcedaccb9261dbc958ec67e8077d7980e496

When ansible goes in to a loop, by default it prints all the keys for
the item it is looping over. Some roles, when setting up the databases,
iterate over an object that includes the database password.

Override the loop label to hide everything but the database name.

Change-Id: I336a81a5ecd824ace7d40e9a35942a1c853554cd

When integrating 3rd party component into openstack with kolla-ansible,
maybe have to mount some extra volumes to container.

Change-Id: I69108209320edad4c4ffa37dabadff62d7340939
Implements: blueprint support-extra-volumes

With Docker CE, the daemon sets the default policy of the iptables
FORWARD chain to DROP. This causes problems for provisioning bare metal
servers when ironic inspector is used with the 'iptables' PXE filter.
It's not entirely clear why these two things interact in this way,
but switching to the 'dnsmasq' filter works around the issue, and is
probably a good move anyway because it is more efficient.

We have added a migration task here to flush and remove the ironic-inspector
iptables chain since inspector does not do this itself currently.

Change-Id: Iceed5a096819203eb2b92466d39575d3adf8e218
Closes-Bug: #1823044

Several config file permissions are incorrect on the host. In general,
files should be 0660, and directories and executables 0770.

Change-Id: Id276ac1864f280554e98b937f2845bb424d521de
Closes-Bug: #1821579

When adding the rolling upgrade support, some upgrade procedures were
modified to pull images explicitly. This is done inconsistently between
services, and is a change in behaviour from Rocky and earlier releases.

This change removes all image pulling from upgrade tasks.

Change-Id: Id0fed17714235e1daed60b83b1f30620f097eb97

This allows ironic service endpoints to use custom hostnames, and adds the
following variables:

* ironic_internal_fqdn
* ironic_external_fqdn
* ironic_inspector_internal_fqdn
* ironic_inspector_external_fqdn

These default to the old values of kolla_internal_fqdn or
kolla_external_fqdn.

This also adds ironic_api_listen_port and ironic_inspector_listen_port
options, which default to ironic_api_port and ironic_inspector_port for
backward compatibility.

These options allow the user to differentiate between the port the
service listens on, and the port the service is reachable on. This is
useful for external load balancers which live on the same host as the
service itself.

Change-Id: I45b175e85866b4cfecad8451b202a5a27f888a84
Implements: blueprint service-hostnames

We're duplicating code to build the keystone URLs in nearly every
config, where we've already done it in group_vars. Replace the
redundancy with a variable that does the same thing.

Change-Id: I207d77870e2535c1cdcbc5eaf704f0448ac85a7a

Adds a new flag, 'enable_openstack_core', which defaults to 'yes'.
Setting this flag to 'no' will disable the core OpenStack services,
including Glance, Heat, Horizon, Keystone, Neutron, and Nova.

Improves the default configuration of OpenStack Ironic when used in
standalone mode. In particular, configures a noauth mode when Keystone
is disabled, and allows the iPXE server to be used for provisioning as
well as inspection if Neutron is disabled.

Documentation for standalone ironic will be updated separately.

This patch was developed and tested using Bikolla [1].

[1] https://github.com/markgoddard/bikolla

Change-Id: Ic47f5ad81b8126a51e52a445097f7950dba233cd
Implements: blueprint standalone-ironic

This allows neutron service endpoints to use custom hostnames, and adds the
following variables:

* neutron_internal_fqdn
* neutron_external_fqdn

These default to the old values of kolla_internal_fqdn or
kolla_external_fqdn.

This also adds a neutron_server_listen_port option, which defaults to
neutron_server_port for backward compatibility.

This option allow the user to differentiate between the port the
service listens on, and the port the service is reachable on. This is
useful for external load balancers which live on the same host as the
service itself.

Change-Id: I87d7387326b6eaa6adae1600b48d480319d10676
Implements: blueprint service-hostnames

This allows glance service endpoints to use custom hostnames, and adds the
following variables:

* glance_internal_fqdn
* glance_external_fqdn

These default to the old values of kolla_internal_fqdn or
kolla_external_fqdn.

This also adds a glance_api_listen_port option, which defaults to
glance_api_port for backward compatibility.

This option allow the user to differentiate between the port the
service listens on, and the port the service is reachable on. This is
useful for external load balancers which live on the same host as the
service itself.

Change-Id: Icb91f728533e2db1908b23dabb0501cf9f8a2b75
Implements: blueprint service-hostnames

The ironic TFTP server should be accessed via the internal API network.
For ironic inspector, dnsmasq.conf advertises this correctly:

dhcp-option=option:tftp-server,'api_interface_address'
dhcp-option=option:server-ip-address,'api_interface_address'

However, ironic conductor does not set the [pxe] tftp_server variable.
This means the TFTP server advertised gets the default value of $my_ip,
which is set by
https://docs.openstack.org/oslo.utils/latest/reference/netutils.html#oslo_utils.netutils.get_my_ipv4,
typically the source IP for the default route.

This change sets [pxe] tftp_server to 'api_interface_address'.

Change-Id: Ic3e688b3f2b92ad9515322f49cd5f4f29d763e49
Closes-Bug: #1808347

Change-Id: I81d56dd850c18298987376740ccfea05fe59267f
Refer: https://storyboard.openstack.org/#!/story/2001842

With this change, an operator may be able to stop a
service container without stopping all services in a host.
This change is the starting point to start
fast-forward upgrades support.
In next changes new flags will be introducced to disable
stop dataplane services during upgrades.

Change-Id: Ifde7a39d7d8596ef0d7405ecf1ac1d49a459d9ef
Implements: blueprint support-stop-containers

The dnsmasq PXE filter [1] provides far better scalability than the
iptables filter typically used. Inspector manages files in a dhcp-hostsdir
directory that is watched by dnsmasq via inotify. Dnsmasq then either
whitelists or blacklists MAC addresses based on the contents of these
files.

This change adds a new variable, ironic_inspector_pxe_filter, that can
be used to configure the PXE filter for ironic inspector. Currently
supported values are 'iptables' and 'dnsmasq', with 'iptables' being the
default for backwards compatibility.

[1]
https://docs.openstack.org/ironic-inspector/latest/admin/dnsmasq-pxe-filter.html

Implements: blueprint ironic-inspector-dnsmasq-pxe-filter
Change-Id: I73cae9c33b49972342cf1984372a5c784df5cbc2

If the [processing] ramdisk_logs_dir option is set, logs returned by the
ironic inspection ramdisk following hardware inspection will be stored
at that location. This enables easier debugging if inspection fails.

Change-Id: I36bdf75c04b088b67b5f54fdf20251c10bdddb63

Known kernel modules are:
- dm-multipath (for multipathd)
- ip_vs (for keepalived)
- iscsi_tcp (for ironic-conductor)
- openvswitch (for openvswitch-vswitchd)

Change-Id: I1841ec30cde142c8019830ad3190847dfe493eb9

The firewall section has been renamed in upstream ironic inspector:

https://github.com/openstack/ironic-inspector/commit/7b2758546376394777c7d36c8f5bb62ff7ef3830

Consequently the iptables pxe filter does not work if the actual
dnsmasq interface name differs from the default (br-ctlplane), as can
be seen from this snippet of iptables-save output:

-A INPUT -i br-ctlplane -p udp -m udp --dport 67 -j ironic-inspector

Change-Id: Ic1d08b85e0b5992fbee489f2f9fd174982b5d493

Having all services in one giant haproxy file makes altering
configuration for a service both painful and dangerous. Each service
should be configured with a simple set of variables and rendered with a
single unified template.

Available are two new templates:

* haproxy_single_service_listen.cfg.j2: close to the original style, but
only one service per file
* haproxy_single_service_split.cfg.j2: using the newer haproxy syntax
for separated frontend and backend

For now the default will be the single listen block, for ease of
transition.

Change-Id: I6e237438fbc0aa3c89a3c8bd706a53b74e71904b

Now kolla dev mode only support clone master branch from git,
add version tag to support clone dedicated branch.

Change-Id: I88de238e5dc7461ba0662a3ecea9a2d80fd0db60

Option auth_uri from group keystone_authtoken is deprecated[1].
Use option www_authenticate_uri from group keystone_authtoken.

[1]https://review.openstack.org/#/c/508522/



Co-Authored-By: confi-surya <singh.surya64mnnit@gmail.com>
Change-Id: Ifd8527d404f1df807ae8196eac2b3849911ddc26
Closes-Bug: #1761907

The variable 'ironic_dnsmasq_interface' is used to configure the interface
used by the ironic inspector dnsmasq service for DHCP on the inspection
network. It is being used correctly in inspector.conf, but not in the
dnsmasq configuration file, which uses api_interface. This change modifies
the dnsmasq configuration file to also use ironic_dnsmasq_interface.

Change-Id: I7670544f4bc41c93ac1d081486502f9ffb8f2286
Closes-Bug: #1785574

Ironic requires the Keystone credentials to communicate with Cinder if
booting from volume.

Change-Id: Id9a90d986e391e84c8ad918af371a5aef33a3524
Closes-Bug: #1785201

This allows you to append additional kernel parameters
to the kernel used for inspection.

Change-Id: Ibc851145a3ffdaaad526ef999c8f024bd222dd5b