Apply public firewalld rules immediately

Previously, firewalld rules were applied to configuration, then
firewalld reloaded to pick up all the new rules. Reloading firewalld can
be disruptive because it sets all chains to a DROP policy while building
up its firewall rules, breaking open connections.

This change switches to applying rules both permanently (to config) and
immediately, such that no reload is required.

Change-Id: I8e48b7827b33bdd2061d0e89c905bea8e29f60e8

ansible/roles/haproxy-config/tasks/main.yml

@@ -24,8 +24,9 @@
 
 - name: "Configuring firewall for {{ project_name }}"
   firewalld:
-    offline: "yes"
-    permanent: "yes"
+    immediate: true
+    offline: true
+    permanent: true
     port: "{{ item.value.port }}/tcp"
     state: "enabled"
     zone: "{{ external_api_firewalld_zone }}"
@@ -38,5 +39,3 @@
     - enable_external_api_firewalld | bool
     - kolla_action != "config"
   with_dict: "{{ project_services | extract_haproxy_services }}"
-  notify:
-    - "Reload firewalld"

ansible/roles/loadbalancer/handlers/main.yml

 ---
-- name: Reload firewalld
-  become: True
-  service:
-    name: "firewalld"
-    state: reloaded
-
 # NOTE(yoctozepto): this handler dance is to ensure we delay restarting master
 # keepalived and haproxy which control VIP address until we have working backups.
 # This could be improved by checking if backup keepalived do not report FAULT state.

releasenotes/notes/firewalld-immediate-c2abf09977c455a9.yaml

+---
+features:
+  - |
+    Modifies public API firewalld rules to be applied immediately to a running
+    firewalld service. This requires firewalld to be running, but avoids
+    reloading firewalld, which is disruptive due to the way in which firewalld
+    builds its firewall chains.