-
Doug Szumski authored
In services which use the Apache HTTP server to service HTTP requests, there exists a TimeOut directive [1] which defaults to 60 seconds. APIs which come under heavy load, such as Cinder, can sometimes exceed this which results in a HTTP 504 Gateway timeout, or similar. However, the request can still be serviced without error. For example, if Nova calls the Cinder API to detach a volume, and this operation takes longer than the shortest of the two timeouts, Nova will emit a stack trace with a 504 Gateway timeout. At some time later, the request to detach the volume will succeed. The Nova and Cinder DBs then become out-of-sync with each other, and frequently DB surgery is required. Although strictly this category of bugs should be fixed in OpenStack services, it is not realistic to expect this to happen in the short term. Therefore, this change makes it easier to set the Apache HTTP timeout via a new variable. An example of a related bug is here: https://bugs.launchpad.net/nova/+bug/1888665 Whilst this timeout can currently be set by overriding the WSGI config for individual services, this change makes it much easier. Change-Id: Ie452516655cbd40d63bdad3635fd66693e40ce34 Closes-Bug: #1917648
Doug Szumski authoredIn services which use the Apache HTTP server to service HTTP requests, there exists a TimeOut directive [1] which defaults to 60 seconds. APIs which come under heavy load, such as Cinder, can sometimes exceed this which results in a HTTP 504 Gateway timeout, or similar. However, the request can still be serviced without error. For example, if Nova calls the Cinder API to detach a volume, and this operation takes longer than the shortest of the two timeouts, Nova will emit a stack trace with a 504 Gateway timeout. At some time later, the request to detach the volume will succeed. The Nova and Cinder DBs then become out-of-sync with each other, and frequently DB surgery is required. Although strictly this category of bugs should be fixed in OpenStack services, it is not realistic to expect this to happen in the short term. Therefore, this change makes it easier to set the Apache HTTP timeout via a new variable. An example of a related bug is here: https://bugs.launchpad.net/nova/+bug/1888665 Whilst this timeout can currently be set by overriding the WSGI config for individual services, this change makes it much easier. Change-Id: Ie452516655cbd40d63bdad3635fd66693e40ce34 Closes-Bug: #1917648
placement-api-wsgi.conf.j2 1.65 KiB
{% set log_dir = '/var/log/kolla/placement' %}
{% set wsgi_directory = '/usr/bin' if placement_install_type == 'binary' else '/var/lib/kolla/venv/bin' %}
{% if placement_enable_tls_backend | bool %}
{% if kolla_base_distro in ['centos'] %}
LoadModule ssl_module /usr/lib64/httpd/modules/mod_ssl.so
{% else %}
LoadModule ssl_module /usr/lib/apache2/modules/mod_ssl.so
{% endif %}
{% endif %}
Listen {{ api_interface_address | put_address_in_context('url') }}:{{ placement_api_listen_port }}
ServerSignature Off
ServerTokens Prod
TraceEnable off
TimeOut {{ kolla_httpd_timeout }}
KeepAliveTimeout {{ kolla_httpd_keep_alive }}
{% if placement_logging_debug | bool %}
LogLevel info
{% endif %}
<VirtualHost *:{{ placement_api_listen_port }}>
WSGIDaemonProcess placement-api processes={{ openstack_service_workers }} threads=1 user=placement group=placement display-name=placement-api
WSGIProcessGroup placement-api
WSGIScriptAlias / {{ wsgi_directory }}/placement-api
WSGIApplicationGroup %{GLOBAL}
WSGIPassAuthorization On
<IfVersion >= 2.4>
ErrorLogFormat "%{cu}t %M"
</IfVersion>
ErrorLog "{{ log_dir }}/placement-api.log"
LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b %D \"%{Referer}i\" \"%{User-Agent}i\"" logformat
CustomLog "{{ log_dir }}/placement-api-access.log" logformat
<Directory {{ wsgi_directory }}>
<Files placement-api>
Require all granted
</Files>
</Directory>
{% if placement_enable_tls_backend | bool %}
SSLEngine on
SSLCertificateFile /etc/placement/certs/placement-cert.pem
SSLCertificateKeyFile /etc/placement/certs/placement-key.pem
{% endif %}
</VirtualHost>