Skip to content
Snippets Groups Projects
  • Mark Goddard's avatar
    56a07702
    Performance: use a single config file for fluentd · 56a07702
    Mark Goddard authored
    Currently we generate multiple fluentd configuration files for inputs,
    filters, formatters and outputs.
    These are then included from the main td-agent.conf configuration file.
    With a large number of hosts, this can take a long time to template.
    
    Benchmarking of templating is available at [1].
    
    This change switches to a single fluentd configuration file, with the
    include done locally. For the default template files included with Kolla
    Ansible we use Jinja includes, but this does not work with templates in
    a different directory. We therefore use the Ansible template lookup
    plugin, which has a slightly higher overhead than a jinja include, but
    far lower than generating multiple templates. This should drastically
    improve the performance of this task.
    
    [1] https://github.com/stackhpc/ansible-scaling/blob/master/doc/template.md
    
    Partially-Implements: blueprint performance-improvements
    
    Change-Id: Ia8623be0aa861fea3e54d2c9e1c971dfd8e3afa9
    56a07702
    History
    Performance: use a single config file for fluentd
    Mark Goddard authored
    Currently we generate multiple fluentd configuration files for inputs,
    filters, formatters and outputs.
    These are then included from the main td-agent.conf configuration file.
    With a large number of hosts, this can take a long time to template.
    
    Benchmarking of templating is available at [1].
    
    This change switches to a single fluentd configuration file, with the
    include done locally. For the default template files included with Kolla
    Ansible we use Jinja includes, but this does not work with templates in
    a different directory. We therefore use the Ansible template lookup
    plugin, which has a slightly higher overhead than a jinja include, but
    far lower than generating multiple templates. This should drastically
    improve the performance of this task.
    
    [1] https://github.com/stackhpc/ansible-scaling/blob/master/doc/template.md
    
    Partially-Implements: blueprint performance-improvements
    
    Change-Id: Ia8623be0aa861fea3e54d2c9e1c971dfd8e3afa9