Skip to content
Snippets Groups Projects
  • John Garbutt's avatar
    70f6f8e4
    Reduce RabbitMQ busy waiting, lowering CPU load · 70f6f8e4
    John Garbutt authored
    On machines with many cores, we were seeing excessive CPU load on systems
    that were not very busy. With the following Erlang VM argument we saw
    RabbitMQ CPU usage drop from about 150% to around 20%, on a system with
    40 hyperthreads.
    
        +S 2:2
    
    By default RabbitMQ starts N schedulers where N is the number of CPU
    cores, including hyper-threaded cores. This is fine when you assume all
    your CPUs are dedicated to RabbitMQ. Its not a good idea in a typical
    Kolla Ansible setup. Here we go for two scheduler threads.
    More details can be found here:
    https://www.rabbitmq.com/runtime.html#scheduling
    and here:
    https://erlang.org/doc/man/erl.html#emulator-flags
    
        +sbwt none
    
    This stops busy waiting of the scheduler, for more details see:
    https://www.rabbitmq.com/runtime.html#busy-waiting
    Newer versions of rabbit may need additional flags:
    "+sbwt none +sbwtdcpu none +sbwtdio none"
    But this patch should be back portable to older versions of RabbitMQ
    used in Train and Stein.
    
    Note that information on this tuning was found by looking at data from:
    rabbitmq-diagnostics runtime_thread_stats
    More details on that can be found here:
    https://www.rabbitmq.com/runtime.html#thread-stats
    
    Related-Bug: #1846467
    
    Change-Id: Iced014acee7e590c10848e73feca166f48b622dc
    70f6f8e4
    History
    Reduce RabbitMQ busy waiting, lowering CPU load
    John Garbutt authored
    On machines with many cores, we were seeing excessive CPU load on systems
    that were not very busy. With the following Erlang VM argument we saw
    RabbitMQ CPU usage drop from about 150% to around 20%, on a system with
    40 hyperthreads.
    
        +S 2:2
    
    By default RabbitMQ starts N schedulers where N is the number of CPU
    cores, including hyper-threaded cores. This is fine when you assume all
    your CPUs are dedicated to RabbitMQ. Its not a good idea in a typical
    Kolla Ansible setup. Here we go for two scheduler threads.
    More details can be found here:
    https://www.rabbitmq.com/runtime.html#scheduling
    and here:
    https://erlang.org/doc/man/erl.html#emulator-flags
    
        +sbwt none
    
    This stops busy waiting of the scheduler, for more details see:
    https://www.rabbitmq.com/runtime.html#busy-waiting
    Newer versions of rabbit may need additional flags:
    "+sbwt none +sbwtdcpu none +sbwtdio none"
    But this patch should be back portable to older versions of RabbitMQ
    used in Train and Stein.
    
    Note that information on this tuning was found by looking at data from:
    rabbitmq-diagnostics runtime_thread_stats
    More details on that can be found here:
    https://www.rabbitmq.com/runtime.html#thread-stats
    
    Related-Bug: #1846467
    
    Change-Id: Iced014acee7e590c10848e73feca166f48b622dc