Skip to content
Snippets Groups Projects
  • Doug Szumski's avatar
    0743a9bf
    Remove Monasca Log Transformer · 0743a9bf
    Doug Szumski authored
    Historically Monasca Log Transformer has been for log
    standardisation and processing. For example, logs from different
    sources may use slightly different error levels such as WARN, 5,
    or WARNING. Monasca Log Transformer is a place where these could
    be 'squashed' into a single error level to simplify log searches
    based on labels such as these.
    
    However, in Kolla Ansible, we do this processing in Fluentd so
    that the simpler Fluentd -> Elastic -> Kibana pipeline also
    benefits. This helps to avoid spreading out log parsing
    configuration over many services, with the Fluentd Monasca output
    plugin being yet another potential place for processing (which
    should be avoided). It therefore makes sense to remove this
    service entirely, and squash any existing configuration which
    can't be moved to Fluentd into the Log Perister service. I.e.
    by removing this pipeline, we don't loose any functionality,
    we encourage log processing to take place in Fluentd, or at least
    outside of Monasca, and we make significant gains in efficiency
    by removing a topic from Kafka which contains a copy of all logs
    in transit.
    
    Finally, users forwarding logs from outside the control plane,
    eg. from tenant instances, should be encouraged to process the
    logs at the point of sending using whichever framework they are
    forwarding them with. This makes sense, because all Logstash
    configuration in Monasca is only accessible by control plane
    admins. A user can't typically do any processing inside Monasca,
    with or without this change.
    
    Change-Id: I65c76d0d1cd488725e4233b7e75a11d03866095c
    0743a9bf
    History
    Remove Monasca Log Transformer
    Doug Szumski authored
    Historically Monasca Log Transformer has been for log
    standardisation and processing. For example, logs from different
    sources may use slightly different error levels such as WARN, 5,
    or WARNING. Monasca Log Transformer is a place where these could
    be 'squashed' into a single error level to simplify log searches
    based on labels such as these.
    
    However, in Kolla Ansible, we do this processing in Fluentd so
    that the simpler Fluentd -> Elastic -> Kibana pipeline also
    benefits. This helps to avoid spreading out log parsing
    configuration over many services, with the Fluentd Monasca output
    plugin being yet another potential place for processing (which
    should be avoided). It therefore makes sense to remove this
    service entirely, and squash any existing configuration which
    can't be moved to Fluentd into the Log Perister service. I.e.
    by removing this pipeline, we don't loose any functionality,
    we encourage log processing to take place in Fluentd, or at least
    outside of Monasca, and we make significant gains in efficiency
    by removing a topic from Kafka which contains a copy of all logs
    in transit.
    
    Finally, users forwarding logs from outside the control plane,
    eg. from tenant instances, should be encouraged to process the
    logs at the point of sending using whichever framework they are
    forwarding them with. This makes sense, because all Logstash
    configuration in Monasca is only accessible by control plane
    admins. A user can't typically do any processing inside Monasca,
    with or without this change.
    
    Change-Id: I65c76d0d1cd488725e4233b7e75a11d03866095c