Configuration

Options for configuring the log handler

host

The host of the Logstash server

Type: string

Default: None

port

The port of the Logstash server

Type: integer

Default: None

database_path

The path to the file containing queued events. To use an in-memory cache instead of a SQLite database, simply pass None. See Persistence for details.

This setting is only relevant for AsynchronousLogstashHandler.

Note

Using multiple instances of AsynchronousLogstashHandler with different database_path settings won’t work because there is only one LogProcessingWorker instance and it is configured with the database_path setting from the first handler which emits a log event.

Type: string

Default: None

transport

Callable or path to a compatible transport class.

You can specify your own transport class, e.g. to implement a transport via Redis or the Beats protocol. If you pass a string, it should be a path to a class which can be imported. If you pass anything else, it should be a callable or an instance of a class with a similar interface as logstash_async.transport.TcpTransport. Especially it should provide a close() and a send() method.

Currently available transports are:

``logstash_async.transport.TcpTransport``
``logstash_async.transport.UdpTransport``
``logstash_async.transport.BeatsTransport``
``logstash_async.transport.HttpTransport``

Type: string

Default: logstash_async.transport.TcpTransport

ssl_enable

Should SSL be enabled for the connection? Only used for logstash_async.transport.TcpTransport, logstash_async.transport.BeatsTransport and logstash_async.transport.HttpTransport.

Type: boolean

Default: False

ssl_verify

Should the server’s SSL certificate be verified? Only used for logstash_async.transport.TcpTransport, logstash_async.transport.BeatsTransport and logstash_async.transport.HttpTransport.

Type: boolean

Default: True

keyfile

The path to client side SSL key file. Only used for logstash_async.transport.TcpTransport and logstash_async.transport.BeatsTransport.

Type: string

Default: None

certfile

The path to client side SSL certificate file. Only used for logstash_async.transport.TcpTransport and logstash_async.transport.BeatsTransport.

Type: string

Default: None

ca_certs

The path to the file containing recognized CA certificates. Only used for logstash_async.transport.TcpTransport and logstash_async.transport.BeatsTransport.

Type: string

Default: None

enable

Flag to enable log processing (disabling might be handy for local testing, etc.)

Type: boolean

Default: True

event_ttl

TTL for messages that are waiting to be published. If a message is beyond it’s TTL, it will be deleted from the cache and will not be published to logstash.

This setting is only relevant for AsynchronousLogstashHandler.

Type: integer

Default: None

Options for configuring the log formatter

The following settings are only valid for the provided formatters logstash_async.handler.LogstashFormatter, logstash_async.handler.DjangoLogstashFormatter and logstash_async.handler.FlaskLogstashFormatter.

You can use any other formatter by configuring Python’s logging system accordingly. Any other formatter’s format() method just should return valid JSON suitable to be sent to Logstash (see Example Logstash Configuration).

Options:

message_type

The type field in the message sent to Logstash

Type: string

Default: python-logstash

tags

Additional tags to include in the Logstash message

Type: list

Default: None

fqdn

Use the system’s FQDN (fully qualified domain name) in the host field of the message sent to Logstash. socket.getfqdn() is used to retrieve the FQDN, otherwise socket.gethostname() is used for the default hostname.

Type: boolean

Default: False

extra_prefix

Name of the field in the resulting message sent to Logstash where all additional fields are grouped into. Consider it as some sort of namespace for all non-standard fields in the log event. This field will take any items passed in as extra fields via the extra configuration option (see below) as well as any extra items passed in the logging call.

To disable grouping of the extra items and have them on the top level of the log event message, simply set this option to None or the empty string.

Type: string

Default: extra

extra

Dictionary with static items to be included in the message sent to Logstash. This dictionary will be merged with any other extra items passed in the logging call.

Type: dict

Default: None

metadata

Dictionary with static items to be included in the message sent to Logstash in the special @metadata field. See https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html#metadata for documentation about the @metadata field and https://www.elastic.co/guide/en/beats/metricbeat/current/logstash-output.html for a common use.

Type: dict

Default: None

ensure_ascii

By default non-ASCII symbols in JSON are escaped with uXXXX sequence. But on some specific settings of Elastic Stack those sequences won’t be transformed back to UTF-8 representation. For those specific cases try to set parameter to ‘False’.

Type: boolean

Default: True

Options for the asynchronous processing and formatting

There are a few constants which are meant to be static but still can be overridden from the calling application by importing the constants variable from the logstash_async.constants which holds all constants as attributes for easy modification.

constants.SOCKET_TIMEOUT

Timeout in seconds for TCP connections

Type: float

Default: 5.0

constants.SOCKET_CLOSE_WAIT_TIMEOUT

Maximum time in seconds to wait for the socket’s write buffer to get empty. Set to 0 to disable waiting for the socket write buffer to get empty.

Type: float

Default: 30.0

constants.QUEUE_CHECK_INTERVAL

Interval in seconds to check the internal queue for new messages to be cached in the database

Type: float

Default: 2.0

constants.QUEUED_EVENTS_FLUSH_INTERVAL

Interval in seconds to send cached events from the database to Logstash

Type: float

Default: 10.0

constants.QUEUED_EVENTS_FLUSH_COUNT

Count of cached events to send from the database to Logstash; events are sent to Logstash whenever QUEUED_EVENTS_FLUSH_COUNT or QUEUED_EVENTS_FLUSH_INTERVAL is reached, whatever happens first

Type: integer

Default: 50

constants.QUEUED_EVENTS_BATCH_SIZE

Maximum number of events to be sent to Logstash in one batch. Depending on the transport, this usually means a new connection to the Logstash is established for the event batch (this is true for the UDP, TCP and Beats transports).

Type: integer

Default: 50

constants.DATABASE_EVENT_CHUNK_SIZE

Maximum number of events to be updated within one SQLite statement

Type: integer

Default: 750

constants.DATABASE_TIMEOUT

Timeout in seconds to “connect” (i.e. open) the SQLite database

Type: float

Default: 5.0

constants.FORMATTER_RECORD_FIELD_SKIP_LIST

List of record attributes which are filtered out from the event sent to Logstash. By default, the list consists of some Python standard LogRecord attributes. Usually this list does not need to be modified. Add/Remove elements to exclude/include them in the Logstash event, for the full list see: http://docs.python.org/library/logging.html#logrecord-attributes

Type: list

Default: <see source code>

constants.FORMATTER_LOGSTASH_MESSAGE_FIELD_LIST

Fields to be set on the top-level of a Logstash event/message, do not modify this unless you know what you are doing

Type: list

Default: <see source code>

constants.ERROR_LOG_RATE_LIMIT

Enable rate limiting for error messages (e.g. network errors) emitted by the logger used in LogProcessingWorker, i.e. when transmitting log messages to the Logstash server. In case the Logstash cannot be reached due to network issues (timeouts, connection refused, …), this may lead to many repeated error log messages which can get annoying, especially if the application’s logging system is configured to send emails or other notifications. For such errors emitted directly from the LogProcessingWorker class, rate limiting of identical errors for some time period can be configured to reduce logging of the same errors. In case rate limiting is in effect, the last message before dropping further messages will contain a hint telling that further messages of this kind will be dropped. To disable set this to None, to enable use a string like ‘5 per minute’, for details see https://limits.readthedocs.io/en/stable/quickstart.html#rate-limit-string-notation.

Note

This rate limit affects only error log messages emitted directly in LogProcessingWorker, if you need a general rate limiting of all log messages, use a filter for the logging framework, e.g. https://github.com/wkeeling/ratelimitingfilter.

Type: string

Default: None

constants.DATABASE_VACUUM_ON_SHUTDOWN

Vacuum SQLite database on shutdown - when enabled, the database will be vacuumed on shutdown to reduce its size on disk.

Type: boolean

Default: False

Example usage:

from logstash_async.constants import constants

constants.SOCKET_TIMEOUT = 10.0

from logstash_async.handler import AsynchronousLogstashHandler
...