DEBOPS.FILEBEAT(5)

NAME
debops.filebeat - Manage Filebeat service using Ansible

SYNOPSIS

debops run service/filebeat [--limit group,host,...] [--diff] [--check] [--tags tag1,tag2,...] [--skip-tags tag1,tag2,...] [<ansible-playbook options>] ...

DESCRIPTION

Filebeat <https://www.elastic.co/beats/filebeat>, from Elastic <https://www.elastic.co/> is part of the Elastic Stack. Filebeat can be used to parse and "ingest" logs from files, syslog, and various other sources, parse them and send them off to Elasticsearch, Logstash or other destinations.

The filebeat Ansible role configures Filebeat on Debian/Ubuntu hosts. The software itself will be installed using the debops.elastic_co Ansible role.

GETTING STARTED

  Upstream package is used by default

The debops.filebeat role depends on the debops.extrepo Ansible role to configure access to the Elastic.co APT repository. This means that usually the latest available Filebeat release will be installed by default. If you require older releases, you can use the debops.apt_preferences role to select the desired package version.

Alternatively, you can "mask" the filebeat__extrepo__dependent_sources variable in the Ansible inventory and configure the APT repositories yourself via the debops.apt role.

  Default configuration

The filebeat role includes a basic configuration for Filebeat which will by default enable inputs for /var/log/*.log files, as well as the system module. Several other Filebeat modules and input log configurations will be enabled conditionally, based on the presence of various Ansible local facts.

The role will also configure Filebeat to read input configuration from the /etc/filebeat/inputs.d/ directory, where other applications can add their own configuration. The directory will not be created by default unless configuration snippets placed in it are configured or enabled conditionally.

The default output is directed at an Elasticsearch database installed on the same host as Filebeat. To change it, you can define in the Ansible inventory:

filebeat__configuration:
  - name: 'output_elasticsearch'
    config:
      output.elasticsearch:
        hosts: [ 'elasticsearch.example.org:9200' ]

See the filebeat__configuration documentation for more details. Elasticsearch database can be installed using the debops.elasticsearch role.

Check the Filebeat configuration <https://www.elastic.co/guide/en/beats/filebeat/current/configuring-howto-filebeat.html> upstream documentation to learn more about configuring Filebeat for your environment and requirements.

  Role debugging tips

Most of the configuration files generated by the role are protected by the Ansible no_log keyword. To make debugging easier, you can use the debops__no_log variable in an inventory (in the development environment) or with Ansible --extra-vars parameter (one time, for production environment) to disable log protection. See the variable documentation for more details.

  Example inventory

To install and configure Filebeat on a host, it needs to be included in the [debops_service_filebeat] Ansible inventory group:

[debops_all_hosts]
hostname
[debops_service_filebeat]
hostname

  Example playbook

If you are using this role without DebOps, here's an example Ansible playbook that uses the debops.filebeat role:

---
- name: Manage Filebeat service
  collections: [ 'debops.debops', 'debops.roles01',
                 'debops.roles02', 'debops.roles03' ]
  hosts: [ 'debops_service_filebeat' ]
  become: True
  environment: '{{ inventory__environment | d({})
                   | combine(inventory__group_environment | d({}))
                   | combine(inventory__host_environment  | d({})) }}'
  roles:
    - role: extrepo
      tags: [ 'role::extrepo', 'skip::extrepo', 'role::filebeat' ]
      extrepo__dependent_sources:
        - '{{ filebeat__extrepo__dependent_sources }}'
    - role: filebeat
      tags: [ 'role::filebeat', 'skip::filebeat' ]

  Ansible tags

You can use Ansible --tags or --skip-tags parameters to limit what tasks are performed during Ansible run. This can be used after host is first configured to speed up playbook execution, when you are sure that most of the configuration has not been changed.

Available role tags:

role::filebeat
    Main role tag, should be used in the playbook to execute all of the role tasks as well as role dependencies.

DEFAULT VARIABLE DETAILS

Some of the debops.filebeat default variables have more extensive configuration than simple strings or lists, here you can find documentation and examples for them.

  filebeat__configuration

The filebeat__*_configuration variables define the contents of the /etc/filebeat/filebeat.yml configuration file. Each variable contains a list of YAML dictionaries; each dictionary defines a part of the configuration which gets merged together during Ansible execution.

You can read the Filebeat configuration documentation <https://www.elastic.co/guide/en/beats/filebeat/current/configuring-howto-filebeat.html> to learn more about configuring Filebeat itself.

  Examples

Extend the default list of Filebeat inputs to include logs from Docker containers (the configuration sections are not merged, but override each other in order of appearance):

filebeat__configuration:
  - name: 'filebeat_inputs'
    config:
      filebeat.inputs:
        - type: 'log'
          enabled: True
          paths:
            - '/var/log/*.log'
            - '/var/log/messages'
        - type: 'container'
          paths:
            - '/var/lib/docker/containers/*/*.log'

Configure Filebeat to output its data to Elasticsearch on another host:

filebeat__configuration:
  - name: 'output_elasticsearch'
    config:
      output.elasticsearch:
        hosts:
          - 'elasticsearch.example.org:9200'

Configure Elasticsearch output, but over an encrypted connection (requires X-Pack support) using certificates managed by the debops.pki role. The access to the cluster is protected by a password, stored in the Filebeat keystore:

filebeat__configuration:
  - name: 'output_elasticsearch'
    config:
      output.elasticsearch:
        hosts:
          - 'https://elasticsearch.example.org:9200'
        ssl:
          certificate_authorities: '/etc/pki/realms/domain/CA.crt'
          certificate: '/etc/pki/realms/domain/default.crt'
          key: '/etc/pki/realms/domain/default.key'
        password: '${ELASTIC_PASSWORD}'

The filebeat__original_configuration variable contains the configuration that comes with the filebeat APT package re-implemented for consumption by the role. The filebeat__default_configuration variable contains some additional configuration enabled by default.

  Syntax

Each configuration entry is a YAML dictionary with specific parameters:

name
    Required. An identifier for a particular configuration entry, not used otherwise. The configuration entries with the same name parameter override each other.

config
    Required. A dictionary which holds the Filebeat configuration written in YAML. The config values from different configuration entries are merged recursively using the combine Ansible filter into a final YAML document.

    YAML keys can be specified in a tree-like structure:

    output:
      elasticsearch:
        hosts:
          - 'elasticsearch.example.org:9200'

    Or, they can be defined on a single line, separated by dots:

    output.elasticsearch.hosts: [ 'elasticsearch.example.org:9200' ]

    The combine Ansible filter does not automatically expand the dot-notation to a tree-like structure. Therefore it's important to use the same style thruought the configuration, otherwise the final YAML document will have duplicate entries.

state
    Optional. If not specified or present, the configuration will be included in the generated /etc/filebeat/filebeat.yml configuration file. if absent, the configuration will not be included in the final file. If ignore, the entry will not be evaluated by Ansible during execution.

  filebeat__snippets

The filebeat__*_snippets variables define the placement and contents of various *.yml files under the /etc/filebeat/ directory. The files can include Filebeat configuration in YAML format.

  Examples

Define an input source for logs generated by a custom application:

filebeat__snippets:
  - name: 'inputs.d/application.yml'
    config:
      type: 'log'
      enabled: True
      paths: [ '/var/log/application/*.log' ]

Add configuration for a built-in Filebeat module:

filebeat__snippets:
  - name: 'modules.d/auditd.yml'
    config:
      - module: 'auditd'
        log:
          enabled: True

You can find more example configurations in the filebeat__default_snippets variable.

  Syntax

Each configuration entry is a YAML dictionary with specific parameters:

name
    Required. Path of the configuration file, relative to the /etc/filebeat/ directory, with all needed subdirectories. The name parameter is also used as an identifier, entries with the same name parameter override each other in order of appearance.

    Role by default configures two subdirectories for input (input.d/) and Filebeat modules (modules.d/) configuration. Don't use the filebeat.yml as the filename, otherwise you will override the main configuration file.

config
    Required. A dictionary which holds the Filebeat configuration written in YAML. The value can either be a dictionary or a list of dictionaries, the result in the generated file will always be a list.

state
    Optional. If not specified or present, the configuration file will be generated. If absent, the configuration file will not be generated, and an existing file will be removed. If ignore, the entry will not be evaluated by Ansible during execution.

comment
    Optional. Comment to be included at the top of the generated file.

mode
    Optional. Specify the filesystem permissions of the generated file. If not specified, 0600 will be used by default.

  filebeat__keys

The filebeat__*_keys variables define the contents of the Filebeat keystore <https://www.elastic.co/guide/en/beats/filebeat/current/keystore.html> used to keep confidential data like passwords or access tokens. The keys can be referenced in the Filebeat configuration files using the ${secret_key} syntax.

  Examples

Add an Elasticsearch password used for access over a secure connection. The password is retrieved from the secret/ directory on the Ansible Controller, managed by the debops.secret Ansible role:

filebeat__keys:
  - ELASTIC_PASSWORD: '{{ lookup("file", secret + "/elastic-stack/elastic/password") }}'
  - KIBANA_PASSWORD:  '{{ lookup("file", secret + "/elastic-stack/kibana/password") }}'

Update an existing key with new content (presence of the force parameter will update the key on each Ansible run):

filebeat__keys:
  - name: 'ELASTIC_PASSWORD'
    value: 'new-elasticsearch-password'
    force: True

Remove a key from the Filebeat keystore:

filebeat__keys:
  - name: 'ELASTIC_PASSWORD'
    state: 'absent'

  Syntax

Each key entry is defined by a YAML dictionary. The keys can be defined using a simple format, with dictionary key being the secret key name, and its value being the secret value. In this case you should avoid the name or value as the secret keys.

Alternatively, secret keys can be defined using YAML dictionaries with specific parameters:

name
    Required. Name of the secret key to store in the Filebeat keystore.

value
    Optional. A string with the value which should be stored under a given key.

state
    Optional. If not specified or present, the key will be inserted into the keystore. If absent, the key will be removed from the keystore.

force
    Optional, boolean. If present and True, the specified key will be updated in the keystore.

AUTHOR
Maciej Delmanowski

COPYRIGHT
2014-2024, Maciej Delmanowski, Nick Janetakis, Robin Schneider and others