DEBOPS.ELASTICSEARCH(5)

NAME
debops.elasticsearch - Install and manage Elasticsearch database clusters

SYNOPSIS

debops run service/elasticsearch [--limit group,host,...] [--diff] [--check] [--tags tag1,tag2,...] [--skip-tags tag1,tag2,...] [<ansible-playbook options>] ...

DESCRIPTION

Elasticsearch <https://en.wikipedia.org/wiki/Elasticsearch> is a distributed search engine and storage system, also a part of the Elastic Stack. The software is developed by Elastic <https://www.elastic.co/>.

The debops.elasticsearch Ansible role can be used to deploy and manage Elasticsearch instances on one or more (3+) hosts. The role can be used as a dependency by other Ansible roles to allow control over their configuration options in the Elasticsearch configuration file.

GETTING STARTED

  Upstream package is used by default

The debops.elasticsearch role depends on the debops.extrepo Ansible role to configure access to the Elastic.co APT repository. This means that usually the latest available Elasticsearch release will be installed by default. If you require older releases, you can use the debops.apt_preferences role to select the desired package version.

Alternatively, you can "mask" the elasticsearch__extrepo__dependent_sources variable in the Ansible inventory and configure the APT repositories yourself via the debops.apt role.

  Elasticsearch is insecure in standalone mode

The Elasticsearch service in the default configuration supports only plaintext connections between the standalone host and clients. There's no client authentication or authorization policies as well. Due to that you should take care to not expose your standalone Elasticsearch service to the outside world without proper encryption and authentication.

In cluster mode, and with debops.pki role configured on the host, the debops.elasticsearch role will automatically enable the X-Pack <https://www.elastic.co/products/x-pack/> plugin and configure TLS encryption for HTTP clients and Transport communication between cluster nodes. This is possible due to changes in the Elastic licensing (the default installation comes with the Basic subscription which provides support for TLS and user/group management via the X-Pack plugin).

With secure cluster communication over TLS, the debops.elasticsearch role can use the Elasticsearch API to manage user accounts and role definitions in the cluster. A default set of built-in users <https://www.elastic.co/guide/en/elasticsearch/reference/current/built-in-users.html> will be created automatically; passwords of these users will be stored in the secret/elasticsearch/credentials/<cluster_name>/built-in/ directory on the Ansible Controller (managed by the debops.secret role). After that, with the base URL of the Elasticsearch API set in the elasticsearch__api_base_url variable, the role can manage Elasticsearch roles and user accounts using the elastic superuser account.

You can install additional plugins that provide encrypted connections, authentication, authorization and access control:

•   X-Pack <https://www.elastic.co/products/x-pack>, an Elastic commercial plugin with free trial period. Supports encryption, authentication, access control, integrates with Kibana and Logstash. Since the plugin is developed by the same team, its releases are in parallel with Elasticsearch.

•   Search Guard <https://search-guard.com/>, an open source third party plugin with commercial support. Has features comparable with X-Pack, with more basic features like HTTP and transport encryption, basic authentication and access control available free of charge.

•   ReadonlyREST <https://readonlyrest.com/>, an open source security plugin focused on the Elasticsearch HTTP REST interface security.

  Standalone deployment or cluster

With the default configuration, the debops.elasticsearch role will deploy the Elasticsearch service in a "standalone" mode without exposing the service to the outside world. This allows easy deployments for development or testing purposes.

In production environment, you are strongly advised to deploy Elasticsearch on at least three (3) nodes to ensure consistency in the cluster. To enable cluster mode, you will need to configure the firewall for Elasticsearch and use one or multiple Ansible inventory groups to design your cluster architecture.

The role currently does not support deployment of multiple Elasticsearch instances on one host. As an alternative, consider setting up an internal container environment with each Elasticsearch instance in a separate container with its own IP address or use Elastic Cloud Elasticsearch Service <https://www.elastic.co/cloud/elasticsearch-service> which does this for you and more.

See the Elasticsearch clustering for more details.

  Use as a role dependency

The Elasticsearch main configuration file does not support an include statement or conf.d directory <https://github.com/elastic/elasticsearch/issues/11362>. To mitigate that and allow multiple Elasticsearch configuration sources from other Ansible roles, the debops.elasticsearch role supports operation as a dependent role. This functionality can be used by other Ansible roles to better manage Elasticsearch plugins or extend the cluster configuration without the need to implement the entire role again and with preserved idempotency.

See the Usage as a role dependency for more details.

  Example inventory

To deploy Elasticsearch in a standalone mode, you can add the host to the [debops_service_elasticsearch] Ansible inventory group:

[debops_service_elasticsearch]
hostname

The default playbook supports use of different Ansible inventory groups for different types of Elasticsearch nodes. See Elasticsearch clustering for more details.

  Example playbook

If you are using this role without DebOps, here's an example Ansible playbook that uses the debops.elasticsearch role:

---
- name: Manage Elasticsearch cluster
  collections: [ 'debops.debops', 'debops.roles01',
                 'debops.roles02', 'debops.roles03' ]
  hosts: [ 'debops_service_elasticsearch',
           'debops_service_elasticsearch_master',
           'debops_service_elasticsearch_data',
           'debops_service_elasticsearch_ingest',
           'debops_service_elasticsearch_lb' ]
  become: True
  environment: '{{ inventory__environment | d({})
                   | combine(inventory__group_environment | d({}))
                   | combine(inventory__host_environment  | d({})) }}'
  pre_tasks:
    - name: Prepare elasticsearch environment
      ansible.builtin.import_role:
        name: 'elasticsearch'
        tasks_from: 'main_env'
      tags: [ 'role::elasticsearch', 'role::secret', 'role::elasticsearch:config' ]
  roles:
    - role: extrepo
      tags: [ 'role::extrepo', 'skip::extrepo', 'role::elasticsearch' ]
      extrepo__dependent_sources:
        - '{{ elasticsearch__extrepo__dependent_sources }}'
    - role: secret
      tags: [ 'role::secret', 'role::elasticsearch', 'role::elasticsearch:config' ]
      secret__directories:
        - '{{ elasticsearch__secret__directories }}'
    - role: etc_services
      tags: [ 'role::etc_services', 'skip::etc_services' ]
      etc_services__dependent_list:
        - '{{ elasticsearch__etc_services__dependent_list }}'
    - role: sysctl
      tags: [ 'role::sysctl', 'skip::sysctl' ]
      sysctl__dependent_parameters:
        - '{{ elasticsearch__sysctl__dependent_parameters }}'
    - role: ferm
      tags: [ 'role::ferm', 'skip::ferm' ]
      ferm__dependent_rules:
        - '{{ elasticsearch__ferm__dependent_rules }}'
    - role: java
      tags: [ 'role::java', 'skip::java' ]
    - role: elasticsearch
      tags: [ 'role::elasticsearch', 'skip::elasticsearch' ]

  Ansible tags

You can use Ansible --tags or --skip-tags parameters to limit what tasks are performed during Ansible run. This can be used after a host was first configured to speed up playbook execution, when you are sure that most of the configuration is already in the desired state.

Available role tags:

role::elasticsearch
    Main role tag, should be used in the playbook to execute all of the role tasks as well as role dependencies.

role::elasticsearch:config
    Generate the Elasticsearch configuration taking into account different configuration sources.

ELASTICSEARCH CLUSTERING

The Elasticsearch service can be deployed either on a single host in a "standalone" mode, or in a cluster of multiple hosts. The cluster mode will be enabled automatically after a few important variables and inventory groups are configured.

  Standalone mode vs cluster mode

In a standalone mode, the Elasticsearch service will not try to talk with any other Elasticsearch nodes. Service will be usable over localhost connection. This mode is good for prototyping, testing and development environments, however it's not very resilient.

In a cluster mode, multiple Elasticsearch nodes talk to each other in a configured network subnet, over TCP. Elasticsearch clients communicate with the cluster over HTTP REST interface, usually via a dedicated host with Kibana and/or Logstash as an intermediary.

  Playbook execution

When multiple Elasticsearch hosts are managed as a cluster, any changes in the cluster configuration should be implemented on all hosts in the cluster at the same time to avoid issues with split-brain or quorum. The role uses inventory groups to compute some specific values for all hosts in the cluster, however using the --limit parameter of the ansible-playbook command will only configure those values on a subset of hosts. Remember to always keep the whole cluster configuration synchronized by running the Elasticsearch playbook on all hosts included in the cluster (without the --limit parameter).

  Ansible inventory groups

The debops.elasticsearch role uses a set of Ansible inventory groups to detect the Elasticsearch node type and change the configuration accordingly.

The main inventory group is [debops_service_elasticsearch]. Hosts in this group are configured to behave in the same way - all of them are eligible to be a master host, all of them can hold data, and all of them can use an ingest pipeline to process the input. This group is useful in small clusters, typically <10 hosts in total.

In larger clusters, the system administrator may want to separate the cluster hosts into separate node types. Each Ansible inventory group enables a separate feature, and hosts can be in multiple groups at once to mix and match the desired features:

[debops_service_elasticsearch_master]
    Hosts in this Ansible inventory group are eligible to become masters.

[debops_service_elasticsearch_data]
    Hosts in this Ansible inventory group can hold data shards.

[debops_service_elasticsearch_ingest]
    Hosts in this Ansible inventory group can process incoming data via an ingest pipeline.

[debops_service_elasticsearch_lb]
    Hosts in this Ansible inventory group do not have any features explicitly enabled, and act as load balancers and coordinators within the cluster.

    You can check the Elasticsearch node documentation <https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-node.html> for more details about node features.

    The inventory groups and their corresponding node functions are defined using default variables. The role uses Ansible inventory groups to automatically determine the list of hosts which will be used for discovery, as well as the number of eligible master hosts, therefore direct changes to the node function variables should be done with care.

  Unicast host discovery, number of master hosts

The role automatically manages the list of hosts which should be contacted for initial host discovery and number of master-eligible nodes based on the Ansible inventory group membership.

If the [debops_service_elasticsearch_master] group is not used, all of the hosts in the [debops_service_elasticsearch] inventory group will be added to the unicast discovery list, and all of them will be eligible to become masters.

When hosts are included in the [debops_service_elasticsearch_master] inventory group, only hosts in this group will be able to become masters, and only hosts in this group will be used for initial unicast discovery. Remember to always include an odd number of master-eligible hosts to achieve quorum majority within the cluster.

  Firewall configuration

The role supports a firewall managed by the debops.ferm Ansible role. When the firewall is enabled, Elasticsearch will be configured to listen to connections on private IP addresses defined on the host along with the localhost; if the firewall is not detected or disabled, Elasticsearch will listen only on the localhost interface by default.

To enable cluster mode, you need to define at least one IP address or a CIDR subnet in the elasticsearch__allow_tcp list. Make sure to only allow access from trusted hosts!

There's also a separate elasticsearch__allow_http variable, but you don't need to enable it unless you need a direct access to the Elasticsearch HTTP REST interface from remote hosts. Kibana and Logstash installed on the same host as an Elasticsearch service should be able to talk to it over localhost with no issues.

  Elasticsearch API access

The debops.elasticsearch role relies on the Elasticsearch API to manage different parts of the cluster, currently user accounts and their roles. This is enabled with X-Pack support and TLS encryption are configured in the cluster. To provide access to the API, define in the Ansible inventory:

elasticsearch__api_base_url: 'https://es.example.com:9200'

When this variable is defined, the role will execute tasks against the API from a single host in the cluster at a time.

USAGE AS A ROLE DEPENDENCY

The debops.elasticsearch role can be used as a dependency by other Ansible roles to manage Elasticsearch main configuration file idempotently. Configuration options from multiple roles can be merged together and included in the configuration file, or removed conditionally.

  Dependent role variables

The role exposes three default variables that can be used by other Ansible roles as dependent variables:

elasticsearch__dependent_role
    Required. Name of the role that uses the debops.elasticsearch as a dependency. This will be used to store the configuration in its own YAML dictionary. The selected name shouldn't be changed, otherwise configuration will be desynchronized.

elasticsearch__dependent_configuration
    Required. List of the Elasticsearch configuration options defined in the same format as the main configuration. See elasticsearch__configuration for more details.

elasticsearch__dependent_state
    Optional. If not specified or present, the configuration will be included in the /etc/elasticsearch/elasticsearch.yml configuration file and stored as Ansible local fact. If absent, the configuration will be removed from the generated configuration file.

  Dependent configuration storage and retrieval

The dependent configuration from other roles is stored in the secret/ directory on the Ansible Controller (see debops.secret for more details) in a JSON file, with each role configuration in a separate dictionary. The debops.elasticsearch reads this file when Ansible local facts indicate that the Elasticsearch service is installed, otherwise a new empty file is created. This ensures that the stale configuration is not present on a new or re-installed host.

The YAML dictionaries from different roles are be merged with the main configuration in the elasticsearch__combined_configuration variable that is used to generate the final configuration. The merge order of the different elasticsearch__*_configuration variables allows to further affect the dependent configuration through Ansible inventory if necessary, therefore the Ansible roles that use this method don't need to provide additional variables for this purpose themselves.

  Example role variables

This file shows an example set of default variables included in a role that uses the debops.elasticsearch role as a dependency:

---
# State of the application deployment
application__deploy_state: 'present'
# Elasticsearch configuration for application
application__elasticsearch__dependent_configuration:
  - name: 'application.option'
    value: True
  - 'application.other.option': False

  Example role playbook

This file shows an example playbook for a role that uses the debops.elasticsearch role as a dependency:

---
- name: Manage application
  collections: [ 'debops.debops' ]
  hosts: [ 'debops_service_elasticsearch_application' ]
  become: True
  environment: '{{ inventory__environment | d({})
                   | combine(inventory__group_environment | d({}))
                   | combine(inventory__host_environment  | d({})) }}'
  pre_tasks:
    - name: Prepare elasticsearch environment
      ansible.builtin.import_role:
        name: 'elasticsearch'
        tasks_from: 'main_env'
      tags: [ 'role::elasticsearch', 'role::secret', 'role::elasticsearch:config' ]
  roles:
    - role: secret
      tags: [ 'role::secret', 'role::elasticsearch', 'role::elasticsearch:config' ]
      secret__directories:
        - '{{ elasticsearch__secret__directories }}'
    - role: elasticsearch
      tags: [ 'role::elasticsearch' ]
      elasticsearch__dependent_role: 'application'
      elasticsearch__dependent_state: '{{ application__deploy_state }}'
      elasticsearch__dependent_configuration:
        - '{{ application__elasticsearch__dependent_configuration }}'
    - role: application
      tags: [ 'role::application' ]

DEFAULT VARIABLE DETAILS

Some of debops.elasticsearch default variables have more extensive configuration than simple strings or lists, here you can find documentation and examples for them.

  elasticsearch__native_roles

The elasticsearch__*_native_roles variables provide a way to manage the Elasticsearch roles used in Role-based access control <https://www.elastic.co/guide/en/elasticsearch/reference/current/authorization.html> mechanisms. Roles can be defined using DebOps' Universal Configuration; different features might require activation of specific Elastic License subscriptions.

This feature requires the X-Pack plugin to be enabled as well as connection to the Elasticsearch cluster secured by the TLS encryption. Both of these will be enabled by the debops.elasticsearch role in a clustered configuration. Native roles will be managed via the Elasticsearch Role API <https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-put-role.html>, the base URL of which needs to be specified using the elasticsearch__api_base_url variable to be available.

  Examples

Create my_admin_role Elasticsearch role, based on the example included in the Roles API documentation:

elasticsearch__native_roles:
  - name: 'my_admin_role'
    data:
      cluster: [ 'all' ]
      indices:
        - names: [ 'index1', 'index2' ]
          privileges: [ 'all' ]
          #field_security:  # requires a license
          #  grant: [ 'title', 'body' ]
          #query: "{\"match\": {\"title\": \"foo\"}}"
      applications:
        - application: 'myapp'
          privileges: [ 'admin', 'read' ]
          resources: [ "*" ]
      run_as: [ 'other_user' ]
      metadata:
        version: 1

  Syntax

The Elasticsearch roles are defined using a list of YAML dictionaries with specific parameters:

name
    Required. The name of the Elasticsearch native role defined via the API. Configuration entries with the same name are merged and can modify each other in order of appearance.

data
    Required. A YAML dictionary with Elasticsearch configuration for the role which will be passed via the API as JSON data. When multiple configuration entries are merged, the data parameter is overwritten by the next entry in the list.

state
    Optional. If not defined or present, the defined Elasticsearch role will be created or updated on each debops.elasticsearch Ansible role execution. If absent, the defined role will be deleted from the cluster configuration. If ignore, a given configuration entry will not be processed by the debops.elasticsearch role during execution - this is a good way to avoid updating the role on each Ansible run, once it is configured.

  elasticsearch__native_users

The elasticsearch__*_native_users variables provide a way to manage the Elasticsearch users <https://www.elastic.co/guide/en/elasticsearch/reference/current/setting-up-authentication.html> used in Role-based access control <https://www.elastic.co/guide/en/elasticsearch/reference/current/authorization.html> authentication. Users can be defined using DebOps' Universal Configuration.

This feature requires the X-Pack plugin to be enabled as well as connection to the Elasticsearch cluster secured by the TLS encryption. Both of these will be enabled by the debops.elasticsearch role in a clustered configuration. Native users will be managed via the Elasticsearch User API <https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-put-user.html>, the base URL of which needs to be specified using the elasticsearch__api_base_url variable to be available.

  Examples

Create jacknich Elasticsearch user, based on the example included in the User API documentation:

elasticsearch__native_users:
  - name: 'jacknich'
    data:
      password: '{{ lookup("password", secret + "/elasticsearch/credentials/"
                           + "native/jacknich/password") }}'
      roles: [ 'admin', 'other_role1' ]
      full_name: 'Jack Nicholson'
      email: 'jacknich@example.com'
      metadata:
        intelligence: 7

An example user account with superuser privileges, equivalent to the elastic user:

- name: 'admin'
  data:
    full_name: 'Elastic Administrator'
    password: 'testpassword'  # don't do this
    email: 'admin@example.net'
    roles: [ 'superuser' ]
  state: 'present'  # change to 'ignore' afterwards

  Syntax

The Elasticsearch users are defined using a list of YAML dictionaries with specific parameters:

name
    Required. The name of the Elasticsearch native user defined via the API. Configuration entries with the same name are merged and can modify each other in order of appearance.

data
    Required. A YAML dictionary with Elasticsearch configuration for the user which will be passed via the API as JSON data. When multiple configuration entries are merged, the data parameter is overwritten by the next entry in the list.

state
    Optional. If not defined or present, the defined Elasticsearch user will be created or updated on each debops.elasticsearch Ansible role execution. If absent, the defined user will be deleted from the cluster configuration. If ignore, a given configuration entry will not be processed by the debops.elasticsearch role during execution - this is a good way to avoid updating the user on each Ansible run, once it is configured.

  elasticsearch__configuration

The elasticsearch__*_configuration variables define the Elasticsearch configuration options that are set in the /etc/elasticsearch/elasticsearch.yml configuration file.

The main Elasticsearch configuration file format is YAML. The reference documentation <https://www.elastic.co/guide/en/elasticsearch/reference/current/settings.html> defines two YAML formats recognized by Elasticsearch, hierarchical (YAML dictionary keys are indented), or flat (YAML dictionary keys are separated by dots). This role focuses only on the latter, flat format since it's used everywhere in the Elasticsearch documentation and seems to be the preferred method for the majority of the configuration options.

For quick reference, the Elasticsearch configuration file contains options in the following format:

cluster.name: example-cluster
node.name: node-1
network.host: [ _local_, _site_ ]
bootstrap.memory_lock: true
discovery.zen.minimum_master_nodes: 3

The elasticsearch__*_configuration variables are YAML lists of dictionaries. Each YAML dictionary defines an option, or redefines a previously defined option (the variables are flattened and then processed in order).

The first YAML dictionary key of each option (in above case, cluster, node, network, bootstrap, discovery is significant, and is used to separate configuration options into sections defined by the elasticsearch__configuration_sections variable.

Configuration options can be defined as YAML dictionaries directly, with the key being the name of the option, and value being its value:

elasticsearch__configuration:
  - 'cluster.name': 'example-cluster'
  - 'node.name': 'node-1'
  - 'network.host': [ '_local_', '_site_' ]
  - 'bootstrap.memory_lock': True
  - 'discovery.zen.minimum_master_nodes': 3

The extended YAML dictionary format is detected if a YAML dictionary contains a name key. The dictionaries support specific parameters:

name
    String. The name of the Elasticsearch option.

value
    The value of the Elasticsearch option. Can be a string, a number, a boolean or a YAML list.

comment
    An optional comment added to the option, either as a string or a YAML text block.

state
    If not specified or present, the option will be included in the configuration. If absent, the option will not be included. If comment, the option will be present but commented out (it's an internal feature and may not work reliably for all cases).

raw Optional, a YAML text block. The name of the configuration option will be discarded and used only as a marker for these parameters. The contents of the raw key will be added as-is to the configuration file. You can use this to include more extensive configuration defined as a hierarchical YAML structure. An example configuration which should be equivalent to the previous example:

    elasticsearch__configuration:
      - name: 'node.meta.host_type'
        raw: |
          # Node type
          node.master: true
          node.data: true
          node.ingest: true

    You should make sure that the indentation of the YAML parameters is consistent through the configuration file.

  elasticsearch__configuration_sections

The /etc/elasticsearch/elasticsearch.yml configuration file is structured in informal 'sections", each section contains configuration options from a specific group (node, cluster, etc.). The elasticsearch__configuration_sections contains a YAML list of sections and option types to associate with them. The order of the entries on the list determines the order of the sections in the finished configuration file.

Each section definition is a YAML dictionary with specific parameters:

name
    Name of the section, stored as a comment.

part or parts
    A string or a YAML list of configuration option prefixes (first YAML dictionary key of a given configuration option). Only the parts defined for a given section will be included in that section.

    After all of the sections are processed, any left over configuration options not matched with a particular section will be added at the end of the configuration file.

  elasticsearch__plugins

The elasticsearch__*_plugins variables are YAML lists that can be used to install or remove Elasticsearch plugins. Support for plugin management using these variables is minimalistic; you can install plugins known by the Elastic plugin repository <https://www.elastic.co/guide/en/elasticsearch/plugins/current/index.html>, or from an URL. More involved management can be done by creating a separate role and using debops.elasticsearch as a role dependency to manage configuration if necessary. See Usage as a role dependency for more details.

Each element of the list is a YAML dictionary with specific parameters:

name
    Required. Name of the plugin that shows up in the output of the

    bin/elasticsearch-plugin list

    command. This parameter will be used to check the state of the plugin.

url Optional. If the plugin is distributed via an URL, you can provide it here for the plugin management script to use instead of the plugin name.

state
    Optional. If not specified or present, the plugin and its configuration will be installed. If absent the plugin and its configuration will be removed.

configuration or config
    Optional. Custom configuration for a given plugin, in the format recognized by the main configuration template.

    See elasticsearch__configuration for more details.

  Examples

Install Java Script language support:

elasticsearch__plugins:
  - name: 'lang-javascript'

AUTHOR
Nick Janetakis, Maciej Delmanowski, Reto Gantenbein

COPYRIGHT
2014-2024, Maciej Delmanowski, Nick Janetakis, Robin Schneider and others