Elasticsearch

You can configure Chef Automate to use Elasticsearch clusters that aren’t deployed with Chef Automate.

Support limitations

This documentation doesn’t provide support for designing or installing custom topologies that Chef products don’t manage and that aren’t already covered in the documentation. If you choose to run external systems, you are responsible for maintaining, managing, and upgrading those systems. For production environments, use a product that performs the installation, maintains configuration and data over time, and tracks the current state of the entire system. This documentation doesn’t make those guarantees.

Configure External Elasticsearch

These configuration directions are intended for in the initial deployment of Chef Automate.

Add the following to your config.toml:

[global.v1.external.elasticsearch]
  enable = true
  nodes = ["http://elastic1.example:9200", "http://elastic2.example:9200", "..." ]

# Uncomment and fill out if using external Elasticsearch with SSL and/or basic auth
# [global.v1.external.elasticsearch.auth]
#   scheme = "basic_auth"
# [global.v1.external.elasticsearch.auth.basic_auth]
## Create this Elasticsearch user before starting the Chef Automate deployment;
## Chef Automate assumes it exists.
#   username = "<admin username>"
#   password = "<admin password>"
# [global.v1.external.elasticsearch.ssl]
#  Specify either a root_cert or a root_cert_file
#  root_cert = """$(cat </path/to/cert_file.crt>)"""
#  server_name = "<elasticsearch server name>"

# Uncomment and fill out if using external Elasticsearch that uses hostname-based routing/load balancing
# [esgateway.v1.sys.ngx.http]
#  proxy_set_header_host = "<your external es hostname>:1234"

# Uncomment and add to change the ssl_verify_depth for the root cert bundle
#  ssl_verify_depth = "2"

Because externally-deployed Elasticsearch nodes won’t have access to Chef Automate’s built-in backup storage services, you must configure Elasticsearch backup settings separately from Chef Automate’s primary backup settings. You can configure backups to use either the local filesystem or S3.

Add Resolvers for Elasticsearch

In case you want to resolve the Elasticsearch node IPs dynamically using DNS servers, you can add resolvers/name servers to the configuration.

Name servers can be added in two ways:

1. Add name servers IPs: Add the name server to your config.toml file to resolve the Elasticsearch nodes.

   [esgateway.v1.sys.ngx.main.resolvers]
     # Multiple resolvers can be specified by adding the resolvers in the list.
     nameservers = ["192.0.2.0:24", "198.51.100.0:24"]
   

2. Set system DNS entries: To use existing system name server entries from /etc/resolv.conf, add the following setting to config.toml:

   [esgateway.v1.sys.ngx.main.resolvers]
     enable_system_nameservers = true
   

If both options are set, name server IPs takes precedence over the system name server entries.

Apply the changes:

sudo chef-automate config patch config.toml

If you wish to reset to the default configuration or to modify the configuration:

1. Run chef-automate config show config.toml.
2. Open config.toml and remove the esgateway.v1.sys.ngx.main.resolvers configuration or change the values.
3. Run chef-automate config set config.toml to apply your changes.

Backup External Elasticsearch

Backup External Elasticsearch to a Local Filesystem

To configure local filesystem backups of Chef Automate data stored in an externally-deployed Elasticsearch cluster:

1. Ensure that the filesystems you intend to use for backups are mounted to the same path on all Elasticsearch master and data nodes.
2. Configure the Elasticsearch path.repo setting on each node as described in the Elasticsearch documentation.
3. Add the following to your config.toml:

[global.v1.external.elasticsearch.backup]
enable = true
location = "fs"

[global.v1.external.elasticsearch.backup.fs]
# The `path.repo` setting you've configured on your Elasticsearch nodes must be
# a parent directory of the setting you configure here:
path = "/var/opt/chef-automate/backups"

Backup External Elasticsearch to AWS S3

To configure AWS S3 backups of Chef Automate data stored in an externally-deployed Elasticsearch cluster:

1. Install the repository-s3 plugin on all nodes in your Elasticsearch cluster.

2. If you wish to use IAM authentication to provide your Elasticsearch nodes access to the S3 bucket, you must apply the appropriate IAM policy to each host system in the cluster.

3. Configure each Elasticsearch node with a S3 client configuration containing the proper S3 endpoint, credentials, and other settings as described in the Elasticsearch documentation.

4. Enable S3 backups by adding the following settings to your config.toml:

   [global.v1.external.elasticsearch.backup]
   enable = true
   location = "s3"
   
   [global.v1.external.elasticsearch.backup.s3]
   
     # bucket (required): The name of the bucket
     bucket = "<bucket name>"
   
     # base_path (optional):  The path within the bucket where backups should be stored
     # If base_path isn't set, backups will be stored at the root of the bucket.
     base_path = "<base path>"
   
     # name of an s3 client configuration you create in your elasticsearch.yml
     # see https://www.elastic.co/guide/en/elasticsearch/plugins/current/repository-s3-client.html
     # for full documentation on how to configure client settings on your
     # Elasticsearch nodes
     client = "<client name>"
   
   [global.v1.external.elasticsearch.backup.s3.settings]
   ## The meaning of these settings is documented in the S3 Repository Plugin
   ## documentation. See the following links:
   ## https://www.elastic.co/guide/en/elasticsearch/plugins/current/repository-s3-repository.html
   
   ## Backup repo settings
   # compress = false
   # server_side_encryption = false
   # buffer_size = "100mb"
   # canned_acl = "private"
   # storage_class = "standard"
   ## Snapshot settings
   # max_snapshot_bytes_per_sec = "40mb"
   # max_restore_bytes_per_sec = "40mb"
   # chunk_size = "null"
   ## S3 client settings
   # read_timeout = "50s"
   # max_retries = 3
   # use_throttle_retries = true
   # protocol = "https"
   

Backup External Elasticsearch to GCS

To configure Google Cloud Storage Bucket (GCS) backups of Chef Automate data stored in an externally-deployed Elasticsearch cluster:

1. Install the repository-gcs plugin on all nodes in your Elasticsearch cluster.

2. Create a storage bucket and configure a service account to access it per the steps described in the Elasticsearch documentation.

3. Configure each Elasticsearch node with a GCS client configuration that contains the proper GCS settings as described in the Elasticsearch documentation.

4. Enable GCS backups by adding the following settings to your config.toml:

   [global.v1.external.elasticsearch]
     enable = true
     nodes = ["https://my-es.cluster"]
     ## If multiple
     # nodes = ["https://my-es.node-1", "https://my-es.node-2", "etc..."]
   
   ## The following settings are required if you have Elasticsearch setup with basic auth
   #[global.v1.external.elasticsearch.auth]
   #  scheme = "basic_auth"
   #
   #[global.v1.external.elasticsearch.auth.basic_auth]
   #  username = "everyuser"
   #  password = "pass123"
   
   [global.v1.external.elasticsearch.backup]
     enable = true
     location = "gcs"
   
   [global.v1.external.elasticsearch.backup.gcs]
     bucket = "<bucket name>"
     # Client name is normally default, but can be set here if you have generated service
     # account credentials with a different client name
     client = "default"
   
   ## GCS Bucket Settings:
   # type = nearline
   # access control = uniform