Elasticsearch Security Architecture: TLS, RBAC and API Keys

Security in Elasticsearch is not one switch. It is five layers that each answer a different question: is the traffic encrypted and are the nodes who they claim to be (TLS), who is this caller (authentication), what may they do (roles and privileges), which documents and fields may they see (document and field level security), and what did they actually do (audit logging). Since 8.0, a new self-managed cluster starts with security on and TLS configured automatically, which removes the worst failure, an open cluster on the internet. It does not give you least privilege. The typical production cluster still has ingest clients running as elastic, Kibana dashboards shared through a superuser, and API keys that never expire. This article lays out the model behind each layer and the concrete configuration that gets a cluster from "secured by default" to "secured by design".

The mental model: two network layers, one authorization model

Every Elasticsearch node listens on two interfaces:

• Transport (port 9300 by default): node-to-node traffic, including shard replication, cluster state publication, and the remote cluster connections used by cross-cluster search and replication in the certificate-based model.
• HTTP (port 9200 by default): the REST API that clients, Kibana, Beats, Elastic Agent and Logstash use.

TLS protects both, but it means different things on each. On transport, TLS is also node authentication: a node that cannot present a certificate trusted by the cluster cannot join. On HTTP, TLS is encryption and server identity; the client is authenticated separately, by password, API key, token or client certificate.

Every request that reaches the HTTP layer goes through the same pipeline:

1. Authentication. The node tries the configured realms in order (or the API key or token service when the request carries one) until one recognizes the credentials. The result is a user with a set of role names.
2. Role resolution. Role names are resolved into role descriptors from built-in roles, the native role store, file-based roles and any role mappings.
3. Authorization. The action and target indices are checked against the union of the user's role privileges. If any role grants the action on the index, it is allowed.
4. Document and field filtering, if any granting role defines a DLS query or FLS fields.
5. Audit, if enabled, at each relevant point.

Privileges are additive. There is no "deny" in a role. A user with two roles has everything both grant. Least privilege therefore comes from giving fewer and narrower roles, never from adding a restrictive one.

TLS on transport and HTTP

On a new 8.x or 9.x self-managed install, the first node startup generates a CA, transport certificates and an HTTP certificate, and writes the relevant settings into elasticsearch.yml. Additional nodes join with an enrollment token. That is sound for a start, but most production estates replace the auto-generated certificates with ones from their own CA, so that rotation, revocation and trust are under their control.

Elasticsearch enforces transport TLS: in production mode (a node bound to a non-loopback address), a cluster with security enabled must have TLS on the transport layer, or the bootstrap checks fail and the node does not start.

A typical explicit configuration with PKCS#12 keystores:

xpack.security.enabled: true

xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: full
xpack.security.transport.ssl.keystore.path: certs/transport.p12
xpack.security.transport.ssl.truststore.path: certs/transport.p12

xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: certs/http.p12

Keystore passwords go into the Elasticsearch keystore, not into the YAML:

bin/elasticsearch-keystore add xpack.security.transport.ssl.keystore.secure_password
bin/elasticsearch-keystore add xpack.security.transport.ssl.truststore.secure_password
bin/elasticsearch-keystore add xpack.security.http.ssl.keystore.secure_password

verification_mode has three values:

certificate is a common, acceptable compromise on transport inside a private network when certificates are issued per cluster rather than per node, because transport trust is anchored by the CA anyway. On HTTP, clients should verify hostnames fully.

elasticsearch-certutil generates a CA and certificates for both layers if you do not have a PKI:

bin/elasticsearch-certutil ca --out elastic-stack-ca.p12
bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12 --out transport.p12
bin/elasticsearch-certutil http

Certificate expiry is an outage

When a transport certificate expires, nodes stop accepting connections from each other. The cluster partitions, masters lose quorum, and the cluster goes down, even though nothing else changed. The defense is boring: record every certificate's expiry, alert at 30 days, and rotate ahead of time. Elasticsearch reloads TLS key and certificate files from disk when they change, so rotation with the same CA does not need a restart. Changing the CA is a multi-step process: add the new CA to every truststore first, then roll new certificates, then remove the old CA.

GET _ssl/certificates

This returns every certificate the node has loaded, with path, subject_dn, serial_number and expiry. Run it per node, since certificates are per node.

Authentication: realms, API keys, service accounts and tokens

Realms are the sources of identity, consulted in a configured order:

Human users should come from an identity provider (SAML or OIDC through Kibana, or LDAP or AD), mapped to roles with role mappings, so access follows group membership and is removed when people leave:

PUT _security/role_mapping/sre-team
{
  "enabled": true,
  "roles": ["sre_operator"],
  "rules": {
    "all": [
      { "field": { "realm.name": "corp-saml" } },
      { "field": { "groups": "es-sre" } }
    ]
  }
}

Machines should not use passwords at all. They should use API keys or service account tokens.

API keys

An API key is a credential bound to the user who created it. Its effective privileges are the intersection of the role descriptors you give it and the owner's privileges, frozen at creation time. If you give no role descriptors, the key gets a snapshot of the owner's privileges, which is almost never what you want for a machine client: create keys with explicit, narrow role descriptors.

POST _security/api_key
{
  "name": "checkout-shipper-prod",
  "expiration": "90d",
  "role_descriptors": {
    "checkout-writer": {
      "cluster": ["monitor"],
      "indices": [
        {
          "names": ["logs-checkout-*"],
          "privileges": ["create_doc", "auto_configure"]
        }
      ]
    }
  },
  "metadata": { "team": "payments", "ticket": "SEC-1432" }
}

{
  "id": "VuaCfGcBCdbkQm-e5aOx",
  "name": "checkout-shipper-prod",
  "expiration": 1800349200000,
  "api_key": "ui2lp2axTNmsyakw9tvNnw",
  "encoded": "VnVhQ2ZHY0JDZGJrUW0tZTVhT3g6dWkybHAyYXhUTm1zeWFrdzl0dk5udw=="
}

The client sends Authorization: ApiKey <encoded>. The secret is returned once; Elasticsearch stores only a hash.

Operational rules:

• Always set expiration. Without it, a key is valid until invalidated. A 90-day expiry with automated renewal is a sound default.
• One key per client and environment. It makes audit trails meaningful and revocation surgical.
• Find and revoke: GET _security/_query/api_key (with a query on name or metadata) lists keys; DELETE _security/api_key with ids or name invalidates them. Invalidated keys stop working immediately.
• Updating privileges of an existing key is possible with the update API key endpoint in recent 8.x releases, without rotating the secret. Rotation is still the safer response to a leaked key.
• Owner deletion: keys are tied to their owner. Create machine keys from a dedicated service user, not from a person's account, so they survive staff changes. Note that disabling or deleting a native user does not necessarily invalidate API keys that user created; invalidate them explicitly.

Cross-cluster API keys are a separate type, created with POST _security/cross_cluster/api_key, used for remote cluster access; see cross-cluster replication and search.

Service accounts

Service accounts are built-in identities for Elastic components, with fixed, appropriate privileges you cannot edit. In 8.x they include elastic/kibana, elastic/fleet-server and a few others; list them with GET _security/service. They authenticate with service tokens:

POST _security/service/elastic/kibana/credential/token/kibana-prod-1

Or, for a token stored in files on the nodes rather than in the cluster:

bin/elasticsearch-service-tokens create elastic/kibana kibana-prod-1

Kibana then uses elasticsearch.serviceAccountToken in kibana.yml instead of a username and password. This replaces the older pattern of Kibana authenticating as the kibana_system user, and removes a password from configuration files. The same applies to Fleet Server.

The built-in superuser

The elastic user has the superuser role: everything, including restricted system indices and the ability to change security itself. Use it to bootstrap, then to create real admin identities, and then lock its password away as a break-glass credential. bin/elasticsearch-reset-password -u elastic resets it if needed. A cluster where applications authenticate as elastic has no meaningful authorization at all.

Authorization: roles and privileges

A role has cluster privileges, index privileges per index pattern, optional application privileges (used by Kibana), and optional run_as.

PUT _security/role/sre_operator
{
  "cluster": ["monitor", "manage_ilm", "manage_slm", "read_ccr"],
  "indices": [
    {
      "names": ["*"],
      "privileges": ["monitor", "view_index_metadata"]
    },
    {
      "names": ["logs-*", "metrics-*"],
      "privileges": ["read", "manage"]
    }
  ],
  "metadata": { "owner": "platform" }
}

The privileges you use most:

Note that system indices such as .security are restricted and are not matched by a wildcard like * unless the index entry sets allow_restricted_indices: true. Grant that only to dedicated administrative roles.

Least privilege for ingest clients

Ingest clients are where privilege creep usually starts, because "it needs to write" turns into all on *. The minimum for a client shipping to a data stream:

• create_doc on the data stream names it writes. Data streams are append-only, so create_doc is exactly enough.
• auto_configure on the same names, so the first write can create the data stream from its template.
• monitor at cluster level only if the client checks cluster version or health at startup (many shippers do).

No read, no delete, no manage. If the client is compromised, the attacker can add documents to its own data streams and nothing else: no reading other teams' logs, no deleting evidence. Logstash, Beats and Elastic Agent documentation each describe their own minimal privilege sets (Fleet manages Agent's keys automatically); start from those and trim.

For clients that update documents by id, index replaces create_doc. Delete privileges should be granted only where deletion is a business function.

Verifying what a caller can do

Two APIs answer access questions without trial and error:

GET _security/user/_privileges

POST _security/user/_has_privileges
{
  "cluster": ["monitor"],
  "index": [ { "names": ["logs-checkout-default"], "privileges": ["create_doc", "read"] } ]
}

The first lists everything the authenticated caller holds. The second answers yes or no per privilege, with has_all_requested at the top. Run them as the client identity, for example with its API key, as part of deployment.

GET _security/_authenticate returns who Elasticsearch thinks you are: the user, realm, roles and authentication type. It is the first call to make when an "unauthorized" error makes no sense.

Document and field level security

Document level security (DLS) restricts which documents a role can see with a query; field level security (FLS) restricts which fields. Both are defined per index entry in a role and require a paid subscription level (Platinum or Enterprise on self-managed).

PUT _security/role/tenant_reader
{
  "indices": [
    {
      "names": ["orders-*"],
      "privileges": ["read"],
      "query": {
        "template": {
          "source": "{\"term\": {\"tenant_id\": \"{{_user.metadata.tenant_id}}\"}}"
        }
      },
      "field_security": {
        "grant": ["*"],
        "except": ["customer.email", "customer.phone", "payment.*"]
      }
    }
  ]
}

The templated query reads the tenant from the authenticated user's metadata, so one role serves every tenant. Each user carries its own tenant_id in its metadata. If tenants will authenticate with API keys, test how the template resolves for an API key in your version before relying on it.

Things to know before relying on DLS and FLS:

• Roles combine permissively. If a user has two roles granting read on the same index, one with a DLS query and one without, the user sees all documents. The union of access applies, as with every other privilege.
• They are for reads. DLS and FLS filter search and get. Elastic's documentation says they are meant for read-only accounts and that users with DLS or FLS on an index should not perform write operations there; the update API, and update requests inside bulk requests, are not supported for them. Keep writing identities separate from DLS-restricted readers.
• Costs are real. DLS adds a filter to every search on the affected indices, and FLS changes what is returned and what can be aggregated. On high query volumes, measure latency with and without the role. See search latency.
• Aggregations respect them. A user without access to a field cannot aggregate on it; a DLS-filtered user's aggregations only count visible documents.

DLS is one of the standard ways to build multi-tenant search on shared indices; the alternatives and trade-offs are in multi-tenancy patterns.

Audit logging

Audit logging records security events: authentication successes and failures, access granted and denied, security configuration changes, tampered requests, connection denials by IP filtering, and run-as events. Enable it per node:

xpack.security.audit.enabled: true

Events go to a JSON log file in the node's logs directory (<clustername>_audit.json). Ship it with the same pipeline as your other logs, ideally to a different cluster than the one being audited, so that an attacker with access to the audited cluster cannot remove the evidence.

Audit logging is noisy if left broad. Tune it with include and exclude lists and ignore filters:

xpack.security.audit.logfile.events.include:
  - authentication_failed
  - access_denied
  - access_granted
  - anonymous_access_denied
  - connection_denied
  - tampered_request
  - run_as_granted
  - run_as_denied
  - security_config_change
xpack.security.audit.logfile.events.emit_request_body: false

xpack.security.audit.logfile.events.ignore_filters.shippers.users: ["svc-logshipper"]
xpack.security.audit.logfile.events.ignore_filters.shippers.indices: ["logs-*"]

Two judgments drive the tuning:

• access_granted for high-volume ingest identities produces at least one event per request (bulk requests also authorize their per-shard sub-requests). With 2,000 bulk requests per second, that is at least 2,000 audit events per second, 172.8 million per day (2,000 x 86,400). Ignore filters for known, narrowly privileged ingest identities keep the log useful. Note that an ignore filter drops every event type that matches it, denials included: the policy above silences svc-logshipper on logs-* entirely, while its denials on any other index are still logged.
• emit_request_body: true writes request bodies, which can include the sensitive data you are trying to protect, and passwords in security API calls. Leave it off unless you need it for a specific investigation.

What to alert on: bursts of authentication_failed for one user or from one origin, any access_denied for a service identity (it means a misconfiguration or a compromised client probing), any security_config_change outside a change window, and tampered_request.

Network controls around the cluster

Security settings inside Elasticsearch are one layer; the network is another, and it remains necessary.

• Do not expose 9200 or 9300 to the internet. Put HTTP behind a load balancer or reverse proxy in a private network, and keep transport restricted to the cluster's own nodes and any remote clusters.
• IP filtering can restrict transport and HTTP connections by address (xpack.security.transport.filter.allow and related settings). It is a useful second layer for transport, not a substitute for network policy.
• Keep the remote cluster port separate. With API key based remote clusters, open port 9443 only to the clusters that need it.
• Anonymous access is off by default. Leave it off; if a use case seems to need it, give that use case an API key with a read-only role instead.

Failure modes and anti-patterns

Applications running as elastic. Symptom: audit logs show elastic authenticating from application hosts. Fix: create per-application API keys with narrow roles, rotate the elastic password, then watch audit logs for authentication failures from the old credential to find stragglers.

API keys without expiry. Symptom: _query/api_key returns keys created years ago, owned by people who left. Fix: invalidate unused keys, set expiration on all new ones, and track owners in metadata.

Wildcard privileges that include more than intended. Symptom: a role with names: ["logs*"] grants access to logs_security_archive. Fix: use precise patterns with the separator (logs-*), and check with _has_privileges against the indices that must be denied.

DLS bypassed by a second role. Symptom: a tenant sees other tenants' data. Confirm with GET _security/user/_privileges as that user: two index entries for the same pattern, one without query. Fix: remove the broader role.

Expired transport certificate. Symptom: nodes cannot connect, logs show TLS handshake failures, the cluster loses its master. Confirm with openssl on the certificate files or GET _ssl/certificates if any node still answers. Fix: deploy renewed certificates signed by the same CA; nodes reload them.

verification_mode: none left from a test. Symptom: none until someone exploits it. Confirm by reviewing elasticsearch.yml on every node. Fix: full or certificate.

Audit logs on the same cluster. Symptom: an incident where the evidence is in the compromised cluster. Fix: ship audit logs to a separate cluster with separate credentials.

Runbook: a security review in one sitting

1. GET _security/_authenticate as yourself, then as each application identity you can obtain. Confirm none is superuser.
2. GET _ssl/certificates on every node: record expiry, alert at 30 days, confirm the chain is your CA.
3. Review elasticsearch.yml on every node: transport and HTTP TLS enabled, no verification_mode: none, audit enabled.
4. GET _security/role and read every custom role: no all or manage for applications, no allow_restricted_indices outside admin roles, no index pattern broader than needed.
5. GET _security/role_mapping: every mapping is tied to a realm and a group, not to everyone in a realm.
6. GET _security/_query/api_key for active keys: each has an expiration, an owner in metadata, and a recent use you can account for. Invalidate the rest.
7. GET _security/service and the token list for each service account in use: Kibana and Fleet Server use tokens, not passwords.
8. For each ingest identity, POST _security/user/_has_privileges asking for read and delete on its own indices. The answer should be false.
9. For DLS roles, test as a tenant user: search for another tenant's known document id and confirm no hit.
10. Confirm audit logs are arriving in a separate cluster, and that alerts exist for authentication failures and security_config_change.

Key takeaways

• Transport TLS authenticates nodes and is mandatory in production mode; HTTP TLS encrypts client traffic, and clients authenticate separately.
• Privileges are additive and there is no deny. Least privilege comes from narrow roles, not restrictive ones.
• Machines use API keys with explicit role descriptors and an expiration, or service account tokens for Elastic components. Never elastic.
• An ingest client writing to data streams needs create_doc and auto_configure on its own patterns, and almost nothing else.
• DLS and FLS are read controls and combine permissively across roles; a second, broader role silently removes the restriction.
• Audit logs belong in another cluster, tuned with ignore filters so the signal survives, and certificate expiry deserves an alert as much as disk does.

How Sparced helps

Sparced reads the cluster's security endpoints as part of its single pass, alongside health, nodes, shards and allocation, so security configuration is in front of you in the same report as cluster state. Findings come with the evidence from the cluster and a fix command with the names filled in, plus a check to confirm the change took. Because Sparced is read-only, installs no agent and writes nothing to the cluster, the identity you run it with never needs write or delete privileges on your data. It does not create roles, rotate keys or change settings for you, and it does not attempt to judge intent behind a role it cannot verify.