Clean up pass-through variables and standardize defaults

docs/roles/elasticsearch.md

@@ -209,17 +209,6 @@ elasticsearch_systemd_override_type_exec: false
 
 `elasticsearch_systemd_override_type_exec` creates a systemd drop-in that changes the Elasticsearch service from `Type=notify` to `Type=exec`. Elasticsearch 8.19+ and 9.x use `Type=notify` with a `systemd-entrypoint` binary that sends `READY=1`. In container environments (Docker-in-Docker, some LXC setups), the sd_notify socket doesn't work -- systemd never receives the ready signal, waits 900 seconds, then kills the process even though Elasticsearch is fully operational. The role's own health-check retries handle readiness detection when this override is active.
 
-### Advanced Cluster Settings
-
-```yaml
-elasticsearch_action_destructive_requires_name: false
-elasticsearch_recovery_max_bytes_per_sec: ""
-```
-
-`elasticsearch_action_destructive_requires_name` sets `action.destructive_requires_name: true` in `elasticsearch.yml`, requiring explicit index names for delete operations. When enabled, wildcard patterns like `_all` or `*` are rejected. Recommended for production to prevent accidental mass deletion.
-
-`elasticsearch_recovery_max_bytes_per_sec` throttles shard recovery bandwidth (e.g., `"100mb"`). When empty (the default), Elasticsearch uses its internal default (40 MB/s). Increase this on fast networks to speed up recovery after node restarts, or decrease it on shared networks to prevent saturation.
-
 ### Persistent Cluster Settings
 
 ```yaml
@@ -237,7 +226,6 @@ elasticsearch_cluster_settings:
 ```
 
 Any setting supported by the [cluster settings API](https://www.elastic.co/docs/reference/elasticsearch/rest-api/cluster/update-cluster-settings) can be used. Values can be strings, numbers, booleans, or nested objects — the YAML dict is serialized to JSON directly.
-
 ### Temperature Attribute
 
 ```yaml
@@ -349,29 +337,6 @@ elasticsearch_cert_will_expire_soon: false
 
 `elasticsearch_cert_will_expire_soon` is an internal flag set by the role during execution. Do not set this manually.
 
-### CORS Configuration
-
-```yaml
-elasticsearch_http_cors_enabled: false
-elasticsearch_http_cors_allow_origin: ""
-elasticsearch_http_cors_allow_methods: "OPTIONS, HEAD, GET, POST, PUT, DELETE"
-elasticsearch_http_cors_allow_headers: "X-Requested-With, Content-Type, Content-Length, Authorization, kbn-xsrf"
-elasticsearch_http_cors_allow_credentials: false
-```
-
-`elasticsearch_http_cors_enabled` enables Cross-Origin Resource Sharing on the Elasticsearch HTTP interface. Only needed when browser-based tools (Elasticsearch Head, Cerebro, custom dashboards) access Elasticsearch directly rather than through Kibana or a reverse proxy.
-
-`elasticsearch_http_cors_allow_origin` sets the permitted origins. Required when CORS is enabled. Use a specific origin (`"https://dashboard.example.com"`) for production, or `"*"` for development (allows all origins). A regex is also accepted: `"/https?://localhost(:[0-9]+)?/"`.
-
-`elasticsearch_http_cors_allow_methods` lists the HTTP methods allowed in CORS requests. The default covers all standard Elasticsearch API methods.
-
-`elasticsearch_http_cors_allow_headers` lists the headers allowed in CORS requests. The default includes `kbn-xsrf` for Kibana compatibility and `Authorization` for basic auth.
-
-`elasticsearch_http_cors_allow_credentials` controls whether the `Access-Control-Allow-Credentials` header is sent. Enable this when clients need to send cookies or HTTP authentication headers in cross-origin requests.
-
-!!! warning
-    Enabling CORS with `allow_origin: "*"` and `allow_credentials: true` is a security risk. Browsers will send authenticated requests to your cluster from any website. Always use specific origins in production.
-
 ### Logging Configuration
 
 These variables control the `log4j2.properties` template. They only take effect when `elasticsearch_manage_logging: true`.
@@ -408,14 +373,17 @@ elasticsearch_logging_audit: true
 ### Extra Configuration
 
 ```yaml
-# elasticsearch_extra_config:     # undefined by default
-#   xpack.monitoring.collection.enabled: true
-#   thread_pool.search.size: 30
+elasticsearch_extra_config:
+  action.destructive_requires_name: true
+  indices.recovery.max_bytes_per_sec: "100mb"
+  http.cors.enabled: true
+  http.cors.allow-origin: "https://dashboard.example.com"
+  thread_pool.search.size: 30
 ```
 
-`elasticsearch_extra_config` is a dictionary of arbitrary `elasticsearch.yml` settings that the role doesn't manage through dedicated variables. The template renders these as top-level YAML keys. Any keys that conflict with settings managed by dedicated role variables (like `cluster.name`, `network.host`, security settings, etc.) are silently filtered out, and the role emits a warning telling you to use the dedicated variable instead.
+`elasticsearch_extra_config` is a dictionary of arbitrary `elasticsearch.yml` settings. Any valid Elasticsearch configuration key can go here — CORS, recovery throttling, destructive action guards, thread pool tuning, monitoring, plugin settings, etc. The template renders these as top-level YAML keys.
 
-This is the escape hatch for settings not covered by other role variables, such as thread pool tuning, monitoring collection, or plugin-specific configuration.
+Keys that conflict with settings managed by dedicated role variables (like `cluster.name`, `network.host`, security/TLS settings, `bootstrap.memory_lock`) are silently filtered out, and the role emits a warning telling you to use the dedicated variable instead.
 
 ### Rolling Upgrades
 

docs/roles/kibana.md

@@ -37,7 +37,7 @@ graph TD
 ```yaml
 kibana_enable: true
 kibana_manage_yaml: true
-kibana_config_backup: true
+kibana_config_backup: false
 ```
 
 `kibana_enable` controls whether the Kibana service is enabled and started. Set to `false` if you want to install and configure Kibana without starting it (for example, during an image build).

examples/full-stack.yml

@@ -87,9 +87,11 @@
     # elasticsearch_http_tls_key: "/path/to/http.key"
     # elasticsearch_tls_ca_certificate: "/path/to/ca.crt"
 
-    # CORS (for direct browser access)
-    # elasticsearch_http_cors_enabled: false
-    # elasticsearch_http_cors_allow_origin: "https://my-app.example.com"
+    # Extra elasticsearch.yml settings (any key the API supports)
+    # elasticsearch_extra_config:
+    #   http.cors.enabled: true
+    #   http.cors.allow-origin: "https://my-app.example.com"
+    #   action.destructive_requires_name: true
 
     # ==========================================================================
     # Kibana

molecule/elasticsearch_custom/converge.yml

@@ -22,11 +22,16 @@
     elasticsearch_fs_repo:
       - /mnt/es-snapshots
     # Extra configuration via elasticsearch_extra_config
-    # Deliberately includes indices.recovery.max_bytes_per_sec to test that the
-    # conflict detection filters it out (dedicated variable takes precedence)
+    # These settings are passed through directly to elasticsearch.yml
     elasticsearch_extra_config:
       action.auto_create_index: ".monitoring*,.watches,.triggered_watches,.watcher-history*,.ml*"
-      indices.recovery.max_bytes_per_sec: "100mb"
+      action.destructive_requires_name: true
+      indices.recovery.max_bytes_per_sec: "50mb"
+      http.cors.enabled: true
+      http.cors.allow-origin: "'*'"
+      http.cors.allow-methods: "OPTIONS, HEAD, GET, POST, PUT, DELETE"
+      http.cors.allow-headers: "X-Requested-With, Content-Type, Content-Length, Authorization, kbn-xsrf"
+      http.cors.allow-credentials: false
     # Cluster name
     elasticsearch_clustername: custom-test-cluster
     # Config backup
@@ -37,13 +42,6 @@
     elasticsearch_pamlimits: true
     # Memory lock
     elasticsearch_memory_lock: true
-    # Destructive requires name
-    elasticsearch_action_destructive_requires_name: true
-    # Recovery throttle
-    elasticsearch_recovery_max_bytes_per_sec: "50mb"
-    # CORS
-    elasticsearch_http_cors_enabled: true
-    elasticsearch_http_cors_allow_origin: "'*'"
   tasks:
     - name: Include Elastic repos
       ansible.builtin.include_role:

molecule/elasticsearch_custom/verify.yml

@@ -160,27 +160,11 @@
           - "'/mnt/es-snapshots' in es_yml.content | b64decode"
         fail_msg: "Snapshot repo path /mnt/es-snapshots not found in elasticsearch.yml"
 
-    - name: Count occurrences of indices.recovery.max_bytes_per_sec
-      ansible.builtin.shell: |
-        grep -c 'indices.recovery.max_bytes_per_sec' /etc/elasticsearch/elasticsearch.yml
-      args:
-        executable: /bin/bash
-      register: recovery_count
-      changed_when: false
-
-    - name: Assert indices.recovery.max_bytes_per_sec appears exactly once (conflict filtering works)
-      ansible.builtin.assert:
-        that:
-          - recovery_count.stdout | int == 1
-        fail_msg: >-
-          indices.recovery.max_bytes_per_sec appears {{ recovery_count.stdout }} times
-          (expected 1 — conflict filtering should remove the duplicate from elasticsearch_extra_config)
-
-    - name: Assert dedicated variable value wins (50mb, not 100mb from extra_config)
+    - name: Assert indices.recovery.max_bytes_per_sec from extra_config
       ansible.builtin.assert:
         that:
-          - "'indices.recovery.max_bytes_per_sec: 50mb' in es_yml.content | b64decode"
-        fail_msg: "Expected 50mb from dedicated variable, not 100mb from elasticsearch_extra_config"
+          - "'indices.recovery.max_bytes_per_sec' in es_yml.content | b64decode"
+        fail_msg: "indices.recovery.max_bytes_per_sec not found (should come from elasticsearch_extra_config)"
 
     # ── Config backup ─────────────────────────────────────────────
 
@@ -246,33 +230,18 @@
           - "'LimitMEMLOCK=infinity' in memlock_content.content | b64decode"
         fail_msg: "LimitMEMLOCK=infinity not in memlock.conf"
 
-    # ── Destructive requires name ─────────────────────────────────
-
-    - name: Verify action.destructive_requires_name is in elasticsearch.yml
-      ansible.builtin.assert:
-        that:
-          - "'action.destructive_requires_name: true' in es_yml.content | b64decode"
-        fail_msg: "action.destructive_requires_name not found in elasticsearch.yml"
-
-    # ── Recovery throttle ─────────────────────────────────────────
-
-    - name: Verify indices.recovery.max_bytes_per_sec is in elasticsearch.yml
-      ansible.builtin.assert:
-        that:
-          - "'indices.recovery.max_bytes_per_sec: 50mb' in es_yml.content | b64decode"
-        fail_msg: "indices.recovery.max_bytes_per_sec not found in elasticsearch.yml"
-
-    # ── CORS settings ─────────────────────────────────────────────
+    # ── extra_config pass-through settings ──────────────────────────
 
-    - name: Verify CORS settings in elasticsearch.yml
+    - name: Verify extra_config settings are rendered in elasticsearch.yml
       ansible.builtin.assert:
         that:
-          - "'http.cors.enabled: true' in es_yml.content | b64decode"
+          - "'action.destructive_requires_name' in es_yml.content | b64decode"
+          - "'http.cors.enabled' in es_yml.content | b64decode"
           - "'http.cors.allow-origin' in es_yml.content | b64decode"
           - "'http.cors.allow-methods' in es_yml.content | b64decode"
           - "'http.cors.allow-headers' in es_yml.content | b64decode"
           - "'http.cors.allow-credentials' in es_yml.content | b64decode"
-        fail_msg: "CORS settings not found in elasticsearch.yml"
+        fail_msg: "extra_config pass-through settings not found in elasticsearch.yml"
 
     # ── File permissions ──────────────────────────────────────────
 

roles/beats/tasks/filebeat.yml

@@ -41,7 +41,7 @@
         group: root
         mode: "0644"
       when:
-        - elasticstack_release | int > 7
+        - elasticstack_release | int >= 8
 
     - name: Enable Ingest Pipelines
       ansible.builtin.shell: >

roles/elasticsearch/defaults/main.yml

@@ -14,7 +14,13 @@ elasticsearch_ml_enabled: true
 elasticsearch_config_backup: false
 # @var elasticsearch_manage_yaml:description: Let the role manage elasticsearch.yml. Set to false to manage it yourself
 elasticsearch_manage_yaml: true
-# @var elasticsearch_manage_logging:description: Let the role manage log4j2.properties for Elasticsearch logging
+# @var elasticsearch_manage_logging:description: >
+#   Let the role manage log4j2.properties for Elasticsearch logging.
+#   Default is false because the Elasticsearch package ships a well-tuned
+#   log4j2.properties that works out of the box. Only set to true when you
+#   need granular control over individual appenders (console, file, JSON,
+#   slow log, deprecation, audit) via the elasticsearch_logging_* variables.
+# @end
 elasticsearch_manage_logging: false
 # @var elasticsearch_logging_console:description: Enable console (stdout) log appender. Usually off for daemon mode
 elasticsearch_logging_console: false
@@ -77,21 +83,6 @@ elasticsearch_jna_workaround: false
 elasticsearch_memory_lock: false
 # @var elasticsearch_systemd_override_type_exec:description: Override systemd service Type to exec. Enable for Docker-in-Docker environments where sd_notify is non-functional
 elasticsearch_systemd_override_type_exec: false
-# @var elasticsearch_action_destructive_requires_name:description: Require explicit index names for destructive operations (prevents accidental wildcard deletes)
-elasticsearch_action_destructive_requires_name: false
-# @var elasticsearch_recovery_max_bytes_per_sec:description: Throttle shard recovery speed to prevent network saturation. Empty = Elasticsearch default
-elasticsearch_recovery_max_bytes_per_sec: ""
-
-# @var elasticsearch_http_cors_enabled:description: Enable CORS on the HTTP interface
-elasticsearch_http_cors_enabled: false
-# @var elasticsearch_http_cors_allow_origin:description: CORS allowed origins. Must be set when CORS is enabled
-elasticsearch_http_cors_allow_origin: ""
-# @var elasticsearch_http_cors_allow_methods:description: CORS allowed HTTP methods
-elasticsearch_http_cors_allow_methods: "OPTIONS, HEAD, GET, POST, PUT, DELETE"
-# @var elasticsearch_http_cors_allow_headers:description: CORS allowed headers
-elasticsearch_http_cors_allow_headers: "X-Requested-With, Content-Type, Content-Length, Authorization, kbn-xsrf"
-# @var elasticsearch_http_cors_allow_credentials:description: Whether to send CORS credentials (cookies, auth headers)
-elasticsearch_http_cors_allow_credentials: false
 
 # @var elasticsearch_cluster_settings:description: >
 #   Persistent cluster settings applied via PUT _cluster/settings after the
@@ -166,6 +157,17 @@ elasticsearch_validate_api_certs: false
 
 elasticsearch_unsafe_upgrade_restart: false
 
+# @var elasticsearch_extra_config:description: >
+#   Additional key-value pairs merged into elasticsearch.yml. Keys that
+#   conflict with dedicated role variables (e.g. cluster.name) are
+#   silently dropped — use the dedicated variables instead.
+# @var elasticsearch_extra_config:example: >
+# elasticsearch_extra_config:
+#   indices.queries.cache.size: "5%"
+#   thread_pool.write.queue_size: 500
+# @end
+elasticsearch_extra_config: {}
+
 # @var elasticsearch_freshstart:description: Internal flag tracking whether this is a fresh installation. Do not set manually
 elasticsearch_freshstart:
   changed: false

roles/elasticsearch/tasks/main.yml

@@ -306,45 +306,6 @@
       Remove them from elasticsearch_extra_config and use the dedicated
       variables instead.
   vars:
-    _elasticsearch_managed_keys:
-      - node.name
-      - path.data
-      - path.logs
-      - cluster.name
-      - network.host
-      - http.port
-      - transport.port
-      - http.publish_host
-      - http.publish_port
-      - node.roles
-      - discovery.type
-      - discovery.seed_hosts
-      - cluster.initial_master_nodes
-      - node.attr.temp
-      - xpack.ml.enabled
-      - xpack.security.enabled
-      - xpack.security.enrollment.enabled
-      - xpack.security.transport.ssl.enabled
-      - xpack.security.transport.ssl.verification_mode
-      - xpack.security.transport.ssl.certificate
-      - xpack.security.transport.ssl.key
-      - xpack.security.transport.ssl.certificate_authorities
-      - xpack.security.transport.ssl.keystore.path
-      - xpack.security.transport.ssl.truststore.path
-      - xpack.security.http.ssl.enabled
-      - xpack.security.http.ssl.certificate
-      - xpack.security.http.ssl.key
-      - xpack.security.http.ssl.certificate_authorities
-      - xpack.security.http.ssl.keystore.path
-      - xpack.security.http.ssl.truststore.path
-      - bootstrap.memory_lock
-      - action.destructive_requires_name
-      - indices.recovery.max_bytes_per_sec
-      - http.cors.enabled
-      - http.cors.allow-origin
-      - http.cors.allow-methods
-      - http.cors.allow-headers
-      - http.cors.allow-credentials
     _elasticsearch_extra_config_conflicts: >-
       {{ elasticsearch_extra_config.keys() | list | intersect(_elasticsearch_managed_keys) }}
   when:

roles/elasticsearch/templates/elasticsearch.yml.j2

@@ -113,19 +113,6 @@ xpack.security.enabled: false
 {% if elasticsearch_memory_lock | bool %}
 bootstrap.memory_lock: true
 {% endif %}
-{% if elasticsearch_action_destructive_requires_name | bool %}
-action.destructive_requires_name: true
-{% endif %}
-{% if elasticsearch_recovery_max_bytes_per_sec | default('', true) | length > 0 %}
-indices.recovery.max_bytes_per_sec: {{ elasticsearch_recovery_max_bytes_per_sec }}
-{% endif %}
-{% if elasticsearch_http_cors_enabled | bool %}
-http.cors.enabled: true
-http.cors.allow-origin: {{ elasticsearch_http_cors_allow_origin }}
-http.cors.allow-methods: {{ elasticsearch_http_cors_allow_methods }}
-http.cors.allow-headers: {{ elasticsearch_http_cors_allow_headers }}
-http.cors.allow-credentials: {{ elasticsearch_http_cors_allow_credentials | lower }}
-{% endif %}
 
 {% if elasticsearch_fs_repo is defined %}
 path:
@@ -137,24 +124,9 @@ path:
 {% endif %}
 
 {# Filter elasticsearch_extra_config to remove keys already managed by dedicated role variables #}
-{% if elasticsearch_extra_config is defined %}
-{%   set _managed_keys = [
-       'node.name', 'path.data', 'path.logs', 'cluster.name', 'network.host',
-       'http.port', 'transport.port', 'http.publish_host', 'http.publish_port',
-       'node.roles', 'discovery.type', 'discovery.seed_hosts', 'cluster.initial_master_nodes',
-       'node.attr.temp', 'xpack.ml.enabled', 'xpack.security.enabled',
-       'xpack.security.enrollment.enabled', 'xpack.security.transport.ssl.enabled',
-       'xpack.security.transport.ssl.verification_mode', 'xpack.security.transport.ssl.certificate',
-       'xpack.security.transport.ssl.key', 'xpack.security.transport.ssl.certificate_authorities',
-       'xpack.security.transport.ssl.keystore.path', 'xpack.security.transport.ssl.truststore.path',
-       'xpack.security.http.ssl.enabled', 'xpack.security.http.ssl.certificate',
-       'xpack.security.http.ssl.key', 'xpack.security.http.ssl.certificate_authorities',
-       'xpack.security.http.ssl.keystore.path', 'xpack.security.http.ssl.truststore.path',
-       'bootstrap.memory_lock', 'action.destructive_requires_name',
-       'indices.recovery.max_bytes_per_sec', 'http.cors.enabled', 'http.cors.allow-origin',
-       'http.cors.allow-methods', 'http.cors.allow-headers', 'http.cors.allow-credentials'
-     ] %}
-{%   set _filtered = elasticsearch_extra_config | dict2items | rejectattr('key', 'in', _managed_keys) | list | items2dict %}
+{# _elasticsearch_managed_keys is defined in vars/main.yml #}
+{% if elasticsearch_extra_config is defined and elasticsearch_extra_config %}
+{%   set _filtered = elasticsearch_extra_config | dict2items | rejectattr('key', 'in', _elasticsearch_managed_keys) | list | items2dict %}
 {%   if _filtered %}
 {{ _filtered | to_nice_yaml(indent=2, sort_keys=False) }}
 {%   endif %}