Skip to content

Widhian Bramantya

coding is an art form

Menu
  • About Me
Menu
elasticsearch

Index Lifecycle Management (ILM) in Elasticsearch: Automatic Data Control Made Simple

Posted on October 5, 2025October 5, 2025 by admin

Example: Full ILM Policy

Here’s a complete ILM policy for logs:

PUT _ilm/policy/logs_policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": { "max_size": "30gb", "max_age": "7d" }
        }
      },
      "warm": {
        "min_age": "30d",
        "actions": {
          "allocate": { "include": { "box_type": "warm" } },
          "forcemerge": { "max_num_segments": 1 }
        }
      },
      "cold": {
        "min_age": "90d",
        "actions": {
          "allocate": { "include": { "box_type": "cold" } },
          "freeze": {}
        }
      },
      "frozen": {
        "min_age": "180d",
        "actions": {
          "searchable_snapshot": {
            "snapshot_repository": "s3_repository"
          }
        }
      },
      "delete": {
        "min_age": "365d",
        "actions": { "delete": {} }
      }
    }
  }
}

Applying the ILM Policy

Attach it to an index template:

PUT _index_template/logs_template
{
  "index_patterns": ["logs-*"],
  "template": {
    "settings": {
      "number_of_shards": 3,
      "index.lifecycle.name": "logs_policy",
      "index.lifecycle.rollover_alias": "logs"
    }
  }
}

Create the first index:

PUT logs-000001
{
  "aliases": {
    "logs": { "is_write_index": true }
  }
}

Elasticsearch will automatically create logs-000002 when the rollover condition is reached.

Checking ILM Status

To see the current ILM phase of an index:

GET logs-000001/_ilm/explain

Example result:

"phase": "cold",
"action": "allocate",
"step": "completed"

List all ILM policies:

GET _ilm/policy

Hot–Warm–Cold–Frozen Architecture

ILM works best when you organize your cluster into node tiers:

TierRoleHardwareData AgeLicense
HotActive dataSSD, fast CPU0–30 daysBasic
WarmOld but used dataHDD30–90 daysBasic
ColdArchived, rarely usedSlow storage90–180 daysBasic
FrozenSnapshot-based, S3Cloud object storage180+ daysPlatinum / Enterprise

You can label nodes like this:

node.attr.box_type: hot

and ILM will move indices automatically between tiers.

Best Practices

  • Use rollover to keep index size balanced (30–50 GB).
  • Always use aliases for write operations.
  • Configure node tiers before applying ILM.
  • Snapshot before delete phase.
  • Monitor with:
GET _cat/indices?v
GET _cluster/health
  • If using frozen, set up an S3 repository and test searchable snapshot first.

Common Problems

IssueCauseSolution
ILM stuck in hot phaseAlias missingAdd rollover alias
Index not movedNode attribute not foundDefine box_type on nodes
Storage fullNo delete phaseAdd delete policy
Frozen not workingNo Platinum licenseUse cold phase or snapshots manually

ILM vs. Curator

FeatureILMCurator
Automatic Phasesyesno
Rollover Supportyesno
Hot–Warm–Coldyesyes (manual)
Frozen / S3 Searchyes (Platinum+)no
Built-in to Elasticsearchyesno (external tool)

ILM is the modern replacement for Curator, smarter, simpler, and integrated.

See also  Log Management at Scale: Integrating Elasticsearch with Beats, Logstash, and Kibana

Conclusion

ILM is your auto-pilot for Elasticsearch data.
It helps you keep your cluster clean, efficient, and affordable, automatically managing indices from hot to frozen.
Even your oldest data can stay searchable from S3, if you have the right license.

“Hot data lives in memory, cold data sleeps on disk, and frozen data dreams in the cloud.”

With ILM, your data never overwhelms you again, it simply flows through its natural lifecycle.

Related posts:

Basic Concept of ElasticSearch (Part 3): Translog, Flush, and Refresh

Log Management at Scale: Integrating Elasticsearch with Beats, Logstash, and Kibana

Advanced Text Search in Elasticsearch: N-Gram, Reverse, Fuzzy, and Search-as-you-type

Pages: 1 2
Category: ElasticSearch

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Linkedin

Widhian Bramantya

Recent Posts

  • Smart Automation in PostgreSQL: Managing Time-Based Data with pg_partman and pg_cron
  • Understanding PostgreSQL WAL, Slot, Publication, LSN, and Replication Lag
  • PostgreSQL Write-Ahead Log (WAL): Durability, Performance Tuning, and Recovery Explained
  • PostgreSQL Replication Deep Dive: From High Availability to Multi-Master Clusters
  • Finding Nearby Merchants in a Ride-Hailing App Using Elasticsearch Polygon Search
  • Advanced Text Search in Elasticsearch: N-Gram, Reverse, Fuzzy, and Search-as-you-type
  • Understanding and Customizing Analyzers in Elasticsearch
  • Log Management at Scale: Integrating Elasticsearch with Beats, Logstash, and Kibana
  • Index Lifecycle Management (ILM) in Elasticsearch: Automatic Data Control Made Simple
  • Blue-Green Deployment in Elasticsearch: Safe Reindexing and Zero-Downtime Upgrades
  • Maintaining Super Large Datasets in Elasticsearch
  • Elasticsearch Best Practices for Beginners
  • Implementing the Outbox Pattern with Debezium
  • Production-Grade Debezium Connector with Kafka (Postgres Outbox Example – E-Commerce Orders)
  • Connecting Debezium with Kafka for Real-Time Streaming
  • Debezium Architecture – How It Works and Core Components
  • What is Debezium? – An Introduction to Change Data Capture
  • Offset Management and Consumer Groups in Kafka
  • Partitions, Replication, and Fault Tolerance in Kafka
  • Delivery Semantics in Kafka: At Most Once, At Least Once, Exactly Once

Recent Comments

No comments to show.

Archives

  • October 2025
  • September 2025
  • August 2025
  • November 2021
  • October 2021
  • August 2021
  • July 2021
  • June 2021
  • March 2021
  • January 2021

Categories

  • Debezium
  • Devops
  • ElasticSearch
  • Golang
  • Kafka
  • Lua
  • NATS
  • PostgreSQL
  • Programming
  • RabbitMQ
  • Redis
  • VPC
© 2026 Widhian Bramantya | Powered by Minimalist Blog WordPress Theme