Skip to content

Widhian Bramantya

coding is an art form

Menu
  • About Me
Menu
elasticsearch

Index Lifecycle Management (ILM) in Elasticsearch: Automatic Data Control Made Simple

Posted on October 5, 2025October 5, 2025 by admin

Example: Full ILM Policy

Here’s a complete ILM policy for logs:

PUT _ilm/policy/logs_policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": { "max_size": "30gb", "max_age": "7d" }
        }
      },
      "warm": {
        "min_age": "30d",
        "actions": {
          "allocate": { "include": { "box_type": "warm" } },
          "forcemerge": { "max_num_segments": 1 }
        }
      },
      "cold": {
        "min_age": "90d",
        "actions": {
          "allocate": { "include": { "box_type": "cold" } },
          "freeze": {}
        }
      },
      "frozen": {
        "min_age": "180d",
        "actions": {
          "searchable_snapshot": {
            "snapshot_repository": "s3_repository"
          }
        }
      },
      "delete": {
        "min_age": "365d",
        "actions": { "delete": {} }
      }
    }
  }
}

Applying the ILM Policy

Attach it to an index template:

PUT _index_template/logs_template
{
  "index_patterns": ["logs-*"],
  "template": {
    "settings": {
      "number_of_shards": 3,
      "index.lifecycle.name": "logs_policy",
      "index.lifecycle.rollover_alias": "logs"
    }
  }
}

Create the first index:

PUT logs-000001
{
  "aliases": {
    "logs": { "is_write_index": true }
  }
}

Elasticsearch will automatically create logs-000002 when the rollover condition is reached.

Checking ILM Status

To see the current ILM phase of an index:

GET logs-000001/_ilm/explain

Example result:

"phase": "cold",
"action": "allocate",
"step": "completed"

List all ILM policies:

GET _ilm/policy

Hot–Warm–Cold–Frozen Architecture

ILM works best when you organize your cluster into node tiers:

TierRoleHardwareData AgeLicense
HotActive dataSSD, fast CPU0–30 daysBasic
WarmOld but used dataHDD30–90 daysBasic
ColdArchived, rarely usedSlow storage90–180 daysBasic
FrozenSnapshot-based, S3Cloud object storage180+ daysPlatinum / Enterprise

You can label nodes like this:

node.attr.box_type: hot

and ILM will move indices automatically between tiers.

Best Practices

  • Use rollover to keep index size balanced (30–50 GB).
  • Always use aliases for write operations.
  • Configure node tiers before applying ILM.
  • Snapshot before delete phase.
  • Monitor with:
GET _cat/indices?v
GET _cluster/health
  • If using frozen, set up an S3 repository and test searchable snapshot first.

Common Problems

IssueCauseSolution
ILM stuck in hot phaseAlias missingAdd rollover alias
Index not movedNode attribute not foundDefine box_type on nodes
Storage fullNo delete phaseAdd delete policy
Frozen not workingNo Platinum licenseUse cold phase or snapshots manually

ILM vs. Curator

FeatureILMCurator
Automatic Phasesyesno
Rollover Supportyesno
Hot–Warm–Coldyesyes (manual)
Frozen / S3 Searchyes (Platinum+)no
Built-in to Elasticsearchyesno (external tool)

ILM is the modern replacement for Curator, smarter, simpler, and integrated.

See also  Basic Concept of ElasticSearch (Part 2): Architectural Perspective

Conclusion

ILM is your auto-pilot for Elasticsearch data.
It helps you keep your cluster clean, efficient, and affordable, automatically managing indices from hot to frozen.
Even your oldest data can stay searchable from S3, if you have the right license.

“Hot data lives in memory, cold data sleeps on disk, and frozen data dreams in the cloud.”

With ILM, your data never overwhelms you again, it simply flows through its natural lifecycle.

Pages: 1 2
Category: ElasticSearch

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Linkedin

Widhian Bramantya

Recent Posts

  • Log Management at Scale: Integrating Elasticsearch with Beats, Logstash, and Kibana
  • Index Lifecycle Management (ILM) in Elasticsearch: Automatic Data Control Made Simple
  • Blue-Green Deployment in Elasticsearch: Safe Reindexing and Zero-Downtime Upgrades
  • Maintaining Super Large Datasets in Elasticsearch
  • Elasticsearch Best Practices for Beginners
  • Implementing the Outbox Pattern with Debezium
  • Production-Grade Debezium Connector with Kafka (Postgres Outbox Example – E-Commerce Orders)
  • Connecting Debezium with Kafka for Real-Time Streaming
  • Debezium Architecture – How It Works and Core Components
  • What is Debezium? – An Introduction to Change Data Capture
  • Offset Management and Consumer Groups in Kafka
  • Partitions, Replication, and Fault Tolerance in Kafka
  • Delivery Semantics in Kafka: At Most Once, At Least Once, Exactly Once
  • Producers and Consumers: How Data Flows in Kafka
  • Kafka Architecture Explained: Brokers, Topics, Partitions, and Offsets
  • Getting Started with Apache Kafka: Core Concepts and Use Cases
  • Security Best Practices for RabbitMQ in Production
  • Understanding RabbitMQ Virtual Hosts (vhosts) and Their Uses
  • RabbitMQ Performance Tuning: Optimizing Throughput and Latency
  • High Availability in RabbitMQ: Clustering and Mirrored Queues Explained

Recent Comments

  1. Playing with VPC AWS (Part 2) – Widhian's Blog on Playing with VPC AWS (Part 1): VPC, Subnet, Internet Gateway, Route Table, NAT, and Security Group
  2. Basic Concept of ElasticSearch (Part 3): Translog, Flush, and Refresh – Widhian's Blog on Basic Concept of ElasticSearch (Part 1): Introduction
  3. Basic Concept of ElasticSearch (Part 2): Architectural Perspective – Widhian's Blog on Basic Concept of ElasticSearch (Part 3): Translog, Flush, and Refresh
  4. Basic Concept of ElasticSearch (Part 3): Translog, Flush, and Refresh – Widhian's Blog on Basic Concept of ElasticSearch (Part 2): Architectural Perspective
  5. Basic Concept of ElasticSearch (Part 1): Introduction – Widhian's Blog on Basic Concept of ElasticSearch (Part 2): Architectural Perspective

Archives

  • October 2025
  • September 2025
  • August 2025
  • November 2021
  • October 2021
  • August 2021
  • July 2021
  • June 2021
  • March 2021
  • January 2021

Categories

  • Debezium
  • Devops
  • ElasticSearch
  • Golang
  • Kafka
  • Lua
  • NATS
  • Programming
  • RabbitMQ
  • Redis
  • VPC
© 2025 Widhian Bramantya | Powered by Minimalist Blog WordPress Theme