Skip to content

Widhian Bramantya

coding is an art form

Menu
  • About Me
Menu
elasticsearch

Basic Concept of ElasticSearch (Part 3): Translog, Flush, and Refresh

Posted on August 30, 2021 by admin

In previous article, Basic Concept of ElasticSearch (Part 2): Architectural Perspective, I have present about elasticsearch from architectural perspective, starting from elasticsearch role node, indexing flow, and searching flow. But, if you are not familiar with ElasticSearch, I suggest to read Basic Concept of ElasticSearch (Part 1): Introduction first.

As I mention in the previous article, ElasticSearch generally consist of two phase, indexing and searching. Indexing is a phase for populating ElasticSearch docs, while searching is a phase for searching docs based on given query. In this article, I will bring you deeper to detailed process of indexing.

Translog

Translog
Translog

During indexing, original doc will be appended in translog and in memory buffer. Translog is refer to transactional log, it stores all requests and its original data. Because of Lucene commits are too expensive to perform on every change then ElasticSearch logs every write operation that is received in translog. It is similar to what we have in redis. In redis we have RDB (Redis Database File) and AOF (Append Only File) to maintain data persistent. RDB is snapshot style persistence format while AOP is change-log style persistence format. Translog is similar to AOF.

If the request is successful on the primary shard, then ElasticSearch will sent the request in parallel to the replica shards. After it is successful fsync’ed on all shards (primary and replica), then client receives acknowledge from ElasticSearch. By default, every 5 seconds translog will be committed to the disk or when flush happen.

Lets take a look this example of directory structure:

/var/lib/elasticsearch/nodes/<node_index>/indices/<es_index_uuid>/<shard_index>/translog

  • node index refers to the index of the respective node, it starts from 0
  • es_index_uuid refers to uuid of specific index, each uuid is unique, example: if we have 3 indexes in one node then it would be 3 uuids.
  • shard_index refers to index of the shard, it starts from 0, example: if we have 3 shard in the same elasticsearch index, then it would be 0, 1, and 2
See also  Maintaining Super Large Datasets in Elasticsearch

As we see, there is translog directory per shard, it means translog is located in disk and it is in shard level. It contains 2 kind of files ckp and tlog. File ckp refers to checkpoint while tlog refers to translog itself. For each shard, we can have multiples translog file with generation index in the file name. ElasticSearch will keep the files depends on the retention config.

In case there is something wrong, as long as translog is not corrupted then we can replay all recorded request in translog. Another case is when one node is restarted, then ElasticSearch automatically replays any unflushed operations from the transaction log.

Refresh

Elasticsearch Refresh docs
Refreshing new docs

In-memory buffer is located in On Heap memory which it is controlled by ElasticSearch. By default the size if 10% of total On Heap memory. The space for in-memory buffer can be reused by GC.

indices.memory.index_buffer_size: 10%

Minimum index buffer can be configured on this param indices.memory.min_index_buffer_size, by default it has 48mb. While maximum index buffer can be configured on this param indices.memory.max_index_buffer_size.

When the index refreshes, data in-memory buffer will be transformed into inverted index in segments memory which is located in Off Heap memory (controlled by Lucene / OS). Refreshing is triggered when in-memory buffer is full or during refresh time interval (default 1 second). This is the reason why searching in ElasticSearch is near time. In addition, refresh also can be triggered by using refresh API:

POST /my-index-000001/_refresh

After it is transformed into inverted index, then the docs are available to be searched. In case search activity happen before refresh time, we only get the latest data if we retrieve a document by its ID. When we retrieve a document by ID, then ElasticSearch will look into translog if it contains the ID that have not committed yet to inverted index.

See also  Elasticsearch Best Practices for Beginners

In case of deletion, Elasticsearch will mark corresponding doc as deleted (soft deleted) in .del file. Every segments in ElasticSearch has this file associated with it. The deleted doc will be filtered during searching. While in update action, Elasticsearch will mark corresponding doc as deleted as well, and insert new doc with higher version. This happen because updating is more expensive than deleting and inserting.

Flush

Elasticsearch Flush docs
Flushing new docs

Flushing is a process for making sure data in translog also stored permanently in the Lucene index (disk). During flushing, translog will be emptied and converted to segments, smaller segments are combined into larger segments, deleted docs are removed, and document frequencies are updated as well. And then segments are persisted in the disk.

Flushing is conducted based on one of these conditions:

  • automatically conducted per 30 minutes, or
  • allowed translog size has been reached, or
  • flush manually by hitting this API
POST /my-index-000001/_flush

Conclusion

In this article, I have present a brief of how docs are getting indexed. It covers translog, refresh, and flush. In the next articles, I am going to present more detail about Term Index, Term Dictionary, and Posting List.

Related posts:

Basic Concept of ElasticSearch (Part 2): Architectural Perspective

Index Lifecycle Management (ILM) in Elasticsearch: Automatic Data Control Made Simple

Understanding and Customizing Analyzers in Elasticsearch

Category: ElasticSearch

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Linkedin

Widhian Bramantya

Recent Posts

  • Smart Automation in PostgreSQL: Managing Time-Based Data with pg_partman and pg_cron
  • Understanding PostgreSQL WAL, Slot, Publication, LSN, and Replication Lag
  • PostgreSQL Write-Ahead Log (WAL): Durability, Performance Tuning, and Recovery Explained
  • PostgreSQL Replication Deep Dive: From High Availability to Multi-Master Clusters
  • Finding Nearby Merchants in a Ride-Hailing App Using Elasticsearch Polygon Search
  • Advanced Text Search in Elasticsearch: N-Gram, Reverse, Fuzzy, and Search-as-you-type
  • Understanding and Customizing Analyzers in Elasticsearch
  • Log Management at Scale: Integrating Elasticsearch with Beats, Logstash, and Kibana
  • Index Lifecycle Management (ILM) in Elasticsearch: Automatic Data Control Made Simple
  • Blue-Green Deployment in Elasticsearch: Safe Reindexing and Zero-Downtime Upgrades
  • Maintaining Super Large Datasets in Elasticsearch
  • Elasticsearch Best Practices for Beginners
  • Implementing the Outbox Pattern with Debezium
  • Production-Grade Debezium Connector with Kafka (Postgres Outbox Example – E-Commerce Orders)
  • Connecting Debezium with Kafka for Real-Time Streaming
  • Debezium Architecture – How It Works and Core Components
  • What is Debezium? – An Introduction to Change Data Capture
  • Offset Management and Consumer Groups in Kafka
  • Partitions, Replication, and Fault Tolerance in Kafka
  • Delivery Semantics in Kafka: At Most Once, At Least Once, Exactly Once

Recent Comments

No comments to show.

Archives

  • October 2025
  • September 2025
  • August 2025
  • November 2021
  • October 2021
  • August 2021
  • July 2021
  • June 2021
  • March 2021
  • January 2021

Categories

  • Debezium
  • Devops
  • ElasticSearch
  • Golang
  • Kafka
  • Lua
  • NATS
  • PostgreSQL
  • Programming
  • RabbitMQ
  • Redis
  • VPC
© 2026 Widhian Bramantya | Powered by Minimalist Blog WordPress Theme