In previous article, Basic Concept of ElasticSearch (Part 2): Architectural Perspective, I have present about elasticsearch from architectural perspective, starting from elasticsearch role node, indexing flow, and searching flow. But, if you are not familiar with ElasticSearch, I suggest to read Basic Concept of ElasticSearch (Part 1): Introduction first.
As I mention in the previous article, ElasticSearch generally consist of two phase, indexing and searching. Indexing is a phase for populating ElasticSearch docs, while searching is a phase for searching docs based on given query. In this article, I will bring you deeper to detailed process of indexing.
Translog

During indexing, original doc will be appended in translog and in memory buffer. Translog is refer to transactional log, it stores all requests and its original data. Because of Lucene commits are too expensive to perform on every change then ElasticSearch logs every write operation that is received in translog. It is similar to what we have in redis. In redis we have RDB (Redis Database File) and AOF (Append Only File) to maintain data persistent. RDB is snapshot style persistence format while AOP is change-log style persistence format. Translog is similar to AOF.
If the request is successful on the primary shard, then ElasticSearch will sent the request in parallel to the replica shards. After it is successful fsync’ed on all shards (primary and replica), then client receives acknowledge from ElasticSearch. By default, every 5 seconds translog will be committed to the disk or when flush happen.
Lets take a look this example of directory structure:
/var/lib/elasticsearch/nodes/<node_index>/indices/<es_index_uuid>/<shard_index>/translog
- node index refers to the index of the respective node, it starts from 0
- es_index_uuid refers to uuid of specific index, each uuid is unique, example: if we have 3 indexes in one node then it would be 3 uuids.
- shard_index refers to index of the shard, it starts from 0, example: if we have 3 shard in the same elasticsearch index, then it would be 0, 1, and 2
As we see, there is translog directory per shard, it means translog is located in disk and it is in shard level. It contains 2 kind of files ckp and tlog. File ckp refers to checkpoint while tlog refers to translog itself. For each shard, we can have multiples translog file with generation index in the file name. ElasticSearch will keep the files depends on the retention config.
In case there is something wrong, as long as translog is not corrupted then we can replay all recorded request in translog. Another case is when one node is restarted, then ElasticSearch automatically replays any unflushed operations from the transaction log.
Refresh

In-memory buffer is located in On Heap memory which it is controlled by ElasticSearch. By default the size if 10% of total On Heap memory. The space for in-memory buffer can be reused by GC.
indices.memory.index_buffer_size: 10%
Minimum index buffer can be configured on this param indices.memory.min_index_buffer_size, by default it has 48mb. While maximum index buffer can be configured on this param indices.memory.max_index_buffer_size.
When the index refreshes, data in-memory buffer will be transformed into inverted index in segments memory which is located in Off Heap memory (controlled by Lucene / OS). Refreshing is triggered when in-memory buffer is full or during refresh time interval (default 1 second). This is the reason why searching in ElasticSearch is near time. In addition, refresh also can be triggered by using refresh API:
POST /my-index-000001/_refresh
After it is transformed into inverted index, then the docs are available to be searched. In case search activity happen before refresh time, we only get the latest data if we retrieve a document by its ID. When we retrieve a document by ID, then ElasticSearch will look into translog if it contains the ID that have not committed yet to inverted index.
In case of deletion, Elasticsearch will mark corresponding doc as deleted (soft deleted) in .del file. Every segments in ElasticSearch has this file associated with it. The deleted doc will be filtered during searching. While in update action, Elasticsearch will mark corresponding doc as deleted as well, and insert new doc with higher version. This happen because updating is more expensive than deleting and inserting.
Flush

Flushing is a process for making sure data in translog also stored permanently in the Lucene index (disk). During flushing, translog will be emptied and converted to segments, smaller segments are combined into larger segments, deleted docs are removed, and document frequencies are updated as well. And then segments are persisted in the disk.
Flushing is conducted based on one of these conditions:
- automatically conducted per 30 minutes, or
- allowed translog size has been reached, or
- flush manually by hitting this API
POST /my-index-000001/_flush
Conclusion
In this article, I have present a brief of how docs are getting indexed. It covers translog, refresh, and flush. In the next articles, I am going to present more detail about Term Index, Term Dictionary, and Posting List.
