Beyond the _reindex API: A Blue-Green Strategy for Zero-Downtime Elasticsearch

Introduction

Our team began adopting Elasticsearch to power advanced search and filtering capabilities in our service. As Elasticsearch is relatively new to us, this migration was not just a technical task but also a great learning opportunity. This post documents the challenges we faced, why the official approach didn’t fit our architecture, and the strategy we designed for a smooth and reliable reindexing process — all without any downtime.

Problem Statement

While working on a new user story in our service, we found that we had accidentally pushed an index mapping which was not able to fulfill our filtering and sorting requirements. Some existing fields needed to be converted into nested fields, but in Elasticsearch the type of a field is immutable—once an index is created, its mapping cannot be changed.

With a deadline to deliver the new functionality just days away, we had to rapidly devise and implement a high-stakes strategy to build a new index from scratch and seamlessly migrate all production data without any service interruption or data loss.

What is the change exactly?

Old mapping:	New mapping:
... "some_object.highest_priority": { "type": "integer" }, ...	... "some_object": { "type": "nested", "properties": { "highest_priority": { "type": "integer" }, "count": { "type": "integer" } } }, ...

Note that by default the `some_object` is of type `object`. We want to change it to a `nested` type so that we can perform advanced filtering on `highest_priority` and `count` together. For more information about the characteristics of nested vs flattened fields, please refer to this article.

Our System Architecture

Before diving into the migration strategy, it’s important to understand how Elasticsearch is integrated within our services.

Key Components:

Our Service: A set of services serving internal and external traffic to manage client cases.
Relational Database: A relational database acting as the primary data store of cases (source of truth).
ES Indexer: A service which reads documents from a Kafka topic and indexes them into Elasticsearch.
Elasticsearch: Secondary storage backing our public Case Search API endpoints.

Indexing Flow:

When Our Service creates or updates a case, it first updates the case in the Relational Database.
The case is published to a Kafka topic using the outbox pattern.
The ES Indexer consumes the Kafka message and indexes the document into Elasticsearch.
Our Service reads Cases directly from Elasticsearch via an internal SDK.

Remarks:

For newly created cases in the source, they always have a complete set of fields being sent to Elasticsearch.
For update operations the set of fields can be partial, meaning only the changed fields are sent to Elasticsearch.
We rely on Kafka to guarantee the order of updates being applied to the documents. We do not rely on optimistic locking or external versioning because the source data is not versioned.

The "Official" Path and Its Pitfalls

The official Elasticsearch recommendation for reindexing is straightforward:

Create a new index with the desired mapping.

PUT /new_index_name
{
  "mappings": {
    "properties": {
      "field_to_change": {
        "type": "new_type"
      },
      "other_field": {
        "type": "text"
      }
      // ... include all other fields from the old index and any new ones
    }
  }
}

Use the _reindex API to copy all documents from the old index to the new one.

POST /_reindex
{
  "source": {
    "index": "old_index_name"
  },
  "dest": {
    "index": "new_index_name"
  }
}

Implement dual-writing in the application, so any new data or updates are written to both the old and new indices simultaneously. This prevents data loss for documents modified during the re-indexing process. Alternatively, if reindexing is fast enough or writes are paused, this step might be skipped.
Switch the alias that the application uses for reads and writes to point from the old index to the new one.

POST /_aliases
{
  "actions": [
    { "remove": { "index": "old_index_name", "alias": "my_app_alias" } },
    { "add": { "index": "new_index_name", "alias": "my_app_alias" } }
  ]
}

Delete the old index once the migration is confirmed to be successful.

DELETE /old_index_name

The official way doesn’t work for us

Unfortunately, this standard approach contained a critical flaw for our system. The problem lies in a potential race condition between the_reindex process and the live dual-writing mechanism.

Consider the situation where a live update from the application and a write from the _reindex job try to modify the same case at nearly the same time.

The two competing writes are:

- Live Update: Case A (Version 2)

- Reindex Job: Case A (Version 1)

Due to the lack of synchronization between these two separate processes, the following sequence can happen:

- The live update (Version 2) arrives first. The application's dual-write successfully saves the newest version of the document to the destination index.

- The reindex job (Version 1) arrives second. The _reindex process, unaware of the recent update, completes its task and overwrites the newer Version 2 with the older Version 1 it was copying.

The final result: The destination index is left in an incorrect state, containing the outdated Version 1. The live update has been lost.

Explored Alternatives

Before diving into the final solution, here are some alternatives which we’ve explored but didn’t consider for the time being:

Use External Versioning: This would require our application to generate and track a unique version for every update and move away from partial updates. This was a significant architectural change we could not undertake at the time.

Pause Online Updates: We could have stopped all writes during the migration, but re-indexing our entire dataset was estimated to take around two hours. Forcing our data to be stale for that long was not an acceptable user experience.

Our Solution: A Blue-Green Strategy for Elasticsearch

Since the standard approach was not feasible, we designed our own solution based on our existing event-driven architecture and the principles of a blue-green deployment.

System components:

ES Indexer Old → The existing indexer writing to the old index (blue deployment).
ES Indexer New → A new indexer configured to write to the new index (green deployment).
cases_index_old → The existing index, serving production traffic.
cases_index_new → The new index, with updated mappings, populated in parallel.

Migration Steps:

Deploy the "Green" Stack: We created the new cases_index_new with the updated mappings. We then deployed a completely new instance of our indexer, ES Indexer New, configured to write to this new index. The existing ES Indexer Old continued to write to cases_index_old as usual.
Mirror the Data Stream: The key to this strategy was Kafka (or any messaging service that provides similar guarantees). Both the old and new ES Indexers subscribed to the same Kafka topic, but they used different consumer groups. This ensures that both indexers receive an identical, complete copy of all messages, allowing them to build their respective indices independently but identically.
Backfill from the Source of Truth: To populate the new index with all historical data, we triggered a scheduled job in the Our Service. This job read every single case from our primary database in batches and published them as documents to the Kafka topic. To prevent dirty reads from concurrent online transactions, the job used exclusive locks when fetching cases.
Maintain Data Consistency: While the backfill was running, any live user updates were also being published to the same Kafka topic. Because Kafka guarantees message order within a partition, we could be certain that both the old and new indices would process historical data and live updates in the exact same sequence. Eventually, both indices would reach a state of complete parity.
The Cutover: Throughout this entire process, Our Service's search APIs continued to read exclusively from the old index, ensuring no impact on users. Once we confirmed that the backfill job had finished and the new index was fully up-to-date, we performed the cutover: we reconfigured the Our Service to use the new index alias for all read operations.
Decommission the "Blue" Stack: After verifying the stability and correctness of the new index in production, the ES Indexer Old and cases_index_old were safely decommissioned.

Future Improvements

This manual blue-green process was a great success, and it has inspired several ideas for future enhancements to our platform:

Support Dual-write in a single ES Indexer: The ES Indexer could be enhanced to support dual-write natively to avoid duplicated deployments and to make automation easier.
Automation and Tooling: The entire process, from deploying the "green" stack to triggering the backfill and performing the final alias switch, could be automated into a CI/CD pipeline. This would reduce the risk of human error and make future migrations much faster.
Automated Consistency Checks: Build automated tools to verify index consistency beyond spot checks.
Version Tracking: Introduce explicit versioning for documents to support more advanced indexing strategies (e.g. revisiting the official _reindex strategy).

Conclusion

Handling breaking changes in a live production system is always a daunting task. While the official Elasticsearch _reindex strategy is a powerful tool, it's not a universal solution. By carefully analyzing the constraints of our own architecture—particularly our use of partial updates and reliance on Kafka for ordering—we were able to identify its pitfalls and engineer a custom solution that fit our needs perfectly.

Our blue-green deployment approach leveraged our existing event-driven infrastructure to build a new index in parallel with the old one, guaranteeing data consistency and achieving a true zero-downtime migration. This experience was a valuable learning opportunity, reinforcing the principle that the best solution is often the one that is most thoughtfully adapted to the system it serves.