Optimizing Elasticsearch Serverless shutdowns with hollow indexing shards

I’m excited to share some of my recent work contributing to the evolution of Elasticsearch Serverless! I had the pleasure of leading the design and implementation of a fascinating new feature: “Hollow Shards”.

💡 What are Hollow Shards?

Idle indexing shards retained Lucene writers and segment readers in memory. This consumed significant heap space and added overhead during node shutdowns, relocations, or autoscaling.

Hollow shards solve this by completely unloading the IndexWriter and segment readers from memory until a new write arrives. The shard remains allocated, but its engine is hollowed, drastically reducing the indexing-tier’s heap footprint.

🛠️ The Implementation Challenge

Implementing this was no small feat! It required tackling fundamental invariants in Elasticsearch:

• Engine Swapping: We had to introduce a mechanism to safely swap between the regular writeable engine and the hollow engine on a live shard, which is something the architecture hadn’t previously supported. This involved careful management of mutexes and read/write locks across various call sites.
• Recovery Logic: We had to redesign recovery paths to skip translog replay and cache prewarming for hollow commits while ensuring data integrity upon unhollowing.

🚀 The Impact

The results have been impressive: in production, fleet-wide indexing-node shutdown times improved by up to 30%, and we saw substantial gains in heap usage across the indexing tier. Search latency remains unaffected!

For those interested in reading the full technical deep dive into this architecture, I highly recommend checking out the official blog post on Elasticsearch Labs: How Elasticsearch Serverless hollow shards cut indexing-node shutdowns by 30%.

Thanks to everyone who contributed to this project!