• Uncategorised
  • 0

Elastic Search 6.0: not that new, but quite improved

Shay Bannon has been called a person who has written more code than what is humanly possible[1]. This has led him from working on a solution for search in his spare time to building an open source framework and a global company around that with clients such as eBay and Verizon[2]. Elastic Search has come a long way and Elastic is about a lot more than search.

Today Elastic is announcing version 6.0 of what is now an entire stack built around the core premise of search, and Bannon as the recently appointed CEO[3] discussed with ZDNet the past, present, and future of Elastic Search and the trends shaping the industry.

Elastic Search as a system of record

what’s hot on zdnet

Our conversation did not start with the new features of version 6.0. If you are part of the Elastic community, you may already know them. If not, you may not get very impressed at first sight.

This is interesting in and by itself, but we thought it would be a good idea to shed some light on what Elastic Search can and cannot do before embarking on a detailed discussion on new features. When Bannon started working on Elastic Search, it was all about storing JSON and having a powerful search language. That was 8 years ago, and as he notes “NoSQL was all the rage.

To me ElasticSearch was something I was passionate about, so I did not want to be part of any hype cycle. That would be shifting the focus from the value Elastic Search can bring as a very powerful search solution. People have been asking — can I replace my MongoDB, or my Oracle database with Elastic Search?

Can it work as a system of record? My answer has always been, if you place Elastic Search next to any of these systems, be it Cassandra or Hadoop or whatever, it will bring value. It has this angle of how to solve challenges under a search prism that no other system has.

But our goal is not to replace these systems.” This “system of record” discussion has been an ongoing one regarding Elastic Search. In previous versions there has been work in the context of the Jepsen project that revealed conditions under which data loss in Elastic Search may occur[4].

Even today, Kyle Kingsbury, Jepsen’s mastermind, says “I would not use this as a system of record, so you would put your data in S3 or Postgres and have a replication tool so that it reiterates the data.”

Elastic Search 6.0: not that new, but quite improved

Not every database is entirely reliable. In fact, most are not. ElasticSearch is not even a database in the strict sense, so it should not be used or judged as one.

Image: Jepsen / Kyle Kingsbury Bannon seems to agree with that in his way. He acknowledges Kingsbury’s contribution in pointing out deficiencies in Elastic Search sharding and says they have worked with him in trying to address them, and this work has made lots of progress and is openly documented.

And if someone wants to use Elastic Search as a core system to store financial transactions on, Bannon would not advise them to do this. In the end, Bannon concedes, Elastic Search has not been around for as long as the Oracles of the world have, and this means it’s by definition less mature. Of course, as he notes, if your data on Elastic Search gets lost or corrupted, this makes for a bad user experience, so they are working on resilience.

For Bannon however resilience is not all about distributed algorithms and sharding, but also about things such as stability and memory footprint: “if you end up writing to a system that causes your runtime to pause, it’s indistinguishable from a network partition.

We have invested heavily in this area and there are many improvements in 6.0.”

Elastic Search 6.0

One such improvement Bannon highlights is based on something called sequence IDs. It’s the ability to have consensus on the sequence of operations between a primary and a replica shard. Bannon says this greatly improves the ability to maintain coherency between data, and helps address a gap Elastic Search has had historically.

Another area that Bannon highlights is what he calls circuit breakers. This is about improving detection of requests that end up consuming lots of resources so they can be isolated without bringing down a cluster. He says a lot of work has gone in the ability to track and stop queries when needed, as well as working with Java off-heap memory techniques and structures.

As a result, memory footprint today is much smaller than it used to be. Many other improvements are in that category as well – things that require expert knowledge not just to implement, but also to comprehend and evaluate the impact of. Things like index sorting[5] or sparse fields storage, which end up trading a few milliseconds of query response time for what Bannon says amounts to between 30 percent and 70 percent of savings in storage space.


Index sorting is a new feature in ElasticSearch 6.0.

It takes some engineering skills and a long blog post to get it, but the end result is better performance. Image: ElasticSearch In the end, if you don’t spend the time to dig into these new features, there’s a good chance you may remain unimpressed by Elastic Search 6.0.

Even though Bannon says they see the new version as something that has been progressively been shared with and explained to the community via a series of blog posts, he acknowledges the fact that not everyone will necessarily have the time and energy for that. For the record, other new features in Elastic Search 6.0 are spread out across the Elastic Search stack, which is comprised of Kibana, Beats and LogStash. These are Elastic Search’s solutions for visualization and dashboards, data ingestion and log storage respectively.

The Elastic Search stack is complemented by X-Pack, a premium set of features that include things like graph visualization and anomaly detection via machine learning.

Listening to users, charging to the future

Elastic Search started as a modest solution centered around making Lucene, the open source framework for indexing and search which is heavily used to this day, usable for efficient search on JSON. Discussing with Bannon the progression that has led to where Elastic Search is today, it becomes clear that what he sees as the key to Elastic Search’s success is also the reason you may remain unimpressed with the new features. For Bannon it’s always been about connecting with and listening to the community. “One of the things i’ve learned about building a successful open source company is that you need to be a good listener”, he says. “After releasing core Elastic Search, it was clear that people wanted to have visualization and dashboards on top of that.

So we brought Kibana in-house and made it part of out stack. When I started working on ElasticSearch, i never imagined one day storing logs would be part of it. But people started doing that, and today we are the number one open source solution and in fact a system of record for this.

People are happy with that, our solution works much better than Splunk for example.”


ElasticSearch is open source and treasures the relationship with its community. But not everyone in the community is entirely happy about every aspect of it. Image: Cohesive Networks[6]

This has been pretty much the story of how Elastic Search has grown, and will apparently continue to be. Bannon does not believe in going away and coming back with radically new things that may be asking people to bet on them, but rather in taking progressive steps. Elastic Search has embraced things such as the cloud, or machine learning, but is not going all-in on them either.

When discussing the move to the cloud, Bannon says Elastic Search was designed to work with AWS from the start, and this has contributed a great deal to its success. Today Elastic Search also runs on Azure and Google cloud, with which there also is a partnership, as well as with Alibaba cloud[7]. There are not many enterprise software providers that are big in China like that, and Bannon sees this as validation of the strategy.

Still, he emphasizes that for them it’s all about empowering users: “when we made the move to offer a managed version in the cloud 3,5 years ago, it was not to force our users, but rather to be there for them. They can run Elastic Search on whatever cloud they want, or use our managed version, or run on premise. We don’t want to leave anyone behind, and with Elastic Cloud Enterprise[8] we run the same code our users run”.

As for the move to IPaaS[9] platforms and machine learning (ML), Bannon says IPaaS is very much in accord with what they do. The progression towards analytics is happening also in Elastic Search, with the recent acquisition of Prelert’s ML technology having been incorporated in the stack. Initially this is used for anomaly detection, and Bannon says it’s already seeing great adoption and the next step is to add forecasting capabilities.

Other areas that Elastic Search will target next are application performance monitoring and fraud detection, security analytics and taking visualization up a notch. This is clearly moving up the stack to domain-specific applications, which may provide a new set of challenges as Elastic Search will have to compete against incumbents. Bannon however believes in the strategy that has been paying off so far:

“Five years ago we were a small company with a relatively popular open source product, and look where we are today.

The way we did it is that we embrace users and listen to them, and make sure that when they innovate on top of our platform, this will soon find its way to the platform.

If we as a company behave in the same way that we have been, I have no worries.”

Tech Pro Research


  1. ^ person who has written more code than what is humanly possible (techcrunch.com)
  2. ^ clients such as eBay and Verizon (www.elastic.co)
  3. ^ recently appointed CEO (www.elastic.co)
  4. ^ work in the context of the Jepsen project that revealed conditions under which data loss in Elastic Search may occur (thenewstack.io)
  5. ^ index sorting (www.elastic.co)
  6. ^ Cohesive Networks (www.slideshare.net)
  7. ^ Alibaba cloud (www.zdnet.com)
  8. ^ Elastic Cloud Enterprise (www.elastic.co)
  9. ^ IPaaS (www.zdnet.com)

You may also like...

Leave a Reply

Your email address will not be published. Required fields are marked *

Apps & Games Clothing Electronics & Photo Large Appliances
Baby Womens Apparel Garden Lighting
Beauty Mens Apparel Outdoors Luggage
Books Girls Apparel Health & Personal Care Pet Supplies
Car Boys Apparel Home Shoes & Bags
Motorbike Computers & Accessories Kitchen Equipment Sports & Outdoors
Fashion DIY & Tools Jewellery Toys & Games