Real-time Statistics With Elasticsearch
Softjourn

Real-time Statistics With Elasticsearch

July 01, 2019 by Softjourn

Search engines have always been the easiest and most natural way to dig up information. Their “snappiness” and the impression that fresh data is immediately available to make them a pleasure to use. However, “think” data cannot be simply flattened in a single search index, the way normal search engines do, but instead requires the join capabilities traditionally offered only by relational databases.

One of Softjourn’s clients turned to us with a request to provide a stand-alone data-tracking solution to complement its existing suite of publishing solutions. The idea was for these programs to receive tracking data from the client's existing technology through a new interface that would allow users to view and export relational data. Before writing the report application, Softjourn’s team started by adjusting ElasticSearch1.

Elasticsearch is a distributed, RESTful search and analytics engine capable of solving a growing number of use cases. As the heart of the Elastic Stack, it centrally stores your data so you can discover the expected and uncover the unexpected.

This search engine was the best solution for searching through all reports using a variety of filters and options.

As a feature rich platform to build search and data intensive solutions, ElasticSearch eliminates the need to design for specific-use cases, the way some NoSQL databases require. That was a big win for us with the client, as it enables teams to iterate on solutions faster than would otherwise be possible:

  • Elastic really means elastic—highly horizontally scalable
  • An open-source search engine built on Apache Lucene
  • Claims to be “real time”—and just about is!
  • Insanely powerful query language (a JSON DSL)
  • Interface all JSON over HTTP, making it easy to use in any language
  • Strong focus on analytics in addition to text search

Introducing the ELK Stack?

Together with Logstash, a tool for collecting and processing logs, and Kibana, a tool for searching and visualizing data in ElasticSearch—known as the “ELK” stack—, ElasticSearch is being adapted more and more widely every day. When it comes to actually using ElasticSearch, tons of metrics are generated.

To sum up:

  • Logstash is a tool for collecting, parsing, and transporting logs for downstream use.
  • Kibana is a web interface that can be used to search and view the logs that Logstash has indexed.
  • ElasticSearch connects Logstash and Kibana, which can store logs in a highly scalable, durable and accessible manner.
  • The ELK stack (ElasticSearch, Logstash, Kibana) provides a powerful set of tools for log aggregation, visualization, and search across all manner of devices, servers and applications. Best of all, it’s Open Source.

This combination, along with ElasticSearch itself, is already known and used by many developers. This proven search engine has been successfully implemented by hundreds of companies. Another tools developed under care of Elastic company are2:

  • Beats — single-purpose data shippers,
  • Elastic Cloud — hosting Elasticsearch clusters,
  • Machine Learning — for discovering data patterns,
  • APM — Application Performance Monitoring,
  • Swiftype — one-click site search.

Softjourn will continue to use the ELK stack in other projects as it’s proved itself to be fast and multifunctional—and it all comes in one package!

1Pagano Dritto G. (2019, March 27). An Overview on Elasticsearch and its usage [Blog post].
2Mydlarz D. (2019, February 4). 6 not so obvious things about Elasticsearch [Blog post].

Softjourn, Inc. is a technology services provider. The world’s leading experts that take pride in developing creative solutions for Fintech, Cards & Payments and Media & Entertainment applications. Contact us to give life to your ideas!