[ELK] An introduction to ELK Stack - Elasticsearch, Logstash, Kibana

... I had more than 5 servers continuously running on my computer when I was in college. My interest in PHP led me to install Apache, MySQL, and a mail server so that I could host multiple sites from my machine. Apart from these there were ftp, vnc etc. Whenever there was a problem with a server, I would go and check logs of that server. Even till 2 years back, I used to do same for other servers also. This time the list added app servers for Liferay.

I used to check logs for each servers individually, until I got to know about Splunk, a tool which my client was using to see all of the logs at one place. It was providing way more than just view, I could search logs for a defined time range etc. It was not collecting logs from just one server machine but almost 20 machines and for few machines, multiple servers. Troubleshooting was easy this way. I was not logging in to 20 different machines to check logs and look for possible problems. Honestly, I had no access to most of them. But I had access to Splunk, and I could query logs already indexed and could diagnose which server had problems.

But, Splunk (full features) is not free.

Then, what are the other options. Let me give you a hint in the image below..

Google (GOD) gave a hint : Look at ELK, once!

Out of curiosity, I just jumped to knowing ELK Stack rather than checking any search results. Instant, Google said once. :P

ELK = Elasticsearch, Logstash, Kibana.

Below image would tell you how this stack works.

 The above is a very very simplified representation of ELK stack.

Elasticsearch - Indexes the data sent to it. The core of elasticsearch is lucene.

Logstash - Its a data pipeline which can read data from a number of sources. There are more than 200 plugins available for logstash classified in four categories - Input, Output, Filter, Codec plugins.

Kibana - Whatever elasticsearch has indexed, Kibana gives opportunity to visualize that data in different forms. The data can be queried, be listed, be drawn as charts.

This was just a very basic introduction to these components. These components offer a lot more than just these. We will get to learn more about these components in future posts.

Until next time :)