[ELK] Reading, Indexing and Visualizing Windows logs with ELK Stack

Hello Friends,

A system administrator knows how much system logs can help to troubleshoot critical problems. And what if those logs are indexed at one place and can be visualized in charts etc. Fun, isn't it?

Let's see what can help us to set it up..

1. Nothing special to be done at Elasticsearch just run it by executing elasticsearch.bat from bin directory.

	D:\Prep\ELK\elasticsearch-2.1.1>bin\elasticsearch.bat
	[2016-01-23 15:06:38,555][WARN ][bootstrap ] unable to install syscall filter: syscall filtering not supported for OS: 'Windows 7'
	[2016-01-23 15:06:39,281][INFO ][node ] [Wild Child] version[2.1.1], pid[17364], build[40e2c53/2015-12-15T13:05:55Z]
	[2016-01-23 15:06:39,282][INFO ][node ] [Wild Child] initializing ...
	[2016-01-23 15:06:39,592][INFO ][plugins ] [Wild Child] loaded [], sites [kopf]
	[2016-01-23 15:06:39,641][INFO ][env ] [Wild Child] using [1] data paths, mounts [[New Volume (D:)]], net usable_space [71.4gb], net total_space [270.4gb]
	, spins? [unknown], types [NTFS]
	[2016-01-23 15:06:43,299][INFO ][node ] [Wild Child] initialized
	[2016-01-23 15:06:43,299][INFO ][node ] [Wild Child] starting ...
	[2016-01-23 15:06:44,750][INFO ][transport ] [Wild Child] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}, {[::1]:9300}
	[2016-01-23 15:06:44,761][INFO ][discovery ] [Wild Child] elasticsearch/NU9kgl8jQcuPJeiDLj4tMg
	[2016-01-23 15:06:48,840][INFO ][cluster.service ] [Wild Child] new_master {Wild Child}{NU9kgl8jQcuPJeiDLj4tMg}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-join(ele
	cted_as_master, [0] joins received)
	[2016-01-23 15:06:49,072][INFO ][gateway ] [Wild Child] recovered [0] indices into cluster_state
	[2016-01-23 15:06:51,750][INFO ][http ] [Wild Child] publish_address {127.0.0.1:9200}, bound_addresses {127.0.0.1:9200}, {[::1]:9200}
	[2016-01-23 15:06:51,751][INFO ][node ] [Wild Child] started

view raw elasticsearch_startup.log hosted with ❤ by GitHub

2. Prepare a config so that Logstash can read windows logs. We need to add config in input plugin.

	input {
	eventlog {
	type => 'Win32-EventLog'
	logfile => 'System'
	}
	}

view raw eventlog.conf hosted with ❤ by GitHub

This eventlog plugin helps logstash to read windows logs. Windows logs are stored in binary format and can be accessed using only Win32 API. This plugin takes several configuration options but all of those are optional. These config options are codec, add_field, logfile, interval, tags, type out of which we are using logfile and type.

a) type has no default value and used for filter activation. The given type is stored as part of event and we can search events in Kibana using this.
b) logfile is an array of String and contains Application, Security, System as default values in array. In the config we have used only System.

3. Store this config to a custom file in conf in Logstash with name logstash-windowslogs.conf
and run logstash using

	>bin\logstash.bat -f conf\logstash-windowslogs.conf
	io/console not supported; tty will not be manipulated
	Settings: Default filter workers: 2
	Logstash startup completed

view raw logstash_startup.log hosted with ❤ by GitHub

where complete config file will be as -

	input {
	eventlog{
	type => 'Win32-EventLog'
	logfile => 'System'
	}
	}
	output {
	elasticsearch {
	hosts => ["localhost:9200"]
	}
	}

view raw logstash-windowslogs.conf hosted with ❤ by GitHub

4. We're done with configuration of Logstash. Next is to read this in kibana. Here is our config to connect Kibana to Elasticsearch to read indices.

	# Kibana is served by a back end server. This controls which port to use.
	server.port: 5601
	# The host to bind the server to.
	server.host: "localhost"
	# If you are running kibana behind a proxy, and want to mount it at a path,
	# specify that path here. The basePath can't end in a slash.
	# server.basePath: ""
	# The Elasticsearch instance to use for all your queries.
	elasticsearch.url: "http://localhost:9200"

view raw kibana-config.yml hosted with ❤ by GitHub

5. As soon as we start Logstash, it will start collecting all the events and index them to elasticsearch. Now lets run Kibana by executing kibana.bat in bin directory.

	>bin\kibana.bat
	log [15:02:40.715] [info][status][plugin:kibana] Status changed from uninitialized to green - Ready
	log [15:02:40.762] [info][status][plugin:elasticsearch] Status changed from uninitialized to green - Ready
	log [15:02:40.784] [info][status][plugin:kbn_vislib_vis_types] Status changed from uninitialized to green - Ready
	log [15:02:40.794] [info][status][plugin:markdown_vis] Status changed from uninitialized to green - Ready
	log [15:02:40.802] [info][status][plugin:metric_vis] Status changed from uninitialized to green - Ready
	log [15:02:40.808] [info][status][plugin:spyModes] Status changed from uninitialized to green - Ready
	log [15:02:40.825] [info][status][plugin:statusPage] Status changed from uninitialized to green - Ready
	log [15:02:40.835] [info][status][plugin:table_vis] Status changed from uninitialized to green - Ready
	log [15:02:40.867] [info][listening] Server running at http://localhost:5601

view raw Kibana-startup.log hosted with ❤ by GitHub

6. Once Kibana is started, we can open its interface in browser using http://localhost:5601/. We need to configure index pattern. By default, it shows logstash-* in index name or pattern field and @timestamp in Time-Field name. Hit create by keeping defaults.

7. Now click Discover on the top menu to see the results. It will show histogram of counts of event.

All the results are also listed below histogram but for security reasons, I have not shown them here..

That's it. Pretty clean, what you say?