iis log analyzer

In October 2015, Netcraft found that after Apache and NGINX, Microsoft IIS is the third most-common web server used by the one million largest websites in the world. Although IIS’s popularity is declining, it’s still the most popular commercial web server and it is understandably popular among Microsoft developers.

Still, it is still difficult to receive relevant and actionable insights from the hundreds or even thousands of log entries that IIS web servers can generate every single second. Here, I wanted to look further into IIS log data to provide three instances of how DevOps engineers and system administrators can use Elasticsearch, Logstash, and Kibana to understand their IIS logs.

For reference: IIS logs can be exported in a W3C format, and the different fields can be customized in the IIS admin user interface.

Elasticsearch, Logstash, and Kibana — commonly known as the ELK Stack — can collect, parse, and store all IIS log data. Then, the information can be shown in the Kibana part of the stack in a way that users can be alerted to specific problems and then fix them immediately.

How to Parse IIS Logs Using Logstash

Often, one of the first things to do is to filter and enhance your IIS logs with Logstash. Here is a sample of a IIS log line and the related Logstash configuration that we happen use in our internal environment.

A sample IIS access log entry:

2015-12-08 06:41:42 GET /handler/someservice.ashx someData= 80 10.223.22.122 HTTP/1.1 Mozilla/5.0+(Windows+NT+6.1;+WOW64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/46.0.2490.86+Safari/537.36 1.2.1005047168.1446986881 http://www.logz.io/testing.aspx www.logz.io 200 638 795 0

The Logstash configuration to parse that IIS access log entry:

grok {
match => [ "message", "%{TIMESTAMP_ISO8601:timestamp} %{NOTSPACE:s-sitename} %{WORD:cs-method} %{URIPATH:cs-uri-stem} %{NOTSPACE:cs-uri-query} %{NUMBER:s-port} %{NOTSPACE:cs-username} %{IPORHOST:c-ip} %{NOTSPACE:cs(User-Agent)} %{NOTSPACE:cs(Cookie)} %{NOTSPACE:cs(Referer)} %{NOTSPACE:cs-host} %{NUMBER:sc-status:int} %{NUMBER:sc-substatus:int} %{NUMBER:sc-win32-status:int} %{NUMBER:sc-bytes:int} %{NUMBER:cs-bytes:int} %{NUMBER:time-taken:int}" ,
"message", "%{TIMESTAMP_ISO8601:timestamp} %{IPORHOST:s-sitename} %{WORD:cs-method} %{URIPATH:cs-uri-stem} %{NOTSPACE:cs-uri-query} %{NUMBER:s-port} %{NOTSPACE:cs-username} %{IPORHOST:c-ip} %{NOTSPACE:cs(User-Agent)} %{NOTSPACE:cs(Referer)} %{NUMBER:response:int} %{NUMBER:sc-substatus:int} %{NUMBER:sc-substatus:int} %{NUMBER:time-taken:int}" ,
"message", "%{TIMESTAMP_ISO8601:timestamp} %{WORD:cs-method} %{URIPATH:cs-uri-stem} %{NOTSPACE:cs-post-data} %{NUMBER:s-port} %{IPORHOST:c-ip} HTTP/%{NUMBER:c-http-version} %{NOTSPACE:cs(User-Agent)} %{NOTSPACE:cs(Cookie)} %{NOTSPACE:cs(Referer)} %{NOTSPACE:cs-host} %{NUMBER:sc-status:int} %{NUMBER:sc-bytes:int} %{NUMBER:cs-bytes:int} %{NUMBER:time-taken:int}"
]
}
geoip {
source => "c-ip"
target => "geoip"
add_tag => [ "iis-geoip" ]
}
useragent {
source => "cs(User-Agent)"
}
}

Now that you’ve seen how to use the ELK Stack to analyze Microsoft IIS log files, I’ll present below some use cases of when to use Elasticsearch, Logstash, and Kibana in this context.

IIS Log Analysis Use Cases

Operations Analysis

iis operational analysis

Whenever traffic significantly exceeds the long-term average of site visits or whenever error rates are higher than normal, the ELK Stack can be used to send alerts to operations teams. This way, slow website response rates can be fixed so that the user experience is not affected.

For example, Elasticsearch, Logstash, and Kibana can be used as a log management stack to see whenever there is a sharp decline in the number of requests for web pages or a significant spike in traffic that caused a server to crash. If both of these things occur in the same dashboard, you could be facing a DDos attack. In such a scenario, ELK can be used to find the origin IP address and block it.

Within our ELK Stack alerts feature, one visualization that we have is the number of log lines that cache responds to disk.

This visualization and more can be found in our ELK Apps library by searching for IIS.

iis technical seo analysis

Technical SEO

In SEO, the need to create quality content is becoming increasingly know. But if Google cannot access and index the content — or if the Googlebot hits its crawl limit before finding the content in the first place — then those marketing materials will be useless.

As the dashboard image shows, IIS log analysis with ELK can tell you when any page on your website was last crawled by Google, how Google prioritizes content in different subdomains and subdirectories, and which URLs are indexed the most and least. In one of our related posts, you can see how to use server log analysis for technical SEO.

Business Intelligence

IIS logs have everything that you need to analyze your application’s users — you can see everything from their geographic locations to the URLs that they visit to the quality of their UX. With ELK, users can correlate the IIS server data with infrastructure-level logs to gain more insight into how your infrastructure is affecting your visitors’ experiences on your website.

For example, memory loads, CPUs, and response times can be analyzed together to see if strong machines might be needed in your overall environment.

Many of these visualizations can be found in our free ELK Apps library by searching for IIS. Here are two examples: one is the response time that we’re getting per response code, and the other is a heat map of all of our visitors.

iis business intelligence

iis user heat map

In Conclusion

elk apps iis

IIS users should analyze their IIS logs regularly. From business intelligence to technical SEO and more, we have dashboards for these operations uses cases and more in our free ELK Apps library.

Have any tips on IIS log file analysis? We’d love to hear your thoughts in the comments below!

Logz.io is a predictive, cloud-based log management platform that is built on top of the open-source ELK Stack and can be used for log analysis, application monitoring, business intelligence, and more. Start your free trial today!

Asaf Yigal is co-founder and VP Product at Logz.io. Prior to Logz.io, Asaf co-founded Currensee, a social-trading platform, which was later acquired by OANDA in 2013. Prior to Currensee, Asaf played executive roles at Akorri in developing an end-to-end performance monitoring platform and at Onaro in developing a storage resource management platform. Both Akorri and Onaro were acquired by NetApp. Prior to Onaro, Asaf headed a research team in the Israeli Navy, taking an artificial intelligence system to military deployment. Asaf holds a B.S. from the Technion and is an Instrument-rated private pilot.