Cool Visualization for Web Server Logs

[lin0]

Fellow Monks,

I wanted to share with you this article I found on SysAdmin. The article is titled: “A New Visualization for Web Server Logs.” I think it presents a really interesting way of analyzing web server requests. Here is the introduction of the article:

“There are well over a hundred web server log analyzers (Google Directory for Log Analysis) or web statistics tools ranging from commercial offerings such as WebTrends to open source ones such as AWStats. These take web server logfiles and display numbers such as page views, visits, and visitors, as well as graphs over various time ranges. This article presents the same data in those logfiles in a very different way: as a 3D plot. By the end of this article, I hope you will agree with me that the visualization described herein is a novel and useful way to view the content of logfiles.

The logfiles of web servers record information on each HTTP request they receive, such as the time, the sender's IP address, the request URL, and the status code. The items in each request are fairly orthogonal to one another. The IP address of a client has no relation to the URL that it requests, nor does the status code of the request to the time of the request. If that is the case, what could be a better way to display these n columns from the logfiles than an n-dimensional plot?

When an administrator observes anomalous behavior on a web server, she reaches out for web statistics reports, as they are usually all there is as a record of past activity. These often prove fruitless, mainly because web statistics is primarily a marketing-oriented view of web server activity. The next step is to take the raw logfiles apart with ad hoc scripts. The sheer mass of data makes it difficult to reduce it to a few numbers that reveal the cause of the problem. Another complication is that you may not quite know what you are looking for other than that it is abnormal behavior. The path this article takes is to provide a visualization of raw data such that cause or causes make themselves visible. This comes from the real-life experience of a client, where crippling performance problems appeared out of nowhere.”

Cheers,

lin0

[moklevat]

This is an appealing approach to log visualization. Thank you for the pointer. One suggestion I might make to monks interested in something like this would be to consider using the 3D plotting modules from PDL (i.e. PDL::Graphics::TriD instead of gnuplot. I like gnuplot for generating static images, but for interactive 3D plots, I find PDL's OpenGL creations to be outstanding.

[renodino]

I saw an even kewler visualization a couple years ago (alas, I can't recall the link). It generated a little "Sim" city type animation of website traffic, with the major pages being tall buildings, and the number of visitors represented as pedestrians, taxis, or buses, visiting each page/building. Not sure just how useful it was, but it was serious eye candy. (sure wish I could find that URL...). And it was all animated so you could see the traffic flow as visitors moved from one page to the next.

Update:
Found it! Visitorville