StoneBlog.stonesoft.com

This article refers to previous post in which I illustrated how to create a logging profile to allow a 3rd party device syslog stream to be received by StoneGate Log Server.

I’ll deepen this information in this post by showing how to go from log collection to centralized log processing and reporting, using an Apache Web Server as log sending device. The ultimate goal is to use some parsed data from Web Server to create a basic report using StoneGate Reporting functionality included in StoneGate Management Center.

Let’s examine a sample log entry from Apache2 access.log file, where standard web requests are logged:

192.168.12.30 – - [05/Jun/2009:01:02:53 +0200] "GET / HTTP/1.1" 200 56 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.2; en-US; rv:1.9.0.10) Gecko/2009042316 Firefox/3.0.10"

This row, when sent via syslog facility (I used syslog-ng on a linux box), is slightly modified as follows:

<13>Jun  5 01:02:55 testbox 192.168.12.30 – - [05/Jun/2009:01:02:53 +0200] "GET / HTTP/1.1" 200 56 "-" "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.0.10) Gecko/2009042316 Firefox/3.0.10 (.NET CLR 3.5.30729)"

Let’s try to identify the parts of datastream that might be interesting for our purpose:

<13>Jun  5 01:02:55 testbox This is not interesting since it is added from syslog and should be skipped

192.168.12.30 This is the address of the client asking for web pages, could be interesting to map it to Source Address field.

[05/Jun/2009:01:02:53 +0200] This is timestamp of the original log, to be mapped to Creation Time field.

GET This is the HTTP request method, which can be GET or POST. We’ll map accordingly later.

/ This is the URI Request, again useful to be mapped to a meaningful field in SMC.

HTTP/1.1 This is the protocol version.

Rest of the stream is not considered relevant for this proof of concept.

As instructed in the previous post, we proceed to create the Logging Profile; important details about this process are:

• we need to skip the first four words since they are meaningless for our purposes. We use space as field separator, leaving the mapping to Ignore.
• fifth word is the HTTP Client IP Address, to be mapped to Src Addr field
• we have then a timestamp information that we need to be mapped to Creation Time field. Since this information needs to be formatted accordingly, we use Field Resolver type of field instead of the Field type. Format used for the information is dd/MMM/yyyy:HH:mm:ss Z Additional information about time formatting is available here.
• finally, we map remaining meaningful information to HTTP Request Method, HTTP Request URI, HTTP Request Version, HTTP Response Code and we leave the rest of stream in Information Message.

The window below shows the final mapping, together with prefixes (= field separators) used. The square in prefix field represents a space character.

Once done with the mapping, we associate the logging profile to the host element configured for the Apache Web Server as shown below:

System is ready, and as soon as web server will send logs to Log Server through the syslog channel, we should see data in StoneGate Log Browser.

Double clicking on a log row will provide additional details:

Since data have been mapped meaningfully, we can use those data to have nice reports as shown below:

 

Enjoy!

written by RoarinPenguin - 2,991 views \\ tags: 3rd party devices, logs, profile