Hello wizards and sorcerers of the WebFOCUS community,

I'm trying to create a report using data from IIS log files (stored in the W3C format), but I'm running into a few problems creating the master(s).

For starters, IIS changes the filename of the active log-file, so I can't point to "the" log-file to create a master, or can I? IIS on my development environment creates files named ex110419.log, ex110420.log, etc. The filename is in the format ex+jjmmdd+.log.
I could set IIS to create one big endless log-file, but that has some obvious drawbacks. I'd rather not.
Is there some way to make IIS less brain-dead about how it names its log-files? Personally I wouldn't mind replacing it with Apache or some-such, but I'm not in charge here...

Then the next problem; I added an adapter for delimited text files to parse these log-files. Then I attempted to create a master for one of the above described log-files. Getting it to accept ' ' (space) as a delimiter wasn't too hard (just a matter of editing the access file by hand, ahem), but the log files in case start with four lines of comments as the header while the master only seems to skip the first line...

Start of log-file:

#Software: Microsoft Internet Information Services 5.1
#Version: 1.0
#Date: 2011-04-20 07:34:22
#Fields: date time c-ip cs-username s-sitename s-computername s-ip s-port cs-method cs-uri-stem cs-uri-query sc-status sc-win32-status cs-host cs(User-Agent) 
2011-04-20 07:34:22 127.0.0.1 ENSCHEDE\hertroys_a W3SVC1 WEP_VR_1059 127.0.0.1 80 GET /ibi_apps/WFServlet IBICFG_action=CFGGET&IBICFG_objtype=INSTALL_INFO 200 0 localhost WFSCOM
2011-04-20 07:34:25 127.0.0.1 ENSCHEDE\hertroys_a W3SVC1 WEP_VR_1059 127.0.0.1 80 POST /ibi_apps/WFServlet - 200 0 localhost WFSCOM
2011-04-20 07:34:25 127.0.0.1 - W3SVC1 WEP_VR_1059 127.0.0.1 80 POST /ibi_apps/WFServlet - 401 5 localhost WFSCOM

The generated master is this:

FILENAME=EX110420, SUFFIX=DFIX    ,
 DATASET=C:\WINDOWS\system32\LogFiles\W3SVC1\ex110420.log, $
  SEGMENT=EX110420, SEGTYPE=S0, $
    FIELDNAME=#SOFTWARE:_MICROSOFT_INTERNET_INFORMATION_SERVICES_5.1, ALIAS='#Software: Microsoft Internet Information Services 5.1', USAGE=A4096V, ACTUAL=A4096BV, $
    FIELDNAME=FIELD_2, ALIAS= , USAGE=A87V, ACTUAL=A87BV, $
    FIELDNAME=FIELD_3, ALIAS= , USAGE=A75V, ACTUAL=A75BV, $
    FIELDNAME=FIELD_4, ALIAS= , USAGE=A4, ACTUAL=A4B, $

With the access file:

SEGNAME=EX110420, DELIMITER=' ', HEADER=YES, $

Any suggestions on how to go about this? What I'm trying is possible, right?

This message has been edited. Last edited by: Wep5622,

I think the best way to read these files, would be to TABLE and HOLD, removing the comments (Lines beginning with #).

Then apply a master to it.

You will need to make sure that the fields are large enought to cater for the varying sizes of the content.

For future reference; It is possible to do this, but having IIS log to a database saves you a lot of head-aches.

Problems you will run into:

• The available fields and the order in which they are defined can/will vary per server. They can even vary on the same server between different dates or within a file! This depends on when and how people altered the logging settings in IIS.
  
• The default for IIS seems to be a log-file per day. It's probably possible to generate a master-file for the day you want to read the log-file for, but even then, you end up with a lot of garbage in your masters (virtual) directory. Whether it's per day or per month or per year doesn't really matter, the log-files are going to be rotated at some point, in which case you need to change your master-file. It's also likely that your log-files are in different locations on different servers, for which you also need to change your master(s).
  

So unless you fancy ending up screaming, hitting your head against walls and being taken away stray-jacketed in a white van, my advice is to log to a database instead. It makes life a LOT easier.

For the hard-headed:

You can create a master that has a single field for an entire line of log-text:

FILENAME=ex110420, SUFFIX=DFIX    ,
 DATASET=C:\WINDOWS\system32\LogFiles\W3SVC1\ex110105.log, $
  SEGMENT=EX110420, SEGTYPE=S0, $
    FIELDNAME=LINE, USAGE=A2048, ACTUAL=A2048, $

Using that master, you can TABLE FILE the log-file and strip off any line starting with '#'.

TABLE FILE EX110420
PRINT LINE
WHERE LINE NOT LIKE '#%';
ON TABLE HOLD AS LOGFILE FORMAT ALPHA
END

Maybe it's even possible to read the field-names (in their particular order) from the appropriate comment line(s) - mind that there can be multiple of those if the logging settings changed during operation (does require an website restart though)!

After that you can apply a master with your preferred field definition. I used:

FILENAME=log, SUFFIX=DFIX    ,
 DATASET=C:\ibi\apps\log\LOGFILE.ftm, $
  SEGMENT=LOG, SEGTYPE=S0, $
    FIELDNAME=DATE, USAGE=A10, ACTUAL=A10,
      TITLE='Date', DESCRIPTION='Log Date', $
    FIELDNAME=TIME, USAGE=A8, ACTUAL=A8,
      TITLE='Time', DESCRIPTION='Log Time', $
    FIELDNAME=CLIENTIP, USAGE=A15V, ACTUAL=A15V,
      TITLE='Client-IP', DESCRIPTION='IP-number of the connecting client', $
    FIELDNAME=USERNAME, USAGE=A255V, ACTUAL=A255V,
      TITLE='Username', DESCRIPTION='Username used by the client', $
    FIELDNAME=SITENAME, USAGE=A255V, ACTUAL=A255V,
      TITLE='Website', DESCRIPTION='Name of the IIS-website this log is about', $
    FIELDNAME=HOSTNAME, USAGE=A255V, ACTUAL=A255V,
      TITLE='Hostname', DESCRIPTION='Host name of the IIS server', $
    FIELDNAME=HOSTIP, USAGE=A15V, ACTUAL=A15V,
      TITLE='Host-IP', DESCRIPTION='IP-number of the IIS server', $
    FIELDNAME=PORT, USAGE=I5, ACTUAL=A5,
      TITLE='Port', DESCRIPTION='Port connected to on the IIS server', $
    FIELDNAME=METHOD, USAGE=A8V, ACTUAL=A8V,
      TITLE='HTTP method', DESCRIPTION='HTTP method', $
    FIELDNAME=URISTEM, USAGE=A255V, ACTUAL=A255V,
      TITLE='Request URI', DESCRIPTION='Requested page name/path', $
    FIELDNAME=URIQUERY, USAGE=A1024V, ACTUAL=A1024V,
      TITLE='Request parameters', DESCRIPTION='Parameters passed to the requested page', $
    FIELDNAME=HTTPCODE, USAGE=I3, ACTUAL=A3V,
      TITLE='HTTP status', DESCRIPTION='HTTP return status (200, 404, 500, etc)', $
    FIELDNAME=WINCODE, USAGE=I6, ACTUAL=A6V,
      TITLE='Windoze code', DESCRIPTION='---', $
    FIELDNAME=CSHOSTNM, USAGE=A255V, ACTUAL=A255V,
      TITLE='Client hostname', DESCRIPTION='Client host name', $
    FIELDNAME=USERAGNT, USAGE=A255V, ACTUAL=A255V,
      TITLE='User-Agent', DESCRIPTION='Client user agent', $
    DEFINE TIMESTAMP/HYYMD-S MISSING ON ALL  WITH DATE=HINPUT(19, DATE | ' ' | TIME, 8, 'HYYMDS');
      TITLE='Timestamp', DESCRIPTION='Timestamp', $

Note that the order and availability of these fields depend on the contents of the aforementioned line of comment. It's NOT set in stone at all!

What you could do to handle the different filenames is go back to the old DOS command things.

DIR *.log > filenames.txt

this creates a file that hold all the filenames

Now you can run a small procedure that goes through this file, takes the first record, renames the file to a default filename, does all the stuff you want to do with it it and then takes the next file.
This is not real IBI software programming but it does what you need.