getstats.c, version 1.2

What is getstats?

Getstats is not in the public domain. However, you may distribute it or its supporting files and documents in any format freely was long as it is unchanged from the original distribution. Getstats may not be resold by itself or as part of a package without express permission of its author (me). Enjoy!

-- Kevin Hughes, kevinh@eit.com

1. Major report types
2. Usage
3. Command-Line Options
4. Internal Options
5. The Getstats Form Interface
6. Frequently Asked Questions
7. Distribution and Compilation Notes
8. Other Statistics Gathering Programs

1. getstats -c (concise report)

This basic set of statistics is always output when getstats runs. Using the -c option will only produce this statistics paragraph.

2. getstats -m (monthly report)

The -m option will produce a monthly report of server use. The dates in the report are the first day of reported activity for that month.

3. getstats -w (weekly report)

The -w option will produce a weekly report of server use. The dates in the report are always the Monday of that particular week.

4. getstats -ds (daily summary)

The -ds option produces a daily summary, which shows the aggregate number of requests for a particular day of the week.

5. getstats -d (daily report)

The -d option produces a daily report, which shows the number of requests per day and the date.

6. getstats -hs (hourly summary)

The -hs option produces an hourly summary, which shows the aggregate number of requests for a particular hour.

7. getstats -h (hourly report)

The -h option produces an hourly report, which shows the number of requests per hour, the day of the week, and the total number of requests for each day.

8. getstats -f (full report)

The -f option tells getstats to create a full report sorted by host name (and IP address). Use the -fa option to make a full report sorted by the number of accesses, the -fd option to create a full report sorted by the last access date, or the -fb option to create a full report sorted by the number of bytes transferred.

9. getstats -r (request report)

The -r option tells getstats to create a report of requests sorted by the request name. Use the -ra option to sort by accesses, -rd to sort by the last access time, -rb to sort by the number of bytes transferred, and -rf to sort by individual file sizes.

10. getstats -dn (domain report)

The -dn option generates a domain report, sorted by domain name. Use -da to sort by the number of requests, -dd to sort by last access date, -db to sort by the number of bytes transferred, or -du to sort by the number of unique domains. The unique domain number is the total number of unique sites under a domain. In the example above, for instance, a total of 3 unique sites came from the .au domain.

11. getstats -dt (directory tree report)

The -dt option generates a directory tree report, which cannot be sorted. The number of requests and last request date for directories and files is displayed. The request count for directories is the amount of requests for that directory plus the sum of all requests for the files and subdirectories under it.

If you find this report is empty, try using the -dr option without specifying a directory. This will tell getstats to make a tree report without verifying that the files and directories reported in the log file actually exist.

12. getstats -e (file) (error report)

-e generates a report of all malformed (or ignored) requests for all dates in the order they were encountered in the log file. If a filename is given as the argument to the option, bad requests will be appended to an error file, where they can be analyzed later.

getstats -a (all reports)

  example: getstats -a -fb

This will create all reports sorted by number of requests, with the exception of the full report, which is sorted by byte traffic, and the error report, which must be specified on the command line.

To see the usage, run getstats with a -z option.

-dr directory, -l file, -C, -N, -P, -G, -A, -O, -M (root directory, logfile and log type)

  example: getstats -dr "/usr/local/www"

The -l option specifies the log file to use. The -C, -N, -P, -G, -A, and -O options will tell getstats to expect the log file to be in either CERN, NCSA, Plexus, GN, MacHTTP, or UNIX Gopher format.

  example: getstats -l my.ncsa.log -N
  example: getstats -l my.plexus.log -P

The -M option tells getstats to expect the log to be in the "common" log file format, a standard that was agreed upon by the World-Wide Web community. If your log looks something like this:

  www.eit.com - - [01/Jan/1994:10:30:00 +0000] "GET /test.html" 200 123

then your server is using the format. Include the option in the command line:

  example: getstats -l cern.common.log -C -M
  example: getstats -l ncsa.common.log -N -M

If you use NCSA's httpd 1.2 or later, or CERN's server version 1.16b or later, you are probably using the common log format.

-sa string, -ss string, -sr string, -sp string (address and request masks)

For these four case-insensitive string masks, the following rules apply:

1. You can use asterisks as wildcards in specifying the string, at each or both ends of the string.
2. Masks without wildcards must match exactly.
3. You can make lists of masks, separated by commas.

  example 1: getstats -sa "*.com, *.edu" -ss "*.eit.com"
  example 2: getstats -sr "*.html, *.gif" -sp "*secret*"
  example 3: getstats -sa "*.*" -sp "/internal/demo.html"

  example 4: getstats -ss "dopey, sneezy, grumpy"

1. Sites from ".com" and ".edu" domains are reported, but anything coming from "eit.com" is skipped.
2. HTML and GIF requests are reported, but any requests with the word "secret" in them are skipped.
3. Addresses with a period in them are reported (this is useful for filtering local machines), and any requests exactly matching "/internal/demo.html" are skipped.
4. Requests coming from "dopey", "sneezy", and "grumpy" are skipped.

-sd string, -sh string, -sw string (date, hour, and day masks)

For these three case-insensitive string masks, the following rules apply:

1. The date string must be in the format "m/d/y". Ranges in brackets, such as "[xx-xx]" can be applied, and asterisks as wildcards can be used only in place of an entire field.
2. Other valid specifications for the date string: today, yesterday, thisweek, lastweek, thismonth, and lastmonth.
3. The hour string must contain numbers from 0 to 23, with 0 being the hour from midnight to 1 am, and 23 being 11 pm. Ranges such as "xx-xx" can be specified.
4. The day string must be a three-letter day of the week, such as "mon", "tue", and "wed". Ranges such as "xxx-xxx" can be specified.
5. Other valid specifications for the day string: weekends and weekdays.

  example 1: getstats -sd "*/[4-10]/93" -sh "6" -sw "mon"
  example 2: getstats -sd "1/[5-30]/1993" -sh "5-17" -sw "wed-sun"
  example 3: getstats -sd "[1-5]/*/[91-94]" -sh "-17" -sw "-thu"
  example 4: getstats -sd "lastweek" -sh "15-" -sw "tue-"

1. Only accesses that occurred from the 4th to the 10th on any month in 1993 on a Monday at 6 pm will be reported.
2. Requests from January 5th to the 30th, 1993 are reported, as are requests from 5 am to 2 pm, on Wednesdays to Sundays.
3. Requests from January to May 1991 to 1994 are reported, as are requests from midnight to 2 pm, on Mondays to Thursdays.
4. Requests from last week are reported, as are requests from 3 pm to 11 pm, on Tuesdays to Sundays.

-i, -p, -ht (input and output)

  example: tail -100 my.log | getstats -i

-p displays a progress meter, so you can see where getstats is in its processing. The -ht option will report all reports in a single-page HTML format, with appropriate links to server support URLs and this documentation.

-dl number, -df file (domain report options)

With the number of levels set at 2:

The -df option allows you to specify a file with descriptions for domain codes, to make the domain report a bit easier to understand. A file with descriptions is available by FTP at ftp.eit.com, in the /pub/web.software/getstats directory. You can also get the list by selecting this.

With domain descriptions:

-b, -ip, -t number (byte reporting, IP lookup, and top lines)

• Redirected URLs
• The sizes of files in personal HTML directories
• The sizes of scripts
• Old files that have been changed
• Requests that have been rejected (perhaps due to access control)

Counts for these requests, or counts for requests that can't be determined, are reported as zero.

The -ip option will make getstats attempt to look up host names from IP addresses. This feature slows processing time, but is useful in analyzing logs from servers that don't look up IP addresses, such as the CERN server.

The -t option allows you to specify how many top lines to report in full, request, and domain reports. Using this, you can easily generate "Top 10" lists and short summaries.

Getstats has a number of options that are not available from the command line which must be specified in the source code:

• The root directory of your web or gopher tree, the URL for your server, and aliases for empty and slash ("/") requests.
• The mark character for graphs, and the number of requests and bytes per mark for different reports. The default settings have been tuned for light server use, or about 200 to 1,000 requests per day.
• The character length to truncate graph reports and requests reports to.
• The option to display all dates in local or GMT time (the default is GMT time), and what time zone the logfile uses.
• Whether or not you want to show numerical domains in the domain report.
• Whether or not you want to report hours, minutes, and seconds of the last access time in non-graph reports.
• Whether or not you wish to show files and their related statistics as well as directories in directory tree report.
• Whether or not to check the actual files to see if they exist in the directory tree report.
• For HTML request reports, whether or not to display requests as selectable URLs.
• If you have a GN server, whether you want to report both Gopher and HTML requests or not.
• Whether or not you want to display an image in your HTML reports (like the one on this page).
• How dates are to be displayed - either as M/D/Y or D/M/Y.

You can get the C program statform.c and the HTML form statform.html by FTP at ftp.eit.com in the /pub/web.software/getstats directory. Or download statform.c and statform.html directly.

To start using the form:

1. Compile getstats with the "CGI" option (you'll need to edit the source code for this).
2. Edit statform.c, specifying the default log file to analyze, the root directory of your Web or Gopher tree, and the path to your getstats program.
3. Compile statform.c and put the executable where you store your other CGI programs.
4. Edit statform.html and specify the URL that will execute the statform program.
5. Make sure both statform and getstats are executable by your server.
6. Load up statform.html and have fun!

Getstats crashes and burns when I run it.

First, double-check that you've specified the correct format for your log file. If it's in the "common" format, add the -M option to the command line. If all else fails, and you think there may be a bad line in your log file, try to narrow down the section under which getstats crashes and email it to me so I can test it out.

How can I use getstats on multiple log files?

A good idea is to use the -i option on previously archived logs, for instance:

zcat log.*.Z | getstats -i

Will getstats be rewritten in perl?

It might be rewritten in perl when I learn perl! However, if someone would like to donate their time towards writing a perl version, please let me know so there's no duplication of effort.

How do I run getstats as a CGI script?

First, uncomment the "#define CGI" line in the source code and compile. To call getstats from a URL with options, use a question mark after the program name, plus characters for spaces, and "%22" for quotes in the command line, such as:

Normal example: getstats -a -fb
   URL example: http://www.eit.com/cgi-bin/getstats?-a+-fb
Normal example: getstats -ss "*.com"
   URL example: http://www.eit.com/cgi-bin/getstats?-ss+%22*.com%22

Why is my directory tree report empty?

If an actual root Web directory is specified, nothing will be reported if getstats can't find the files. Use the -dr option with no arguments to generate a report if no physical directory exists.

Why are some files reported as directories in my tree report?

Unless you tell getstats where the actual files exist, it can only make a "best guess" as to whether a file is a directory or not. Use the -dr option with your root Web directory to tell it where your files are. See the above documentation for more details.

The reported errors from my "common" format log file look OK to me!

The second to last field on each line indicates the status code the server returned. Codes not in the "2xx" range mean some sort of error occurred. You can get a list of these codes at http://info.cern.ch/hypertext/WWW/Protocols/HTTP/HTRESP.html.

Can I run getstats with my Windows NT server?

It appears that the EMWACS Windows NT server uses the CERN format. For their server, include the -C option in the command line.

You can get the program by FTP'ing ftp.eit.com and going to the /pub/web.software/getstats directory. Some files you'll need for compiling under VMS, as well as the getstats icon (which you can use in your HTML reports) are there also.

Other environments that getstats has reportedly compiled and run under with no problems are Solaris 2.3, DEC Ultrix, DEC OSF/1 V1.3 in Alpha AXP, SGI with Irix 4.0, AIX, NeXTstep, HPUX, A/UX, and VMS (with or without UCX). If you have compiled successfully on another platform or experience problems, please let me know.

Some nice examples of what people have done with getstats (without or without statform) are below:

• Harvey Mudd College's Web statistics
• Backgrounds Data Center - Naval Research Laboratory
• University of Florida Computer and Information Sciences (combined with other programs)

Wanted!

• Code to generate graphs (via gnuplot and/or pbm utilities) either internally or from completed reports.
• Suggestions for features that are most needed for administrators.
• Ways I can speed up getstats and/or make it more efficient.
• Comments, improvements, patches, and suggestions are greatly appreciated - you can send them all to kevinh@eit.com.

• WWWstat, a full-featured NCSA server log analyzer written in Perl.
• wusage, a C program that can generate graphs and nifty graph icons along with basic CERN and NCSA log statistics.
• WebReport, a Perl NCSA log analyzer that comes with NCSA's httpd.
• gnlog, a Perl GN log analyzing program.
• glog, a Gopher log analyzer that can sort statistics in many ways. This and other Gopher utility software is available at the boombox.micro.umn.edu Gopher.