WD-logfile-960221
WD-logfile-960221This is a W3C Working Draft for review by W3C members and other interested parties. It is a draft document and may be updated, replaced or obsoleted by other documents at any time. It is inappropriate to use W3C Working Drafts as reference material or to cite them as other than "work in progress". A list of current W3C working drafts can be found at: http://www.w3.org/pub/WWW/TR
Note: since working drafts are subject to frequent change, you are advised to reference the above URL, rather than the URLs for working drafts themselves.
An improved format for Web server log files is presented. The format is extensible, permitting a wider range of data to be captured. This proposal is motivated by the need to capture a wider range of data for demographic analysis and also the needs of proxy caches.
Most Web servers offer the option to store logfiles in either the common log format or a proprietary format. The common log file format is supported by the majority of analysis tools but the information about each server transaction is fixed. In many cases it is desirable to record more information. Sites sensitive to personal data issues may wish to omit the recording of certain data. In addition ambiguities arise in analysing the common log file format since field separator characters may in some cases occur within fields. The extended log file format is designed to meet the following needs:
The log file format described permits customized logfiles to be recorded in a format readable by generic analysis tools. A header specifying the data types recorded is written out at the start of each log.
This work is in part motivated by the need to support collection of demographic data. This work is discussed at greater length in companion drafts describing session identifier URIs [Hallam96a] and more consistent proxy behaviour [Hallam96b].
A extended log file contains a sequence of lines containing ASCII characters terminated by either the sequence CR or CRLF. Log file generators should follow the line termination convention for the platform on which they are executed. Analysers should accept either form. Each line may contain either a directive or an entry.
Entries consist of a sequence of fields relating to a single HTTP transaction. Fields are separated by whitespace, the use of tab characters for this purpose is encouraged. If a field is unused in a particular entry dash "-" marks the omitted field. Directives record information about the logging process itself.
The following directives are defined:
The directives Version and Fields are required and should preceed all entries in the log. The Fields directive specifies the data recorded in the fields of each entry.
The following is an example file in the extended log format:
#Version: 1.0 #Date: 12-Jan-1996 #Fields: time cs-method cs-uri 00:34:23 GET /foo/bar.html 12:21:16 GET /foo/bar.html 12:45:52 GET /foo/bar.html 12:57:34 GET /foo/bar.html
The #Fields directive lists a sequence of field identifiers specifying the information recorded in each entry. Field identifiers may have one of the following forms:
The following prefixes are defined:
The identifier cs-method thus refers to the method in the request sent by the client to the server while sc(Content-Type) refers to the content type field of the reply.
The following identifiers do not require a prefix
The following identifiers require a prefix
Analysis tools may generate log summaries. A log summary lists the number of operations performed on a URI but omits information specific to a particular transaction.
The following field is mandatory and must preceed all others:
The following fields may be used in place of time to allow aggregation of log file entries over intervals of time.
This section describes the data formats for log file field entries. These formats are chosen so as to avoid ambiguity, minimize the difficulty of generation and parsing and provide for human readability.
Each logfile entry consists of a sequence of fields separated by whitespace and terminated by a CR or CRLF sequence. The meanings of the fields are defined by a preceeding #Fields directive. If a field is ommitted for a particular entry a single dash "-" is substituted.
Log file parsers should be tolerant of errors. If an entry contains corrupt data or is terminated unexpectedly the parser should resynchronize using the end of line marker and continue to parse the following entries. Entries must not contain any ASCII control characters.
<entry> = <field>* <end-of-line> <field> = <integer> | <fixed> | <uri> | <date> | <time> | <string> <digit = "0" | "1" | "2" | "3" | "4" | "5" | "6" | "7" | "8" | "9"
<integer> = <digit> +
Integers are represented as a sequence of digits.
<float> = <digit> [. <digit>*]
A URI as specified by RFC1738, relative URIs are specified by RFC1808. URIs cannot by definition include whitespace or ASCII control characters. Consequently no ambiguity arises from their use.
<date> = <digit> <digit> <digit> <digit> "-" <digit> <digit> "-" <digit> <digit>
Dates are recorded in the format YYYY-MM-DD where YYYY, MM and DD stand for the numeric year, month and day respectively. This format is chosen to assist collation using sort.
<time> = <digit> <digit> ":" <digit> <digit> [":" <digit> <digit> ["." <digit>*]
Times are recorded in the form HH:MM, HH:MM:SS or HH:MM:SS.S where HH is the hour in 24 hour format, MM is minutes and SS is seconds.
<string> = '"' <schar>* '"' <schar> = xchar | '"' '"'
Strings are output in quoted form. If a string contains a quotation character the character is repeated. This format is unambiguous since fields are by definition separated by whitespace.
No mechanism for incorporating control characters is defined.
<text> = <char>*
The text field is used only by directives.
<name> = <alpha> [ "." <alpha> * ]
DNS name.
<name> = <integer> [ "." <integer> * ] [ ":" <integer> ]
Numeric IP address and optional port specifier.
Robert Thau provided usefull advice and some code. John Mallery and Roger Hurwitz helped develop many of the ideas.