Full Online Ordered Training is a learning method for email classifiers where each incoming email is learned as soon as it arrives, thereby always keeping category descriptions up to date for the next classification. This directly models the way that some email classifiers are used in practice.
FOOT's error rates depend directly on the order in which emails are seen. A small change in ordering, as might happen due to networking delays, can have an impact on the number of misclassifications. Consequently, mailfoot does not give meaningful results, unless the sample emails are chosen carefully. However, as this method is commonly used by spam filters, it is still worth computing to foster comparisons. Other methods (see mailcross(1),mailtoe(1)) attempt to capture the behaviour of classification errors in other ways.
To improve and stabilize the error rate calculation,
performs the FOOT simulations several times on slightly reordered email streams, and
averages the results. The reorderings occur by multiplexing the emails from each
category mailbox in random order. Thus if there are three categories, the first email
classified is chosen randomly from the front of the sample email streams of each type.
The second email is also chosen randomly among the three types, from the front of the
streams after the first email was removed. Simulation stops when all sample streams are exhausted.
mailfoot uses the environment variable MAILFOOT_FILTER when executing, which permits the simulation of arbitrary filters, provided these satisfy the compatibility conditions stated in the ENVIRONMENT section below.
For convenience, mailfoot implements a testsuite framework with predefined wrappers for several open source classifiers. This permits the direct comparison of dbacl(1) with competing classifiers on the same set of email samples. See the USAGE section below.
During preparation, mailfoot builds a subdirectory named mailfoot.d in the current working directory. All needed calculations are performed inside this subdirectory.
The normal usage pattern is the following: first, you should separate your email collection into several categories (manually or otherwise). Each category should be associated with one or more folders, but each folder should not contain more than one category. Next, you should decide how many runs to use, say 10. The more runs you use, the better the predicted error rates. However, more runs take more time. Now you can type
% mailfoot add spam spam.mbox
% mailfoot add work work.mbox
% mailfoot add play play.mbox
You should aim for a similar number of emails in each category, as the random multiplexing will be unbalanced otherwise. The ordering of the email messages in each *.mbox file is important, and is preserved during each simulation. If you repeatedly add to the same category, the later mailboxes will be appended to the first, preserving the implied ordering.
You can now perform as many FOOT simulations as desired. The multiplexed emails are classified and learned one at a time, by executing the command given in the environment variable MAILFOOT_FILTER. If not set, a default value is used.
% mailfoot run
% mailfoot summarize
The testsuite commands are designed to simplify the above steps and allow comparison of a wide range of email classifiers, including but not limited to dbacl. Classifiers are supported through wrapper scripts, which are located in the /usr/share/dbacl/testsuite directory.
The first stage when using the testsuite is deciding which classifiers to compare. You can view a list of available wrappers by typing:
% mailfoot testsuite list
Note that the wrapper scripts are NOT the actual email classifiers, which must be installed separately by your system administrator or otherwise. Once this is done, you can select one or more wrappers for the simulation by typing, for example:
% mailfoot testsuite select dbaclA ifile
If some of the selected classifiers cannot be found on the system, they are not selected. Note also that some wrappers can have hard-coded category names, e.g. if the classifier only supports binary classification. Heed the warning messages.
It remains only to run the simulation. Beware, this can take a long time (several hours depending on the classifier).
% mailfoot testsuite run
% mailfoot testsuite summarize
Once you are all done, you can delete the working files, log files etc. by typing
% mailfoot clean
mailfoot testsuite takes care of learning and classifying your prepared email corpora for each selected classifier. Since classifiers have widely varying interfaces, this is only possible by wrapping those interfaces individually into a standard form which can be used by mailfoot testsuite.
Each wrapper script is a command line tool which accepts a single command followed by zero or more optional arguments, in the standard form:
wrapper command [argument]...
Each wrapper script also makes use of STDIN and STDOUT in a well defined way. If no behaviour is described, then no output or input should be used. The possible commands are described below:
Right after loading, mailfoot reads the hidden file .mailfootrc in the $HOME directory, if it exists, so this would be a good place to define custom values for environment variables.
The subdirectory mailfoot.d can grow quite large. It contains a full copy of the training corpora, as well as learning files for size times all the added categories, and various log files.
FOOT simulations for dbacl(1) are very, very slow (order n squared) and will take all night to perform. This is not easy to improve.
Because the ordering of emails within the added mailboxes matters, the estimated error rates are not well defined or even meaningful in an objective sense. However, if the sample emails represent an actual snapshot of a user's incoming email, then the error rates are somewhat meaningful. The simulations can then be interpreted as alternate realities where a given classifier would have intercepted the incoming mail.
The source code for the latest version of this program is available at the following locations:
Laird A. Breyer <email@example.com>
bayesol(1) dbacl(1), mailcross(1), mailinspect(1), mailtoe(1), regex(7)