tenshi is a log monitoring program, designed to watch one or more log files for lines matching user defined regular expressions and report on the matches. The regular expressions are assigned to queues which have an alert interval and a list of mail recipients.
Queues can be set to send a notification as soon as there is a log line assigned to it, or to send periodic reports.
Additionally, uninteresting fields in the log lines (such as PID numbers) can be masked with the standard regular expression grouping operators ( ). This allows cleaner and more readable reports. All reports are separated by hostname and all messages are condensed when possible.
The program reads a configuration file (tenshi.conf) and then forks a daemon for monitoring the specified log files.
All directives are shown with the standard default value where applicable, if
omitted the default value will be used.
EXTERNAL CONFIGURATION FILES
All configuration directives can be optionally split into different
configuration files and then read with the two following statements.
STATIC OPTIONS
These options will be read the first time tenshi reads its config file. They
cannot be changed by re-reading the config file. If you change one of these
options and HUP tenshi it will die. You have been warned.
DYNAMIC OPTIONS
These options are set each time the config file is read. tenshi reads its
config file once on start-up and whenever it receives a HUP.
QUEUES OPTIONS
All messages are assigned to queues. Every queue is processed periodically
according to its notification interval. There are four default builtin queues,
trash
to which unwanted messages can be assigned (think /dev/null),
repeat
which is used for smart repeat messages handling,
group
and
group_host
, see
REGEXP DEFINITIONS
for details. There's also a special
noprefix
queue, read further for details about it.
All queues are automatically flushed before shutdown when a SIGTERM is received. Please see section SIGNALS for additional information.
The syntax is the following:
The special noprefix queue can be used and defined like any other queue with the difference that it will get all messages that don't match any configured prefix.
Examples:
set queue report tenshi@localhost sysadmin@localhost [0 9-17 * * *]
set queue report tenshi@localhost sysadmin@localhost [30 18 * * *]
set queue report tenshi@localhost sysadmin@localhost [*/10 * * * *]
set queue critical tenshi@localhost sysadmin@localhost,noc@localhost [now] CRITICAL WARNING -
set queue pager tenshi@localhost pager:sysadmin_pager@localhost,pager:noc_pager@localhost [now] ALERT
REGEXP DEFINITIONS
All valid syslog messages are matched against standard perl regexps, all
regexps are defined with the following syntax:
The regexps are evaluated in order so a matched message is not checked against the subsequent regexps. Keep this in mind when assembling the configuration file. It's advisable to catch all messages by placing an all matching regexp at the end of the configuration file. It's also good for performance having trash rules not logically connected with other matching rules at the beginning of the section. Multiple queues can be defined with a comma separated list, builtin queues cannot be used when using this syntax.
If an escalation number is provided for a queue, the matched message will only
be placed into the queue when <escalation_number> messages have matched the
regexp. The queue will receive the message that matched the regexp at the time
of escalation, with a count equal to the escalation number. The count of
messages matching the regexp will be reset when the left most queue mentioned
in the queue list is mailed. The left most queue cannot have an escalation
number unless it is the only queue listed. When the number of messages that
match the regexp reaches the greatest escalation number mentioned, escalation
will begin again into the escalation queues, modulus the greatest escalation
number. For example, using the queues `a,b:10,c:50', when 10 messages match the
regexp, a message will go into b, when 50 match, one will go into c. At 60,
another will go into b, and at 100, another into c, 110 to b, 150 to c, and so
on. Escalation numbers must be positive integers greater than zero and must be
listed in increasing order from left to right. All queues without escalation
numbers must be listed more left than the queues with escalation numbers.
The standard grouping operators
( )
can be used for string masking, literal "(" and ")" can be protected with the
standard quotation operator "\". There's a lot of documentation about regular
expressions, a good start could be perl
perlre
and
perlretut
manual pages.
You can also use the (?: ) operators to use groups without masking. This allows
you to match, for example, output from several programs in a similar format.
There is an example of this below (the sudo/su line).
The builtin queue
repeat
can be used for special handling of "last message repeated x times" style log
lines. When the assigned regexps are matched the line count for the last line
received from the same host is incremented by the first grouped string. Keep in
mind that it is possible for syslog lines to be received from remote hosts out
of order. If this happens you should not use this feature because tenshi will
mis-report line counts.
The builtin queue
group
can be used to group sets of regex together to speed up line matching. If a
line fails to match a regex assigned to the group queue then tenshi will skip
all the regex up until the next
group_end
statement. Nested groups are allowed. An example of this is included below.
The builtin
group_host
queue can be used for selective hostname matching. Like the
group
queue it is also terminated with the
group_end
statement. All regex definitions within that group will only apply if the hostname
associated to the log entries matches the regex passed to the
group_host
definition.
The regexs below assume
hidepid
is turned on. If you have it turned off then you will need to add in \[(.+)\]
to the regex following the progam name to get them to work.
For example:
mail ^sendmail: (.+): to=(.+),(.+)delay=(.+)
becomes:
mail ^sendmail\[(.+)\]: (.+): to=(.+),(.+)delay=(.+)
Examples:
trash ^xinetd
repeat ^(?:last message repeated|above message repeats) (\d+) time
group ^sendmail:
mail ^sendmail: (.+): to=(.+),(.+)delay=(.+)
mail ^sendmail: (.+): to=(.+),(.+)relay=(.+),(.+)stat=Sent
group_end
group_host mailserver1
mail1 ^sendmail
mail1 ^sendmail:.+
critical,mail1 ^sendmail:.+SYSERR.+
group_end
mail ^ipop3d: Login user=(.+)
critical,report ^sshd: Illegal user
general,urgent:200,critical:1000 ^sshd: Illegal user
root ^sshd\(pam_unix\): session opened for user root by root\(uid=0\)
report ^sshd: Accepted rsa for (.+) from (.+) port (.+)
trash ^sshd
critical ^(?:sudo|su):
critical,pager ^Oops
misc .*
WARNING: If you change a STATIC OPTION in the config file and send tenshi a HUP it will die. You will need to restart tenshi for changes to STATIC OPTIONs to take effect.
tenshi needs a working 'tail' implementation when not using FIFO mode.
It also requires Net::SMTP module for mailing reports, which should be included in your perl installation, and IO::BufferedSelect. If you miss any of them you can grab them at CPAN (http://www.cpan.org) or using the CPAN shell (`perl -e shell -MCPAN`).
Please report any bugs you find at <tenshi@inversepath.com>
Any volunteers ? ;)
Permission to use, copy, modify, and distribute this software for any purpose with or without fee is hereby granted, provided that the above copyright notice and this permission notice appear in all copies.
THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
The tenshi project page is http://dev.inversepath.com/trac/tenshi
tenshi was formerly known as wasabi but the name was changed as we were informed that wasabi is a registered a trademark relating to another piece of software.
Two mailing lists are also available:
<tenshi-user@lists.inversepath.com> for general discussion, subscribe by sending a message to <tenshi-user+subscribe@lists.inversepath.com>
<tenshi-announce@lists.inversepath.com> for announcements, subscribe by sending a message to <tenshi-announce+subscribe@lists.inversepath.com>
It should be noted that tenshi was initially a perl rewrite of oak (http://www.ktools.org).
Friedl, Jeffrey E. F. Mastering Regular Expressions, 2nd Edition. O'Reilly
Copyright 2004-2009 Andrea Barisani <andrea@inversepath.com>
Rob Holland <rob@inversepath.com>