Taming p0f by chunk processing STDIN

P0f is a useful, but slightly outdated, tool for passive operating system fingerprinting. Unfortunately it generates fingerprint information for every identified packet matching the BPF filter provided by the user. There is no built in option to aggregate the data so that it is framed per host within a given time interval or set of contiguous records. This means that when p0f is attached to the SPAN port or tap of a high volume pipe the log files that it generates grow very quickly as they are full of repeated information.

The following little script wraps the p0f process and removes the redundant information from each 1000 record chunk of output:

#! /bin/sh

exec 3>&-
exec 2>&-
exec 1>&-
cd /

nohup p0f -i eth2 -u p0f -N -U -q -p -t -l 'src net 143.210.0.0/16' | \
sed -n -e 's/^<\([A-Za-z0-9: ]*\)> \([0-9.]\{7,15\}\):[0-9]\{1,5\} - \
\(.*\)/\2 \3/p' | gawk 'ORS=NR%1000?"\n":"\000"' | xargs -0 -i bash -c \
'date +"*** %c ***"; echo "$0" | sort | uniq' {} >> /srv/p0f/os.log &

p0f aside, the interesting part boils down to this useful Unix shell programming paradigm:

$INPUT_CMD | gawk "ORS=NR%$BLOCK_LINES?'\n':'\000'" | xargs -0 -i $PROCESS_CMD {}

It splits the streamed output of $INPUT_CMD down into chunks of $BLOCK_LINES lines which are immediately independently processed by $PROCESS_CMD. It chunks the data by replacing the ordinary line separator on every $BLOCK_LINES line into an ASCII 0 character which xargs -0 uses as the argument separator.

Apache accesslog to syslog

Apache allows its error logs to be written to the local syslog, however it does not natively support the directing of access logs to the syslog. How frustrating!

It does however allow access logs to be written to a pipe and I have seen a number of home-brew scripts that essentially redirect the Apache access log data from STDIN to syslog.

I've yet to see anything quite as simple as the following directive that I cooked up today:

CustomLog "|exec /usr/bin/logger -t apache -i -p local6.notice" combined

It pipes the access log data to the BSD logger(1) utility that is installed by default on almost any Unix system. No need for any more of those STDIN wrapper scripts!