Trigger (automated packet capture)

The following links show various outputs associated with the trigger.
mon.lite
event_today
event_previous_day


1 trigger overview:


the trigger mechanism allows the probe to dynamically store a set of "interesting" packets which may be filtered internally with various BPF expressions and then written out to a tcpdump style dump file.

Typically this is done when a back-end graphics "curve" exceeds some configuration-time threshold integer peak where both the threshold and current values are calculated in terms of back-end values, either in bits per second, or packets per second. A per trigger type dump directory must be specified (say /usr/dumps) and each trigger places a unique instance of a dump file filled with the specified count of packets. The instance file name is always of the form: trigger_tag..dmp. The timestamp records both the time the dump was made and allows the dump to have a unique filename. Only one trigger of each kind may be active at a time.

For example, one can have both the udp error weight trigger and the tworm triggers active, but it is not possible to have two tworm triggers. Each trigger has some sort of BPF expression capability associated with it, which is not user specifiable (except in the case of BPF filter-set triggers). For example the udp weight trigger dynamically determines the top IP host address, and stores count packets of the form: "host IP and (udp or icmp)", thus storing udp packets and any ICMP errors associated with the host IP that caused the trigger to activate. In general triggers are associated with some topn lists, and the BPF filter-set BPF expressions.

Current trigger types include:

topn_ip - topn IP (any IP protocol) trigger on total flow count.
tworm - total worm count.
topn_icmperror (udplist) - udp error weight for a specific IP host.
topn_scans - 1 trigger that is actually 4 triggers,
one per topn_scan list type (IP, L4, TCP, UDP). This is a per IP host trigger. (pkts)drops - triggers on too many drops from the pkt capture filter. bpf - any BPF expression used in a BPF filter-set may have a trigger.
Note: there is a maximum count of triggers. (MAXTRIGGERS see trigger.h).

It is important to note that triggers may be more useful in terms of the packets captured if the particular BPF expression (implicit as with tworm, or explicit as with a BPF filter-set expression) is more specific. For example, a BPF expression that captures all TCP packets may not show anything useful. On the other hand, the udp error weight trigger only captures UDP packets for a particular IP source. Thus the tcpdump in question may show the exact nature of an attack. This problem can be called the "signal to noise problem". A BPF expression on the order of "net X and tcp" may be more useful than just "tcp" or "net X" by themselves.

In ourmon.conf, triggers in general have the following syntax:

trigger_name threshold packet_count dump_dirname.

threshold - specify in bits/sec or pkts/sec a trigger threshold. When the named filter reaches that threshold, packet capture will begin.
packet_count - terminate when packets stored.
dump_dirname - directory in which to store the dump file.

Packet dumping is turned off when either the count is exceeded or the trigger threshold flow count is less than the current runtime value -- which ever comes first. Packet dumping after being triggered cannot recur between trigger on and trigger off times which are based on the threshold being crossed (either above it or below it). For example assume the packet capture count is 100000, and the value is 2000. If at mon.lite creation time, the value is 2000, a trigger_on event is started and 100000 packets (barring trigger_off occuring because threshold < value) will be stored. If the count is exhausted and the threshold is still > than v, no more packets will be stored in the file. Thus it is not possible for one trigger_on event to cause more than one output file. Only one trigger on event is possible for the topn_ip type. However this does not prevent other triggers from working (say with tworm). Also when this trigger is turned off, it may be retriggered later in which case data will be put into a separate file according to the timestamp.

Trigger_on and trigger_off events are put into mon.lite as elog messages and are recorded in the back-end daily event log. As a result it is possible to more or less determine when the trigger event occurred and to easily see the event from the web interface. The trigger dump filename is passed to the event log. The messages that show the trigger event in mon.lite will be like:

elog: tworm trigger on, current count: 20, threshold 10, dumpfile: /home/mrourmon/dumps/tworm.<03.05.2005|10:02:54>.dmp

The message means that the trigger for tworm is on because current packets number 20 is greater than threshold value 10. The trigger will capture and store the packets into file '/home/mrourmon/dumps/tworm.<03.05.2005|10:02:54>.dmp. The syntax and semantics of the packet file will be explained later.

elog: tworm trigger off, current count: 0, threshold 10.

The message means that the trigger for tworm is turned off because current packets number has dropped below the threshold value.

The log entries in the event file will be like:

Tue Mar 1 09:37:01 PST 2005: ourmon front-end event: tworm trigger on, current count: 45, threshold 40, dumpfile: /usr/dumps/tworm.<03.01.2005|09:36:53>.dmp Tue Mar 1 09:38:01 PST 2005: ourmon front-end event: tworm trigger OFF, current count is 20, threshold: 40

Note: at this time ourmon has no way to determine if the dynamically created file has been created successfully or if there is runtime storage room within the file system in question. If the file cannot be created an elog message will be sent. ourmon will not exit.


2. trigger_topn_ip threshold packet_count dump_dirname


The flow count is graphed in the back-end graphics flow.html page.

This trigger will dump packets when the flow count for topn_ip flows exceeds the threshold. Packets are stored in a file with the form:

topn_ip.< timestamp >.dmp.

This particular trigger uses an internal dynamic BPF expression to determine if packets should be stored. At trigger_on time the trigger code looks to see if tcp, udp, or icmp events are greater than the trigger, if so, the BPF expression is made more narrow, e.g., "tcp". If this is NOT the case, the expression "ip" which is the default for this expression is used. The BPF expression is only made more narrow if for example the TCP flow count per second is greater than the threshold but the UDP and ICMP flow counts are both less than the threshold. This logic also holds for UDP and ICMP flow counts. When in doubt the trigger expression is "ip".

Note: this trigger is aimed at the total flow count, not any individual single flow instance. The flow count is the second parameter in topn_ip, topn_tcp, etc. mon.lite outputs.


3. trigger_worm threshold packet_count dump_dirname


This trigger applies to the tworm total count (not us, not them, total). If the total value for tworm counts (us+them) exceeds the supplied threshold value, a packet dump will begin for count packets in the dump_dirname in a dynamically created filename. The filename has the form:
tworm.< timestamp >.dmp.

The semantics for this trigger are similar to the semantics for the preceeding trigger. However only TCP SYN packets are stored. Note it can be useful to use the saved back-end portreport.txt file to try and reproduce the port signature report at the trigger time. This can easily help narrow down the attacking instances in terms of IP source addresses or destination TCP port numbers. Put another way, it is useful to know the IP address of the attacker a priori or know a port signature in the case of multiple attackers. The IP address of the attackers can then be used to "search" the tcpdump file to see exactly what the attackers were up to. Also one may simply use ports of interest as indicated in the portreport.txt file. For example, assume that the portsignature.txt file shows that port 6666 was attacked, then one can do:

tcpdump -n -r "*dump*file" tcp and port 6666

to narrow down the packet dump to a more interesting set of packets.

bpf_trigger bpf_graph_name bpf_line_label threshold packet_count dump_dirname This trigger works with any expression found in its corresponding BPF filter-set. Ourmon allows user specified BPF expressions which result in graphics-engine RRDTOOL-based graphs. Every BPF filter is identified by 'bpf_graph_name' and 'bpf_line_label'. For any pre-declared BPF filter, a trigger can be associated with it by declaring the filter's 'bpf_graph_name' and 'bpf_line_label'. Please see the BPF section above for the syntax and semantics of bpf filter-sets. Note the threshold value is compared to the internal count in terms of packets or bytes -- whichever was used when the BPF filter-set was declared. The dynamically created filename has the following form:

bpf_< bpf_graph_name >_< bpf_line_label >.< timestamp >.dmp.

The semantics for this trigger are similar to that for the preceeding trigger. Only those packets which match BPF filter spec are stored.

It is useful here to refer to the back-end/graphics graphs in order to determine a threshold value. One may watch the graphs for a week or so and then determine a threshold. Note that in general the more specific the BPF expression, the better as this will tend to help you filter "signal" as opposed to "noise".


4. topn_scans_trigger threshold packet_count dump_dirname


The topn_scans_trigger works for four kinds of topn_scan list and stores packets for each kind of topn_scan list separately. Please see the topn_scan section for the semantics of topn_scan. The trigger (set) for topn_scan will dynamically configure its BPF fiilter expression, and capture filename. For the basic ip_scan list, the trigger is applied to the number of destination addresses related to the IP src address which is the top one in the ip_scans list. If the number of destinations this src address has exceeds the supplied threshold value, a packet dump will begin. The capture filename has the form:

ip_scans.< timestamp >.dmp

This trigger will capture and store any IP packets sent by the IP source in question.

The trigger for the topn_scan_port list is applied to the total count of L4 ports for the first host in the ip_portscan list. If the number of ports is greater than the threshold value, a packet dump will begin and the filename has the form:

ip_portscan.< timestamp >.dmp

The trigger for the topn_scan_port_tcp list is applied to the total count of TCP dst ports. If the count of TCP ports for the top IP src in the TCP scan list exceeds the threshold, packets will be stored in a filename with the form:

tcp_portscan.< timestamp >.dmp

The trigger for the topn_scan_port_udp list is applied to the total count of UDP dst ports. If the count of UDP ports for the top IP src address in the udp_portscan list exceeds the threshold value, packets will be stored in a filename with the form:

udp_portscan.< timestamp >.dmp

Note: there is really only one trigger capability here, although it can be dynamically narrowed a bit. Only one scan trigger can be triggered at one time. For example, it is not possible to have the tcp_portscan and the udp_portscan both running concurrently. They can however run in a serial fashion, one after the other.


5. drop_trigger threshold packet_count dump_dirname


This trigger is triggered when the current drop count (as seen in pkts.html) exceeds the trigger value. If packets are not dropped due to mundane reasons (like too much ourmon work or not enough CPU power), packets may be dropped due to DOS attacks. Thus it is possible that this trigger might actually capture a DOS attack. The capture filename has the form:

drop.< timestamp >.dump.

The semantics for this trigger are similiar to that for the preceeding trigger. The default BPF filter configured for drop_trigger makes it store only 'ip' packets.


6. udperror_trigger threshold packet_count dump_dirname


This trigger is based on the udperror_list (topn_icmperror). Too many packets might mean a UDP port scan attack. See the topn_icmperror section for more information on the udperror_list. If the udperror weight calculated for the first host in udperror_list is greater than the threshold, a packet dump will begin for the requested number of packets in a dynamically created filename in the supplied directory. The capture filename has the form:

topn_udp_err.< timestamp >.dump

This trigger will capture and store "udp" packets sent to and from the top host in udperror_list. The internal BPF expression used is: "ip_src and (udp or icmp)".

The log entries for trigger of topn_udperror in the event file will be like:

Fri Mar 4 01:17:33 PST 2005: udpweight threshold exceeded:10.0.0.1 14841330 7284 1194 1218 0 10: [1649,0][7674,80][22321,18][33068,0][40167,0][54156,0][55662,0][61131,0][64734,0][ 65021,0] Fri Mar 4 01:17:33 PST 2005: ourmon front-end event: topn_udp_err: bpf=(src 10.0.0.1 and (udp or icmp)) trigger on, current count: 14841330, threshold 10000000, dumpfile: /usr/dumps/topn_udp_err.<03.04.2005|01:17:05>.dmp