SENSOR.CONF(5)

NAME
sensor.conf - Sensor Configuration file for rwflowpack and flowcap

DESCRIPTION
As part of collecting flow data, the rwflowpack(8) and flowcap(8) daemons need to know what type of data they are collecting and how to collect it (e.g., listen on 10000/udp for NetFlow v5; listen on 4740/tcp for IPFIX). In addition, the rwflowpack daemon needs information on how to categorize the flow to specify its flowtype (for example, to label the flows collected at a border router as incoming or outgoing). The Sensor Configuration file, sensor.conf, contains this information, and this manual page describes the syntax of the file.

The sensor.conf file may have any name, and it does not have to reside in a particular location. The location of the the file is specified by the --sensor-configuration switch to rwflowpack and flowcap.

The Sensor Configuration File defines the following concepts:

probe
    A probe specifies a source for flow data. The source could be a port on which flowcap or rwflowpack collects NetFlow or IPFIX data from a flow generator such as a router or the YAF software. In rwflowpack, the source can be a file containing NetFlow v5 PDUs or a directory to periodically poll to find such files. When defining a probe, you must specify a unique name for the probe and the probe's type.

sensor
    A sensor represents a logical collection point for the purposes of analysis. Usually one sensor corresponds to one probe; however, a sensor could be comprised of multiple probes, or the flow data collected at a single probe could be divided into multiple sensors. The sensor contains the information that rwflowpack uses to categorize each flow depending on how the flow moves between networks at the collection point. Since the sensors and the categories (known as flowtypes or as class/type pairs) are also used for analysis, they are defined in the Site Configuration file, as described silk.conf(5). In the Sensor Configuration file, you specify how sensors map to probes, and give the rules required to categorize the data.

SYNTAX
When parsing the Sensor Configuration file, blank lines are ignored. At any location in a line, the character "#" indicates the beginning of a comment, which continues to the end of the line. These comments are ignored.

All other lines begin with optional leading whitespace, a command name, and one or more arguments to the command. Command names are a sequence of non-whitespace characters, not including the character "#". Arguments are textual atoms: any sequence of non-whitespace, non-"#" characters, including numerals and punctuation.

There are three contexts for commands: top-level, probe block, and sensor block. The probe block and sensor block contexts are used to describe individual features of probes and sensors.

The valid commands for each context are described below.

  Top-Level Commands
In addition to the commands to begin a probe or sensor block, the top-level context supports the following command:

include "path"
    The include command is used to include the contents of another file whose location is path. This may be used to separate large configurations into logical units.

  Probe Block
The first command below is used at top-level to begin a probe definition block, and the remaining commands are accepted within the probe context. Note that one and only one of listen-on-port, listen-on-unix-socket, read-from-file, poll-directory must be specified.

probe probe-name probe-type
    The probe command begins a new probe block and it continues to the end probe command. The arguments to the probe command are the name of the probe being defined and the probe type. The probe-name must be unique among all probes. It must begin with a letter, and it may not contain whitespace characters or the slash character ("/"). When a probe is associated with a single sensor, it is good practice to give the probe the same name as the sensor; however, when rwflowpack from SiLK-1.0 is reading flowcap files created by earlier versions of SiLK, the probe name must follow the rules specified in the rwflowpack(8) manual page. The probe-type must be one of the following:

    netflow-v5
        This probe processes NetFlow v5 protocol data units (PDU) that the daemon reads from a UDP port or from a file. NetFlow may be generated by a router or by software that reads packet capture (pcap(3)) data and generates NetFlow v5 records.

    netflow
        This is an alias for netflow-v5 for backwards compatibility. This alias is deprecated, and it may be removed in a future release.

    ipfix
        An IPFIX probe processes Internal Protocol Flow Information eXchange records that the daemon reads over the network from an IPFIX source such as YAF. To support IPFIX probes, SiLK must be built with support for the external library libfixbuf. Both YAF and libfixbuf are available from <http://tools.netsa.cert.org/>.

    netflow-v9
        This probe processes NetFlow v9 protocol data units (PDU) that the daemon reads from a UDP port from a router. To support NetFlow v9 probes, SiLK must be built with support for the external library libfixbuf, version 0.8 or later.

end probe
    The end probe command ends the definition of a probe. Following an end probe command, top-level commands are again accepted.

listen-on-port port
    This command specifies the network port where the probe should collect flow data. When listening to NetFlow from a Cisco router, this is the port that was specified to the Cisco ISO command

      ip flow-export [ip-address] [port]

protocol { tcp | udp }
    This command, required when listen-on-port is given, specifies whether the port is a "tcp" or "udp" port. IPFIX probes support both types; the only permitted value for all other probe types is "udp".

accept-from-host host-name
    This optional command specifies the IP or name of the host that is allowed to connect to the port where the probe is listening. When this command is not present, any host may connect. The command may only be specified when the listen-on-port command is also present. When listening for NetFlow, this attribute would be the IP address of the router as seen from the machine running rwflowpack or flowcap.

listen-as-host ip-address
    This optional command is used on a multi-homed machine to specify the address the probe should listen on (bind(2) to). Its value is a network IP address in dotted-decimal notation. If not present, the program will listen on all the machine's addresses. The command may only be specified when the listen-on-port command is also present. For listening to NetFlow, the value would be the ip-address that was specified to the Cisco ISO command

      ip flow-export [ip-address] [port]

listen-on-unix-socket path-to-unix-socket
    The value contains the path name to a UNIX domain socket where the flow generator writes its data. The parent directory of path-to-unix-socket must exist.

poll-directory directory-path
    When this command is given, rwflowpack will periodically poll the directory-path to look for files containing NetFlow v5 PDUs. The format of these files is described in the next command entry. flowcap will exit with an error if you attempt to use probes that contain this command since flowcap does not support reading data from files.

read-from-file dummy-value
    When this command is given, rwflowpack will read NetFlow v5 records from the the file specified by the --netflow-file command line switch. The value to the read-from-file command is completely ignored, and we recommend you use "/dev/null" as the value. flowcap will exit with an error if you attempt to use probes that contain this command since flowcap does not support reading data from files. The format of a NetFlow v5 file is that the file's length should be an integer multiple of 1464 bytes, where 1464 is the maximum length of the NetFlow v5 PDU. Each 1464 block should contain the 24-byte NetFlow v5 header and space for thirty 48-byte flow records, even if fewer NetFlow records are valid.

log-flags { none | all | { bad | missing ... } }
    This optional command gives specific logging instructions for the probe. Some probe types will keep track of the sequence numbers in the packets they receive; these probes will write messages to the log file about missing or bad packets. By default, all messages are written by the probes that keep track of this information, but sometimes it is useful to reduce the verbosity of this information or eliminate it entirely. The log-flags command accepts a list of the following values:

    none
        Log nothing.

    all Log all the conditions listed below; this is the default.

    bad Write messages about bad packets. An example of a bad packet is a NetFlow v5 PDU that reports no flow records.

    missing
        Examine the sequence numbers and write messages about missing packets.

priority value
    This optional command is used by flowcap when it is running as a server. The value may be an integer between 1 and 100, inclusive, where a value of 1-50 is considered low priority, and 51-100 is high. flowcap accumulates flow data from probes into files, and these files are closed after a certain amount of time. Closed high-priority files are always sent to a client before closed low-priority files.

  Sensor Block
The information from the sensor block is used by rwflowpack to determine how to categorize a flow; that is, in which file the flow record is stored.

When the Sensor Configuration file is used with flowcap, no sensors need to be defined. In fact, flowcap completely ignores all text inside the Sensor Block.

The logic to categorize a flow for the twoway site is described below in the "PACKING LOGIC" section. The packing logic is normally loaded into rwflowpack as a run-time plug-in. The packing logic for your version of rwflowpack may be different.

The packing logic plug-in specifies a list of network names, and you will refer to these networks when you configure the sensor block. Most plug-ins provide the "external", "internal", and "null" names, where internal refers to network being monitored, null are flows that were blocked by the router's access control list, and external is everything else.

The first command below is used at top-level to begin a sensor configuration block, and the remaining commands are accepted within the sensor block.

sensor sensor-name
    The sensor command begins a new sensor configuration block. It takes as an argument the name of the sensor being configured, and that sensor must be defined in the Site Configuration file (see silk.conf(5)). A sensor block is closed with the end sensor command. You may have multiple sensor blocks that have the same sensor-name, but the list of probes within each sensor block must be unique.

end sensor
    The end sensor command ends the configuration of a sensor. Following an end sensor command, top-level commands are again accepted.

probe-type-probes probe-name [probe-name ...]
    This command associates the listed probe names of the given probe type with the sensor. The probes do not have to be defined before they are used. (Note this also means that a mistyped probe-name will not be detected.) For example, netflow-v5-probes S1 says that S1 is a netflow-v5 probe; whenever flow data arrives on the S1 probe, the sensor associated with the probe notices that data is available and processes it.

source-network network-name
    This command causes the sensor to assume that all flows originated from the network named network-name. For example, if a sensor is associated a probe that only monitors incoming traffic, you could use "source-network external" to specify that all traffic originated from the external network.

destination-network network-name
    This command causes the sensor to assume that all flows were sent to the network named network-name.

network-name-ipblocks {cidr-block [cidr-block...] | remainder}
    The value of this command is a list of CIDR blocks specifying the IP-space of the network named network-name. If the value is the keyword remainder, the IP-space is taken as all IPs not assigned to other networks. The remainder keyword can only appear one time within a sensor block.

network-name-interfaces {interface [interface...] | remainder}
    This command is similar to the above. The value is a list of positive integers from 0 to 65535 that represent the SNMP interface index(es) that connect to routers or machines on the network named network-name. If the value is the keyword remainder, the interface list is computed by finding all interface values not assigned to other networks. The remainder keyword can only appear one time within a sensor block. The command is only valid on sensors that are associated with netflow-v5 probes.

isp-ip ip-address [ip-address ...]
    This optional command may be used for a sensor that processes data from a netflow-v5 probe. The value to the command is a list of IP addresses in dotted-decimal notation, where the IPs are the addresses of the NICs on the router. For traffic that doesn't leave the router (and thus was sent to the router's null-interface), some packing-logic plug-ins use these IPs to distinguish legitimate traffic for the router (e.g., routing protocol traffic, whose destination address would be in this list) from traffic that violated the router's access control list (ACL).

PACKING LOGIC
The packing logic plug-in that rwflowpack loads uses the values in the Sensor Block to determine how to categorize a flow and where on disk to store it. This section describes the how a flow's category (or flowtype) is set when the logic in the packlogic-twoway.so file is used.

The packlogic-twoway.so plug-in specifies three network names: the internal network refers to network being monitored, the null network is the destination for flows that were blocked by the router's access control list, and the external network is everything else.

You may categorize a flow by specifying the source-network and destination-network. This can be used when the physical network device only sees one sort of traffic; for example, if NetFlow is only enabled on a router's external-facing interfaces. Most of the time, more complex categorization is required.

You may categorize any flow record, regardless of the type of probe that collected it, based on the IP addresses the flow contains and the IP space of the monitored network. Flows collected by a NetFlow probe may instead be categorized using the SNMP interface information in the flow.

The packlogic-twoway.so packing logic requires you specify for each sensor either the IP space outside the monitored network (external-ipblock) or, for sensors that use NetFlow probes, the SNMP interfaces that face outside the network (external-interfaces). In addition to specifying the external side, you may specify the values on the internal side (internal-ipblock or internal-interfaces). When the internal value is not specified, it is taken to be all values not used on the external side. It is an error to specify both the IP space and SNMP interfaces on a sensor.

When a router generates a flow record for packets that the router did not route (that never left the router), the outgoing SNMP interface is set to the "null" interface. A packet may be not-routed because the packet is meant for the router itself (e.g., a BGP message) or because it matches an ACL violation. On a CISCO router, the null interface has the destination ID of 0. You may specify the null-interfaces to use this destination in your packing logic. (For consistency, there is a null-ipblock that can be set.)

Note: As of SiLK-1.0, the null-interface must be explicitly set. It is no longer set by default.

Given a flow travelling between the external, internal, and null networks, the following rules specify the flowtype that is assigned to the flow. This description uses IP-based splitting; for SNMP-based splitting, the logic is identical, except the SNMP interfaces are checked. These rules also apply when the source- and destination-networks have been set explicitly.

ext2ext
    Contains flows where both the source IP and destination IP are listed in the external-ipblock.

innull
    Contains flows where the source IP is listed in the external-ipblock and the destination IP is listed in the null-ipblock.

in,inicmp,inweb
    Contains flows where the source IP is listed in the external-ipblock and the destination IP is listed in the internal-ipblock. A flow's flowtype is inicmp if the protocol is ICMP (i.e., 1). The flow's flowtype is inweb if the protocol is TCP (6) and either the source port or destination port is one of 80, 443, 8080. Otherwise the flow's flowtype is in.

int2int
    Contains flows where both the source IP and destination IP are listed in the internal-ipblock.

outnull
    Contains flows where the source IP is listed in the internal-ipblock and the destination IP is listed in the null-ipblock.

out,outicmp,outweb
    Contains flows where the source IP is listed in the internal-ipblock and the destination IP is listed in the external-ipblock.

other
    Contains flows where either

    •   the source IP is not listed in the external-ipblock or internal-ipblock (note that the source could be listed in the null-ipblock)

    •   the destination IP is not listed in the external-ipblock, internal-ipblock, or null-ipblock

EXAMPLES
All these examples assume you are using the packlogic-twoway.so packing logic plug-in to rwflowpack.

  NetFlow Categorized by SNMP Interface
The following two blocks define a probe that listens on 9900/udp for NetFlow v5 from a router. The probe only accepts traffic originating from 172.16.22.22. The associated sensor uses the SNMP interfaces to categorize the flows, where traffic from 1 to 8 is "in", 8 to 1 is "out", 1 to 0 is "innull", 8 to 8 is "ext2ext", etc.

  probe S1 netflow-v5
      listen-on-port 9900
      protocol udp
      accept-from-host 172.16.22.22
  end probe

  sensor S1
      netflow-v5-probes S1
      external-interface 1
      internal-interface 8
      null-interface 0
  end sensor

  NetFlow Categorized by IP Address
The probe is the same as above, but the flows are categorized based on the IP addresses. The internal network is defined as 128.2.0.0/16, and all other IPs are defined as external.

  probe S2 netflow-v5
      listen-on-port 9901
      protocol udp
      accept-from-host 172.16.22.22
  end probe

  sensor S2
      netflow-v5-probes S2
      internal-ipblock 128.2.0.0/16
      external-ipblock remainder
  end sensor

  IPFIX Categorized by IP Address
This example uses the same sensor configuration as above, but it uses an IPFIX probe to collect the flows on port 9902/tcp. Also, we instruct the probe to bind to address 192.168.1.92. Since there is no SNMP information for IPFIX flows, one must use CIDR blocks to categorize the flows.

  probe S3 ipfix
      listen-on-port 9902
      protocol tcp
      listen-as-host 192.168.1.92
  end probe

  sensor S3
      ipfix-probes S3
      internal-ipblock 128.2.0.0/16
      external-ipblock remainder
  end sensor

  Multiple IPFIX
Consider the case of using YAF on a monitor at the border of a network where all traffic entering the network enters the monitor on one network interface card (NIC) and all traffic leaving the network enters the monitor on a different NIC. Since YAF does not support multiple interfaces yet, you must run two YAF processes, one for each NIC. The sensor configuration for this monitor would list two probes, each listening on a different port, and two sensor blocks both packing to the same sensor.

  probe S4-in ipfix
      listen-on-port 9902
      protocol tcp
  end probe

  probe S4-out ipfix
      listen-on-port 9907
      protocol tcp
  end probe

  sensor S4
      ipfix-probes S4-in
      source-network external
      destination-network internal
  end sensor

  sensor S4
      ipfix-probes S4-out
      source-network internal
      destination-network external
  end sensor

  NetFlow From a File Categorized by SNMP Interfaces
Instead of listening on a UDP port for NetFlow traffic, you can configure the probe to process a file containing NetFlow v5 PDUs. This example assumes you are running rwflowpack with the switches --input-mode=file --netflow-file=FILENAME. The --netflow-file switch overrides the read-from-file command on the probe.

  probe S5 netflow-v5
      log-flags bad                     # ignore missing pkts
      read-from-file /dev/null          # use --netflow-file=<file>
  end probe

  sensor S5
      netflow-v5-probes S5
      external-interface 2
      internal-interface 9
      null-interface 0
  end sensor

SEE ALSO
SiLK Installation Handbook, silk.conf(5), rwflowpack(8), flowcap(8), silk(7),

NOTES
You may need to use specific probe names when rwflowpack is processing files created by a pre-SiLK-1.0 version of flowcap. See the NOTES section of the rwflowpack man page for details.

The syntax of sensor.conf differs greatly between SiLK-0.11.x and SiLK-1.0. As of SiLK-1.0, defining temporary sensors in sensor.conf is no longer supported. Instead, the sensors should be defined in the silk.conf file.

SiLK-1.0 provides the update-sensor-conf script to upgrade older versions of sensor.conf to the new syntax. When making the change by hand, this old sensor-probe block:

 sensor-probe S01
      probe-name netflow
      probe-type netflow
      priority 8
      log-flags none all bad missing
      listen-on-port 9999
      protocol udp
      listen-as-host 192.168.1.1
      listen-on-unix-domain-socket /tmp/sock
      read-from-file /var/tmp/flow-file
      accept-from-host 172.16.22.22
      null-index 0
      isp-ip 10.10.10.10,10.10.10.11
      external-index 1,2,3,4
      internal-index 8,9
      null-ipblock 10.0.0.0/8,172.16.0.0/12,192.168.0.0/16
      internal-ipblock 128.2.0.0/16
      external-ipblock remainder

becomes:

 probe S01 netflow-v5
      priority 8
      log-flags none all bad missing
      listen-on-port 9999
      protocol udp
      listen-as-host 192.168.1.1
      listen-on-unix-domain-socket /tmp/sock
      read-from-file /var/tmp/flow-file
      accept-from-host 172.16.22.22
 end probe

 sensor S01
      netflow-v5-probes S01
      isp-ip 10.10.10.10,10.10.10.11
      null-interface 0
      external-interface 1,2,3,4
      internal-interface 8,9
      null-ipblock 10.0.0.0/8,172.16.0.0/12,192.168.0.0/16
      internal-ipblock 128.2.0.0/16
      external-ipblock remainder
 end sensor

Note the following:

•   Move the probe-type to the same line as the name of the probe.

•   Change the probe-type from netflow to netflow-v5.

•   Add a probe-type-probes statement to the sensor.

•   Change any occurrence of -index or -indexes to -interface.

•   For netflow-v5 probes, rwflowpack no longer sets the null-interface to 0 by default. You must specify it explicitly.

•   The protocol statement is now required for all probes whenever listen-on-port is specified.

•   The probe block gets all the statements except for those that define ipblocks, interfaces, or isp-ips.

•   Each block must have an end statement.