SILK(7)

NAME
SiLK - the System for Internet-Level Knowledge

DESCRIPTION
SiLK is a collection of traffic analysis tools developed by the CERT Network Situational Awareness Team (CERT NetSA) to facilitate security analysis of large networks. The SiLK tool suite supports the efficient collection, storage, and analysis of network flow data, enabling network security analysts to rapidly query large historical traffic data sets. SiLK is ideally suited for analyzing traffic on the backbone or border of a large, distributed enterprise or mid-sized ISP.

There are two categories of applications that comprise a SiLK installation:

  Analysis tools
Analysis tools read binary files containing SiLK flow records and partition, sort, and count these records. The most important analysis tool is rwfilter(1), an application for querying the central data repository for SiLK Flow records that satisfy a set of filtering options.

The tools are intended to be composed in various ways to perform an analysis task. A typical analysis uses UNIX pipes and intermediate data files to share data between invocations of tools.

The tools, configuration files, and plug-in modules that make up the analysis tools are listed below, roughly grouped by functionality.

Filtering, sorting, and display

rwfilter(1) partitions SiLK Flow records into one or more 'pass' and/or 'fail' output streams. It is the primary tool for pulling flows from the data store.

silk.conf(5) is the configuration file naming the Classes, Types, and Sensors available at your installation.

rwsort(1) reads SiLK Flow records, sorts them by the specified key, and writes the records to the named output path or to the standard output.

rwcut(1) prints SiLK Flow records in a pipe-delimited ("|"), columnar, human-readable format.

SiLK Python Extension

pysilk(3). PySiLK, the SiLK Python extension, allows one to read, manipulate, and write SiLK Flow records and IPsets from within Python. PySiLK may be used in a stand-alone Python script or as a plug-in. This document describes the objects, methods, and functions that PySiLK provides. The next entry describes using PySiLK from within a plug-in.

silkpython(3). The SiLK Python plug-in provides a way to use the PySiLK to define arbitrary partitioning rules for rwfilter(1) and arbitary fields for rwcut(1), rwuniq(1), and rwsort(1).

Counting and statistics

rwuniq(1) bins SiLK Flow records by a user-defined key and prints the total byte, packet, and flow counts for each bin. rwuniq can also print distinct source IP and destination IP counts.

rwcount(1) summarizes SiLK Flow records across time, producing textual output with counts of bytes, packets, and flow records for each time bin.

rwstats(1) summarizes SiLK Flow records by one of a limited number of key/value pairs and displays the results as a Top-N or Bottom-N list.

rwtotal(1) bins SiLK Flow records by a specified key and prints the total byte, packet, and flow counts for each bin.

rwaddrcount(1) summarizes SiLK flow records by source or destination IP and prints the byte, packet, and flow counts for each IP.

IPset, Bag, and Prefix Map manipulation

rwset(1) reads SiLK Flow records and generates binary IPset file(s).

rwsetbuild(1) reads IP addresses in dotted-quad or CIDR notation from input-file and writes a binary IPset file.

rwsetcat(1) prints the contents and/or additional information about IPset files.

rwsettool(1) performs union, intersection, difference, and sampling functions on the input IPset files, generating a new IPset file.

rwsetmember(1) determines if the specified IP address is contained in an IPset.

rwsetintersect(1) generates a new IPset file by performing intersection and difference options on the IPset files listed on the command line. This command is deprecated; use rwsettool instead.

rwsetunion(1) merges the input binary IPset files into the output IPset; an IP in any input file will be in the output file. This command is deprecated; use rwsettool instead.

rwbag(1) reads SiLK Flow records and builds binary Bag(s) containing key-count pairs.

rwbagbuild(1) creates a binary Bag file from binary IPset file or a textual input file.

rwbagcat(1) prints binary Bag files as text.

rwbagtool(1) performs operations on Bag files.

rwpmapbuild(1) reads textual input and creates a binary prefix map file for use with the Address Type (addrtype(3)) and Prefix Map (pmapfilter(3)) Plug-Ins.

rwpmapcat(1) prints information about a prefix map (pmap) file. By default, print each IP range in the pmap and its label.

rwipaimport(1) imports a SiLK IPset, Bag, or Prefix Map file into the IP Address Association (IPA <http://tools.netsa.cert.org/ipa/>) library.

rwipaexport(1) exports a set of IP addresses from the IP Address Association (IPA) library to a SiLK IPset, Bag, or Prefix Map.

Run time plug-ins

addrtype(3). The Address Type plug-in provides a way to map an IP address to an integer denoting the IP as internal, external, or non-routable.

ccfilter(3). The Country Code plug-in provides a mapping from an IP address to two-letter, lowercase abbreviation of the country that owns the IP address.

flowrate(3). The Flowrate plug-in, which must be loaded explicitly, adds switches and fields to compute packets/second, bytes/second, bytes/packet, payload-bytes, and payload-bytes/second.

pmapfilter(3). The Prefix map plug-in provides a way to map field values to string labels based on a user-defined map file.

Record grouping and masking

rwgroup(1) groups flows together by specified id-fields and delta-field and marks the records with a group ID that is stored in the next hop IP field. rwgroup requires that its input is sorted.

rwmatch(1) groups records as queries and responses. rwmatch requires that its input is sorted.

rwnetmask(1) reads SiLK Flow records from the standard input, masks the lower bits of the specified IP address(es), and writes the resulting records to stdout.

Packet and external flow-format processing

rwpcut(1) outputs a pcap file as ASCII, in a form similar to rwcut.

rwpdedupe(1) detects and eliminates duplicate records.

rwpmatch(1) filters a pcap file by outputting only packets whose five-tuple and timestamp match corresponding flows in a SiLK Flow file.

rwptoflow(1) reads a pcap file and generates a SiLK Flow record for every packet.

rwp2yaf2silk(1) converts a pcap file to a single file of SiLK Flow records; rwp2yaf2silk assumes that the yaf(1) (<http://tools.netsa.cert.org/yaf/>) and rwipfix2silk(1) commands are available on your system.

rwipfix2silk(1) converts a stream of IPFIX (Internet Protocol Flow Information eXport) records to the SiLK Flow record format.

rwsilk2ipfix(1) converts a stream of SiLK Flow records to the IPFIX (Internet Protocol Flow Information eXport) format.

Scan detection

rwscan(1) attempts to detect scanning activity from SiLK Flow records. rwscan can produce files that can be loaded into a database and queried with rwscanquery.

rwscanquery(1) queries the scan database which has been populated from database load files generated by rwscan.

Utilities

mapsid(1) maps between sensor names and sensor IDs using the values specified in the silk.conf(5) file.

num2dot(1) reads pipe-delimited ("|") text from the standard input, converts integer values in the specified column(s) (default first column) to dotted-decimal IP address, and prints the result to the standard output.

rwappend(1) appends the SiLK Flow records contained in the second through final file name arguments to the records contained in the first file name argument.

rwcat(1) reads SiLK Flow records from the files named on the command line, or from the standard input when no files are provided, and writes the SiLK records to the specified output file or to the standard output if it is not connected to a terminal.

rwdedupe(1) reads SiLK Flow records from files named on the command line or from the standard input and writes the records to the named output path or to the standard output, removing any duplicate flow records. Note that the order of records is not maintained as the records are sorted.

rwfglob(1) prints to the standard output the list of files that rwfilter would normally process for a given set of file selection switches.

rwfileinfo(1) prints information (type, version, etc.) about a SiLK Flow, IPset, or Bag file.

rwgeoip2ccmap(1) creates the country code prefix map required by the ccfilter(3) plug-in from the MaxMind GeoIP database.

rwidsquery(1) invokes rwfilter to find flow records matching Snort signatures.

rwip2cc(1) maps a (textual) list of IP addresses to country.

rwrandomizeip(1) generates a new flow file by substituting a pseudo-random IP address for the source and destination IP addresses in given input file.

rwresolve(1) reads pipe-delimited ("|") text from the standard input, attempts to resolve the IP addresses in the specified column(s) to host names, and prints result to the standard output.

rwsplit(1) generates a set of sub-files from the input. The sub-files can be limited by flow, byte, packet, or unique IP count, and the sub-file may contain all the flows or only a sample of them.

rwswapbytes(1) generates a new flow file by changing the byte order of values in a given input file.

rwtuc(1) generates SiLK flow records from textual input; the input should be in a form similar to what rwcut(1) generates.

  Packing tools
The packing tools run as daemons to collect flow records from a flow generator (e.g., a router producing NetFlow), convert the records to the SiLK flow format, categorize the flows as incoming or outgoing, and write the records to their final destination in binary flat files for use by the analysis tools.

The tools that make up the SiLK Packing System are:

flowcap(8) listens to network devices which produce flow data (flow generators) and stores the data in incremental files in a flat directory for transfer to a remote machine for processing by rwflowpack.

rwflowpack(8) reads flow data either directly from a flow generator or from a file generated by flowcap, converts the data to the SiLK flow format, categorizes the flow records, and writes records into either hourly flat-files organized in a time-based directory structure or into files for transfer to a remote machine for processing by rwflowappend.

rwflowappend(8) watches a directory for files containing small numbers of SiLK flow records and append those records to hourly files organized in a time-based directory tree.

rwsender(8) watches an incoming directory for files, moves the files into a processing directory, and transfers the files to one or more rwreceiver processes. Either rwsender or rwreceiver may act as the server with the other acting as the client.

rwreceiver(8) accepts files transferred from one or more rwsender processes and stores them in a destination directory.

sensor.conf(5) is a configuration file for sensors and probes used by rwflowpack and flowcap.

ENVIRONMENT
The following environment variables affect the tools in the SiLK tool suite.

SILK_CONFIG_FILE
    This environment variable contains the location of the site configuration file, silk.conf(5). This variable has precedence over all methods of finding the site file except for the --site-config-file switch on the applications.

SILK_DATA_ROOTDIR
    This variable gives the root of directory tree where the data store of SiLK Flow files is maintained, overriding the location that is compiled into the tools. The rwfilter(1) and rwfglob(1) tools use this value when selecting which flow files to process unless the user passes the --data-rootdir switch to the application. In addition, the SiLK tools will search for the site configuration file, silk.conf, in this directory.

SILK_PATH
    This environment variable gives the root of the directory tree where the tools are installed. As part of its search for the SiLK site configuration file, the SiLK applications check for a file named silk.conf in the directories $SILK_PATH/share/silk and $SILK_PATH/share. These directories are also searched when any other configuration file is required (e.g., the country code map required by ccfilter(3)). In addition, the applications that support plug-ins will search for them in $SILK_PATH/lib/silk, $SILK_PATH/share/lib and $SILK_PATH/lib.

SILK_IPV6_POLICY
    For tools that implement the --ipv6-policy switch, this environment variable is used as the value for that switch when it is not provided.

SILK_PAGER
    When set to a non-empty string, most of the applications that produce textual output (e.g., rwcut(1)) will automatically invoke this program to display their output a screen at a time. If set to an empty string, no paging of the output is performed. The --pager switch on the application overrides this value.

PAGER
    The applications that support paging their output will use the value in this environment variable when SILK_PAGER is not set.

SILK_TMPDIR
    This variable is used by tools that write temporary files (e.g., rwsort(1)) as the directory in which to store those files. SILK_TMPDIR overrides the value of TMPDIR.

TMPDIR
    When set and SILK_TMPDIR is not set, temporary files are created in this directory.

PYTHONPATH
    The Python modules and library files required to use PySiLK from rwfilter(1), rwcut(1), rwsort(1), and rwuniq(1) as well as from Python itself are installed under SiLK's installation tree by default. It may be necessary to set or modify the PYTHONPATH environment variable so Python can find these files. For information on using the PySiLK module, see silkpython(3) as well as the SiLK in Python handbook.

SILK_PYTHON_TRACEBACK
    When set, Python plug-ins will output traceback information on Python errors to stderr.

SILK_RWFILTER_THREADS
    The number of threads rwfilter uses while reading input files or files selected from the data store.

RWRECEIVER_TLS_PASSWORD, RWSENDER_TLS_PASSWORD
    Used by rwreceiver(8) and rwsender(8), this variable specifies the password to use to decrypt the PKCS#12 file specified in the --tls-pkcs12 switch.

SILK_LOGSTATS_RWFILTER
    When set to a non-empty value, rwfilter(1) will treat the value as the path to a program to execute with information about this rwfilter invocation. Its purpose is to provide the SiLK administrator with information on how the SiLK tool set is being used.

SILK_LOGSTATS
    This environment variable is currently an alias for SILK_LOGSTATS_RWFILTER. The ability to log invocations may be extended to other SiLK tools in future releases.

SILK_LOGSTATS_DEBUG
    If the environment variable is set to a non-empty value, rwfilter will print messages to the standard error about the SILK_LOGSTATS value being used and either the reason why the value cannot be used or the arguments to the external program being executed.

SILK_DYNLIB_DEBUG
    When set to a non-empty value, an application that supports plug-ins will print status messages to the standard error as it tries to open each of its plug-ins.

SEE ALSO
Analysts' Handbook: Using SiLK for Network Traffic Analysis, The SiLK Reference Guide, SiLK Installation Handbook, <http://tools.netsa.cert.org/silk/>