Thursday, September 10, 2015

Analyzing Bro Logs with Sagan

Overview

Applying log analysis tools to Bro logs can improve event correlation and expedite the use of pattern based signatures for quick detection. Sagan, a multithreaded log analysis engine written by Champ Clark "Da Beave" is a good candidate for performing log analysis on Bro for at the following reasons:

  • Sagan supports the Bro intel file format through a preprocessor
  • Contains existing rules for detections from Bro logs
  • Uses the familiar snort-like rule language
  • Ability to output data in unified2, integrates with existing tools like Snorby, Sguil, Squert, etc.
OSSEC is another popular tool that can perform log analysis and is able to integrate with other tools.

Implementation

To implement this Bro logs can be forwarded to a system running Sagan via a syslog daemon that supports forwarding on the Bro manager. There is also a syslog daemon listening on the Sagan host which accepts the Bro logs and writes them to a FIFO (named pipe) from which sagan reads. For each log sagan reads it will create a worker thread to perform analysis by applying its rules to the logs. See the documentation for details on installing and configuring Sagan. A few examples are given below for forwarding Bro logs to a remote syslog host where Sagan lives.

A forwarding example with Rsyslog (rainer-script):
module(load="imfile" PollingInterval="1")

# Read in dns.log on Bro manager
input(type="imfile"
      File="/usr/local/bro/logs/current/dns.log"
      Tag="bro_dns"
      Severity="debug"
      Facility="local6")

# Forward logs to sagan system
action(Name="sagan"
   Type="omfwd"
   Target="sagan.xyz.org"
   Port="514"
   queue.FileName="sagan-buffer"
   queue.SaveOnShutdown="on"
   queue.Type="LinkedList"
   queue.size="10000000"
   queue.maxdiskspace="10000000000"
   queue.syncqueuefiles="on"
   queue.checkpointinterval="100"
   Action.ResumeInterval="30"
   Action.ResumeRetryCount="-1")
A forwarding example using the older Rsyslog syntax:
$ModLoad imfile

# Read in dns.log on Bro manager
$InputFileName /usr/local/bro/logs/current/dns.log
$InputFileTag bro_dns:
$InputFileStateFile stat-bro_dns
$InputFileSeverity debug
$InputFileFacility local6
$InputRunFileMonitor
local6.debug @@sagan.xyz.org:514
Use multiple input configurations to send more Bro logs.

Rules

Sagan has a rules package that includes existing snort-like rules for bro logs, this file is named bro-ids.rules. Below are a few custom rules written by Jon Schipp which were presented at BroCon'15.

The first rule detects hosts which made at least 10 DNS requests in an hour (3600s) for names that are present in an intelligence feed. For this to work Sagan's Bro Intel preprocessor needs to be enabled and pointing to Bro formatted intel files in sagan.conf.
# sagan.conf
processor bro-intel: /usr/local/etc/ip.intel,/usr/local/etc/domain.intel,/usr/local/etc/url.intel
# Include custom bro rules
include $RULE_PATH/local-bro.rules
The magic of the rule is done with the bro-intel: domain option which matches all feed entries with type Intel::DOMAIN across the logs Sagan is analyzing.

# local-bro.rules
alert udp $EXTERNAL_NET any -> $HOME_NET $DNS_PORT (msg: "[BRO] Excessive Bad Domains (10+)"; bro-intel: domain; after: track by_src, count 10, seconds 3600; parse_src_ip: 1; parse_dst_ip: 2; classtype: suspicious-traffic; sid: 13000000;rev:1;)
Another example is a more advanced detection that works across multiple rules using flowbits, a feature of the snort-like rule language. Doing this we can perform basic correlation across different logs files such as checking for an HTTP request in http.log and then looking for a file that was carried across the connection in files.log. A practical example of this is to test for the successful use of a proxy by matching a HTTP request that is indicative of proxy behavior and then checking files.log after for a transferred file of a guessable significant size within 60 seconds of the original HTTP request. Having arithmetic operators like less than or greater than would be helpful in expressions where we want to compare byte counts but there are problems

  1. Expressions like these are not available in this rule language
  2. This language doesn't understand Bro fields (e.g. files.log has multiple fields of type count)

For example, because of current conditions we cannot tell Sagan to match on files.log where the value of the seen_bytes field is greater than say 1024 because arithmetic operators are not available, and even if they were Sagan wouldn't know which field to use to evaluate the expression (it could compare it to an IP address which contains numbers because it doesn't have the concept of data types). Though with some ingenuity and luck we can make due with a detection like this. There's a field in files.log called duration with a unique type of interval which looks similar to a floating point number. We can do a match on the decimal point and some digits that would indicate that the file took longer to transfer than x seconds. I chose 0.00 as the value to perform a negation match on i.e. the rule will only match if 0.00 is not present in the log line e.g. a duration of 1.24 will trigger an alert. This is a best guess indicator that the file transferred over the HTTP connection was large enough to take more than 0 seconds. Note that it does not take account of network conditions which makes it imperfect like most of the rules we have today.

The first rule matches on the HTTP CONNECT method commonly used by clients accessing a proxy. It does not alert due to its use of the noalert options. Its purpose is to trigger an event via a flowbit once a proxy attempt using CONNECT occurs, when this happens the next rule which is watching for the flowbit named bro_possible_proxy_connect will be called.
# Match on HTTP CONNECT methods
alert tcp $EXTERNAL_NET any -> $HOME_NET $HTTP_PORT (msg: "[BRO] Possible Proxyvia CONNECT"; content: " CONNECT "; content: "ROXY-CONNECTION"; parse_src_ip: 1; parse_dst_ip: 2; flowbits: set, bro_possible_proxy_connect, 60; flowbits: noalert; classtype: suspicious-traffic; sid: 11000002; rev:1;)
The second rule is called to match on files.log, it uses a few content matches to identify the log by its content such as containing hashes of files e.g. SHA. The parse ip instructions are used to noramalize the direction of the IP addresses used in the original HTTP request which will be in reverse given that the files will most likely becoming from the server. This is needed to provide correlation by tracking the offender IP across the HTTP request and transferred file, we don't want to alert from a transfer indicated in files.log by some other host which never made the HTTP CONNECT method!
# Follow up rule to validate the use of a proxy
alert tcp $EXTERNAL_NET any -> $HOME_NET $HTTP_PORT (msg: "[BRO] Proxy Detectedvia CONNECT"; content: "SHA"; content:!"0.00"; pcre: "/SSL|HTTP|FTP/"; parse_src_ip: 2; parse_dst_ip: 1; flowbits: isset,by_src,bro_possible_proxy_connect;
classtype: suspicious-traffic; sid: 11000004; rev:1;)
Syslog tags can be used in rules to identify different log files. There's plenty more to detect and explore - if you're performing log analysis on Bro share your rules and experiences with the community.

References

For more information see the Bro Integrations slides and talk at BroCon '15

Tuesday, September 8, 2015

Bro 2.4.1 Release

Bro 2.4.1 has been released. This release addresses a few potential DOS vectors using specially crafted connections. The release also contains minor updates to analyzers to reduce the number of messages in reporter.log. The source distribution is available on the download page. Our binary packages will be updated later today - users should be able to automatically update the package using their system package manager.

See CHANGES for the full list of changes in the release.

Since this is a bug fix release, we encourage users to update at their earliest convenience.

The Bro Team