Thursday, January 23, 2014

Intelligence Data and Bro

Overview

Intelligence data, or feeds, are an important source of network security information. Many internet security research centers, non-profit organizations, and commercial organizations provide intellegence data sets freely available to the public. (e.g. Emerging Threats, Shadow Server, etc.)

A solid solution for handling multiple intelligence feeds and acting upon them is to use Bro's Intel Framework in conjunction with its Input Framework to log hits seen on your network.

Hits, or matches, are logged and stored in intel.log.

$ head /bro/logs/current/intel.log
#fields ts      uid     id.orig_h       id.orig_p       id.resp_h       id.resp_p       fuid    file_mime_type  file_desc       seen.indicator  seen.indicator_type     seen.where      sources
#types  time    string  addr    port    addr    port    string  string  string  string  enum    enum    table[string]
1389646914.469513       Cb1T1Y3r56rjSRr0ac      70.173.122.62   49379   93.184.215.163  80      -       -       -       93.184.215.163  Intel::ADDR     Conn::IN_RESP   snort
1389646925.095172       CeHuLr2IfATgjhQPUg      195.178.109.14  43471   227.146.143.88  5900    -       -       -       195.178.109.14  Intel::ADDR     Conn::IN_ORIG   CIF - need-to-know
1389646979.298695       Cj9bjq43KThmM8TOA6      194.160.95.196  60430   119.254.16.21   80      -       -       -       www.dhgate.com  Intel::DOMAIN   HTTP::IN_HOST_HEADER    CIF - need-to-know
1389636016.229788       CNXlzu1VdMhKVEDCPf      42.82.115.14    52518   184.72.106.52   443     -       -       -       184.72.106.52   Intel::ADDR     Conn::IN_RESP   tor
1389636187.690988       CsGYCjrhwI313rWob       72.49.141.144   41270   118.219.221.39  3389      -       -       -     72.49.141.144   Intel::ADDR     Conn::IN_ORIG   ciarmy
...

Each log entry consists of a number of fields describing the connection:

ts:Timestamp
uid:Unique ID of connection
id.orig_h:Connection originator's endpoint IP address
id.orig_p:Connection originator's endpoint TCP/UDP or ICMP code
id.resp_h:Connection responder's endpoint IP address
id.resp_p:Connection responder's endpoint TCP/UDP or ICMP code
fuid:File indentifier (if file was found in connection)
file_mime_type:Libmagic file type (e.g. application/x-dosexec)
file_desc:Optional file type description
seen.indicator:The indicator that triggered the match (e.g. IP)
seen.indicator_type:
 The type of indicator (e.g. ADDR, DOMAIN)
seen.where:Location in Bro's where the event triggered (e.g. DNS::REQUEST)
sources:Intel data source description (e.g. emergingthreats.net)

Intel Framework

The Intel Framework provides the facilities to handle intelligence data in a meaningful manner (e.g. categorization, types, source, etc.). [1]

Supported types of intelligence data indicators are:

Intel::ADDR
Intel::URL
Intel::SOFTWARE
Intel::EMAIL
Intel::DOMAIN
Intel::USER_NAME
Intel::FILE_HASH
Intel::FILE_NAME
Intel::CERT_HASH

Before we can use the framework we must load it by adding the following lines to local.bro or equivalent:

@load frameworks/intel/seen
@load frameworks/intel/do_notice

The next step is to properly format the data set. Each field must be separated by a single tab character.

#fields indicator       indicator_type  meta.source     meta.url        meta.do_notice  meta.if_in
1.182.117.119   Intel::ADDR     ciarmy  http://www.ciarmy.com/list/ci-badguys.txt       T       -
2.187.28.215    Intel::ADDR     ciarmy  http://www.ciarmy.com/list/ci-badguys.txt       T       -
2.229.117.159   Intel::ADDR     ciarmy  http://www.ciarmy.com/list/ci-badguys.txt       T       -
4.35.96.216     Intel::ADDR     ciarmy  http://www.ciarmy.com/list/ci-badguys.txt       T       -
5.79.69.204     Intel::ADDR     ciarmy  http://www.ciarmy.com/list/ci-badguys.txt       T       -
5.135.146.0     Intel::ADDR     ciarmy  http://www.ciarmy.com/list/ci-badguys.txt       T       -
5.135.240.133   Intel::ADDR     ciarmy  http://www.ciarmy.com/list/ci-badguys.txt       T       -
5.153.54.130    Intel::ADDR     ciarmy  http://www.ciarmy.com/list/ci-badguys.txt       T       -
5.254.101.69    Intel::ADDR     ciarmy  http://www.ciarmy.com/list/ci-badguys.txt       T       -

Also, each field has a particular type of value. Two worthy of mention are meta.do_notice and meta.if_in because they work with the Notice Framework. meta.do_notice expects a boolean value which determines whether matches will be sent to the Notice Framework where entries will end up in notice.log, in addition to the normal intel.log.

$ grep Intel notice.log
1389636016.229788       CNXlzu1VdMhKVEDCPf      23.58.90.219  52518   184.72.106.52   443     -       -       -       tcp     Intel::Notice   Intel hit on 184.72.106.52 at Conn::IN_RESP     184.72.106.52   23.58.90.219  184.72.106.52   443
1389636016.517027       Cr1u7g2pFxgSzHBsRf      234.136.242.173 52519   184.72.106.52   443     -       -       -       tcp     Intel::Notice   Intel hit on 184.72.106.52 at Conn::IN_RESP     184.72.106.52  234.136.242.173 184.72.106.52   443
...

meta.if_in concerns itself with where a particular match was found in network traffic (e.g. HTTP::IN_HOST_HEADER) and serves as a restriction for matches sent to the Notice Framework i.e. they must be found in the location of meta.if_in.

Note: meta.if_in takes a single location value. If you would to like to specify more than one location for an indicator you will need an entry for each location you desire to be seen. e.g.

#fields indicator       indicator_type  meta.source     meta.url        meta.do_notice  meta.if_in
aolon1ine.com   Intel::DOMAIN   mandiant        -       T       DNS::IN_REQUEST
aolon1ine.com   Intel::DOMAIN   mandiant        -       T       HTTP::IN_HOST_HEADER
aolon1ine.com   Intel::DOMAIN   mandiant        -       T       SMTP::IN_FROM

The list of possible meta.if_in locations are: [2]

Conn::IN_ORIG
Conn::IN_RESP
Files::IN_HASH
Files::IN_NAME
DNS::IN_REQUEST
DNS::IN_RESPONSE
HTTP::IN_HOST_HEADER
HTTP::IN_REFERRER_HEADER
HTTP::IN_USER_AGENT_HEADER
HTTP::IN_X_FORWARDED_FOR_HEADER
HTTP::IN_URL
SMTP::IN_MAIL_FROM
SMTP::IN_RCPT_TO
SMTP::IN_FROM
SMTP::IN_TO
SMTP::IN_RECEIVED_HEADER
SMTP::IN_REPLY_TO
SMTP::IN_X_ORIGINATING_IP_HEADER
SMTP::IN_MESSAGE
SSL::IN_SERVER_CERT
SSL::IN_CLIENT_CERT
SSL::IN_SERVER_NAME
SMTP::IN_HEADER

Input Framework

The Input Framework [3] provides the facilities to read information from a text file, and in our case send it to the Intel Framework for processing.

Note: If you're running a Bro cluster, the intelligence files only need to be stored and read on the manager.

An absolute file path is required to read from a file. Three ways to do this are:

  1. Specify the file's full path

  2. Append the file name to @DIR which stores the directory path of the calling script. If the script with Intel::read_files is not in the same directory as the intel data sets this will not work.

  3. Create a constant that's value is the directory path in local.bro or equivalent.

    const feed_directory = "/opt/bro/feeds";
    

Here's an example using all three methods:

redef Intel::read_files += {
        @DIR + "/botcc.intel",
        @DIR + "/ciarmy.intel",
        "/opt/bro/feeds/malhosts.intel",
        "/opt/bro/feeds/malips.intel",
        feed_directory + "/tor.intel",
        feed_directory + "/snort.intel",
};

Obtaining Feeds

Two solutions for obtaining pre-formatted feeds for Bro to use are mal-dns2bro, a helper script for mal-dnssearch, or the Collective Intelligence Framework and its Bro output plug-in.

Mal-dns2bro

Mal-dnssearch [4] is a shell script I wrote that downloads, parses, and compares intelligence feeds against a number of popular application log files, reporting any matches. mal-dns2bro [5] is a helper script included with mal-dnssearch that formats feeds for Bro's Intel Framework to extend the application of intelligence data directly against live network traffic. mal-dns2bro has the ability to customize Intel Framework fields like setting meta.source, meta.url, meta.do_notice, and meta.if_in.

mal-dnssearch supports a number of feeds, ``mal-dnssearch -h'':

Malware List Options:
        -M <list>               Name of list, e.g. \`\`-M snort\'\'

        List:      |     Description:
        snort      -     http://labs.snort.org/feeds/ip-filter.blf (IP)
        et_ips     -     http://rules.emergingthreats.net/open/suricata/rules/compromised-ips.txt (IP)
        alienvault -     http://reputation.alienvault.com/reputation.generic (BIG file) (IP)
        botcc      -     http://rules.emergingthreats.net/open/suricata/rules/botcc.rules (IP)
        tor        -     http://rules.emergingthreats.net/open/suricata/rules/tor.rules (IP)
        rbn        -     http://rules.emergingthreats.net/blockrules/emerging-rbn.rules (IP)
        malhosts   -     http://www.malwaredomainlist.com/hostslist/hosts.txt (DNS)
        malips     -     http://www.malwaredomainlist.com/hostslist/ip.txt (IP)
        ciarmy     -     http://www.ciarmy.com/list/ci-badguys.txt (IP)
        mayhemic   -     http://secure.mayhemiclabs.com/malhosts/malhosts.txt (DNS)
        mandiant   -     https://raw.github.com/jonschipp/mal-dnssearch/master/mandiant_apt1.dns (DNS)

Download and install mal-dnssearch:

$ git clone https://github.com/jonschipp/mal-dnssearch
$ cd mal-dnssearch
$ sudo make install

Download Mandiant's APT1 data set, parse, and pipe (``-p'') it to mal-dns2bro for formatting:

$ mal-dnssearch -M mandiant -p | mal-dns2bro -T dns -s mandiant > mandiant.intel

Sample file output:

#fields indicator       indicator_type  meta.source     meta.url        meta.do_notice  meta.if_in
advanbusiness.com       Intel::DOMAIN   mandiant        -       F       -
aoldaily.com    Intel::DOMAIN   mandiant        -       F       -
aolon1ine.com   Intel::DOMAIN   mandiant        -       F       -
applesoftupdate.com     Intel::DOMAIN   mandiant        -       F       -

In the example above, mal-dns2bro reads in the mandiant list from stdin and sets the indicator type (``-T'') to DNS because the mandiant list consists of only DNS names. The source (``-s'') field is also set which is a short description of where the intelligence data came from.

mal-dns2bro will add the necessary tab separated columns for the Intel Framework. It accepts a list of a specific indicator type, but supports all of them, with one entry per line. It can read from stdin or from a file (``-f''). If you don't want to use mal-dnssearch, you can create your own lists with a text editor or other program and have mal-dns2bro format them for Bro.

Another example, downloading a list of tor nodes and customizing all fields:

$ mal-dnssearch -M tor -p | mal-dns2bro -T ip -s tor -n true -u http://rules.emergingthreats.net/open/suricata/rules/tor.rules > tor.intel

Sample file output:

#fields indicator       indicator_type  meta.source     meta.url        meta.do_notice  meta.if_in
103.10.197.50   Intel::ADDR     tor     http://rules.emergingthreats.net/open/suricata/rules/tor.rules  T       Conn::IN_RESP
103.3.188.169   Intel::ADDR     tor     http://rules.emergingthreats.net/open/suricata/rules/tor.rules  T       Conn::IN_RESP
105.237.43.166  Intel::ADDR     tor     http://rules.emergingthreats.net/open/suricata/rules/tor.rules  T       Conn::IN_RESP
106.186.21.31   Intel::ADDR     tor     http://rules.emergingthreats.net/open/suricata/rules/tor.rules  T       Conn::IN_RESP

In this last example, two new options are introduced: mal-dns2bro sets meta.do.notice to true (``-n'') which will send intel matches to the notice framework: the source URL, when applicable, is set (``-u'').

Collective Intelligence Framework

The Collective Intelligence Framework (CIF) is a cyber threat intelligence management system that pulls and stores feeds into a database for querying with CIF's tools. [6] CIF comes with a number of output plug-ins including one for Bro.

Installing CIF is pretty involved so I will refer the reader to its documentation. [7]

The CIF plug-in for Bro outputs a few different fields than we've seen so far: meta.cif_impact, meta.cif_severity, and meta,cif_confidence. Because of this we will need to load a Bro script, in addition to those required for the Intel Framework, that can handle these.

Add the following line to local.bro or equivalent:

@load policy/integration/collective-intel

Now, lets output domains related to botnets with a confidence level of 85 or greater formatted for Bro.

$ cif -q domain/botnet -c 85 -p bro > domain_botnet.intel

Sample file output:

#fields indicator       indicator_type  meta.source     meta.desc       meta.url        meta.cif_impact meta.cif_severity       meta.cif_confidence
computo164.laweb.es     Intel::DOMAIN   CIF - need-to-know      palevo  https://palevotracker.abuse.ch/?host=computo164.laweb.es (public)       -       high    85
ms4all.twoplayers.net   Intel::DOMAIN   CIF - need-to-know      palevo  https://palevotracker.abuse.ch/?host=ms4all.twoplayers.net (public)     -       high    85
hcuewgbbnfdu1ew.com     Intel::DOMAIN   CIF - need-to-know      palevo  https://palevotracker.abuse.ch/?host=hcuewgbbnfdu1ew.com (public)       -       high    85
arta.romail3arnest.info Intel::DOMAIN   CIF - need-to-know      palevo  https://palevotracker.abuse.ch/?host=arta.romail3arnest.info (public)   -       high    85

Create a list of machines known for scanning the internet:

$ cif -q infrastructure/scan -c 85 -p bro > intrastructure_scan.intel

Sample file output:

#fields indicator       indicator_type  meta.source     meta.desc       meta.url        meta.cif_impact meta.cif_severity       meta.cif_confidence
192.74.251.115  Intel::ADDR     CIF - need-to-know      ssh     http://danger.rulez.sk/projects/bruteforceblocker/blist.php (public)    -       medium  85
117.41.247.212  Intel::ADDR     CIF - need-to-know      ssh     http://danger.rulez.sk/projects/bruteforceblocker/blist.php (public)    -       medium  85
133.242.171.75  Intel::ADDR     CIF - need-to-know      ssh     http://danger.rulez.sk/projects/bruteforceblocker/blist.php (public)    -       medium  85

E-mailing Notices

This section presumes that Intel events are sent to the Notice Framework [8], described in the Intel section of this article.

Each time a Intel::Match event is generated, the intel data for that match is sent to the Notice Framework where a Notice is raised that has a type of Intel::Notice. To receive e-mail notifications upon a match add the following Notice type to emailed_types in local.bro or equivalent:

redef Notice::emailed_types += {
        Intel::Notice,
};

Suppression can be used to reduce the number of events generated for a specific Intel match. To limit each particular match to once per day add the following lines to local.bro or equivalent:

redef Notice::type_suppression_intervals += {
        [Intel::Notice] = 1day,
};

Log Queries

Bro-cut can be used to print fields in Bro logs. The following example will print a list of indicators only i.e. the host, hash, e-mail, etc. that triggered the match.

$ bro-cut seen.indicator < intel.log
61.36.24.57
193.107.16.206
192.0.72.2
...

To print a unique list of all ADDR types and the number of times they were seen try the following query. If you count be the number of fields 11 is seen.indicator_type and field 10 is seen.indicator:

$ awk '$11 == "Intel::ADDR" { print $10 }' intel.log | sort -t . -n -k 1,1 -k 2,2 -k 3,3 -k 4,4 | uniq -c
190 61.36.24.57
1 93.120.27.62
3 93.184.215.163
1 96.45.82.69
7 192.0.72.2
2 193.107.16.206

The next example illustrates an intel match found in the host field of the HTTP header.

1390425478.178039       CQzyg512Q8Mm2UrTc4       53.80.23.123   51220   65.52.128.33    80      -       -       -       tipranks.com    Intel::DOMAIN   HTTP::IN_HOST_HEADER    malhosts

Since the above connection is using HTTP there are bound to be more logs related to the connection such as those produced by the HTTP analyzer. We can search all Bro's logs for the connection identifier, a 4-tuple flow hash, to find other logs related to that connection.

$ zgrep CQzyg512Q8Mm2UrTc4 *.log.gz
1390425478.076356 CQzyg512Q8Mm2UrTc4      53.80.23.123 51220   65.52.128.33    80      tcp     http    123.242140      397     981     RSTR    T       0       ShADadr 12      1302    12      2514    (empty) US      US      nids-31-2
1390425478.398364        FQSaaGSHQsLn11w8j       65.52.128.33    53.80.23.123 CQzyg512Q8Mm2UrTc4      HTTP    0       SHA1,MD5        text/plain      -       0.000000        F       F       595     -       0       0       F       -       d6923c591dfa3cb616327a0bb44375b3        aa5438c500083ee69297d8c1a2ab4f82655c4632      -       -
1390425478.179990        CQzyg512Q8Mm2UrTc4      53.80.23.123 51220   65.52.128.33    80      1       GET     tipranks.com    /valuewalk/tipranks.js  http://www.valuewalk.com/2014/01/advanced-micro-devices-inc-amd-shares-dive-after-earnings/     Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.76 Safari/537.36  0       595     200     OK      -       -       -       (empty) -       -       -       -       -       FQSaaGSHQsLn11w8j       text/plain
1390425478.178039        CQzyg512Q8Mm2UrTc4      53.80.23.123 51220   65.52.128.33    80      -       -       -       65.52.128.33    Intel::ADDR     Conn::IN_RESP   malips
1390425478.178039        CQzyg512Q8Mm2UrTc4      53.80.23.123 51220   65.52.128.33    80      -       -       -       tipranks.com    Intel::DOMAIN   HTTP::IN_HOST_HEADER    malhosts

We found a number of events detailing the connection, these include the URL the user visited, the type of file that was transferred in the session, and another intel match from a different intel source. This is especially useful in cases where there are more than one HTTP request in a single TCP session.

Putting it all together

A powerful but simple solution is to write a daily cronjob that downloads and formats the latest version of each feed from which Bro continuously reads and then restarts Bro, if necessary. A restart is required if you want to purge entries that have been removed from the feeds, but not if you only want the new entries because Bro keeps the file open and will pick up any new additions.

If you come across any trouble be sure to check reporter.log.

References:

[1]http://www.bro.org/sphinx-git/frameworks/intel.html
[2]http://www.bro.org/sphinx-git/_downloads/where-locations.bro
[3]http://www.bro.org/sphinx/frameworks/input.html
[4]https://github.com/jonschipp/mal-dnssearch
[5]https://github.com/jonschipp/mal-dnssearch/blob/master/tools/mal-dns2bro.sh
[6]https://code.google.com/p/collective-intelligence-framework/
[7]https://code.google.com/p/collective-intelligence-framework/wiki/ServerInstall_v1
[8]http://www.bro.org/sphinx-git/frameworks/notice.html

2 comments:

  1. Hello,

    what kind of cif Setup is necessary for this bro integration? Is a client instance sufficient or do I require a cif Server instance?

    Regards
    Sebastian

    ReplyDelete
  2. Sorry for the late reply, I didn't see this message but for any others that want to know - you can use the client as long as it's configured to use a CIF server

    ReplyDelete