Monday, December 5, 2016

The Intelligence Framework Update

Note: This is a guest blog post by Jan Grashöfer, the original post may be found here

Recently Bro's intelligence framework was refactored and extended with a couple of new features. This post will discuss the updates and tries to clear some of the backgrounds that turned out to be common pitfalls in the past.

The Intelligence Framework Data Model

Understanding the intel framework's data model is the key for exploiting its full potential, so let's have a closer look: The core of an intelligence datum is the indicator (also indicator of compromise, IoC), e.g. an IP, hash or domain name (for a list of available types see Bro's script reference). The indicator can be enriched by meta data of different kinds, e.g. a description, url or severity level. The same indicator can be obtained from different intelligence sources, providing different meta data. Thus in Bro's intelligence framework, a plain indicator can be described by multiple meta data records. A meta data record is uniquely identified by its source. Figure 1 illustrates the described relation.

Figure 1: Data Model of Bro's Intelligence Framework

Keeping this in mind now let's have a look at the intel files. Each line represents what is called an intelligence item (Intel::Item). An intelligence item consists of the indicator, the indicator's type and a meta data record (fields prefixed by meta.), including the meta data source. In terms of the data model this is equal to the internal representation with n=1. So what about the relations? The second thing to keep in mind regarding the intelligence framework's data model is the fact that intelligence files are only the supply mechanism to feed intelligence data into Bro. The "database" Bro uses for matching is kept in memory following the described model. So how do they interact? Bro uses the input framework to read intelligence files. Each line triggers a corresponding insert into the in-memory data structure. Imagine the same indicator was obtained from two different sources, each supplying different meta data. Thus the same indicator occurs multiple times (in a single file or in different files). Ingesting the files, Bro will store the indicator only once, associate both meta data records to it but will not duplicate it (see Figure 2).

#field indicator indicator_type meta.source meta.desc meta.url Intel::DOMAIN test-source1 domain for testing Intel::DOMAIN test-source2 domain for testing

Figure 2: Internal representation of the example items

In case a file is changed (Note: Changes have to be atomic e.g. using mv), Bro will reread the whole file and update the in-memory data structure. Changing some meta data values will cause a corresponding update but changing the meta data source, Bro will assume the indicator was obtained again from a new source, causing Bro to add another meta data record and assign it to the given indicator. Likewise Bro will add a new intelligence datum if the indicator or indicator type was changed, while keeping the original item in the in-memory data structure. Accordingly deleting a line from an intel file will not delete the corresponding intelligence item from Bro's database (see the next section on how to get rid of inserted intelligence data). This means in particular that the intelligence files on disk do not necessarily reflect the actual database Bro uses for matching.

Now that we have the intelligence indicators at hand, let's have a quick look how matching works. In theory, every piece of data, that is made available by Bro's events, can be used for matching. Once there is something that should be checked against the database of indicators, the datum is wrapped inside a Intel::Seen record and sent to the intel framework. The record contains the seen indicator, its type and additional information, e.g. where the indicator was seen. Bro comes with a set of policy scripts located in intel/seen/ that report indicators by evaluating well-known events. For example connection_established provides IP addresses or dns_request is used to extract domains. Figure 3 illustrates the data flow.

Figure 3: Data Flow of Bro's Intelligence Framework

Finally there is one detail left, which might be not that intuitive. When it comes to interacting with the intelligence framework, most of the functions, hooks and events use Intel::Item to pass information about indicators. In case an indicator is associated with more than one meta data records, it will be unrolled into a set of multiple items. For example the Intel::match event's items will contain an Intel::Item record for every meta data record that is associated with the matched indicator. So we have been talking about a yet simple but still relational data model and every time it is accessed it gets denormalized. Doesn't seem very smart, right? The reason behind is that the import (intel files) as well as the output (Bro's logfiles) is based on CSV-like plain text files (although writing JSON is possible, nested structures are not supported). All in all the intelligence framework's design realizes an easy to use interface while providing as much flexibility as possible. Theory done. Time for some new features.

Removing Intelligence Items

Prior to the refactoring, the only way to get rid of an intelligence item was to whitelist it using Seth Hall's intel-extensions (we will come back to that). The only way to purge an item from the in-memory datastructure was to restart Bro. In case of long running live systems or frequently changing intelligence data that was a major handicap. In context of the framework update a new function has been added: remove: remove: function(item: Item, purge_indicator: bool &default = F); The Intel::Item type represents a single line of an intelligence file and thus just contains a single meta data record. But keeping in mind the data model there might be multiple meta data records associated to an indicator. In this case, only the meta data record matching the given meta data will be deleted. As meta data records are identified by source, it would be sufficient to specify only the source name inside the item that is passed to the function. In case there is no meta data left, the whole indicator is removed. If purge_indicator is set, the given metadata is ignored and the indicator is removed including all possible instances of meta data associated.

In principal the new remove function allows any script to delete an intel item. Imagine you have accidentally added your webserver's IP and alerts start flooding. Now a small tool would be great to remove that IP from your Bro instance without shutting down Bro. These extensions contain a small python script (utils/ that connects to Bro using broker and triggers item removal. The corresponding Bro script (scripts/remote_delete.bro) sets up broker and handles incoming deletion requests (Note: As broker is under development, there is a high probability that the scripts do not work with current master as of reading these lines):

event Intel::remote_remove(indicator: string, indicator_type: string)
    local item: Item = [
        $indicator = indicator,
        $indicator_type = type_tbl[indicator_type],
        $meta = record($source = "")
    remove(item, T);
The only thing that's done here is the composition of an Intel::Item record using the values sent by the python script to call the remove function (type_tbl is a string-indexed table to map a string to the corresponding Intel::Type). Instead of this broker-based solution, one could also write a script, analog to the intelligence import, that reads files containing indicators to delete. While these possibilities are already quite useful, the following new feature provides another excellent use case. So let's continue.

Intelligence Expiration

Intelligence expiration is the new feature I like most. Imagine we are ingesting a large intelligence feed of probably bad IPs into Bro. On the first day there is a hit, that indicates some malware is calling home. On the second day there is nothing. But on the third day the owner of the IP changed (think of agile cloud environments) and the system behind now offers a legitimate service. As users start to use that service, false positives pop up. The bottom line is that most intelligence data has a natural half life (e.g. hashes might be an exception here). So let's put an expiration date on it.

Bro's intelligence framework now allows to configure the Intel::item_expiration interval. Once an indicator expires, the intel framework executes the item_expired hook passing the indicator, its type and the associated meta data as arguments. The hook can be used to handle expiration. By default the intel framework won't do anything except executing the hook, so we are free to use that mechanism for whatever we like. But in case the hook chain is broken (see Bro's script reference for details about hooks), the expired indicator will be removed automatically. Coming back to the IP feed example, all we need to do is configuring the expiration interval and break the hook chain to remove expired indicators. As this is somewhat the default case, Bro ships with a new policy script do_expire.bro:

##! This script enables expiration for intelligence items.

@load base/frameworks/intel

module Intel;

redef Intel::item_expiration = 10min;

hook item_expired(indicator: string, indicator_type: Type,
    metas: set[MetaData]) &priority=-10
    # Trigger removal of the expired item.
So all that is left to do for us is loading that script and adapt the expiration interval according to our needs:

@load intel/do_expire
redef Intel::item_expiration = 2days;
Neat, isn't it? Something to keep in mind is, that expiration time runs as soon as an item is inserted into Bro. In case the item is "reinserted", the expiration timer is reset. Note: Whenever an intelligence file changes, all items listed in the file are reinserted! Technically this allows to keep the intel files and the intel database inside Bro in sync. For example one could define an expiration interval of 1 hour plus 30 minutes buffer. Now scheduling an update of the intel files every hour would cause an expiration timer reset of all indicators corresponding to items contained in the files, while indicators of items that have been removed from the files will expire in the given time frame.

Extending the Intelligence Framework

The next feature that is worth to discuss is the new extension mechanism. To be precise this feature is not completely new as it is based on the intel extensions created by Seth Hall (see The idea is to allow reacting to an intelligence match. As it turned out to be very useful this was integrated into the intelligence framework. Now it is possible to influence the framework's matching behavior via the extend_match hook. The hook receives the info record to log, the seen record that was observed and a set of items that have been matched (remember, the set of items is the unrolled internal representation). A hook may change these values and thus can influence what is logged. Additionally breaking the hook chain will prevent the intelligence framework from logging the match at all. A good example how to use the new mechanism would be whitelisting indicators. That means indicators are kept in memory for matching but logging of matches gets prevented. Bro already ships with a policy script (whitelist.bro, see Bro repository) that implements whitelisting. But as this was already part of Seth's intel extensions, let's discuss another thing that can be achieved using the new extension mechanism.

A desirable functionality would be to allow enriching item's meta data with some extra information, aggregate this information in case of a match and extend the intel log accordingly. For example one could add identifiers for the local Security Information and Event Management system (SIEM). The following script does the job:

module Intel;

export {
    redef record MetaData += {
        ## My SIEM identifier.
        siem_id: string &optional;

    redef record Info += {
        ## Set of SIEM IDs involved.
        siem_ids: set[string] &optional &log;

hook extend_match(info: Info, s: Seen, items: set[Item])
    info$siem_ids = set();
    for ( item in items )
        if ( item$meta?$siem_id )
            add info$siem_ids[item$meta$siem_id];
At first the Intel::MetaData record is extended with a SIEM identifier. Then the Intel::Info record is extended to allow logging a set of SIEM identifiers (keep in mind that a single indicator could have been obtained from multiple sources resulting into multiple meta data records associated). Note that the fields added to the info record have to be optional or defined with a default value as the record gets created by the intel framework, which does not know about the field. Finally the hook implements the aggregation logic. It initializes the set, loops the matched items and adds the SIEM identifier, if present, to the set. That's it.

Using this feature, the intelligence framework can be extended in multiple ways. Let me just give one last example. Let's assume we have a feed that publishes domains generated by Domain Generation Algorithms (DGA). These domains are only valid for a certain time window. However, the time window might differ for each of the DGAs. Just ingesting the feed would blow memory and performance sooner or later. So what to do? We could combine the expiration feature and the extension feature and implement per item expiration. The do_item_expire.bro script implements per item expiration by allowing to define individual expiration timeouts using the new meta data value expire.

The small things

Last but not least there are some minor improvements to the intelligence framework. Minor in terms of visible effect, which is definitely just the tip of the iceberg! We don't need to dive into the details. Let's just keep it with the good news: The intelligence framework supports subnets. The new type Intel::SUBNET can be used to ingest subnets in CIDR notation. The subnets are matched against seen addresses. Thus a hit on a single IP could be triggered by an intelligence item describing the exact address or a subnet containing the address or both. To distinguish these cases the new field matched was added to the intel log. As the name might suggest, matched is a set of intelligence types that triggered the hit. Furthermore subnets might overlap. Assume Bro's intelligence database contains and In this case seeing would as well trigger a single hit, caused by multiple indicators. At this point it could be useful to recall the data model again. A single Intel::Seen record that is reported can trigger multiple indicators. Each indicator can have multiple meta data records attached, as the same indicator can be obtained from different sources. So in case of addresses there are two levels of indirection.

Another couple of small changes hides inside the do_notice.bro policy script. Notice emails are extended to contain the service(s) inferred for the connection that triggered the hit as well as the intel source of the matched indicator. Additionally an identifier is added. Having an identifier allows notice suppression in the notice framework. To suppress intel notices for 12 hours we just need a simple redef:

redef Notice::type_suppression_intervals += {
    [Intel::Notice] = 12hr,
The notice identifier is composed of indicator, originator's and responder's IP without considering the direction of the flow. Thus all connections between two IPs regarding the given indicator will ignored for the defined suppression interval. Note that only the corresponding notices are suppressed. The intel log will still contain all hits.


This blog post discussed the data model of Bro's intelligence framework and the new remove function. Furthermore the intelligence expiration and match extension mechanisms have been explained. Finally the new type for subnets and the changes to the do_notice.bro script have been reviewed. I hope this post could shed some light on the ideas behind Bro's intelligence framework. Have fun integrating the framework into your Bro deployment!

Tuesday, November 29, 2016

Donate to The Bro Project

Bro Community,

As 2016 comes to a close, please consider adding The Bro Project to your list of charitable donations. We are managed by Software Freedom Conservancy, which is a 501(c)(3) organization and therefore exempt from US taxes.

To donate via credit card, click on the "Donate" button below, or go our site for more payment options:

If your commercial organization is considering sponsoring the Bro Project, you can find more information about it on our sponsorship page.

Thank you. And have a happy new year.

The Bro Project

Monday, November 28, 2016

New additions to the Bro Leadership Team

Last year when we announced The Bro Project had joined Software Freedom Conservancy we also announced the formation of Bro Leadership Team. The team consists of key contributors and community representatives working with SFC to set the direction of the project.

The Team has recently added two more members to the group: Johanna Amann and Martin van Hensbergen. As you may know, Johanna is a Bro developer and works on ICSI SSL Notary Service. Martin is a Threat and Malware analyst and the creator of the Bro (RFB)VNC parser.

Welcome Johanna and Martin, thank you for your contributions to the Bro Project!

The complete Leadership Team is now:
  • Johanna Amann, International Computer Science Institute 
  • Seth Hall, International Computer Science Institute
  • Keith Lehigh, Indiana University 
  • Vern Paxson, University of California at Berkeley
  • Michal Purzynski, Mozilla Foundation
  • Aashish Sharma, Lawrence Berkeley Lab 
  • Adam Slagell, National Center for Supercomputing Applications 
  • Robin Sommer, International Computer Science Institute
  • Martin van Hensbergen, Fox-IT

Monday, November 21, 2016

Bro4Pros 2017: February 2nd in San Francisco

Mark your calendars! Bro4Pros 2017 will be on Thursday, February 2nd in San Francisco, CA at Salesforce's Spear St. office (map).

Bro4Pros is a one-day workshop for advanced Bro users (i.e., those who use Bro on a daily basis, feel comfortable customizing its configuration, and have written scripts on their own).

This is a joined community effort and get-together, and the program will depend to a large degree on what people want to talk about. Attendance is limited to ensure an interactive and productive atmosphere.

We scheduled this year's workshop immediately after the Usenix Enigma conference to make travel a little more convenient for out-of-towners.

Registration is free and will open at 11am PST on Thursday, December 1st. Seats are limited and are first-come, first serve. If you have to cancel your registration please contact us to release your seat.

Call for presentations:

We have a few spots available for community presentations.

Send abstracts (max 500 words) to:
Subject: Bro4Pros 2017 Call for Presentations
Submission due date: January 6th, 2017

More details about the event can be found on our event page.

Thank you to Salesforce for sponsoring this event.

Thursday, November 17, 2016

Bro 2.5 released

We are very happy to announce the release of Bro v2.5. The new version is now available for download! Here is a brief summary of some of the new features and improvements:
  • Bro now includes the NetControl framework. This framework allows easy interaction with hard- and software switches, firewalls, etc. 
  • Support for the SMB protocol (SMB1 and SMB2), including GSSAPI and NTLM.
  • Support for the remote framebuffer protocol (RFB), that is used by VNC servers for remote graphical display.
  • The Intelligence framework was refactored and extended. It now supports, for example subnet indicators and item deletion/expiration.
See NEWS for release notes and CHANGES for the exhaustive commit list. Since the first beta version a series of smaller problems were fixed, in particular with the new SMB analyzer. We also added preliminary support for TLS 1.3. There were no significant changes anymore since the second beta version.

Bro 2.5 is currently available as source code, binary packages will come soon.

Thursday, November 3, 2016

Bro 2.5 Beta2

We are happy to announce that the second beta of Bro v2.5 is available for download. The main changes since the first beta are:
  • Lots of small fixes to the SMB analyzer. (Note that the analyzer is disabled by default)
  • Preliminary TLS 1.3 support.
  • Lots of other small fixes.
For a full list of changes see the CHANGES file. For information on the main features added in Bro v2.5, see our NEWS file and the earlier blog post.

Feedback is encouraged and should be sent to the Bro mailing list. As previously stated, we do not recommend using a beta release for production use.

Thursday, October 20, 2016

Contributing to the Bro Project

Recently we have had a number of community members ask us for suggestions for contributing back to the Bro Project. We have updated the Community page on our website to reflect the new options available.

Custom Scripts and/or Plugins

We encourage Bro users to make their custom scripts and/or plugins available to the community by creating a package and submitting it to the Bro Package Source. See the README file of that GitHub repo for more instructions on how to create a package and submit it. Once your package is accepted, it becomes installable via the Bro Package Manager.

Patches and New Functionality

For working on the Bro codebase itself, work from our official GitHub mirrors or clone the master repositories directly from git://<repo>. See our contribution guidelines for more information.

Writing Documentation

We are grateful for any corrections or contributions to documentation. Send documentation to or submit a ticket to our issue tracker.

Provide community support

Respond to user questions on the Mailing List, Twitter, IRC, and Gitter.

Financial Support

Become a Bro Future Fund sponsor, make an individual donation, or sponsor Bro events like BroCon.