Thursday, December 18, 2014

Bro Rewind 2014

Bro 2014

Welcome to the Bro monthly newsletter, which for the month of December features the Bro annual newsletter, recapping the events of 2014.
We will talk about:
  • Bro events in 2014. 
  • New resources for the community: 2014 the Bro community gained many new resources to learn, teach, and get help with using Bro. 
  • Bro impact: an overview of Bro's visibility and impact in terms of statistics and awards. 
  • Security is broken: highlights of the security breaches we saw in 2014. 
  • Bro dev: new developments, major releases.
  • Bro research: along with being a widely used NSM, Bro also both enables research and itself inspires research and innovation. 

Bro Events

  • Cybersecurity Summit 2014 — August 2014 
  • BroCon ‘14 — August 2014 
  • Floss Weekly — May 2014 
  • DOE Network Monitoring Group Meeting — May 2014, Lawrence Berkeley National Laboratory 
  • BSides Cincinnati — May 2014 
  • Troopers 2014 — March 2014. TalkBro: A Flexible Open-Source Platform for Comprehensive Network Security Monitoring.  Slides.
  • The Bro Team presented a members-only two-day Bro training workshop for the Department of Energy.

Bro Resources


Over the last year the Bro team continued to improve the usability of Bro, also in terms of resources, such as documentation. In this section we want to focus on new developments.

Bro Center of Expertise


The Bro Center of Expertise is a central point of contact for institutions funded by the National Science Foundation (NSF) that bundles the Bro Team’s expertise and offers it to NSF-supported sites seeking advice.

The Center provides the umbrella for many of the efforts we discuss in the following.

The More You Bro


The More You Bro is a series about various features of Bro, taught in the format of a hands-on tutorial. The episodes are intentionally brief to keep the content focused and approachable to new learners. Interested in suggesting a topic? Send us an email at info@bro.org or Tweet us @Bro_IDS.

The Bro Teaching Community


The Bro Teaching Community aims to create a knowledge base and resource collection for educators, ranging from example curricula and slide sets to exercises for all purposes and skills levels. By coordinating and synchronizing existing and future teaching efforts, we want to help share materials, and exchange “lessons learned” from different settings. With members of the core Bro team involved, the Community also helps with technical questions and provides guidance on using Bro effectively.

The Bro Teaching Community offers a bi-weekly meeting as well as access to a restricted git repository where we collect reusable teaching material. In the meetings we discuss possible curricula, technical problems, and other related topics.  The Bro Teaching Community is a collaboration of the Bro core team and interested university faculty.

Get in touch via info@bro.org.

The Bro Playground


Teaching, learning, and testing are activities that are sometimes hard to distinguish. They all need a non-disruptive space to enable exploration without risking harm to productive systems.

In 2014 the Bro team released two new tools that allow you and your students to explore Bro in a safe way.

Bro Live!


Bro Live! is a training system that gives users hands-on access to a Bro learning environment without having to install a virtual machine and deal with associated dependencies. Bro Live! can be built with exercises for a given class or workshop, with access to the environment limited to the duration of the event if desired. All the user needs is an SSH client and Internet access.

Bro Live! is a Linux-based sandbox system, relying on Linux containers, OpenSSH, and Docker. It places the user in their own isolated environment with shell access to Bro, the exercises, and the standard Unix toolset. The user's work is saved in their container typically for the duration of the training event and can be easily re-attached at anytime during the event to continue their work.

To use Bro Live! download it from Github.

ISLET


The work on Bro Live! led to the development of ISLET.

ISLET, the Isolated, Scalable, and Lightweight Environment for Training, is a platform used to teach Linux-based software that reduces the administrative overheard of building training environments and ensures a smoother training experience for users than comparable virtual appliance-based training. We intend ISLET to provide an improved replacement to event training that relies on virtual machines. It excels at quickly providing user's with shell access to containers to play with network security and other Linux based tools.

The official BroLive! training image works with ISLET, and we launched a precursor at BroCon14.

https://registry.hub.docker.com/u/broplatform/brolive/
https://github.com/jonschipp/ISLET

Try.Bro


Try.bro.org is a web-based scripting sandbox made freely available to users on our site. No login. No installation. No trouble.

We have included a few basic scripts and pcaps to help users get started. Users can paste their own scripts or upload their own pcaps. The environment includes version control to test your scripts against current or previous versions of Bro. In addition, Try.bro caches a user's work and generates unique URLs to enable sharing with others. No more copying and pasting scripts or log files, just send the link. We store code fragments for three days and pcaps for one hour, resetting the timeout when the link is used.

Bro Impact


Since the make-over of Bro in 2012, which specifically targeted better usability for production deployments, Bro has attracted more users every year. All the Bro 2.x releases were enabled by an NSF award that ended in 2014, with the NSF Center now continuing that work. 

With the number of users, the number of contributors and third party scripts and extensions grew as well. In this section we sketch these developments.

InfoWorld awarded Bro a 2014 "Bossie Award" in the category "The best open source networking and security software", and they also included Bro into their list of "11 open source security tools catching fire on GitHub". Indeed, Bro is at the top of GitHub's security showcases list now and has more than 640 stars.

We typically see about 10,000 direct downloads per version from our main server.  These tend to come from a couple thousand unique ASNs across about 150 countries. These numbers do not include downloads from GitHub, nor what has arguably become the most common way for new users to get started with Bro: Security Onion, a Linux-based live DVD environment tailored to security monitoring, which includes Bro as a key component.

During the recent years attendance at our annual Bro user meetings grew from originally 30-50 people to 150 attendees from 60 different institutions at the 2014 event.

Our Twitter account shows almost 3,000 followers, and the main Bro mailing list now reaches close to 1,000 people.

Security is Bro-ken


2014 saw a number severe security incidents, many of them concerning TLS/SSL. Here is a collection of some of the most important cases.

Heartbleed


The Heartbleed vulnerability in the widely used OpenSSL library can reveal memory contents of processes running OpenSSL, which can include highly sensitive data such as encryption key material. Due to the ease of exploiting it and the large number of vulnerable servers, the vulnerability was very widely reported, and represents one of the most serious security problems in the Internet this year. Bro includes a thorough detection script that can alert users if Heartbleed exploits are performed on their network.

To enable Heartbleed detection, load the policy/protocols/ssl/heartbleed.bro script.  If you use broctl, it will be loaded by default in a new installation using this branch. If using Bro on the command line, e.g., to read a trace, specify it directly:

bro -r [trace] policy/protocols/ssl/heartbleed

As usual, Bro will write the corresponding notices to notice.log.

Shellshock


Another significant vulnerability was first "announced" a patch message.  On September 24th, news went viral about a Bash patch that revealed a very serious vulnerability in Bash: ''... the vulnerability arises from the fact that you can create environment variables with specially-crafted values before calling the Bash shell. These variables can contain code, which gets executed as soon as the shell is invoked. The name of these crafted variables does not matter, only their contents.'' [source]

In other words, this bug allows anyone to execute their own code on affected remote hosts!  Even worse, if a vulnerable server runs as root, an attacker exploiting the vulnerability can immediately, and trivially, acquire full control over the server's system.

A Bro Shellshock detector was released September 25th by Broala.

SSLv3 - Poodle


SSL headaches were still not done for the year. October saw the discovery of a protocol flaw in SSLv3.  To find SSLv3 servers in Bro logs:

cat ssl.log | bro-cut version id.resp_h | grep "^SSLv3" | awk '{print $2}'| sort | uniq -c | sort -nr

Blog post: The SSLv3 #Poodle Attack & current SSL usage statistics from the ICSI SSL Notary  (http://t.co/lJeHc1DGNc). — ICSI Notary (@ICSInotary) October 17, 2014.

Bro Dev


During the last year Bro was developed further and extended. This section presents a snippet of the coding news from the Bro universe.

Bro 2.3


The release of 2.3 freed Bro from its dependency on Libmagic. The release brought (among other things) new SSL functionality, e.g., to detect the Heartbleed vulnerability; analyzers for SNMP and Radius; and extended capabilities for PF_Ring.

Users still operating on 2.2 go can find out what's exciting about 2.3.

Packet Bricks


We are happy to announce an initial prototype of Packet Bricks, a new Bro-related project written by Asim Jamshed from KAIST, who visited the Bro team in Berkeley over the summer.

Packet Bricks - which is still under active development - is a Linux/FreeBSD daemon that is capable of receiving and distributing ingress traffic to user-land applications. Its main responsibilities will eventually include: (i) load-balancing, (ii) duplicating, and/or (iii) filtering ingress traffic across all registered applications. The distribution is flow-aware (i.e., packets of one connection will always end up in the same application). Packet Bricks leverages the netmap packet I/O framework for handling packets efficiently, and employs netmap pipes to forward packets to user-land applications.

Packet Bricks is available on github. It's still a very early piece of software, and we announce it at this time primarily for users willing to help us collect some first experiences with it. If you have any feedback, please send it to the Bro development mailing list. If you aren't subscribed yet, do so here.

Bro Research


Bro's powerful capabilities to analyze traffic makes it a powerful research tool, as well as itself serving as a domain for research.

BinPAC++ Release


In the context of ICSI's ongoing research projects, we developed a prototype of BinPAC++.

BinPAC++ is a next-generation parser generator that makes it easy to build parsers for network protocols, file formats, and more.  It provides a comprehensive system that enables developers to write attributed grammars defining both syntax and semantics of an input format inside a single comprehensive scripting language.

The BinPAC++ toolchain, built on top of HILTI, turns such grammars into efficient parsing code that exposes a well-defined C interface to its host application for feeding in input and retrieving results. At runtime, parsing proceeds fully incrementally—and potentially in parallel—on input streams of arbitrary size. Compilation takes place either statically at build time, or just-in-time at startup.

You might have seen Robin Sommer's demo at BroCon'14. If you want to try it out, you can fetch the code, though keep in mind that it is still a prototype and not yet production-ready.

Bro Related Publications


Here is an overview of this year's research output. The list contains publications of Bro team members but also external publications that use or extend Bro for their work. Please note that we do NOT know if this list is complete, there might be more publications out there and you are invited to let us know about each of them.

You can find a more complete list of Bro-related publication here


Wednesday, November 19, 2014

Bro Monthly #3

Bro Monthly #3


Welcome to the 3rd Bro Monthly newsletter.
This month we cover the following topics:
  • Bro Meet-ups: a new monthly category for Bro related gatherings and groups,
  • Bro teaching and training,
  • Bro in research,
  • Bro in the wild,
  • Bro-active: current exploits, attacks, and how Bro can help, and other everyday Bro.

Call for news:


If you want to point us on anything that should be in the next monthly just let us know, send mail to news@bro.org or tweet it to @Bro_IDS.

Bro Meet-ups


This new category lists all meet-ups we hear of that are somehow related to Bro. If you send us the information we can list your event here. Just write to info@bro.org.

OpenNSM


OpenNSM aims to provide a place for network security analysts and those interested in information security with a network security and incident response focus to share tricks, solutions, work on projects, and other knowledge about the subject. We're not aware of any other active NSM user groups in the United States, and have the ambitious goal of being a premier place for students, professionals, and hobbyists, from all over to share their research, tools, and techniques in a laid back and friendly environment. Remote attendance is available. Join the mailing list or Facebook group for meeting info.

They've had 3 presentations from Bro Team members so far and more to come!

More info: http://opennsm.ncsa.illinois.edu/

Bro teaching and training


ISLET


The Isolated, Scalable, & Lightweight Environment for Training is container system for teaching Linux based software with minimal participation and configuration effort. You can use ISLET to teach Bro by installing the BroLive! environment ('make install-brolive-config') after install ISLET.

https://github.com/jonschipp/islet
https://registry.hub.docker.com/u/broplatform/brolive/


Bro research


HILTI


When developing networking systems such as firewalls, routers, andintrusion detection systems, one faces a striking gap between the easewith which one can often describe a desired analysis in high-levelterms, and the tremendous amount of low-level implementation detailsthat one must still grapple with to come to a robust solution. At thisyear's Internet Measurement Conference (IMC) we presented a prototypeof "HILTI", a platform that bridges this divide by providing much ofthe standard low-level functionality, without tying it to any specificanalysis structure.


Beyond pattern matching: a concurrency model for stateful deep packet inspection


On modern multi-core processing platforms, intrusion detection systems need to scale across a large number of processing units--a challenge, as distributing their analysis must not come at the cost of decreased effectiveness in attack detection. At ACM's Conference on Computer and Communications Security (CCS) we presented a novel domain-specific concurrency model that facilities concurrent traffic analysis by partionining input according to fine-granular analysis scopes.


Bro in the wild




Bro-active


SSLv3


SSL continues to produce headaches, last month's hick-up was a protocol mistake in SSLv3. 

To find SSLv3 servers in your Bro logs this line helps you:

cat ssl.log | bro-cut version id.resp_h | grep "^SSLv3" | awk '{print $2}'|  sort | uniq -c | sort -nr


FireEye APT28


Bro Passive DNS tool


Friday, November 7, 2014

Using Bro to Build a Passive DNS Database

Searching DNS logs became a lot faster with the launch of our Passive DNS tool for Bro. It uses Bro's DNS logs to build a database that is more compact, and therefore a lot easier to search.
See how we did it by checking it out on GitHub.

Friday, October 10, 2014

Bro Monthly #2

Bro Monthly #2

Welcome to the 2nd Bro Monthly newsletter.
This month we cover the followoing topics:
  • Bro won a Bossie,
  • Bro.org needs help,
  • the Shellshock incident,
  • new features in the Intel framework,
  • news on BinPAC++,
  • Bro in research,
  • Bro in the wild,
  • Bro on demand.

Bro.org Needs Help


Bro has changed -- and improved -- a lot during the last years. Bro.org needs to keep pace with our developers and engineers, so we are looking for a web developer who can help us to give bro.org a facelift.

Please find all details on our jobs site.

Current Developments


Shellshock


The topic of the month was for sure the shellshock.
On September 24th the news went viral about a Bash patch that revealed a very bad vulnerability in Bash:
''...the vulnerability arises from the fact that you can create environment variables with specially-crafted values before calling the Bash shell. These variables can contain code, which gets executed as soon as the shell is invoked. The name of these crafted variables does not matter, only their contents.'' [from]
In other words, this bug allows anyone to execute their own code on affected remote hosts, in some cases even as root.

A Bro shellshock detector was released September 25th by Broala.

If this is all news to you, please stop reading here!
Patch your system NOW and use the Bro detector to see if you were attacked. You are welcome to continue reading afterwards.

BinPAC++ Code Release


BinPAC++ is a next-generation parser generator that makes it easy to build parsers for network protocols, file formats, and more.
BinPAC++ is more than just a "yacc for protocols": it's an all-in-one system that enables developers to write attributed grammars defining both syntax and semantics of an input format inside a single comprehensive scripting language.

The BinPAC++ toolchain, built on top of HILTI, turns such grammars into efficient parsing code that exposes an well-defined C interface to its host application for feeding in input and retrieving results. At runtime, parsing proceeds fully incrementally—and potentially in parallel—on input streams of arbitrary size. Compilation takes place either statically at build time, or or just-in-time at startup.

You might have seen the name BinPac++ in the last Bro Monthly or even seen Robin Sommer's demo at BroCon'14. If not watch the video of the demo, get excited, and fetch the code from here.

Don't get too excited, though, because this is all still in prototype state, and not production-ready yet.

Intel Framework - New Features


Seth Hall published two new features for the Intel Framework this month:
  • The ability to extend the Intel log by handling the new Intel::extend_match event.
  • The ability to whitelist items with a new intel item field named "whitelist". If you want to start whitelisting intel items at runtime, you should create a new intel file with an extra "meta.whitelist" field and set the field value to "T" (there is a test that shows this). As you add elements to this intel file, those items won't show up in your log file.
Note that the script renames the intel.log to intel-ext.log.

Bro Research


Summary Statistics Framework


Last month, we presented our research paper on the Bro Summary Statistics Framework at the International Symposium of Research in Attacks, Intrusions and Defenses (RAID) in Gothenburg, Sweden. The Summary Statistics Framework allows the easy calculation of a wide array of statistics in real-time, independent for the underlying data. It is, e.g., used in Bro to detect port scans and brute-force attacks. It has a wide array of applications, like finding top traffic sources in your network, getting lists of the top DNS requests, etc.
For more details please Read the paper.

Heartbleed Study


Another research paper that was recently released is a measurement study about the Heartbleed SSL bug.
Among other measurement methods, the paper uses Bro to examine pre- and post-exploit network traces of several research institutions. The study used the Bro SSL analyzer to detect Heartbleed traffic in those traces.
For more details please refer to the publication.

Bro On Demand


Your call for better/more documentation is heard.
This month we improved the script language reference
and the documentation on the default logs.
We are constantly working on further documentation improvements.
Please use the community channels to let us know what is still missing in the
documentation.

Bro In The Wild


BruCon 0x06 presents their findings reviewing their network
using Bro. For everyone who always wanted to know what weird.log is good for, we recommend this blog post.
They find a lot by analyzing weird.log.

NodeJS has started a nice project called nodejs that can help you to get more out of your Bro logs.
''The idea is to do processing events from BRO IDS in nodejs - this is a simple first step by parsing the bro log files
'online' and generate new events when any of the logs gets modified.''

Bro Plugins from the outside
Anthony Kasza (OpenDNS) wrote another Brolog entry, this time about his first experience porting a script to a plugin. We are always working on our documentation, but being a part of the Bro team sometimes conflicts with writing down the ''right'' things to help others using Bro.
If you have trouble getting started with Bro plugins, Anthony's approach from the outside might help you.

Note that plugins require the current development version of Bro.
There's also some initial documentation on our web site.

Bro Teaching Community


The Bro Teaching Meeting is moved from every Tuesday to every second Friday 10 AM PST, starting 10/10.
If you want to join the Teaching community to learn more about teaching (with) Bro and share your experiences, write to info@bro.org.
The next Teaching Meeting will then be at 11/07 due to vacation.

Tuesday, September 16, 2014

Bro Monthly #1

Bro Monthly

Welcome to the 1st Bro Monthly, our new monthly newsletter covering the latest developments in the Bro universe. This newsletter will appear every month, around the 15th, as a Bro blog post.
Please send feedback, wishes, and suggestions to info@bro.org or @Bro_IDS on Twitter.

Events


BroCon'14


BroCon'14  was held at NCSA from August 18th - 20th.
This year we received almost 150 attendees, our largest Bro event ever!

At this point we want thank again our sponsors:
Arista, Northrop Grumman, NSF, Reservoir Labs, and Security Onion Solutions.
A big thank you goes to NCSA who helped organizing the event.

We had great talks, presentations, and demos:
  • BroCon was opened by Adam Slagell, introducing the Bro Center of Expertise , an NSF project that enables a lot of new developments in the Bro universe,
    such as Bro Live! and Try.Bro (see below).
  • Nick Buraglio from ESnet talked about "Best practices for securing the science DMZ".
  • Bob Rotsted from Reservoir Labs discussed the "Value of context when detecting adversaries".
  • Johanna Amann from ICSI presented the new SSL analyzer in Bro 2.3 that is also capable of detecting the Heartbleed exploit.
  • Michael Pananen from Vigilant Technology Solutions showed how he automated Bro's installation, upgrade, and configuration using puppet.
  • Kurt Grutzmacher from Cisco Security Solutions presented OpenSOC, a Hadoop solution to extend Bro's ingestion capacity to 1.2 million packets per second and more.
  • Aashish Sharma gave some very entertaining insights into his day-to-day work fighting off attacks at LBNL.
  • Matthias Vallentin from ICSI introduced VAST (Visibility Across Space and Time), a large-scale network forensics platform.
  • Robin Sommer's (ICSI/LBNL/Broala) live demonstration of the new BinPAC++ parser generator was one of the most resonating contributions. He implemented a full protocol parser in less than half an hour in front of the audience.
  • To conclude the day Seth Hall (ICSI/LBNL/Broala) talked about the future of Bro, giving insights into long term and short term plans.
  • The third day was opened by Bob Bregant from the University of Illinois, who talked about how Arista's "DANZ" software can be used in combination with Bro to balance the costs when monitoring large high speed networks, working around problems arising from aggregation and traffic splitting.
  • The third day was wrapped up by a panel discussion in which the audience had the chance to pick the Bro team's brains about their visions for the Bro project.

Apart from the talks and demos we had five exercises ranging from beginner level to quite advanced scripting challenges. The exercises can be found at the event site of BroCon'14 . The solutions will be given out on demand. Please contact info@bro.org.

The videos of most of the BroCon'14  talks are now online. The Bro team respects the privacy preferences of our speakers, so when a speaker opted to not being recorded, we do not offer a video of the talk.

2014 NSF Cybersecurity Summit for Large Facilities and Cyberinfrastructure


The CTSC Summit was held in Arlington, VA on August 26th - 28th.

The Bro team presented a one-day training to a smaller group of attendees affiliated with NSF projects.  The training consisted of a couple exercises from BroCon '14 as well as some presentations.

Robin Sommer also gave an overview of the Bro Center of Expertise at the main event on August 27, 2014, in which he presented our latest efforts for making Bro more accessible to the community, and enable people and institutions to use Bro more effectively.

Bro Commits


Bro 2.3.1


Bro v2.3.1 has been released. This release addresses a potential DOS vector using specially crafted DNS packets. It also fixes a bug in the OCSP validation code that could lead to crashes as well as a memory leak. The source distribution and binary packages are available on our downloads page. See CHANGES for the full commit list. Since this release addresses a bug fix, we encourage users to review and install at their earliest convenience. Feedback is encouraged and should be sent to the Bro mailing list .

Bro's new dynamic plugin infrastructure


Any who has tried to add a new protocol analyzer to Bro will havenoticed that so far that has required touching a lot of pieces of Bro, as well as a complete rebuild of the Bro code base. We have just added a new comprehensive plugin infrastructure to Bro that makes this process much easier by allowing to write protocol analyzers externally, without *any* changes to the Bro core, by compiling them into a shared library that Bro will then load at runtime.  That way, the custom code remains self-contained, and can be maintained and installed independently.

This new infrastructure is in fact not limited to protocol analyzers, but supports other components of Bro as well. Developers can now use plugins also to provide custom file analyzers, log writers, input readers, packet sources and dumpers, as well as new built-in functions. For more information, see the introduction to writing Bro plugins.

Packet Bricks


We are happy to present an initial prototype of Packet Bricks, a new Bro-related project written by Asim Jamshed from KAIST, who visited the Bro team in Berkeley over the summer.

Packet Bricks - which is still under active development - is a Linux/FreeBSD daemon that is capable of receiving and distributing ingress traffic to userland applications. Its main responsibilities will eventually include (i) load-balancing, (ii) duplicating and/or (iii) filtering ingress traffic across all registered applications. The distribution is flow-aware (i.e., packets of one connection will always end up in the same application). Packet Bricks leverages the netmap packet I/O framework for handling packets efficiently, and employs netmap pipes to forward packets to userland applications.

Packet Bricks is available on github. It's still a very early piece of software, and we announce it at this time primarily for user's willing to help us collect some first experiences with it. If you have any feedback, please send it to the Bro development mailing list. If you aren't subscribed yet, you can do so here.

Bro On Demand




Bro Teaching and Trying


In August we launched three new projects that aim at helping the Bro community use, learn, and teach (with) Bro.

The Bro Teaching Community


We are happy to announce the newly started Bro Teaching Community, a community project of educators interested in collaboratively exploring Bro's use as a teaching tool, and sharing experiences and material. The goal is to create a knowledge base and resource collection for educators, ranging from example curricula and slide sets to exercises for all purposes and skills levels. We provide logistics and technical advice, e.g., weekly calls, a mailing list, a repository with seed material, and access to the Bro team. To learn more please visit our Teaching Site.

The Bro Playground


The Bro Playground  is a new part of the Bro Community resources.

It is a collection of tools and toys to assist you.

Whether you want to teach Bro, use Bro for teaching others, teach yourself, or try something out “quickly” without impacting your live system, this is the place to look for the right tool for your use case.

Try.Bro


Try.Bro - as simple as that!

Try.bro is a web-based Bro scripting sandbox made freely available to users on our site

No login.
No installation.
No trouble.

We have included a few basic scripts and pcaps to help get you started. You can paste your own scripts or upload your own pcaps, too. We even included the option of chosing different Bro versions to test your scripts against current or previous releases. And, last but not least, Try.bro temporarily caches your work and generates a unique URL to share with others. No more copying and pasting scripts or log files, just send the link. We store code fragments for three days and pcaps for one hour. The timeout is reset when the link is used.

To learn more please refer to the blog post.

Bro Live!

We are excited to announce the public release of Bro Live!

Bro Live! is a training system that gives users hands-on access to a Bro learning environment without having to download a virtual machine or its required dependencies.  Bro Live! may be built with exercises for a given class or workshop and access to the environment may be limited to the duration of the event. All the user needs is an SSH client with access to the Internet.

Please read our latest Blog post.

Cool Stuff


Exfil Framework by Reservoir Labs

Robert Rotsted from Reservoir Labs posted on the Bro mailing list about the new Exfil Framework.

"The Exfil Framework is a suite of Bro scripts that detect file uploads in TCP connections. The Exfil Framework can detect file uploads in most TCP sessions including sessions that have encrypted payloads (SCP,SFTP,HTTPS).
The scripts are located at:

https://github.com/reservoirlabs/bro-scripts/tree/master/exfil-detection-framework"

Monday, September 15, 2014

Announcing Bro Live!

We are excited to announce the public release of Bro Live!

Bro Live! is a training system that gives users hands-on access to a Bro learning environment without having to download a virtual machine or its required dependencies.  Bro Live! may be built with exercises for a given class or workshop and access to the environment may be limited to the duration of the event. All the user needs is an SSH client with access to the Internet.

The inspiration for Bro Live! came from the frequent frustrations we faced when building and distributing virtual machines for our training events: files are too large, the variety of user platforms and virtualization products complicate the troubleshooting process, pushing last minute updates are difficult, too much time spent not learning Bro, etc.  Bro Live! overcomes all of these limitations by putting the responsibility back in the hands of the system engineers. Users simply open a terminal, connect to the server via ssh, create an account, and they are up and running.

How Does Bro Live! work? Bro Live! was built using Linux, OpenSSH, Docker, and Bash to glue everything together. It relies on Linux-based containers, managed by Docker, to place users in their own isolated environments with shell access to Bro, the exercises, and the standard Unix toolset. The user's work is saved in their container typically for the duration of the training event and can be easily re-attached at anytime during the event to continue their work.

Want to host your own Bro training event with a system like this? See our work in Github for more information.  As always we welcome your feedback and bug reports, contact us at info@bro.org, follow us on Twitter, or like us on Facebook.

Thursday, September 11, 2014

Bro and Chrome's Sunsetting of SHA-1

A few days ago, Google announced their plans for sunsetting certificates using the SHA-1 hash algorithm in the near future. Google does not think SHA-1 certificates should be considered secure in the future anymore as collision attacks against SHA-1 get more realistic.

For these reasons, they will start reducing the security indicators if the certificate chain uses SHA-1 and the host-certificate is valid beyond the end of 2015. Depending on the exact validity in the certificate, the visual display will first change to a symbol showing slight errors in the HTTPS connection to symbols showing the connection to be not secure.

Chrome will start reducing the security indications for such certificates at the end of September 2014.

We created a Bro script that can detect servers in your local network that contain such certificates.

To use the script, simply download it from https://github.com/0xxon/bro-scripts/blob/master/chrome-sha1.bro and either load it in your local.bro (when using broctl) or add it to the Bro command line when starting it manually. By default the script only checks servers that are in your Site::local_nets list.

The script will output notices for each server containing such certificates to notice.log. An example output is included in the git repository.

Please let us know any issues you encounter when using this script.

Monday, September 8, 2014

Bro 2.3.1 Release

Bro v2.3.1 has been released.  This release addresses a potential DOS vector using specially crafted DNS packets.  It also fixes a bug in the OCSP validation code that could lead to crashes as well as a memory leak.  The source distribution and binary packages are available on our downloads page.

See CHANGES for the full commit list.

Since this release addresses a bug fix, we encourage users to review and install at their earliest convenience.

Feedback is encouraged and should be sent to the Bro mailing list.


The Bro Team

Friday, September 5, 2014

Announcing Try.bro

We are very excited to announce the official launch of Try.bro.org!



Try.bro is a web-based scripting sandbox made freely available to users on our site.  No login.  No installation.  No trouble.

We have included a few basic scripts and pcaps to help get you started.  You can paste your own scripts or upload your own pcaps, too.  We even included the option of version control to test your scripts against current or previous versions of Bro.  And, last but not least, Try.bro temporarily caches your work and generates a unique URL to share with others.  No more copying and pasting scripts or log files, just send the link.  We store code fragments for three days and pcaps for one hour.  The timeout is reset when the link is used.

How does Try.bro work?  Mainly we use Docker to run Bro in an isolated container.

In the few weeks we have been beta testing Try.bro it has become an extremely useful tool for collaboration, troubleshooting problems with users, and more.

We have additional features planned for Try.bro, including making it embeddable and building scripted tutorials around it for guided learning.

If you find a cool use for Try.bro or want to request a feature/report a bug, please let us know.

Contact us at info@bro.org, follow us on Twitter, or like us on Facebook.

Thursday, August 7, 2014

Meet the Bro Teaching Community

We are happy to announce the newly started Bro Teaching Community, a community project of educators interested in collaboratively exploring Bro's use as a teaching tool, and sharing experiences and material. The goal is to create a knowledge base and resource collection for educators, ranging from example curricula and slide sets to exercises for all purposes and skills levels.

We invite you to participate in our open discussion every Tuesday at 10:00 AM PDT. In these meetings we discuss planned curricula, practical and technical topics around exercises, slide sets, and general questions related to teaching security, networks and systems with Bro.

For details see www.bro.org/teaching/ or contact us directly via info<at>bro.org

Tuesday, June 17, 2014

Bro 2.3 Release

We are happy to announce the release of Bro v2.3.  The source distribution and binary packages are available on our downloads page.  For a brief overview of new features and bug fixes you may review our previous blog post about the v2.3 beta.

See NEWS for the preliminary release notes and CHANGES for the exhaustive commit list.

Feedback is encouraged and should be sent to the Bro mailing list.

We extend sincere thanks to all who have helped make this release possible, especially those members of the community who have given us their feedback and support.



The Bro Team

Friday, May 23, 2014

Bro 2.3 Public Beta

We are happy to announce the public beta of Bro v2.3 is available for download! The majority of our development time has been focused on improving performance, reliability, and memory use.

Here is a brief summary of the new features and improvements:
  • Support for GRE tunnel decapsulation 
  • Heartbleed detector on the SSL analyzer 
  • Analyzers for SNMP and Radius 
  • BroControl now supports PF_RING+DNA 
  • File type detection via Bro’s own signature engine instead of external libraries 

See NEWS for the preliminary release notes and CHANGES for the exhaustive commit list.

Feedback is encouraged and should be sent to the Bro mailing list. As we have stated in the past, we do not recommend using a beta release for production use.

Tuesday, April 8, 2014

Detecting the heartbleed bug using Bro

Update: The heartbleed detector is now part of bro master. You should switch if you are still using the development branch. You still have to load policy/protocols/ssl/heartbleed.bro

We just added support to Bro to detect the recent heartbleed attack on TLS servers that are using OpenSSL 1.0.1a-f.

In TLS the payload size of a heartbeat packet and the size of the whole packet is specified in two different places. The heartbleed attack exploits the fact - vulnerable OpenSSL versions return random bits of server memory, when the request packet specified that the payload size is bigger then the size of the whole data packet.

Bro can detect this attack in several different ways. In the simplest incarnation, which is the only one we have seen in the wild so far, the heartbeat message is sent very early, before the TLS encryption kicks in.

In these cases, Bro just compares the payload and message sizes. If there is a mismatch, we know that an exploit has been tried. If the server responds to the message it very probably was vulnerable to the attack.

In theory, the attack also can take place after the TLS encryption started. In this case, we only know the message size, the payload size in the request is encrypted.

In these cases, we use several different heuristics to deduce when an attack is taking place. We count the numbers of heartbeats sent by the client and the server. If there is a divergence of more than a few packets, an attack is likely.

We also check if TLS heartbeat messages are smaller then the minimal length they are required to have, which probably also is only the case in an attack. Furthermore, we check if the encrypted heartbeat packets returned by the server have the same size as the packets sent by the client. If they diverge, the server is returning more data than the client sent in the first place -- which only can happen due to this attack.

To enable the heartbeat detection, you have to load the policy/protocols/ssl/heartbleed script. If you use broctl, it will be loaded by default in a new installation using this branch. If using bro on the command line, e.g. to read a trace, you have to specify it directly like this:

bro -r [trace] policy/protocols/ssl/heartbleed

The alerts will be written into notice.log.

Wednesday, March 5, 2014

Dissecting the GnuTLS Bug

Update: we now host a test server at gnutls.notary.icsi.berkeley.edu. See gnutls command lines below.

The recent  GnuTLS certificate verification bug made it possible to craft an arbitrary certificate in a way that GnuTLS would validate correctly against a given CA root certificate store.

Since we started our SSL Notary service about a year ago, whenever a big SSL or TLS security incident happens, we scan all the certificates we have seen so far to see if they could have been used for an attack. For this bug, we dived into the GnuTLS source, examined the conditions that trigger the bug, and figured how an attacker would exploit the bug. Based on our analysis, we believe that there exist multiple methods to trigger the bug. In the following, we describe one that worked.

Looking at the patch that fixes this bug, we observe that the return codes for error cases were not set properly in a few examples. For example, the function _gnutls_verify_certificate2 performs the actual certificate verification starting at line 465. GnuTLS first attempts to extract the part of the certificate that is signed, the signature, and the signature algorithm. If any of these lookups fail, the verification still succeeds due to missing adjustments of the variable result, which denotes the function return value. On success, the function should return a non-zero value, but the code lacked checks to set the value to zero.

From an attacker perspective, making the signature lookup fail is probably the easiest method to exploit the bug. According to the function _gnutls_x509_get_signature, the signature verification fails if the number of bits in the certificate signature is not divisible by 8. Certificate signatures have an ASN.1 bit string encoding, which theoretically may exhibit an arbitrary number of zeros and ones, and may not necessarily be divisible by 8. But the GnuTLS function assumes a byte-aligned signature, hence the check. In the ASN.1 DER encoding, used in X509 certificates, a bit string begins with a single byte which denotes the number of missing bits in the last byte. For all normal certificates, this value is always zero, and the function continues. But if we change this byte to a non-zero value, the function fails the divisibility check and returns with wrong error code, causing the certificate verification to succeed.

Moreover, we also looked at a few other entry points with the goal to make the signature algorithm verification fail. However, none of them seemed to be as easy to exploit as the signature algorithm.

To trigger the bug, we patched the certificate for bro.org, exchanged the private key, and tried to validate it with GnuTLS. Indeed, the verification succeeds with gnutls-certtool and we could establish a secure connection to the server using gnutls-cli.

You can test the vulnerability of your GnuTLS installation by trying to connect to our test server:

    gnutls-cli -p 443 gnutls.notary.icsi.berkeley.edu --x509cafile [root store]

You also can test it by downloading the following certificate chain and verifying it using

    gnutls-certtool --verify --infile exploit.pem --load-ca-certificate [root store]

Usually, you can find your operating system root store in /etc/ssl, or you can use our root-store copy. Note that the output of gnutls-certtool is slightly truncated when using a chain that exploits the bug, nevertheless certificate verification (including hostname verification) works.

At the moment we are scanning all the certificates contained in the ICSI SSL Notary to see if there is any certificate that specifies a non 8-bit-divisible signature. Due to the size of our data set - at the moment it consists or more than 1.8 million certificates extracted from more than 50 billion connections - the scan is still running; so far there have been no hits.

Thursday, January 23, 2014

Intelligence Data and Bro

Overview

Intelligence data, or feeds, are an important source of network security information. Many internet security research centers, non-profit organizations, and commercial organizations provide intellegence data sets freely available to the public. (e.g. Emerging Threats, Shadow Server, etc.)

A solid solution for handling multiple intelligence feeds and acting upon them is to use Bro's Intel Framework in conjunction with its Input Framework to log hits seen on your network.

Hits, or matches, are logged and stored in intel.log.

$ head /bro/logs/current/intel.log
#fields ts      uid     id.orig_h       id.orig_p       id.resp_h       id.resp_p       fuid    file_mime_type  file_desc       seen.indicator  seen.indicator_type     seen.where      sources
#types  time    string  addr    port    addr    port    string  string  string  string  enum    enum    table[string]
1389646914.469513       Cb1T1Y3r56rjSRr0ac      70.173.122.62   49379   93.184.215.163  80      -       -       -       93.184.215.163  Intel::ADDR     Conn::IN_RESP   snort
1389646925.095172       CeHuLr2IfATgjhQPUg      195.178.109.14  43471   227.146.143.88  5900    -       -       -       195.178.109.14  Intel::ADDR     Conn::IN_ORIG   CIF - need-to-know
1389646979.298695       Cj9bjq43KThmM8TOA6      194.160.95.196  60430   119.254.16.21   80      -       -       -       www.dhgate.com  Intel::DOMAIN   HTTP::IN_HOST_HEADER    CIF - need-to-know
1389636016.229788       CNXlzu1VdMhKVEDCPf      42.82.115.14    52518   184.72.106.52   443     -       -       -       184.72.106.52   Intel::ADDR     Conn::IN_RESP   tor
1389636187.690988       CsGYCjrhwI313rWob       72.49.141.144   41270   118.219.221.39  3389      -       -       -     72.49.141.144   Intel::ADDR     Conn::IN_ORIG   ciarmy
...

Each log entry consists of a number of fields describing the connection:

ts:Timestamp
uid:Unique ID of connection
id.orig_h:Connection originator's endpoint IP address
id.orig_p:Connection originator's endpoint TCP/UDP or ICMP code
id.resp_h:Connection responder's endpoint IP address
id.resp_p:Connection responder's endpoint TCP/UDP or ICMP code
fuid:File indentifier (if file was found in connection)
file_mime_type:Libmagic file type (e.g. application/x-dosexec)
file_desc:Optional file type description
seen.indicator:The indicator that triggered the match (e.g. IP)
seen.indicator_type:
 The type of indicator (e.g. ADDR, DOMAIN)
seen.where:Location in Bro's where the event triggered (e.g. DNS::REQUEST)
sources:Intel data source description (e.g. emergingthreats.net)

Intel Framework

The Intel Framework provides the facilities to handle intelligence data in a meaningful manner (e.g. categorization, types, source, etc.). [1]

Supported types of intelligence data indicators are:

Intel::ADDR
Intel::URL
Intel::SOFTWARE
Intel::EMAIL
Intel::DOMAIN
Intel::USER_NAME
Intel::FILE_HASH
Intel::FILE_NAME
Intel::CERT_HASH

Before we can use the framework we must load it by adding the following lines to local.bro or equivalent:

@load frameworks/intel/seen
@load frameworks/intel/do_notice

The next step is to properly format the data set. Each field must be separated by a single tab character.

#fields indicator       indicator_type  meta.source     meta.url        meta.do_notice  meta.if_in
1.182.117.119   Intel::ADDR     ciarmy  http://www.ciarmy.com/list/ci-badguys.txt       T       -
2.187.28.215    Intel::ADDR     ciarmy  http://www.ciarmy.com/list/ci-badguys.txt       T       -
2.229.117.159   Intel::ADDR     ciarmy  http://www.ciarmy.com/list/ci-badguys.txt       T       -
4.35.96.216     Intel::ADDR     ciarmy  http://www.ciarmy.com/list/ci-badguys.txt       T       -
5.79.69.204     Intel::ADDR     ciarmy  http://www.ciarmy.com/list/ci-badguys.txt       T       -
5.135.146.0     Intel::ADDR     ciarmy  http://www.ciarmy.com/list/ci-badguys.txt       T       -
5.135.240.133   Intel::ADDR     ciarmy  http://www.ciarmy.com/list/ci-badguys.txt       T       -
5.153.54.130    Intel::ADDR     ciarmy  http://www.ciarmy.com/list/ci-badguys.txt       T       -
5.254.101.69    Intel::ADDR     ciarmy  http://www.ciarmy.com/list/ci-badguys.txt       T       -

Also, each field has a particular type of value. Two worthy of mention are meta.do_notice and meta.if_in because they work with the Notice Framework. meta.do_notice expects a boolean value which determines whether matches will be sent to the Notice Framework where entries will end up in notice.log, in addition to the normal intel.log.

$ grep Intel notice.log
1389636016.229788       CNXlzu1VdMhKVEDCPf      23.58.90.219  52518   184.72.106.52   443     -       -       -       tcp     Intel::Notice   Intel hit on 184.72.106.52 at Conn::IN_RESP     184.72.106.52   23.58.90.219  184.72.106.52   443
1389636016.517027       Cr1u7g2pFxgSzHBsRf      234.136.242.173 52519   184.72.106.52   443     -       -       -       tcp     Intel::Notice   Intel hit on 184.72.106.52 at Conn::IN_RESP     184.72.106.52  234.136.242.173 184.72.106.52   443
...

meta.if_in concerns itself with where a particular match was found in network traffic (e.g. HTTP::IN_HOST_HEADER) and serves as a restriction for matches sent to the Notice Framework i.e. they must be found in the location of meta.if_in.

Note: meta.if_in takes a single location value. If you would to like to specify more than one location for an indicator you will need an entry for each location you desire to be seen. e.g.

#fields indicator       indicator_type  meta.source     meta.url        meta.do_notice  meta.if_in
aolon1ine.com   Intel::DOMAIN   mandiant        -       T       DNS::IN_REQUEST
aolon1ine.com   Intel::DOMAIN   mandiant        -       T       HTTP::IN_HOST_HEADER
aolon1ine.com   Intel::DOMAIN   mandiant        -       T       SMTP::IN_FROM

The list of possible meta.if_in locations are: [2]

Conn::IN_ORIG
Conn::IN_RESP
Files::IN_HASH
Files::IN_NAME
DNS::IN_REQUEST
DNS::IN_RESPONSE
HTTP::IN_HOST_HEADER
HTTP::IN_REFERRER_HEADER
HTTP::IN_USER_AGENT_HEADER
HTTP::IN_X_FORWARDED_FOR_HEADER
HTTP::IN_URL
SMTP::IN_MAIL_FROM
SMTP::IN_RCPT_TO
SMTP::IN_FROM
SMTP::IN_TO
SMTP::IN_RECEIVED_HEADER
SMTP::IN_REPLY_TO
SMTP::IN_X_ORIGINATING_IP_HEADER
SMTP::IN_MESSAGE
SSL::IN_SERVER_CERT
SSL::IN_CLIENT_CERT
SSL::IN_SERVER_NAME
SMTP::IN_HEADER

Input Framework

The Input Framework [3] provides the facilities to read information from a text file, and in our case send it to the Intel Framework for processing.

Note: If you're running a Bro cluster, the intelligence files only need to be stored and read on the manager.

An absolute file path is required to read from a file. Three ways to do this are:

  1. Specify the file's full path

  2. Append the file name to @DIR which stores the directory path of the calling script. If the script with Intel::read_files is not in the same directory as the intel data sets this will not work.

  3. Create a constant that's value is the directory path in local.bro or equivalent.

    const feed_directory = "/opt/bro/feeds";
    

Here's an example using all three methods:

redef Intel::read_files += {
        @DIR + "/botcc.intel",
        @DIR + "/ciarmy.intel",
        "/opt/bro/feeds/malhosts.intel",
        "/opt/bro/feeds/malips.intel",
        feed_directory + "/tor.intel",
        feed_directory + "/snort.intel",
};

Obtaining Feeds

Two solutions for obtaining pre-formatted feeds for Bro to use are mal-dns2bro, a helper script for mal-dnssearch, or the Collective Intelligence Framework and its Bro output plug-in.

Mal-dns2bro

Mal-dnssearch [4] is a shell script I wrote that downloads, parses, and compares intelligence feeds against a number of popular application log files, reporting any matches. mal-dns2bro [5] is a helper script included with mal-dnssearch that formats feeds for Bro's Intel Framework to extend the application of intelligence data directly against live network traffic. mal-dns2bro has the ability to customize Intel Framework fields like setting meta.source, meta.url, meta.do_notice, and meta.if_in.

mal-dnssearch supports a number of feeds, ``mal-dnssearch -h'':

Malware List Options:
        -M <list>               Name of list, e.g. \`\`-M snort\'\'

        List:      |     Description:
        snort      -     http://labs.snort.org/feeds/ip-filter.blf (IP)
        et_ips     -     http://rules.emergingthreats.net/open/suricata/rules/compromised-ips.txt (IP)
        alienvault -     http://reputation.alienvault.com/reputation.generic (BIG file) (IP)
        botcc      -     http://rules.emergingthreats.net/open/suricata/rules/botcc.rules (IP)
        tor        -     http://rules.emergingthreats.net/open/suricata/rules/tor.rules (IP)
        rbn        -     http://rules.emergingthreats.net/blockrules/emerging-rbn.rules (IP)
        malhosts   -     http://www.malwaredomainlist.com/hostslist/hosts.txt (DNS)
        malips     -     http://www.malwaredomainlist.com/hostslist/ip.txt (IP)
        ciarmy     -     http://www.ciarmy.com/list/ci-badguys.txt (IP)
        mayhemic   -     http://secure.mayhemiclabs.com/malhosts/malhosts.txt (DNS)
        mandiant   -     https://raw.github.com/jonschipp/mal-dnssearch/master/mandiant_apt1.dns (DNS)

Download and install mal-dnssearch:

$ git clone https://github.com/jonschipp/mal-dnssearch
$ cd mal-dnssearch
$ sudo make install

Download Mandiant's APT1 data set, parse, and pipe (``-p'') it to mal-dns2bro for formatting:

$ mal-dnssearch -M mandiant -p | mal-dns2bro -T dns -s mandiant > mandiant.intel

Sample file output:

#fields indicator       indicator_type  meta.source     meta.url        meta.do_notice  meta.if_in
advanbusiness.com       Intel::DOMAIN   mandiant        -       F       -
aoldaily.com    Intel::DOMAIN   mandiant        -       F       -
aolon1ine.com   Intel::DOMAIN   mandiant        -       F       -
applesoftupdate.com     Intel::DOMAIN   mandiant        -       F       -

In the example above, mal-dns2bro reads in the mandiant list from stdin and sets the indicator type (``-T'') to DNS because the mandiant list consists of only DNS names. The source (``-s'') field is also set which is a short description of where the intelligence data came from.

mal-dns2bro will add the necessary tab separated columns for the Intel Framework. It accepts a list of a specific indicator type, but supports all of them, with one entry per line. It can read from stdin or from a file (``-f''). If you don't want to use mal-dnssearch, you can create your own lists with a text editor or other program and have mal-dns2bro format them for Bro.

Another example, downloading a list of tor nodes and customizing all fields:

$ mal-dnssearch -M tor -p | mal-dns2bro -T ip -s tor -n true -u http://rules.emergingthreats.net/open/suricata/rules/tor.rules > tor.intel

Sample file output:

#fields indicator       indicator_type  meta.source     meta.url        meta.do_notice  meta.if_in
103.10.197.50   Intel::ADDR     tor     http://rules.emergingthreats.net/open/suricata/rules/tor.rules  T       Conn::IN_RESP
103.3.188.169   Intel::ADDR     tor     http://rules.emergingthreats.net/open/suricata/rules/tor.rules  T       Conn::IN_RESP
105.237.43.166  Intel::ADDR     tor     http://rules.emergingthreats.net/open/suricata/rules/tor.rules  T       Conn::IN_RESP
106.186.21.31   Intel::ADDR     tor     http://rules.emergingthreats.net/open/suricata/rules/tor.rules  T       Conn::IN_RESP

In this last example, two new options are introduced: mal-dns2bro sets meta.do.notice to true (``-n'') which will send intel matches to the notice framework: the source URL, when applicable, is set (``-u'').

Collective Intelligence Framework

The Collective Intelligence Framework (CIF) is a cyber threat intelligence management system that pulls and stores feeds into a database for querying with CIF's tools. [6] CIF comes with a number of output plug-ins including one for Bro.

Installing CIF is pretty involved so I will refer the reader to its documentation. [7]

The CIF plug-in for Bro outputs a few different fields than we've seen so far: meta.cif_impact, meta.cif_severity, and meta,cif_confidence. Because of this we will need to load a Bro script, in addition to those required for the Intel Framework, that can handle these.

Add the following line to local.bro or equivalent:

@load policy/integration/collective-intel

Now, lets output domains related to botnets with a confidence level of 85 or greater formatted for Bro.

$ cif -q domain/botnet -c 85 -p bro > domain_botnet.intel

Sample file output:

#fields indicator       indicator_type  meta.source     meta.desc       meta.url        meta.cif_impact meta.cif_severity       meta.cif_confidence
computo164.laweb.es     Intel::DOMAIN   CIF - need-to-know      palevo  https://palevotracker.abuse.ch/?host=computo164.laweb.es (public)       -       high    85
ms4all.twoplayers.net   Intel::DOMAIN   CIF - need-to-know      palevo  https://palevotracker.abuse.ch/?host=ms4all.twoplayers.net (public)     -       high    85
hcuewgbbnfdu1ew.com     Intel::DOMAIN   CIF - need-to-know      palevo  https://palevotracker.abuse.ch/?host=hcuewgbbnfdu1ew.com (public)       -       high    85
arta.romail3arnest.info Intel::DOMAIN   CIF - need-to-know      palevo  https://palevotracker.abuse.ch/?host=arta.romail3arnest.info (public)   -       high    85

Create a list of machines known for scanning the internet:

$ cif -q infrastructure/scan -c 85 -p bro > intrastructure_scan.intel

Sample file output:

#fields indicator       indicator_type  meta.source     meta.desc       meta.url        meta.cif_impact meta.cif_severity       meta.cif_confidence
192.74.251.115  Intel::ADDR     CIF - need-to-know      ssh     http://danger.rulez.sk/projects/bruteforceblocker/blist.php (public)    -       medium  85
117.41.247.212  Intel::ADDR     CIF - need-to-know      ssh     http://danger.rulez.sk/projects/bruteforceblocker/blist.php (public)    -       medium  85
133.242.171.75  Intel::ADDR     CIF - need-to-know      ssh     http://danger.rulez.sk/projects/bruteforceblocker/blist.php (public)    -       medium  85

E-mailing Notices

This section presumes that Intel events are sent to the Notice Framework [8], described in the Intel section of this article.

Each time a Intel::Match event is generated, the intel data for that match is sent to the Notice Framework where a Notice is raised that has a type of Intel::Notice. To receive e-mail notifications upon a match add the following Notice type to emailed_types in local.bro or equivalent:

redef Notice::emailed_types += {
        Intel::Notice,
};

Suppression can be used to reduce the number of events generated for a specific Intel match. To limit each particular match to once per day add the following lines to local.bro or equivalent:

redef Notice::type_suppression_intervals += {
        [Intel::Notice] = 1day,
};

Log Queries

Bro-cut can be used to print fields in Bro logs. The following example will print a list of indicators only i.e. the host, hash, e-mail, etc. that triggered the match.

$ bro-cut seen.indicator < intel.log
61.36.24.57
193.107.16.206
192.0.72.2
...

To print a unique list of all ADDR types and the number of times they were seen try the following query. If you count be the number of fields 11 is seen.indicator_type and field 10 is seen.indicator:

$ awk '$11 == "Intel::ADDR" { print $10 }' intel.log | sort -t . -n -k 1,1 -k 2,2 -k 3,3 -k 4,4 | uniq -c
190 61.36.24.57
1 93.120.27.62
3 93.184.215.163
1 96.45.82.69
7 192.0.72.2
2 193.107.16.206

The next example illustrates an intel match found in the host field of the HTTP header.

1390425478.178039       CQzyg512Q8Mm2UrTc4       53.80.23.123   51220   65.52.128.33    80      -       -       -       tipranks.com    Intel::DOMAIN   HTTP::IN_HOST_HEADER    malhosts

Since the above connection is using HTTP there are bound to be more logs related to the connection such as those produced by the HTTP analyzer. We can search all Bro's logs for the connection identifier, a 4-tuple flow hash, to find other logs related to that connection.

$ zgrep CQzyg512Q8Mm2UrTc4 *.log.gz
1390425478.076356 CQzyg512Q8Mm2UrTc4      53.80.23.123 51220   65.52.128.33    80      tcp     http    123.242140      397     981     RSTR    T       0       ShADadr 12      1302    12      2514    (empty) US      US      nids-31-2
1390425478.398364        FQSaaGSHQsLn11w8j       65.52.128.33    53.80.23.123 CQzyg512Q8Mm2UrTc4      HTTP    0       SHA1,MD5        text/plain      -       0.000000        F       F       595     -       0       0       F       -       d6923c591dfa3cb616327a0bb44375b3        aa5438c500083ee69297d8c1a2ab4f82655c4632      -       -
1390425478.179990        CQzyg512Q8Mm2UrTc4      53.80.23.123 51220   65.52.128.33    80      1       GET     tipranks.com    /valuewalk/tipranks.js  http://www.valuewalk.com/2014/01/advanced-micro-devices-inc-amd-shares-dive-after-earnings/     Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.76 Safari/537.36  0       595     200     OK      -       -       -       (empty) -       -       -       -       -       FQSaaGSHQsLn11w8j       text/plain
1390425478.178039        CQzyg512Q8Mm2UrTc4      53.80.23.123 51220   65.52.128.33    80      -       -       -       65.52.128.33    Intel::ADDR     Conn::IN_RESP   malips
1390425478.178039        CQzyg512Q8Mm2UrTc4      53.80.23.123 51220   65.52.128.33    80      -       -       -       tipranks.com    Intel::DOMAIN   HTTP::IN_HOST_HEADER    malhosts

We found a number of events detailing the connection, these include the URL the user visited, the type of file that was transferred in the session, and another intel match from a different intel source. This is especially useful in cases where there are more than one HTTP request in a single TCP session.

Putting it all together

A powerful but simple solution is to write a daily cronjob that downloads and formats the latest version of each feed from which Bro continuously reads and then restarts Bro, if necessary. A restart is required if you want to purge entries that have been removed from the feeds, but not if you only want the new entries because Bro keeps the file open and will pick up any new additions.

If you come across any trouble be sure to check reporter.log.

References:

[1]http://www.bro.org/sphinx-git/frameworks/intel.html
[2]http://www.bro.org/sphinx-git/_downloads/where-locations.bro
[3]http://www.bro.org/sphinx/frameworks/input.html
[4]https://github.com/jonschipp/mal-dnssearch
[5]https://github.com/jonschipp/mal-dnssearch/blob/master/tools/mal-dns2bro.sh
[6]https://code.google.com/p/collective-intelligence-framework/
[7]https://code.google.com/p/collective-intelligence-framework/wiki/ServerInstall_v1
[8]http://www.bro.org/sphinx-git/frameworks/notice.html