Thursday, October 11, 2018

Renaming the Bro Project

More than 20 years ago I chose the name "Bro" as "an Orwellian reminder that monitoring comes hand in hand with the potential for privacy violations", as the original Bro paper put it. Today that warning is needed more than ever ... but it's clear that now the name "Bro" is alas much more of a distraction than a reminder.

On the Leadership Team of the Bro Project, we heard clear concerns from the Bro community that the name "Bro" has taken on strongly negative connotations, such as "Bro culture". These send a sharp, anti-inclusive - and wholly unintended and undesirable - message to those who might use Bro. The problems were significant enough that during BroCon community sessions, several people have mentioned substantial difficulties in getting their upper management to even consider using open-source software with such a seemingly ill-chosen, off-putting name.

Accordingly, in 2017 the Leadership Team undertook to find a new name for the project. We solicited suggestions for new names from the Bro community, receiving several hundred that covered a wide range of sensibilities and rationales. The LT extensively discussed the candidates internally but was unable to come close to a consensus on a satisfactory choice. Names that some LT members quite liked, others found quite deficient. This process proved particularly hard because some well-liked names had trademark issues and such.

Given that impasse, the LT engaged with a professional naming/branding consultancy to identify other possible names. The process elicited reflection on just what we would like the name to convey, which included notions of insight/visibility, soundness, flexibility, and Bro's heritage.

As the process proceeded, a number of LT members identified their fondness for quirky, pithy names for open-source projects. One name in particular dates all the way back to the very beginning of Bro in 1995 at the Lawrence Berkeley National Laboratory: Zeek. At LBL, the production Bro monitoring ran as a pseudo-user named "zeek" - this included both running the Bro process itself, and also the batch jobs and parallel tcpdump captures used to ensure robust 24x7 operation - a usage that continued for decades.

Why Zeek? The name was inspired by Gary Larson's use of Zeek characters in various "The Far Side" cartoons. We were big Far Side fans at LBL!

As Bro's originator, I have to say that I find switching the system's name to Zeek not only timely, but - in its quirkiness and history - endearing. I am thrilled that the Leadership Team identified and subsequently strongly backed the choice. (It was great, too, to find that we could secure the domain zeek.org.)

The name "Bro" offered numerous opportunities for modest-but-memorable wordplay, such as referring to the project's "Broadmap", beginning conferences with a "Broverview", and coining the term "brogrammer". We've only begun exploring the possibilities with Zeek. However, the speed with which we readily found our first slogan holds promise in this regard. In looking for a new name, we had particularly identified wanting to find one that underscores the system's ability to provide deep insight into network traffic. We put that goal aside for our final selection. But shortly after we settled on our choice, a project member offered:

Zeek, and ye shall find!

Thursday, July 19, 2018

Broker is Coming, Part 2: Replacing &synchronized


As a quick followup to Part1, I want to bring attention to the Reminder about Events and Module Namespaces.  This is something that I'd forgotten about (or never knew) that will save you a lot of headaches when converting scripts to use Broker.  While I'm at it, when testing data stores, it's important to keep in mind any data expiration times you have set.  More than once I got back to testing the next morning and wondered where my persistent data had gone only to eventually realize it had just been correctly expired. 

In Part 1, I discussed using Broker to interact with persistent data stores, so now I want to go over a couple options for synchronizing data across your Bro cluster. We'll also look at how to debug Broker communication to verify things are working as you expected. As a reminder, below I've included a basic example of the old method of using &sychronized, which is now depreciated.


Publish Data to the Manager


There are several common use cases for Broker communication and it comes down to where your data needs to be for your policy to work as expected. Do the workers need access to the full view of the data or can the proxies or manager make the decisions? The first case we'll examine is using Broker to publish the addresses to the manager, check out the &sychronized example near the bottom to see the script we started from.


Broker uses a pub/sub model in combination with events to communicate between nodes so we need to send a message to the recipient, but also define the event that will be processed as a result. If you aren't familiar with pub/sub and the concept of topics, here is an overview.

@load base/frameworks/cluster module EX; export { global known_clients: set[addr] &create_expire=1day &synchronized; global add_client: event(newaddr: addr); } event add_client(newaddr: addr){ if(newaddr !in known_clients){ add known_clients[newaddr]; } } event connection_established(c: connection){ if(c$id$resp_h == 192.150.187.43){ Broker::publish(Cluster::manager_topic,EX::add_client, c$id$orig_h); } }

The event for adding the address to the known_clients set will be handled by the manager, so we moved the check for existence in the set to just before adding the client data. The event definition is exported for global use. Then in the connection_established event, we replace the add to the set with our Broker::publish call. We need to specify the pub/sub topic, but Bro pre-defines Cluster::manager_topic for us already and the manager is automatically subscribed to it. The only other arguments are the event we want the manager to execute, along with any arguments. Don't forget to remove &synchronized! Re-deploying the new script on the cluster, let's see the result:


[bro@host ~]$ broctl deploy
[bro@host ~]$ wget -q https://www.bro.org
[bro@host ~]$ broctl print EX::known_clients
      logger   EX::known_clients = {
}
     manager   EX::known_clients = {
198.128.111.111
}
     proxy-1   EX::known_clients = {
}
     proxy-2   EX::known_clients = {
}
    worker-1   EX::known_clients = {
}
    worker-2   EX::known_clients = {
}
    worker-3   EX::known_clients = {
}
    worker-4   EX::known_clients = {
}

This method of offloading work to the manager has pretty clear tradeoffs. The workers aren't all storing copies of the set in memory and they also don't have to do any additional processing work. However, you're putting that load on the manager. There may be a better way, which we'll look at when we discuss distributing work among the proxies.

Making it Work in Standalone Mode

If you do any testing by running Bro in a standalone mode, or would like your policies to support people who do, you'll need to do a little more work to make your script do both.

event connection_established(c: connection){
      if(c$id$resp_h == 192.150.187.43){
            @if( Cluster::is_enabled())
Broker::publish(Cluster::manager_topic,EX::add_client,
    c$id$orig_h);
@else
event EX::add_client(c$id$orig_h);
@endif
      }

}

Sometimes you'll see additional checks for which kind of cluster node is processing the event, but that's an added layer of complication we won't discuss right now. Example: @if ( ! Cluster::is_enabled() || Cluster::local_node_type() == Cluster::MANAGER ) For the rest of these examples I will skip the standalone code to make things simpler.

Distribute Work Among Proxies


Another option when the workers don't need access to the data is to distribute the processing amongst the proxy nodes. We could publish the data as we did with the manager, but as there may be more than one proxy, we don't want them to multiply the effort. Conveniently, Bro provides a "highest random weight" hashing algorithm to spread the load evenly. All we have to do is change the Broker::publish line:

event connection_established(c: connection){ if(c$id$resp_h == 192.150.187.43){ Cluster::publish_hrw(Cluster::proxy_pool, c$id$orig_h, EX::add_client, c$id$orig_h); } }

The Cluster::publish_hrw function distributes events across a set of nodes, here we're using the proxy pool. The second argument is the value that will be hashed to determine the distribution. Finally, we give it the event to send along with any arguments.

[bro@host ~]$ broctl deploy [bro@host ~]$ wget -q https://www.bro.org [bro@host ~]$ broctl print EX::known_clients logger EX::known_clients = { } manager EX::known_clients = { } proxy-1 EX::known_clients = { } proxy-2 EX::known_clients = { 198.128.111.111 } worker-1 EX::known_clients = { } worker-2 EX::known_clients = { } worker-3 EX::known_clients = { } worker-4 EX::known_clients = { }

Note that it only wrote to one of the proxies as expected.  If you test the policy from multiple clients you should end up with a fairly even distribution across proxies.  Obviously, all we're doing in the example is adding an address to a set, but in policies where additional logic is needed, this could be a huge relief off both the workers and the manager.  A good example of this usage is in the core policies known-hosts.bro, known-services.bro, and known-certs.bro.

Syncing Data Between Workers

Instead of the manager or proxies, let's imagine we want the workers to all have the same data so they can make decisions on it immediately.  We need to be cautious with this method because it means duplicating the memory footprint across all workers, but there are cases where being able to lookup data in a set or table will save more additional work.

Workers aren't connected directly to each other so in order to publish data to them we need to relay it through either the manager or the proxies.  To do this, we need to establish two things: a topic for the worker to publish to and we need to handle that event on the manager or proxies to forward the event to the other workers.  I'll be using a method that can be seen in the core Bro distribution as part of scripts/base/protocols/irc/dcc-send.bro.

First let's discuss the Broker topic.  We could relay the event through the master node, but ideally we should keep the load on the proxies as that's what they're for.  It would also be nice to distribute the load evenly across all available proxies which we can do using a provided round robin function, Cluster::rr_topic().  The result is that we can build a new function that determines which proxy or manager topic to use:

function example_relay_topic(): string{
   local rval = Cluster::rr_topic(Cluster::proxy_pool, "example_rr_key");


   if ( rval == "" )
       # No proxy is alive, so relay via manager instead.
       return Cluster::manager_topic;


   return rval;

}

This simple function tries to round robin through our proxies, but if for some reason we don't have any available, it will fall back to using the manager node.  The second parameter is just a key that the Cluster framework uses internally.  With the topic deciding function in place, we can modify the core Broker::publish line.

event connection_established(c: connection){
     if(c$id$resp_h == 192.150.187.43){
Broker::publish(example_relay_topic(),
EX::add_client, c$id$orig_h);
     }
}

Note that all we've really done at this point is send the EX::add_client event to one of the proxies, we still need to have them forward it to the workers.  That can be accomplished within that same event:

event add_client(newaddr: addr){
@if ( Cluster::local_node_type() == Cluster::PROXY ||
     Cluster::local_node_type() == Cluster::MANAGER )
     Broker::publish(Cluster::worker_topic,EX::add_client,newaddr);
@else
     if(newaddr !in known_clients){
       add known_clients[newaddr];
     }
@endif

}

And then we can test just like before:

[bro@host ~]$ wget -q https://www.bro.org
[bro@host ~]$ broctl print EX::known_clients
      logger   EX::known_clients = {
}
     manager   EX::known_clients = {
}
     proxy-1   EX::known_clients = {
}
     proxy-2   EX::known_clients = {
}
    worker-1   EX::known_clients = {
198.128.111.111
}
    worker-2   EX::known_clients = {
198.128.111.111
}
    worker-3   EX::known_clients = {
198.128.111.111
}
    worker-4   EX::known_clients = {
198.128.111.111
}

So there we have it, the known client address was distributed to all of the workers.

Debugging Broker Communication


Clearly there's a lot more to Broker than I've covered here and it can start to get confusing so let's look at a way to dig deeper into where these messages are going.  To start, you'll need to recompile Bro from source and include the --enable-debug flag.  

Now we need to pass the -B broker argument to bro which we can do by setting BroArgs in $BROPATH/etc/broctl.cfg.  (Other debugging options can be see with bro -B help.)

[bro@host ~]$ head -n 4 /usr/local/bro/etc/broctl.cfg
## Global BroControl configuration file.

BroArgs = -B broker

[bro@host ~]$ broctl deploy
[bro@host ~]$ wget -q https://www.bro.org

We need to look at the debug logs for the individual nodes which will be found in the $BROPATH/spool/[nodename]/ directories while the cluster is running.  Let's look at the proxies first:

[bro@host ~]$ grep add_client /usr/local/bro/spool/proxy*/debug.log
/usr/local/bro/spool/proxy-1/debug.log:1530925958.459474/1530925958.460614 [broker] Process event: EX::add_client: [198.128.111.111]
/usr/local/bro/spool/proxy-1/debug.log:1530925958.459474/1530925958.460633 [broker] Publishing event: EX::add_client([198.128.111.111]) -> bro/cluster/worker

Here we can see where proxy-1 received the relay event and then published the event to the workers.  The second proxy never saw anything because of the round robin nature of the relay.  The logs also contain the event called and arguments which can be very useful.  Now the workers:

[bro@host ~]$ grep add_client /usr/local/bro/spool/worker*/debug.log
/usr/local/bro/spool/worker-1/debug.log:1530925958.531330/1530925958.531552 [broker] Process event: EX::add_client [198.128.111.111]
/usr/local/bro/spool/worker-2/debug.log:1530925958.532718/1530925958.533077 [broker] Process event: EX::add_client [198.128.111.111]
/usr/local/bro/spool/worker-3/debug.log:1530925958.527955/1530925958.528265 [broker] Process event: EX::add_client [198.128.111.111]
/usr/local/bro/spool/worker-4/debug.log:1530925958.531130/1530925958.531427 [broker] Process event: EX::add_client [198.128.111.111]

As you can see, all of the workers received the event as expected.

&synchronized


As mentioned at the start, I'm including this section simply as a reminder of what a simple policy would have looked like with the old &synchronized attribute.

@load base/frameworks/cluster
module EX; export { global known_clients: set[addr] &create_expire=1day &synchronized; } event connection_established(c: connection){ # bro.org clients if(c$id$resp_h == 192.150.187.43 && c$id$orig_h !in known_clients){ add known_clients[c$id$orig_h]; } }
All we're doing is creating a set of addresses that we see as clients connecting to bro.org.  To test this, I've setup a cluster with four workers and two proxies (realistically we only need one, but two will be important for a later example).  We can use broctl print on the command line to check values on a running cluster:

[bro@host ~]$ wget -q https://www.bro.org [bro@host ~]$ broctl print EX::known_clients logger EX::known_clients = { } manager EX::known_clients = { } proxy-1 EX::known_clients = { } proxy-2 EX::known_clients = { } worker-1 EX::known_clients = { 198.128.111.111 } worker-2 EX::known_clients = { } worker-3 EX::known_clients = { } worker-4 EX::known_clients = { }

We made one connection and only one of the workers shows an address in the set proving that &synchronized is depreciated.

Wrap-Up


I hope this helps folks get started with Broker. There's so much more to learn with Broker, but I felt it was a good idea to help ease the transition with the most common aspects.

Monday, June 4, 2018

Conservancy and Bro Announce End to Bro's Member Project Status

Bro Moves Back to ICSI; Makes $10k Donation To Conservancy

Software Freedom Conservancy, a charity that provides a home to free and open source software projects, and the Bro Leadership Team announce that the Bro Project, an open source network traffic analysis framework, will end its status as a Conservancy member project. 

During its time with Conservancy the Bro project successfully raised funds and spent them effectively to support the community. For example, Conservancy helped Bro manage a substantial MOSS grant, which created an ecosystem for Bro community contributions through the new Bro package manager & repository. Conservancy also supported three conferences as well as smaller workshops, helped acquire trademarks for the project, and assisted in many other ways. In recognition of all of this work, the Bro Leadership Team is donating $10,000 to the Conservancy’s general fund to aid them in their ongoing efforts to promote and support software freedom and provide a home to other member projects.

The mutual decision for Bro to leave Conservancy is a result of the changing nature of Bro’s community of core contributors, and the diminished fit between the rapidly growing project and Conservancy’s charitable goals and corresponding services. Conservancy will assist Bro moving back to the International Computer Science Institute (ICSI)—the project’s previous home for more than a decade.

When the Bro project first joined Conservancy more than three years ago, the project was primarily a collaboration between two different academic institutions: ICSI and the National Center for Supercomputing Applications (NCSA). At that time, Bro’s development was funded mostly through substantial awards by the U.S. National Science Foundation (NSF), who set out to advance Bro into a powerful security tool for the nation’s education environments and scientific institutions. 

Today, the Bro community looks different. With the NSF funding winding down, the team at the NCSA that heavily contributed to Bro for nearly a decade has significantly reduced their work on the project. Most of the core team of Bro is now affiliated with Corelight, placing the company at the center of Bro’s future development—which mismatches Conservancy’s charitable mission. While Bro’s strong footing in the academic community remains, the Bro user community overall has expanded from the public sector to the private sector. This shift has also been reflected in Bro conference attendance. These successes and rapid changes have led to an evolution of the project such that its trajectory is less of an apt match to Conservancy’s goals and services.

Going forward, ICSI will once again provide the Bro Leadership Team with asset and financial management as the project moves into a new phase of its life cycle. The Bro Leadership Team will continue to steer the project’s overall direction as an independent entity working in the best interest of Bro’s large and diverse open-source community, and Conservancy is fully committed to helping Bro transition smoothly to its new home.

Friday, May 25, 2018

Broker is Coming: Persistent Stores

Note: This is a guest blog post by Mike Dopheide.

----------------------------------------------------------------------------------------

Disclaimer:  If you aren't familiar with the Bro IDS software, this is going to make zero sense.

The Bro development team has been hard at work and, Broker, the new communication framework isn't far off.  I started looking into it to solve a problem I was having and learned quite a bit along the way.  Given that the documentation and examples aren't all done yet, I thought it might be nice to share a basic example of how data stores can be used.

Getting started, you'll need to be using the master branch.  Of note, Bro now ships with libcaf so if you don't specify --with-caf during configure, Bro will just build it for you which is super nice.

Problem Statement


To give you a little bit of context for where my head was at, we'd noticed our worker nodes continuously doing thousands of reverse DNS requests.  I tracked this down to the built-in protocols/ssh/interesting-hostnames.bro policy which tries to tell you if there is a successful ssh login to a host with an interesting name, like 'mail' or 'dns'.  Our issue was that internally processes were ssh'ing into all of our systems on a regular basis causing tons of redundant queries for local systems.  Wouldn't it be nice if we cached those locally on the worker?  Since we already have a list of locally known hosts provided by protocols/conn/known-hosts.bro, I set out to extend/replace that to also do the DNS lookups.  Might as well figure out how to do it with the upcoming Broker framework so it doesn't have to be re-written later.

What I'm going to show isn't that full extension (Part 2?), but an example of how to interact with persistent Broker stores.  If you want to follow along, now is a good time to make sure your known_hosts.log is working correctly[1] to begin with.

Making known_hosts Persistent


Let's look at the newly rewritten protocols/conn/known-hosts.bro to point out a few things.  First, let's take note of the variable defining the name of the data store name, this will be relevant in a little bit:

global host_store: Cluster::StoreInfo;
...
const host_store_name = "bro/known/hosts" &redef;

Next, inside bro_init() we see this:

Known::host_store = Cluster::create_store(Known::host_store_name);

The data store is created when Bro initializes and this needs to happen before you can access data in the store, regardless of whether the store is persistent or not[2].  We'll see later that this may be a bit counter intuitive as you have to 'create' a store that may already exist on disk.

In order to make Known::host_store persistent, it's helpful to understand what the Cluster::create_store function is doing. From
share/bro/base/frameworks/cluster/main.bro:


global create_store: function(name: string, persistent: bool &default=F): StoreInfo;
...
const default_backend = Broker::MEMORY &redef;
...
const default_persistent_backend = Broker::SQLITE &redef;

Here we notice that the default for create_store is to create a store that is not persistent, which results in simply using memory for your backend.  If we could change the call in known-hosts.bro to be Cluster::create_store(Known::host_store_name, T), we'd be all set.  However, that's not an option without modifying shipped code, fortunately there's another option.

Recall above that I didn't talk about host_store being allocated as Cluster::StoreInfo.  This is a record that contains a backend element.

type StoreInfo: record {
...
               ## The type of backend used for storing data.
               backend: Broker::BackendType &default=default_backend;
...

So the backend for our store is set to the default of Broker::Memory before it's actually created and this storeinfo record is redef'able.

Add these lines to a new test script (broker-test.bro in my case) or local.bro, as long as it happens after loading known-hosts.bro:


redef Cluster::stores += {
 [Known::host_store_name] = Cluster::StoreInfo($backend = Broker::SQLITE)
};

Checking to Make Sure the Data Store Exists via CLI


First, let's examine what we should see at this point.  Back in protocols/conn/known-hosts.bro we can find the call to Broker::put_unique():

event Known::host_found(info: HostsInfo)
       {
       when ( local r = Broker::put_unique(Known::host_store$store,
info$host, T, Known::host_store_expiry) )

There are two things to notice here.  First, Broker data store functions need to happen through asynchronous when()calls, that's important to know for later.  The second thing is the arguments to put_unique() where we see the store, key, value, and expiration.  What this means is we should see keys in the store with values that are simply the T bool.

Now run Bro.  If you run Bro in standalone mode, you will find the data store off of your current directory in a name that matches the name of the data store from above, bro/known/hosts.sqlite, or as a cluster the default store location will be $BROPATH/spool/stores/bro/known/hosts.sqlite

Hopefully you've got sqlite3 so you can use it on the command line to verify the data store:

[bro@hostname known]$ sqlite3 hosts.sqlite
SQLite version 3.7.17 2013-05-20 00:56:22
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
sqlite> .tables
meta   store
sqlite> .schema store
CREATE TABLE store(key blob primary key, value blob, expiry integer);
sqlite> select * from store;
||1526488517607812295
||1526488517609842240
||1526488517611842000
...
||1526488517613785435
||1526488517615733229


sqlite>

We see the expiration times, but it clearly looks like something may be wrong.  The keys and values don't appear to exist, but I assure you they're there.  The 'blob's can be seen more clearly in hex:

sqlite> select hex(key) from store limit 1;
0600000000000000000000FFFFAABBCCDD
sqlite> select hex(value) from store limit 1;
0101

The hex values in red are the addr for our keys and 0101 is our bool T value.  Perfect, that's exactly what we wanted to see.

Accessing an Existing Data Store


Let's let take a look at accessing that data store outside of the original policy that created it.  I'll be building a short new Bro policy and show the difference whether or not policy/conn/known-hosts.bro is loaded first or not.


@load base/frameworks/cluster
module Test;
event bro_init() {
     for(store in Cluster::stores){
               print store;
     }
}

The Cluster::stores variable should hold all of the data stores Bro is aware of.  However, when we run Bro with this script there's no output.  Recall I mentioned the the store needs to be created when Bro initializes, regardless of whatever it's persistent or not.  Let's try that:

@load base/frameworks/cluster
module Test;


export {
      global host_store: Cluster::StoreInfo;
}


event bro_init() {
host_store = Cluster::create_store("bro/known/hosts",T);
     for(store in Cluster::stores){
               print store;
     }
}

Here we can create the store adding the extra T argument to make the store persistent without having to do the redef.  Now when we run Bro, the store is initiated and we can see "bro/known/hosts" in the output.  However, there's a bit cleaner way given that we assume policy/protocols/known-hosts will already be loaded.  Make sure we remember to redef the backend in this case (you may have done this in local.bro previously).

@load base/frameworks/cluster
@load protocols/conn/known-hosts
module Test;


redef Cluster::stores += {
   [Known::host_store_name] = Cluster::StoreInfo($backend = Broker::SQLITE)
};


event bro_init() &priority=-5 {
     for(store in Cluster::stores){
               print store;
     }
}

Note that by loading known-hosts and setting a negative priority, we allow the create_store()call to happen before we get to this point.  We also want to make sure we still do the redef for the store backend.  That doesn't really count as accessing the store, we've just verified that it's loaded, so let's add a little more:


@load base/frameworks/cluster
@load protocols/conn/known-hosts
module Test;


redef Cluster::stores += {
   [Known::host_store_name] = Cluster::StoreInfo($backend = Broker::SQLITE)
};


event bro_init() &priority=-5 {


when ( local r = Broker::keys(Known::host_store$store)){
if ( r$status == Broker::SUCCESS ){
print r$result
}
}timeout Known::host_store_timeout{
print "Broker timeout\n";
}
}

The Broker::keys() function, as the name implies, returns all of the keys from a data store.  It's important to note that all when() calls that involve Broker require having a timeout defined. Known::host_store$store and Known::host_store_timeout are available to us as a result of known-hosts being loaded.  Your output should include broker::data that includes a set of your keys:

[data=broker::data{{192.168.1.1, 192.168.2.2, 192.168.3.3}}]

Obviously, if we had values in our store other than just T, we'd want to be able to access those as well.  For that we can use the Broker::get() function.  If we already know the key we need the value for, it's fairly simple:

      when ( local r2 = Broker::get(Known::host_store$store,
192.168.1.1)){
          print r2$result;

     }timeout Known::host_store_timeout{

              print "Broker timeout\n";
     }

The output in this case should be:

[data=broker::data{T}]

This is a good time to introduce that variables can now be cast to other types.

     when ( local r2 = Broker::get(Known::host_store$store,192.168.1.1)){
          print r2$result as bool;
     }timeout Known::host_store_timeout{
              print "Broker timeout\n";
     }

That will give you just the T value.  So that's if you know the key you want to get the value for, but let's say you need to iterate over all of the keys, that's possible as well by embedding the get() inside of the result of keys().

when ( local r = Broker::keys(Known::host_store$store)){
if ( r$status == Broker::SUCCESS ){


for (ip in r$result as addr_set) {
when ( local r2 = Broker::get(Known::host_store$store,ip)){
print fmt("%s %s",ip,r2$result as bool);
}timeout Known::host_store_timeout{
print "Broker get() timeout\n";
}
}
}
}timeout Known::host_store_timeout{
print "Broker timeout\n";
}

And there you have it, iterating over your Broker data store.  Obviously this becomes much more useful and has less overhead when calls to Broker are made through events and not layers of embedded when()'s, but I think this example helps illustrate how things are working at a basic level.

Acknowledgements:  Thanks to Justin Azoff and Jon Siwek for answering my questions and Samson Hille for providing the initial motivation.

Footnotes:

1)  Common problems are not having protocols/conn/known-hosts loaded or not having your Site::local_nets defined (or listed in networks.cfg).
2)  Persistence is just whether or not your data will be saved across Bro restarts.  By default, if you restart Bro, your known_hosts will be empty and all of the hosts you knew before will be re-logged.