You can download the script on the yahsnarf project page.

Yahsnarf requires Ruby, ruby-pcap and bit-struct (Thanks Matasano for introducing me to bit-struct, made this script take about 1/4rd the time to write)

I’m also currently working on an NSM-Console module for Yahsnarf.

This script is a little different than Aimsnarf, mostly because Aimsnarf was the first program I ever wrote in Ruby, so it tended to be just a little rusty, without the best design practices. For one, Yahsnarf is way smaller than Aimsnarf (70 lines to around 150), and Yahsnarf follows an object-oriented design. Enough of that, here’s what you can expect to see:

shell> sudo ./yahsnarf.rb -i en1
Use '-h' to display usage
Capture/Decoding...
buddy1 --> buddy2: This is a test of yahsnarf
buddy2 --> buddy1: A test this is of yahsnarf; it's awesome!
buddy1 --> buddy2: thanks for the help :)

You can also use ./yahsnarf.rb -r <pcapfile> to read and extract from a network capture file.

Pretty simple eh? Replace buddy1 and buddy2 with the screen names of the conversationalists. There are a few issues I’m still working out, like usernames not always showing up (they could for the most part). Also, this obviously does not work on encrypted messages (OTR or otherwise), so if you value your privacy, use encryption.

Remember, don’t ever say anything over IM that you wouldn’t mind the world knowing, you never know who could be listening in

In conclusion, I’d also like to thank Yahoo, for making their protocol so much less of a pain to decode than AOL’s.

NSM-Console (Network Security Monitoring Console) is a framework for performing analysis on packet capture files. It implements a modular structure to allow for an analyst to quickly write modules of their own without any programming language experience. Using these modules a large amount of pcap analysis can be performed quickly using a set of global (as well as per-module) options. It aims to be simple to run and easy to understand without a lot of learning time.

The recently posted NSM-Console whitepaper is a good place to start if you want an introduction to NSM-Console.

You can download NSM-Console 0.6 from the project page.

Here are the highlights for the changes in this version, check the TODO and CHANGELOG for a full list of changes.

The dump command
The dump command lets you dump either the payload (in ascii format), or the full packet (with the -f flag) to a file, you can specify ranges like 1-20, 5,7,9-* and 1-* just like you can with the print command, extremely useful for picking and choosing packets to dump to a file. Here’s an example:

nsm> dump 1-5,10-* file.txt
Writing (append) packet(s) 1 through 5 from /Users/hinmanm/data.pcap to file.txt...
Writing (append) packet(s) 10 through * from /Users/hinmanm/data.pcap to file.txt...

And another of the full dump

nsm> dump -f 20,56-59 full.out
New args: 20,56-59 full.out
Dumping full packet, not just payload...
Writing (append) packet(s) 20 through 20 from /Users/hinmanm/data.pcap to full.out...
Writing (append) packet(s) 56 through 59 from /Users/hinmanm/data.pcap to full.out...

Performance improvements for harimau and checkip
The Harimau module, as well as the checkip command, now use wget (if it’s installed) instead of Ruby’s HTTP request library, this results is a much much faster query time. Another performance tweak was that reverse dns lookups for most modules is now turned off by default because it was taking extremely long for large pcaps

New modules: clamscan, foremost, trace-summary
New modules have been included in this release, clamscan, which scans extracted files (from either the tcpxtract or foremost modules) for viruses, foremost, which is another file extractor and trace-summary, which provides an extremely nice summary of traffic in the pcap

NSM-Console handle’s gzip’d files
You can now specify a gzip’d file directly to NSM-Console, it will attempt to decompress the file into a temporary file when it is loaded. Note that directories of gzip’d files is still not supported entirely.

Bro-ids and Snort changes
Bro-IDS has been split into 2 modules, one for protocol analysis and the other for generating content streams.
Snort has had its community and emerging rules updated to the latest revisions.

As always, bugfixes and minor enhancements, check out the full changelog for a list of what’s changed. If you have any requests for additional features or modules, be sure to leave a command and let me know

Also, if anyone out there wants to see what my ~/.nsmcrc looks like, you can find it here.

For example, let’s say you discover a new binary malware that one of your honeypots caught, here’s how I envision this would work out:

1. You register an account at the collaboration website, you can additionally assign your pgp key to your name, security people like to know who they’re actually talking with.
2. You upload the file, in this case it’s a .exe file, tagging it with a basic description (“nepenthes honeypot caught this transferred over ftp, I think it’s a trojan, etc, etc”) and tags so it becomes searchable (exe, malware, binary, ftp).
3. The file/pcap is anonymized (optional, but would be extremely nice)
4. After the initial upload, the collaboration server performs super-basic, but good baseline analysis on the file, saving the results for later. For a .exe file, it could be things like md5sum, clamscan and strings. For other types of files, different tools could be used (*cough* an automated NSM-Console session *cough*), etc
5. The malware is displayed on the page, security gurus log into their account, have the ability to download the binary to play with it themselves, and are encouraged to share what they found when doing their analysis (and how). They have the ability to upload screenshots, short video clips, textfiles, whatever would help with the analysis. This of it like a traditional website ‘shoutbox’, but with comments on a particular piece of malware or network capture.
6. Users can also create correlations between different submissions, Example: “This is the link to the network capture for the worm exploiting this particular binary malware”, now we can draw pretty graphs!
7. Discussion continues until the file has been “figure out”. Give people ‘karma’ or whatever to encourage posting.
8. ????
9. Profit!

In all seriousness, you know what I think would be great about this? The community as a whole benefits from the knowledge and talent of people who are good at an individual skill. For instance, I might suck at binary malware analysis, but I can help decode what’s going on with a network trace picked up by an IDS. Community is created, knowledge is shared, security can be improved, people become familiar with the parts of security in which they lack knowledge, everyone is happy.

Make the framework distributable, small groups of people can set up their own collaboration for working with extremely confidential files, think Trac, but instead of bug reports and svn tracking, malware/pcap collaboration and research.

There are projects already like this, I’m excited for the direction that OpenPacket is going with packet captures, upload a file and it’s automatically run through tshark, giving you a baseline to start working with. I think that if the idea is expanded, we can get a lot of different people involved. I know I’d certainly like to get better at doing binary analysis.

Does this sound interesting? It certainly does to me. I’m curious if anyone else is interested, leave me a comment and let me know if you’d be interested in something like this! (Maybe if 40 hours suddenly appear out of nowhere I’m start working on it…)

P.S. I didn’t think of all of this myself, thanks to all the people in #rawpacket for their ideas Just want to give credit where it’s due…

Flowtime is a script written in Ruby that produces a timeline of the network flows in a pcap file. Everything is better with a picture, so here’s a picture: (warning, this picture is 3000×2000 pixels, kind of large)

Each bar on the left is a IP address along with a port, the timeline on the bottom is the time it was capture (in seconds). The different colors represent different kinds of traffic, http is blue, ssl is red, yellow is other, etc. At the moment there isn’t any legend, but I already think it’s useful just to see patterns in the traffic over time.

Download Flowtime here.

Requirements to run Flowtime:

• Argus (version 3 only)
• Ploticus (you should have ‘pl’ in your path. You may have to symlink ‘ploticus’ to ‘pl’)
• EasyTimeline (you should have ‘EasyTimeline’ in your path)

Yea, I know, lots of dependencies, I’m hoping to rewrite this into a MUCH better version in the future, consider this the prototype

Here’s how to run it:

flowtime [-w #] [-h #] [-g] [--help] <pcapfile> <ipaddr> <outfile_base>
-w specify the width, default: 2000
-h specify the height, default: 2000
-g automatically try generate a png (requires 'EasyTimeline' and 'pl' in path)
<pcapfile> the packet file to generate a graph of
<ipaddr> source address to generate a graph for, 'all' for all IPs
<outfile_base> basename for the output file

To generate a basic graph, just do something like ‘flowtime -g data.pcap all data-out‘. After running this (if everything works okay), you should have a few files in your directory, if you open data-out.png you should be able to see the data as a timeline. If there are errors instead of data-out.png, there will be a file called data-out.err.

I know this script has issues, isn’t very user-friendly and doesn’t always work, here’s what I’m hoping to improve for the next version:

• Generate an image of the entire timeline, in addition to showing it in a window
• The ability to zoom in to a particular range of time to show only that time
• Export an image of the current view
• Select a bar to get more information about that flow

I’m going to have to use a different framework though, I’m considering Tk, but I haven’t ever done anything with it before.

Questions? Concerns? Comments? Suggestions?

P.S. I already made an NSM-Console module for flowtime too

Firstly, download the files here.
The static page for nsm-console is here.

I finally got around to releasing the next version of the nsm-console. This version incorporates a large amount of bug fixes and additional features, first, I’ll start with some of the features I’m the most happy about Most of these features are in the new Hex 1.0.2 release which came out yesterday (go download it now!)

• Categories
  • You can now toggle certain categories on and off, for instance, one category shipped with the new release is the ‘flow’ categories, you treat them just like a regular module. Simply use “toggle flow” to toggle the flow category (and all of it’s modules) on and off.
  • You can easily add your own categories to customize your work environment, all you have to do is create a file named the same name as the category name in the modules/categories directory.
• Directory analysis
  • When you normally run the nsm-console, you would specify a single pcap file to perform analysis on, now you can use the same “file” command to specify a directory full of files  instead of a single file. When the “run” command is executed, all the toggled module’s operations will be executed on each file in the directory (recursively)
  • To better accommodate this type of operation, I encourage anyone that is writing any modules to write them to output the results into an output file named something like ${PCAP_BASE}.tcpdstat.out (so if you had more than 1 file, the output will go into more than one file)
• The ‘exec’ command
  • I added the exec command because I was tired of spawning an additional shell in order to run a simple ‘tcpdump’. I hope this helps with the automation that I’m going to talk about below.

The directory functionality and the exec command isn’t in the current Hex release, but hopefully it will be in the next release.

Part of the reason I think nsm-console is neat is the ease of automation you can do using simple text files. For instance, if you created a text file called “automate.txt” and put the following lines in it:

file /pcap/data.pcap
output automated-output
toggle aimsnarf
toggle tcpdstat
toggle chaosreader
run
quit

Then, you can run the command:

./nsm < automate.txt > output.txt

Which will run all the commands in the text file automatically, placing all the output in output.txt, simple eh?

If you have any questions, comments or suggestions, feel free to leave a comment or send an email I’d love to hear if/how you’re using nsm-console

<ip address>,<country>,<city and state>,<latitude>,<longitude>,<packet count>

First off, get the script here.

It requires ruby-pcap and wget (like most of the Ruby scripts I write :P). Each unique ip address will have the script query the hostip.info database, storing the info in a temp file. If the ip already exists in the script’s list, it won’t query for the information (to save time and bandwidth).

iploc can either run in live capture mode, or read from a pcap file. Here’s how you would run it live:

./iploc -i eth0 >> output.csv

iploc will run until it receives a CTRL-C, at which point it will aggregate the data and dump it to STDOUT. Alternatively, you can read from a pcap file like this:

./iploc -r ~/data.pcap >> output.csv

Here’s an example of what the CSV file looks like after being opened in Numbers (or Excel):

Since it’s a CSV file, it should be easy to parse for any other program. I’m planning on hopefully writing another program to take out the latitude/longitude and plot the points on a map (for a more visual representation).

IPloc also supports BPF filters, just use it like you would tcpdump, for instance, if you only cared about http data:

./iploc -i eth0 port 80 > output.csv

Questions? Leave me a comment below! Now go forth, and visualize your honeypot data!

P.S. Thanks to the hostip guys for having a really nice web api to get this data quickly.

If you get this error:

cc -I. -I../libpcap-0.7.1 -DLINUX -D__FAVOR_BSD -D_LARGEFILE_SOURCE=1 -D_FILE_OFFSET_BITS=64 -L../libpcap-0.7.1 -c stat.c
cc -I. -I../libpcap-0.7.1 -DLINUX -D__FAVOR_BSD -D_LARGEFILE_SOURCE=1 -D_FILE_OFFSET_BITS=64 -L../libpcap-0.7.1 -c net_read.c
net_read.c:74:1: warning: "__FAVOR_BSD" redefined
<command line>:1:1: warning: this is the location of the previous definition
net_read.c:149: error: static declaration of ‘packet_length’ follows non-static declaration
tcpdstat.h:415: error: previous declaration of ‘packet_length’ was here
make: *** [net_read.o] Error 1

Edit the net_read.c file and change line 149 from this:

static int packet_length; /* length of current packet */

to this:

int packet_length; /* length of current packet */

Simple eh? Just type “make” again and tcpdstat should compile just fine. Simple fix.

Behold! I present NSM-Console! (read more about it here, watch a screencast here)

Download the framework here.
Keep in mind this framework only includes 3 modules (mostly used just for testing)

NSM-Console in a small (< 500 1000 1500 lines) framework for running nsm modules. Essentially, it’s a framework for running things (but we don’t call it that because it sounds like it wasn’t any work :P). Here’s the breakdown:

Users will run the ‘nsm’ program to enter into a console, by which they can select modules that have options to run against a packet capture file. Let’s talk a little bit about how it works. When nsm starts, it looks in the directory given by $MODULE_DIR (hardcoded at the moment, will be moved later) for all directories that follow the naming convention “<name>.module”, where <name> is the name of the module. Following so far?

For explanation’s sake, let’s call our module “aimsnarf”, after the aim-sniffing program I wrote a while back. So inside the module directory, there is another directory called “aimsnarf.module”. Inside the aimsnarf.module directory, there are a few files that define how our module will act. First is a file just called “aimsnarf”, this is the file that dictates commands to be run in the framework. It can have multiple commands separated by lines in the file. (If the module were called “bathtub”, the file would be called “bathtub” also, it’s named after the module name). Let’s take a look at an example line from the aimsnarf.module/aimsnarf file:

aimsnarf -r ${PCAP_FILE} > ${OUTPUT_DIR}/${OUTPUT_FILE}

In this line, you can see some variables that get changed out when the module is executed from nsm console. ${PCAP_FILE} is always set to the pcap file being analyzed, ${OUTPUT_DIR} will be set to the module-specific directory output will be directed into (set by nsm console), the last option, ${OUTPUT_FILE} is not set by the nsm console, it is a user-defined variable. But where is it defined? Let’s take a look at the next file: aimsnarf.module/defaults:

OUTPUT_FILE=${PCAP_BASE}.aim

The defaults file defines the user-settable options for the module, in this case, there is only 1, the “OUTPUT_FILE” option. The defaults file also provides a default value for that option (in this case, ${PCAP_BASE}.aim). The default value is provided so the module can be run without having to set every option. ${PCAP_BASE} is another variable that is provided by the nsm console, it is the basename of the pcap file (so it the pcap file was “~/pcap/file.pcap”, the ${PCAP_BASE} would be “file.pcap”).

The next file is aimsnarf.module/description, this file provides a 1-line description of the module to be listed when the user lists all the modules. Here is the ‘description’ for the aimsnarf module:

Extracts AIM messages from a pcap file

Simple eh? Still following? Okay, there is one more file, the aimsnarf.module/info file, this file provides a more detailed explanation of the module, it should also define what each option means, here is what aimsnarf’s ‘info’ file looks like:

Aimsnarf is a program to extract aim conversations from a pcap file

The following options are enabled:
OUTPUT_FILE (default $PCAP_DATA}.aim)
This option specifies the output file.

So how does this all work together? Well, first, you would start the NSM console you would see the following screen:

You can see that the aimsnarf module has been loaded (as well as a couple of others) and a prompt has been presented. From here there are a few different options, here’s the output of the ‘help’ command:

I think the help is pretty self-explanatory, hopefully it makes sense to everyone else as well.

Let’s run the aimsnarf module on a pcap file as an example. First, we need to specify a pcap file to work on with the ‘file’ command, so we type:

nsm> file /Users/hinmanm/data.pcap
Setting ${PCAP_FILE} = /Users/hinmanm/data.pcap
Setting ${PCAP_BASE} = data.pcap

Next, we will specify a different output directory (the default is ‘output’) using the ‘output’ command:

nsm> output data-output
Setting ${OUTPUT_DIR} = data-output

Next, let’s make sure the only module that is going to run is the aimsnarf module:

nsm> toggle none
All modules turned off.
nsm> toggle aimsnarf
aimsnarf module turned on.

And make sure our options are set correctly for the aimsnarf module:

nsm> options aimsnarf
Global options:
-----------------------------------
${PCAP_FILE}: /Users/hinmanm/data.pcap
${PCAP_BASE}: data.pcap
${OUTPUT_DIR}: data-output
${MODULE_DIR}: modules
Options for module aimsnarf:
-----------------------------------
${OUTPUT_FILE} = ${PCAP_BASE}.aim

That looks good to me! Now, let’s run and see what happens:

nsm> run
Executing analysis...
===> module aimsnarf running...
Creating directory data-output/aimsnarf
Result: true
--> aimsnarf -r /Users/hinmanm/data.pcap > data-output/aimsnarf/data.pcap.aim
/Library/Ruby/Site/1.8/universal-darwin9.0/pcap.bundle: warning: do not use Fixnums as Symbols
/Library/Ruby/Site/1.8/universal-darwin9.0/pcap.bundle: warning: do not use Fixnums as Symbols
/Library/Ruby/Site/1.8/universal-darwin9.0/pcap.bundle: warning: do not use Fixnums as Symbols
Result: true
===> module aimsnarf finished.
===> module hash skipped.
===> module ngrep skipped.

Looks like it ran the command! (ignore the warnings, they’re generated by aimsnarf, not nsm) Simple enough, right?

So here’s the real question: why write something like this? Well, I believe this tool will help to perform large amounts of analysis (think 20-30 plugins instead of 3) on packet capture files without having to manually manage each tool that does analysis. Instead of having to change a hardcoded script, you can now edit (and create) modules to do exactly what you need, but still having the flexibility of global varaibles (ie: ${OUTPUT_DIR}) and module specific variables (ie: ${OUTPUT_FILE} in aimsnarf). Being able to toggle each module on and off allows you to do a flexible amount of analysis on a file on the fly. In the future I hope to be able to extend the framework to run on a directory of files, allowing you to automate running a vast number of pcap analysis tools on a large number of pcap files very easily.

While this is not a replacement for running each command individually (and it shouldn’t be), I hope this tool will be useful to people. I hope to be able to make it useful for the Hex liveCD as well

Questions? Comments? Leave me a comment below or send me an email!

Download the script here.

Read about aimsnarf in the previous post about it.

Changes in this version:

• Trillian is now supported, as well as AOL’s AIM client. Most other clients should be supported too, I figured out the variable length/number of TLV fields in the packet, so aimsnarf is much smarter about decoding them
• Code cleaned up to be more readable
• Fixed some misc messages that were showing up, you still might see a few

Todos:

• Figure out what the heck iChat is doing, it doesn’t seem to be sending the same kind of data as all the other AIM clients
• Still do OTR stuff
• Maybe add support for different protocols?
• More testing!

If you find any bugs, send me a note or leave a comment. If you really want to help, you can send me some pcap data to analyze   If you have any feature requests, lemme know!

Firstly, download the script here.

aimsnarf.rb is a small (~200 lines) Ruby script that I’ve written to sniff and dump AOL IM messages to STDOUT. I wrote this an as alternative to aimsniff, because I really dislike having to install aimsniff and all of it’s dependancies when all I want is a simple text transcript. I really felt like the dsniff toolkit should have had something like this (they already have urlsnarf, filesnarf, etc) to be used for penetration testing.

The only thing aimsnarf requires is Ruby and the ruby-pcap library (which is waaay easier to install than the 10+ CPAN modules that aimsniff requires). After installing the pcap library, simply run aimsnarf.rb on the console, here’s the usage:

Use '-h' to display usage
Usage: aimsnarf.rb [ -dnv ] [ -i interface | -r file ] [ -c count ] [ -s snaplen ] [ filter ]
Options:
-n do not convert address to name
-d debug mode
-v verbose mode

Due to the way that ruby-pcap works, I don’t have control over the usage displayed, currently the only real options you should mess with are ‘-i interface‘ and ‘-r file‘, changing anything else might produce “unknown” consequences ;). If you want to see hex dumps of the AIM data, edit the script and change the line “ap.data_debug(0)” to be “ap.data_debug(1)“, this will display the hex data as it is received.

Ignore the “pcap.bundle: warning: do not use Fixnums as Symbols” warnings you get when you run the program, the warning lies with the ruby-pcap library, so it’s out of my hands to fix. When run correctly, you should see something like this:

****** --> <you>: <HTML>what're you up to?</HTML>
<you> --> ******: <HTML>doing some stuff</HTML>
****** --> <you>: <HTML>awesome</HTML>
<you> --> ******: <HTML>talkity talk talk</HTML>
etc, etc

“******” will be the screen name of the person that’s talking. Yes, AIM sends the HTML tags, I don’t put those on.

Tangent:
Let’s talk a little bit about how much I hate the AIM protocol
Take a look at the protocol listing as given from ethereal, you can see that each AIM packet actually holds a pretty good amount of information, turns out, AOL decided to make a ton of their fields variable length, which means a headache for me in decoding it, because the length has to be read, translated, then used to set the offset for reading the data, this is the reason the code for the script is incredibly messy, I plan on cleaning it up at a later time. In a future post, I’ll also go into more detail about how this particular script decodes the protocol (very much hackish at the moment).

Known Issues:

• Messages received by people who are away don’t get intercepted due to the packet being different than a regular incoming message packet
• Different clients might not work (depending on the features supported). Right now I’ve tested with GAIM/Pidgin and Adium, it looks like Trillian isn’t working correctly yet, although I’ve collected some data for analysis so I can get it working.
• This is probably the first *useful* script I’ve written in Ruby. I am not a ruby master so the code is really messy and probably badly written, have a problem with it? Send a patch!
• OTR encrypted chat interception doesn’t work (duh)

TODO (no particular order):

• Clean up code to make it easier to extend to different protocol/clients
• Fix the Trillian problem
• Test with AOL’s AIM client
• Fix the incoming/away message
• Correctly detect OTR chat and do (something?) about it

Remember people, don’t send credit card numbers, social security numbers, passwords, PIN numbers, etc over IM, ESPECIALLY when you’re somewhere like a coffeeshop using public wifi.

Thanks to the HeX LiveCD team for putting out a great release, already having the tools installed for use in a system is super helpful

Questions? Problems? Patches? Hatemail? Email me or leave a comment below!