Yahsnarf – Sniff Yahoo IM conversations
Remember way back, when I released Aimsnarf? Well, it turns out that people were interested in one for Yahoo IM, so I’m happy to present Yahsnarf, the Yahoo messenger sniffing script.
You can download the script on the yahsnarf project page.
Yahsnarf requires Ruby, ruby-pcap and bit-struct (Thanks Matasano for introducing me to bit-struct, made this script take about 1/4rd the time to write)
I’m also currently working on an NSM-Console module for Yahsnarf.
This script is a little different than Aimsnarf, mostly because Aimsnarf was the first program I ever wrote in Ruby, so it tended to be just a little rusty, without the best design practices. For one, Yahsnarf is way smaller than Aimsnarf (70 lines to around 150), and Yahsnarf follows an object-oriented design. Enough of that, here’s what you can expect to see:
shell> sudo ./yahsnarf.rb -i en1
Use '-h' to display usage
Capture/Decoding...
buddy1 --> buddy2: This is a test of yahsnarf
buddy2 --> buddy1: A test this is of yahsnarf; it's awesome!
buddy1 --> buddy2: thanks for the help 
You can also use ./yahsnarf.rb -r <pcapfile> to read and extract from a network capture file.
Pretty simple eh? Replace buddy1 and buddy2 with the screen names of the conversationalists. There are a few issues I’m still working out, like usernames not always showing up (they could for the most part). Also, this obviously does not work on encrypted messages (OTR or otherwise), so if you value your privacy, use encryption.
Remember, don’t ever say anything over IM that you wouldn’t mind the world knowing, you never know who could be listening in
In conclusion, I’d also like to thank Yahoo, for making their protocol so much less of a pain to decode than AOL’s.
NSM-Console version 0.6 release
I’m happy to announce the release of the next version of NSM-Console. Version 0.6. If you are unfamiliar with NSM-Console, here’s the synopsis from the project page:
NSM-Console (Network Security Monitoring Console) is a framework for performing analysis on packet capture files. It implements a modular structure to allow for an analyst to quickly write modules of their own without any programming language experience. Using these modules a large amount of pcap analysis can be performed quickly using a set of global (as well as per-module) options. It aims to be simple to run and easy to understand without a lot of learning time.
The recently posted NSM-Console whitepaper is a good place to start if you want an introduction to NSM-Console.
You can download NSM-Console 0.6 from the project page.
Here are the highlights for the changes in this version, check the TODO and CHANGELOG for a full list of changes.
The dump command
The dump command lets you dump either the payload (in ascii format), or the full packet (with the -f flag) to a file, you can specify ranges like 1-20, 5,7,9-* and 1-* just like you can with the print command, extremely useful for picking and choosing packets to dump to a file. Here’s an example:
nsm> dump 1-5,10-* file.txt
Writing (append) packet(s) 1 through 5 from /Users/hinmanm/data.pcap to file.txt...
Writing (append) packet(s) 10 through * from /Users/hinmanm/data.pcap to file.txt...
And another of the full dump
nsm> dump -f 20,56-59 full.out
New args: 20,56-59 full.out
Dumping full packet, not just payload...
Writing (append) packet(s) 20 through 20 from /Users/hinmanm/data.pcap to full.out...
Writing (append) packet(s) 56 through 59 from /Users/hinmanm/data.pcap to full.out...
Performance improvements for harimau and checkip
The Harimau module, as well as the checkip command, now use wget (if it’s installed) instead of Ruby’s HTTP request library, this results is a much much faster query time. Another performance tweak was that reverse dns lookups for most modules is now turned off by default because it was taking extremely long for large pcaps
New modules: clamscan, foremost, trace-summary
New modules have been included in this release, clamscan, which scans extracted files (from either the tcpxtract or foremost modules) for viruses, foremost, which is another file extractor and trace-summary, which provides an extremely nice summary of traffic in the pcap
NSM-Console handle’s gzip’d files
You can now specify a gzip’d file directly to NSM-Console, it will attempt to decompress the file into a temporary file when it is loaded. Note that directories of gzip’d files is still not supported entirely.
Bro-ids and Snort changes
Bro-IDS has been split into 2 modules, one for protocol analysis and the other for generating content streams.
Snort has had its community and emerging rules updated to the latest revisions.
As always, bugfixes and minor enhancements, check out the full changelog for a list of what’s changed. If you have any requests for additional features or modules, be sure to leave a command and let me know
Also, if anyone out there wants to see what my ~/.nsmcrc looks like, you can find it here.
Collaborative analysis efforts with simple to use interfaces
You know what would be really helpful? I mean, actually helpful to people in the security industry as a whole? We need some kind of collaboration tool that allows many different users to view, download, analyze, tag, describe and ask questions about any and all kinds of malware, network captures and security logs. I’ve been talking to some of the #rawpacket guys/gals about how it would work, so now I’m stealing their ideas for a blog post 
For example, let’s say you discover a new binary malware that one of your honeypots caught, here’s how I envision this would work out:
- You register an account at the collaboration website, you can additionally assign your pgp key to your name, security people like to know who they’re actually talking with.
- You upload the file, in this case it’s a .exe file, tagging it with a basic description (“nepenthes honeypot caught this transferred over ftp, I think it’s a trojan, etc, etc”) and tags so it becomes searchable (exe, malware, binary, ftp).
- The file/pcap is anonymized (optional, but would be extremely nice)
- After the initial upload, the collaboration server performs super-basic, but good baseline analysis on the file, saving the results for later. For a .exe file, it could be things like md5sum, clamscan and strings. For other types of files, different tools could be used (*cough* an automated NSM-Console session *cough*), etc
- The malware is displayed on the page, security gurus log into their account, have the ability to download the binary to play with it themselves, and are encouraged to share what they found when doing their analysis (and how). They have the ability to upload screenshots, short video clips, textfiles, whatever would help with the analysis. This of it like a traditional website ‘shoutbox’, but with comments on a particular piece of malware or network capture.
- Users can also create correlations between different submissions, Example: “This is the link to the network capture for the worm exploiting this particular binary malware”, now we can draw pretty graphs!
- Discussion continues until the file has been “figure out”. Give people ‘karma’ or whatever to encourage posting.
- ????
- Profit!
In all seriousness, you know what I think would be great about this? The community as a whole benefits from the knowledge and talent of people who are good at an individual skill. For instance, I might suck at binary malware analysis, but I can help decode what’s going on with a network trace picked up by an IDS. Community is created, knowledge is shared, security can be improved, people become familiar with the parts of security in which they lack knowledge, everyone is happy.
Make the framework distributable, small groups of people can set up their own collaboration for working with extremely confidential files, think Trac, but instead of bug reports and svn tracking, malware/pcap collaboration and research.
There are projects already like this, I’m excited for the direction that OpenPacket is going with packet captures, upload a file and it’s automatically run through tshark, giving you a baseline to start working with. I think that if the idea is expanded, we can get a lot of different people involved. I know I’d certainly like to get better at doing binary analysis.
Does this sound interesting? It certainly does to me. I’m curious if anyone else is interested, leave me a comment and let me know if you’d be interested in something like this! (Maybe if 40 hours suddenly appear out of nowhere I’m start working on it…)
P.S. I didn’t think of all of this myself, thanks to all the people in #rawpacket for their ideas Just want to give credit where it’s due…
Flowtime – Create a timeline for packet flow
You can never have too many tools for pcap visualization
Flowtime is a script written in Ruby that produces a timeline of the network flows in a pcap file. Everything is better with a picture, so here’s a picture: (warning, this picture is 3000×2000 pixels, kind of large)
Each bar on the left is a IP address along with a port, the timeline on the bottom is the time it was capture (in seconds). The different colors represent different kinds of traffic, http is blue, ssl is red, yellow is other, etc. At the moment there isn’t any legend, but I already think it’s useful just to see patterns in the traffic over time.
Requirements to run Flowtime:
- Argus (version 3 only)
- Ploticus (you should have ‘pl’ in your path. You may have to symlink ‘ploticus’ to ‘pl’)
- EasyTimeline (you should have ‘EasyTimeline’ in your path)
Yea, I know, lots of dependencies, I’m hoping to rewrite this into a MUCH better version in the future, consider this the prototype
Here’s how to run it:
flowtime [-w #] [-h #] [-g] [--help] <pcapfile> <ipaddr> <outfile_base>
-w specify the width, default: 2000
-h specify the height, default: 2000
-g automatically try generate a png (requires 'EasyTimeline' and 'pl' in path)
<pcapfile> the packet file to generate a graph of
<ipaddr> source address to generate a graph for, 'all' for all IPs
<outfile_base> basename for the output file
To generate a basic graph, just do something like ‘flowtime -g data.pcap all data-out‘. After running this (if everything works okay), you should have a few files in your directory, if you open data-out.png you should be able to see the data as a timeline. If there are errors instead of data-out.png, there will be a file called data-out.err.
I know this script has issues, isn’t very user-friendly and doesn’t always work, here’s what I’m hoping to improve for the next version:
- Generate an image of the entire timeline, in addition to showing it in a window
- The ability to zoom in to a particular range of time to show only that time
- Export an image of the current view
- Select a bar to get more information about that flow
I’m going to have to use a different framework though, I’m considering Tk, but I haven’t ever done anything with it before.
Questions? Concerns? Comments? Suggestions?
P.S. I already made an NSM-Console module for flowtime too
NSM-console version 0.2 release
I found out there is internet here, so I’m finally able to post some code changes I was working on while on the airplane.
Firstly, download the files here.
The static page for nsm-console is here.
I finally got around to releasing the next version of the nsm-console. This version incorporates a large amount of bug fixes and additional features, first, I’ll start with some of the features I’m the most happy about Most of these features are in the new Hex 1.0.2 release which came out yesterday (go download it now!)
- Categories
- You can now toggle certain categories on and off, for instance, one category shipped with the new release is the ‘flow’ categories, you treat them just like a regular module. Simply use “
toggle flow” to toggle the flow category (and all of it’s modules) on and off. - You can easily add your own categories to customize your work environment, all you have to do is create a file named the same name as the category name in the
modules/categoriesdirectory.
- You can now toggle certain categories on and off, for instance, one category shipped with the new release is the ‘flow’ categories, you treat them just like a regular module. Simply use “
- Directory analysis
- When you normally run the nsm-console, you would specify a single pcap file to perform analysis on, now you can use the same “
file” command to specify a directory full of files instead of a single file. When the “run” command is executed, all the toggled module’s operations will be executed on each file in the directory (recursively) - To better accommodate this type of operation, I encourage anyone that is writing any modules to write them to output the results into an output file named something like
${PCAP_BASE}.tcpdstat.out(so if you had more than 1 file, the output will go into more than one file)
- When you normally run the nsm-console, you would specify a single pcap file to perform analysis on, now you can use the same “
- The ‘exec’ command
- I added the exec command because I was tired of spawning an additional shell in order to run a simple ‘tcpdump’. I hope this helps with the automation that I’m going to talk about below.
The directory functionality and the exec command isn’t in the current Hex release, but hopefully it will be in the next release.
Part of the reason I think nsm-console is neat is the ease of automation you can do using simple text files. For instance, if you created a text file called “automate.txt” and put the following lines in it:
file /pcap/data.pcap
output automated-output
toggle aimsnarf
toggle tcpdstat
toggle chaosreader
run
quit
Then, you can run the command:
./nsm < automate.txt > output.txt
Which will run all the commands in the text file automatically, placing all the output in output.txt, simple eh?
If you have any questions, comments or suggestions, feel free to leave a comment or send an email I’d love to hear if/how you’re using nsm-console
Network traffic IP Location aggregator (iploc)
Have you ever been looking through your pcap files (or live captures) and wondered where all the traffic was coming from (or going to)? I have! Well, I’ve written a small (< 150 lines) script to aggregate all of the packet source addresses into a neatly separated CSV (comma-separated values) file. It includes
<ip address>,<country>,<city and state>,<latitude>,<longitude>,<packet count>
First off, get the script here.
It requires ruby-pcap and wget (like most of the Ruby scripts I write 
). Each unique ip address will have the script query the hostip.info database, storing the info in a temp file. If the ip already exists in the script’s list, it won’t query for the information (to save time and bandwidth).
iploc can either run in live capture mode, or read from a pcap file. Here’s how you would run it live:
./iploc -i eth0 >> output.csv
iploc will run until it receives a CTRL-C, at which point it will aggregate the data and dump it to STDOUT. Alternatively, you can read from a pcap file like this:
./iploc -r ~/data.pcap >> output.csv
Here’s an example of what the CSV file looks like after being opened in Numbers (or Excel):
Since it’s a CSV file, it should be easy to parse for any other program. I’m planning on hopefully writing another program to take out the latitude/longitude and plot the points on a map (for a more visual representation).
IPloc also supports BPF filters, just use it like you would tcpdump, for instance, if you only cared about http data:
./iploc -i eth0 port 80 > output.csv
Questions? Leave me a comment below! Now go forth, and visualize your honeypot data!
P.S. Thanks to the hostip guys for having a really nice web api to get this data quickly.
Compiling tcpdstat on Mac OSX (quick fix)
Quick fix for compiling tcpdstat on Mac OSX (Leopard, although probably works for Tiger too).
If you get this error:
cc -I. -I../libpcap-0.7.1 -DLINUX -D__FAVOR_BSD -D_LARGEFILE_SOURCE=1 -D_FILE_OFFSET_BITS=64 -L../libpcap-0.7.1 -c stat.c
cc -I. -I../libpcap-0.7.1 -DLINUX -D__FAVOR_BSD -D_LARGEFILE_SOURCE=1 -D_FILE_OFFSET_BITS=64 -L../libpcap-0.7.1 -c net_read.c
net_read.c:74:1: warning: "__FAVOR_BSD" redefined
<command line>:1:1: warning: this is the location of the previous definition
net_read.c:149: error: static declaration of ‘packet_length’ follows non-static declaration
tcpdstat.h:415: error: previous declaration of ‘packet_length’ was here
make: *** [net_read.o] Error 1
Edit the net_read.c file and change line 149 from this:
static int packet_length; /* length of current packet */
to this:
int packet_length; /* length of current packet */
Simple eh? Just type “make” again and tcpdstat should compile just fine. Simple fix.
NSM Console – A framework for running things
Well, I’ve been hard at work for the last couple of days working on a (hopefully) useful tool for aiding in NSM file analysis (for pcap files, live analysis doesn’t work).
Behold! I present NSM-Console! (read more about it here, watch a screencast here)
Download the framework here.
Keep in mind this framework only includes 3 modules (mostly used just for testing)
NSM-Console in a small (< 500 1000 1500 lines) framework for running nsm modules. Essentially, it’s a framework for running things (but we don’t call it that because it sounds like it wasn’t any work ). Here’s the breakdown: Read more
aimsnarf version 0.11 released
Yea yea, I know, it’s only been a few hours since the first release. Well, here’s the new release with a couple of major todos taken care of:
Download the script here.
Read about aimsnarf in the previous post about it.
Changes in this version:
- Trillian is now supported, as well as AOL’s AIM client. Most other clients should be supported too, I figured out the variable length/number of TLV fields in the packet, so aimsnarf is much smarter about decoding them
- Code cleaned up to be more readable
- Fixed some misc messages that were showing up, you still might see a few
Todos:
- Figure out what the heck iChat is doing, it doesn’t seem to be sending the same kind of data as all the other AIM clients
- Still do OTR stuff
- Maybe add support for different protocols?
- More testing!
If you find any bugs, send me a note or leave a comment. If you really want to help, you can send me some pcap data to analyze If you have any feature requests, lemme know!
Introducing ‘aimsnarf.rb’ => A simple AIM sniffing tool written in Ruby
[UPDATE 11/13/07] : version 0.11 released
Firstly, download the script here.
aimsnarf.rb is a small (~200 lines) Ruby script that I’ve written to sniff and dump AOL IM messages to STDOUT. I wrote this an as alternative to aimsniff, because I really dislike having to install aimsniff and all of it’s dependancies when all I want is a simple text transcript. I really felt like the dsniff toolkit should have had something like this (they already have urlsnarf, filesnarf, etc) to be used for penetration testing.
The only thing aimsnarf requires is Ruby and the ruby-pcap library (which is waaay easier to install than the 10+ CPAN modules that aimsniff requires). After installing the pcap library, simply run aimsnarf.rb on the console, here’s the usage:
Use '-h' to display usage
Usage: aimsnarf.rb [ -dnv ] [ -i interface | -r file ] [ -c count ] [ -s snaplen ] [ filter ]
Options:
-n do not convert address to name
-d debug mode
-v verbose mode
Due to the way that ruby-pcap works, I don’t have control over the usage displayed, currently the only real options you should mess with are ‘-i interface‘ and ‘-r file‘, changing anything else might produce “unknown” consequences . If you want to see hex dumps of the AIM data, edit the script and change the line “ap.data_debug(0)” to be “ap.data_debug(1)“, this will display the hex data as it is received.
Ignore the “pcap.bundle: warning: do not use Fixnums as Symbols” warnings you get when you run the program, the warning lies with the ruby-pcap library, so it’s out of my hands to fix. When run correctly, you should see something like this:
****** --> <you>: <HTML>what're you up to?</HTML>
<you> --> ******: <HTML>doing some stuff</HTML>
****** --> <you>: <HTML>awesome</HTML>
<you> --> ******: <HTML>talkity talk talk</HTML>
etc, etc
“******” will be the screen name of the person that’s talking. Yes, AIM sends the HTML tags, I don’t put those on.
Tangent:
Let’s talk a little bit about how much I hate the AIM protocol 
Take a look at the protocol listing as given from ethereal, you can see that each AIM packet actually holds a pretty good amount of information, turns out, AOL decided to make a ton of their fields variable length, which means a headache for me in decoding it, because the length has to be read, translated, then used to set the offset for reading the data, this is the reason the code for the script is incredibly messy, I plan on cleaning it up at a later time. In a future post, I’ll also go into more detail about how this particular script decodes the protocol (very much hackish at the moment).
Known Issues:
- Messages received by people who are away don’t get intercepted due to the packet being different than a regular incoming message packet
- Different clients might not work (depending on the features supported). Right now I’ve tested with GAIM/Pidgin and Adium, it looks like Trillian isn’t working correctly yet, although I’ve collected some data for analysis so I can get it working.
- This is probably the first *useful* script I’ve written in Ruby. I am not a ruby master so the code is really messy and probably badly written, have a problem with it? Send a patch!
- OTR encrypted chat interception doesn’t work (duh)
TODO (no particular order):
- Clean up code to make it easier to extend to different protocol/clients
- Fix the Trillian problem
- Test with AOL’s AIM client
- Fix the incoming/away message
- Correctly detect OTR chat and do (something?) about it
Remember people, don’t send credit card numbers, social security numbers, passwords, PIN numbers, etc over IM, ESPECIALLY when you’re somewhere like a coffeeshop using public wifi.
Thanks to the HeX LiveCD team for putting out a great release, already having the tools installed for use in a system is super helpful
Questions? Problems? Patches? Hatemail? Email me or leave a comment below!
-
My links on del.icio.us
Tumblr
Look me up on Twitter
Get a copy of my gpg public key
AIM: thnetos
Yahoo: okenezak
Join me on irc in #rawpacket on Freenode! 
Twitter feed:- thnetos: @technomancy The interesting parts of your setup are on Amazon? Is that the right link?
- thnetos: @fiftyfootshadow minor bug report for youdisappear.net, http://i.imgur.com/JOGw0.png
- thnetos: @stuartsierra GNU Screen? :)
- thnetos: @fogus Yea, I was underwhelmed by it also.
- thnetos: Consider me... SO TROLLED: http://pastie.org/1110154 /sarcasm. Warning: minor language.
- On Tumblr
- Latest del.icio.us links:
- Recet last.fm tracks
Subscribe in an RSS reader