NSM-Console version 0.6 release

nsmmonkeyI’m happy to announce the release of the next version of NSM-Console. Version 0.6. If you are unfamiliar with NSM-Console, here’s the synopsis from the project page:

NSM-Console (Network Security Monitoring Console) is a framework for performing analysis on packet capture files. It implements a modular structure to allow for an analyst to quickly write modules of their own without any programming language experience. Using these modules a large amount of pcap analysis can be performed quickly using a set of global (as well as per-module) options. It aims to be simple to run and easy to understand without a lot of learning time.

The recently posted NSM-Console whitepaper is a good place to start if you want an introduction to NSM-Console.

You can download NSM-Console 0.6 from the project page.

Here are the highlights for the changes in this version, check the TODO and CHANGELOG for a full list of changes.

The dump command
The dump command lets you dump either the payload (in ascii format), or the full packet (with the -f flag) to a file, you can specify ranges like 1-20, 5,7,9-* and 1-* just like you can with the print command, extremely useful for picking and choosing packets to dump to a file. Here’s an example:

nsm> dump 1-5,10-* file.txt
Writing (append) packet(s) 1 through 5 from /Users/hinmanm/data.pcap to file.txt...
Writing (append) packet(s) 10 through * from /Users/hinmanm/data.pcap to file.txt...

And another of the full dump

nsm> dump -f 20,56-59 full.out
New args: 20,56-59 full.out
Dumping full packet, not just payload...
Writing (append) packet(s) 20 through 20 from /Users/hinmanm/data.pcap to full.out...
Writing (append) packet(s) 56 through 59 from /Users/hinmanm/data.pcap to full.out...

Performance improvements for harimau and checkip
The Harimau module, as well as the checkip command, now use wget (if it’s installed) instead of Ruby’s HTTP request library, this results is a much much faster query time. Another performance tweak was that reverse dns lookups for most modules is now turned off by default because it was taking extremely long for large pcaps

New modules: clamscan, foremost, trace-summary
New modules have been included in this release, clamscan, which scans extracted files (from either the tcpxtract or foremost modules) for viruses, foremost, which is another file extractor and trace-summary, which provides an extremely nice summary of traffic in the pcap

NSM-Console handle’s gzip’d files
You can now specify a gzip’d file directly to NSM-Console, it will attempt to decompress the file into a temporary file when it is loaded. Note that directories of gzip’d files is still not supported entirely.

Bro-ids and Snort changes
Bro-IDS has been split into 2 modules, one for protocol analysis and the other for generating content streams.
Snort has had its community and emerging rules updated to the latest revisions.

As always, bugfixes and minor enhancements, check out the full changelog for a list of what’s changed. If you have any requests for additional features or modules, be sure to leave a command and let me know :)

Also, if anyone out there wants to see what my ~/.nsmcrc looks like, you can find it here.

March 14, 2008 | Filed Under 0.6, bro-ids, clamscan, dump, foremost, gzip, harimau, monitoring, network, nsm, nsm console, packet, pcap, security, snort, trace-summary, whitepaper | Leave a Comment 

Flowtime – Create a timeline for packet flow

You can never have too many tools for pcap visualization :)

Flowtime is a script written in Ruby that produces a timeline of the network flows in a pcap file. Everything is better with a picture, so here’s a picture: (warning, this picture is 3000×2000 pixels, kind of large)

ink2

Each bar on the left is a IP address along with a port, the timeline on the bottom is the time it was capture (in seconds). The different colors represent different kinds of traffic, http is blue, ssl is red, yellow is other, etc. At the moment there isn’t any legend, but I already think it’s useful just to see patterns in the traffic over time.

Download Flowtime here.

Requirements to run Flowtime:

Yea, I know, lots of dependencies, I’m hoping to rewrite this into a MUCH better version in the future, consider this the prototype :)

Here’s how to run it:

flowtime [-w #] [-h #] [-g] [--help] <pcapfile> <ipaddr> <outfile_base>
-w specify the width, default: 2000
-h specify the height, default: 2000
-g automatically try generate a png (requires 'EasyTimeline' and 'pl' in path)
<pcapfile> the packet file to generate a graph of
<ipaddr> source address to generate a graph for, 'all' for all IPs
<outfile_base> basename for the output file

To generate a basic graph, just do something like ‘flowtime -g data.pcap all data-out‘. After running this (if everything works okay), you should have a few files in your directory, if you open data-out.png you should be able to see the data as a timeline. If there are errors instead of data-out.png, there will be a file called data-out.err.

I know this script has issues, isn’t very user-friendly and doesn’t always work, here’s what I’m hoping to improve for the next version:

I’m going to have to use a different framework though, I’m considering Tk, but I haven’t ever done anything with it before.

Questions? Concerns? Comments? Suggestions?

P.S. I already made an NSM-Console module for flowtime too ;)

January 24, 2008 | Filed Under argus, easytimeline, flow, flowtime, packet, pcap, plot, ploticus, ruby, script, timeline, visualization | 4 Comments 

NSM-Console version 0.4 release

smallmonkeyWell, it has barely been any length of time and there’s already a new release of NSM-Console, there are so many features that I’ve been coding like crazy to get them all done. First, let’s start with the downloading:

http://writequit.org/projects/nsm-console/files/nsm-console-0.4.tar.gz

And, for anyone interested, here’s a rundown of the most notable new features:

Additional encoding/decoding options
You can now do uuencode and uudecode using encode and decode. In addition I’ve added octal and char decoding, more to come in the future!

The ‘print’ command (or just ‘p’)
NSM-Console now supports reading and printing pcap file connections as well as payloads in a variety of multiple formats. If you use the ‘print’ or ‘p’ command without any arguments, usage is displayed. You can print just connection information, or you can print the payload in either ascii or hex. The print command also supports ranges, *s and commas. For example, all of these are valid commands:
p -x 100
p -h 10-15
p 100-*
p -x 10,53-64,102,2037-*
To see the print command in more action, take a look at how it is used to decode the sans packet challenge.

Color terminal text
Yep, nsm-console now supports color to brighten up your packet analysis experience :) If you are too dull to enjoy color (or don’t have a terminal that supports it), you can turn it off by using the ‘color off’ command.

Added modules
I’m slowing down for finding modules to easily add, but I still managed to add 2: tcptrace and tcpick. Note that tcpick isn’t installed by default on a Hex 1.0.2 install, so you’ll need to install it yourself. In order to view the graphs generated by tcptrace, you’ll need to install xplot too. (Hopefully these will be included in the next version of Hex)

~/.nsmcrc configuration file
When NSM-Console starts up, it will now check to see if the ~/.nsmcrc file exists, if it does, it will read through it and execute the commands found. For example:

[hinmanm@Euclid] $ cat ~/.nsmcrc
color off
eval $PROMPT="#{$GREEN}nsm#{$RESET}> "
# This is a comment
set honeysnap HOST_LIST 192.168.1.101,192.168.1.102

would set the color to be off, change the prompt to be green (yea, I know, a contradiction, it’s just an example) and set up some of our honeysnap options. Any line starting with a “#” will be ignored as a comment. This should help with some of the tediousness of having the set the same options every time you start up nsm-console.

Bugfixes, always bugfixes
Who doesn’t like bugfixes?

You can see the full list of all the changes in this version here, as well as future TODOs here. I’d like to give a big thanks to Scholar for letting me use his pcap parsing library, this way nsm-console doesn’t depend on any external libraries. Thanks Scholar!

I’ve also set up an NSM-Console wiki page over on the trac, in case you’re having trouble finding any of the information or want to download an older release.

January 16, 2008 | Filed Under analysis, framework, hex, monitoring, monkey, network, nsm, nsm console, packet, ruby, script, security, tcpick, tcptrace | 3 Comments 

Decoding the SANS Christmas packet challenge using only NSM-Console

In my never-ending quest to find justification for writing NSM-Console, I hereby present the following tutorial on how to decode the SANS Christmas packet challenge using nothing but NSM-Console:

I’m going to be using NSM-Console version 0.4-DEVEL, which adds the features that allow this analysis to be performed without external tools. You can get the development version here. Alright, let’s get this party started:

First things first, the fellows at SANS point you to the first packet in the xmas_Starter.pcap file, so let’s load up NSM-Console with the packet capture

./nsm ~/xmas_Starter.pcap

Next, let’s do a printout of all the packets in this dump (since it’s a small file, there shouldn’t be too many)

Read more

January 11, 2008 | Filed Under analysis, base64, challenge, christmas, console, decode, encode, fun, geek00l, geekery, hex, nsm, nsm console, packet, ruby, sans, terminal, urlescape | 2 Comments 

Bad Behavior has blocked 250 access attempts in the last 7 days.