:wq - blog » packet

NSM-Console version 0.6 release

Lee — Fri, 14 Mar 2008 19:50:29 +0000

I’m happy to announce the release of the next version of NSM-Console. Version 0.6. If you are unfamiliar with NSM-Console, here’s the synopsis from the project page:

NSM-Console (Network Security Monitoring Console) is a framework for performing analysis on packet capture files. It implements a modular structure to allow for an analyst to quickly write modules of their own without any programming language experience. Using these modules a large amount of pcap analysis can be performed quickly using a set of global (as well as per-module) options. It aims to be simple to run and easy to understand without a lot of learning time.

The recently posted NSM-Console whitepaper is a good place to start if you want an introduction to NSM-Console.

You can download NSM-Console 0.6 from the project page.

Here are the highlights for the changes in this version, check the TODO and CHANGELOG for a full list of changes.

The dump command
The dump command lets you dump either the payload (in ascii format), or the full packet (with the -f flag) to a file, you can specify ranges like 1-20, 5,7,9-* and 1-* just like you can with the print command, extremely useful for picking and choosing packets to dump to a file. Here’s an example:

nsm> dump 1-5,10-* file.txt Writing (append) packet(s) 1 through 5 from /Users/hinmanm/data.pcap to file.txt... Writing (append) packet(s) 10 through * from /Users/hinmanm/data.pcap to file.txt...

And another of the full dump

nsm> dump -f 20,56-59 full.out New args: 20,56-59 full.out Dumping full packet, not just payload... Writing (append) packet(s) 20 through 20 from /Users/hinmanm/data.pcap to full.out... Writing (append) packet(s) 56 through 59 from /Users/hinmanm/data.pcap to full.out...

Performance improvements for harimau and checkip
The Harimau module, as well as the checkip command, now use wget (if it’s installed) instead of Ruby’s HTTP request library, this results is a much much faster query time. Another performance tweak was that reverse dns lookups for most modules is now turned off by default because it was taking extremely long for large pcaps

New modules: clamscan, foremost, trace-summary
New modules have been included in this release, clamscan, which scans extracted files (from either the tcpxtract or foremost modules) for viruses, foremost, which is another file extractor and trace-summary, which provides an extremely nice summary of traffic in the pcap

NSM-Console handle’s gzip’d files
You can now specify a gzip’d file directly to NSM-Console, it will attempt to decompress the file into a temporary file when it is loaded. Note that directories of gzip’d files is still not supported entirely.

Bro-ids and Snort changes
Bro-IDS has been split into 2 modules, one for protocol analysis and the other for generating content streams.
Snort has had its community and emerging rules updated to the latest revisions.

As always, bugfixes and minor enhancements, check out the full changelog for a list of what’s changed. If you have any requests for additional features or modules, be sure to leave a command and let me know

Also, if anyone out there wants to see what my ~/.nsmcrc looks like, you can find it here.

Flowtime – Create a timeline for packet flow

Lee — Thu, 24 Jan 2008 16:36:28 +0000

You can never have too many tools for pcap visualization

Flowtime is a script written in Ruby that produces a timeline of the network flows in a pcap file. Everything is better with a picture, so here’s a picture: (warning, this picture is 3000×2000 pixels, kind of large)

Each bar on the left is a IP address along with a port, the timeline on the bottom is the time it was capture (in seconds). The different colors represent different kinds of traffic, http is blue, ssl is red, yellow is other, etc. At the moment there isn’t any legend, but I already think it’s useful just to see patterns in the traffic over time.

Download Flowtime here.

Requirements to run Flowtime:

Argus (version 3 only)
Ploticus (you should have ‘pl’ in your path. You may have to symlink ‘ploticus’ to ‘pl’)
EasyTimeline (you should have ‘EasyTimeline’ in your path)

Yea, I know, lots of dependencies, I’m hoping to rewrite this into a MUCH better version in the future, consider this the prototype

Here’s how to run it:

flowtime [-w #] [-h #] [-g] [--help] -w specify the width, default: 2000 -h specify the height, default: 2000 -g automatically try generate a png (requires 'EasyTimeline' and 'pl' in path) the packet file to generate a graph of source address to generate a graph for, 'all' for all IPs basename for the output file

To generate a basic graph, just do something like ‘flowtime -g data.pcap all data-out‘. After running this (if everything works okay), you should have a few files in your directory, if you open data-out.png you should be able to see the data as a timeline. If there are errors instead of data-out.png, there will be a file called data-out.err.

I know this script has issues, isn’t very user-friendly and doesn’t always work, here’s what I’m hoping to improve for the next version:

Generate an image of the entire timeline, in addition to showing it in a window
The ability to zoom in to a particular range of time to show only that time
Export an image of the current view
Select a bar to get more information about that flow

I’m going to have to use a different framework though, I’m considering Tk, but I haven’t ever done anything with it before.

Questions? Concerns? Comments? Suggestions?

P.S. I already made an NSM-Console module for flowtime too

NSM-Console version 0.4 release

Lee — Thu, 17 Jan 2008 00:47:01 +0000

Well, it has barely been any length of time and there’s already a new release of NSM-Console, there are so many features that I’ve been coding like crazy to get them all done. First, let’s start with the downloading:

http://writequit.org/projects/nsm-console/files/nsm-console-0.4.tar.gz

And, for anyone interested, here’s a rundown of the most notable new features:

Additional encoding/decoding options
You can now do uuencode and uudecode using encode and decode. In addition I’ve added octal and char decoding, more to come in the future!

The ‘print’ command (or just ‘p’)
NSM-Console now supports reading and printing pcap file connections as well as payloads in a variety of multiple formats. If you use the ‘print’ or ‘p’ command without any arguments, usage is displayed. You can print just connection information, or you can print the payload in either ascii or hex. The print command also supports ranges, *s and commas. For example, all of these are valid commands:
p -x 100 p -h 10-15 p 100-* p -x 10,53-64,102,2037-*
To see the print command in more action, take a look at how it is used to decode the sans packet challenge.

Color terminal text
Yep, nsm-console now supports color to brighten up your packet analysis experience If you are too dull to enjoy color (or don’t have a terminal that supports it), you can turn it off by using the ‘color off’ command.

Added modules
I’m slowing down for finding modules to easily add, but I still managed to add 2: tcptrace and tcpick. Note that tcpick isn’t installed by default on a Hex 1.0.2 install, so you’ll need to install it yourself. In order to view the graphs generated by tcptrace, you’ll need to install xplot too. (Hopefully these will be included in the next version of Hex)

~/.nsmcrc configuration file
When NSM-Console starts up, it will now check to see if the ~/.nsmcrc file exists, if it does, it will read through it and execute the commands found. For example:

[hinmanm@Euclid] $ cat ~/.nsmcrc color off eval $PROMPT="#{$GREEN}nsm#{$RESET}> " # This is a comment set honeysnap HOST_LIST 192.168.1.101,192.168.1.102

would set the color to be off, change the prompt to be green (yea, I know, a contradiction, it’s just an example) and set up some of our honeysnap options. Any line starting with a “#” will be ignored as a comment. This should help with some of the tediousness of having the set the same options every time you start up nsm-console.

Bugfixes, always bugfixes
Who doesn’t like bugfixes?

You can see the full list of all the changes in this version here, as well as future TODOs here. I’d like to give a big thanks to Scholar for letting me use his pcap parsing library, this way nsm-console doesn’t depend on any external libraries. Thanks Scholar!

I’ve also set up an NSM-Console wiki page over on the trac, in case you’re having trouble finding any of the information or want to download an older release.

Decoding the SANS Christmas packet challenge using only NSM-Console

Lee — Sat, 12 Jan 2008 00:12:30 +0000

In my never-ending quest to find justification for writing NSM-Console, I hereby present the following tutorial on how to decode the SANS Christmas packet challenge using nothing but NSM-Console:

I’m going to be using NSM-Console version 0.4-DEVEL, which adds the features that allow this analysis to be performed without external tools. You can get the development version here. Alright, let’s get this party started:

First things first, the fellows at SANS point you to the first packet in the xmas_Starter.pcap file, so let’s load up NSM-Console with the packet capture

./nsm ~/xmas_Starter.pcap

Next, let’s do a printout of all the packets in this dump (since it’s a small file, there shouldn’t be too many)

nsm> p * Args: * Filename: /Users/hinmanm/xmas_Starter.pcap list from 1 to * 1 1198471642.61773 192.168.25.1 -> 192.168.25.255 UDP 138 > 138 Len=243 2 1198471662.79806 192.168.25.100 -> 192.168.25.128 TCP 7337 > 1000 Len=254 3 1198471662.79813 192.168.25.128 -> 192.168.25.100 TCP 1000 > 7337 Len=58 4 1198471662.79877 192.168.25.100 -> 192.168.25.128 TCP 7337 > 1000 Len=60 5 1198471663.79691 192.168.25.100 -> 192.168.25.128 TCP 7337 > 1000 Len=254 6 1198471663.79697 192.168.25.128 -> 192.168.25.100 TCP 1000 > 7337 Len=58
… etc, etc, etc, to a total of 25 packets

Well, the SANS guide says to start at packet #1, so let’s take a look at it

nsm> p -x 1
(see image for output)

Whoops, this doesn’t look like useful output, maybe they meant the first TCP packet? (Instead of UDP). Looks like the first TCP packet is #2, so let’s look at that one:

nsm> p -x 2 Args: 2 Filename: /Users/hinmanm/xmas_Starter.pcap full from 2 to 2 2 1198471662.79806 192.168.25.100 -> 192.168.25.128 TCP 7337 > 1000 Len=254 0010 53 57 34 67 64 47 68 6c 49 47 31 76 64 6d 6c 6c SW4gdGhlIG1vdmll 0020 49 45 45 67 51 32 68 79 61 58 4e 30 62 57 46 7a IEEgQ2hyaXN0bWFz 0030 49 45 4e 68 63 6d 39 73 4c 43 42 6f 62 33 63 67 IENhcm9sLCBob3cg 0040 62 57 46 75 65 53 42 75 61 57 64 6f 64 43 68 7a bWFueSBuaWdodChz 0050 4b 53 42 6b 61 57 51 67 64 47 68 6c 49 48 52 6f KSBkaWQgdGhlIHRo 0060 63 6d 56 6c 49 48 4e 77 61 58 4a 70 64 48 4d 67 cmVlIHNwaXJpdHMg 0070 59 32 39 74 5a 53 42 30 62 79 42 32 61 58 4e 70 Y29tZSB0byB2aXNp 0080 64 44 38 3d 00 00 00 00 00 00 00 00 00 00 00 00 dD8=............ 0090 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00a0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00c0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................

Ahh, that looks like more readable output, due to the string ending with “=”, it’s most likely encoded in base64 (since = is used for padding). Let’s output the string in just plain ascii so we can decode it easier

nsm> p -a 2 Args: 2 Filename: /Users/hinmanm/xmas_Starter.pcap ascii from 2 to 2 2 1198471662.79806 192.168.25.100 -> 192.168.25.128 TCP 7337 > 1000 Len=254 SW4gdGhlIG1vdmllIEEgQ2hyaXN0bWFzIENhcm9sLCBob3cgbWFueSBuaWdodChz KSBkaWQgdGhlIHRocmVlIHNwaXJpdHMgY29tZSB0byB2aXNpdD8=...................... ..............................................................

(I put line breaks in so it wouldn’t distort the page)
Okay, let’s take this line and decode it to get our first clue

nsm> decode base64 SW4gdGhlIG1vdmllIEEgQ2hyaXN0bWFzIENhcm9sLCBob3cgbWFueSBuaWdodChz KSBkaWQgdGhlIHRocmVlIHNwaXJpdHMgY29tZSB0byB2aXNpdD8= Decoding base64 --> ascii... Output ([]'s added to show beginning and end):

[In the movie A Christmas Carol, how many night(s) did the three spirits come to visit?]

Alright! Got the clue! Now let’s load up our answer file and get the next one

nsm> file /Users/hinmanm/xmas_challenge_2007.pcap Setting ${PCAP_FILE} = /Users/hinmanm/xmas_challenge_2007.pcap Setting ${PCAP_BASE} = xmas_challenge_2007.pcap

Since the 3 spirits came on 1 night, we know our next answer is in packet #1, so let’s take a look

nsm> p -a 1 Args: 1 Filename: /Users/hinmanm/xmas_challenge_2007.pcap ascii from 1 to 1 1 1194153111.12232 192.168.25.100 -> 192.168.25.128 TCP 7337 > 1000 Len=154 QWxsIEkgd2FudCBmb3IgQ2hyaXN0bWFzIGlzIG15IF9fX18gRnJvbnQgVGVldG gu....................................

Aha! Another base64 encoding, let’s decode it

nsm> decode base64 QWxsIEkgd2FudCBmb3IgQ2hyaXN0bWFzIGlzIG15 IF9fX18gRnJvbnQgVGVldGgu Decoding base64 --> ascii... Output ([]'s added to show beginning and end):

[All I want for Christmas is my ____ Front Teeth.]

Alright, I think this shows the basic idea, and since this post is to illustrate some of the new features of NSM-Console (rather than the solution to the puzzle), let’s skip on ahead.

Packet 2 leads you to packet 3
Packet 3 leads you to packet 9
Packet 9 leads you to packet 11
Packet 11 leads you to packet 12
Packet 12 leads you to packet 359

Here’s where things start to get a little more interesting, printing out packet 359 shows that it doesn’t look like it’s encoded base64 anymore, in fact it looks like it’s urlescaped

nsm> p -x 359 Args: 359 Filename: /Users/hinmanm/xmas_challenge_2007.pcap full from 359 to 359 359 1194153771.83615 192.168.25.100 -> 192.168.25.128 TCP 7337 > 1000 Len=154 0010 38 37 25 32 30 31 30 31 25 32 30 4e 55 4c 4c 25 87%20101%20NULL% 0020 32 30 31 31 39 25 32 30 31 30 35 25 32 30 31 31 20119%20105%2011 0030 35 25 32 30 31 30 34 25 32 30 4e 55 4c 4c 25 32 5%20104%20NULL%2 0040 30 31 32 31 25 32 30 31 31 31 25 32 30 31 31 37 0121%20111%20117 0050 25 32 30 4e 55 4c 4c 25 32 30 39 37 25 32 30 4e %20NULL%2097%20N 0060 55 4c 4c 25 32 30 37 37 25 32 30 31 30 31 25 32 ULL%2077%20101%2

Also, the SANS challenge mentions that the message may or may not be in multiple packets, so let’s check the next one

nsm> p -x 360 Args: 360 Filename: /Users/hinmanm/xmas_challenge_2007.pcap full from 360 to 360 360 1194153772.83062 192.168.25.100 -> 192.168.25.128 TCP 7337 > 1000 Len=154 0010 25 32 30 31 31 34 25 32 30 31 32 31 25 32 30 4e %20114%20121%20N 0020 55 4c 4c 25 32 30 36 37 25 32 30 31 30 34 25 32 ULL%2067%20104%2 0030 30 31 31 34 25 32 30 31 30 35 25 32 30 31 31 35 0114%20105%20115 0040 25 32 30 31 31 36 25 32 30 31 30 39 25 32 30 39 %20116%20109%209 0050 37 25 32 30 31 31 35 25 32 30 34 34 25 30 44 25 7%20115%2044%0D% 0060 30 41 38 37 25 32 30 31 30 31 25 32 30 4e 55 4c 0A87%20101%20NUL

Looks like a continuation, if you print out a few more, you find out that the data stops in packet #365, let’s print out all the packets so we can see what the data looks like

nsm> p -x 359-365
(see picture for output, too long to paste here)

Alright, looks like we’ve got our message, let’s decode it

nsm> p -a 359-365
gives us:

nsm> decode urlescape
shows:

That looks like decimal ascii values, let’s decode the values using the “char” decoding

nsm> decode char
Here’s the output from decoding each of the 3 lines:

And look! There’s our message (I replaced all the ‘NULL’s with spaces):

We wish you a Merry Christmas, We wish you a Merry Christmas, We wish you a Merry Christmas and a Happy New Year!!!

Merry Christmas to you too SANS, and thanks for the awesome challenge!

You can check out SANS’ solution for the challenge here.
You can check out Geek00l’s full solution for the challenge here.

If it wasn’t for this challenge, we might not have had the idea to include these features in NSM-Console, I’m glad we did and I hope it proves useful to the rest of the packet monkeys out there

P.S. A big thank-you to Scholar for letting me use his pcap parsing library, thus eliminating the dependency on any 3rd party libraries to do all the packet reading, thanks a bunch!

Have another idea for a feature that should be in NSM-Console? Shoot me an email or leave a comment!