:wq - blog » console

NSM-Console version 0.7 release

Lee — Sun, 27 Apr 2008 18:19:34 +0000

First off, I apologize for the lack of posts here lately, I’ve been trying to come up with something good to post, because I’m just not a fan of rehashing things other blogs post, or commenting on news stories. Hopefully I’ll be able to contribute more soon

Now down to the real post, NSM-Console 0.7 has been released, there are a lot of cool features in this release, but first, go download NSM-Console!

As always, you can check out the TODO and CHANGELOG from svn.

Now, let’s cover some of the newest features in this release:

Encode/Decode enhancements
The encode and decode methods have had a few enhancements added to them, most notably, you can now specify a file to encode or decode, instead of specifying just a string, so you could do:

nsm> encode -f base64 testfile.txt Encoding ascii --> base64... Output ([]'s added to show beginning and end): [TlNNLUNvbnNvbGUgaXMgYXdlc29tZSwgeW91IHNob3VsZCB1c2UgaXQgOikK]

Also, you can specify a variety of hex encodings, because I was noticing that it was delineated in a variety of ways, \x, space (or not delineated at all). I’ve also added the default hex and binary methods, so you don’t have to specify endianness, they default to little-endian.

IP->ASN mapping
As per Scholar’s suggestion, there is now both a module and a command for translating an ip into it’s ASN, you can either use the module to get a listing from each IP in the pcap, or use the below command to get the ASN for just one address:

nsm> ip2asn 203.223.154.86 Bulk mode; whois.cymru.com [2008-04-27 17:53:32 +0000] 17992 | 203.223.154.86 | AIMS-MY-DIA-AS AIMS Data Centre

Thanks to Team Cymru for their ASN servers

‘Print’ command supports flags
The print command now supports printing TCP flags, still uses Scholar’s pcapparser library.

New command: ‘iplist’
Generate a list of the ips in a pcap file, sorted by the number of occurrences in the file, see below:

nsm> iplist === IP list for data.pcap === 192.168.1.123 1507 64.233.179.109 260 192.168.1.136 141 204.245.162.17 126 216.178.38.133 102 208.67.217.230 92 209.225.0.103 88 .. etc etc

Pipes now supported
One feature geek00l has been bugging me about is getting piping to work in the nsm shell, I’m happy to announce that they finally work, you might run into a few bugs (broken pipes with less), but for the most part they work, now output can be piped into files and programs:

nsm> p -x 1-* | less (display all the packets and hex output, piped into less) nsm> iplist > iplist.txt (output the list of ips into iplist.txt) nsm> ip2asn 203.223.154.86 >> iplist.txt (append the ip2asn output to iplist.txt)

Etc, etc, you get the idea. The ‘<‘ pipe hasn’t been implemented yet, perhaps if it’s needed in the future.

New modules, bro-ids-connection and yahsnarf
Geek00l committed his bro-ids-connection module for generating only connection information from a pcap, a yahsnarf module was also committed, to enable extract yahoo IM conversations from a pcap file. Thanks geek00l!

Automatic updating of NSM-Console
Users desiring to be on the bleeding edge of NSM-Console development (is there anyone that actually desires this? :P) can now use the “update” command from within NSM-Console to automatically update from the latest subversion commit. You can also use the -v for verbose output, see below:

nsm> update -v Updating NSM-Console from svn... Fetching newest revision from svn... etc, etc

Still a few kinks to work out, but should work pretty well.

Bugfixes
I fixed some bugs related to gzip’d pcap files as well as some bugs in the encode and decode methods. I also introduced some bugs (hurray!) with pipes, but it’s still usable.

Like I always say, check out the full TODO and CHANGELOG for complete details, and send me any feedback you have

Decoding the SANS Christmas packet challenge using only NSM-Console

Lee — Sat, 12 Jan 2008 00:12:30 +0000

In my never-ending quest to find justification for writing NSM-Console, I hereby present the following tutorial on how to decode the SANS Christmas packet challenge using nothing but NSM-Console:

I’m going to be using NSM-Console version 0.4-DEVEL, which adds the features that allow this analysis to be performed without external tools. You can get the development version here. Alright, let’s get this party started:

First things first, the fellows at SANS point you to the first packet in the xmas_Starter.pcap file, so let’s load up NSM-Console with the packet capture

./nsm ~/xmas_Starter.pcap

Next, let’s do a printout of all the packets in this dump (since it’s a small file, there shouldn’t be too many)

nsm> p * Args: * Filename: /Users/hinmanm/xmas_Starter.pcap list from 1 to * 1 1198471642.61773 192.168.25.1 -> 192.168.25.255 UDP 138 > 138 Len=243 2 1198471662.79806 192.168.25.100 -> 192.168.25.128 TCP 7337 > 1000 Len=254 3 1198471662.79813 192.168.25.128 -> 192.168.25.100 TCP 1000 > 7337 Len=58 4 1198471662.79877 192.168.25.100 -> 192.168.25.128 TCP 7337 > 1000 Len=60 5 1198471663.79691 192.168.25.100 -> 192.168.25.128 TCP 7337 > 1000 Len=254 6 1198471663.79697 192.168.25.128 -> 192.168.25.100 TCP 1000 > 7337 Len=58
… etc, etc, etc, to a total of 25 packets

Well, the SANS guide says to start at packet #1, so let’s take a look at it

nsm> p -x 1
(see image for output)

Whoops, this doesn’t look like useful output, maybe they meant the first TCP packet? (Instead of UDP). Looks like the first TCP packet is #2, so let’s look at that one:

nsm> p -x 2 Args: 2 Filename: /Users/hinmanm/xmas_Starter.pcap full from 2 to 2 2 1198471662.79806 192.168.25.100 -> 192.168.25.128 TCP 7337 > 1000 Len=254 0010 53 57 34 67 64 47 68 6c 49 47 31 76 64 6d 6c 6c SW4gdGhlIG1vdmll 0020 49 45 45 67 51 32 68 79 61 58 4e 30 62 57 46 7a IEEgQ2hyaXN0bWFz 0030 49 45 4e 68 63 6d 39 73 4c 43 42 6f 62 33 63 67 IENhcm9sLCBob3cg 0040 62 57 46 75 65 53 42 75 61 57 64 6f 64 43 68 7a bWFueSBuaWdodChz 0050 4b 53 42 6b 61 57 51 67 64 47 68 6c 49 48 52 6f KSBkaWQgdGhlIHRo 0060 63 6d 56 6c 49 48 4e 77 61 58 4a 70 64 48 4d 67 cmVlIHNwaXJpdHMg 0070 59 32 39 74 5a 53 42 30 62 79 42 32 61 58 4e 70 Y29tZSB0byB2aXNp 0080 64 44 38 3d 00 00 00 00 00 00 00 00 00 00 00 00 dD8=............ 0090 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00a0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00c0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................

Ahh, that looks like more readable output, due to the string ending with “=”, it’s most likely encoded in base64 (since = is used for padding). Let’s output the string in just plain ascii so we can decode it easier

nsm> p -a 2 Args: 2 Filename: /Users/hinmanm/xmas_Starter.pcap ascii from 2 to 2 2 1198471662.79806 192.168.25.100 -> 192.168.25.128 TCP 7337 > 1000 Len=254 SW4gdGhlIG1vdmllIEEgQ2hyaXN0bWFzIENhcm9sLCBob3cgbWFueSBuaWdodChz KSBkaWQgdGhlIHRocmVlIHNwaXJpdHMgY29tZSB0byB2aXNpdD8=...................... ..............................................................

(I put line breaks in so it wouldn’t distort the page)
Okay, let’s take this line and decode it to get our first clue

nsm> decode base64 SW4gdGhlIG1vdmllIEEgQ2hyaXN0bWFzIENhcm9sLCBob3cgbWFueSBuaWdodChz KSBkaWQgdGhlIHRocmVlIHNwaXJpdHMgY29tZSB0byB2aXNpdD8= Decoding base64 --> ascii... Output ([]'s added to show beginning and end):

[In the movie A Christmas Carol, how many night(s) did the three spirits come to visit?]

Alright! Got the clue! Now let’s load up our answer file and get the next one

nsm> file /Users/hinmanm/xmas_challenge_2007.pcap Setting ${PCAP_FILE} = /Users/hinmanm/xmas_challenge_2007.pcap Setting ${PCAP_BASE} = xmas_challenge_2007.pcap

Since the 3 spirits came on 1 night, we know our next answer is in packet #1, so let’s take a look

nsm> p -a 1 Args: 1 Filename: /Users/hinmanm/xmas_challenge_2007.pcap ascii from 1 to 1 1 1194153111.12232 192.168.25.100 -> 192.168.25.128 TCP 7337 > 1000 Len=154 QWxsIEkgd2FudCBmb3IgQ2hyaXN0bWFzIGlzIG15IF9fX18gRnJvbnQgVGVldG gu....................................

Aha! Another base64 encoding, let’s decode it

nsm> decode base64 QWxsIEkgd2FudCBmb3IgQ2hyaXN0bWFzIGlzIG15 IF9fX18gRnJvbnQgVGVldGgu Decoding base64 --> ascii... Output ([]'s added to show beginning and end):

[All I want for Christmas is my ____ Front Teeth.]

Alright, I think this shows the basic idea, and since this post is to illustrate some of the new features of NSM-Console (rather than the solution to the puzzle), let’s skip on ahead.

Packet 2 leads you to packet 3
Packet 3 leads you to packet 9
Packet 9 leads you to packet 11
Packet 11 leads you to packet 12
Packet 12 leads you to packet 359

Here’s where things start to get a little more interesting, printing out packet 359 shows that it doesn’t look like it’s encoded base64 anymore, in fact it looks like it’s urlescaped

nsm> p -x 359 Args: 359 Filename: /Users/hinmanm/xmas_challenge_2007.pcap full from 359 to 359 359 1194153771.83615 192.168.25.100 -> 192.168.25.128 TCP 7337 > 1000 Len=154 0010 38 37 25 32 30 31 30 31 25 32 30 4e 55 4c 4c 25 87%20101%20NULL% 0020 32 30 31 31 39 25 32 30 31 30 35 25 32 30 31 31 20119%20105%2011 0030 35 25 32 30 31 30 34 25 32 30 4e 55 4c 4c 25 32 5%20104%20NULL%2 0040 30 31 32 31 25 32 30 31 31 31 25 32 30 31 31 37 0121%20111%20117 0050 25 32 30 4e 55 4c 4c 25 32 30 39 37 25 32 30 4e %20NULL%2097%20N 0060 55 4c 4c 25 32 30 37 37 25 32 30 31 30 31 25 32 ULL%2077%20101%2

Also, the SANS challenge mentions that the message may or may not be in multiple packets, so let’s check the next one

nsm> p -x 360 Args: 360 Filename: /Users/hinmanm/xmas_challenge_2007.pcap full from 360 to 360 360 1194153772.83062 192.168.25.100 -> 192.168.25.128 TCP 7337 > 1000 Len=154 0010 25 32 30 31 31 34 25 32 30 31 32 31 25 32 30 4e %20114%20121%20N 0020 55 4c 4c 25 32 30 36 37 25 32 30 31 30 34 25 32 ULL%2067%20104%2 0030 30 31 31 34 25 32 30 31 30 35 25 32 30 31 31 35 0114%20105%20115 0040 25 32 30 31 31 36 25 32 30 31 30 39 25 32 30 39 %20116%20109%209 0050 37 25 32 30 31 31 35 25 32 30 34 34 25 30 44 25 7%20115%2044%0D% 0060 30 41 38 37 25 32 30 31 30 31 25 32 30 4e 55 4c 0A87%20101%20NUL

Looks like a continuation, if you print out a few more, you find out that the data stops in packet #365, let’s print out all the packets so we can see what the data looks like

nsm> p -x 359-365
(see picture for output, too long to paste here)

Alright, looks like we’ve got our message, let’s decode it

nsm> p -a 359-365
gives us:

nsm> decode urlescape
shows:

That looks like decimal ascii values, let’s decode the values using the “char” decoding

nsm> decode char
Here’s the output from decoding each of the 3 lines:

And look! There’s our message (I replaced all the ‘NULL’s with spaces):

We wish you a Merry Christmas, We wish you a Merry Christmas, We wish you a Merry Christmas and a Happy New Year!!!

Merry Christmas to you too SANS, and thanks for the awesome challenge!

You can check out SANS’ solution for the challenge here.
You can check out Geek00l’s full solution for the challenge here.

If it wasn’t for this challenge, we might not have had the idea to include these features in NSM-Console, I’m glad we did and I hope it proves useful to the rest of the packet monkeys out there

P.S. A big thank-you to Scholar for letting me use his pcap parsing library, thus eliminating the dependency on any 3rd party libraries to do all the packet reading, thanks a bunch!

Have another idea for a feature that should be in NSM-Console? Shoot me an email or leave a comment!

NSM-Console version 0.3 release

Lee — Wed, 09 Jan 2008 06:29:44 +0000

Yep, I’ve just been cranking out code lately, so I am proud to present the 0.3 release of nsm-console!

You can download NSM-Console here:

http://writequit.org/projects/nsm-console/files/nsm-console-0.3.tar.gz

This release was focused a bit more on usability, features and bugfixes rather than the addition of new modules, however, there were still a couple that were added. Since this release has some pretty big changes, let’s start by going over some of the notable ones:

- Logfile changes
I decided I wanted a way to track the exit status of each module command that was executed, so the logfile will now prepend each executed command with “[exit: #]” (# is the exit status of the command). I’m hoping this will help with module debugging.

- Output directory changes
The default output directory has been changed from “output” to “${PCAP_BASE}-output”, I’m hoping this helps to provide a more meaningful output directory name. Note that when you perform analysis on a directory of files, multiple output directories will be created, if you want all the output in 1 folder, change output not to have ${PCAP_BASE} in it.

- Modules
The following modules were added, use “info ” to get more information about a module: iploc, fl0p, argus

- e command
“e” is now a shortcut for exec, who likes typing that all out anyway?

- eval command
The “eval” command allows an analyst to evaluate a line of ruby, this is extremely powerful to anyone that knows ruby. Note that all the variables and methods I’ve written for nsm-console are available for use with eval also. Here are two examples:

nsm> eval 9 * 6 => 54

nsm> eval m = get_mod_by_name("aimsnarf"); puts m.get_commands => aimsnarf -r ${PCAP_FILE} > ${OUTPUT_DIR}/${OUTPUT_FILE}

- Better tab completion
I added things like “PCAP_FILE” and “OUTPUT_DIR” to the list of tab-completed words, there is a much larger list now. (If you really want a list you can use: “eval puts $tabstrings” to print them all out)

- encode/decode commands
Here are the two commands I think a lot of packet analysts are going to find extremely useful. They allow someone to easily translate from one encoding to a different encoding. Rather than explain with text, let me show you a screenshot:

Right now encode and decode support a handful of formats (see screenshot for the list), but I’m hard at work on many many more formats to encode and decode. Thanks go to Geek00l, who gave me the idea for this feature (especially how it would be useful in something like this). This should allow for quicker analysis as performing inline encoding and encoding should be much easier to perform. Take a look at the TODO file in the tarball and let me know if there is an encoding missing that should be available.

- Licensing
NSM-Console is now released under a LGPL version 2.1 license, you can read the license agreement in the LICENSE file or online here.

- Bugfixes
Whitespace handling fixes, better handling of commands not found, category reading fix and many more

- Code cleanup and organization
Cleanup is always good.

See the CHANGELOG file for more detail about what has changed. The TODO file lists features that I am currently working on. If you aren’t sure what nsm-console is, I recommend you watch the screencast I recently created.

As always, I welcome any feedback, comments, criticism, support and patches.

Screencast: An introduction to NSM-Console

Lee — Sun, 06 Jan 2008 04:23:56 +0000

Well, I’ve been working on this for the last week or so, trying to get it all working the way I wanted, and after around 15 takes, I finally have a screencast for anyone interested in the idea behind and usage of nsm-console.

The version of nsm-console used in the screencast is the 0.3-DEVEL version. UPDATE: Version 0.3 is now out!

The video is in .mov format and is 12 minutes and 40 seconds, it is around 17MB. Don’t forget to right-click and “Download As”!

mirror 1 [thanks enhanced]
mirror 1 w/ssl [thanks again enhanced]
mirror 2 [my server]

I’m hoping to have a flash version created soon, I’ll update this entry when I do.

If you have any questions, comments or criticisms, feel free to leave a comment below or email me.

I also updated the “about me” page if you absolutely must know what I look like.

Oh, one more thing, ignore the fact that I say “so” around 30 times in this one video, this is my first screencast, gimme a break.

NSM Console projected module list

Lee — Wed, 28 Nov 2007 21:43:28 +0000

Here’s a list of all the planned modules and completed (struck-out) modules for nsm-console: (if a module is struck out, it’s because I’ve finished making a module for it, it isn’t necessarily in the tarball for download)

aimsnarf
ngrep (gif/jpg/pdf/exe/pe/ne/elf/3pg/torrent)
tcpxtract
tcpflow
chaosreader
bro-IDS
snort
tcpdstat
capinfos
tshark
argus
ragator
racount
rahosts
hash (md5 & sha256)
ra
honeysnap
p0f
pads
fl0p
iploc
foremost – thanks shadowbq!
flowgrep
tcptrace
tcpick
flowtime
flowtag
harimau
clamscan

Think of any other useful modules? Leave me a comment and let me know!

P.S. I’m also brainstorming for some pcap/real-time network visualization tools, stay tuned!

NSM Console – A framework for running things

Lee — Wed, 28 Nov 2007 02:06:26 +0000

Well, I’ve been hard at work for the last couple of days working on a (hopefully) useful tool for aiding in NSM file analysis (for pcap files, live analysis doesn’t work).

Behold! I present NSM-Console! (read more about it here, watch a screencast here)

Download the framework here.
Keep in mind this framework only includes 3 modules (mostly used just for testing)

NSM-Console in a small (< 500 1000 1500 lines) framework for running nsm modules. Essentially, it’s a framework for running things (but we don’t call it that because it sounds like it wasn’t any work :P). Here’s the breakdown:

Users will run the ‘nsm’ program to enter into a console, by which they can select modules that have options to run against a packet capture file. Let’s talk a little bit about how it works. When nsm starts, it looks in the directory given by $MODULE_DIR (hardcoded at the moment, will be moved later) for all directories that follow the naming convention “.module”, where is the name of the module. Following so far?

For explanation’s sake, let’s call our module “aimsnarf”, after the aim-sniffing program I wrote a while back. So inside the module directory, there is another directory called “aimsnarf.module”. Inside the aimsnarf.module directory, there are a few files that define how our module will act. First is a file just called “aimsnarf”, this is the file that dictates commands to be run in the framework. It can have multiple commands separated by lines in the file. (If the module were called “bathtub”, the file would be called “bathtub” also, it’s named after the module name). Let’s take a look at an example line from the aimsnarf.module/aimsnarf file:

aimsnarf -r ${PCAP_FILE} > ${OUTPUT_DIR}/${OUTPUT_FILE}

In this line, you can see some variables that get changed out when the module is executed from nsm console. ${PCAP_FILE} is always set to the pcap file being analyzed, ${OUTPUT_DIR} will be set to the module-specific directory output will be directed into (set by nsm console), the last option, ${OUTPUT_FILE} is not set by the nsm console, it is a user-defined variable. But where is it defined? Let’s take a look at the next file: aimsnarf.module/defaults:

OUTPUT_FILE=${PCAP_BASE}.aim

The defaults file defines the user-settable options for the module, in this case, there is only 1, the “OUTPUT_FILE” option. The defaults file also provides a default value for that option (in this case, ${PCAP_BASE}.aim). The default value is provided so the module can be run without having to set every option. ${PCAP_BASE} is another variable that is provided by the nsm console, it is the basename of the pcap file (so it the pcap file was “~/pcap/file.pcap”, the ${PCAP_BASE} would be “file.pcap”).

The next file is aimsnarf.module/description, this file provides a 1-line description of the module to be listed when the user lists all the modules. Here is the ‘description’ for the aimsnarf module:

Extracts AIM messages from a pcap file

Simple eh? Still following? Okay, there is one more file, the aimsnarf.module/info file, this file provides a more detailed explanation of the module, it should also define what each option means, here is what aimsnarf’s ‘info’ file looks like:

Aimsnarf is a program to extract aim conversations from a pcap file

The following options are enabled: OUTPUT_FILE (default $PCAP_DATA}.aim) This option specifies the output file.

So how does this all work together? Well, first, you would start the NSM console you would see the following screen:

You can see that the aimsnarf module has been loaded (as well as a couple of others) and a prompt has been presented. From here there are a few different options, here’s the output of the ‘help’ command:

I think the help is pretty self-explanatory, hopefully it makes sense to everyone else as well.

Let’s run the aimsnarf module on a pcap file as an example. First, we need to specify a pcap file to work on with the ‘file’ command, so we type:

nsm> file /Users/hinmanm/data.pcap Setting ${PCAP_FILE} = /Users/hinmanm/data.pcap Setting ${PCAP_BASE} = data.pcap

Next, we will specify a different output directory (the default is ‘output’) using the ‘output’ command:

nsm> output data-output Setting ${OUTPUT_DIR} = data-output

Next, let’s make sure the only module that is going to run is the aimsnarf module:

nsm> toggle none All modules turned off. nsm> toggle aimsnarf aimsnarf module turned on.

And make sure our options are set correctly for the aimsnarf module:

nsm> options aimsnarf Global options: ----------------------------------- ${PCAP_FILE}: /Users/hinmanm/data.pcap ${PCAP_BASE}: data.pcap ${OUTPUT_DIR}: data-output ${MODULE_DIR}: modules Options for module aimsnarf: ----------------------------------- ${OUTPUT_FILE} = ${PCAP_BASE}.aim

That looks good to me! Now, let’s run and see what happens:

nsm> run Executing analysis... ===> module aimsnarf running... Creating directory data-output/aimsnarf Result: true --> aimsnarf -r /Users/hinmanm/data.pcap > data-output/aimsnarf/data.pcap.aim /Library/Ruby/Site/1.8/universal-darwin9.0/pcap.bundle: warning: do not use Fixnums as Symbols /Library/Ruby/Site/1.8/universal-darwin9.0/pcap.bundle: warning: do not use Fixnums as Symbols /Library/Ruby/Site/1.8/universal-darwin9.0/pcap.bundle: warning: do not use Fixnums as Symbols Result: true ===> module aimsnarf finished. ===> module hash skipped. ===> module ngrep skipped.

Looks like it ran the command! (ignore the warnings, they’re generated by aimsnarf, not nsm) Simple enough, right?

So here’s the real question: why write something like this? Well, I believe this tool will help to perform large amounts of analysis (think 20-30 plugins instead of 3) on packet capture files without having to manually manage each tool that does analysis. Instead of having to change a hardcoded script, you can now edit (and create) modules to do exactly what you need, but still having the flexibility of global varaibles (ie: ${OUTPUT_DIR}) and module specific variables (ie: ${OUTPUT_FILE} in aimsnarf). Being able to toggle each module on and off allows you to do a flexible amount of analysis on a file on the fly. In the future I hope to be able to extend the framework to run on a directory of files, allowing you to automate running a vast number of pcap analysis tools on a large number of pcap files very easily.

While this is not a replacement for running each command individually (and it shouldn’t be), I hope this tool will be useful to people. I hope to be able to make it useful for the Hex liveCD as well

Questions? Comments? Leave me a comment below or send me an email!