git clone git://github.com/dakrone/nsm-console.git

I really like git more than subversion, and I’m glad that any nsm-console changes that I check in aren’t going to break the version of nsm-console in Hex.

After around 8 months of development, HeX 2.0, codename “Bonobo

HeX is a liveCD developed by the rawpacket team that is based on FreeBSD 7.0 and designed to be used for network security monitoring.

There are a lot of new features and a lot of bugfixes that went into this release, but before we get into that, you can grab the iso here:

• mirror 1 (Georgia Tech, USA) [iso] [md5] [sha]
• mirror 2 (Rawpacket, USA) [iso] [md5] [sha] (md5 and sha are broken links right now, use the links from a different mirror until the files propagate)
• mirror 3 (Rawpacket, Malaysia) [iso] [md5] [sha]

Some of the new features include:

• Moved from FreeBSD 6.2 to 7.0 using the newer ULE scheduler instead of the 4BSD scheduler.
• Integration with UnionFS.
• Addition of many new tools like:
  • silktools
  • argus v3
  • dhcpdump
  • rtpbreak
  • pktstat
  • isic
  • tftpgrab
  • flowtag
  • yahsnarf
  • lots more!
• Inclusion of zsh in addition to bash.
• Lots of base package upgrades.
• Additional signatures for fl0p, pads and tcpXtract.
• Additional scripts for packet statistics and analysis.
• NSM-Console updated to 0.8-dev.
• Addition of Firefox plugins and bookmarks for network security.
• New fluxbox themes and desktop wallpapers

Check out the HeX Trac page for the full list of what’s changed in this release. You can also check out some screenshots of the new release on Geek00l’s blog post. In addition to this release, we’ve begun working on the HeX Sensor project, for a drop-in NSM sensor, hopefully we’ll have a release of that soon!

Thanks to the development team for all the work that went into this awesome release! Feel free to leave a comment about any new features you’d like to see, or join us on the mailing list or irc (#rawpacket on Freenode) to hang out and talk about ideas for the next release. Enjoy!

Now down to the real post, NSM-Console 0.7 has been released, there are a lot of cool features in this release, but first, go download NSM-Console!

As always, you can check out the TODO and CHANGELOG from svn.

Now, let’s cover some of the newest features in this release:

Encode/Decode enhancements
The encode and decode methods have had a few enhancements added to them, most notably, you can now specify a file to encode or decode, instead of specifying just a string, so you could do:

nsm> encode -f base64 testfile.txt
Encoding ascii --> base64...
Output ([]'s added to show beginning and end):
[TlNNLUNvbnNvbGUgaXMgYXdlc29tZSwgeW91IHNob3VsZCB1c2UgaXQgOikK]

Also, you can specify a variety of hex encodings, because I was noticing that it was delineated in a variety of ways, \x, space (or not delineated at all). I’ve also added the default hex and binary methods, so you don’t have to specify endianness, they default to little-endian.

IP->ASN mapping
As per Scholar’s suggestion, there is now both a module and a command for translating an ip into it’s ASN,  you can either use the module to get a listing from each IP in the pcap, or use the below command to get the ASN for just one address:

nsm> ip2asn 203.223.154.86
Bulk mode; whois.cymru.com [2008-04-27 17:53:32 +0000]
17992   | 203.223.154.86   | AIMS-MY-DIA-AS AIMS Data Centre

Thanks to Team Cymru for their ASN servers

‘Print’ command supports flags
The print command now supports printing TCP flags, still uses Scholar’s pcapparser library.

New command: ‘iplist’
Generate a list of the ips in a pcap file, sorted by the number of occurrences in the file, see below:

nsm> iplist
=== IP list for data.pcap ===
192.168.1.123   1507
64.233.179.109  260
192.168.1.136   141
204.245.162.17  126
216.178.38.133  102
208.67.217.230  92
209.225.0.103   88
.. etc etc

Pipes now supported
One feature geek00l has been bugging me about is getting piping to work in the nsm shell, I’m happy to announce that they finally work, you might run into a few bugs (broken pipes with less), but for the most part they work, now output can be piped into files and programs:

nsm> p -x 1-* | less
(display all the packets and hex output, piped into less)
nsm> iplist > iplist.txt
(output the list of ips into iplist.txt)
nsm> ip2asn 203.223.154.86 >> iplist.txt
(append the ip2asn output to iplist.txt)

Etc, etc, you get the idea. The ‘<‘ pipe hasn’t been implemented yet, perhaps if it’s needed in the future.

New modules, bro-ids-connection and yahsnarf
Geek00l committed his bro-ids-connection module for generating only connection information from a pcap, a yahsnarf module was also committed, to enable extract yahoo IM conversations from a pcap file. Thanks geek00l!

Automatic updating of NSM-Console
Users desiring to be on the bleeding edge of NSM-Console development (is there anyone that actually desires this? :P) can now use the “update” command from within NSM-Console to automatically update from the latest subversion commit. You can also use the -v for verbose output, see below:

nsm> update -v
Updating NSM-Console from svn...
Fetching newest revision from svn...
etc, etc

Still a few kinks to work out, but should work pretty well.

Bugfixes
I fixed some bugs related to gzip’d pcap files as well as some bugs in the encode and decode methods. I also introduced some bugs (hurray!) with pipes, but it’s still usable.

Like I always say, check out the full TODO and CHANGELOG for complete details, and send me any feedback you have

NSM-Console (Network Security Monitoring Console) is a framework for performing analysis on packet capture files. It implements a modular structure to allow for an analyst to quickly write modules of their own without any programming language experience. Using these modules a large amount of pcap analysis can be performed quickly using a set of global (as well as per-module) options. It aims to be simple to run and easy to understand without a lot of learning time.

The recently posted NSM-Console whitepaper is a good place to start if you want an introduction to NSM-Console.

You can download NSM-Console 0.6 from the project page.

Here are the highlights for the changes in this version, check the TODO and CHANGELOG for a full list of changes.

The dump command
The dump command lets you dump either the payload (in ascii format), or the full packet (with the -f flag) to a file, you can specify ranges like 1-20, 5,7,9-* and 1-* just like you can with the print command, extremely useful for picking and choosing packets to dump to a file. Here’s an example:

nsm> dump 1-5,10-* file.txt
Writing (append) packet(s) 1 through 5 from /Users/hinmanm/data.pcap to file.txt...
Writing (append) packet(s) 10 through * from /Users/hinmanm/data.pcap to file.txt...

And another of the full dump

nsm> dump -f 20,56-59 full.out
New args: 20,56-59 full.out
Dumping full packet, not just payload...
Writing (append) packet(s) 20 through 20 from /Users/hinmanm/data.pcap to full.out...
Writing (append) packet(s) 56 through 59 from /Users/hinmanm/data.pcap to full.out...

Performance improvements for harimau and checkip
The Harimau module, as well as the checkip command, now use wget (if it’s installed) instead of Ruby’s HTTP request library, this results is a much much faster query time. Another performance tweak was that reverse dns lookups for most modules is now turned off by default because it was taking extremely long for large pcaps

New modules: clamscan, foremost, trace-summary
New modules have been included in this release, clamscan, which scans extracted files (from either the tcpxtract or foremost modules) for viruses, foremost, which is another file extractor and trace-summary, which provides an extremely nice summary of traffic in the pcap

NSM-Console handle’s gzip’d files
You can now specify a gzip’d file directly to NSM-Console, it will attempt to decompress the file into a temporary file when it is loaded. Note that directories of gzip’d files is still not supported entirely.

Bro-ids and Snort changes
Bro-IDS has been split into 2 modules, one for protocol analysis and the other for generating content streams.
Snort has had its community and emerging rules updated to the latest revisions.

As always, bugfixes and minor enhancements, check out the full changelog for a list of what’s changed. If you have any requests for additional features or modules, be sure to leave a command and let me know

Also, if anyone out there wants to see what my ~/.nsmcrc looks like, you can find it here.

Excerpt from the abstract:

“With the proliferation of dozens of different packet analysis tools, a network
traffic analyst has a dizzying amount of tools to choose from to analyze
network data. As the number of tools will only increase, a framework to
unite and manage each of these tools is necessary. This framework should
provide a central and uni?ed way to change the options for each of the
tools. My solution to this problem is NSM-Console, or the Network Security
Monitoring Console. “

You can grab the iso here. [md5] [sha256]
Or, grab the iso from the mirror. [md5] [sha256]

Since Geek00l already covered a list of the most important changes in his blog post, I’ll just echo the changes in NSM-Console, which is the software that I develop. The version of NSM-Console in Hex 1.0.3 is 0.6-DEVEL, which can be obtained either by checking out the code from subversion (svn co http://svn.security.org.my/trunk/rawpacket-root/usr/home/analyzt/rp-NSM/nsm-console nsm-console), or waiting until I release 0.6

New Features:
– ‘dump‘ command added, you can now dump packet payloads into a binary file for later analysis
– Significant speedups in the harimau module and ‘checkip‘ command if wget is installed
– tcpxtract configuration file changed to extract more types of files
– Added foremost module
– Added clamscan module (Thanks JohnQPublic)
– Argus and tcptrace have reverse dns turned off by default now, it was causing the module to hang for extremely large pcap files. Can be switched on by changed the module options
– rot13 encoding and decoding added
Bugfixes:
– alias command
– urlescape (en|de)coding
– file existence check
– many other things
All the other enhancements, bugfixes and additions.

Since Hex 1.0.2 had NSM-Console version 0.2 in it, if you haven’t downloaded the console or checked it out from subversion since the last hex release, here’s what else has been added since then:

• Release notes for NSM-Console version 0.5

• Release notes for NSM-Console version 0.4

• Release notes for NSM-Console version 0.3

Or you can view the TODO and CHANGELOG for NSM-Console directly. If you want to check out more information about either of the projects, check out the Trac pages for Hex and NSM-Console:

Now, onward to FreeBSD 7.0!

Firstly, scholar01 has created a ‘flowtag’ module for NSM-Console to use Chris Lee’s  excellent Flowtag software for categorizing and tagging network flow for a packet capture. Thanks for the submission scholar01!

Secondly, JohnQPublic has created a ‘clamscan’ module to in order to scan the files extracted by either tcpxtract or foremost for viruses. The clamscan module uses the popular open-source antivirus ClamAV software. Thanks JohnQPublic!

Both of these modules have been committed into NSM-Console’s code, and while only flowtag is included in the 0.5 release, you can try them out by checking NSM-Console out of SVN with the following command:

svn co http://svn.security.org.my/trunk/rawpacket-root/usr/home/analyzt/rp-NSM/nsm-console nsm-console

Note that the majority of the code I commit to svn is stable enough for regular usage, it just doesn’t undergo the regular testing that the point-releases do before they are released.

Thanks to both authors for submitting modules, they’re now included in the ‘credits’ command.

Firstly, you can download NSM-Console version 0.5 here:
http://writequit.org/projects/nsm-console/files/nsm-console-0.5.tar.gz

Mirror here:
https://secure.redsphereglobal.com/data/dakrone/files/nsm-console-0.5.tar.gz

Like always, let’s go over some of the new features in this release:

Alias command
You can now alias a command whatever else you would like to, the syntax is the same as regular bash alias syntax, for instance, here are my aliases from my ~/.nsmcrc:
alias ls = list
alias ll = list
alias serv = e cat /etc/services | grep
So as an example, if I wanted to look up a service port, now I just type “serv 5190” and see if /etc/services has an entry for that port. (I have a habit of hitting ‘ll’ or ‘ls’ all the time, so now at least they’re useful)

Additional modules: flowtime and harimau
I added a couple of modules, the first is flowtime, which is a packet timeliner that I wrote about in this post. The second is the Harimau module, which will query the Harimau watchlist for all the IPs in a pcap file and print out the matching entries. Thanks go to Spoonfork and the Security.org.my team for the awesome tool.
Note: flowtime won’t work out of the box in Hex unless you install Argus version 3 (not version 2, which is what Hex comes with) as well as symlink ‘ploticus’ to ‘pl’ somewhere in your path.

Checkip command
Speaking of the Harimau watchlist, it has also been integrated as an NSM-Console command. You can see an example here:
nsm> checkip 209.177.146.34
209.177.146.34,www.emergingthreats.net/rules/bleeding-botcc.rules,botcc,2008-02-05 00:03:10

Module improvements
The snort module now uses the ac-bnfa search algorithm, which should help on systems with lower amounts of RAM (*cough* like my own). In addition, the bro-ids module now actually generates many more helpful reports and actually performs intrusion detection instead of just generating flow content. Some modules have been added to categories to make them easier to toggle.

Other minor improvements
Toggle handles multiple module names, space separated
All NSM-Console errors finally go to STDERR instead of STDOUT
Help command is much more readable and supports argument to get help about a particular command.
~./nsmcrc is read extremely quietly now, so it doesn’t fill up the screen
Bugfixes.

You can read the entire changelog here.

As always, please please please let me know if you have any comments, criticisms or suggestions Feel free to email me or leave a comment below.

http://writequit.org/projects/nsm-console/files/nsm-console-0.4.tar.gz

And, for anyone interested, here’s a rundown of the most notable new features:

Additional encoding/decoding options
You can now do uuencode and uudecode using encode and decode. In addition I’ve added octal and char decoding, more to come in the future!

The ‘print’ command (or just ‘p’)
NSM-Console now supports reading and printing pcap file connections as well as payloads in a variety of multiple formats. If you use the ‘print’ or ‘p’ command without any arguments, usage is displayed. You can print just connection information, or you can print the payload in either ascii or hex. The print command also supports ranges, *s and commas. For example, all of these are valid commands:
p -x 100
p -h 10-15
p 100-*
p -x 10,53-64,102,2037-*
To see the print command in more action, take a look at how it is used to decode the sans packet challenge.

Color terminal text
Yep, nsm-console now supports color to brighten up your packet analysis experience If you are too dull to enjoy color (or don’t have a terminal that supports it), you can turn it off by using the ‘color off’ command.

Added modules
I’m slowing down for finding modules to easily add, but I still managed to add 2: tcptrace and tcpick. Note that tcpick isn’t installed by default on a Hex 1.0.2 install, so you’ll need to install it yourself. In order to view the graphs generated by tcptrace, you’ll need to install xplot too. (Hopefully these will be included in the next version of Hex)

~/.nsmcrc configuration file
When NSM-Console starts up, it will now check to see if the ~/.nsmcrc file exists, if it does, it will read through it and execute the commands found. For example:

[hinmanm@Euclid] $ cat ~/.nsmcrc
color off
eval $PROMPT="#{$GREEN}nsm#{$RESET}> "
# This is a comment
set honeysnap HOST_LIST 192.168.1.101,192.168.1.102

would set the color to be off, change the prompt to be green (yea, I know, a contradiction, it’s just an example) and set up some of our honeysnap options. Any line starting with a “#” will be ignored as a comment. This should help with some of the tediousness of having the set the same options every time you start up nsm-console.

Bugfixes, always bugfixes
Who doesn’t like bugfixes?

You can see the full list of all the changes in this version here, as well as future TODOs here. I’d like to give a big thanks to Scholar for letting me use his pcap parsing library, this way nsm-console doesn’t depend on any external libraries. Thanks Scholar!

I’ve also set up an NSM-Console wiki page over on the trac, in case you’re having trouble finding any of the information or want to download an older release.

You can download NSM-Console here:

http://writequit.org/projects/nsm-console/files/nsm-console-0.3.tar.gz

This release was focused a bit more on usability, features and bugfixes rather than the addition of new modules, however, there were still a couple that were added. Since this release has some pretty big changes, let’s start by going over some of the notable ones:

- Logfile changes
I decided I wanted a way to track the exit status of each module command that was executed, so the logfile will now prepend each executed command with “[exit: #]” (# is the exit status of the command). I’m hoping this will help with module debugging.

- Output directory changes
The default output directory has been changed from “output” to “${PCAP_BASE}-output”, I’m hoping this helps to provide a more meaningful output directory name. Note that when you perform analysis on a directory of files, multiple output directories will be created, if you want all the output in 1 folder, change output not to have ${PCAP_BASE} in it.

- Modules
The following modules were added, use “info <module>” to get more information about a module: iploc, fl0p, argus

- e command
“e” is now a shortcut for exec, who likes typing that all out anyway?

- eval command
The “eval” command allows an analyst to evaluate a line of ruby, this is extremely powerful to anyone that knows ruby. Note that all the variables and methods I’ve written for nsm-console are available for use with eval also. Here are two examples:

nsm> eval 9 * 6
=> 54

nsm> eval m = get_mod_by_name("aimsnarf"); puts m.get_commands
=> aimsnarf -r ${PCAP_FILE} > ${OUTPUT_DIR}/${OUTPUT_FILE}

- Better tab completion
I added things like “PCAP_FILE” and “OUTPUT_DIR” to the list of tab-completed words, there is a much larger list now. (If you really want a list you can use: “eval puts $tabstrings” to print them all out)

- encode/decode commands
Here are the two commands I think a lot of packet analysts are going to find extremely useful. They allow someone to easily translate from one encoding to a different encoding. Rather than explain with text, let me show you a screenshot:

Right now encode and decode support a handful of formats (see screenshot for the list), but I’m hard at work on many many more formats to encode and decode. Thanks go to Geek00l, who gave me the idea for this feature (especially how it would be useful in something like this). This should allow for quicker analysis as performing inline encoding and encoding should be much easier to perform. Take a look at the TODO file in the tarball and let me know if there is an encoding missing that should be available.

- Licensing
NSM-Console is now released under a LGPL version 2.1 license, you can read the license agreement in the LICENSE file or online here.

- Bugfixes
Whitespace handling fixes, better handling of commands not found, category reading fix and many more

- Code cleanup and organization
Cleanup is always good.

See the CHANGELOG file for more detail about what has changed. The TODO file lists features that I am currently working on. If you aren’t sure what nsm-console is, I recommend you watch the screencast I recently created.

As always, I welcome any feedback, comments, criticism, support and patches.