git clone git://github.com/dakrone/nsm-console.git

I really like git more than subversion, and I’m glad that any nsm-console changes that I check in aren’t going to break the version of nsm-console in Hex.

After around 8 months of development, HeX 2.0, codename “Bonobo

HeX is a liveCD developed by the rawpacket team that is based on FreeBSD 7.0 and designed to be used for network security monitoring.

There are a lot of new features and a lot of bugfixes that went into this release, but before we get into that, you can grab the iso here:

• mirror 1 (Georgia Tech, USA) [iso] [md5] [sha]
• mirror 2 (Rawpacket, USA) [iso] [md5] [sha] (md5 and sha are broken links right now, use the links from a different mirror until the files propagate)
• mirror 3 (Rawpacket, Malaysia) [iso] [md5] [sha]

Some of the new features include:

• Moved from FreeBSD 6.2 to 7.0 using the newer ULE scheduler instead of the 4BSD scheduler.
• Integration with UnionFS.
• Addition of many new tools like:
  • silktools
  • argus v3
  • dhcpdump
  • rtpbreak
  • pktstat
  • isic
  • tftpgrab
  • flowtag
  • yahsnarf
  • lots more!
• Inclusion of zsh in addition to bash.
• Lots of base package upgrades.
• Additional signatures for fl0p, pads and tcpXtract.
• Additional scripts for packet statistics and analysis.
• NSM-Console updated to 0.8-dev.
• Addition of Firefox plugins and bookmarks for network security.
• New fluxbox themes and desktop wallpapers

Check out the HeX Trac page for the full list of what’s changed in this release. You can also check out some screenshots of the new release on Geek00l’s blog post. In addition to this release, we’ve begun working on the HeX Sensor project, for a drop-in NSM sensor, hopefully we’ll have a release of that soon!

Thanks to the development team for all the work that went into this awesome release! Feel free to leave a comment about any new features you’d like to see, or join us on the mailing list or irc (#rawpacket on Freenode) to hang out and talk about ideas for the next release. Enjoy!

Now down to the real post, NSM-Console 0.7 has been released, there are a lot of cool features in this release, but first, go download NSM-Console!

As always, you can check out the TODO and CHANGELOG from svn.

Now, let’s cover some of the newest features in this release:

Encode/Decode enhancements
The encode and decode methods have had a few enhancements added to them, most notably, you can now specify a file to encode or decode, instead of specifying just a string, so you could do:

nsm> encode -f base64 testfile.txt
Encoding ascii --> base64...
Output ([]'s added to show beginning and end):
[TlNNLUNvbnNvbGUgaXMgYXdlc29tZSwgeW91IHNob3VsZCB1c2UgaXQgOikK]

Also, you can specify a variety of hex encodings, because I was noticing that it was delineated in a variety of ways, \x, space (or not delineated at all). I’ve also added the default hex and binary methods, so you don’t have to specify endianness, they default to little-endian.

IP->ASN mapping
As per Scholar’s suggestion, there is now both a module and a command for translating an ip into it’s ASN,  you can either use the module to get a listing from each IP in the pcap, or use the below command to get the ASN for just one address:

nsm> ip2asn 203.223.154.86
Bulk mode; whois.cymru.com [2008-04-27 17:53:32 +0000]
17992   | 203.223.154.86   | AIMS-MY-DIA-AS AIMS Data Centre

Thanks to Team Cymru for their ASN servers

‘Print’ command supports flags
The print command now supports printing TCP flags, still uses Scholar’s pcapparser library.

New command: ‘iplist’
Generate a list of the ips in a pcap file, sorted by the number of occurrences in the file, see below:

nsm> iplist
=== IP list for data.pcap ===
192.168.1.123   1507
64.233.179.109  260
192.168.1.136   141
204.245.162.17  126
216.178.38.133  102
208.67.217.230  92
209.225.0.103   88
.. etc etc

Pipes now supported
One feature geek00l has been bugging me about is getting piping to work in the nsm shell, I’m happy to announce that they finally work, you might run into a few bugs (broken pipes with less), but for the most part they work, now output can be piped into files and programs:

nsm> p -x 1-* | less
(display all the packets and hex output, piped into less)
nsm> iplist > iplist.txt
(output the list of ips into iplist.txt)
nsm> ip2asn 203.223.154.86 >> iplist.txt
(append the ip2asn output to iplist.txt)

Etc, etc, you get the idea. The ‘<‘ pipe hasn’t been implemented yet, perhaps if it’s needed in the future.

New modules, bro-ids-connection and yahsnarf
Geek00l committed his bro-ids-connection module for generating only connection information from a pcap, a yahsnarf module was also committed, to enable extract yahoo IM conversations from a pcap file. Thanks geek00l!

Automatic updating of NSM-Console
Users desiring to be on the bleeding edge of NSM-Console development (is there anyone that actually desires this? :P) can now use the “update” command from within NSM-Console to automatically update from the latest subversion commit. You can also use the -v for verbose output, see below:

nsm> update -v
Updating NSM-Console from svn...
Fetching newest revision from svn...
etc, etc

Still a few kinks to work out, but should work pretty well.

Bugfixes
I fixed some bugs related to gzip’d pcap files as well as some bugs in the encode and decode methods. I also introduced some bugs (hurray!) with pipes, but it’s still usable.

Like I always say, check out the full TODO and CHANGELOG for complete details, and send me any feedback you have

NSM-Console (Network Security Monitoring Console) is a framework for performing analysis on packet capture files. It implements a modular structure to allow for an analyst to quickly write modules of their own without any programming language experience. Using these modules a large amount of pcap analysis can be performed quickly using a set of global (as well as per-module) options. It aims to be simple to run and easy to understand without a lot of learning time.

The recently posted NSM-Console whitepaper is a good place to start if you want an introduction to NSM-Console.

You can download NSM-Console 0.6 from the project page.

Here are the highlights for the changes in this version, check the TODO and CHANGELOG for a full list of changes.

The dump command
The dump command lets you dump either the payload (in ascii format), or the full packet (with the -f flag) to a file, you can specify ranges like 1-20, 5,7,9-* and 1-* just like you can with the print command, extremely useful for picking and choosing packets to dump to a file. Here’s an example:

nsm> dump 1-5,10-* file.txt
Writing (append) packet(s) 1 through 5 from /Users/hinmanm/data.pcap to file.txt...
Writing (append) packet(s) 10 through * from /Users/hinmanm/data.pcap to file.txt...

And another of the full dump

nsm> dump -f 20,56-59 full.out
New args: 20,56-59 full.out
Dumping full packet, not just payload...
Writing (append) packet(s) 20 through 20 from /Users/hinmanm/data.pcap to full.out...
Writing (append) packet(s) 56 through 59 from /Users/hinmanm/data.pcap to full.out...

Performance improvements for harimau and checkip
The Harimau module, as well as the checkip command, now use wget (if it’s installed) instead of Ruby’s HTTP request library, this results is a much much faster query time. Another performance tweak was that reverse dns lookups for most modules is now turned off by default because it was taking extremely long for large pcaps

New modules: clamscan, foremost, trace-summary
New modules have been included in this release, clamscan, which scans extracted files (from either the tcpxtract or foremost modules) for viruses, foremost, which is another file extractor and trace-summary, which provides an extremely nice summary of traffic in the pcap

NSM-Console handle’s gzip’d files
You can now specify a gzip’d file directly to NSM-Console, it will attempt to decompress the file into a temporary file when it is loaded. Note that directories of gzip’d files is still not supported entirely.

Bro-ids and Snort changes
Bro-IDS has been split into 2 modules, one for protocol analysis and the other for generating content streams.
Snort has had its community and emerging rules updated to the latest revisions.

As always, bugfixes and minor enhancements, check out the full changelog for a list of what’s changed. If you have any requests for additional features or modules, be sure to leave a command and let me know

Also, if anyone out there wants to see what my ~/.nsmcrc looks like, you can find it here.

Excerpt from the abstract:

“With the proliferation of dozens of different packet analysis tools, a network
traffic analyst has a dizzying amount of tools to choose from to analyze
network data. As the number of tools will only increase, a framework to
unite and manage each of these tools is necessary. This framework should
provide a central and uni?ed way to change the options for each of the
tools. My solution to this problem is NSM-Console, or the Network Security
Monitoring Console. “

You can grab the iso here. [md5] [sha256]
Or, grab the iso from the mirror. [md5] [sha256]

Since Geek00l already covered a list of the most important changes in his blog post, I’ll just echo the changes in NSM-Console, which is the software that I develop. The version of NSM-Console in Hex 1.0.3 is 0.6-DEVEL, which can be obtained either by checking out the code from subversion (svn co http://svn.security.org.my/trunk/rawpacket-root/usr/home/analyzt/rp-NSM/nsm-console nsm-console), or waiting until I release 0.6

New Features:
– ‘dump‘ command added, you can now dump packet payloads into a binary file for later analysis
– Significant speedups in the harimau module and ‘checkip‘ command if wget is installed
– tcpxtract configuration file changed to extract more types of files
– Added foremost module
– Added clamscan module (Thanks JohnQPublic)
– Argus and tcptrace have reverse dns turned off by default now, it was causing the module to hang for extremely large pcap files. Can be switched on by changed the module options
– rot13 encoding and decoding added
Bugfixes:
– alias command
– urlescape (en|de)coding
– file existence check
– many other things
All the other enhancements, bugfixes and additions.

Since Hex 1.0.2 had NSM-Console version 0.2 in it, if you haven’t downloaded the console or checked it out from subversion since the last hex release, here’s what else has been added since then:

• Release notes for NSM-Console version 0.5

• Release notes for NSM-Console version 0.4

• Release notes for NSM-Console version 0.3

Or you can view the TODO and CHANGELOG for NSM-Console directly. If you want to check out more information about either of the projects, check out the Trac pages for Hex and NSM-Console:

Now, onward to FreeBSD 7.0!

Firstly, scholar01 has created a ‘flowtag’ module for NSM-Console to use Chris Lee’s  excellent Flowtag software for categorizing and tagging network flow for a packet capture. Thanks for the submission scholar01!

Secondly, JohnQPublic has created a ‘clamscan’ module to in order to scan the files extracted by either tcpxtract or foremost for viruses. The clamscan module uses the popular open-source antivirus ClamAV software. Thanks JohnQPublic!

Both of these modules have been committed into NSM-Console’s code, and while only flowtag is included in the 0.5 release, you can try them out by checking NSM-Console out of SVN with the following command:

svn co http://svn.security.org.my/trunk/rawpacket-root/usr/home/analyzt/rp-NSM/nsm-console nsm-console

Note that the majority of the code I commit to svn is stable enough for regular usage, it just doesn’t undergo the regular testing that the point-releases do before they are released.

Thanks to both authors for submitting modules, they’re now included in the ‘credits’ command.

Firstly, you can download NSM-Console version 0.5 here:
http://writequit.org/projects/nsm-console/files/nsm-console-0.5.tar.gz

Mirror here:
https://secure.redsphereglobal.com/data/dakrone/files/nsm-console-0.5.tar.gz

Like always, let’s go over some of the new features in this release:

Alias command
You can now alias a command whatever else you would like to, the syntax is the same as regular bash alias syntax, for instance, here are my aliases from my ~/.nsmcrc:
alias ls = list
alias ll = list
alias serv = e cat /etc/services | grep
So as an example, if I wanted to look up a service port, now I just type “serv 5190” and see if /etc/services has an entry for that port. (I have a habit of hitting ‘ll’ or ‘ls’ all the time, so now at least they’re useful)

Additional modules: flowtime and harimau
I added a couple of modules, the first is flowtime, which is a packet timeliner that I wrote about in this post. The second is the Harimau module, which will query the Harimau watchlist for all the IPs in a pcap file and print out the matching entries. Thanks go to Spoonfork and the Security.org.my team for the awesome tool.
Note: flowtime won’t work out of the box in Hex unless you install Argus version 3 (not version 2, which is what Hex comes with) as well as symlink ‘ploticus’ to ‘pl’ somewhere in your path.

Checkip command
Speaking of the Harimau watchlist, it has also been integrated as an NSM-Console command. You can see an example here:
nsm> checkip 209.177.146.34
209.177.146.34,www.emergingthreats.net/rules/bleeding-botcc.rules,botcc,2008-02-05 00:03:10

Module improvements
The snort module now uses the ac-bnfa search algorithm, which should help on systems with lower amounts of RAM (*cough* like my own). In addition, the bro-ids module now actually generates many more helpful reports and actually performs intrusion detection instead of just generating flow content. Some modules have been added to categories to make them easier to toggle.

Other minor improvements
Toggle handles multiple module names, space separated
All NSM-Console errors finally go to STDERR instead of STDOUT
Help command is much more readable and supports argument to get help about a particular command.
~./nsmcrc is read extremely quietly now, so it doesn’t fill up the screen
Bugfixes.

You can read the entire changelog here.

As always, please please please let me know if you have any comments, criticisms or suggestions Feel free to email me or leave a comment below.

Although it looks like the February version of the ISSA journal hasn’t been pushed out of the website just yet, you can check out Russ’ columns here, or download February’s column directly here.

You can get the screencast here (right-click and download, don’t stream):

NSM-Console project page.

It’s under 10 mb and clocks in at 7 minutes and 14 seconds.

If you’re looking to make your own module, I highly recommend first reading this README file (found in the modules directory) and watching the screencast.

If you do make a module and would like it included with the NSM-Console distribution, let me know by sending me an email or leaving a comment.

The version of NSM-Console used in the screencast is verion 0.4