Firstly, for anyone unfamiliar with Catalyst, Catalyst is a MVC (Model-View-Controller) framework using Perl, for anyone familiar with Ruby on Rails, Catalyst is extremely similar.

The book is written by Jonathan Rockway, who is one of the core team members of the Catalyst team, which gives the book a solid technical background. The book’s text is easy to read and understandable.

Catalyst starts by giving a brief introduction to the MVC methodology and instructions for downloading and installing the Catalyst run-time. One thing to note is that this book is not for readers new to Perl, the book assumes you have a solid knowledge of Perl intricacies and experience using CPAN, which is required to install the multitudes of modules required for Catalyst development. I found that a certain level of problem-solving was required, as not all CPAN modules installed cleanly. I had to force-install a few of the modules and manually resolve a number of dependencies in order to get all the required modules. This barrier to entry might discourage some beginners just starting out with Perl and Catalyst. Other than that, all the installation instructions were very clear.

Developers don’t learn very well with just theory, so Catalyst does a good job of providing a sample application to develop and extend throughout the book. The bulk of the development centers around creating an Address Book application, which is then extended to have features like authentication and AJAX later on during the book. Personally, I’m happy the same application is used, rather than switching applications every chapter. The book does change applications for a couple of chapters (which I find is good, to give a different app’s perspective), then returns to the first application afterwards.

The book discusses a lot of the cool features of Catalyst, such as FormBuilder, templates, REST APIs, Jemplates, AJAX and RSS feeds. The book describes ways to use these features to help speed up development and move away from repetitive code creation. In chapters 8 and 9, testing and deployment are discussed. Personally, I would have liked to see the testing and deployment chapters moved up and integrated into the earlier chapters to enforce a methodology that includes testings from the beginning of development, but I can understand moving it to the end to make entry into the framework a little easier.

One other thing to note, as with almost any programming book, syntax and spelling errors can be killer when attempting to replicate code written in a book, the Catalyst book has a list of errors pertaining to the code in the book, which caused me a great deal of frustration until I looked up the errata. I recommend that anyone use the code example either downloaded from the website or directly from the book take a look and make sure they get the correct syntax. The only other thing I would have liked to see was a chapter on security concerns of the Catalyst framework (being concerned about security myself), I was disappointed they didn’t show up in the book, perhaps in the next edition. I have noticed (from googling for errors I ran into), that Jonathan is very vocal on forums about helping people, so I have no doubt that the syntax errors will be corrected in a future edition

Alright, enough blathering from me, here’s the short and sweet summary: This book is great for Perl or Ruby programmers who are already familiar with MVC development and are interested in the Catalyst framework. The book is not exhaustive by any means, and is not overly long (I would say this is a good thing), it provides a very solid foundation to work with, and a developer should be able to develop his/her own app after reading it. If you’re a Ruby developer interested in a Perl alternative to Rails, or if you’re a Perl developer jealous of Rails, I encourage you to check it out.

P.S. I have also posted this review to Amazon here.

Firstly, you can download NSM-Console version 0.5 here:
http://writequit.org/projects/nsm-console/files/nsm-console-0.5.tar.gz

Mirror here:
https://secure.redsphereglobal.com/data/dakrone/files/nsm-console-0.5.tar.gz

Like always, let’s go over some of the new features in this release:

Alias command
You can now alias a command whatever else you would like to, the syntax is the same as regular bash alias syntax, for instance, here are my aliases from my ~/.nsmcrc:
alias ls = list
alias ll = list
alias serv = e cat /etc/services | grep
So as an example, if I wanted to look up a service port, now I just type “serv 5190” and see if /etc/services has an entry for that port. (I have a habit of hitting ‘ll’ or ‘ls’ all the time, so now at least they’re useful)

Additional modules: flowtime and harimau
I added a couple of modules, the first is flowtime, which is a packet timeliner that I wrote about in this post. The second is the Harimau module, which will query the Harimau watchlist for all the IPs in a pcap file and print out the matching entries. Thanks go to Spoonfork and the Security.org.my team for the awesome tool.
Note: flowtime won’t work out of the box in Hex unless you install Argus version 3 (not version 2, which is what Hex comes with) as well as symlink ‘ploticus’ to ‘pl’ somewhere in your path.

Checkip command
Speaking of the Harimau watchlist, it has also been integrated as an NSM-Console command. You can see an example here:
nsm> checkip 209.177.146.34
209.177.146.34,www.emergingthreats.net/rules/bleeding-botcc.rules,botcc,2008-02-05 00:03:10

Module improvements
The snort module now uses the ac-bnfa search algorithm, which should help on systems with lower amounts of RAM (*cough* like my own). In addition, the bro-ids module now actually generates many more helpful reports and actually performs intrusion detection instead of just generating flow content. Some modules have been added to categories to make them easier to toggle.

Other minor improvements
Toggle handles multiple module names, space separated
All NSM-Console errors finally go to STDERR instead of STDOUT
Help command is much more readable and supports argument to get help about a particular command.
~./nsmcrc is read extremely quietly now, so it doesn’t fill up the screen
Bugfixes.

You can read the entire changelog here.

As always, please please please let me know if you have any comments, criticisms or suggestions Feel free to email me or leave a comment below.

http://writequit.org/projects/nsm-console/files/nsm-console-0.4.tar.gz

And, for anyone interested, here’s a rundown of the most notable new features:

Additional encoding/decoding options
You can now do uuencode and uudecode using encode and decode. In addition I’ve added octal and char decoding, more to come in the future!

The ‘print’ command (or just ‘p’)
NSM-Console now supports reading and printing pcap file connections as well as payloads in a variety of multiple formats. If you use the ‘print’ or ‘p’ command without any arguments, usage is displayed. You can print just connection information, or you can print the payload in either ascii or hex. The print command also supports ranges, *s and commas. For example, all of these are valid commands:
p -x 100
p -h 10-15
p 100-*
p -x 10,53-64,102,2037-*
To see the print command in more action, take a look at how it is used to decode the sans packet challenge.

Color terminal text
Yep, nsm-console now supports color to brighten up your packet analysis experience If you are too dull to enjoy color (or don’t have a terminal that supports it), you can turn it off by using the ‘color off’ command.

Added modules
I’m slowing down for finding modules to easily add, but I still managed to add 2: tcptrace and tcpick. Note that tcpick isn’t installed by default on a Hex 1.0.2 install, so you’ll need to install it yourself. In order to view the graphs generated by tcptrace, you’ll need to install xplot too. (Hopefully these will be included in the next version of Hex)

~/.nsmcrc configuration file
When NSM-Console starts up, it will now check to see if the ~/.nsmcrc file exists, if it does, it will read through it and execute the commands found. For example:

[hinmanm@Euclid] $ cat ~/.nsmcrc
color off
eval $PROMPT="#{$GREEN}nsm#{$RESET}> "
# This is a comment
set honeysnap HOST_LIST 192.168.1.101,192.168.1.102

would set the color to be off, change the prompt to be green (yea, I know, a contradiction, it’s just an example) and set up some of our honeysnap options. Any line starting with a “#” will be ignored as a comment. This should help with some of the tediousness of having the set the same options every time you start up nsm-console.

Bugfixes, always bugfixes
Who doesn’t like bugfixes?

You can see the full list of all the changes in this version here, as well as future TODOs here. I’d like to give a big thanks to Scholar for letting me use his pcap parsing library, this way nsm-console doesn’t depend on any external libraries. Thanks Scholar!

I’ve also set up an NSM-Console wiki page over on the trac, in case you’re having trouble finding any of the information or want to download an older release.

You can download NSM-Console here:

http://writequit.org/projects/nsm-console/files/nsm-console-0.3.tar.gz

This release was focused a bit more on usability, features and bugfixes rather than the addition of new modules, however, there were still a couple that were added. Since this release has some pretty big changes, let’s start by going over some of the notable ones:

- Logfile changes
I decided I wanted a way to track the exit status of each module command that was executed, so the logfile will now prepend each executed command with “[exit: #]” (# is the exit status of the command). I’m hoping this will help with module debugging.

- Output directory changes
The default output directory has been changed from “output” to “${PCAP_BASE}-output”, I’m hoping this helps to provide a more meaningful output directory name. Note that when you perform analysis on a directory of files, multiple output directories will be created, if you want all the output in 1 folder, change output not to have ${PCAP_BASE} in it.

- Modules
The following modules were added, use “info <module>” to get more information about a module: iploc, fl0p, argus

- e command
“e” is now a shortcut for exec, who likes typing that all out anyway?

- eval command
The “eval” command allows an analyst to evaluate a line of ruby, this is extremely powerful to anyone that knows ruby. Note that all the variables and methods I’ve written for nsm-console are available for use with eval also. Here are two examples:

nsm> eval 9 * 6
=> 54

nsm> eval m = get_mod_by_name("aimsnarf"); puts m.get_commands
=> aimsnarf -r ${PCAP_FILE} > ${OUTPUT_DIR}/${OUTPUT_FILE}

- Better tab completion
I added things like “PCAP_FILE” and “OUTPUT_DIR” to the list of tab-completed words, there is a much larger list now. (If you really want a list you can use: “eval puts $tabstrings” to print them all out)

- encode/decode commands
Here are the two commands I think a lot of packet analysts are going to find extremely useful. They allow someone to easily translate from one encoding to a different encoding. Rather than explain with text, let me show you a screenshot:

Right now encode and decode support a handful of formats (see screenshot for the list), but I’m hard at work on many many more formats to encode and decode. Thanks go to Geek00l, who gave me the idea for this feature (especially how it would be useful in something like this). This should allow for quicker analysis as performing inline encoding and encoding should be much easier to perform. Take a look at the TODO file in the tarball and let me know if there is an encoding missing that should be available.

- Licensing
NSM-Console is now released under a LGPL version 2.1 license, you can read the license agreement in the LICENSE file or online here.

- Bugfixes
Whitespace handling fixes, better handling of commands not found, category reading fix and many more

- Code cleanup and organization
Cleanup is always good.

See the CHANGELOG file for more detail about what has changed. The TODO file lists features that I am currently working on. If you aren’t sure what nsm-console is, I recommend you watch the screencast I recently created.

As always, I welcome any feedback, comments, criticism, support and patches.

Firstly, download the files here.
The static page for nsm-console is here.

I finally got around to releasing the next version of the nsm-console. This version incorporates a large amount of bug fixes and additional features, first, I’ll start with some of the features I’m the most happy about Most of these features are in the new Hex 1.0.2 release which came out yesterday (go download it now!)

• Categories
  • You can now toggle certain categories on and off, for instance, one category shipped with the new release is the ‘flow’ categories, you treat them just like a regular module. Simply use “toggle flow” to toggle the flow category (and all of it’s modules) on and off.
  • You can easily add your own categories to customize your work environment, all you have to do is create a file named the same name as the category name in the modules/categories directory.
• Directory analysis
  • When you normally run the nsm-console, you would specify a single pcap file to perform analysis on, now you can use the same “file” command to specify a directory full of files  instead of a single file. When the “run” command is executed, all the toggled module’s operations will be executed on each file in the directory (recursively)
  • To better accommodate this type of operation, I encourage anyone that is writing any modules to write them to output the results into an output file named something like ${PCAP_BASE}.tcpdstat.out (so if you had more than 1 file, the output will go into more than one file)
• The ‘exec’ command
  • I added the exec command because I was tired of spawning an additional shell in order to run a simple ‘tcpdump’. I hope this helps with the automation that I’m going to talk about below.

The directory functionality and the exec command isn’t in the current Hex release, but hopefully it will be in the next release.

Part of the reason I think nsm-console is neat is the ease of automation you can do using simple text files. For instance, if you created a text file called “automate.txt” and put the following lines in it:

file /pcap/data.pcap
output automated-output
toggle aimsnarf
toggle tcpdstat
toggle chaosreader
run
quit

Then, you can run the command:

./nsm < automate.txt > output.txt

Which will run all the commands in the text file automatically, placing all the output in output.txt, simple eh?

If you have any questions, comments or suggestions, feel free to leave a comment or send an email I’d love to hear if/how you’re using nsm-console

• About the author
• Aimsnarf
• NSM-Console

Hopefully this makes it easier to link to a particular project. Take a look and let me know if you find anything missing! I’m hoping to add a link for packages I’ve created soon also!

Also, I’m contemplating future posts, are there any requests for particular posts on a certain topic? More ZFS posts? More security tool posts? More how-to posts? Leave me a comment and let me know!

Behold! I present NSM-Console! (read more about it here, watch a screencast here)

Download the framework here.
Keep in mind this framework only includes 3 modules (mostly used just for testing)

NSM-Console in a small (< 500 1000 1500 lines) framework for running nsm modules. Essentially, it’s a framework for running things (but we don’t call it that because it sounds like it wasn’t any work :P). Here’s the breakdown:

Users will run the ‘nsm’ program to enter into a console, by which they can select modules that have options to run against a packet capture file. Let’s talk a little bit about how it works. When nsm starts, it looks in the directory given by $MODULE_DIR (hardcoded at the moment, will be moved later) for all directories that follow the naming convention “<name>.module”, where <name> is the name of the module. Following so far?

For explanation’s sake, let’s call our module “aimsnarf”, after the aim-sniffing program I wrote a while back. So inside the module directory, there is another directory called “aimsnarf.module”. Inside the aimsnarf.module directory, there are a few files that define how our module will act. First is a file just called “aimsnarf”, this is the file that dictates commands to be run in the framework. It can have multiple commands separated by lines in the file. (If the module were called “bathtub”, the file would be called “bathtub” also, it’s named after the module name). Let’s take a look at an example line from the aimsnarf.module/aimsnarf file:

aimsnarf -r ${PCAP_FILE} > ${OUTPUT_DIR}/${OUTPUT_FILE}

In this line, you can see some variables that get changed out when the module is executed from nsm console. ${PCAP_FILE} is always set to the pcap file being analyzed, ${OUTPUT_DIR} will be set to the module-specific directory output will be directed into (set by nsm console), the last option, ${OUTPUT_FILE} is not set by the nsm console, it is a user-defined variable. But where is it defined? Let’s take a look at the next file: aimsnarf.module/defaults:

OUTPUT_FILE=${PCAP_BASE}.aim

The defaults file defines the user-settable options for the module, in this case, there is only 1, the “OUTPUT_FILE” option. The defaults file also provides a default value for that option (in this case, ${PCAP_BASE}.aim). The default value is provided so the module can be run without having to set every option. ${PCAP_BASE} is another variable that is provided by the nsm console, it is the basename of the pcap file (so it the pcap file was “~/pcap/file.pcap”, the ${PCAP_BASE} would be “file.pcap”).

The next file is aimsnarf.module/description, this file provides a 1-line description of the module to be listed when the user lists all the modules. Here is the ‘description’ for the aimsnarf module:

Extracts AIM messages from a pcap file

Simple eh? Still following? Okay, there is one more file, the aimsnarf.module/info file, this file provides a more detailed explanation of the module, it should also define what each option means, here is what aimsnarf’s ‘info’ file looks like:

Aimsnarf is a program to extract aim conversations from a pcap file

The following options are enabled:
OUTPUT_FILE (default $PCAP_DATA}.aim)
This option specifies the output file.

So how does this all work together? Well, first, you would start the NSM console you would see the following screen:

You can see that the aimsnarf module has been loaded (as well as a couple of others) and a prompt has been presented. From here there are a few different options, here’s the output of the ‘help’ command:

I think the help is pretty self-explanatory, hopefully it makes sense to everyone else as well.

Let’s run the aimsnarf module on a pcap file as an example. First, we need to specify a pcap file to work on with the ‘file’ command, so we type:

nsm> file /Users/hinmanm/data.pcap
Setting ${PCAP_FILE} = /Users/hinmanm/data.pcap
Setting ${PCAP_BASE} = data.pcap

Next, we will specify a different output directory (the default is ‘output’) using the ‘output’ command:

nsm> output data-output
Setting ${OUTPUT_DIR} = data-output

Next, let’s make sure the only module that is going to run is the aimsnarf module:

nsm> toggle none
All modules turned off.
nsm> toggle aimsnarf
aimsnarf module turned on.

And make sure our options are set correctly for the aimsnarf module:

nsm> options aimsnarf
Global options:
-----------------------------------
${PCAP_FILE}: /Users/hinmanm/data.pcap
${PCAP_BASE}: data.pcap
${OUTPUT_DIR}: data-output
${MODULE_DIR}: modules
Options for module aimsnarf:
-----------------------------------
${OUTPUT_FILE} = ${PCAP_BASE}.aim

That looks good to me! Now, let’s run and see what happens:

nsm> run
Executing analysis...
===> module aimsnarf running...
Creating directory data-output/aimsnarf
Result: true
--> aimsnarf -r /Users/hinmanm/data.pcap > data-output/aimsnarf/data.pcap.aim
/Library/Ruby/Site/1.8/universal-darwin9.0/pcap.bundle: warning: do not use Fixnums as Symbols
/Library/Ruby/Site/1.8/universal-darwin9.0/pcap.bundle: warning: do not use Fixnums as Symbols
/Library/Ruby/Site/1.8/universal-darwin9.0/pcap.bundle: warning: do not use Fixnums as Symbols
Result: true
===> module aimsnarf finished.
===> module hash skipped.
===> module ngrep skipped.

Looks like it ran the command! (ignore the warnings, they’re generated by aimsnarf, not nsm) Simple enough, right?

So here’s the real question: why write something like this? Well, I believe this tool will help to perform large amounts of analysis (think 20-30 plugins instead of 3) on packet capture files without having to manually manage each tool that does analysis. Instead of having to change a hardcoded script, you can now edit (and create) modules to do exactly what you need, but still having the flexibility of global varaibles (ie: ${OUTPUT_DIR}) and module specific variables (ie: ${OUTPUT_FILE} in aimsnarf). Being able to toggle each module on and off allows you to do a flexible amount of analysis on a file on the fly. In the future I hope to be able to extend the framework to run on a directory of files, allowing you to automate running a vast number of pcap analysis tools on a large number of pcap files very easily.

While this is not a replacement for running each command individually (and it shouldn’t be), I hope this tool will be useful to people. I hope to be able to make it useful for the Hex liveCD as well

Questions? Comments? Leave me a comment below or send me an email!