After around 8 months of development, HeX 2.0, codename “Bonobo

HeX is a liveCD developed by the rawpacket team that is based on FreeBSD 7.0 and designed to be used for network security monitoring.

There are a lot of new features and a lot of bugfixes that went into this release, but before we get into that, you can grab the iso here:

• mirror 1 (Georgia Tech, USA) [iso] [md5] [sha]
• mirror 2 (Rawpacket, USA) [iso] [md5] [sha] (md5 and sha are broken links right now, use the links from a different mirror until the files propagate)
• mirror 3 (Rawpacket, Malaysia) [iso] [md5] [sha]

Some of the new features include:

• Moved from FreeBSD 6.2 to 7.0 using the newer ULE scheduler instead of the 4BSD scheduler.
• Integration with UnionFS.
• Addition of many new tools like:
  • silktools
  • argus v3
  • dhcpdump
  • rtpbreak
  • pktstat
  • isic
  • tftpgrab
  • flowtag
  • yahsnarf
  • lots more!
• Inclusion of zsh in addition to bash.
• Lots of base package upgrades.
• Additional signatures for fl0p, pads and tcpXtract.
• Additional scripts for packet statistics and analysis.
• NSM-Console updated to 0.8-dev.
• Addition of Firefox plugins and bookmarks for network security.
• New fluxbox themes and desktop wallpapers

Check out the HeX Trac page for the full list of what’s changed in this release. You can also check out some screenshots of the new release on Geek00l’s blog post. In addition to this release, we’ve begun working on the HeX Sensor project, for a drop-in NSM sensor, hopefully we’ll have a release of that soon!

Thanks to the development team for all the work that went into this awesome release! Feel free to leave a comment about any new features you’d like to see, or join us on the mailing list or irc (#rawpacket on Freenode) to hang out and talk about ideas for the next release. Enjoy!

In other news, development on Hex 2.0 continues forward, we’re trying to get ports finished for inclusion in the ISO, (which is what spawns the flowtag porting), malware analysis took a backseat to sysadmin work due to a rather large project that just finished up. Hopefully I’ll have more time to post here soon, here’s what I’ve been working on:

– A simple C program to measure the index of coincidence of a file (for binary data instead of strings). This was going great until I tried it on a non-OSX OS, now I’m running into segfaults trying to get it to run on Linux (does read() behave differently or something?). I’ve also ported it to Windows (which works), but was a giant pain due to the fact that I don’t ever use Visual Studio.

– A binary file to hex string beautifier; basically, take binary data and print it in really nice strings for either a ruby or a C program. Why? Because I’m tired of manually formatting data for programs.

– A program to generate data with variable amounts of randomness (this is really used for work, but I might end up posting it here depending on whether I think it’s neat enough).

– Rewriting labview (our internal machine allocation management software). Okay, so Jon’s really doing most of the work, but that’s mostly because I don’t know how to do SQL “relations”, I’ll do more of the development soon enough.

– Biking to work. Yea, this isn’t technical, but I got a new bike so I’ve been trying to cut down on the commute, save gas, all that kind of stuff.

UPDATE: Just got an email the flowtag port has been committed, it should be showing up the next time you do a “cvsup”

Anyone that might have read my blog for a long time might remember this post about how I do GTD (Getting To Done) on my machine(s). Well, I decided it’s been quite a while since I’ve talked about that, so I’ll go through some of my workflow, tools and quirks.

If you read my blog soley for security content, you can skip this post Personally I like to read other people’s workflows/setup to see if there is an interesting way I can fit useful tools into my own setup. Anyhow, on with the details

I have 3 machines that I primarily use. My main machine is a MacBook Pro with an attached Dell LCD monitor running OSX (what else?), the second machine is a Dell Optiplex tower running Ubuntu 8.04. The third machine is a Dell 610 laptop that dual-boots Hex and Windows. Here’s a picture of my default setup on the machines:

OSX

Ubuntu

Hex

Most of my work is done on the MacBook (my personal machine, not supplied by work), I use the Ubuntu machine for tasks that span multiple days, so I can leave a screen term logged in and attach/detach whenever I need to take my laptop home. The Hex machine is mainly used for security purposes, packet analysis, nsm-console development testing, etc.

I like to keep a pretty clean desktop, clutter really tends to irritate me. I try as hard as I can to separate everything that would be on the desktop into either a “papers”, “media”, “code” or “misc” folder. In the previous post about GTD, I mentioned that we use the Mantis ticketing system to keep track of administration tickets. We still use Mantis, just recently upgraded, you can check out a shot here:

If my firefox looks a little strange, it’s because I use the vimperator extension to give firefox Vim-like keybindings and layout. It helps someone who spends a large amount of time in a vim editor to transition quickly between text editing and web-browsing. Speaking of text editing, I absolutely love MacVim and really recommend it to anyone using OSX who likes Vim:

I’ve also transitioned from Mutt to Thunderbird, mostly because Mutt’s 2-way IMAP settings are not so great if you have to check mail between multiple computers. I use the pre-release muttator extension to give Thunderbird mutt-like keybindings:

I use the standard GTD folder naming, which mostly consists of “Inbox”, “Archive” and “Do” folders. (I hope the idea behind each folder is obvious). I still use Gmail for personal email, but can’t access it from inside the work network over POP3 or IMAP due to firewall restrictions, so I use the web client.

Instead of using iCal for meetings, like I was doing, I moved to using Google Calendar, which is aided by the notifier for events, so I don’t have to set up reminders for meetings. It also lets me share my wife’s calendar and see what she’s up to As well as google calendar, I also use gReader to browse my 222 RSS subscriptions, nothing else comes close as far as I’m concerned (yay more vim-like keybindings).

I do still use a moleskine, but lately I’ve transitioned to the next larger notebook (5′ x 7′), so that I can write out larger notes and more information (writting assembly notation in a tiny notebook gets annoying). I make use of the same sort of TODO list as in my last post ( – for an incomplete task, + for a completed task ). I take it to all meetings I go to and doodle in it sometimes I still use the smaller book for lists and small notes.

One of the things I’ve found is that despite having a widescreen 2nd monitor, the ‘wide’ part doesn’t really help any for what I do, so I flipped the monitor up to make it vertical, which is really helpful for using IDA, since I can see more instructions at once, here’s the malware I’ve working on at the moment:

Since I normally have at least 1 VM running, as well as mail, 1 or 2 browsers, X11, Adium, MacVim and any other number of things, I cut down on RAM usage by using cmus (hurray more vim keybindings) for a music playing instead of iTunes (which is such a memory hog for simple things). I also use irssi instead of a graphical client for IRC since it’s a little lighter weight:

I rigged Quicksilver up to the cmus-remote command-line utility so I can have system-wide keybindings for player control, pretty much the other thing I use iTunes for anymore is heavy music folder management (I do like that feature). I spend a great deal of time at the command-line, if you’re interested in configuration files, you can find my zshrc, screenrc, vimrc, gvimrc and nsmcrc at dotfiles.org.

Finally, here’s an overview of my basic whole-desktop setup:

My GTD process has kind of evolved into a stripped down version of the multiple-step instructions from some of the more famous practitioners (receive email, process until inbox has 0 messages, perform actions). I make up a large amount of time by unifying my tools around a certain look and feel (Vim-keybindings for everything), allowing my brain not to have to shift gears for different tasks. By keeping a strict set of rules on which application goes where (as in for virtual spaces), I always know where I can find what I am looking for (application wise). In my opinion, this adds to my efficiency more than any particular tool could have.

Well, that’s about it, hopefully you found something interesting in this post. If you have a neat tool or helpful suggestion, leave a comment below (especially if it has Vim keybindings )

Now down to the real post, NSM-Console 0.7 has been released, there are a lot of cool features in this release, but first, go download NSM-Console!

As always, you can check out the TODO and CHANGELOG from svn.

Now, let’s cover some of the newest features in this release:

Encode/Decode enhancements
The encode and decode methods have had a few enhancements added to them, most notably, you can now specify a file to encode or decode, instead of specifying just a string, so you could do:

nsm> encode -f base64 testfile.txt
Encoding ascii --> base64...
Output ([]'s added to show beginning and end):
[TlNNLUNvbnNvbGUgaXMgYXdlc29tZSwgeW91IHNob3VsZCB1c2UgaXQgOikK]

Also, you can specify a variety of hex encodings, because I was noticing that it was delineated in a variety of ways, \x, space (or not delineated at all). I’ve also added the default hex and binary methods, so you don’t have to specify endianness, they default to little-endian.

IP->ASN mapping
As per Scholar’s suggestion, there is now both a module and a command for translating an ip into it’s ASN,  you can either use the module to get a listing from each IP in the pcap, or use the below command to get the ASN for just one address:

nsm> ip2asn 203.223.154.86
Bulk mode; whois.cymru.com [2008-04-27 17:53:32 +0000]
17992   | 203.223.154.86   | AIMS-MY-DIA-AS AIMS Data Centre

Thanks to Team Cymru for their ASN servers

‘Print’ command supports flags
The print command now supports printing TCP flags, still uses Scholar’s pcapparser library.

New command: ‘iplist’
Generate a list of the ips in a pcap file, sorted by the number of occurrences in the file, see below:

nsm> iplist
=== IP list for data.pcap ===
192.168.1.123   1507
64.233.179.109  260
192.168.1.136   141
204.245.162.17  126
216.178.38.133  102
208.67.217.230  92
209.225.0.103   88
.. etc etc

Pipes now supported
One feature geek00l has been bugging me about is getting piping to work in the nsm shell, I’m happy to announce that they finally work, you might run into a few bugs (broken pipes with less), but for the most part they work, now output can be piped into files and programs:

nsm> p -x 1-* | less
(display all the packets and hex output, piped into less)
nsm> iplist > iplist.txt
(output the list of ips into iplist.txt)
nsm> ip2asn 203.223.154.86 >> iplist.txt
(append the ip2asn output to iplist.txt)

Etc, etc, you get the idea. The ‘<‘ pipe hasn’t been implemented yet, perhaps if it’s needed in the future.

New modules, bro-ids-connection and yahsnarf
Geek00l committed his bro-ids-connection module for generating only connection information from a pcap, a yahsnarf module was also committed, to enable extract yahoo IM conversations from a pcap file. Thanks geek00l!

Automatic updating of NSM-Console
Users desiring to be on the bleeding edge of NSM-Console development (is there anyone that actually desires this? :P) can now use the “update” command from within NSM-Console to automatically update from the latest subversion commit. You can also use the -v for verbose output, see below:

nsm> update -v
Updating NSM-Console from svn...
Fetching newest revision from svn...
etc, etc

Still a few kinks to work out, but should work pretty well.

Bugfixes
I fixed some bugs related to gzip’d pcap files as well as some bugs in the encode and decode methods. I also introduced some bugs (hurray!) with pipes, but it’s still usable.

Like I always say, check out the full TODO and CHANGELOG for complete details, and send me any feedback you have

You can grab the iso here. [md5] [sha256]
Or, grab the iso from the mirror. [md5] [sha256]

Since Geek00l already covered a list of the most important changes in his blog post, I’ll just echo the changes in NSM-Console, which is the software that I develop. The version of NSM-Console in Hex 1.0.3 is 0.6-DEVEL, which can be obtained either by checking out the code from subversion (svn co http://svn.security.org.my/trunk/rawpacket-root/usr/home/analyzt/rp-NSM/nsm-console nsm-console), or waiting until I release 0.6

New Features:
– ‘dump‘ command added, you can now dump packet payloads into a binary file for later analysis
– Significant speedups in the harimau module and ‘checkip‘ command if wget is installed
– tcpxtract configuration file changed to extract more types of files
– Added foremost module
– Added clamscan module (Thanks JohnQPublic)
– Argus and tcptrace have reverse dns turned off by default now, it was causing the module to hang for extremely large pcap files. Can be switched on by changed the module options
– rot13 encoding and decoding added
Bugfixes:
– alias command
– urlescape (en|de)coding
– file existence check
– many other things
All the other enhancements, bugfixes and additions.

Since Hex 1.0.2 had NSM-Console version 0.2 in it, if you haven’t downloaded the console or checked it out from subversion since the last hex release, here’s what else has been added since then:

• Release notes for NSM-Console version 0.5

• Release notes for NSM-Console version 0.4

• Release notes for NSM-Console version 0.3

Or you can view the TODO and CHANGELOG for NSM-Console directly. If you want to check out more information about either of the projects, check out the Trac pages for Hex and NSM-Console:

Now, onward to FreeBSD 7.0!

Firstly, scholar01 has created a ‘flowtag’ module for NSM-Console to use Chris Lee’s  excellent Flowtag software for categorizing and tagging network flow for a packet capture. Thanks for the submission scholar01!

Secondly, JohnQPublic has created a ‘clamscan’ module to in order to scan the files extracted by either tcpxtract or foremost for viruses. The clamscan module uses the popular open-source antivirus ClamAV software. Thanks JohnQPublic!

Both of these modules have been committed into NSM-Console’s code, and while only flowtag is included in the 0.5 release, you can try them out by checking NSM-Console out of SVN with the following command:

svn co http://svn.security.org.my/trunk/rawpacket-root/usr/home/analyzt/rp-NSM/nsm-console nsm-console

Note that the majority of the code I commit to svn is stable enough for regular usage, it just doesn’t undergo the regular testing that the point-releases do before they are released.

Thanks to both authors for submitting modules, they’re now included in the ‘credits’ command.

Firstly, you can download NSM-Console version 0.5 here:
http://writequit.org/projects/nsm-console/files/nsm-console-0.5.tar.gz

Mirror here:
https://secure.redsphereglobal.com/data/dakrone/files/nsm-console-0.5.tar.gz

Like always, let’s go over some of the new features in this release:

Alias command
You can now alias a command whatever else you would like to, the syntax is the same as regular bash alias syntax, for instance, here are my aliases from my ~/.nsmcrc:
alias ls = list
alias ll = list
alias serv = e cat /etc/services | grep
So as an example, if I wanted to look up a service port, now I just type “serv 5190” and see if /etc/services has an entry for that port. (I have a habit of hitting ‘ll’ or ‘ls’ all the time, so now at least they’re useful)

Additional modules: flowtime and harimau
I added a couple of modules, the first is flowtime, which is a packet timeliner that I wrote about in this post. The second is the Harimau module, which will query the Harimau watchlist for all the IPs in a pcap file and print out the matching entries. Thanks go to Spoonfork and the Security.org.my team for the awesome tool.
Note: flowtime won’t work out of the box in Hex unless you install Argus version 3 (not version 2, which is what Hex comes with) as well as symlink ‘ploticus’ to ‘pl’ somewhere in your path.

Checkip command
Speaking of the Harimau watchlist, it has also been integrated as an NSM-Console command. You can see an example here:
nsm> checkip 209.177.146.34
209.177.146.34,www.emergingthreats.net/rules/bleeding-botcc.rules,botcc,2008-02-05 00:03:10

Module improvements
The snort module now uses the ac-bnfa search algorithm, which should help on systems with lower amounts of RAM (*cough* like my own). In addition, the bro-ids module now actually generates many more helpful reports and actually performs intrusion detection instead of just generating flow content. Some modules have been added to categories to make them easier to toggle.

Other minor improvements
Toggle handles multiple module names, space separated
All NSM-Console errors finally go to STDERR instead of STDOUT
Help command is much more readable and supports argument to get help about a particular command.
~./nsmcrc is read extremely quietly now, so it doesn’t fill up the screen
Bugfixes.

You can read the entire changelog here.

As always, please please please let me know if you have any comments, criticisms or suggestions Feel free to email me or leave a comment below.

Although it looks like the February version of the ISSA journal hasn’t been pushed out of the website just yet, you can check out Russ’ columns here, or download February’s column directly here.

Firstly, power down the image and edit the configuration options for your image, click on the “Video options”. Check ‘Enable custom screen resolutions’ and add the resolution (in this case, 1280×800). Make sure the resolution is enabled (checked). See the screenshot below for an example of what it should look like:

Next, boot into the VM image and let’s take a look at the xorg.conf file, here are the lines I changed that actually matter:

#HorizSync 31.5 - 48.5
HorizSync 30.0 - 82.0
#VertRefresh 50.0 - 90.0
VertRefresh 50.0 - 90.0
#Option "DPMS"
ModeLine "1280x800" 80.58 1280 1344 1480 1680 800 801 804 827 -HSync -VSync

These should be in the section right after ‘Section “Monitor”‘. After changing the hsync and vsync as well as adding the mode, I changed the display section from:

SubSection "Display"
Depth 24
Modes "1024x768" "800x600" "640x480"
to:
SubSection "Display"
Depth 24
Modes "1280x800" "1024x768" "800x600" "640x480"

It’s a good idea to change the modes for each of the depths (at least 8, 15, 16 and 24) also.

After rebooting (or killing X with Ctrl+Alt+Backspace), your screen should come up in 1280×800 resolution. Hurray!

You can see an example of my desktop setup for Hex 1.0.3-RC2 here:

You can get a copy of my entire xorg.conf file here. (Note that in this xorg.conf, CapsLock is remapped to additional control because I hate capslock with a passion).

Hope this helps someone out there

Step 1: Get the packages here: http://navi.eight7.org/~hinmanm/files/hex/wmii/, you will need all 4 packages. (hopefully my server doesn’t go down

Step 2: Install the packages. With all the packages in the same directory do:
pkg_add -v ./wmii-3.5.1.tbz
The dependencies will automatically be installed along with it.

Step 3: Hex utilizes .bash_profile to actually do the starting of X11, if you take a look at the last line in ~/.bash_profile, you’ll see:
...
if [ -z "$DISPLAY" ] && [ -z "$SSH_CLIENT" ]; then
exec startx
fi
Since X is already started, the easiest way to have wmii started instead of fluxbox is to simply change your ~/.xinitrc file to read:
while wmii; do
true
done

Step 4: Restart X, kill X with a Ctrl+Alt+Backspace, if everything works correctly, you should be staring at an extremely plain desktop. Hit Alt+Enter to open an xterm.

Step 5: wmii uses the /usr/local/etc/wmii-3.5/wmiirc file to store its configuration options, there are a few lines that need to be changed in order to fit wmii into hex a little better:

WMII_TERM="xterm"
changes to:
WMII_TERM="mrxvt"

xsetroot -solid $WMII_BACKGROUND
changes to whatever background-setting command you like to use, mine is set to:
Esetroot -center /home/analyzt/rp-Wallpapers/rp-team.jpg

In addition, to make using Alt as the MODKEY non-annoying (change the MODKEY=Mod1 setting if you want to use something other than Alt), there are a few changes that should go into ~/.mrxvtrc:

The line:
Mrxvt.macro.Alt+1: GotoTab 1
changed to:
Mrxvt.macro.Ctrl+1: GotoTab 1

Repeat for all the GotoTab # commands. Since wmii uses Alt+# to switch workspaces, using them to switch mrxvt tabs doesn’t work either.

Step 6: Restart wmii, hitting ‘Alt+p’ and selecting ‘quit’ should prompt wmii to restart, with the new settings, now hitting Alt+Enter should open the standard mrxvt terminal.

Check out the guide for additional info on how to use wmii. I am loving the lightweight feel and speedy response I’m getting so far, it works great for running Hex in virtualization, as it’s even lighter weight than Fluxbox is.

Good luck!

P.S. Hex 1.0.3 should be out any day now, It will have NSM-Console 0.5-DEVEL version on it, which I will be releasing additionally for download at the same time, look forward to it!