Collaborative analysis efforts with simple to use interfaces

February 12, 2008

You know what would be really helpful? I mean, actually helpful to people in the security industry as a whole? We need some kind of collaboration tool that allows many different users to view, download, analyze, tag, describe and ask questions about any and all kinds of malware, network captures and security logs. I’ve been talking to some of the #rawpacket guys/gals about how it would work, so now I’m stealing their ideas for a blog post ;)

For example, let’s say you discover a new binary malware that one of your honeypots caught, here’s how I envision this would work out:

  1. You register an account at the collaboration website, you can additionally assign your pgp key to your name, security people like to know who they’re actually talking with.
  2. You upload the file, in this case it’s a .exe file, tagging it with a basic description (“nepenthes honeypot caught this transferred over ftp, I think it’s a trojan, etc, etc”) and tags so it becomes searchable (exe, malware, binary, ftp).
  3. The file/pcap is anonymized (optional, but would be extremely nice)
  4. After the initial upload, the collaboration server performs super-basic, but good baseline analysis on the file, saving the results for later. For a .exe file, it could be things like md5sum, clamscan and strings. For other types of files, different tools could be used (*cough* an automated NSM-Console session *cough*), etc
  5. The malware is displayed on the page, security gurus log into their account, have the ability to download the binary to play with it themselves, and are encouraged to share what they found when doing their analysis (and how). They have the ability to upload screenshots, short video clips, textfiles, whatever would help with the analysis. This of it like a traditional website ‘shoutbox’, but with comments on a particular piece of malware or network capture.
  6. Users can also create correlations between different submissions, Example: “This is the link to the network capture for the worm exploiting this particular binary malware”, now we can draw pretty graphs!
  7. Discussion continues until the file has been “figure out”. Give people ‘karma’ or whatever to encourage posting.
  8. ????
  9. Profit!

In all seriousness, you know what I think would be great about this? The community as a whole benefits from the knowledge and talent of people who are good at an individual skill. For instance, I might suck at binary malware analysis, but I can help decode what’s going on with a network trace picked up by an IDS. Community is created, knowledge is shared, security can be improved, people become familiar with the parts of security in which they lack knowledge, everyone is happy.

Make the framework distributable, small groups of people can set up their own collaboration for working with extremely confidential files, think Trac, but instead of bug reports and svn tracking, malware/pcap collaboration and research.

There are projects already like this, I’m excited for the direction that OpenPacket is going with packet captures, upload a file and it’s automatically run through tshark, giving you a baseline to start working with. I think that if the idea is expanded, we can get a lot of different people involved. I know I’d certainly like to get better at doing binary analysis.

Does this sound interesting? It certainly does to me. I’m curious if anyone else is interested, leave me a comment and let me know if you’d be interested in something like this! (Maybe if 40 hours suddenly appear out of nowhere I’m start working on it…)

P.S. I didn’t think of all of this myself, thanks to all the people in #rawpacket for their ideas :) Just want to give credit where it’s due… ;)

User-submitted modules: flowtag and clamscan

February 11, 2008

I’d like to point out a couple of user-submitted modules for NSM-Console that are now included in the distribution.

Firstly, scholar01 has created a ‘flowtag’ module for NSM-Console to use Chris Lee’s  excellent Flowtag software for categorizing and tagging network flow for a packet capture. Thanks for the submission scholar01!

Secondly, JohnQPublic has created a ‘clamscan’ module to in order to scan the files extracted by either tcpxtract or foremost for viruses. The clamscan module uses the popular open-source antivirus ClamAV software. Thanks JohnQPublic!

Both of these modules have been committed into NSM-Console’s code, and while only flowtag is included in the 0.5 release, you can try them out by checking NSM-Console out of SVN with the following command:

svn co http://svn.security.org.my/trunk/rawpacket-root/usr/home/analyzt/rp-NSM/nsm-console nsm-console

Note that the majority of the code I commit to svn is stable enough for regular usage, it just doesn’t undergo the regular testing that the point-releases do before they are released.

Thanks to both authors for submitting modules, they’re now included in the ‘credits’ command. :)

NSM-Console version 0.5 release

February 5, 2008

smallmonkeyThat’s right, no development release this time around. I’ve been trying to get version 0.5 all finished for the Hex 1.0.3 release, and I’m happy to present the newest NSM-Console release!

Firstly, you can download NSM-Console version 0.5 here:
http://writequit.org/projects/nsm-console/files/nsm-console-0.5.tar.gz

Mirror here:
https://secure.redsphereglobal.com/data/dakrone/files/nsm-console-0.5.tar.gz

Like always, let’s go over some of the new features in this release:

Alias command
You can now alias a command whatever else you would like to, the syntax is the same as regular bash alias syntax, for instance, here are my aliases from my ~/.nsmcrc:
alias ls = list
alias ll = list
alias serv = e cat /etc/services | grep

So as an example, if I wanted to look up a service port, now I just type “serv 5190” and see if /etc/services has an entry for that port. (I have a habit of hitting ‘ll’ or ‘ls’ all the time, so now at least they’re useful)

Additional modules: flowtime and harimau
I added a couple of modules, the first is flowtime, which is a packet timeliner that I wrote about in this post. The second is the Harimau module, which will query the Harimau watchlist for all the IPs in a pcap file and print out the matching entries. Thanks go to Spoonfork and the Security.org.my team for the awesome tool.
Note: flowtime won’t work out of the box in Hex unless you install Argus version 3 (not version 2, which is what Hex comes with) as well as symlink ‘ploticus’ to ‘pl’ somewhere in your path.

Checkip command
Speaking of the Harimau watchlist, it has also been integrated as an NSM-Console command. You can see an example here:
nsm> checkip 209.177.146.34
209.177.146.34,www.emergingthreats.net/rules/bleeding-botcc.rules,botcc,2008-02-05 00:03:10

Module improvements
The snort module now uses the ac-bnfa search algorithm, which should help on systems with lower amounts of RAM (*cough* like my own). In addition, the bro-ids module now actually generates many more helpful reports and actually performs intrusion detection instead of just generating flow content. Some modules have been added to categories to make them easier to toggle.

Other minor improvements
Toggle handles multiple module names, space separated
All NSM-Console errors finally go to STDERR instead of STDOUT
Help command is much more readable and supports argument to get help about a particular command.
~./nsmcrc is read extremely quietly now, so it doesn’t fill up the screen
Bugfixes.

You can read the entire changelog here.

As always, please please please let me know if you have any comments, criticisms or suggestions :) Feel free to email me or leave a comment below.

Hex and NSM-Console writeup in February ISSA Journal

February 4, 2008

Russ, the author of holisticinfosec.org has kindly written up a review of the Hex NSM-liveCD in the February edition of his ‘toolsmith’ column for the ISSA journal. The column is a good 3-4 pages about Hex as well as some of the tools included on the distribution. There’s even a page dedicated to NSM-Console (Although the review is using the older 0.2 and 0.3 versions and there’s been lots of improvements in NSM-Console since). Thanks for the awesome review Russ!

Although it looks like the February version of the ISSA journal hasn’t been pushed out of the website just yet, you can check out Russ’ columns here, or download February’s column directly here.

How to enable 1280×800 resolution in Parallels for X11

January 31, 2008

This topic really sucks to search for, way too many different results without any actual clarity, so here’s how I was able to get it working:

Firstly, power down the image and edit the configuration options for your image, click on the “Video options”. Check ‘Enable custom screen resolutions’ and add the resolution (in this case, 1280×800). Make sure the resolution is enabled (checked). See the screenshot below for an example of what it should look like:

custom1280

Next, boot into the VM image and let’s take a look at the xorg.conf file, here are the lines I changed that actually matter:

#HorizSync 31.5 - 48.5
HorizSync 30.0 - 82.0
#VertRefresh 50.0 - 90.0
VertRefresh 50.0 - 90.0
#Option "DPMS"
ModeLine "1280x800" 80.58 1280 1344 1480 1680 800 801 804 827 -HSync -VSync

These should be in the section right after ‘Section “Monitor”‘. After changing the hsync and vsync as well as adding the mode, I changed the display section from:

SubSection "Display"
Depth 24
Modes "1024x768" "800x600" "640x480"

to:
SubSection "Display"
Depth 24
Modes "1280x800" "1024x768" "800x600" "640x480"

It’s a good idea to change the modes for each of the depths (at least 8, 15, 16 and 24) also.

After rebooting (or killing X with Ctrl+Alt+Backspace), your screen should come up in 1280×800 resolution. Hurray!

You can see an example of my desktop setup for Hex 1.0.3-RC2 here:

wholedesk

You can get a copy of my entire xorg.conf file here. (Note that in this xorg.conf, CapsLock is remapped to additional control because I hate capslock with a passion).

Hope this helps someone out there :)

Switching from fluxbox to wmii on Hex

January 30, 2008

wmiiDon’t get me wrong, I love fluxbox, I just enjoy experimenting with other window managers and decided I’d finally try the daunting wmii (turns out, not very daunting at all). So, here’s a quick rundown on getting wmii working on Hex 1.0.3BETA (this will work on pretty much any other Hex 1.* release as well). They should always work about the same for a standard 6.2 FreeBSD release, although the steps may vary a little.

Step 1: Get the packages here: http://navi.eight7.org/~hinmanm/files/hex/wmii/, you will need all 4 packages. (hopefully my server doesn’t go down ;)

Step 2: Install the packages. With all the packages in the same directory do:
pkg_add -v ./wmii-3.5.1.tbz
The dependencies will automatically be installed along with it.

Step 3: Hex utilizes .bash_profile to actually do the starting of X11, if you take a look at the last line in ~/.bash_profile, you’ll see:
...
if [ -z "$DISPLAY" ] && [ -z "$SSH_CLIENT" ]; then
exec startx
fi

Since X is already started, the easiest way to have wmii started instead of fluxbox is to simply change your ~/.xinitrc file to read:
while wmii; do
true
done

Step 4: Restart X, kill X with a Ctrl+Alt+Backspace, if everything works correctly, you should be staring at an extremely plain desktop. Hit Alt+Enter to open an xterm.

Step 5: wmii uses the /usr/local/etc/wmii-3.5/wmiirc file to store its configuration options, there are a few lines that need to be changed in order to fit wmii into hex a little better:

WMII_TERM="xterm"
changes to:
WMII_TERM="mrxvt"

xsetroot -solid $WMII_BACKGROUND
changes to whatever background-setting command you like to use, mine is set to:
Esetroot -center /home/analyzt/rp-Wallpapers/rp-team.jpg

In addition, to make using Alt as the MODKEY non-annoying (change the MODKEY=Mod1 setting if you want to use something other than Alt), there are a few changes that should go into ~/.mrxvtrc:

The line:
Mrxvt.macro.Alt+1: GotoTab 1
changed to:
Mrxvt.macro.Ctrl+1: GotoTab 1

Repeat for all the GotoTab # commands. Since wmii uses Alt+# to switch workspaces, using them to switch mrxvt tabs doesn’t work either.

Step 6: Restart wmii, hitting ‘Alt+p’ and selecting ‘quit’ should prompt wmii to restart, with the new settings, now hitting Alt+Enter should open the standard mrxvt terminal.

Check out the guide for additional info on how to use wmii. I am loving the lightweight feel and speedy response I’m getting so far, it works great for running Hex in virtualization, as it’s even lighter weight than Fluxbox is.

Good luck!

P.S. Hex 1.0.3 should be out any day now, It will have NSM-Console 0.5-DEVEL version on it, which I will be releasing additionally for download at the same time, look forward to it!

How to convert a non-SMP RedHat VM into a SMP RedHat VM

January 28, 2008

Today I got an interesting request, a user needed to change his 64 bit non-SMP VM image (running RedHat 4.0) to a SMP machine, problem is, I don’t really want to have to go through reconfiguring the kernel and rebuilding it, so here’s the easy way to do it (it’s pretty simple):

  1. Power the image down
  2. Right-click and edit the settings for the VM image (if you don’t know how to do this, this article is beyond your scope)
  3. Change the CPU settings from 1 to >1 (2 or 4 or 8 or whatever you want to use) (see picture)
    changecpu
  4. Power the VM image back on
  5. Put the RedHat CD #2 in your desktop CD drive (or using the ISO), connect the disc to the VM image (see picture, note that I selected the wrong ISO, should be disc #2)
    mountisoselectiso
  6. On the vmware image, mount the cd with: mount /media/cdrom
  7. Enter the directory: cd /media/cdrom/RedHat/RPMS
  8. In my case, I’m using a 64-bit kernel, so I would use the 64-bit SMP kernel:
    [root@lava2057 RPMS]# rpm -Uvh kernel-smp-2.6.9-42.EL.x86_64.rpm
  9. Unmount the CD, dettach the CD/ISO
  10. Run up2date to make sure the new kernel you installed is up to date (chances are that it isn’t, since RedHat has pushed out a new kernel since putting out the CDs)
  11. Edit the /boot/grub/grub.conf file, mine looks something like this before changes:
    default=2
    timeout=5
    splashimage=(hd0,0)/grub/splash.xpm.gz
    hiddenmenu
    title Red Hat Enterprise Linux ES (2.6.9-67.0.1.ELsmp)
    root (hd0,0)
    kernel /vmlinuz-2.6.9-67.0.1.ELsmp ro root=LABEL=/ rhgb quiet
    initrd /initrd-2.6.9-67.0.1.ELsmp.img
    title Red Hat Enterprise Linux ES (2.6.9-42.ELsmp)
    root (hd0,0)
    kernel /vmlinuz-2.6.9-42.ELsmp ro root=LABEL=/ rhgb quiet
    initrd /initrd-2.6.9-42.ELsmp.img
    title Red Hat Enterprise Linux ES (2.6.9-67.0.1.EL)
    root (hd0,0)
    kernel /vmlinuz-2.6.9-67.0.1.EL ro root=LABEL=/ rhgb quiet
    initrd /initrd-2.6.9-67.0.1.EL.img
    title Red Hat Enterprise Linux ES (2.6.9-42.EL)
    root (hd0,0)
    kernel /vmlinuz-2.6.9-42.EL ro root=LABEL=/ rhgb quiet
    initrd /initrd-2.6.9-42.EL.img
  12. I have 4 kernels installed, the initial install 64-bit (non-SMP), the 64-bit SMP I just installed and the most recently updated versions of each of these, as you can see, the line “default=2” is still pointing to our non-SMP image. We need to use the 2.6.9-67.0.1.ELsmp kernel, so I’m going to change it to “default=0
  13. Reboot the machine
  14. After it’s rebooted, log in and check ‘uname -a‘, mine shows:
    Linux lava2057.lss.emc.com 2.6.9-67.0.1.ELsmp #1 SMP Fri Nov 30 11:57:43 EST 2007 x86_64 x86_64 x86_64 GNU/Linux

Simple eh? I know it’s not related to security, but I figured it’d be helpful to someone out there. :)

Flowtime – Create a timeline for packet flow

January 24, 2008

You can never have too many tools for pcap visualization :)

Flowtime is a script written in Ruby that produces a timeline of the network flows in a pcap file. Everything is better with a picture, so here’s a picture: (warning, this picture is 3000×2000 pixels, kind of large)

ink2

Each bar on the left is a IP address along with a port, the timeline on the bottom is the time it was capture (in seconds). The different colors represent different kinds of traffic, http is blue, ssl is red, yellow is other, etc. At the moment there isn’t any legend, but I already think it’s useful just to see patterns in the traffic over time.

Download Flowtime here.

Requirements to run Flowtime:

  • Argus (version 3 only)
  • Ploticus (you should have ‘pl’ in your path. You may have to symlink ‘ploticus’ to ‘pl’)
  • EasyTimeline (you should have ‘EasyTimeline’ in your path)

Yea, I know, lots of dependencies, I’m hoping to rewrite this into a MUCH better version in the future, consider this the prototype :)

Here’s how to run it:

flowtime [-w #] [-h #] [-g] [--help] <pcapfile> <ipaddr> <outfile_base>
-w specify the width, default: 2000
-h specify the height, default: 2000
-g automatically try generate a png (requires 'EasyTimeline' and 'pl' in path)
<pcapfile> the packet file to generate a graph of
<ipaddr> source address to generate a graph for, 'all' for all IPs
<outfile_base> basename for the output file

To generate a basic graph, just do something like ‘flowtime -g data.pcap all data-out‘. After running this (if everything works okay), you should have a few files in your directory, if you open data-out.png you should be able to see the data as a timeline. If there are errors instead of data-out.png, there will be a file called data-out.err.

I know this script has issues, isn’t very user-friendly and doesn’t always work, here’s what I’m hoping to improve for the next version:

  • Generate an image of the entire timeline, in addition to showing it in a window
  • The ability to zoom in to a particular range of time to show only that time
  • Export an image of the current view
  • Select a bar to get more information about that flow

I’m going to have to use a different framework though, I’m considering Tk, but I haven’t ever done anything with it before.

Questions? Concerns? Comments? Suggestions?

P.S. I already made an NSM-Console module for flowtime too ;)

Screencast: Creating a module for NSM-Console

January 16, 2008

It hasn’t been that long since my last screencast, but I thought I’d do another, this time showing how to create a module for NSM-Console (so now you have no excuse for not contributing!).

You can get the screencast here (right-click and download, don’t stream):

NSM-Console project page.

It’s under 10 mb and clocks in at 7 minutes and 14 seconds.

If you’re looking to make your own module, I highly recommend first reading this README file (found in the modules directory) and watching the screencast.

If you do make a module and would like it included with the NSM-Console distribution, let me know by sending me an email or leaving a comment.

The version of NSM-Console used in the screencast is verion 0.4

NSM-Console version 0.4 release

January 16, 2008

smallmonkeyWell, it has barely been any length of time and there’s already a new release of NSM-Console, there are so many features that I’ve been coding like crazy to get them all done. First, let’s start with the downloading:

http://writequit.org/projects/nsm-console/files/nsm-console-0.4.tar.gz

And, for anyone interested, here’s a rundown of the most notable new features:

Additional encoding/decoding options
You can now do uuencode and uudecode using encode and decode. In addition I’ve added octal and char decoding, more to come in the future!

The ‘print’ command (or just ‘p’)
NSM-Console now supports reading and printing pcap file connections as well as payloads in a variety of multiple formats. If you use the ‘print’ or ‘p’ command without any arguments, usage is displayed. You can print just connection information, or you can print the payload in either ascii or hex. The print command also supports ranges, *s and commas. For example, all of these are valid commands:
p -x 100
p -h 10-15
p 100-*
p -x 10,53-64,102,2037-*

To see the print command in more action, take a look at how it is used to decode the sans packet challenge.

Color terminal text
Yep, nsm-console now supports color to brighten up your packet analysis experience :) If you are too dull to enjoy color (or don’t have a terminal that supports it), you can turn it off by using the ‘color off’ command.

Added modules
I’m slowing down for finding modules to easily add, but I still managed to add 2: tcptrace and tcpick. Note that tcpick isn’t installed by default on a Hex 1.0.2 install, so you’ll need to install it yourself. In order to view the graphs generated by tcptrace, you’ll need to install xplot too. (Hopefully these will be included in the next version of Hex)

~/.nsmcrc configuration file
When NSM-Console starts up, it will now check to see if the ~/.nsmcrc file exists, if it does, it will read through it and execute the commands found. For example:

[hinmanm@Euclid] $ cat ~/.nsmcrc
color off
eval $PROMPT="#{$GREEN}nsm#{$RESET}> "
# This is a comment
set honeysnap HOST_LIST 192.168.1.101,192.168.1.102

would set the color to be off, change the prompt to be green (yea, I know, a contradiction, it’s just an example) and set up some of our honeysnap options. Any line starting with a “#” will be ignored as a comment. This should help with some of the tediousness of having the set the same options every time you start up nsm-console.

Bugfixes, always bugfixes
Who doesn’t like bugfixes?

You can see the full list of all the changes in this version here, as well as future TODOs here. I’d like to give a big thanks to Scholar for letting me use his pcap parsing library, this way nsm-console doesn’t depend on any external libraries. Thanks Scholar!

I’ve also set up an NSM-Console wiki page over on the trac, in case you’re having trouble finding any of the information or want to download an older release.

 
Powered by Wordpress and MySQL. Theme by Shlomi Noach, openark.org