Firstly, for anyone unfamiliar with Catalyst, Catalyst is a MVC (Model-View-Controller) framework using Perl, for anyone familiar with Ruby on Rails, Catalyst is extremely similar.

The book is written by Jonathan Rockway, who is one of the core team members of the Catalyst team, which gives the book a solid technical background. The book’s text is easy to read and understandable.

Catalyst starts by giving a brief introduction to the MVC methodology and instructions for downloading and installing the Catalyst run-time. One thing to note is that this book is not for readers new to Perl, the book assumes you have a solid knowledge of Perl intricacies and experience using CPAN, which is required to install the multitudes of modules required for Catalyst development. I found that a certain level of problem-solving was required, as not all CPAN modules installed cleanly. I had to force-install a few of the modules and manually resolve a number of dependencies in order to get all the required modules. This barrier to entry might discourage some beginners just starting out with Perl and Catalyst. Other than that, all the installation instructions were very clear.

Developers don’t learn very well with just theory, so Catalyst does a good job of providing a sample application to develop and extend throughout the book. The bulk of the development centers around creating an Address Book application, which is then extended to have features like authentication and AJAX later on during the book. Personally, I’m happy the same application is used, rather than switching applications every chapter. The book does change applications for a couple of chapters (which I find is good, to give a different app’s perspective), then returns to the first application afterwards.

The book discusses a lot of the cool features of Catalyst, such as FormBuilder, templates, REST APIs, Jemplates, AJAX and RSS feeds. The book describes ways to use these features to help speed up development and move away from repetitive code creation. In chapters 8 and 9, testing and deployment are discussed. Personally, I would have liked to see the testing and deployment chapters moved up and integrated into the earlier chapters to enforce a methodology that includes testings from the beginning of development, but I can understand moving it to the end to make entry into the framework a little easier.

One other thing to note, as with almost any programming book, syntax and spelling errors can be killer when attempting to replicate code written in a book, the Catalyst book has a list of errors pertaining to the code in the book, which caused me a great deal of frustration until I looked up the errata. I recommend that anyone use the code example either downloaded from the website or directly from the book take a look and make sure they get the correct syntax. The only other thing I would have liked to see was a chapter on security concerns of the Catalyst framework (being concerned about security myself), I was disappointed they didn’t show up in the book, perhaps in the next edition. I have noticed (from googling for errors I ran into), that Jonathan is very vocal on forums about helping people, so I have no doubt that the syntax errors will be corrected in a future edition

Alright, enough blathering from me, here’s the short and sweet summary: This book is great for Perl or Ruby programmers who are already familiar with MVC development and are interested in the Catalyst framework. The book is not exhaustive by any means, and is not overly long (I would say this is a good thing), it provides a very solid foundation to work with, and a developer should be able to develop his/her own app after reading it. If you’re a Ruby developer interested in a Perl alternative to Rails, or if you’re a Perl developer jealous of Rails, I encourage you to check it out.

P.S. I have also posted this review to Amazon here.

./Command/failure......................ok 1/0
./Command/failure......................NOK 24/0# Failed test at ./Command/failure.t line 85.
# ''
# !=
# '0'
./Command/failure......................ok 41/0# Looks like you failed 1 test of 49.
./Command/failure......................dubious
Test returned status 1 (wstat 256, 0x100)
DIED. FAILED test 24
Failed 1/49 tests, 97.96% okay

… snip …

Failed 1/39 test programs. 1/905 subtests failed.
make: *** [test] Error 1
### execution of make failed, exit code 2
phase compiling: fink-0.27.8-41 failed

Edit the following file:

<bootstrap_dir>/t/Command/failure.t
Replace <bootstrap_dir> with the directory you untar’d the fink-0.27.* tarball into.

Comment out line 85 by changing:
cmp_ok( $!, '!=', 0 );
To be:
#cmp_ok( $!, '!=', 0 );

Rebootstrap the fink and it should install without a problem.

Note: This is totally unsupported by the Fink team, and might produce unintended results According to the mailinglists, the error seems like it’s actually caused by perl 5.8.8 on Leopard instead of 5.8.6 on Tiger, so the test fails. If you want to install perl 5.8.6, you can link /usr/bin/perl to the older version and it should pass the tests.

I started out writing the script in Perl, which at this point is the scripting language I know the best, I then decided that now is as good a time as any to learn Ruby (which I’ve been interested in for a while now), so I re-wrote the entire program in Ruby (first *useful* script I’ve actually written in Ruby). Then, one day at work I was told that I needed to extend the program to support 5000 simultaneous threads doing disk I/O. I thought about this for a while and (after talking with my friend Jon about it) decided on using Java, as the threading was much more robust (something I had problems with using Ruby and Perl). Well, I’ve got working versions of all 3 programs and I thought I’d share my perspective on the pro’s and con’s of each one:

Java pro’s:

• Most robust thread implementation of the 3 languages
• Handles SMP much better than ruby
• Code is portable with minimum requirements to run
• OO language (a bigger pro to actual developers who this matters more to)

Java con’s:

• JVM overhead (not really that much nowadays)
• More difficult to read due to Java’s extreme verbosity
• Requires jdk 1.5+ (1.4 is still the only actual “supported” JDK in my company)

Ruby pro’s:

• Most readable code of all 3 (shortest too)
• I got to learn Ruby
• Ruby implementation available for most platforms
• More OO than Perl (not that I used OO…)

Ruby con’s:

• Ruby only took advantage of 1 of my CPU cores (Java used both)
• Ruby is slower than Perl (maybe one day they’ll be just as fast?)
• Almost no one in my department has heard of Ruby

Perl pro’s:

• Super-easy to install with ActiveState for windows, comes default with most *nix
• Super-easy to install the required module: perl -MCPAN -e shell ; install File::Random
• Allows fine-grain tuning of thread parameters (adjustable thread stack size)

Perl con’s:

• If you don’t have threaded perl, gotta reinstall (/cry @ Solaris)
• Least readable code (unless you loooovvve punctuation)
• Perl doesn’t like me spawning millions of threads and detaching all of them

Overall, since I need code that’s portable to multiple platforms easily, while allowing for very large amounts of IO, I’ll probably stick to the java version (which was renamed CIOSim [Customer I/O Simulator] because you actually pronounce it :P), followed by the Ruby version (so easy to write), and then the Perl version, which, actually has the largest amount of features.

I haven’t written all the features into each version yet (except for the Perl one), but, if you’d like to take a look at them, here they are:

Java version
Ruby version
Perl version

Next tool I need to write, I’ll probably be looking at Ruby

Anyone out there use anything different for sysadmin tools? Python? Lisp? Assembler? Leave a comment and let me know

./phfos.pl -d <dir> -r -v --min=2 --max=3 -n 1000

We’re actually guessing that after allocating too many threads, since we aren’t immediately exiting the thread after reading (what –readstop does), we allocate past the amount of stack space perl allocated us, giving us the EXC_BAD_ACCESS (0x0001) error.

Well, how do we fix that? Turns out perl has an option to set individual stack sizes per thread, so if you decrease the size of the stack per thread, we could allocate more threads. Only 1 problem, on OSX

perl -e'use threads; print(threads->get_stack_size(), "\n")'

Gives you:

Can't locate auto/threads/get_stack_s.al in @INC (@INC contains: /System/Library/Perl/5.8.8/darwin-thread-multi-2level /System/Library/Perl/5.8.8 /Library/Perl/5.8.8/darwin-thread-multi-2level /Library/Perl/5.8.8 /Library/Perl /Network/Library/Perl/5.8.8/darwin-thread-multi-2level /Network/Library/Perl/5.8.8 /Network/Library/Perl /System/Library/Perl/Extras/5.8.8/darwin-thread-multi-2level /System/Library/Perl/Extras/5.8.8 /Library/Perl/5.8.6 /Library/Perl/5.8.1 .) at -e line 1

Anyone else can read about setting stack sizes for perl scripts here (look for the THREAD STACK SIZE section)

Now I need to somehow figure out how I’m going to set thread stack size without being able to see what the values are…

Suggestions?

Just a small update, you *can* actually use more than ~20 threads when using the phfos script. The key is to make sure you are using the --readstop option so the thread doesn’t sit there waiting for rand(maxTime-minTime+1) + minTime seconds keeping the file open. I was successfully able to run it with 1500 threads on my MacBook Pro and again with 1500 threads on my desktop Ubuntu machine.

Hopefully this will make the script infinitely more useful, as a lot more connections can be simulated now.

It’s quick, it’s dirty, but here it is “PHFOS” (Note to self: get better at naming scripts). So here’s what it does.

Basically, you specify a directory with some kind of files in it, the script then spawns <n> threads that each keep a random file open for a random amount of time (to simulate customers accessing files in a random manner). There are options to change the maximum number of threads, the minimum time to keep a file open, the maximum time to keep a file open, whether to read the contents of the file, or to shortcircuit and immediately close after reading the file, etc. Take a look at the “-h” option to see what they do.

***WARNING ***
Don’t run this from a production machine, it can (and will, if you aren’t careful) take down an entire machine in a matter of seconds depending on the command-line options. I accidently ran this with the number of threads=1550 and killed my MacBook Pro laptop immediately. Experiment with low settings on the numbers until you reach the sweet spot, if you run into bus errors or kernel protection errors, try decreasing the numbers. Here are some good starter numbers:

If you have small (<10mb) files, try this:
./phfos.pl -r -v -d <dir> -n 20 --min=2 --max=3 --readstop

If you have larger files, you might try this:
./phfos.pl -r -v -d <dir> -n 15 --min=10 --max=30

Of course, you can take out the -r if you don’t want the files actually read, just opened. Read past the end if you want to read a bit about how it works and the problems you might run into.

Download the script here if you can’t copy it from below (and because WordPress mangles indentions).

#!/usr/bin/perl

use warnings;
use strict;

# PHFOS – Perl Hold File Open Script
use Getopt::Long;
use threads;
use threads::shared;
use POSIX;

sub print_usage {
print “Flags:\n”;
print ” -d <directory>\t\tDirectory to read files from (REQUIRED)\n”;
print ” -n <number>\t\tMaximum number of threads to spawn (default 10)\n”;
print ” -r\t\t\tRead the contents of the files in addition to opening\n”;
print ” –min=<seconds>\tMinimum number of seconds to keep a file open (default 5)\n”;
print ” –max=<seconds>\tMaximum number of seconds to keep a file open (default 10)\n”;
print ” –readstop\t\tIf set, immediately exit thread after reading contents of the file\n”;
print ” -v\t\t\tVerbose mode, tell me what files are open and for how long\n”;
print ” -h\t\t\tDisplay this usage\n”;
exit(0);
}

my %options = ();
my $verbose = 0;

GetOptions(“verbose!” => \$verbose, # verbose option
“d:s” => \$options{dir}, # directory to read files from
“n:i” => \$options{numOpen}, # number of threads to open at the same time
“min:i” => \$options{minTime}, # minimum time to keep a file open
“max:i” => \$options{maxTime}, # maximum time to keep a file open
“readstop” => \$options{readStop},
“r” => \$options{readFile}, # display usage
“h” => \$options{help} # display usage
);

if ($options{help}) { print_usage(); } # if help is set, display usage and exit
if (!defined($options{dir})) { print_usage(); }
if (!defined($options{numOpen})) { $options{numOpen} = 10; }

my $dir = $options{dir};
my @threadList;
my $maxThreads = $options{numOpen} || 10;
my $minTime = $options{minTime} || 5;
my $maxTime = $options{maxTime} || 10;
my @filelist = get_dir_list($dir);
my $index = 0;
our $readfile : shared = $options{readFile};
our $readstop : shared = $options{readStop};

while (0 == 0) {
my $file = $filelist[ rand @filelist ];
my $time = int( rand($maxTime-$minTime + 1)) + $minTime;
if (!$readstop) {
print “Randomly selected file: $file will be opened for $time seconds\n” if $verbose;
}
my $filename = $dir . “/” . $file;

my $newthread = threads->new(\&hold_file_open, $filename, $time);
$newthread->detach;
$index++;

if ($index >= $maxThreads) {
# give the OS a chance to recover
sleep(int(($minTime+$maxTime)/2));
$index = int($index / 2);
}

}

sub get_dir_list {
my $dirname = shift;
opendir(DIR, $dirname) || die “can’t opendir $dirname: $!”;
my @files = grep { /[^\.]/ && -f “$dirname/$_” } readdir(DIR);
closedir DIR;
return @files;

}

sub hold_file_open {
my $filename = shift; # which file to hold open
my $openlength = shift; # the length in seconds to keep it open
my $data;
my $size = 0;
my $bytesread = 0;
my $FIN = POSIX::open($filename);
if (!defined($FIN)) { die “Unable to open $filename\n”; }
if ($readfile) {
my $starttime = time();
while (($bytesread = POSIX::read($FIN,$data,65536)) > 0) {
$size = $size + $bytesread;
#print “[” . $data . “]”;
}
my $endtime = time();
my $readtime = $endtime – $starttime;
print “thread[” . threads->self->tid . “] read $size bytes from $filename in $readtime seconds\n” if $verbose;
if ($readstop) {
sleep(1); # give the OS a bit of time to play catchup
POSIX::close($FIN);
return 0;
}
my $newsleeptime = $openlength – $readtime;
if ($newsleeptime < 1) { exit(0); }
$openlength = $newsleeptime;
}
sleep($openlength);
POSIX::close($FIN);
return 0;
}

Okay, so basically, how it works is that it traverses the directory, looking for file and adding them to an array, if then randomly spawning a thread to open/read the file (up to “n” number of threads). Once it reaches the max number of threads as specified, it waits for ((minTime+maxTime)/2) seconds before halving the thread counter. Basically it waits the average amount of time (assuming true randomness) because by then, statistically half of the threads *should* have finished (I can’t keep track of this because of the thread->detach). At some times you will have slightly larger than “n” threads and sometimes slightly less.

Let’s also talk about a problem I’ve run into that I can’t seem to figure out, here’s the crashdump from Perl:

Exception: EXC_BAD_ACCESS (0x0001)
Codes: KERN_PROTECTION_FAILURE (0x0002) at 0x00000038

Thread 0 Crashed:
0 libSystem.B.dylib 0x90025c82 flockfile + 18
1 libSystem.B.dylib 0x900017c5 fileno + 37
2 libperl.dylib 0x97035b03 PerlIOStdio_dup + 159
3 libperl.dylib 0x970374da PerlIO_fdupopen + 156
4 libperl.dylib 0x96fd6bce Perl_fp_dup + 102
5 libperl.dylib 0x97037789 PerlIO_clone + 442
6 libperl.dylib 0x96fdb246 perl_clone + 1979
7 threads.bundle 0x0000e8bb Perl_ithread_create + 557
8 threads.bundle 0x0000ef01 XS_threads_new + 351
9 libperl.dylib 0x96fc11ad Perl_pp_entersub + 897
10 libperl.dylib 0x96fb8277 Perl_runops_standard + 19
11 libperl.dylib 0x96f4b5d8 perl_run + 724
12 perl 0x000020d2 0x1000 + 4306
13 perl 0x00001f92 0x1000 + 3986
14 perl 0x00001eb9 0x1000 + 3769


For some reason, after a random amount of time, the dispatch thread dies because it attempts to access bad memory. I’ve run ktrace and kdump to see if I could figure it out and it *looks* like it might be a file descriptor problem, however, I can’t figure out why. You shouldn’t run into the problem unless you run the program with a high thread count and a small min/max time (which you’re welcome to do, if you are masochistic).

Anyone out there that’s better at perl than I am, do you know what could be causing this problem? Send me an email or leave a comment!

Firstly, the script: http://lee.hinmanphoto.com/files/zdiff.txt (formatting long scripts in wordpress’ crazy editor is a very long and arduous process, thus I’m just linking to the script in this case, if anyone knows of a better place to stick it let me know). chmod +x it and away you go!

Edit: Sun was nice enough to host the file for me, here’s a link to their version in case the other one goes down: http://www.sun.com/bigadmin/scripts/submittedScripts/zdiff.txt

In a nutshell, here’s what it does:

• Allows you to diff a file inside a ZFS snapshot with the current file in the filesystem and (optionally) print out the line differences
• Recursively diff an entire snapshot using md5 sums and (optionally) printing out the line differences
• Display the md5 sums for each file in a ZFS snapshot and filesystem (this can get old to look at very quickly)

Basically, that doesn’t mean a whole lot, here’s the output from the -h option:

ZFS Snapshot diff
./zdiff.pl [-dhirv] <zfs shapshot name> [filename]

-d Display the lines that are different (diff output)
-h Display this usage
-i Ignore files that don't exist in the snapshot (only necessary for recursing)
-r Recursively diff every file in the snapshot (filename not required)
-v Verbose mode

[filename] is the filename RELATIVE to the ZFS snapshot root. For example, if
I had a filesystem snapshot called pool/data/zone@initial. The filename '/etc/passwd'
would refer to the filename /pool/data/zone/etc/passwd in the filesystem and filename
/pool/data/zone/.zfs/snapshot/initial/etc/passwd in the snapshot.

A couple of examples:
./zdiff.pl -v -r -i pool/zones/lava2019@Fri
Checks the current pool/zones/lava2019 filesystem against the snapshot
returning the md5sum difference of any files (ignore files that don't
exist in the snapshot). With verbose mode

./zdiff.pl -d pool/zones/lava2019@Mon /root/etc/passwd
Check the md5sum for /pool/zones/lava2019/root/etc/passwd and compare
it to /pool/zones/lava2019/.zfs/snapshot/Mon/root/etc/passwd. Display
the lines that are different also.

Here’s what the output is going to look like:

-bash-3.00# ./zdiff.pl -d -v -r -i pool/zones/lava2019@Fri
Recursive diff on pool/zones/lava2019@Fri
Filesystem: /pool/zones/lava2019, Snapshot: Fri
Comparing: /pool/zones/lava2019/
to: /pool/zones/lava2019/.zfs/snapshot/Fri/
** /pool/zones/lava2019/root/etc/shadow is different
** MD5(/pool/zones/lava2019/root/etc/shadow)= 04fa68e7f9dbc0afbf8950bbb84650a6
** MD5(/pool/zones/lava2019/.zfs/snapshot/Fri/root/etc/shadow)= 4fc845ff7729e804806d8129852fa494
17d16
< tom:*LK*:::::::
** /pool/zones/lava2019/root/etc/dfs/dfstab is different
** MD5(/pool/zones/lava2019/root/etc/dfs/dfstab)= 8426d34aa7aae5a512a0c576ca2977b7
** MD5(/pool/zones/lava2019/.zfs/snapshot/Fri/root/etc/dfs/dfstab)= c3803f151cb3018f77f42226f699ee1b
13d12
< share -F nfs -o rw -d "Data" /data

etc, etc, etc.

I am planning on using it so I can audit certain files on different zones (like /etc/passwd) against an initial ZFS snapshot to see what’s changed. Nice little way to keep track of stuff. Email me with any bugs. Matthew dot hinman at gmail dot com.

#!/usr/bin/perl

use warnings;
use strict;

my $zfsname = shift || die "Need a filesystem name\n";

my $day = `date`;

$day =~ s/(Sun|Mon|Tue|Wed|Thu|Fri|Sat)[\S\s]+/$1/gi;

my $snapname = "$zfsname\@$day";
my $exist = system("zfs list $snapname");

# if it already exists, delete it
if($exist == 0) {
print "Destroying previous weeks snapshot...";
system("zfs destroy $snapname");
print "done.\n";
}

print "Creating daily snapshot...";
system("zfs snapshot $snapname\n");
print "done.\n";

And here’s the crontab entry:
0 2 * * * /usr/sbin/zsnap.pl pool/zones/lava2019
(replace “pool/zones/lava2019″ with whichever zfs you want a snapshot of)

Easy as cake, you’ll always have a zfs snapshot called <zfsname>@Sun through <zfsname>@Sat to rollback to!