I’m going to be using NSM-Console version 0.4-DEVEL, which adds the features that allow this analysis to be performed without external tools. You can get the development version here. Alright, let’s get this party started:

First things first, the fellows at SANS point you to the first packet in the xmas_Starter.pcap file, so let’s load up NSM-Console with the packet capture

./nsm ~/xmas_Starter.pcap

Next, let’s do a printout of all the packets in this dump (since it’s a small file, there shouldn’t be too many)

nsm> p *
Args: *
Filename: /Users/hinmanm/xmas_Starter.pcap
list from 1 to *
1 1198471642.61773 192.168.25.1 -> 192.168.25.255 UDP 138 > 138 Len=243
2 1198471662.79806 192.168.25.100 -> 192.168.25.128 TCP 7337 > 1000 Len=254
3 1198471662.79813 192.168.25.128 -> 192.168.25.100 TCP 1000 > 7337 Len=58
4 1198471662.79877 192.168.25.100 -> 192.168.25.128 TCP 7337 > 1000 Len=60
5 1198471663.79691 192.168.25.100 -> 192.168.25.128 TCP 7337 > 1000 Len=254
6 1198471663.79697 192.168.25.128 -> 192.168.25.100 TCP 1000 > 7337 Len=58
… etc, etc, etc, to a total of 25 packets

Well, the SANS guide says to start at packet #1, so let’s take a look at it

nsm> p -x 1
(see image for output)

Whoops, this doesn’t look like useful output, maybe they meant the first TCP packet? (Instead of UDP). Looks like the first TCP packet is #2, so let’s look at that one:

nsm> p -x 2
Args: 2
Filename: /Users/hinmanm/xmas_Starter.pcap
full from 2 to 2
2 1198471662.79806 192.168.25.100 -> 192.168.25.128 TCP 7337 > 1000 Len=254
0010 53 57 34 67 64 47 68 6c 49 47 31 76 64 6d 6c 6c SW4gdGhlIG1vdmll
0020 49 45 45 67 51 32 68 79 61 58 4e 30 62 57 46 7a IEEgQ2hyaXN0bWFz
0030 49 45 4e 68 63 6d 39 73 4c 43 42 6f 62 33 63 67 IENhcm9sLCBob3cg
0040 62 57 46 75 65 53 42 75 61 57 64 6f 64 43 68 7a bWFueSBuaWdodChz
0050 4b 53 42 6b 61 57 51 67 64 47 68 6c 49 48 52 6f KSBkaWQgdGhlIHRo
0060 63 6d 56 6c 49 48 4e 77 61 58 4a 70 64 48 4d 67 cmVlIHNwaXJpdHMg
0070 59 32 39 74 5a 53 42 30 62 79 42 32 61 58 4e 70 Y29tZSB0byB2aXNp
0080 64 44 38 3d 00 00 00 00 00 00 00 00 00 00 00 00 dD8=............
0090 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00a0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00c0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................

Ahh, that looks like more readable output, due to the string ending with “=”, it’s most likely encoded in base64 (since = is used for padding). Let’s output the string in just plain ascii so we can decode it easier

nsm> p -a 2
Args: 2
Filename: /Users/hinmanm/xmas_Starter.pcap
ascii from 2 to 2
2 1198471662.79806 192.168.25.100 -> 192.168.25.128 TCP 7337 > 1000 Len=254
SW4gdGhlIG1vdmllIEEgQ2hyaXN0bWFzIENhcm9sLCBob3cgbWFueSBuaWdodChz KSBkaWQgdGhlIHRocmVlIHNwaXJpdHMgY29tZSB0byB2aXNpdD8=...................... ..............................................................

(I put line breaks in so it wouldn’t distort the page)
Okay, let’s take this line and decode it to get our first clue

nsm> decode base64 SW4gdGhlIG1vdmllIEEgQ2hyaXN0bWFzIENhcm9sLCBob3cgbWFueSBuaWdodChz KSBkaWQgdGhlIHRocmVlIHNwaXJpdHMgY29tZSB0byB2aXNpdD8=
Decoding base64 --> ascii...
Output ([]'s added to show beginning and end):

[In the movie A Christmas Carol, how many night(s) did the three spirits come to visit?]

Alright! Got the clue! Now let’s load up our answer file and get the next one

nsm> file /Users/hinmanm/xmas_challenge_2007.pcap
Setting ${PCAP_FILE} = /Users/hinmanm/xmas_challenge_2007.pcap
Setting ${PCAP_BASE} = xmas_challenge_2007.pcap

Since the 3 spirits came on 1 night, we know our next answer is in packet #1, so let’s take a look

nsm> p -a 1
Args: 1
Filename: /Users/hinmanm/xmas_challenge_2007.pcap
ascii from 1 to 1
1 1194153111.12232 192.168.25.100 -> 192.168.25.128 TCP 7337 > 1000 Len=154
QWxsIEkgd2FudCBmb3IgQ2hyaXN0bWFzIGlzIG15IF9fX18gRnJvbnQgVGVldG gu....................................

Aha! Another base64 encoding, let’s decode it

nsm> decode base64 QWxsIEkgd2FudCBmb3IgQ2hyaXN0bWFzIGlzIG15 IF9fX18gRnJvbnQgVGVldGgu
Decoding base64 --> ascii...
Output ([]'s added to show beginning and end):

[All I want for Christmas is my ____ Front Teeth.]

Alright, I think this shows the basic idea, and since this post is to illustrate some of the new features of NSM-Console (rather than the solution to the puzzle), let’s skip on ahead.

Packet 2 leads you to packet 3
Packet 3 leads you to packet 9
Packet 9 leads you to packet 11
Packet 11 leads you to packet 12
Packet 12 leads you to packet 359

Here’s where things start to get a little more interesting, printing out packet 359 shows that it doesn’t look like it’s encoded base64 anymore, in fact it looks like it’s urlescaped

nsm> p -x 359
Args: 359
Filename: /Users/hinmanm/xmas_challenge_2007.pcap
full from 359 to 359
359 1194153771.83615 192.168.25.100 -> 192.168.25.128 TCP 7337 > 1000 Len=154
0010 38 37 25 32 30 31 30 31 25 32 30 4e 55 4c 4c 25 87%20101%20NULL%
0020 32 30 31 31 39 25 32 30 31 30 35 25 32 30 31 31 20119%20105%2011
0030 35 25 32 30 31 30 34 25 32 30 4e 55 4c 4c 25 32 5%20104%20NULL%2
0040 30 31 32 31 25 32 30 31 31 31 25 32 30 31 31 37 0121%20111%20117
0050 25 32 30 4e 55 4c 4c 25 32 30 39 37 25 32 30 4e %20NULL%2097%20N
0060 55 4c 4c 25 32 30 37 37 25 32 30 31 30 31 25 32 ULL%2077%20101%2

Also, the SANS challenge mentions that the message may or may not be in multiple packets, so let’s check the next one

nsm> p -x 360
Args: 360
Filename: /Users/hinmanm/xmas_challenge_2007.pcap
full from 360 to 360
360 1194153772.83062 192.168.25.100 -> 192.168.25.128 TCP 7337 > 1000 Len=154
0010 25 32 30 31 31 34 25 32 30 31 32 31 25 32 30 4e %20114%20121%20N
0020 55 4c 4c 25 32 30 36 37 25 32 30 31 30 34 25 32 ULL%2067%20104%2
0030 30 31 31 34 25 32 30 31 30 35 25 32 30 31 31 35 0114%20105%20115
0040 25 32 30 31 31 36 25 32 30 31 30 39 25 32 30 39 %20116%20109%209
0050 37 25 32 30 31 31 35 25 32 30 34 34 25 30 44 25 7%20115%2044%0D%
0060 30 41 38 37 25 32 30 31 30 31 25 32 30 4e 55 4c 0A87%20101%20NUL

Looks like a continuation, if you print out a few more, you find out that the data stops in packet #365, let’s print out all the packets so we can see what the data looks like

nsm> p -x 359-365
(see picture for output, too long to paste here)

Alright, looks like we’ve got our message, let’s decode it

nsm> p -a 359-365
gives us:

nsm> decode urlescape <big long escaped text>
shows:

That looks like decimal ascii values, let’s decode the values using the “char” decoding

nsm> decode char <space separated char codes>
Here’s the output from decoding each of the 3 lines:

And look! There’s our message (I replaced all the ‘NULL’s with spaces):

We wish you a Merry Christmas,
We wish you a Merry Christmas,
We wish you a Merry Christmas
and a Happy New Year!!!

Merry Christmas to you too SANS, and thanks for the awesome challenge!

You can check out SANS’ solution for the challenge here.
You can check out Geek00l’s full solution for the challenge here.

If it wasn’t for this challenge, we might not have had the idea to include these features in NSM-Console, I’m glad we did and I hope it proves useful to the rest of the packet monkeys out there

P.S. A big thank-you to Scholar for letting me use his pcap parsing library, thus eliminating the dependency on any 3rd party libraries to do all the packet reading, thanks a bunch!

Have another idea for a feature that should be in NSM-Console? Shoot me an email or leave a comment!

Don’t use Java for your installer.

Yea, sure, it’s fine if you use Java for the GUI installer *if* the machine can support it, but what about when I want to install on a machine with a minimal amount of RAM? I mean, even your text-based installer uses Java, and for what? Replace your Java text-based installer with something like Curses (or something equivalent). Make it easier to install Solaris for people who are in college. The install for Solaris almost assumes you’ve been through the install before and know what you’re doing. If you really want more adoption in a learner’s market, you need to make it simpler to install.

In other news: Happy Birthday OpenSolaris (you’re 2! whee!). Now if I could only install you on all of my really really old hardware so I could make headless servers. Alas needing much RAM to work.

Anyone know a good PCI SATA card that will work in my Blade 150? I’m tempted to get this, but I’m not sure if it’ll support JBODs without flashing the BIOS on the card, which would be a pain to do on a SPARC system.

From the vmcore file:
ZFS: I/O failure (write on <unknown> off 0: zio 6000620cd40 [L0 ZIL intent log] 1000L/1000P DVA[0]=<0:1300cb9000:1000> zilog uncompressed BE contiguous birth=208621 fill=0 cksum=8eafa7df8b7cb3e:f2fd0

From the /var/adm/messages file:
Jun 5 12:01:11 lava2051 fctl: [ID 517869 kern.warning] WARNING: fp(0)::GPN_ID for D_ID=650700 failed
Jun 5 12:01:11 lava2051 fctl: [ID 517869 kern.warning] WARNING: fp(0)::N_x Port with D_ID=650700, PWWN=5006016841e019a7 disappeared from fabric
Jun 5 12:01:30 lava2051 scsi: [ID 243001 kern.info] /pci@1c,600000/fibre-channel@1/fp@0,0 (fcp0):
Jun 5 12:01:30 lava2051 offlining lun=0 (trace=0), target=650700 (trace=2800004)
Jun 5 12:06:28 lava2051 unix: [ID 836849 kern.notice]
Jun 5 12:06:28 lava2051 ^Mpanic[cpu2]/thread=2a101061cc0:
Jun 5 12:06:28 lava2051 unix: [ID 809409 kern.notice] ZFS: I/O failure (write on <unknown> off 0: zio 6000620cd40 [L0 ZIL intent log] 1000L/1000P DVA[0]=<0:1300cb9000:1000> zilog uncompressed BE contiguous birth=208621 fill=0 cksum=8eafa7df8b7cb3e:f2fd0a04af0e949e:1a:f3): error 5)
... some stuff ...
Jun 5 12:09:55 lava2051 savecore: [ID 570001 auth.error] reboot after panic: ZFS: I/O failure (write on <unknown> of
f 0: zio 6000620cd40 [L0 ZIL intent log] 1000L/1000P DVA[0]=<0:1300cb9000:1000> zilog uncompressed BE contiguous birth=208621 fill=0 cksum=8eafa7df8b7cb3e:f2fd0
Jun 5 12:09:55 lava2051 savecore: [ID 748169 auth.error] saving system crash dump in /var/crash/lava2051/*.1

Repeat x 3 so far. Like I said, extremely annoying.

Here’s what I think the problem is so far: I have a 500g ZFS pool built on a single Clariion LUN that is exported to this machine. From the looks of it the machine is having trouble seeing the LUN all the time, when it disappears ZFS freaks out and panics because of a I/O failure. Now that I know what the problem is, I have no idea how to make the LUN stop disappearing. Guess I’m off to check some Clariion logs and see where that gets me. Anyone out there have any other suggestions on how I could go about fixing this problem? I have little experience in working with core dumps. I would be extremely grateful

P.S. Yes, I know I should have mirrored the ZFS pool on 2 or more devices in case of a problem like this. This is more my “proof-of-concept” machine where I try out new things and see how developers/QA react to them.

UPDATE:
It looks like the problem was a problem on the Clariion side, for the meantime, we exported a LUN from a different clariion, did a zfs attach, waited for the data to be mirrored and then detached the old one. Fixed! <3 ZFS

UPDATE 2:
Now the data is mirrored to a different Clariion. fun fun. Interestingly enough, EMC doesn’t officially support ZFS on Clariion, only on Symmetrix.

#!/bin/bash
# backup_zfssnap.sh, (c) 2007 ralf [dot] ramge [at] webde [dot] de

BACKUPDIR="/export/backup/snapshots"
DSTAMP=`date '+%y%m%d-%H%M%S'`
FILESYS=$1
DEST=$2
REPLICA=$3
BACKUPNAME=`echo $FILESYS | sed 's/\//_/g'`
BACKUPFILE=$BACKUPNAME"-"$DSTAMP".zfs"
SNAPSHOT=$FILESYS"@backup-"$DSTAMP

if [ ! -d $BACKUPDIR ]; then
echo "Backup Directory doesn't exist"
exit 1
fi

cd $BACKUPDIR

# Check here if we have 7 backup files, create them if we don't
COUNT_FILES=`ls -1 $BACKUPNAME* | wc -l`
if [ $COUNT_FILES -le 1 ]; then
for COUNT in 1 2 3 4 5 6 7
do
if [ ! -f $BACKUPNAME"-000000-00000"$COUNT".zfs" ]; then
touch $BACKUPNAME"-000000-00000"$COUNT".zfs"
sleep 1
fi
done
fi

# Check here that we have less than 8 backup files
COUNT_FILES=`ls -1 $BACKUPNAME* | wc -l`
if [ $COUNT_FILES -gt 7 ]; then
# echo "More than 7 backup files exist"
# exit 1
while [ $COUNT_FILES -gt 7 ]
do
OLDEST_BACKUP_FILE=`ls -rt1 $BACKUPNAME* | head -1`
rm $OLDEST_BACKUP_FILE
let COUNT_FILES=COUNT_FILES-1
done
fi

# Find the oldest backup file to delete
OLDEST_BACKUP_FILE=`ls -rt1 $BACKUPNAME* | head -1`

# Create the snapshot
zfs snapshot $SNAPSHOT

# Create a filesystem image in the local backup directory
zfs send $SNAPSHOT > $BACKUPDIR"/"$BACKUPFILE

# Check for $2 and, if exists, create a second copy on a remote host for tape archival
if [ ! -z $2 ]; then
`zfs send $SNAPSHOT | ssh root@$2 "cat >$BACKUPDIR/$BACKUPFILE"`
fi

# Check for $3 and, if exists, mirror the filesystem on the remote host
if [ ! -z $3 ]; then
`ssh root@$2 "zfs receive $3 < $BACKUPDIR/$BACKUPFILE"`
fi

# Check for $4 and, if exists, clone and promote the filesystem on the remote host
if [ ! -z $4 ]; then
`ssh root@$2 "zfs clone $SNAPSHOT $4; sleep 30; zfs promote $4"`
fi

# Get the trash out of the house
rm $OLDEST_BACKUP_FILE
if [ ! -z $2 ]; then
ssh root@$2 "rm $BACKUPDIR/$OLDEST_BACKUP_FILE"
fi

SNAPLIST=`zfs list -H | grep $FILESYS | grep @backup | cut -f1`
for i in $SNAPLIST; do
zfs destroy $i
done

# Exit cleanly
exit 0

Thanks for the submission Ralf! (I changed your email address in the script comments so you wouldn’t get spam)

I had expected to hear a pretty good discussion around the “linuxification” of Solaris and how Ian Murdock plans to approach it, turns out about halfway through I was a little disappointed by the zealotry of some of the audience members. To *me* at least, it seems like they were arguing trivial points that led to the discussion going way off track. About 2/3 of the way through I turned if off so I could concentrate of a perl script I was writing (see below :P). Here’s what my opinion is about the subject:

• Who cares if you call the Linux userland “Linux” instead of “GNU”?? Most managers and people engaged in casual conversation reference the entire userland as Linux anyway, it makes it easier to talk about. Yes, everyone that is involved in OSS knows that Linux is just the kernel, but that seems like a pretty trivial point to make when you’re not even discussing that in the first place.
•  Solaris needs a better userland. This, I agree with, I used to hate Solaris because I didn’t know how to do things with it, I think Ian makes a good point in that in college, the majority of students that ran an “alternative” operating system were running Linux, they knew it, they loved it, they wanted to use it when they got out of college (at least, I did). I certainly wish I had been experimenting with Solaris in college (I think I only did once). Now that I’ve been administering Solaris for the last year, it is by far my favorite administration platform, it might not be great for everything, but I certainly love it for my sysadmin work. Now if only the rest of the world would come to see the way I feel…
• I commented about GNU having a better userland on a post on OSnews.com some time ago and someone alerted me to the fact that Solaris utilities have a better POSIX standardization than GNU utilities. After doing some poking around I definitely agree with that, I mean, in Linux, do you use -option? –option -option=? Is the manpage helpful ? (Hint: no). What I really miss are the features of the GNU tools, windowing in grep, -iname option for find. Things like that.
• I read an article a couple days ago about how Solaris has a more powerful administration interface, however, Linux has an easier administration interface. I would say that’s about true. When it comes down to it, a lot of people are going to choose what they think will the best and easiest to administer. More education is needed. That and Linux is beating Solaris in online documentation by about 1000 webpages for every 1. Finding what I need for Solaris has always been a more extensive challenge to my GoogleFu than with Linux.
• Almost every person that commented in the forum with Ian reminded me of that annoying guy from CS classes in college who thought he knew everything and was very elitist. Ugh, I just want to hit someone.

There you go, personal opinions that have almost no logical reason other than personal preference, way to go internet.

Ugh, re-reading this it is clear I am not an english major. Sorry for the disjointedness.

Firstly, the script: http://lee.hinmanphoto.com/files/zdiff.txt (formatting long scripts in wordpress’ crazy editor is a very long and arduous process, thus I’m just linking to the script in this case, if anyone knows of a better place to stick it let me know). chmod +x it and away you go!

Edit: Sun was nice enough to host the file for me, here’s a link to their version in case the other one goes down: http://www.sun.com/bigadmin/scripts/submittedScripts/zdiff.txt

In a nutshell, here’s what it does:

• Allows you to diff a file inside a ZFS snapshot with the current file in the filesystem and (optionally) print out the line differences
• Recursively diff an entire snapshot using md5 sums and (optionally) printing out the line differences
• Display the md5 sums for each file in a ZFS snapshot and filesystem (this can get old to look at very quickly)

Basically, that doesn’t mean a whole lot, here’s the output from the -h option:

ZFS Snapshot diff
./zdiff.pl [-dhirv] <zfs shapshot name> [filename]

-d Display the lines that are different (diff output)
-h Display this usage
-i Ignore files that don't exist in the snapshot (only necessary for recursing)
-r Recursively diff every file in the snapshot (filename not required)
-v Verbose mode

[filename] is the filename RELATIVE to the ZFS snapshot root. For example, if
I had a filesystem snapshot called pool/data/zone@initial. The filename '/etc/passwd'
would refer to the filename /pool/data/zone/etc/passwd in the filesystem and filename
/pool/data/zone/.zfs/snapshot/initial/etc/passwd in the snapshot.

A couple of examples:
./zdiff.pl -v -r -i pool/zones/lava2019@Fri
Checks the current pool/zones/lava2019 filesystem against the snapshot
returning the md5sum difference of any files (ignore files that don't
exist in the snapshot). With verbose mode

./zdiff.pl -d pool/zones/lava2019@Mon /root/etc/passwd
Check the md5sum for /pool/zones/lava2019/root/etc/passwd and compare
it to /pool/zones/lava2019/.zfs/snapshot/Mon/root/etc/passwd. Display
the lines that are different also.

Here’s what the output is going to look like:

-bash-3.00# ./zdiff.pl -d -v -r -i pool/zones/lava2019@Fri
Recursive diff on pool/zones/lava2019@Fri
Filesystem: /pool/zones/lava2019, Snapshot: Fri
Comparing: /pool/zones/lava2019/
to: /pool/zones/lava2019/.zfs/snapshot/Fri/
** /pool/zones/lava2019/root/etc/shadow is different
** MD5(/pool/zones/lava2019/root/etc/shadow)= 04fa68e7f9dbc0afbf8950bbb84650a6
** MD5(/pool/zones/lava2019/.zfs/snapshot/Fri/root/etc/shadow)= 4fc845ff7729e804806d8129852fa494
17d16
< tom:*LK*:::::::
** /pool/zones/lava2019/root/etc/dfs/dfstab is different
** MD5(/pool/zones/lava2019/root/etc/dfs/dfstab)= 8426d34aa7aae5a512a0c576ca2977b7
** MD5(/pool/zones/lava2019/.zfs/snapshot/Fri/root/etc/dfs/dfstab)= c3803f151cb3018f77f42226f699ee1b
13d12
< share -F nfs -o rw -d "Data" /data

etc, etc, etc.

I am planning on using it so I can audit certain files on different zones (like /etc/passwd) against an initial ZFS snapshot to see what’s changed. Nice little way to keep track of stuff. Email me with any bugs. Matthew dot hinman at gmail dot com.

#!/usr/bin/perl

use warnings;
use strict;

my $zfsname = shift || die "Need a filesystem name\n";

my $day = `date`;

$day =~ s/(Sun|Mon|Tue|Wed|Thu|Fri|Sat)[\S\s]+/$1/gi;

my $snapname = "$zfsname\@$day";
my $exist = system("zfs list $snapname");

# if it already exists, delete it
if($exist == 0) {
print "Destroying previous weeks snapshot...";
system("zfs destroy $snapname");
print "done.\n";
}

print "Creating daily snapshot...";
system("zfs snapshot $snapname\n");
print "done.\n";

And here’s the crontab entry:
0 2 * * * /usr/sbin/zsnap.pl pool/zones/lava2019
(replace “pool/zones/lava2019″ with whichever zfs you want a snapshot of)

Easy as cake, you’ll always have a zfs snapshot called <zfsname>@Sun through <zfsname>@Sat to rollback to!

# Generated by iptables-save v1.2.11 on Thu May 17 14:52:04 2007
*filter
:INPUT DROP [13164:946396]
:FORWARD ACCEPT [0:0]
:OUTPUT DROP [0:0]
-A INPUT -p tcp -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p tcp -m tcp --dport 22 -j ACCEPT
-A INPUT -s 128.222.228.235 -p tcp -j ACCEPT
-A INPUT -s 128.222.228.235 -p udp -j ACCEPT
-A INPUT -s 128.222.228.236 -p tcp -j ACCEPT
-A INPUT -s 128.222.228.236 -p udp -j ACCEPT
-A INPUT -s 128.222.12.10 -p tcp -j ACCEPT
-A INPUT -s 128.222.12.10 -p udp -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A OUTPUT -p tcp -m state --state RELATED,ESTABLISHED -j ACCEPT
-A OUTPUT -p tcp -j ACCEPT
-A OUTPUT -p udp -j ACCEPT
-A OUTPUT -p icmp -j ACCEPT
COMMIT
# Completed on Thu May 17 14:52:04 2007

(128.222.228.235/236 and 128.221.12.10 are our DNS servers, I also accept pings too because I’m nice like that and people around here tend to freak out if they can’t ping their machine. I also let anything out, easy to comment out to deny outbound traffic.)

RAID0:
Given 4 slices, each ~5g:

First, need a metadb, I created a 100MB slice on c1t1d0s0 (which I am NOT using for the RAID, entirely separate drive) and ran this command to initiate the database. It is a good idea to mirror the database in a minimum of 3 positions, but that is beyond the scope of this tutorial
metadb -a -f c1t1d0s0

Then, it’s as easy as 1 command to bring multiple drives into one slice/partition with the following command:
metainit d100 1 4 c2t2d0s0 c2t3d0s0 c2t4d0s0 c2t5d0s0
NOTE: I already created slice 0 on each of the drives.

To see the status of your meta-slice:
metastat d100
d100: Concat/Stripe
Size: 40878080 blocks (19 GB)
Stripe 0: (interlace: 32 blocks)
Device Start Block Dbase Reloc
c2t2d0s0 0 No Yes
c2t3d0s0 4096 No Yes
c2t4d0s0 4096 No Yes
c2t5d0s0 4096 No Yes

Device Relocation Information:
Device Reloc Device ID
c2t2d0 Yes id1,sd@n6006048cb0ca0ceeef67fa7a33ce4c94
c2t3d0 Yes id1,sd@n6006048cb275dda20f654d7248d17197
c2t4d0 Yes id1,sd@n6006048c5aa658e3c69370f2bad75bc0
c2t5d0 Yes id1,sd@n6006048cc092136a695a21eeaa948f88

See? Now we’ve got a 19GB slice. Feel free to newfs /dev/md/dsk/d100 and mount it somewhere fun.

Next up: RAID1
This is actually not as hard as it looks. First, make sure you init your database like the first step from above. Then initialize your first meta slice:
metainit d101 1 1 c2t2d0s0

Then, create the mirror for that slice which will become your final RAID1 slice by issuing the following command:
metainit d100 -m d101

Then initialize the other slices in your mirror, in this care there are 3 additional slices:
metainit d102 1 1 c2t3d0s0
metainit d103 1 1 c2t4d0s0
metainit d104 1 1 c2t5d0s0

From there, it’s quite easy to finish it up by attaching the mirrors:
metattach d100 d102
metattach d100 d103
metattach d100 d104

Then, monitor metastat for the sync progress percentage until all the mirrors are sync’d. Finished!
metastat d100

Basically, for my own categorization:

1. Celerra-side:
Create filesystems (I am using 4 because I want to stripe across all 4:
nas_fs -n iscsiRAID1_5g -c size=5G pool=clar_r5_performance
nas_fs -n iscsiRAID2_5g -c size=5G pool=clar_r5_performance
nas_fs -n iscsiRAID3_5g -c size=5G pool=clar_r5_performance
nas_fs -n iscsiRAID4_5g -c size=5G pool=clar_r5_performance

Mount filesystems:
server_mount server_2 iscsiRAID1_5g /iscsiRAID1_5g
(repeat for all 4 filesystems)

Create iscsi target:
server_iscsi server_2 -target -alias target_3 -create 1000:np=10.5.140.151
(10.5.140.151 is the datamover IP for this Celerra, “target_3″ is the target name)

Create iscsi LUNs:
server_iscsi server_2 -lun -number 1 -create target_3 -size 5000 -fs iscsiRAID1_5g
server_iscsi server_2 -lun -number 2 -create target_3 -size 5000 -fs iscsiRAID2_5g
server_iscsi server_2 -lun -number 3-create target_3 -size 5000 -fs iscsiRAID3_5g
server_iscsi server_2 -lun -number 4 -create target_3 -size 5000 -fs iscsiRAID4_5g
I am creating 4 luns, 1 for each of the 4 filesystems

2. On the Sun side:
iscsiadm modify discovery --sendtargets enable
iscsiadm add discovery-address 10.5.140.151:3260
(10.5.140.151 is the datamover for our Celerra, it will be our iscsi target)

Run this command so you can get the initiator node name:
iscsiadm list initiator-node
It’ll spit out something that looks like this:
Initiator node name: iqn.1986-03.com.sun:01:ba88a3f5ffff.4648d8d8
Initiator node alias: -
Login Parameters (Default/Configured):
Header Digest: NONE/-
Data Digest: NONE/-
Authentication Type: NONE
RADIUS Server: NONE
RADIUS access: unknown
Configured Sessions: 1
We’re interested in the bold part up there, the part that starts with iqn.blahblahblah

Back on the Celerra:
server_iscsi server_2 -mask -set target_3 -initiator iqn.1986-03.com.sun:01:ba88a3f5ffff.4648d8d8 -grant 1-4
(use the initiator you got from the previous command, we are granting access to LUNs 1 through 4 (our raid LUNs))
And start the iscsi service if it hasn’t been started already:
server_iscsi server_2 -service -start
You are now completely done on the Celerra side, you can log off.

Back on the Sun:
Run this command to make sure you can see your targets alright
iscsiadm list target
Target: iqn.1992-05.com.emc:apm000650039080000-3
Alias: target_3
TPGT: 1000
ISID: 4000002a0000
Connections: 1
You should see something similar to the above. If you do, you now have a successful connection to the Celerra for iscsi. Don’t forget to create device nodes for your drives by running this:
devfsadm -i iscsi
Now run “format” and you should be able to see your drives show up. Don’t forget to open port 3260 in your firewall so that iscsi traffic can get through.

You should now be in business with your 4 drives. I’m still working on the RAID/mirror/striping part. I will add another post once I figure this out.

If you run into an error where the iscsi driver will not online, take a look at this link.