Here’s something that doesn’t make me happy to see:

[root@lava2054 ~]# tail /var/log/messages
Segmentation fault
[root@lava2054 ~]# dmesg
-bash: dmesg: command not found
[root@lava2054 ~]# sync;reboot
-bash: sync: command not found
reboot: error while loading shared libraries: libattr.so.1: cannot open shared object file: No such file or directory

Ack. *runs off to find the problem*

Update:

Turns out that one of the partitions in the LVM that was the / partition (I *hate* RedHat’s default partitioning) died, which caused the machine to panic every time I booted. Since I needed the particular machine by the end of the day and didn’t have a lot of time to debug what went wrong, I just performed a fresh install. Problem Fixed (If a little ungracefully).

Also known as “Why you need some kind of iSCSI security”

Okay, after reading Himanshu Dwivedi’s presentation[PDF] on iSCSI security (insecure-SCSI hur hur hur) I decided to try and replicate one of the attacks that he mentioned in the presentation. Following is how I managed to get the data shown of a different machine.

Firstly, I needed to get the initiatorname for the iscsi daemon on the target host. In this case the /etc/initiatorname.iscsi file is -rw-------, so I needed a way to find out the initiator name without root privileges. In this case I used wireshark (used to be ethereal) to sniff the traffic for a plain-text initiator name. Okay, so here’s what I did:

Fire up Wireshark (Ethereal) and set it to promiscuous mode, with a filter for port 3260 (the iscsid port), feel free to filter by host, etc. Run the live capture for a while, what you’re going to be looking for is a sequence of packets that look more like this:
…
iSCSI Login Command
TCP [PSH,ACK] <other information>
TCP [ACK] <other information>
iSCSI Login Response (Success)
…
I can’t say how long it’s going to take this, but it’s much easier to get when the iscsi service is being started on the machine you’re trying to sniff, therefore, if you can sniff while a machine is coming online from a reboot you will most likely have a much better chance of detecting this.

There’s another easier way of getting what you want just doing a string search. Search for the string “Initiator” below you can see a picture what you should be looking for in Wireshark:

Note the highlighted text at the bottom, this is what you’re looking for. Copied straight out you get something like this:
`
7LrEN@@
By"p/
InitiatorName=iqn.1987-05.com.cisco:01.87956e84f925InitiatorAlias=lava2163SessionType=DiscoveryHeaderDigest=NoneDataDigest=NoneMaxRecvDataSegmentLength=8192DefaultTime2Wait=0DefaultTime2Retain=0IFMarker=NoOFMarker=NoErrorRecoveryLevel=0X-com.cisco.PingTimeout=5X-com.cisco.sendAsyncText=YesX-com.cisco.protocol=draft20

All we really care about in that text is the part that is bolded, using this, we can manually set the /etc/initiatorname.iscsi file on a different Linux server to have the line “InitiatorName=iqn.1987-05.com.cisco:01.87956e84f925“. Don’t forget to change the /etc/iscsi.conf file to have the following line in it:
DiscoveryAddress=<ip of iscsi target host>
Fill in the host with the IP address that your sniffing showed (in this case, it was 10.5.140.229 as you can see in the picture)

After this step, if this were a real attack it would probably be a good idea to preform a DOS attack on the original target to knock it out of connection with the server (you don’t really want 2 hosts attempting to get the same information from an iSCSI target). Then start the iscsi daemon with “/etc/init.d/iscsi start” and you should be seeing the data originally meant for the other host.

This is really a simple attack and barely requires any technical knowledge of iSCSI to exploit it. It’s nothing special, but it does show that you really need to implement some kind of security in your network (CHAP or whatever else suits you).

What kind of security do you use for iSCSI? CHAP? None? Leave a comment and let me know!

EDIT: Blog O’Matty has an article on the Solaris iSCSI stack in the August issue of SysAdmin magazine if you’re interested. I find his articles to be very insightful and I highly recommend checking out some of the other ones at prefetch.net. Check it out!

The following comes to you from Ralf Ramge, who has graciously allowed me to post his script and all the instructions below:

“I have a small update. I’ve made the number of backups of each
filesystem easier to handle by replacing the hardcoded number with a
variable. I also added some comments so everybody should be able adjust
both the path of the snapshot directory and the number of backups easily.

I decided to show you the disaster scenario for which this script is
being used.

Let’s take a simple server with the following ZFS file systems:

—
root@static:/> zfs list
NAME                   USED   AVAIL   REFER  MOUNTPOINT
tank                   8,78G  101G   28,5K   /export
tank/backup            4,93G  101G   27,5K   /export/backup
tank/backup/snapshots  4,93G  101G   4,93G   /export/backup/snapshots
tank/backup/sysdata    68K    101G   41,5K   /export/backup/sysdata
tank/repository        205M   101G   205M    repository/packages
tank/zones             3,65G  101G   25,5K   /export/zones
tank/zones/ffxi-sites  3,65G  101G   3,63G   /export/zones/ffxi-sites
root@static:/>
—

tank/backup: usual file system, but with compression=on. Very useful for
snapshots, compression rate is 1.66:1
tank/backup/snapshots: The snapshot directory I use in the scripts
tank/backup/sysdata: That’s a backup directory in which essential system
data is stored. Most important: the complete contents of /etc/zones and
perhaps some stuff like /etc/netmasks, /etc/nsswitch.conf, whatever
could be of importance during reconstruction of the host.
tank/repository/packages: That’s where I keep my packages, scripts,
whatever. Just a repository which is shared (sharenfs=ro,anon=o) and
re-mounted in the local zone as `/var/spool/pkg` … makes things easier.
tank/zones/ffxi-sites: That’s the zonepath of the zone `ffxi-sites-zone`.

I bet it’s getting interesting now, because I’ll explain you how to
backup this zone using the script … and how to reconstruct it from
scratch, on another hardware with different network drivers and such.

Okay. I make backups of tank/backup/sysdata (we need the contents of
/etc/zones for reconstruction), tank/repository/packages and, of
course, the entire local zone itself by backing up
tank/zones/ffxi-sites. Use my script for it, e.g. by executing it for
each file system in your crontab and sending it to a backup server.

This will result in two copies. the local copy in
/export/backup/snapshots (you can edit this path in the script) and one
on a remote server. We us the local copy in case someone shot the
database in the local zone or whatever. And the remote copy is needed in
case of a necessary re-installation of the entire server.

Btw: My private ISP only offers plain FTP for accessing backup servers.
FTP isn’t supported by the backup script. I make the local backups
between 1 an 3 am, and I use the following (imperfect) script at 5am to
transfer all backups to the FTP server:

—
#!/bin/bash
HOST='<ip of ftp server>'
USER='<user>'
PASSWD='<pass>'

BACKUPDIR="/export/backup/snapshots"

if [ ! -d $BACKUPDIR ]; then
echo "Backup Directory doesn't exist"
exit 1
fi

cd $BACKUPDIR

# cleanup ftp server, kamikaze style

ftp -n $HOST << END_CLEANUP > /dev/null
quote USER $USER
quote PASS $PASSWD
mdel *.zfs
bye
END_CLEANUP

# transfer all ZFS images to the ftp server

#for FILE in `ls -rt1 *.zfs`; do
# ftp -n $HOST << END_SCRIPT
# quote USER $USER
# quote PASS $PASSWD
# binary
# put $FILE
# bye
# END_SCRIPT
#done

#for FILE in `ls -rt1 *.zfs`; do
ftp -n $HOST << END_SCRIPT >/dev/null
quote USER $USER
quote PASS $PASSWD
binary
mput *.zfs
# put $FILE
quit
END_SCRIPT
#done

exit 0
—

This script isn’t recommended for production use, there’s a plain text
password in it. I only use it because the ftp server is firewalled and
cannot be accessed from other servers using this login – and if someone
hacked into my machine, he won’t need the backups anyway.

Okay, now disaster happens. We have to reconstruct *everything*. So
insert the Solaris DVD into the drive, install Solaris as usual, apply
the Recommend Patches cluster, and so on. Finally, we’re ready create
the ZFS pool again, we create tank/backup/snapshots and copy the ZFS
images from our remote server backup into this directory. We have our
local copies back.

Now: deploy the three filesystems using `zfs receive`, e.g. `zfs receive
tank/backup/sysdata < tank_backup_sysdata-070611-033000.zfs`

We have our zone configuration back now. We `cd` to
/export/backup/sysdata. Then we’re going to copy the `index` file plus
the ?.xml` files back to /etc/zones, replacing the default ones.

We still have to fix the network interface, which changed due to the
(imaginary) new hardware we have to use now. Enter the zone
configuration using `zonecfg`, e.g.:

—-
root@static:/> zonecfg -z ffxi-sites-zone
zonecfg:ffxi-sites-zone> info
zonename: ffxi-sites-zone
zonepath: /export/zones/ffxi-sites
autoboot: false
pool: pool_default
limitpriv:
net:
address: <your zone ip>
physical: rtls0
rctl:
name: zone.cpu-shares
value: (priv=privileged,limit=10,action=none)
attr:
name: comment
type: string
value: ffxi-sites
zonecfg:ffxi-sites-zone>
—

We have to change the “physical” entry to something else, let’s say
we’re using a X4100 now and so we need `e1000g0` instead of `rtls0`.
`ifconfig -a` shows us the device name.

Type `select net address=<your zone ip>`. Then `set pyhsical=e1000g0`.
Then `end`.

We still have to commit the changes. We do this by typing `commit` and
then we `exit`.

All we have to do now is `zoneadm boot ffxi-sites-zone` and we’re online
again, without having to deploy the zone, too.

Okay, done. We’re online again.

Only continue reading if you want to join some unsupported “hacking
procedures”.

Q: What to do in case our zonepath changed, or we want it to be changed?
A: That’s easy. Grab your `vi` and edit /etc/zones/index. Ignore the “DO
NOT EDIT” warning, that’s for girls only. See below on what to do.

Q: I only want to clone a zone, what to do now?
A: `zfs send <source filesystem> | zfs receive <destination
filesystem>`. Or clone and promote a filesystem, your choice. Then grab
you `vi` and edit /etc/zones/index again. Change the IP like we I shows
you earlier before you boot the cloned zone. And don’t forget chmod 700
your new zonepath.

(Matthew Says: Another way to clone as zone, depending on your patch level, can be seen further down here: http://uadmin.blogspot.com/2006/08/day-in-life-of-solaris-11-admin.html)

Let’s have a look at the index file:

—
root@static:/> cat /etc/zones/index
# Copyright 2004 Sun Microsystems, Inc. All rights reserved.
# Use is subject to license terms.
#
# ident "@(#)zones-index 1.2 04/04/01 SMI"
#
# DO NOT EDIT: this file is automatically generated by zoneadm(1M)
# and zonecfg(1M). Any manual changes will be lost.
#
global:installed:/
ffxi-sites-zone:installed:/export/zones/ffxi-sites:6cffc060-de2f-c972-f548-f36320bcfccf
root@static:/>
—

“ffxi-sites-zone” is the name of my zone. enter the name of your clone here.
“installed” is the zone’s status. Our filesystems contains a bootable
zone, so “installed” is okay. If you configured a new zone manually and
didn’t install it using `zoneadm` yet, it’ll be “configured” … then
deploy a new, bootable zonepath and set the mode to “installed”. I use
this because I use completely installed zone templates for deploying new
zones, that’s much faster than the official (and supported) `zoneadm -z
<zone> install` way.
/export/zones/ffxi-sites is the zonepath. you may change it. Just make
sure the new zonepath exists and it has mode 700. The latter is very
important.

Make sure you made a backup of your orginal before editing it manually.
You edit it on your own risk.

The entry for a cloned zone may look like this:

—
ffxi-sites2-zone:installed:/export/zones/ffxi-sites2:6cffc060-de2f-c972-f548-f36320bcfccf
—

Okay, and now forget what I told you, because Sun won’t give you the
least bit of support if something goes wrong The fact that I use
such methods for production use won’t help you much if you have a typo
where you shouldn’t have made one.

Hope you had some fun.”

Thanks for the submission Ralf!

I’ve been trying to get a HA solution put together for one of our software projects here at EMC and I figured I’d share the configuration of these two products in the environment that we’re using. I have to write the documentation for it anyway, so I might as well post it here for everyone else to see and learn from first

We are going to configure 2 machines to be in a Active/Passive failover situation, which means that if the primary machine dies, the secondary will take over its identity and continue functioning as previously.

Primary: lava2042 (10.5.140.42) (192.168.1.1 for crossover interface)
Secondary: lava2138 (10.5.140.138) (192.168.1.2 for crossover interface)
HA-address: lava2222 (10.5.140.222)

Configuring heartbeat

Step 1.
Install Heartbeat and DRBD on BOTH machines that you are planning on configuring. This should be a very straightforward step and I’m not going to go into detail.

Step 2.
We’re going to need a way to connect the machines, you can use either a crossover cable from an additional ethernet port to the other or you can use a serial cable. In this example I’m using a crossover cable.

Step 3.
Now we’re going to configure the /etc/ha.d/ha.cf file for our machine. Here is what I’ve put into the /etc/ha.d/ha.cf file ON EACH MACHINE:
bcast eth1
keepalive 2
warntime 10
deadtime 30
initdead 120
udpport 694
auto_failback on
node lava2042
node lava2138
Check this page if you have trouble or are using a serial connection instead of a crossover cable. It has instructions on how to configure this file for a serial interface.

Step 4.
Now configure the /etc/ha.d/authkeys file ON EACH MACHINE for what kind of security and file checking you want, I don’t care about security in this example so I put this is the file since it’s the fastest:
auth 2
2 crc
(See here for more information)

We’ll also need to configure the /etc/ha.d/haresources file, but we won’t do that until we get DRBD working correctly.

Configuring DRBD

Step 1.
The /etc/drbd.conf file needs to be configured. It should already have an example setup in the file. I used the already existing resource r0 and edited the nodes. Inside the “resource r0 {” bracket there should be a part that says “on <something>”. Here is what I put for my 2 nodes:
on lava2042 {
device /dev/drbd0;
disk /dev/sda1;
address 10.5.140.42:7788;
meta-disk internal;
}

on lava2138 {
device /dev/drbd0;
disk /dev/sda8;
address 10.5.140.138:7788;
meta-disk internal;
}

Now let me give a little background. I had already made the /dev/sda1 partition on lava2042 and the /dev/sda8 partition on lava2138, each 1 gig to store the data that was going to be shared. /dev/drbd0 is the device that will actually be mounted and read from. Other than that, I left the entire file to be it’s defaults. Make sure to comment out any other resources unless you need more than one filesystem replicated.

Step 2.
Make sure to load the drbd module by doing a modprobe drbd and check the dmesg command to make sure the output looks correct (Sorry, I don’t have what it should look like, I’ll keep better notes in the future).

Step 3.
Now we need to initialize our metadata for DRBD. We do this by running this on EACH machine:
drbdmeta create-md r0
Where r0 is the name of the resource from the /etc/drbd.conf file. You should now be able to run the following on each machine:
drbdadm up all
After running these two commands, you should be able to check dmesg and /proc/drbd to see the status of your filesystem.

Step 4.
The next step is to force one of the machines to be the primary and create a filesystem. In this case I’m choosing lava2042 as the primary, so I will run this on the machine:
lava2042# drbdsetup /dev/drbd0 primary -o
This will do the initial sync between the machines, you should only need to do this once. After that, run this command:
lava2042# drbdadm primary all
To force lava2042 into the primary state and make /etc/drbd0 usable. From here you can create a filesystem by doing a:
lava2042# mkfs.ext3 /dev/drbd0 (or whatever filesystem you want)
And mount the filesystem to check it out (make sure to unmount it after you’re done)

You should now be able to do a drbdadm primary all on either machine (while in a Secondary/Secondary state (check /proc/drbd)) and mount the filesystem

Step 5.
Okay, now let’s drop back into secondary mode for lava2042 by doing this:
lava2042# drbdadm secondary all
The /proc/drbd file should look something like this:
version: 8.0.3 (api:86/proto:86)
SVN Revision: 2881 build by root@lava2138, 2007-06-18 09:50:33
0: cs:Connected st:Secondary/Secondary ds:UpToDate/UpToDate C r---
ns:316952 nr:1221300 dw:1222380 dr:346211 al:8 bm:107 lo:0 pe:0 ua:0 ap:0
resync: used:0/31 hits:81456 misses:98 starving:0 dirty:0 changed:98
act_log: used:0/257 hits:262 misses:8 starving:0 dirty:0 changed:8
(Important part bolded) The filesystem needs to be in a Secondary state for both machines in order for heartbeat to work properly

And now we’re going to edit the /etc/ha.d/haresources file to take care of sharing the filesystem. Here’s what I have in the file:
lava2042 drbddisk::r0 Filesystem::/dev/drbd0::/opt/EMC::ext3 10.5.140.222 httpd
Let’s go through it line by line:
lava2042 – the machine that will be the primary node
drbddisk::r0 – activate the r0 resource disk (make sure r0 corresponds to whatever your resource is named)
Filesystem::/dev/drbd0::/opt/EMC::ext3 – mount /dev/drbd0 on /opt/EMC as an ext3 filesystem
10.5.140.222 – the IP address for our solution (see the beginning of the post)
httpd – the service we’re going to watch over and take care of, in this case httpd (which wasn’t really what I was configuring, but it’s the easiest to show as an example)
Don’t forget this file has to be the same on BOTH MACHINES.

Step 6.
Make sure heartbeat and the service(s) you’re watching DO NOT start at boot, otherwise things get really ugly if when you screw up:
chkconfig heartbeat off
chkconfig httpd off
/etc/init.d/httpd stop
(on both machines)

Step 7. (The cross your fingers step)
Alright, it’s finally time to test your failover configuration. First, we need to start heartbeat on the primary machine:
lava2042# /etc/init.d/heartbeat start
Then, start it on the secondary machine
lava2138# /etc/init.d/heartbeat start

You should now be able to ping the cluster IP (lava2222 or 10.5.140.222). You can also check that the /dev/drbd0 filesystem is mounted on the primary node using df. Check the /var/log/messages file on either machine for debugging information.

The moment of truth
Go to your primary node and yank the power cable out of the back. Head back to your machine and carefully watch the /var/log/messages file on the secondary node. You should see information about the link being down, the drbd having trouble accessing the filesystem, then heartbeat should kick in and start taking over, mounting the filesystem and finally starting your httpd service. Congratulations, you have now successfully failed over.

If you have an error, check the error messages and see if you can figure out what to do, if you need any help leave a comment or email me and I’ll try and help. Hopefully this helps somebody as this took me quite a while to figure out, having never worked with either piece of software.

Additional links:
Information mostly pulled from:
http://linux-ha.org/GettingStarted
http://www.linux-ha.org/DRBD/GettingStarted
http://www.linux-ha.org/DRBD/HowTo

P.S. Ralf Ramge emailed me an updated version of his bash zfs backup script. I am still working on getting it put together to post. Thanks for the email Ralf

Okay, so almost everyone has heard about Project Indiana right? The one where SUN tries to make Solaris like Linux so they can compete in more areas and get all the wonderful features of Solaris on more platforms. Well, I have a suggestion for you:

Don’t use Java for your installer.

Yea, sure, it’s fine if you use Java for the GUI installer *if* the machine can support it, but what about when I want to install on a machine with a minimal amount of RAM? I mean, even your text-based installer uses Java, and for what? Replace your Java text-based installer with something like Curses (or something equivalent). Make it easier to install Solaris for people who are in college. The install for Solaris almost assumes you’ve been through the install before and know what you’re doing. If you really want more adoption in a learner’s market, you need to make it simpler to install.

In other news: Happy Birthday OpenSolaris (you’re 2! whee!). Now if I could only install you on all of my really really old hardware so I could make headless servers. Alas needing much RAM to work.

Anyone know a good PCI SATA card that will work in my Blade 150? I’m tempted to get this, but I’m not sure if it’ll support JBODs without flashing the BIOS on the card, which would be a pain to do on a SPARC system.

Alright, so for the last 3 days or so, my main Solaris machine has been going crazy and kernel panicing about once every day or so, which is extremely annoying because every time it panics the machine reboots (and this machine has 3 zones that are in current use, so I get 3 calls about “why did my machine reboot”). Luckily, none of our servers here are production, so I get calls from development and not angry customers. So, I’m setting out to try and figure out why the machine is panicing. Here’s what I’m getting from the logs:

From the vmcore file:
ZFS: I/O failure (write on <unknown> off 0: zio 6000620cd40 [L0 ZIL intent log] 1000L/1000P DVA[0]=<0:1300cb9000:1000> zilog uncompressed BE contiguous birth=208621 fill=0 cksum=8eafa7df8b7cb3e:f2fd0

From the /var/adm/messages file:
Jun 5 12:01:11 lava2051 fctl: [ID 517869 kern.warning] WARNING: fp(0)::GPN_ID for D_ID=650700 failed
Jun 5 12:01:11 lava2051 fctl: [ID 517869 kern.warning] WARNING: fp(0)::N_x Port with D_ID=650700, PWWN=5006016841e019a7 disappeared from fabric
Jun 5 12:01:30 lava2051 scsi: [ID 243001 kern.info] /pci@1c,600000/fibre-channel@1/fp@0,0 (fcp0):
Jun 5 12:01:30 lava2051 offlining lun=0 (trace=0), target=650700 (trace=2800004)
Jun 5 12:06:28 lava2051 unix: [ID 836849 kern.notice]
Jun 5 12:06:28 lava2051 ^Mpanic[cpu2]/thread=2a101061cc0:
Jun 5 12:06:28 lava2051 unix: [ID 809409 kern.notice] ZFS: I/O failure (write on <unknown> off 0: zio 6000620cd40 [L0 ZIL intent log] 1000L/1000P DVA[0]=<0:1300cb9000:1000> zilog uncompressed BE contiguous birth=208621 fill=0 cksum=8eafa7df8b7cb3e:f2fd0a04af0e949e:1a:f3): error 5)
... some stuff ...
Jun 5 12:09:55 lava2051 savecore: [ID 570001 auth.error] reboot after panic: ZFS: I/O failure (write on <unknown> of
f 0: zio 6000620cd40 [L0 ZIL intent log] 1000L/1000P DVA[0]=<0:1300cb9000:1000> zilog uncompressed BE contiguous birth=208621 fill=0 cksum=8eafa7df8b7cb3e:f2fd0
Jun 5 12:09:55 lava2051 savecore: [ID 748169 auth.error] saving system crash dump in /var/crash/lava2051/*.1

Repeat x 3 so far. Like I said, extremely annoying.

Here’s what I think the problem is so far: I have a 500g ZFS pool built on a single Clariion LUN that is exported to this machine. From the looks of it the machine is having trouble seeing the LUN all the time, when it disappears ZFS freaks out and panics because of a I/O failure. Now that I know what the problem is, I have no idea how to make the LUN stop disappearing. Guess I’m off to check some Clariion logs and see where that gets me. Anyone out there have any other suggestions on how I could go about fixing this problem? I have little experience in working with core dumps. I would be extremely grateful

P.S. Yes, I know I should have mirrored the ZFS pool on 2 or more devices in case of a problem like this. This is more my “proof-of-concept” machine where I try out new things and see how developers/QA react to them.

UPDATE:
It looks like the problem was a problem on the Clariion side, for the meantime, we exported a LUN from a different clariion, did a zfs attach, waited for the data to be mirrored and then detached the old one. Fixed! <3 ZFS

UPDATE 2:
Now the data is mirrored to a different Clariion. fun fun. Interestingly enough, EMC doesn’t officially support ZFS on Clariion, only on Symmetrix.

Here’s a nifty little submission from Ralf Ramge. It will do a ZFS snapshot backup to a local directory, a remote machine and also clone and promote the filesystem on the remote machine. It keeps the last 7 backups around. Take a look:

#!/bin/bash
# backup_zfssnap.sh, (c) 2007 ralf [dot] ramge [at] webde [dot] de

BACKUPDIR="/export/backup/snapshots"
DSTAMP=`date '+%y%m%d-%H%M%S'`
FILESYS=$1
DEST=$2
REPLICA=$3
BACKUPNAME=`echo $FILESYS | sed 's/\//_/g'`
BACKUPFILE=$BACKUPNAME"-"$DSTAMP".zfs"
SNAPSHOT=$FILESYS"@backup-"$DSTAMP

if [ ! -d $BACKUPDIR ]; then
echo "Backup Directory doesn't exist"
exit 1
fi

cd $BACKUPDIR

# Check here if we have 7 backup files, create them if we don't
COUNT_FILES=`ls -1 $BACKUPNAME* | wc -l`
if [ $COUNT_FILES -le 1 ]; then
for COUNT in 1 2 3 4 5 6 7
do
if [ ! -f $BACKUPNAME"-000000-00000"$COUNT".zfs" ]; then
touch $BACKUPNAME"-000000-00000"$COUNT".zfs"
sleep 1
fi
done
fi

# Check here that we have less than 8 backup files
COUNT_FILES=`ls -1 $BACKUPNAME* | wc -l`
if [ $COUNT_FILES -gt 7 ]; then
# echo "More than 7 backup files exist"
# exit 1
while [ $COUNT_FILES -gt 7 ]
do
OLDEST_BACKUP_FILE=`ls -rt1 $BACKUPNAME* | head -1`
rm $OLDEST_BACKUP_FILE
let COUNT_FILES=COUNT_FILES-1
done
fi

# Find the oldest backup file to delete
OLDEST_BACKUP_FILE=`ls -rt1 $BACKUPNAME* | head -1`

# Create the snapshot
zfs snapshot $SNAPSHOT

# Create a filesystem image in the local backup directory
zfs send $SNAPSHOT > $BACKUPDIR"/"$BACKUPFILE

# Check for $2 and, if exists, create a second copy on a remote host for tape archival
if [ ! -z $2 ]; then
`zfs send $SNAPSHOT | ssh root@$2 "cat >$BACKUPDIR/$BACKUPFILE"`
fi

# Check for $3 and, if exists, mirror the filesystem on the remote host
if [ ! -z $3 ]; then
`ssh root@$2 "zfs receive $3 < $BACKUPDIR/$BACKUPFILE"`
fi

# Check for $4 and, if exists, clone and promote the filesystem on the remote host
if [ ! -z $4 ]; then
`ssh root@$2 "zfs clone $SNAPSHOT $4; sleep 30; zfs promote $4"`
fi

# Get the trash out of the house
rm $OLDEST_BACKUP_FILE
if [ ! -z $2 ]; then
ssh root@$2 "rm $BACKUPDIR/$OLDEST_BACKUP_FILE"
fi

SNAPLIST=`zfs list -H | grep $FILESYS | grep @backup | cut -f1`
for i in $SNAPLIST; do
zfs destroy $i
done

# Exit cleanly
exit 0

Thanks for the submission Ralf! (I changed your email address in the script comments so you wouldn’t get spam)

So I read quite a few Solaris blogs and when this popped up this morning I decide to take a look (warning, the movie that the post links to is >500 megs)

I had expected to hear a pretty good discussion around the “linuxification” of Solaris and how Ian Murdock plans to approach it, turns out about halfway through I was a little disappointed by the zealotry of some of the audience members. To *me* at least, it seems like they were arguing trivial points that led to the discussion going way off track. About 2/3 of the way through I turned if off so I could concentrate of a perl script I was writing (see below :P). Here’s what my opinion is about the subject:

• Who cares if you call the Linux userland “Linux” instead of “GNU”?? Most managers and people engaged in casual conversation reference the entire userland as Linux anyway, it makes it easier to talk about. Yes, everyone that is involved in OSS knows that Linux is just the kernel, but that seems like a pretty trivial point to make when you’re not even discussing that in the first place.
•  Solaris needs a better userland. This, I agree with, I used to hate Solaris because I didn’t know how to do things with it, I think Ian makes a good point in that in college, the majority of students that ran an “alternative” operating system were running Linux, they knew it, they loved it, they wanted to use it when they got out of college (at least, I did). I certainly wish I had been experimenting with Solaris in college (I think I only did once). Now that I’ve been administering Solaris for the last year, it is by far my favorite administration platform, it might not be great for everything, but I certainly love it for my sysadmin work. Now if only the rest of the world would come to see the way I feel…
• I commented about GNU having a better userland on a post on OSnews.com some time ago and someone alerted me to the fact that Solaris utilities have a better POSIX standardization than GNU utilities. After doing some poking around I definitely agree with that, I mean, in Linux, do you use -option? –option -option=? Is the manpage helpful ? (Hint: no). What I really miss are the features of the GNU tools, windowing in grep, -iname option for find. Things like that.
• I read an article a couple days ago about how Solaris has a more powerful administration interface, however, Linux has an easier administration interface. I would say that’s about true. When it comes down to it, a lot of people are going to choose what they think will the best and easiest to administer. More education is needed. That and Linux is beating Solaris in online documentation by about 1000 webpages for every 1. Finding what I need for Solaris has always been a more extensive challenge to my GoogleFu than with Linux.
• Almost every person that commented in the forum with Ian reminded me of that annoying guy from CS classes in college who thought he knew everything and was very elitist. Ugh, I just want to hit someone.

There you go, personal opinions that have almost no logical reason other than personal preference, way to go internet.

Ugh, re-reading this it is clear I am not an english major. Sorry for the disjointedness.

Hi everyone, I’m back again with another perl script to hopefully be useful to a few of you.

Firstly, the script: http://lee.hinmanphoto.com/files/zdiff.txt (formatting long scripts in wordpress’ crazy editor is a very long and arduous process, thus I’m just linking to the script in this case, if anyone knows of a better place to stick it let me know). chmod +x it and away you go!

Edit: Sun was nice enough to host the file for me, here’s a link to their version in case the other one goes down: http://www.sun.com/bigadmin/scripts/submittedScripts/zdiff.txt

In a nutshell, here’s what it does:

• Allows you to diff a file inside a ZFS snapshot with the current file in the filesystem and (optionally) print out the line differences
• Recursively diff an entire snapshot using md5 sums and (optionally) printing out the line differences
• Display the md5 sums for each file in a ZFS snapshot and filesystem (this can get old to look at very quickly)

Basically, that doesn’t mean a whole lot, here’s the output from the -h option:

ZFS Snapshot diff
./zdiff.pl [-dhirv] <zfs shapshot name> [filename]

-d Display the lines that are different (diff output)
-h Display this usage
-i Ignore files that don't exist in the snapshot (only necessary for recursing)
-r Recursively diff every file in the snapshot (filename not required)
-v Verbose mode

[filename] is the filename RELATIVE to the ZFS snapshot root. For example, if
I had a filesystem snapshot called pool/data/zone@initial. The filename '/etc/passwd'
would refer to the filename /pool/data/zone/etc/passwd in the filesystem and filename
/pool/data/zone/.zfs/snapshot/initial/etc/passwd in the snapshot.

A couple of examples:
./zdiff.pl -v -r -i pool/zones/lava2019@Fri
Checks the current pool/zones/lava2019 filesystem against the snapshot
returning the md5sum difference of any files (ignore files that don't
exist in the snapshot). With verbose mode

./zdiff.pl -d pool/zones/lava2019@Mon /root/etc/passwd
Check the md5sum for /pool/zones/lava2019/root/etc/passwd and compare
it to /pool/zones/lava2019/.zfs/snapshot/Mon/root/etc/passwd. Display
the lines that are different also.

Here’s what the output is going to look like:

-bash-3.00# ./zdiff.pl -d -v -r -i pool/zones/lava2019@Fri
Recursive diff on pool/zones/lava2019@Fri
Filesystem: /pool/zones/lava2019, Snapshot: Fri
Comparing: /pool/zones/lava2019/
to: /pool/zones/lava2019/.zfs/snapshot/Fri/
** /pool/zones/lava2019/root/etc/shadow is different
** MD5(/pool/zones/lava2019/root/etc/shadow)= 04fa68e7f9dbc0afbf8950bbb84650a6
** MD5(/pool/zones/lava2019/.zfs/snapshot/Fri/root/etc/shadow)= 4fc845ff7729e804806d8129852fa494
17d16
< tom:*LK*:::::::
** /pool/zones/lava2019/root/etc/dfs/dfstab is different
** MD5(/pool/zones/lava2019/root/etc/dfs/dfstab)= 8426d34aa7aae5a512a0c576ca2977b7
** MD5(/pool/zones/lava2019/.zfs/snapshot/Fri/root/etc/dfs/dfstab)= c3803f151cb3018f77f42226f699ee1b
13d12
< share -F nfs -o rw -d "Data" /data

etc, etc, etc.

I am planning on using it so I can audit certain files on different zones (like /etc/passwd) against an initial ZFS snapshot to see what’s changed. Nice little way to keep track of stuff. Email me with any bugs. Matthew dot hinman at gmail dot com.