Sunday, May 1, 2011

ZFS and Scrubbing

So, I've been on call most of the last week, which basically means I can't stray from a PC for long, and so I've been messing with zfs scrub automation.

One of the nice things about zfs is that you can run a scrub, which will check for consistency, checksum and mirroring errors.  Since I'm using consumer grade drives, I want to run it once a week or so.  Using bits and pieces of scripts from across the interwebs, I've managed to cobble together a bash script that I run out of crontab.  Original was from here.  It runs once a week on sundays early in the morning and scrubs each of the files systems:


#!/usr/bin/env bash
#VERSION: 0.2
#AUTHOR: gimpe
#EMAIL: gimpe [at] hype-o-thetic.com
#WEBSITE: http://hype-o-thetic.com
#DESCRIPTION: Created on FreeNAS 0.7RC1 (Sardaukar)
# This script will start a scrub on each ZFS pool (one at a time) and
# will send an e-mail or display the result when everyting is completed.
#CHANGELOG
# 0.2: 2009-08-27 Code clean up
# 0.1: 2009-08-25 Make it work
#SOURCES:
# http://aspiringsysadmin.com/blog/2007/06/07/
# scrub-your-zfs-file-systems-regularly/
# http://www.sun.com/bigadmin/scripts/sunScripts/zfs_completion.bash.txt
# http://www.packetwatch.net/documents/guides/2009073001.php
# e-mail variables
FROM=root@smaug.censored.com
TO=jeff@censored.com
SUBJECT="Scrub of Pools on smaug"
BODY=""


# arguments
VERBOSE=1
SENDEMAIL=1
# work variables
ERROR=0
SEP=" - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - "
RUNNING=1

# commands & configuration
ZPOOL=/usr/sbin/zpool
PRINTF=/usr/bin/printf
MSMTP=/usr/bin/mailx

# print a message
function _log {
   DATE="`date +"%Y-%m-%d %H:%M:%S"`"
   # add message to e-mail body
   BODY="${BODY}$DATE: $1\n"
   # output to console if verbose mode
   if [ $VERBOSE = 1 ]; then
      echo "$DATE: $1"
   fi

}

# find all pools
pools=$($ZPOOL list -H -o name)
# for each pool
for pool in $pools; do
   # start scrub for $pool
   _log "starting scrub on $pool"
   zpool scrub $pool
   RUNNING=1
   # wait until scrub for $pool has finished running
   while [ $RUNNING = 1 ]; do
   # still running?
   if $ZPOOL status -v $pool | /usr/xpg4/bin/grep -q "scrub in progress"; then
     sleep 60
   # not running
   else
      # finished with this pool, exit
      _log "scrub ended on $pool"
      _log "`$ZPOOL status -v $pool`"
      _log "$SEP"
      RUNNING=0
      # check for errors
      if ! $ZPOOL status -v $pool | /usr/xpg4/bin/grep -q "No known data errors"; then
         _log "data errors detected on $pool"
         ERROR=1
      fi
   fi
   done
done

# change e-mail subject if there was error
if [ $ERROR = 1 ]; then
   SUBJECT="${SUBJECT}: Error(s) Detected"
fi
# send e-mail
if [ $SENDEMAIL = 1 ]; then
   $PRINTF "$BODY" | $MSMTP -r $FROM -s "$SUBJECT" $TO
fi

So, of course, I wanted it to send me an email with the results of the scrub.  Since I didn't want to end up in sendmail hell, I ended up installing postfix, which is MUCH easier to use, and setup only took a couple of hours.  More on setting up postfix in another post.

1 comment:

  1. Awesome script. I'm using it with on Debian Wheezy + ZFS on Linux.

    Just had to replace
    "zpool scrub $pool"
    with
    "$ZPOOL scrub $pool"

    modify paths (including grep)
    and all is well.
    Thanks a bunch.

    ReplyDelete