Monday, September 8, 2008

Bacula with USB disks, vchanger altered

Last month I've successfully implemented a new backup solution using Bacula. The old backup consist in a combination of Linux and Windows scripts; we had abandoned tapes two years ago, in favor of USB removable disks.
Now I've wanted to re implement everything using Bacula for all the backup process: starting from db dump scripts and similar, ending with the writing of the backup on the USB disks. I've also wanted Bacula to alert someone every day, in order to regularly rotate disks with daily frequency.
Bacula doesn't natively support a system to cycle USB disks, which is a lack when removable disks with big capacities are cheap and they're lowering their price every day. Anyway: it can use tapes autochangers, and someone started to write fake scripts that uses USB disks as tapes , simulating the autochanger.
It was difficult to find a script that works well: after a few study and research I've found one written by Josh Fisher in 2006. It seemed to me that was the best one so I'd started to implement it. Unfortunately I've found something wrong with this script, so I've altered 2 or 3 lines of code in order to make it work.
I've started to test it two weeks ago and it seems to works very well.
Below is the altered script, you can find other useful informations with the Joe's howto posted there.
Enjoy:

#!/bin/sh
#
# Bacula interface to virtual autochanger using removable disk drives
#
# Based (somewhat) on the "disk-changer" script from bacula 1.39.26
#
# Vchanger is a Bacula autochanger script that emulates a conventional
# magazine-based tape library device using removable disk drives.
# Partitions on the removable drives are used as virtual magazines,
# where each "magazine" contains the same number of virtual slots. Each
# "slot" holds one virtual tape, where a "tape" is a regular file that
# Bacula treats as a "Device Type = File" volume.
#
# This script will be invoked by Bacula using the Bacula Autochanger
# Interface and will be passed the following arguments:
#
# vchanger "changer-device" "command" "slot" "archive-device" "drive-index"
# $1 $2 $3 #4 #5
#
# See the Bacula documentation for Autochanger Interface details
#
# Copyright (C) 2006 Josh Fisher
#
# Modified by Simone Tregnago on 07-20-2008
#
# Permission to use, copy, modify, distribute, and sell this software
# and its documentation for any purpose is hereby granted without fee,
# provided that the above copyright notice appears in all copies. This
# software is provided "as is" without express or implied warranty.
#
# This software is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
#

#
# log whats done
#
dbgfile="/var/bacula/vchanger.log"
# to turn on logging, uncomment the following line
touch $dbgfile
#

#
# Write to a log file
# To log debugging info, create file /var/bacula/vchanger.log
# with write permission for bacula-sd user. To stop logging,
# delete file /var/bacula/vchanger.log
#
function debug()
{
if test -e $dbgfile; then
echo "`date +\"%Y%m%d-%H:%M:%S\"` $*" >> $dbgfile
fi
}

#
# Return length of string $1
#
function strlen ()
{
expr length $1
}

#
# Prepend zeros to $1 and return a string that is $2 characters long
#
function mklen ()
{
o1=$1
while [ `eval strlen ${o1}` -lt ${2} ]; do
o1="0${o1}"
done
echo $o1
}

#
# Initialize autochanger's state directory if not already initialized
#
function init_statedir() {
debug "Initializing $statedir"
# Create state dir if needed
if [ ! -d "${statedir}" ]; then
mkdir "${statedir}"
if [ $? -ne 0 ]; then
echo "Could not create ${statedir}"
exit 1
fi
fi
chmod 770 "${statedir}"
if [ $? -ne 0 ]; then
echo "Could not chmod ${statedir}"
exit 1
fi
# Create nextmag file to hold max magazine index used
if [ ! -f "${statedir}/nextmag" ]; then
echo 1 >"${statedir}/nextmag"
if [ $? -ne 0 ]; then
echo "Could not create ${statedir}/nextmag"
exit 1
fi
fi
chmod 660 "${statedir}/nextmag"
if [ $? -ne 0 ]; then
echo "Could not chmod ${statedir}/nextmag"
exit 1
fi
# Check nextmag value
nextmag=`cat "${statedir}/nextmag"`
if [ $? -ne 0 -o "${nextmag}" == "" -o $nextmag -lt 1 -o $nextmag -gt 99 ]; then
echo "${statedir}/nextmag has invalid value"
return 1
fi
# Create 'loaded' files for each virtual drive that hold the slot
# number currently loaded in that 'drive'
i=0
while [ $i -le $maxdrive ]; do
if [ ! -f "${statedir}/loaded${i}" ]; then
echo "0" 2>/dev/null >"${statedir}/loaded${i}"
if [ $? -ne 0 ]; then
echo "Could not create ${statedir}/loaded${i}"
exit 1
fi
chmod 660 "${statedir}/loaded${i}"
if [ $? -ne 0 ]; then
echo "Could not chmod ${statedir}/loaded${i}"
exit 1
fi
fi
i=`expr ${i} + 1`
done
}


#
# Initialize magazine if not already initialized
#
function init_magazine() {
debug "Initializing magazine"
# Get max magazine index that has been used
nextmag=`cat "${statedir}/nextmag"`
if [ $? -ne 0 -o "${nextmag}" == "" ]; then
echo "Failed to read ${statedir}/nextmag"
exit 1
fi
# Check magazine for existing index
if [ -f "${mountpoint}/index" ]; then
# retrieve existing magazine index
mi=`cat "${mountpoint}/index"`
if [ $? -ne 0 ]; then
echo "Failed to read ${mountpoint}/index"
exit 1
fi
# must be 1-99
if [ $mi -lt 1 -o $mi -gt 99 ]; then
echo "Magazine has invalid index ${mi}"
exit 1
fi
else
# new magazine, so assign it the next avail index
mi=`expr ${nextmag} + 1`
if [ $mi -lt 0 -o $mi -gt 99 ]; then
echo "Max magazines exceeded"
exit 1
fi
echo $mi 2>/dev/null >"${mountpoint}/index"
if [ $? -ne 0 ]; then
echo "Failed to write ${mountpoint}/index"
exit 1
fi
fi
# make sure max index used is up to date
if [ $mi -gt $nextmag ]; then
echo $mi 2>/dev/null >"${statedir}/nextmag"
if [ $? -ne 0 ]; then
echo "Failed to update ${statedir}/nextmag"
exit 1
fi
fi
# make magazine index 2 digits
magindex=`eval mklen ${mi} 2`
# setup slot files (ie. virtual tapes)
i=1
while [ $i -le $magslots ]; do
s=`eval mklen ${i} 3`
f="${mountpoint}/m${magindex}s${s}"
if [ ! -f "${f}" ]; then
touch "${f}" 2>/dev/null >/dev/null
if [ $? -ne 0 ]; then
echo "Failed to create ${f}"
exit 1
fi
fi
i=`expr ${i} + 1`
done
return 0
}


#
# check parameter count on commandline
#
function check_parm_count() {
pCount=$1
pCountNeed=$2
if test $pCount -lt $pCountNeed; then
echo "usage: vchanger ctl-device command [slot archive-device drive-index]"
echo " Insufficient number of arguments arguments given."
if test $pCount -lt 2; then
echo " Mimimum usage is first two arguments ..."
else
echo " Command expected $pCountNeed arguments"
fi
exit 1
fi
}


# Setup arguments
ctl=$1
cmd="$2"
slot=$3
device=$4
drive=$5

# Setup default config values
magslots=10
maxdrive=0
statedir="/var/bacula/vchanger"
mountpoint=

# Pull in conf file
if [ -f $ctl ]; then
. $ctl
else
echo "Config file ${ctl} not found"
exit 1
fi

# check for required config values
if [ "${mountpoint}" == "" ]; then
echo "Required variable 'mountpoint' not defined in ${ctl}"
exit 1
fi
if [ "${magslots}" == "" -o $magslots -lt 1 -o $magslots -gt 999 ]; then
echo "Ivalid value for 'magslots' in ${ctl}"
exit 1
fi
if [ "${maxdrive}" == "" -o $maxdrive -lt 0 -o $maxdrive -ge $magslots ]; then
echo "Invalid value for 'maxdrive' in ${ctl}"
exit 1
fi
if [ "${statedir}" == "" ]; then
echo "Invalid value for 'statedir' in ${ctl}"
exit 1
fi

# Initialize state directory for this autochanger
init_statedir

# Check for special cases where only 2 arguments are needed,
# all others are a minimum of 5
#
case $2 in
list)
check_parm_count $# 2
;;
slots)
check_parm_count $# 2
;;
*)
check_parm_count $# 5
if [ $drive -gt $maxdrive ]; then
echo "Drive ($drive) out of range (0-${maxdrive})"
exit 1
fi
if [ $slot -gt $magslots ]; then
echo "Slot ($slot) out of range (1-$magslots)"
exit 1
fi
;;
esac

debug "Parms: $ctl $cmd $slot $device $drive"

case $cmd in
unload)
debug "Doing vchanger -f $ctl unload $slot $device $drive"
ld=`cat "${statedir}/loaded${drive}"`
if [ $? -ne 0 ]; then
echo "Failed to read ${statedir}/loaded${drive}"
exit 1
fi
if [ $slot -eq $ld ]; then
echo "0" >"${statedir}/loaded${drive}"
if [ $? -ne 0 ]; then
echo "Failed to write ${statedir}/loaded${drive}"
exit 1
fi
unlink "${device}" 2>/dev/null >/dev/null
exit 0
fi
if [ $ld -eq 0 ]; then
echo "Drive ${drive} Is Empty"
else
echo "Storage Element ${slot} is Already Full"
fi
exit 1
;;

load)
debug "Doing vchanger $ctl load $slot $device $drive"
ld=`cat "${statedir}/loaded${drive}"`
if [ $? -ne 0 ]; then
echo "Failed to read ${statedir}/loaded${drive}"
exit 1
fi
if [ $ld -eq 0 ]; then
unlink "${device}" 2>/dev/null >/dev/null
# make sure slot is not loaded in another drive
i=0
while [ $i -le $maxdrive ]; do
if [ $i -ne $drive ]; then
ldi=`cat "${statedir}/loaded${i}"`
if [ $ldi -eq $slot ]; then
echo "Storage Element ${slot} Empty (loaded in drive ${i})"
exit 1
fi
fi
i=`expr ${i} + 1`
done
init_magazine
if [ $? -ne 0 ]; then
echo "Magazine Not Loaded"
exit 1
fi
s=`eval mklen ${slot} 3`
ln -s "${mountpoint}/m${magindex}s${s}" "${device}"
echo $slot >"${statedir}/loaded${drive}"
exit 0
else
echo "Drive ${drive} Full (Storage element ${ld} loaded)"
exit 1
fi
;;

list)
debug "Doing vchanger -f $ctl -- to list volumes"
init_magazine
if [ $? -ne 0 ]; then
echo "Magazine Not Loaded"
exit 1
fi
i=1
while [ $i -le $magslots ]; do
s=`eval mklen ${i} 3`
echo "${i}:m${magindex}s${s}"
i=`expr ${i} + 1`
done
exit 0
;;

loaded)
debug "Doing vchanger -f $ctl $drive -- to find what is loaded"
cat "${statedir}/loaded${drive}"
exit 0
;;

slots)
debug "Doing vchanger -f $ctl -- to get count of slots"
echo $magslots
exit 0
;;
esac
# eof


Here is the config of the 1st changer(vchanger1.conf):

# /etc/bacula/vchanger1.conf
baculasd="backup-sd"
baculasd_user=bacula
bconsole=/usr/bin/bconsole

magslots=1
maxdrive=0

mountpoint=/mnt/vchanger1/magazine
statedir=/var/bacula/vchanger1
# eof

P.S.: remember to take care of file permissions, since Bacula needs to read them from its user; so I suggest to chown the USB disks and vchanger folder with user 'bacula' and group 'tape' (or whatever you use to run the bacula-sd daemon ). For example, considering the folder used in the config above, you could run:
chown bacula:tape /mnt/vchanger1/magazine/ -R
and
chown bacula:tape /var/bacula/vchanger1/ -R

Update:
You can download script and configuration files using the following links:
vchanger
vchanger1.conf

7 comments:

Anonymous said...

This is great info to know.

Tim Small said...

Hello. Thanks for posting the modified script, but blogger has made a bit of a mess of it...

Any chance you could post the file somewhere?

Cheers,

Tim.

Simone Tregnago said...

Tim,
I've updated the post with two links that point to script and configuration files

brian_252 said...

Simone,

I copied the Bacula and Removable Disk HOWTO to the Bacula wiki. This is the version i found from Jim Fisher's attachment in the mailing list.

I also posted some suggestions on pool configuration on my blog here Bacula Removable Disk configuration suggestions

Brian

Tim Small said...

BTW, vchanger has been rewritten by the original author in C++...

http://sourceforge.net/projects/vchanger/

I'm currently giving it a spin (or would be if my esata <-> ide converter hadn't just died - awaiting a new one at the moment).

Simone Tregnago said...

tim,
so a new vchanger is coming, rewritten in C++... that's very good!

Elliot 'Zulu' Mokoena said...

I have followed the tutorial and all is working ok except I can't label the volumes. I get the error" Unable to open device "usb-changer-1-drive-0" (/var/spool/bacula/usbchanger1/drive0): ERR=dev.c:549 Could not open: /var/spool/bacula/usbchanger1/drive0, ERR=Is a directory"