[CLUE-Tech] tape backups

[Ed Hill]

On Wed, 2002-08-14 at 21:26, David Anselmi wrote:
> Jim Ockers wrote:
> > 
> > Does anyone have any ideas how tar can be used but still achieve native
> > drive speeds on the backup subsystem?  I've seen various schemes for
> > buffering the tar input and output using dd (through the system RAM)
> > but I've tried that and was underwhelmed by the speed increase.
> > 
> > Part of the optimization on commercial backup programs is that they
> > stream several backups to the tape at the same time.  This means
> > that the software has to keep a catalog of what's on the tape and
> > where it is - otherwise it can be problematic to restore.
> 
> As usual, I don't know but I'll say something anyway :-)
> 
> Do you have any idea what it is that can't keep up with the tape?  If 
> the bottleneck is disk heads or cpu, writing several streams at once 
> doesn't seem to improve things--unless you're writing from several disks.
> 
> Could you have tar write into a big fifo?  Or is that the dd bit you 
> mentioned?
> 
> Maybe amanda has some optimization--it's supposed to be pretty fancy.




<DISCLAIMER>  I'm not a kernel driver hacker.  </DISCLAIMER>


That said, this problem doesn't sound like something that will get
solved from user-space.  It sounds like a kernel driver performance
issue.  Looking in the kernel docs (/usr/src/linux-2.4/Documentation/)
you'll find:

   st=             [HW,SCSI] SCSI tape parameters (buffers, etc.).

in the "kernel-parameters.txt" file.  Ah, good!  This means that you can
send boot-time and module-load-time parameters for the scsi tape (st)
driver.

Digging into the source files for the scsi tape driver:

  /usr/src/linux-2.4.18/drivers/scsi/st.h
  /usr/src/linux-2.4.18/drivers/scsi/st.c
  /usr/src/linux-2.4.18/drivers/scsi/st_options.h

you'll find four load-time options as described in "st.c":

  MODULE_PARM_DESC(buffer_kbs, "Default driver buffer size 
    (KB; 32)");
  MODULE_PARM_DESC(write_threshold_kbs, "Asynchronous write 
    threshold (KB; 30)");
  MODULE_PARM_DESC(max_buffers, "Maximum number of buffer 
    allocated at initialisation (4)");
  MODULE_PARM_DESC(max_sg_segs, "Maximum number of scatter/
    gather segments to use (32)");

and just less than twenty compile-time params in "st_options.h".

Now, I *don't* have a scsi tape drive handy at the moment to test this,
but I'll bet that:

  1) the default options are *SLOW* since they have most 
     likely been set to maximize compatibility at the cost
     of performance

  2) increasing the buffer sizes (per Jim Ockers) may help

  3) tweaking some of the compile-time options (such as 
     ST_FAST_MTEOM) may help

So Jim, do you have a scsi tape handy to try these params?

Ed

ps - The syntax is:

  insmod /lib/modules/2.4.18-5/kernel/drivers/scsi/st.o \
    buffer_kbs=64 write_threshold_kbs=50


-- 
Edward H. Hill III, PhD 
Post-Doctoral Researcher   |  Email:  ed at eh3.com,  ehill at mines.edu
Division of ESE            |  URLs:   http://www.eh3.com
Colorado School of Mines   |    http://cesep.mines.edu/people/edhill.php
Golden, CO  80401          |  Phone:  303-273-3483    Fax: 303-273-3311