[CLUE-Tech] RE: [CLUE-Talk] swapping tapes in a jukebox with GNU tar

[Ed Hill]

On Wed, 2002-02-13 at 19:04, Jim Ockers wrote: 
> 
> http://dmst.aueb.gr/dds/pubs/trade/1999-login-dd/html/dd.html
> 


Hi Jim, 

Thanks for the note and the URL.  I skimmed the double-buffering article
and it looked like a neat idea.


> My question is: I think that if tar is writing to standard output you can't
> use the -M option because tar does not own the write-data-to-tape process,
> so it would have no way of releasing the tape drive to change tapes and
> re-attaching itself to the tape drive so that we could continue writing data
> to the next tape.
> 
> Do you agree?  Can you think of some way that I could use the
> double-buffering method to speed up the write-to-tape but still use my tape
> library with changer?


Ok, there seems to be two questions: 

  1) How do we get the maximal space usage out of each tape? 

     Since GNU tar cannot know what the actual compression ratio 
     is for the hardware compression algorithm and since no 
     compression algorithm will work on every input stream, I'd 
     consider setting the maximum size to the actual media 
     capacity.  If you can convince yourself that the data stream
     will *always* result in a min of x% compression, then you 
     could rely on that.  But I'd tend to be conservative.

     The best solution to this would be an extension to GNU tar 
     that allows it to communicate with the underlying hardware 
     so that it can know how full each tape actually *is* when 
     using the hardware compression.  We have the tar source, so 
     it could be done, AFAIK.


  2) How to use the double-buffering trick in conjunction with 
     GNU tar tape-rotation? 

     Ok, this is possible since GNU tar allows for the execution 
     of arbitrary commands at the time of the tape swap.  Below 
     is a quickly-written PERL demo showing one way to implement 
     it.

If you or your company wants, I could put together a production-quality
C or PERL routine that does this with sanity checking on the inputs,
better error reporting, efficient handling of the data stream, etc.

And I do charge a very reasonable rate for consulting...  ;-)

Ed


#=== BEGIN ===

#!/usr/local/bin/perl
#
# Copyright (C) 2002 Edward H. Hill III <ed at eh3.com>
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
# General Public License for more details.

use vars qw($pipe_in $pipe_out $num);

$SIG{HUP} = 'handler';

sub handler {
    local($sig) = @_;
    print "Caught a SIG$sig\n";
    close($pipe_out);
    $pipe_out++;
    $num++;
    open $pipe_out, "| dd obs=512 | dd obs=512 of=./out_file_$num"
	or die "can't open ./out_file_$num";
}

sub writer {
    $pipe_out++;
    $num++;
    open $pipe_out, "| dd obs=512 | dd obs=512 of=./out_file_$num"
	or die "can't open ./out_file_$num";
    local $SIG{PIPE} = 'IGNORE';
    while (<$pipe_in>) {
	print $pipe_out $_;
    }
}

open RSCRIPT, "./rotate_script" or die "can't open \"./rotate_script\"";
print RSCRIPT "#!/bin/bash\n mtx_command\n kill -HUP $my_pid\n";
chmod 500, "./rotate_script";

open $pipe_in, "tar -M -L 10000 -F ./rotate_script -cvf - /home |"
	or die "can't open tar command";

writer();

#=== END ===



-- 
Edward H. Hill III, PhD
Post-Doctoral Researcher   |  Email:       ed at eh3.com, ehill at mines.edu
Division of ESE            |  URL:         http://www.eh3.com
Colorado School of Mines   |  Phone:       303-273-3483
Golden, CO  80401          |  Fax:         303-273-3311
Key fingerprint = 5BDE 4DA1 66BE 4F7B BC17  3A0C 932B 7266 1E76 F123
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 232 bytes
Desc: This is a digitally signed message part
Url : http://cluedenver.org/pipermail/clue-tech/attachments/20020215/334aceff/attachment.bin