[clue-tech] Mirroring Debian testing, i386.

[David L. Anselmi]

Warning: this is boring and esoteric.  But if you're in to mirroring 
debian maybe you can tell me an easier way (is Sean R listening?). 
'Course it may be useful if you can't sleep too.

So I've been in the habit of keeping .isos of 5 of the debian testing 
CDs and updating them from the weekly build:

http://cdimage.debian.org/cdimage/weekly-builds/

Jigdo's a good tool for that.

But the weekly builds aren't being done now since the installer is 
broken and the daily builds are installer only.  So I thought I'd break 
down and get the whole repository (just testing, and just i386, since 
that's all I need).

There seem to be 3 tools in the repository: debmirror, apt-mirror, and 
debpartial-mirror (in popcon order).  There's also apt-move that I tried 
first, to see whether it would move packages from the CD images into the 
right place and reduce the amount of downloading.  That was a bust 
because the docs aren't very detailed about what each config setting is 
trying to do, so I kept getting errors.

I copied all the pool/ stuff from the CDs by hand (should be close 
enough) and tried apt-mirror.  It insists on putting all the files under 
the URL you're mirroring, so using it with multiple sources is a pain. 
It also doesn't create the 3 directories it needs.  The install scripts 
make them under /var but if you decide you want the mirror put on your 
big external drive you get errors until you make them yourself.  Ought 
to file a bug on that.

Then (based on popcon) I tried debmirror.  It checks sigs on the Release 
file but doesn't use either the installed keyrings nor apt's and though 
I followed the directions to extract from the keyring to the file it 
uses it didn't find the keys it wanted.  I gave up at that point.

Based on popcon, the bugs open, and the lack of recent update I skipped 
trying debpartial-mirror.  The gforge site for it didn't make it seem 
very exciting either.

So finally I got the official mirror script:

http://www.debian.org/mirror/ftpmirror

That doesn't work because lockfile and lockfile-create (which I already 
had, rather than installing procmail) use hard links, which I don't have 
on my FAT32 external drive.  (I wonder whether rename would serve the 
same purpose as link and unlink for lock files?)  But it turns out that 
failing to create the file isn't as fatal as failing because the file 
exists.  So ignoring the warning gets what you want.

Rather than download from my usual mirrors (Berkeley or USC) I tried MSU 
since it's a push mirror.  That ran for 18-ish hours and only got 9GB of 
mirror (not even half-way).  Seemed to be lots of throttling (0 window 
packets).  Wonder if that's a bandwidth shaper or just load on the server.

So I gave up on that (actually, rsync crashed when a voltage sag 
confused my external drive) and put in the USC mirror and now I've got 
4GB more in less than a few hours.  It doesn't seem to be doing the 
usual ~160KB/s.  I ought to see if rsync -P gives more info.

Because of the crash I don't know that I'll find out how much the CD 
image pre-seeding helped.  Oh well.

Compared to RPM based distros the Debian repository seems pretty 
sophisticated.  So it does what the ftpmasters want pretty well, but 
working with it otherwise kind of sucks.  Once the initial sync is done 
it may not be that bad.  Then the next trick is to see whether it solves 
all my woes net booting the debian-installer.  (I think the answer will 
be yes, though perhaps the testing version of installer is broken more 
often than not.  So then I'll have to figure out how to hack the stable 
installer to install testing (but really, hacking the installer is 
likely to be easier than hacking the repository)).

So there it is, the story of my weekend (sans the painful work on 
proprietary software and the runaway horse part, but those are OT). 
Thanks for listening.

Dave