[CLUE-Tech] Re: Jigdo [was: CD burning]

[David Anselmi]

Sean LeBlanc wrote:
[...]
> 
> Has anyone tried out jigdo? All this talk about ISOs made me wonder if
> anyone ever found a jigdo file for downloading Knoppix, or anything else
> beyond Debian. I'm using it to download Debian. I'm surprised there isn't
> more effort on projects like these. 
> 

I use jigdo regularly.  It is an important tool for the debian mirror 
admins, but for other mirrors or end users isn't so significant (other 
than indirectly by making mirroring easier).  I've offered to do a KISS 
on it, but that hasn't been scheduled yet.

For those who don't know, jigdo allows you to download packages from a 
debian mirror and assemble them into an iso image.  The overhead is that 
you need a jigdo file (small) to list the packages, and a template 
(~20MB) to describe the assembly.

The big win is for the mirrors who don't need to provide iso images for 
download.  They already maintain 5+GB of packages per architecture and 
maintaining iso images for each would double that (or triple if they 
have DVD images too).

For users of debian stable, jigdo doesn't do much.  You have to download 
all the bytes anyway, and once downloaded they don't change much.  You 
do get a wider selection of download sites and it is fairly robust as 
far as resuming dropped connections and checksumming the image (but that 
takes time).

For users of debian testing, jigdo allows you to update your iso so you 
only have to download what's changed (20-30 packages a week, per CD for 
CDs 1-3).  That's easier than downloading the whole thing, but you still 
don't want to do it on a dial-up.  Also, in order to re-use packages 
that haven't changed they have to be copied from your old CD (or loop 
mounted iso) to the new iso so you wind up with two copies around during 
the download.

I also had a problem with testing once.  The jigdo and template files 
are updated weekly to match what is on the mirrors (there are 
'unofficial' sites that are updated nightly).  I tried to download at 
the end of the week (and the 'unofficial' sites were that old, too) and 
some of the packages in the jigdo weren't on the mirror any more.  So I 
got download errors.  I was still able to burn a CD, but of course 
trying to install the missing packages was useless.  So now I only 
update my testing images at the beginning of the week.

It would be nice if there were a way to use rsync to pull only changed 
bytes into an iso, but doing that is relatively slow.  It seems that 
rsync, besides checksumming each block (slow), also makes a temp file 
and copies blocks into it either from the local copy or the remote one. 
  So it is easier on the network but takes time and extra disk to 
operate.  The slower your link, the better it is, of course.

Dave