[clue-tech] CLUE Talk Mailing list mbox file too big to rsynch

[Jed S. Baer]

On Mon, 2 Feb 2009 16:52:32 -0700
Kevin Fenzi wrote:

> > I'll see what I can find to get more logging going on the server side.
> > Turning up the verbosity on the issuing end hasn't turned up anything.
> 
> It does seem like an issue on the server end... 
> 
> Check for any OOM messages there? Perhaps something is taking up a
> bunch of memory and the rsync doesn't have enough to gather a
> filelist... 

Well, it's just amazing what you can find at the official rsync website:

http://samba.anu.edu.au/rsync/FAQ.html#5

That said, what's there is not particularly helpful right now.

I did make an annoying discovery. When I set this whole thing up, I
pretty much ignored the concept of "modules", since they seemed
extraneous to the task at hand. I'm not sure why the rsync folks decided
to use the term "module" to refer to what is just an alias for a target
directory (and I think it violates the principle of least surprise, at
least a little). Anyways, the max verbosity defaults to 1, and, according
to the fine manual, the only way to override that it inside a "module"
section in the rsyncd.conf file. Why that isn't available as a global
option, I don't know. I might try it anyway.

However, after looking at our hosting package, I'm 99.9% sure that the
rsync is just banging it's head against a hard memory limit, and there
ain't squat I can do about it without doing an ugly renoberation of the
backup script. Okay, it could be made elegant with sufficient work, but
this was supposed to be simple. The solution seems to be to break up the
task into smaller pieces. But that's somewhat complicated by the fact that
Mailman creates a new directory and html index file for each month. Again,
I can script around that, I think, but I was trying not to have to work at
this. :) I can patch around part of it by using exclude patterns, since
the monthly directories prior to 2009 should never change anyway --
probably, I wouldn't have to muck with the exclude pattern more than
every few years.

However, despite the FAQ saying that rsync can run out of memory on large
file lists, but NOT large files, I still see the error on the mbox files,
and there's no way to split those, that I can think of, short of
"retiring" CLUE-Talk and CLUE-Tech, and starting fresh lists, which would
have been a good thing to have done before this error situation. Well,
the mbox files seem to work ok at least some of the time.

Anyways, just thought I'd pass on an update.

jed