[clue-tech] CLUE Talk Mailing list mbox file too big to rsynch

[Jed S. Baer]

On Thu, 05 Feb 2009 18:36:32 -0700
David L. Anselmi wrote:

> Jed S. Baer wrote:
> > Well, part of me wishes I knew enough about Mailman internals to
> > answer that question. The other just wants an OTS remote backup that
> > I can install, point at the directories, and have it just work,
> > without my having to muck with it too much.
> 
> rsync.  Or maybe rdiff-backup doesn't have the same footprint.

Uh, rsync? :-)

> > I'm also thinking of just setting the dovecot service to shutdown
> > overnight. Might free up enough memory for the rsync to work again.
> > That might be as good a solution as any. Until I know more about how
> > Mailman uses its files (who knows when that might be), that might
> > have to be good enough, if it works.
> 
> Well, the rsync FAQ says 800k files take about 80MB of memory.  We're 
> not anywhere near 800k files, are we?  So then maybe actually running 
> out of memory isn't really the problem.

Well, I'm not sure what else to make of the rsync program reporting that
it can't allocate memory. Yeah, it could be an invalid error message, I
guess.

Just for fun, I ran top in one window while running the backup script in
another. Hard to catch such a thing staring at the top display, but I did
see free memory drop to 64k at one point.

> It looks like we have 3 spamd processes that are using most of the 
> memory.  Guess I'm not sure how top's numbers translate to total
> memory. Maybe fewer httpd or smtp processes would help.

Well, I'm no Apache tuner. My main reason for thinking about just
shutting down dovecot overnight is it's a know zero impact thing that's
easy to do. (Other than being, at most, a minor inconvenience to Dennis
and myself.)

I can certainly tweak the httpd.conf for whatever param it is that sets
the number of pre-allocated child processes. At worst, site performance
would drag. IIRC, there's also a param for max child processes.

I'm less optimistic about being able to reduce the Spamassasin footprint.
Sadly, we get lots of crap e-mail. I can lighten the spamd load by
increasing the Postfix load, and I honestly don't know where the balance
point on that should be.

jed