[CLUE-Tech] Re: [lug] how to handle open files - upload server

[marcus hall]

On Tue, Jan 13, 2004 at 08:16:48PM +0100, rm at fabula.de wrote:
>  For the first i'd suggest to first get the directory entry for the
>  file (via opendir etc.) from there  get the inode number and access
>  the inode structure by that. The inode structure should have a 'link
>  count' field (too far away from my refences :) which tells you how
>  many filehandles/directory references point to the file.

Apart from having to understand every file system that could be mounted,
the only information on the disk is how many directory references there
are to the file.  The in-core inode has the reference count from file
handles and that is not easily accessible.

>  A _big_ caveat: all these techniques use non-atomic operations and
>  hence are vulnerable to race conditions (well, i'm not shure about
>  exclusive locking allone, i'd have to look it up).

Then, "Alek O. Komarnitsky (N-CSC)" <alek at ast.lmco.com> also wrote:
>I'm fairly certain that exclusive locking is "atomic" ... somewhat
>defeats the purpose otherwise. And yes, to echo Ralf's comments,
>you better make sure that EVERY application is using locking;
>the analogy from the Perl cookbook is a stoplight - only works
>if everyone stops when the light is red ... and this might be
>kinda hard if it is coming in via FTP.

You *also* have to avoid deadlock situations.  Suppose that program "1"
holds a write lock on file "A" and is currently running.  Pretty soon,
it is going to need a write lock on file "B".  Now, the backup program "2"
starts up and happens to get a read lock on file "B" and then sleeps
requesting a read lock on file "A".  When program "1" gets to the point
that it wants a write lock on file "B", the two programs will be deadlocked,
since program "1" is blocking "2" by holding the write lock on file "A",
while program "2" is blocking "1" by holding the read lock on file "B".
Neither can proceed, so they both sit and wait forever.

I would suggest that the backup program should attempt to lock all of the
files to backup with a non-blocking request.  If the request fails, it
should release all locks, sleep a short time, and re-try.  This is not
perfect, but without the ability to enforce a locking protocol with
the various random programs that may be running, it's a pretty good
heuristic.  If there are a lot of conflicts, the backup may take some
time to successfully lock all files, but this is probably OK since backing
up is typically a "lower priority" operation.

>BTW, be careful about using the old "touch a file, and if it exists,
>then assume things are locked" ... since that is NOT an atomic
>operation ...

Actually, the idiom is to create a file with zero permission, as in:

	fd = creat(lockfile, 0);

If that succeeds, then you are guaranteed that things are locked.  The
problem is that if it fails with errno == EEXIST, it could be because
somebody else has things locked, or somebody has things locked and crashed
without cleaning up the lockfile.  To clean *that* up, typically one
wrote their process ID to the lockfile when you get the lock.  If it
exists and contains a process ID, you can test if the PID is valid
with kill().  There are still two holes, though, if the lockfile does
not have a PID in it, perhaps the locker just hasn't written it yet,
or perhaps they died before writing it.  Also, if the original locker
died a long time ago and their PID was subsequently re-used by another
process, it may appear to be still valid...  Thus file locking was
created and it was good.  Use it, it's much simplier.

marcus hall