[CLUE-Tech] Re: [CO-SAGE] how to handle open files - upload server

[Tres Hofmeister]

	The approach we generally take here is to put the onus of
offering only complete files on the application which creates or
receives them.  We do this by having the application receive or create
the file in a temporary directory, then link(2) (usually using ln(1))
to a directory *in the same filesystem*, which the clients can then
read whenever.  The application then unlinks the temporary file.
This way, clients only ever see complete files.  An added benefit is
that the clients can check for new files by polling the directory
modification time, which is an inexpensive operation and so can be
done frequently and by many clients.

	In situations where we are stuck with something like sftp, we
have sometimes used a hack where the sending system copies the files to
a temporary directory, then remotely links and unlinks as above using
a separate ssh command.  Alternately, if there are enough clients,
we hack the above behavior by writing a small daemon which makes a
best-effort attempt to move files from the temporary directory based
on a priori knowledge of what the files are supposed to look like,
or we upload a status file with a list of completed files.  The method
above is much tidier, of course... ;)

-- 
Tres Hofmeister <tres at rap.ucar.edu>      http://www.rap.ucar.edu/~tres/
Research Applications Program, National Center for Atmospheric Research