[clue-tech] CLUE Talk Mailing list mbox file too big to rsynch

[David L. Anselmi]

Angelo Bertolli wrote:
> I wrote a findd (find duplicate) script for this sort of thing.  It's old
> and probably makes excessive use of files, but it works.

If you can stand to have less verbose output you could write that as:

find . -type f -print0 | xargs -0 md5sum | sort | uniq -D -w 32

xargs is frequently as good as a loop.  uniq can show dupes, and can 
limit how much of a line it looks at (the 32 goes with using md5sum, 
cksum would be different).

The difference isn't just in getting rid of temp files.  It's also in 
taking advantage of more features of the commands you used.

Of course in Jed's case he probably prefers diff output (he's looking 
for differences, not dupes).  And the directories to compare are on 
different machines.

Dave