Removing duplicate files (e.g Music_track.mp3, Music_track 1.mp3)

[ddogg777]

For some reason or another I have duplicate mp3 files of my entire music collection on my Freenas system. What would be the easiest way to get rid of all the music_track x.mp3 copies? Should I try to mount it as a scsi extent and use any of the freeware programs to do it?

 

[cyberjock]

Did you copy the duplicates on accident or something? If you did that's not a FreeNAS problem and you'd likely be better off asking that question in IRC(if you think the FreeNAS guys are smart and know the answer) or in a more general forum. If these duplicate files have must appeared out of thin air, then you probably have a serious problem with your FreeNAS implementation.

 

[Mixel]

I am wondering exactly the same thing... Did you get this resolved?

 

[fracai]

I installed "duff" in a jail. It's great.

 

[Dusan]

I installed "duff" in a jail. It's great.

Or you can install fdupes (http://svnweb.freebsd.org/ports/head/sysutils/fdupes/pkg-descr?revision=HEAD). It can also delete duplicates.

 

[fracai]

There's also rdfind and rmlint, among others. One of those I (rmlint?) couldn't get to compile though.

I like duff because all it does is find the files. You can the pipe the output to delete duplicates:

Code:

duff -re0 <path ...> | xargs -0 rm -v

Before I went through trying a bunch of these I was using my own perl script. And then it started deleting files that weren't dupes. I trust other's work more than my own. Plus, the performance went way up.

 

[simon67]

DuplicateFilesDeleter did a great job with me

 

[emk2203]

rmlint (which I use on Linux often) is now officially in the ports of FreeBSD. According to the author's comparisons, it's orders of magnitude faster than fdupes, for example. What were the problems in compiling? If it's in the ports tree now, it should be simple, no?

 

[cyberjock]

This thread is 2 years old. Sorry but i'm gonna lock this thread.

 

[fracai]

I don't recall the specific errors that I saw with rmlint, but they should be resolved now that it's in the port tree. It's also nice in that rmlint supports colorized output, but I still prefer the simplicity and piping of duff.