is it possible? duplicate file search?

I had an external drive with my video and image files on.  It's getting full.

Is there an easy way to search it for duplicate files - but the files might have different names as I changed my naming system.

There are some tools that can help, even if you've renamed them. Since I'm a linux guy, I would use the find command with the -ls flag and then sort by file size. Any two files that have exactly the same size could very well be duplicates.

Like this:

find <directory> -type f -ls | sort -n -k7

This will output a list of everything in <directory> and sort from smallest to largest.

If you're using a Mac, you have all the Unix tools at your disposal. Just type them in a terminal window. If you're on Windows, the find command is different (and pretty useless for stuff like this) but you can either install Cygwin or figure out how to use Get-Children in Powershell (there are helpful posts on stackoverflow.com and reddit describing how to do it).

I forgot to mention,  they are sorted into different folders.

If all the folders are on that external drive, then running the find command at the root of the disk (the very top level) will search everything underneath.

find <external drive label> -type f -ls | sort -n -k7

never used it, but i like the sound of fdupes

from man fdupes "Searches the given path for duplicate files. Such files are found by comparing file sizes and MD5 signatures, followed by a byte-by-byte comparison."

for dealing with numerous subdirectories, invoke the recursive search flag (-r), as in: fdupes -r /topdirectory/

if you don't normally use linux you could likely boot your machine with a live usb to handle some file work

credits:
https://www.makeuseof.com/find-and-remove-duplicate-files-linux-using-fdupes/
https://unix.stackexchange.com/questions/277697/whats-the-quickest-way-to-find-duplicated-files

Alas, no penguins yet.  I'm still recovering from the loss of DOS (I used to do this easily in DOS) and just hanging out with windows until I can recover from the trama of losing DOS and learn a new operating system.

I've found a few tools already in windows (an example: https://helpcenter.trendmicro.com/en-us/article/tmka-20817 )

But it seems that they either require the file be the same name, or they automatically delete what they think is the extra files.

I just want to locate them and then use the human (aka, me) to decide which ones to keep.

I'd definitely give Jonathan's suggestion of a bootable USB stick some serious consideration. That gives you all the cool toys without changing your working system. A Linux Mint live USB is super easy to download and create.

What I'm going to do is to take the drive and just search by extension then search by size (largest first).  Starting with MP4.

4TB is a lot, so this is going to take a while.  I should have done it when the disk still had a quarter empty so it had room to think about the files better.  

Phil Stevens wrote:I'd definitely give Jonathan's suggestion of a bootable USB stick some serious consideration. That gives you all the cool toys without changing your working system. A Linux Mint live USB is super easy to download and create.

One day.

But for now, I lack the courage.

My super-power is to break computers.  It's all I can do to keep Windows from blowing up my PC (only had it happen three times) and something open source tends to have a lot of chaff in the code and tends not to be safe for someone of my special skills.

In other news, my going rate for breaking computers is only $50 an hour.  I test programmes, hardware, and if you are in town, will come to your place of work or education and can usually down a network by walking in the door (but not the apple store because I'm pretty sure I'm still banned from that one, and some of the cell phone shops)

look at that, I just cleared up 3.4Mb with a single file.  On a roll.  

As the asker doesn’t use Linux, this won’t really be helpful to them, but I saw linux and I’m a huge linux geek of 23 years now.
I wrote a script way back in 2005 to do just this and its a bit overkill but hey its got options!

This script uses MD5 hashes so if the files are *exact* matches it’ll identify it and allow you to either move them to a dups dirs or delete them. It’s from my earlier days so it only handles 1 match a time and doesn’t let you choose which of the 2 to remove.
I wrote it to clean up an old photo library so they were not exactly organized anyway, so removing either of the duplicates didn’t matter to me.

While I don’t remember any names, I do recall seeing an image duplicate checker a *long* time ago (I’ve not ran windows since XP days!)

(I’d attach it but apparently even a .txt extension isn’t allowed.)
Dupchecker:
#!/bin/bash # Author Stephen Leaf # Version: 1.1 # Date: 2005-04-17 # Last updated: 2011-07-03 # # This script will recursively md5sum all files within the given argument # and show a list of files which are safe to remove. # finally it will prompt you for deletion. VERBOSE=false CHECK=false MOVE=false DELETE=false FILE="files.md5" LFILE="files.lst" CFILE="files.md5" VERSION="1.1" cleanUp() { # if [[ $CHECK != true ]]; then # rm -f $FILE || exit $? # fi if [[ $VERBOSE == true ]]; then rm -f $LFILE || exit $? fi } parse_cmdline() { # Referenced mozilla-launcher script. while [[ $# -ne 0 ]] ; do if [[ $1 == -* ]] ; then case "${1#-}" in v|-verbose) VERBOSE=true shift 1 ;; d|-delete) DELETE=true shift 1 ;; m|-move) MOVE=true shift 1 ;; c|-check) CHECK=true CFILE=$2 shift 2 ;; f|fileout) LFILE=$2 shift 2 ;; mf|md5out) FILE=$2 shift 2 ;; h|-help) echo "dupcheck by Stephen Leaf version $VERSION" echo "Usage: $0 [Arguments]" echo " --md5out, -mf [file] Sets the output file for the MD5 hashes." echo " --fileout, -f [file] Sets the output file for the file list hashes." echo "                      Only used in verbose mode." echo " --check, -c [file]   Checks md5 of files listed within a MD5 file. (default: files.md5)" echo " --delete, -d         Deletes duplicates automatically" echo " --move, -m           Moves duplicates to ./dups automatically" echo " --verbose, -v        Make the output more verbose" echo " --help, -h           This Help Message" exit $? ;; esac else DIR=$1 shift fi done } hashFiles() { echo -n "Hashing... " if [[ $VERBOSE == true ]]; then find $DIR -type f > $LFILE NUMFILES=`cat $LFILE | wc -l` POS=0 while read line do POS=`echo "$POS + 1"|bc` md5sum "${line}" >> $FILE PERCENT=`echo "${POS}0000 / ${NUMFILES}"|bc|sed -e "s/[0-9]\{1,2\}$/\.\0/"` if [[ $PERCENT < 1 ]]; then PERCENT="0$PERCENT" fi echo -ne "\rHashing... ${PERCENT}%" done < $LFILE echo else find $DIR -type f -exec md5sum {} \; > $FILE echo " ...done" fi } moveFiles() { # Create it if it doesn't exist if [ ! -e "dups/" ]; then echo "Creating ./dups/" mkdir dups fi echo "${DUPS}" | while read count hash file do # Move the file and force an overwrite. echo -n "Moving ${file} ... " mv -f "$file" dups/ && echo "Ok" || echo "Failed" done } deleteFiles() { echo "${DUPS}" | while read count hash file do echo -n "Removing ${file} ... " rm -f "$file" && echo "Ok" || echo "Failed" done } parse_cmdline "$@" if [[ $CHECK == true ]]; then md5sum -c $CFILE else if [ -e $FILE ]; then echo "Removing existing hash file and filelist." cleanUp fi hashFiles DUPS=`sort $FILE | uniq -cdw 33` if [[ $DUPS != "" ]]; then echo "These files are duplicates and can be safely removed" echo "  Count Hash                              File" echo "${DUPS}" echo echo "`echo "${DUPS}" | wc -l ` Files found." if [[ $DELETE == true ]]; then deleteFiles elif [[ $MOVE == true ]]; then moveFiles else echo echo -n "Would you like to delete these files? (y/n): " read del if [[ $del == "y" || $del == "Y" ]]; then deleteFiles else echo -n "Would you like to move them into ./dups? (y/n): " read move if [[ $move == "y" || $move == "Y" ]]; then moveFiles fi fi fi else echo "No duplicates where found." fi echo -n "Cleaning up tmp files..." cleanUp echo " done" fi

Phil Stevens wrote: Since I'm a linux guy, .

Just another "Linux Guy" with words of encouragement to anyone who may read.

I too had Windows Nightmares

The Windows Nightmare will never happen again when you "Wake Up" to Linux.
There's a reason why it runs most of the servers in the world.
The irony is you can customize it to "look" like Windows, but it will always be far superior. With a file-based Os you can customize almost everything exactly how you want and/or need for that particular moment without the Windows Whining Police knockin at your door or the "blue screen of death"  is that still a thing???  lol

The programs; If there isn't a GUI for it , there's a code. If you can code, automation and customization  is endless. There's more documentation out there than you can imagine for any problem just about.

RedHat and Debian are now coming out with the immutable desktops. I'm an OpenSUSE'r running Tumbleweed a rolling release, not a hiccup ever. I actually had to run TW for a short time before I had the choice. It was the only Distro that would run my new AMD 5700g I built on top of when they first came out before it was for individual retail. Before that it was OpenSUSE Leap. Now I'm considering switching to MicroOS, OpenSUSEs' Immutable.
I've tried many different distributions.
I am not a coder. I'm 10+yrs, nightmare free, with close to total control.
It just works.

I encourage everyone to take the "Leap" to Linux. So you can end the Windows Nightmares. =D

If anything get an old PC and load up Linux Mint Cinnamon edition to play with, it's what I started with, looks just like Windows 7 desktop.

r ranson wrote:Alas, no penguins yet.  I'm still recovering from the loss of DOS (I used to do this easily in DOS) and just hanging out with windows until I can recover from the trama of losing DOS and learn a new operating system.

I've found a few tools already in windows (an example: https://helpcenter.trendmicro.com/en-us/article/tmka-20817 )

But it seems that they either require the file be the same name, or they automatically delete what they think is the extra files.

I just want to locate them and then use the human (aka, me) to decide which ones to keep.

For windows I like the free Puran Utilities tool.  Comes with a lot of great cleanup stuff.