How to Find and Delete Duplicate Photos or Other Files in Windows/Mac

As time goes by, photos, videos, documents, downloads, and other data will continue to accumulate on our computers. Duplicate files are inevitable. They not only take up extra storage space but also make it difficult to manage files. In severe cases, they also affect the computer's operating speed. Therefore, it is very necessary to find and delete duplicate files to maintain the good status of your computer and improve work efficiency. This post will share some practical tips to help you easily manage files on your computer.

Table of contents

About Duplicates

How to Delete Duplicate Files in Windows 12/11/10

1. Use File Explorer
2. Use the Command Line
3. Try DIY Script
4. Use Free Duplicate File Finder

How to Remove Duplicate Files on Mac

1. Use Finder
2. Via Smart Folder
3. Try A Mac Bash Script
4. Use MacClean

Conclusion

FAQs

About Duplicates

Duplicates are files with the same or highly similar content but not necessarily the same size, name, or hash value. Usually, these files have the same extension but may differ for specific types, such as phto.png and photo.jpg. The following situations may generate them:

1. Photos or videos taken become duplicated due to import, export, backup, multiple copies, etc.
2. Multiple copies are saved during the file editing process.
3. Repeat the download footage or documents to different folders.
4. Some programs automatically create backup or cache files to ensure a smooth operation. But they don't clean up the redundant after.

How to Delete Duplicate Files in Windows 12/11/10

Although these duplicate files may seem harmless, they take up hard disk space, causing the computer to run slowly or even become unable to save new data. To keep the computer clean and perform well, removing these items is very important. So, please check out the methods below.

Way 1. Use File Explorer

When there are a few duplicate files in the computer, mainly photos, we can locate and delete them manually via File Explorer, but it will be a tedious process.

1. Right-click the "Start" icon and choose "Settings." Then, go to "Privacy & Security" > "Searching Windows."
2. Scroll down and find "Advanced indexing options" and click it.
3. After clicking the "Modify" button, click "Show all locations" in the window that opens.
4. In the "Change selected locations" area, select all items and click "OK."
5. Open File Explorer and go to the location where you need to find duplicate files.
6. In the search box, enter the following extensions according to your needs to check for duplicates.

   Data Type	File Extension
   Pictures	jpg, jpeg, png, webp, svg, bmp, cr2, crw, nef, tif, tiif, ico, gif, avif, raw, and more.
   Video	mp4, avi, mov, rem flv, 3gp, mpg, rm, mpeg, wmv, and more.
   Documents	Word: docx, doc, docm, dot, dotx, etc PowerPoint: ppt, pps, ppa, ppam, pptx, pptm, pot, potx, potm, ppsx, etc Excel: xls, xlt, xlm, xlsx, xlsm, xltx, xltm, etc Portable Document: pdf Plain Text :txt More file name extensions
   Audio	mp3, aif, cda, mid, mpa, ogg, wav, wpl, and more.
   Other Files	Executable Files: exe, apk, iso, bat, bin, cgi, etc Font Files: fnt, fon, otf, ttf, pcf, etc

7. After completing the search, check for possible duplicates with file name, size, date, etc. If you want to delete duplicate photos, click "View > Extra large icons" in the top menu to view thumbnails.
8. Hold down the Ctrl key and click if you need to remove ones. After that, click the Delete key. Finally, go to the Recycle Bin to delete them permanently.

Way 2. Use the Command Line

It's time-consuming and inflexible to check duplicate files with File Explorer. It's much better to use a command line to make it. Though it seems complicated to use the command line, you can delete duplicate files in Windows 10/11/12 by simply copying the command to the command prompt, and it's totally free.

1. Right-click the Start icon at the bottom of the screen. For Windows 11/12, select "Terminal (Admin)," and for Windows 10, select "Windows PowerShell (Admin)." When the User Account Control dialog box appears, click "Yes".
2. Type "set-location –path #:\" and press Enter.(# represents the drive letter to be scanned for duplicate files, such as "D")
3. Type the following command.

   ls "directory with duplicate files" -recurse | get-file hash | group -property hash | where { $_.count -gt 1 } | % { $_.group } | Out-File -FilePath "save the search result directory and file name."

4. When a new line of commands appears, it means the scan is complete. At this time, please open the files which are displayed in the results to check for duplicate items.
5. After confirming it is a duplicate file， type the "ls" directory to the duplicate files" -recurse | get-file hash | group -property hash | where { $_.count -gt 1 } | % { $_.group | select -skip 1 } | del" to automatically remove duplicate files on Windows 10/11/12.

Way 3. Try DIY Script

Did you find that both File Explorer and the command line can't handle similar files? Taking advantage of the system's native features and file names really can't make it. To remedy this shortcoming, you can try the script written by our technicians. With detailed comments, it will help you clean up the redundant items on your computer.

A. Batch script

Script example description:

• @echo off
• set local to enable delayed expansion
• :: Set the folder path to scan
• set "folderPath=< Locations that need to be scanned for duplicate files >"
• :: Initializing variables
• set "oldFileName="
• set "oldFileSize=0"
• :: View all files in the directory
• for /f "delims=" %%f in ('dir /b /a-d "%folderPath%*"') do (
• set "currentFileName=%%~nf"
• set "currentFileSize=%%~zf"
• :: Checks whether the current file has duplicate content with the previous file (based on the file name and file size)
• if "!currentFileName!"=="!oldFileName!" if "!currentFileSize!"=="!oldFileSize!" (
• echo Deleting duplicate file: "%%f."
• del "%folderPath%\%%f"
• ) else (
• :: Update the previous file information
• set "oldFileName=!currentFileName!"
• set "oldFileSize=!currentFileSize!"
• )
• )
• Echo Scan completed.
• Pause

How to delete duplicate files on PC using the Batch script:

1. Right-click on the desktop, then choose "New > Text Document". For easy identification, please name it correctly, such as "remove duplicates."
2. Paste the above example into this text, and then replace "<location to scan for duplicate files>" with what you need, such as "C:\Users\Name\Pictures." Finally, save and close it.
3. Right-click the text file and select "Properties".
4. In the window that opens, change its suffix "txt" to "bat," and then click "OK." The system will display a prompt box; please click "Yes."
5. Double-click this BAT file to remove duplicate photos and other items.

B. Python Script

This is an improved version of the previous script, which combines image-hashing libraries (ImageHash and Pillow). This is more effective for users who want to delete duplicate and similar images.

Script example description:

• import os
• from PIL import Image
• from image hash import ImageHash, average_hash
• from collections import defaultdict
• # Set the folder path to scan
• folder_path = 'path_to_your_images_folder'
• # Set similarity threshold (between 0 and 1; lower is more stringent)
• similarity_threshold = 0.1
• # Create a dictionary to store the hash value and the corresponding list of image paths
• hash_dict = defaultdict(list)
• # View all images in the folder
• For root, dirs, files in os.walk(folder_path):
• For file_name in files:
• If file_name.lower().endswith(('.png,' '.jpg,' '.jpeg,' '.bmp,' '.gif')):
• file_path = os.path.join(root, file_name)
• Try:
• # Open the image and generate a hash value
• image = Image.open(file_path)
• image_hash = average_hash(image)
• # Add the hash value and image path to the dictionary
• hash_dict[image_hash].append(file_path)
• Except for Exception as e:
• print(f" Unable to process file {file_path}: {e}")
• # Search for similar images
• similar_images = []
• for hash_value, file_list in hash_dict.items():
• if len(file_list) > 1:
• # For each hash value, if there is more than one image, they are similar (or identical)
• # But we also need to compare further the similarities between them (for images that are not the same)
• # Here, we simplify the process and list only the pictures with the same hash value. For a more detailed comparison, other hash methods or algorithms can be used.
• similar_images.append((hash_value, file_list))
• # Print out similar pictures
• for hash_value, file_list in similar_images:
• print(f" Hash value: {hash_value}")
• print("File path of similar images:")
• for file_path in file_list:
• print(file_path)
• print()

Way 4. Use Free Duplicate File Finder

Using tools will make dealing with everything on the computer efficient. This is also true for deleting duplicate photos, videos, documents, etc. iBeesoft Duplicate File Finder is developed for this purpose. With its help, you can complete this process with just a few simple clicks.

Simple steps to delete duplicate and similar files in Windows 12/11/10 with iBeesoft Duplicate File Finder

1. Download and install iBeesoft Duplicate File Finder on your computer.
2. Launch it, then click on the four options at the bottom of the first window to set search parameters for faster and more accurate finding of duplicate files.
3. Click the "+" icon to select the folder or disk to scan.
4. After the scan is completed, click the file type on the left side of the window to view these files.
5. Select the ones you want to delete, then click "Remove". If there are too many duplicates, you can also click "Select All Duplicates" to quickly select them all.

How to Remove Duplicate Files on Mac

Since Mac users cannot expand storage space, cleaning up duplicates and similar items will be more necessary than for Windows users. If you want to gain more free space, here are some solutions.

Way 1. Use Finder

Photos and videos are the most recognizable files. If you want to find duplicates among them manually, using Finder is the most direct way.

1. Open Finder and click "Downloads" or "Documents" on the left column.
2. Type the file type in the search box in the upper right corner, such as ".jpg," ".mp4", or ".docx."
3. After the search results appear, click the sorting icon at the top. Then, select an arrangement in the dialog box to view these files.
4. When you find duplicate items, please delete them one by one and then empty the Trash.

Way 2. Via Smart Folder

As its name suggests, this method is smarter in finding duplicate files. To do this, you can open Finder and click "File" > "New Smart Folder" in the top menu bar. Then, follow the same steps as in Method 1 of this section.

Way 3. Try A Mac Bash Script

The above methods take time and are inefficient, especially when you have thousands of files on your computer. If you want to remove duplicate files on Mac automatically but without software, then check out the script below.

Here, we combine find, md5, sort, unique, and other commands to automatically identify duplicates to save you time.

Code example:

• #!/bin/bash
• # Specify directories to search
• SEARCH_DIR="/path/to/your/directory."
• #Temporary file, store the MD5 and path of the file
• TEMP_FILE="/tmp/md5sums.txt"
• # Find all files, calculate MD5, and store them in a temporary file
• find "$SEARCH_DIR" -type f -exec md5 {} \; | awk '{print $1, $2}' > "$TEMP_FILE"
• # Sort MD5 and find duplicate MD5
• sort "$TEMP_FILE" | uniq -d | while read -r md5; do
• # Find all files with the same MD5
• grep "^$md5 " "$TEMP_FILE" | awk '{print $2}'
• done

Instructions:

1. Copy the above code and replace the SEARCH_DIR variable with the directory you want to search. Then, save it as a .sh file.
2. Open a console and type "chmod +x <the name of the SH file>," then run the script.

Way 4. Use MacClean

For most users, applications are still the first choice when dealing with duplicate files in macOS. MacClean is a complete set of cleaning and maintenance tools that is worth your try. Here are the steps for how to use it.

1. Download and install MacClean.
2. After opening it, click "Cleanup Tools" in the sidebar, then select "Duplicates Finder."
3. Choose a folder to be searched and click "Scan."
4. Select the duplicate files found and click Delete.

Conclusion

We believe you can successfully delete duplicate files on both Windows and Mac through this comprehensive guide. Whether you want to clean up manually, try automated scripts, or use tools. Finally, to avoid the recurrence of redundant items, please improve the way you manage files.

FAQs