PHOTO DUPLICATE CLEANER

FeaturesScreenshotsVideosFrequently Asked QuestionsRoadmap

INTRODUCTION 
Over the last eight years I’ve collection hundreds of thousands of photos and videos, from my very first iPhone, to all my friends sending me theirs of events and travel. Every time I upgraded, or felt like my digital life was in danger, I backed up all my photos and videos. I ended up with an archive of over a million files, all stored on my network drive (a Synology NAS box). The time came when I had to sort it all out, partly because I couldn’t find anything, but also because I needed to replace my NAS and copying the terabytes of data was too long and expensive on storage. I bought several duplicate photo cleaners and I ran into the same problems with all of them:

1. They couldn’t handle the number of files, ran out of memory, and were painfully slow.
2. They crashed when trying to show me all the duplicates they found, all 870,000 of them.
3. Painfully slow. One of them took 3 days to just find all the files, and I’ve got a decent Mac and Network.

So, I decided to write my own with the specific aim to handle this size of archive. To this end I had three main goals:

1. No preview. Previews are pointless with so many duplicates. No one is ever going to go through them, so just skip that bit.
2. Scalability. It had to handle millions of files, from networks to local SSD drive using little memory and be as fast as Apple can make it.
3. Simplicity. No complex UI, but powerful under the hood to be able to work out duplicates and intelligently remove the right one.

And here it is, now at Version 3. I created it to fix a problem and thought “why not put it on the app store and share it”; I got so much feedback that people loved the concept and goals, but could I do this, or that. It feels great to know other people have the same issues, so I’ve invested over a year so far making it fulfil those requests while keeping it as simple as 1,2,3.

This is an on-going piece of work and something I use in my everyday life. But if there’s anything I’ve missed, or something you’d like then just let me know and I’ll try and get it in. Oh, I also have a roadmap section below so you can see what’s coming up :) - Thanks to everyone.

FAQ

How many files can you actually handle?
We’ve tested to 2 million files but there are no real limits with the exception of your hardware, a MacBook Air can process less, and will take longer than a Macbook Pro

Why don’t you have a preview?
Ask yourself “if it finds a thousand duplicates, do I want to go through each one and verify it?” - If the answers yes, then this isn't the app for you. If the answers No (or “I have better things to be doing with my time”) then you’ll appreciate the sophistication that’s gone into building an duplicate detection engine that can work out which files to keep, and which to remove.

What is Merge?
Removing duplicates without a preview is a scary prospect for the newcomer, so the default mode is ‘Merge’ - this is 100% safe, we don’t actually delete anything. The app scans all you photos and videos and then merges them into a new folder, with the duplicates removed. You can do this as many times as you want, adding more archives into the merged folder, no duplicates are ever added, only newly discovered photos and videos. This of course requires additional disk space.

What is Delete?
Removes the actual duplicates from your archives. Of course you can have them moved to a Trash folder just in case, but this is designed to remove those duplicates when disk space is at a premium and you just need the job done. As part of the intelligence, the app will also remove any empty folders where all the files were duplicates, so it tidies up as it goes along.

How do you detect duplicates?
It’s a complex process that has to be balanced between performance, memory, and accuracy. We detect what is known as binary duplicates, that is, two photos that are identical in every way; if you even so much as change the metadata title, or one pixel, then they are not classed as a duplicate (those changes are often referred to as a like). This process uses cryptographic hashes, a way to convert any file down to a simple, much smaller, and manageable size. 

Do you remove duplicate from the Apple Photos App?
No, not yet anyway. Apple don’t make it possible to do this. Some apps claim to be able to, but all they do is move duplicates to another photos folder for you to delete; even worse, if you use “Optimise Mac Storage”, they are not testing the real photos, just a small thumbnail version. This can lead to a lot of duplicates that are not, and should be kept. I am always looking at new features and as soon as Apple make this possible I’ll implement it - I’m also looking at other ways to do this, but it will add a lot of time if each photo needs to be downloaded from iCloud to be checked correctly. 

ROADMAP

As with all roadmaps, I can’t say when or the true order of these items, but it’ll give you an indication of where I’m focusing development, and how you can influence the feature set, these are major features and you’ll see many small updates in-between that make UI changes, add small features and of course, bug fix. All new features have a prime set of goals: They must keep things simple and be useful to everyone.

Locked Folders.
This will allow you to ‘tag’ the folders you drop into the app as ‘locked’, meaning any duplicates found anywhere else, that are in a locked folder as well, the locked folder version is the original. Basically a way of you influencing the removal engine. There is a lot of work for this, mostly in the UI to keep it simple, but also in performance and a way to remember locked folders between use.

Control over the duplicate detection algorithm
There are different algorithms available and I’ll implement different ones as they have some advantages.

Themes
Want dark mode, or your own color scheme?

Mac Photos App
Deduplicate photos and videos in the photos app, and iCloud

Cloud
Facebook, othe iCloud accounts, twitter, etc. Merging these into a central archive

Merge Move
Same as merge mode, but moving the files instead of copying to reduce disk space needed

Rotate, Resize, Image Mode 
Instead of a binary duplicate check, this will allow for some modification of images to be detected as duplicates, e.g just check the image and ignore the metadata.

Zip files
Opening zip files and searching for photos and videos. Probably only useful for merge mode.