Finding duplicate files sounds like a straightforward space-saving task until you realise that "duplicate" is doing a lot of work in that sentence. Two files with the same name aren't necessarily the same content. Two files with the same content aren't necessarily safe to reduce to one. And some files that look like two copies on disk are actually the same data counted twice, where deleting either one changes nothing about your free space at all. Getting duplicate cleanup right means understanding which of these situations you're actually looking at before you delete anything.
What "duplicate" actually means, precisely
There are three distinct things people call duplicates, and they need entirely different handling:
True duplicates — two separate files, in separate locations, that happen to contain identical bytes. These are genuine duplicates: deleting one frees real space and loses nothing, because the other copy is complete and independent.
Hard-linked files — two directory entries that point to the same underlying data on disk (the same inode). These look like two separate files in Finder, sometimes even with different names, but they are the same data. Deleting one does not free any space, because the other link still references the same bytes — the space is only freed once every link to that data is removed.
Same-name, different-content files — files that share a name but aren't duplicates at all, like report.pdf from two different projects that happen to share a filename. These are not candidates for duplicate removal; they're just a naming coincidence.
Why filename matching isn't reliable
The simplest duplicate detection compares file names, which catches almost nothing useful in practice. A downloaded file often gets renamed by the browser (invoice.pdf becomes invoice (1).pdf on a second download), while genuinely identical content can arrive under completely different names (an exported photo saved once as IMG_4021.jpg and once as beach-photo.jpg). Reliable duplicate detection has to look at content, not names.
How content hashing actually works
A cryptographic hash function (commonly SHA-256) reads a file's full contents and produces a fixed-length fingerprint. Two files with identical content produce an identical hash; a single byte of difference produces a completely different one. This is the correct basis for duplicate detection because it doesn't care about filename, location, or modification date — only the actual bytes.
From the command line, you can hash a single file to sanity-check a suspected duplicate manually:
shasum -a 256 /path/to/file-one.jpg
shasum -a 256 /path/to/file-two.jpg
If the two hashes match exactly, the files are byte-for-byte identical. This doesn't scale to scanning a whole disk by hand — a proper duplicate finder does this hashing automatically across every file — but it's a useful way to verify a specific pair before trusting any tool's conclusion.
Distinguishing true duplicates from hard links before you delete
Because hard-linked files can have identical content and different names, a content-hash duplicate finder will report them as duplicates too — which is technically true about their content but misleading about the space you'd recover by deleting one. Before deleting a suspected duplicate, check whether it shares an inode with the other copy:
ls -i /path/to/file-one.jpg /path/to/file-two.jpg
If both files report the same inode number, they're hard-linked: deleting one frees no space, because the data is still referenced by the other link. If the inode numbers differ, they're genuinely separate copies of the same content, and deleting one frees real, measurable space.
Why deleting the wrong copy can break references
Some applications rely on a specific file existing at a specific path, even when a byte-identical copy exists elsewhere. A project's build tooling might reference ./assets/logo.png by that exact relative path; a Photos library might reference an original file location internally even after you've exported a copy elsewhere. Deleting the "duplicate" you didn't check first can silently break the application depending on it, while the actual duplicate you meant to remove sits untouched.
The safe order of operations is: identify the duplicate pair, confirm which one is referenced by anything (a project, a library, a shortcut), and only then delete the copy that nothing depends on.
Where duplicates tend to accumulate
A few categories account for most real duplicate clutter:
- Photos exported or shared multiple times, especially from Messages, AirDrop, and manual exports from a Photos library
- Downloaded documents re-downloaded because the original location was forgotten
- Project assets copied between folders during refactors rather than referenced from one place
- Backup exports made "just in case" before an edit, then never removed afterward
A sensible approach, in order
- Run a content-hash-based scan across the folders most likely to have duplicates (Downloads, Photos exports, Documents) rather than the whole disk at once.
- For each match, check inode numbers if the files are large enough to matter, to rule out hard links.
- Check whether either copy is referenced by a project, library, or shortcut before deleting.
- Delete the copy with the less useful location or name (a Downloads copy over a properly filed one), not automatically the older or newer one.
Near-duplicates aren't the same problem
Everything above deals with byte-identical files, which have a clean, verifiable answer: the hashes either match or they don't. A separate and much harder problem is near-duplicates — a photo exported at two different resolutions, a document saved with minor edits under a new name, three takes of the same screenshot a second apart. These don't hash identically, and no hashing approach will group them automatically, because they genuinely aren't the same data.
Resolving near-duplicates is a judgment call rather than a mechanical one: you're deciding which version is worth keeping, not confirming that two files are interchangeable. Treat this as a separate, manual review pass — usually with Photos' own tools for image bursts — rather than expecting a duplicate finder built around content hashing to catch it, since it fundamentally isn't looking for the same thing.
A note on why scope matters for large hashing scans
Hashing every file on a large drive to find duplicates is computationally expensive — reading a file's full contents to fingerprint it costs time proportional to its size, and doing this across a whole disk with hundreds of gigabytes of data takes meaningfully longer than scanning a specific folder. This is the practical reason to scope a duplicate scan to where duplicates are actually likely (Downloads, Photos exports, a specific project's asset folders) rather than running one across an entire home directory by default.
Where VolumeLens comes in
Duplicate detection is one of the areas where guessing is genuinely risky, which is why we built VolumeLens's approach around what's verifiable: real disk usage per file, hard links counted once rather than double-counted, and nothing removed except through an explicit review sheet before Move to Trash. See how VolumeLens handles file-level detail if you're working through a duplicate cleanup and want the underlying numbers to be trustworthy rather than approximate.