How to clean up duplicate files in a NAS? I scanned through 40,000 photos, cleared out 18GB, and ordered me to copy directly

last month I packed up my NAS and casually checked the photo library: 46,000 photos, 186GB. This number is off—my phone album has at most 13,000 photos, and after four years of backup, it shouldn't have tripled. After flipping through the table of contents for half an hour, the truth was striking: the same photo of my daughter gnawing corn was lying in the library, three copies with different file names and identical contents. This article will describe the entire process of cleaning duplicate files this time, including which tool to use, how to type commands, and which pitfalls to avoid. In the end, I cleared 18.9GB.

Where did

duplicate files come from: three sources, I took them all by number

after checking their own library, duplicate files basically come from three sources.

the first is to replace the device and back up all the data again. My photos are automatically backed up to the NAS. I wrote about this process the phone photo backup article. The problem was when switching phones: when the new phone logged into the app, it couldn't recognize the existing cloud part and reopened the album as is. Switching machines twice means two batches of full-scale replicas. That's the biggest one—two-thirds of Curry's repeats come from this.

second is WeChat Pictures. If you save the chat image to your album, the album will be backed up again—but by the time the other person sent me the picture, I might have already saved it once. If you split the same document into two albums, you'll have two copies on your NAS. I get confused by the WeChat directory: the elderly at home still log in to the same account on their phones, and the photos they saved will also be mixed in through the shared album. This part can't be sorted out by visually flipping through the directory; it can only be left to the tools.

The third is repeatedly importing the camera's SD card. Every time I finish shooting and import to the computer, my hand shakes and I create a new folder. Whenever I fail to import everything properly, I have to import it all again. Video files are large, with a single 4K material costing hundreds of MB; repeating two or three times means 1GB is lost. Later, I counted that 217 duplicate videos accounted for 11.2GB, more than the 6,625 duplicate photos (7.7GB) combined. Before clearing photos, keep an eye on the video to make the most of it.

Before

start, think carefully: the same hash code counts as duplicate, and "looking alike" doesn't count

you can't figure this out, you'll easily delete the wrong stuff later.

fdupes such tools determine duplicate based on file content —counting the file as a hash once, and only if the hash is identical does it count as duplicate. So no matter how ridiculous the file name is, it doesn't matter. IMG_20240601.jpg and WeChat images_20240601102000.jpg as long as the byte is the same, they'll be extracted. Conversely, if you crop, compress, or filter photos, even if the image is the same and the hash is completely different, the tool won't touch it. This is actually a form of protection: the files you have been moved are assigned to your memory to judge, and the untouched ones are handed over to the tools.

my principle is simple: just scan first, not delete. Prepare the list as a document and review it thoroughly, then it yourself. By default, command-line tools only list duplicates and don't change any files; deleting or deleting is decided by another parameter. This design is quite safe.

NAS local device running FDUPES: Feiniu gives three commands, Synology runs Docker

start with Feiniu. Feiniu runs on Debian at the core; logging in SSH to install tools is just an apt task:

sudo apt install fdupes

after installation, I ran first, only scan, not deleted, and saved the results as a checklist:

fdupes -r /vol1/1000/photo > /tmp/dup.txt

-r is a recursive sweep directory. After finishing and opening the dup.txt, duplicate files will be grouped one by one. You can decide which group to keep for each group. This time, I scanned the 186GB in 3 hours and 40 minutes. With the N100 plus a mechanical drive, I had 46,000 small files—no rush. Here's a pitfall: turn off hard drive sleep before scanning, otherwise if it sleeps halfway through, it either freezes or ruins all your previously set sleep plans.

check the list and confirm everything is fine, then delete it using interaction mode:

fdupes -r -d /vol1/1000/photo

-d will ask you in groups to keep which one to keep, list the serial numbers, and have you type out the numbers. There are 6,842 files to be clicked one by one, so my approach is to first use the interaction mode to check the first few dozen groups, confirm that the scans are indeed duplicates, and only then go back to batch process them according to the list. The batch method is very simple:dup.txt the paths within the same group are lined up side by side. From what I saw, most of the duplicates are concentrated in the old directories from the two phone upgrades. Batch old directories are processed in batches, leaving the new directories untouched. This is much easier than picking up multiple sets and less likely to be accidentally hit.

Synology, it's a bit more troublesome. There is no ready-made deduplication suite in the DSM kit center, and no apt in SSH either. There are two options: either download a static compiled version of FDUPES and run it on your machine, or use Docker to create a Linux container and attach shared directories to scan. If you're not familiar with Docker, refer to my previous Feiniu Docker Getting Started , where the directory mounting section is exactly the same.

want to find "similar" photos: dupeGuru needs to run on the computer

fdupes has a natural blind spot: continuous shooting. In the same scene, pressing the shutter five times showed different bytes and hashes in five photos, and FDUPES couldn't recognize a single one. This kind of "looking alike but not the same file" depends on dupeGuru.

dupeGuru is installed on computers, available on Windows and Mac. The method is to mount the NAS shared directory on the computer—Windows maps the network drive, Mac directly accesses the SMB — then selects this directory in dupeGuru to scan. It has a dedicated Picture mode, which compares images by content. Continuous shooting, different compression rates, and resolutions can basically be grouped together. Each group will mark reference images for you, and you just check the remaining ones to delete.

price is slowness. Scanning small files over the network is very costly with SMB protocols; I spent most of the night scanning through it. So my suggestion is to divide the work: the exact same repeats are left to FDUPES to clear on the NAS locally, and if you find it messy, burst shoot can be slowly in dupeGuru. Additionally, Feiniu Album and Synology Photos apps also have plagiarism checking portals that can process some identical photos—whether the machine's computing power is sufficient is another matter; my device's foundation was explained in face recognition test —but they basically can't handle burst shooting. I tried using my own library, and the recognition rate is not as good as dupeGuru's Picture mode.

don't want to delete files but still want to save space: hard links are a trick to the

here's an advanced gameplay tip: if you're worried about deleting the wrong files, you can check it out.

jdupes (an optimized branch of fdupes, usually found in Feiniu Source as well) has a -L parameter. Instead of deleting them, it turns duplicate files into hard links: Each group of duplicates only occupies one portion of space, but every path remains. You can see this photo from any directory. For album-type apps, nothing has happened, but the size has truly decreased.

sounds perfect, but the limitation is that hard links can only be made within same file system ; if you have one copy of two disks, you can't link them together; The sub-volumes of BTRFS and scenes involving snapshots each have their own unique awkwardness. I only tried a small part on the download drive—the origin of that drive written the SSD cache was retired and used as a download drive. The game images and installation packages inside were heavily duplicated, saving 4GB after linking. I didn't dare touch the album's main library at all. If you're not confident, don't try remedies on production data—that's the sentence for you.

three types of people are assigned accordingly, with my complete order attached

choose the route according to your situation:

my complete order for your reference: first, turn off the hard drive for hibernation, then run the FDUPES to complete the list; In the checklist, videos and photos are counted separately, starting with video processing (11GB is more cost-effective than the photo side); The photo group randomly checked about twenty groups to confirm no misjudgments before batch clearing; Finally, it cured the problem of repeated backups when switching devices—when logging into the backup app again, I watched the first sync and stopped manually making up errors once I found a full retweet. I kept that dup.txt and didn't delete it. I plan to run the FDUPES again in half a year to compare and see if the duplicates are quietly accumulating again. After freeing up 18.9GB, the pressure on my 4TB drive has become much less. But honestly, the usable space of a NAS is like a wardrobe—after cleaning, it's just a temporary comfort, but next time I buy , I'll have to settle the score sooner or later.