Is Btrfs in the NAS getting slower the more I use it? Eighty percent of it is caused by debris. After the treatment, the load of my download seeding machine has been reduced by half.

This is a bit embarrassing to say. My N100 small host has been equipped with Feiniu for almost two years. The disk is a mix of ext4 and Btrfs - I even wrote an article about this selection. How to choose the account . The Btrfs photo gallery and several containers started to go wrong in the winter last year.: No one is using it in the early morning, but the load is always above 2. When I look at iostat, %util is 80% higher, and the disk is clearly not bad and SMART is all green. I took a detour in the log and memory for less than half a month, and almost pressed the button The article about the container being unable to rise The idea is to reinstall Docker entirely. Later, a filefrag went down and the real culprit was revealed.: A database file of more than 200 megabytes has more than 2,700 fragments.

Make it clear first: Fragmentation is not a bug in Btrfs, it is its nature

In a file system like ext4, files are basically modified in place. The data blocks are next to each other. When reading, the magnetic head just scans along. Btrfs doesn't do this. Its signature is copy-on-write (CoW).: If you want to change a block, it does not touch the original block. First find a new place to write a new one, and then point the pointer there. Snapshots, verification, compression and other technologies all rely on this hand - the old version before the change is naturally still there, and the snapshots are just picked up for free.

This is also the price. Change a little bit and copy a piece. The more fragmented the changes are, the more scattered the data blocks on the disk will be. What’s even worse is that copy-on-write inherently cannot fill holes in place.: If the file is changed to 4K, Btrfs may have to rearrange the entire logical block. Downloading and doing this kind of small pen work all day long is equivalent to using a chisel to carve words all over the plate. After a year of carving, a "logically continuous" file is physically scattered. Mechanical disk seeking is a physical action, the head runs back and forth, and the delay increases. ； No matter how diligent the controller is, he cannot withstand this kind of reading.

So this has a clear picture: It's all about photos and videos. You can write them once and don't touch them very much. They're basically useless. Don't mess around with them. ； Disk fragmentation is a chronic disease that needs to be cured sooner or later when seeding and running database containers. In my family, there are two fortunes and two fortunes, which is how the work is divided.

first step: See a doctor first, don’t take medicine right away

SSH to Feiniu (how to open that door, The article about opening SSH There are steps), it doesn’t matter if e2fsprogs is installed or not, filefrag is basically standard. Shoot suspicious files:

filefrag -v /vol1/@containers/immich/db/immich.db

The number of extents on the last line of the output is the number of fragments. Let me give you a rough guideline in my mind.: Files of several hundred megabytes, extents in double digits, healthy ； Three digits, start saving ； Four thousand and up, but it didn't run away. I have 2,700 Immich databases on my disk, more than 800 qBittorrent resume files, and 1,200 Jellyfin caches - all of them are repeat offenders.

If you want to check in batches, you can scan the entire directory and sort it by the number of fragments.:

find /vol1/@containers -type f -size +10M -exec filefrag {} \; | sort -t: -k2 -rn | head -20

By the way, don’t panic too much just by looking at the number of fragments. Btrfs small files have naturally high extents. Whether a small file of several K is broken or not has no impact on performance. Don’t be intimidated by the numbers and fix it blindly. Staring at large files and files that have been written over and over again is serious medical treatment.

There is also a free supporting evidence, iostat. If not pretending yum install sysstat Or just have apt and run. iostat -x 5 Watch refresh in five seconds: If %util is always high, r_The await number is also ugly, there is not much traffic on the shared folder, and there is no abnormality in SMART. The disk is doing unnecessary work. Nine times out of ten, the disk is being driven around by fragments. I was staring at this set of numbers at two o'clock in the morning and became suspicious - no one was using the computer, so why was the disk busy?

Let me tell you more, there is a free control group for this disease.: My ext4 download disk on the same machine is seeded and run in the same way, and %util is about 20% all year round. One of the two plates is busy and the other is idle, but they still do the same job. This comparison is basically a self-confession. If you also have a mixed plate in your hand, it will be easier to take a look at each other than to guess by yourself.

Step 2: The trick that treats the root cause is chattr +C

The root cause is copy-on-write, and the cure is to turn off copy-on-write for this directory. Just one command on Linux:

chattr +C /vol1/@containers/immich/db

After adding the C attribute, newly created files in this directory will no longer be copied on write, and the data will be changed on the spot, and the fragments will be broken from the root. But there are two dead rules that must be engraved in your mind first.

Rule 1: C only handles new files. An old file that has been broken into pieces will not be put back together even if you add C ten thousand times. So the correct order is: First move the data in the directory (mv to another disk or temporary directory), set chattr +C, and then move the data back - the move back is equivalent to rewriting the file, and this time it is "defragmenting by the way." This is how I moved my habitual criminal files. After moving the 2,700 extents, they returned to double digits.

Rule 2: With CoW turned off, snapshot verification compression will be completely ineffective for this directory. Snapshots eat the cost of copy-on-write, and without snapshots CoW is just empty talk. So don’t add C to the entire Btrfs disk without thinking. In a place like the photo library that takes snapshots to save your life, don’t touch any attribute. Only "fragmentable but reproducible" directories such as database, download, and cache are disabled. This is what I am doing. Selective essay The half of the sentence I didn’t say in Pit 2 will be completed this time.

Step 3: Move the container directory and settle the accounts along the way

The most worry-free way in actual operation is to move the entire directory of the container data volume. Stop the container, mv the data directory out, chattr +C, mv back, and start the container, it only takes five minutes. The key to stopping a container is Container upgrade article As written in, don’t confuse the data volume and the configuration directory. If you move the container to the wrong place, it will be an empty shell. If your container is managed by Compose, it only takes one command to stop it. Compose the article There are ready-made copying methods.

Don’t rush to call it a day after moving, observe it for a week or two before drawing any conclusions. My actual test results: During the seeding period in the early morning, the system load dropped from 2.3 to about 1.1, and the disk's %util dropped from 80% to 30%. I dare not say that everyone has this number - load is related to the number of containers and memory size. Container memory account It’s a disease from another dimension, don’t treat it at the same time. But the direction is certain: The broken pen can no longer write CoW, so I don’t have to go back and forth with it in the middle of the night.

There are two types of discs, and there is no need to treat this disease at all.

One type is highly dependent on snapshot/verification. For non-renewable things like photo libraries and family documents, snapshots are the life-saving layer. The snapshot article He had calculated his account and removed the protection for a few fragments, putting the cart before the horse. At most, the fragments will slow down the reading, but losing them is a big deal.

The other type is solid state disk. Solid state does not have the physical action of head seeking, and random reads caused by fragmentation have limited damage to it. My suggestion is not to mess with Btrfs on the cache disk and system disk - if you really want to care about the life of the solid state, what you should look at is the amount of writes and slowdown. The speed test article and TBW article This is a symptomatic physical examination form.

This disease can be explained in one sentence: Mechanical disk, CoW, high-frequency broken pen, all three will be committed. If you have everything, follow the three steps above. If you don’t have everything, don’t scare yourself.

Prevent recurrence: Three things you should think about when creating a new directory

After treating this time, we can prevent the next time. This is how I arranged it myself:

Just point the data volume to the directory with C added on the day you build the container, which is ten times easier than moving it later. No one told me this when I was pretending to be Immich; The article about pretending to be Immich I suffered this loss only a few months after writing it. In addition, don’t confuse dishwashing inspection and debris management: Scrub controls whether the data is correct or not. It controls whether you can read this article quickly or not. Each of the two levels of accounts is responsible for its own maintenance. It is the maintenance that Btrfs should do. Just stagger the runs during the shift schedule.

End of order: First, check the filefrag number to see if it is fragmented. If so, select the database and download directory chattr +C and move it. Don’t move a finger on the photo library, and don’t mess with the solid state disk. Check again every two weeks. If the load drops, it is correct. If it is still stuck after the treatment, it is not the cause of the fragment. Go back and check the memory and network lines again. Don't hang yourself on a tree.