How to accurately measure the speed of a SSD in a NAS? The night I dropped from 230MB/s to 80MB/s, I figured out whether I was measuring the cache or the disk.

The cause was my 480G SATA solid state. It used to be used as a cache disk on my N100 Feiniu, and later it was used as a cache disk. Is caching worth that article? The conclusion was removed and changed into a download disk specially fed to qBittorrent. Last month I wanted to do an annual inspection on it and check the speed. I plugged it back into the computer and ran DiskSpeedTest. It read 530MB/s and wrote 470MB/s, which was pretty cool. But I clearly remember that after the third or fourth hour of BT downloading, the speed dropped from more than 200 to early 80s, and then it lay there motionless.

On the same disk, two numbers are six times different. Which one is true? That night I turned over the theory of this matter and found that it was not that the disk was bad at all, but that the things measured twice were not the same at all. I think everyone who puts a solid state into a NAS will run into this trap sooner or later, so I’ll write it down.

Let’s talk about the conclusion first: 80% of what you measured is not the disk, but the cache.

The biggest pitfall in the SSD speed test is not that the software is inaccurate, but that the "cache layer" is standing between you and the disk. And it has more than one layer - there is a layer of SLC cache in the disk and a layer of page cache in the system memory. Both layers will dress up your speed test results.

Let’s talk about the layer on the plate first. Nowadays, most SATA solid-state chips are made of TLC chips, but they claim to be fast because they use part of the capacity as SLC mode. The data is poured into this high-speed area first, and the writing speed is 400 to 500 MB/s. It looks very cool. ； When the high-speed area is full, the master controller has to move it to the real TLC area and slowly digest it. This action is called folding. When the folding occurs, you continue to write, and the speed drops directly to 70 or 80 MB/s. This is the "no makeup speed" of this disk. For a 480G disk, the SLC cache can generally provide 50 or 60 GB, so if you copy a movie of more than 10 GB in Windows, you will never see a slowdown, and the cache is enough to hold it in place. The BT download can write more than 100 gigabytes in one night. The eighty or so after dawn is the real level of the disk.

Let’s talk about the memory layer. The hole is deeper because it is given to you by the system for free. Linux will use the free memory as a file cache. The files you have read will remain in the memory. If you read them next time, they will be spit out to you directly from the memory without the disk being involved at all. When measuring speed on a NAS, if the files tested have been touched before, the numbers can be ridiculously high.

Three commands, each testing one thing

After understanding caching, it is easy to choose a speed testing tool. I have three permanent commands on Feiniu, each in charge of one thing. If you look at each other, you will not be deceived.

The first article, hdparm, pure reading test:

hdparm -Tt /dev/sdb

It will spit out two numbers, -T That one is buffer reading. What is actually measured is the memory speed, which is three to four thousand MB/s. It’s fun to watch. ； -t That is the reading speed of the device. The normal SATA solid state is between 450 and 520. This command is fast, and the result is output in two seconds. It is suitable for quickly checking whether the disk is still in status after each restart.

Article 2, dd write test, this is the key to debugging the cache:

dd if=/dev/zero of=/vol1/tmp/speedtest bs=1M count=20000 oflag=direct

The point is oflag=direct , bypassing the memory page cache write-through disk. When writing 20G, the first dozen or so gigabytes of SATA solid state is fast, and then it drops off. The drop off is the real writing speed. There is a pit here that I stepped on the first time: Do not put the test file in the partition that is being used by the system. I tried to save trouble by writing it to the root directory. Feiniu's system partition suddenly became tight and an alarm popped up. Later, I honestly built a temporary shared directory specifically for this purpose. Another point, remember to delete the files after the test. 20G of zero files are thrown on the download disk, eating up the quota in vain.

The third item is not a speed test, it is fio, looking at the performance of small files.:

fio --name=randrw --rw=randrw --bs=4k --size=2G --numjobs=1 --runtime=60 --directory=/vol1/tmp

A mechanical disk's 4K random reading and writing can support one or two MB/s, while a solid state's can be a few hundred, a gap of several hundred times. This is not usually used, but if you put the Docker container and database on a solid state, The article about containers eating up memory For the combination of Immich and postgres, the overwhelming thing is that small IO is not large files, and then this number will make sense. It's just a disk that plays movies and photos, so don't worry about it.

Why did my download disk drop from 230 to 80 overnight?

Now that I know how to use the command, I'll talk about my disk later. Use dd to bypass the cache and test it. After the cache is full, it stabilizes at around 82MB/s, which is on par with BT's speed in the middle of the night. Things are clear.: The disk is not broken. The 470 in Windows is due to the SLC cache. The cache cannot hold the overnight capacity, so the secret is revealed.

Can this disk still be used? Yes, it actually does a pretty good job of downloading the disk. The upper limit of BT download speed is generally tens of megabytes, and the 82MB/s after slowdown is more than enough. ； The fatal flaw of download disks is never the speed, but the amount of writing. The amount of writing is 200G in one night, and more than 70 TB in a year. The mechanical disk is so hard to wear, but the solid state can handle it - this amount of writing is not enough for a disk with a nominal capacity of 300 to 600TBW in five years, and the account is The amount of writing If you are too careful, it is never the download party that you should worry about.

But there is one thing that is much more important than speed. It is the unique account of solid state on the download disk.: The disk doesn't know what was deleted. Deleting a file on a mechanical disk means overwriting it, but not on a solid state disk. The main control needs to rely on the TRIM command to know which blocks are invalid and can be recycled. The system does not send TRIM packets, and the main control just moves a bunch of garbage blocks around as valid data. The more it is used, the slower it becomes, and the write amplification becomes more and more ugly. I ignored this matter for the first half of the year, and for a while the response from the download disk was obviously sluggish. After checking, I found that the account at this level was not settled.

How did the TRIM on the download disk land on Feiniu?

The good news is that you basically don’t have to worry about this, there is only one step to confirm. Debian-based systems such as Feiniu all come with systemd’s fstrim.timer, which runs once a week on the mount point by default. fstrim -av , notify all mounted file systems that support TRIM. SSH up and type:

systemctl status fstrim.timer

When you see active (waiting), you are all alive. In the next line, you can also see the time of the last run. That's when I checked here. I found that I manually mounted this disk, and discard was not enabled in the mount parameters (it is not recommended to enable continuous TRIM, which sends a package every time it is deleted, which interferes with front-end writing. A timer plan that runs in batches once a week is more stable), and the fstrim timer only takes care of the mount points registered in fstab. My manual disk was missed. After writing it into the mount configuration of storage management, the speed returned in the second week.

There is a stupid but effective way to verify whether TRIM is effective.: After deleting a bunch of files, wait for the timer to run out, and then test the writing speed again. The garbage block is recycled, and the "high-speed segment" before the write speed drops will become significantly longer. If you don’t want to wait a week, you can manually reissue SSH. fstrim -v /vol1/downloads , -v will print out the amount of recycling this time, and the number of tens of gigabytes looks very decompressed.

Let me know by the way: Some pitfalls are not configuration issues, but hardware simply does not cooperate. When the solid state is plugged into a USB hard drive box or some SATA expansion cards, the TRIM command cannot be transmitted. fstrim It will directly report "the disk does not support discard". I am here The article about second-hand disk cleaning When I wiped the second solid state, I encountered the pitfall of the USB bridge chip not transmitting transparently, and I felt lonely. TRIM disks in the NAS must be directly connected to the motherboard's SATA port, or they must be disk slots from the NAS manufacturer. This is a waste of money.

In addition to the speed test, I also look at three things during the solid-state annual inspection:

To put it bluntly, speed testing is "performance-level data." What annual inspections should really focus on is the foundation. If the speed drops, it will be slow at best. The following three problems will cost you money.

How to read these numbers, which RAW values ​​should be panicked and which ones should not? SMART interpretation of that article There is an item-by-item human version. The judgment standards for solid-state and mechanical are different. Don’t scare yourself with the mechanical disc. If you can't remember the location, Feiniu has laid out the key items on the hard drive details page of storage management, just click on it and take a look.

wrap up: Don’t be fooled into buying the wrong one by the beautiful speed test chart

After I sorted this out, the rules I set for myself were very simple.: When choosing a solid-state NAS, you can have fun looking at the nominal speed, and first think about what it is used for. Make a cache disk, watch 4K random and TBW, anyway After three months of tossing, the part of my house was finally demolished. ； When making a download disk, smoothness of TBW and TRIM is ten times more important than speed. ； When installing the system and Docker, priority should be given to capacity and stability. Dropping the disk once will cause more harm than slowing it down ten times. Two speed test tools, dd and hdparm, are enough. Next time if anyone sends me a screenshot of more than 400 MB/s, I will ask first.: How big a file was written?

The 480G disk is still in service at the download location, the annual inspection numbers are all normal, and the fstrim timer clocks in at 2 a.m. every Wednesday. After I figured out the speed issue, I no longer had to worry about it - the cache gives me face, and 82MB/s is the lee. As long as the lee is enough for downloading.