What should I do if the SSD of my NAS disappears every now and then? It took me a month to catch up on the small disk that I used as a cache. The root cause of the problem is probably not on the disk.

The Feiniu NAS at home is equipped with a 256G solid state as a cache. Is solid state cache worth it? The one mentioned in that article. One day I logged in to the backend to check the cache hit rate, but it disappeared from the hard disk list - only four mechanical disks were left neatly arranged, and it was as if the solid state had never existed. My heart sank. My first reaction was that the disk was down. A 256G disk was not worth a lot of money. But without the storage pool, the cache was running naked, so I had to deal with it immediately.

As a result, before I could place an order for a new disk, I restarted the machine and it came back. The health information looks normal, the temperature is normal, and the writing volume is normal. I took it as a metaphysics and didn't care. In the next month, it fell off twice more: One time was in the middle of the night, and the next morning I saw a cache exception reported by the storage pool. ； One time, I was checking my phone next to me and heard no movement in the case. When I logged in, I saw that no one was there. If it goes off, just restart it, but it will go off again after a few days. This kind of "on-again, off-again" problem is more annoying than if it breaks down directly, because you never know when it will not come back completely.

Later, I turned over the system logs and found the current situation in an inconspicuous corner. In this article, I will write down this set of ideas for grabbing the disk.: I will first talk about how to obtain evidence, and then talk about the root causes of the diseases I encountered and the order of investigation based on probability. If the solid-state disk disappears from the list, most likely the disk will die, so don't throw money away in a hurry.

Get evidence first: The system actually shouted at the moment the disk was dropped.

The disappearance of SSDs is not silent. The moment it disconnects from the SATA link, the kernel will record an entry in the log, but this account is not in the pages you often look at, so you have to look it up yourself.

If SSH is enabled on both Feiniu and Synology, you can directly look at the kernel log.: dmesg | grep -i "sata\|link\|reset" . When my disk went offline, the logs looked like this:: First SATA link down (the link is down), followed by ata3.00: disabled (The device on this SATA port is disabled), and then the next series hard resetting link (The system is repeatedly trying to pull the link back up). The timestamp is clear, exactly the same as the moment when I saw the disk disappear in the background. This is the real deal - the link went down, and the system tried to pull it back, so it was removed from the device list.

There is a way to use Synology without opening SSH. Click on the disk in the storage manager. In some versions, you can see the "hard disk log"」 ； If there really isn't, just take a step back and use the time comparison method.: At the moment when the disk was dropped, the background management page still remembered the system event, and matched the event time with the abnormal time in the hard disk list. Although it is not as accurate as dmesg, it at least confirmed that "it was broken at that minute" and not that the disk was broken slowly. My disk was later confirmed to show that the log times of the three disk drops were all between 2 a.m. and 4 a.m. This pattern directly helped me identify the root cause of the problem, which I will discuss in detail below.

By the way, what you want to see in the log is not "bad", but "broken". Even if the SMART health items are all green, the disk will still be lost - the disk loss is a matter of the link layer, while the physical examination report is a matter of the physical layer, which are two different things. I wrote an article on how to read hard drive SMART data Don’t scare yourself if you look at the RAW value , don’t use SMART as a criterion when it comes to losing orders, as it will lead to misjudgment.

Root cause 1: The power-saving mechanism of the disk, the number one habitual offender of solid-state disk failure

After the log pattern came out, I thought about it all night: Why is it early in the morning? It wouldn't fall off even if the whole family was using the NAS hard during the day, but it would fall off in the middle of the night when no one touched it. Then I came up with a word - power saving.

SSDs have their own set of low-power state machines. SATA disks are called DEVSLP and Partial low-power states, and NVMe disks are called APST. After being idle for a period of time, the disk will drop to a low power consumption state and sleep until it is used again. This mechanism works fine on desktop computers, but there is a hidden pitfall on NAS.: The disks of the NAS are idle all year round (the cache disk is especially idle and idle when there are no reading and writing tasks), and the number of times it enters the low-power state is dozens of times that of an ordinary computer. ； And some main control firmware is poorly written, or has a bad temper with the motherboard's SATA controller. It can't be woken up after falling asleep - the link cannot be pulled up, and what the system sees is that the disk is dropped.

Verification costs are ridiculously low: In the storage pool or cache settings where this disk is located, turn off the "hard disk hibernation" related options, or turn off this disk in SSH to save power ( hdparm -B 254 /dev/sdX Pin the APM level to a position that can’t help but save power, use the specific drive letter lsblk Check it out first), keep it awake and run for a week or two. If it doesn’t fall off, the root cause of the disease will be determined. After my disk was turned off to save power, it didn't drop once in a month, and the regular disk drops in the early morning disappeared. Zero cost, just one or two more electricity, which can cost more than ten yuan a year.

This step is cousin to the mechanical disk problem of "failed to wake up from hibernation". I wrote down the troubleshooting of the mechanical disk set of stolen disks in The article about hibernation always failing Here, you can read the two articles side by side: Mechanical disks can’t sleep, and solid-state drives can’t wake up. These are power-saving mechanisms that conflict with 7×24 scenarios like NAS.

Root cause 2: SATA port and cable, a six-dollar pitfall

If the disk disconnection time is irregular and random during the day and night, then put aside the power saving issue first and look at the link itself.

Symptoms of poor mechanical disk contact: I am The disk cannot be recognized when the computer is turned on written in: The SATA cable is not plugged in properly, and even half a millimeter can cause the link to be intermittent, and the log is full of link down. Solid-state disks inherit this problem one by one, and there are two places where they are more likely to be caught. One is the disk position: The solid state of multi-bay machines is mostly installed on the adapter bracket of the 3.5-inch bracket. If the screws of the bracket are not tightened, the contact between the adapter plate and the backplane will shake when the machine shakes. ； The second is the SATA power supply head that is separated from the power supply line. Several disks are strung together on one line. If the connector is oxidized or loose, the link will fall off if the power supply is shaken. If what you see in the log is link power management Related error reports are mixed hard reset , the link layer is more suspicious.

The judgment method is the same as that of the mechanical disk.: Change cables, ports, and swap disks with other disks to see who causes the fault. Following the market is a matter of the market or the adapter. ； Following the position is a matter of mouth or line. SATA cables cost six yuan each. Change the cable first before talking about anything else. This is the ancestral motto of the hard drive circle. The cache disk of my colleague's machine was lost for a quarter, and the problem was solved in ten minutes by changing the line.

Root cause three: The main control firmware is too rough and the incidence of off-brand disks is high.

This disease is the most irritating because it cannot be repaired, only recognized.

The main control firmware of the SSD has to handle a lot of concurrent tasks: Garbage collection, wear leveling, bad block mapping, and maintaining the link state machine. The firmware of big-name disks (Samsung, Chitai, Kioxia, etc.) is much more polished. In order to reduce costs, white-brand disks use public versions of master controllers and rough firmware. The link state machine is particularly prone to glitches.: If corner scenarios such as low-power switching, exception recovery, and error correction and retry are not handled well, the disk will be dropped. My cousin's off-brand disk reported a remaining life of 3%, and this was included in the fault chain - the firmware reported wear and tear based on the worst-case scenario, which is a minor matter, but the link breaking every now and then is a big deal. I'll include that account in this article. At what TBW should you change? Inside.

How to tell if it’s the firmware’s fault?: Switch to the computer, change the cable, change the port, still the same problem ； Even turning it off to save power doesn't work. ； SMART couldn't find anything bad - the disk was not faulty, the behavior was faulty, and the blame basically rested on the firmware. If you encounter this kind of disk, don’t mess around. There is only one way to save your data.: Data can be moved quickly. There is a particularly bad trend when the disk is dropped at the firmware level, that is, whenever I fall asleep and never wake up, it becomes a brick directly, and the main controller can leave as soon as it is told. If the system disk is damaged, will the data be lost? As I learned in that article, mechanical disks at least give a warning, but solid-state disks often don't.

So when choosing solid state for NAS, my advice has never changed.: For places like cache disks and system disks that are "very expensive", you'd rather buy a big-name one for thirty or fifty yuan. For a disk in this position, stability is more important than speed, and more important than capacity. How do you measure the speed so that it is not false? Solid state speed test As mentioned in that article, don’t just focus on running points when selecting games.

Root cause four: Temperature and power supply side accounts

Two things that are unlikely but worth knowing.

temperature: Solid-state disks have better heat resistance than mechanical disks, but the underclocking protection of off-brand disks is poorly written. For M2 disks installed close to the CPU heat sink, or SATA disks without air ducts inserted into a mechanical disk stack, when the temperature exceeds 70 degrees, some firmwares will directly cut off the link to protect themselves, and the log will show that the disk has dropped. I wrote an article about what is the normal hard drive temperature? 52 degrees and 58 degrees are two different things , the solid-state scale is looser, but when the disk is dropped and there is no other reason, touch the disk case and if it is hot, move it to a windy position first.

powered by: Solid-state disks consume little power and generally do not bear the burden of power supply, except for one situation - they are connected to the same SATA power supply as several mechanical disks. When the mechanical disks collectively spin up during a cold boot, the current rushes very fiercely (a 3.5-inch disk consumes more than 20 watts to spin up, which is calculated in the article about not recognizing the disk). The solid-state disks on the same road are subject to voltage collapse and flash, which can cause the link to be lost or the connection to be lost directly. If you have a lot of disks at home and the disks often fall out, check the wiring of the power supply lines and feed the solid state separately.

wrap up: Check in this order, do it first and then your wallet

Collect the whole set of ideas into a table and follow it when the disk is dropped.:

One final reminder: Don’t ignore the collateral damage caused by the cache disk being dropped - it plays a role in the storage pool architecture. When the cache disk is dropped, the pool will report an exception, which is very similar to the reporting method of a mechanical disk being dropped, but the processing is completely different (the storage pool side of the mechanical disk drop is handled in The downgrade article , when encountering a solid-state cache disk, just dismantle the cache and rebuild it. Don't pull out the disk in a panic like RAID). My 256G one has been stable since I turned off the power saving, and the cache hit rate is still increasing. The evidence for the disk drop problem is all in the logs. Look through the logs first, and then touch the wallet.