What should I do if the Docker log of the NAS consumes 40 gigabytes in half a year? I locked it in three steps and never touched this mine again.

Last month, my mother's backup NAS sent me an alert that the storage pool was 92% occupied. I remotely connected to df -h and saw that the 4T pool only had more than 300 Gs left, but I remembered that only two more Ts were used. I didn't dare to touch the family portrait and the database with a finger. I went down layer by layer and walked around in the big head and found a familiar face - the Docker section took up more than 40 gigabytes. Among them, I wrote more than 8 gigabytes of log files for one qBittorrent container. I treated this disease once with my own machine, and I recorded it at that time Don’t delete randomly when the storage pool is full. In that article, I encountered it again on someone else's machine this time, which shows that it is really not a small probability. This article will dismantle the Docker log from the beginning.: Why does it grow so big, how to clear it to really free up space, and how to lock it so it never grows bigger.

Make it clear first: Why can the log grow to several gigabytes?

Many people think that container logs are the same as ordinary software logs, which can last for hundreds of megabytes with just a few bits of text. The pitfall of Docker is its default behavior: The json-file log driver does not limit the size. The logs are saved as long as the container is running. A file is written from beginning to end. The log volume that refreshes the screen every few seconds on the seed protection client is on the order of 8 G in half a year - this is not an exaggeration, it is a real number that comes out of my df and du.

What's even worse is that this matter is extremely hidden. When you look at the container on Feiniu or Synology's interface, you can only see the words "Running". The log files are under the hash directory of /var/lib/docker/containers/, and not a single line of volume is shown to you on the interface. The storage pool slowly turns red, and your first reaction is to delete old movies and clear the recycle bin. It is not your turn to doubt Docker - after all, on my mother's machine, the log expansion was caused by qB-ittorrent, but the logs of the rsync backup machine are also growing quietly, but it is refreshed slowly, only 2G in two years. Two machines, two speeds, the same conclusion: Without locking, each container is a time bomb with different fuse lengths. .

By the way, Docker eats more than just logs. The old images saved from previous container upgrades, abandoned containers that have been stopped but not deleted, and dangling volumes often add up to more fat than the logs. I'll take care of that part of the account Container upgrade article I’ve done the calculations before, so I won’t repeat them here. This article only focuses on the diary.

first step: Find out which container is eating, and reconcile df and du

The idea of ​​​​positioning is with me. Storage pool is full The overall order of writing that article is the same: Look at the total quantity first, and then look for the bulk one layer at a time. SSH into NAS (how to open SSH, pitfalls of port 22 forwarding, see Enable SSH That article), start with two commands:

df -h First look at how much is left in the pool and have a general ledger in mind. docker system df Let’s look at Docker’s internal account and see how many images, containers, and volumes each occupy on one screen. Note that the container line on this screen counts the writable layer. In most cases, logs are not listed in separate columns. , so when you see that the container only occupies a few hundred megabytes, don’t rush to rule out the suspicion. You have to continue to drill down.:

This article will list the log file size of each container, sorted by size, and it will be clear who is stealing. On my mother’s machine, qBittorrent’s log of 8.4G ranks first, followed by rsync’s 2.1G. The remaining seven containers add up to less than 500M – a typical structure of “one or two repeat offenders + a group of good people”. If you see that the log list is evenly large as a whole, you have to go back and check if there is a root cause of the uniform flushing (for example, a certain container is reporting an error and retrying the loop). Fix the root cause first and then clear the logs, otherwise the logs will grow back the next day after clearing.

Step 2: There is a pitfall when clearing logs. The space is not released after rm deletes it.

If you find a repeat offender, can you rm that log file? This is what I did for the first time. After deleting df -h, I saw that the occupation was not moving at all. I was stunned on the spot. Only later did I understand: When the container is running, the log file handle is held by the process. What you rm remove is the directory entry. The file itself is still referenced and the space is not released. . You have to use truncate to clear it. The file is still there and the handle is still held, but the content is reset to zero.:

On my mother's machine, I truncate the 8.4G log, and df -h immediately returned 8G, and the pool dropped from 92% to 90% - a small number, but this is an immediately visible release, which is much more practical than deleting ten movies. When I was writing this article, I looked through the records and found that the most frequent flush in qBittorrent’s log was the duplicate entries pulled from RSS. This was purely caused by incorrect configuration and writing. If this root problem is solved, the log will grow more slowly.

If you find it troublesome to write files one by one, there is a lazy way to write it in a loop, but I won't give it directly - because the logs of the good people container actually have troubleshooting value, I am Troubleshooting if the container cannot be opened As written in that article, looking through container logs is the first place to locate startup failure. Cleaning up all the records in one go is like burning the medical records, and you won’t be able to look them up next time something happens. My own habit is to only start with projects exceeding 1G, and save the small ones.

Step 3: locked, daemon.json Cap the log

Cleaning it up is just to stop the bleeding, the root cause is to set the upper limit on the size of the log. Docker has a global configuration file daemon.json , it can be configured directly in Feiniu’s Docker setting interface. Use the command line to change /etc/docker/daemon.json and add two lines of log-opts.:

This means that the log file of each container can be up to 100M, with a maximum of 3 copies. It will be automatically rotated when it is full, and the log file will be capped at 300M. qBittorrent's 8G flash method for half a year will always stop at 300M after being locked. I took the two numbers 100m and 3 based on the measurements of my own machine.: The log should be enough for troubleshooting, but not so big that it hurts the disk. If your machine is small and logs are refreshed slowly, you can set max-file to 2 and save 100M.

After the change is completed, restart the Docker service to take effect. There is a pitfall here that most people have fallen into. I must write it boldly.: These two lines only take effect for newly created containers. Old containers will not take advantage of the new rules. . The old container must be deleted and rebuilt before the lock parameters are used - don't panic when you hear "delete container", what is deleted is the container shell, and the data volume and configuration directory you mounted will not be touched at all. This is the original design intention of Docker. I am here Container upgrade In that article, we discussed in detail how to copy and reconstruct parameters. It is more worry-free to install with Compose. The parameters in the configuration file can be easily pulled and built. How does Compose manage the container obediently? See Docker Compose That article.

There is also a thinner keyhole: Set individual caps for individual containers. daemon.json It controls the global default. If individual containers want to overwrite it individually on the command line, add --log-opt max-size=50m when docker run, which is tighter than the global one. I uploaded 50m of qBittorrent alone at home - it is a repeat offender and needs to be tightened.

Closing accounts for my mom's machine

The entire process was run on her backup machine.: Truncate cleared just over 10 GB (qBittorrent's 8.4G plus rsync's 2.1G). I pruned the abandoned containers and dangling images that had been idle for half a year and returned 7 GB. The docker system df was reduced from more than 40 GB to 23 GB. The pool occupancy returned from 92% to 86%, and the alarm was lifted. After the log locking was completed and the old container was rebuilt, I kept an eye on it for another half month, and the volume of the Docker stall did not increase at all. This is the difference before and after locking. It used to be a thunder that grew up at any time, but now it is a dead thunder.

Unexpectedly, there is a joint benefit: The hibernation of the hard disk at night is more stable than before. While the log is being written, the hard disk cannot sleep well. Even if the 8.4G log is written slowly, it is still writing continuously. I am not aware of the fact that secretly writing to the disk disrupts sleep. Troubleshooting hibernation failure I was named in that article. After locking, a single file of 100M is rotated after being written, and the writing density is reduced. The disk of my mother's machine really slept the whole night. Scheduled tasks should not compete with the quiet period after the log is locked. How to schedule automatic updates and snapshots, please refer to Do you want to restart regularly? The period account in that article.

Who should do this set immediately? I will collect the list according to the pitfalls I have stepped on.:

End in one sentence: There are only three steps to debunk the Docker log mine - find out the habitual offenders, truncate to stop the bleeding, daemon.json Lock and rebuild the old container. The new container will be locked from the first day, and you will never have the chance to encounter the case of "the pool is inexplicably full". If you are still looking at the door of Docker, take a look first Getting started with Docker That article clarified the three words image, port, and mount. The logging part is an advanced course. It’s not too late to come back after getting started.