ENOSPC with 107 GB free: thin provisioning lies to df

For two days my CI pipeline failed on every image push with no space left on device, against a host where df reported 107 GB free. Both things were true. The filesystem had 107 GB of addressable blocks and the thin pool underneath it had zero blocks left to back them with. df measures a guest’s own accounting, two layers above the storage that does the allocating.

If you run anything on thin-provisioned storage - LVM thin pools, a VM disk on a SAN, a cloud volume with overcommit, a container on someone else’s host - this failure is available to you, and the number you normally trust will not warn you.

The symptom

The push died at blob upload with an HTTP 500 the registry could only describe as UNKNOWN:

failed to send blob (put), digest sha256:f660bd0b...
request failed: unexpected http status code: Internal Server Error [http 500]
{"errors":[{"code":"UNKNOWN","message":"unknown error","detail":{
    "Op":"sync",
    "Path":"/var/lib/registry/.../_uploads/3bcdc111-.../data",
    "Err":28
}}]}

Err: 28 is a raw syscall.Errno. It is ENOSPC. The registry called fsync() on an uploaded chunk and the kernel refused. Docker distribution has no error code for “out of disk”, so it degrades to UNKNOWN and a 500, which is why the message reads like a registry bug instead of a storage one.

That mismatch is the general shape of the problem: the component that reports the failure is rarely the component that has it.

Why every tool said there was space

A write on this host passes through five layers of indirection before a block lands on the NVMe. Each layer reports capacity honestly, about the layer directly below it. Only the bottom one knows the truth.

   guest    ┌─────────────────────────────────────────┐
            │  registry process - fsync() the upload  │
            └───────────────────┬─────────────────────┘
                                v
            ┌─────────────────────────────────────────┐
            │  docker volume                          │
            └───────────────────┬─────────────────────┘
                                v
            ╔═════════════════════════════════════════╗
            ║  guest ext4 - 295G, df says 107G free   ║ <- df reads here,
            ╚═══════════════════╤═════════════════════╝    reports free OK
                                v
   host     ┌─────────────────────────────────────────┐
            │  LV vm-101-disk-0 - 300G virtual        │
            └───────────────────┬─────────────────────┘
                                v
            ╔═════════════════════════════════════════╗
            ║  thin pool - 337.86G, Data% 100         ║ <- allocation
            ╚═══════════════════╤═════════════════════╝    fails here
                                v
            ┌─────────────────────────────────────────┐
            │  nvme0n1                                │
            └─────────────────────────────────────────┘

The gap between those two marked rows is the entire incident. 480 GB of virtual disks were provisioned against a 337.86 GB pool, which is fine right up until the guests stop being sparse.

Same moment, two irreconcilable answers:

The guest’s 107 GB was real as far as ext4 was concerned. Those blocks were simply never backed by anything.

The false trail, and why it was convincing

My first diagnosis was wrong. The registry volume was genuinely bloated: 65 GB across 34 repositories, with hundreds of megabytes of orphaned upload sessions, and a retention script that existed but had never been wired into cron. Every piece of evidence agreed that the registry had filled the disk.

I fixed it. It freed 24 GB, 65 GB down to 41 GB. The pushes still failed.

That was the moment the theory should have died. Freeing a quarter of the disk changed nothing, which means the disk was never the binding constraint. I spent another few hours before I took my own evidence seriously.

Worse, on this stack the cleanup could not have worked even in principle. Deleting files inside a container returns nothing to the thin pool - the blocks stay mapped to the guest, and the guest cannot release them itself. The 24 GB went onto a free list the pool never heard about.

The test that settled it

The breakthrough was noticing that writes did not fail uniformly. A 2 GB write had succeeded minutes before a 300 MB write failed. Running the variants side by side made it obvious:

Incompressible data failing on every path while zeroes sailed through is not a filesystem problem. That is thin-provisioned storage with no blocks left to hand out. Note that two of the three standard “am I out of space” checks lie here. Zeroes are compressed away and fallocate only touches metadata.

On the host, one command ended the investigation:

$ lvs -o lv_name,lv_size,data_percent,lv_attr
  LV                LSize   Data%  Attr
  data              337.86g 100.00 twi-aotzD-
                            ^^^^^^         ^
                            pool full      out-of-data-space
  vm-101-disk-0     300.00g  87.92 Vwi-aotz--

The D in twi-aotzD- is LVM saying plainly that the pool is out of data space. It had been saying so for two days to nobody.

The fix

$ pct fstrim 101
/var/lib/lxc/101/rootfs/: 119.2 GiB (128020983808 bytes) trimmed

119.2 GiB, the accumulated residue of every file ever deleted in that container since it was created, including my 24 GB of registry cleanup. The pool dropped from 100% to 75.4%, the out-of-data-space flag cleared, and pushes worked immediately.

I verified it with the exact operation that had been failing rather than with a glance at a dashboard: a rebuild and a repush of 200 MB of freshly generated random data, so that no blob could already exist server-side and quietly succeed.

What I changed

Two traps in that prune logic worth stealing if you write one. Deleting by digest removes every tag on that digest, so a naive “delete the sha-tagged images” pass takes your staging tag down with it when they point at the same manifest. And garbage collection must run with the registry stopped, because GC racing a live push can delete a blob that was uploaded but not yet linked to a manifest. A colliding push should get a clean connection refused, not a silently corrupt image.

What is still exposed

I am not going to pretend this is closed. Three things remain true:

The pool has no autoextend configured and the volume group has 16 GB free, so it cannot grow far even if I told it to. Nothing alerts on pool utilisation, which means this was detected by a failing deploy pipeline, the most expensive monitor available. And the overcommit arithmetic has not changed: 480 GB provisioned on a 337.86 GB pool. Trimming buys time, it does not change the sum.

What I would tell someone starting

Three things transfer off this stack entirely.

Find out whether anything under you is thin-provisioned, before you need to know. Virtual disks, LVM thin pools, overcommitted hypervisors, most cloud block storage, and every container you run on hardware you don’t own. If yes, df is a report about bookkeeping, not about whether your next write lands.

When a fix frees resources and the symptom does not move, the theory is dead. Not “partially right”, not “needs more of the same”. I lost hours to respecting a diagnosis that had already been falsified by my own evidence.

Keep one cheap test that the abstraction cannot fake. For this class of problem it is a single line, it takes two seconds, and it is the only check in that table that told me the truth:

dd if=/dev/urandom of=/tmp/.spacetest bs=1M count=300 conv=fsync && rm -f /tmp/.spacetest

Every layer between your process and the physical device is a place where a comfortable number can be reported by something that is not responsible for delivering it. Know which layer answers the question you are asking.