docker logs out of control

While researching out of space message, I found logs way over the set limit

sudo -i                                                                                                                
du -sh /var/lib/docker/containers/*/*.log | sort -hr | head -n 5

7.0G	/var/lib/docker/containers/..../....cbfc4409d310e-json.log
5.3G	/var/lib/docker/containers/..../....3f9f37fd55928-json.log
2.8G	/var/lib/docker/containers/..../....0c11bd07bacf1-json.log
1.6G	/var/lib/docker/containers/..../....f9dcfb2966a6c-json.log
1.6G	/var/lib/docker/containers/..../....d7e5ebbcd7e01-json.log

The logs were supposed to be limited by /etc/docker/daemon.json file:

cat /etc/docker/daemon.json
{
    "insecure-registries" : ["localhost:32000"]
}

{
	"exec-opts": ["native.cgroupdriver=systemd"],
	"log-driver": "json-file",
	"log-opts": {
		"max-size": "100m"
	},
	"storage-driver": "overlay2"
}

However; because the json syntax has two {} {} sections,
docker was skipping the second {} with the actual limiter!
Here’s the corrected file:

{
    "insecure-registries": ["localhost:32000"],
    "exec-opts": ["native.cgroupdriver=systemd"],
    "log-driver": "json-file",
    "log-opts": {
        "max-size": "100m",
        "max-file": "3"
    },
    "storage-driver": "overlay2"
}

Use following to truncate log files:

truncate -s 0 /var/lib/docker/containers/*/*.log

In my case recovering 18GB

Some useful commands:

Check for Hidden Large Files

Sometimes a deleted file is still being “held open” by a process (like a massive log file),
so the space isn’t actually freed. Check for deleted but open files:
If a container or process deleted a massive log file but hasn’t closed the file handle, 
df shows the space as “Used,”
but the OS won’t let new files (like your Postgres lock file) be written to that “ghost” space.

sudo lsof +L1 | grep -i docker

If you see large files with (deleted) next to them, you need to restart the Docker service or the specific container to actually free that space.

sudo systemctl restart docker

sudo lsof +L1 | grep -i docker                                                                                           
docker-co  873291            frank    0u   CHR  136,2      0t0     0        5 /dev/pts/2 (deleted)
docker-co  873291            frank    1u   CHR  136,2      0t0     0        5 /dev/pts/2 (deleted)
docker-co  873291            frank    2u   CHR  136,2      0t0     0        5 /dev/pts/2 (deleted)
docker-co 3914942            frank    0u   CHR  136,0      0t0     0        3 /dev/pts/0 (deleted)
docker-co 3914942            frank    1u   CHR  136,0      0t0     0        3 /dev/pts/0 (deleted)
docker-co 3914942            frank    2u   CHR  136,0      0t0     0        3 /dev/pts/0 (deleted)

Check Inode Usage:

df -i

If IUse% for / is 100%, you cannot create any new files, even tiny ones like the PostgreSQL postmaster.pid lock file.

df -i                                                                                                                    
Filesystem       Inodes   IUsed    IFree IUse% Mounted on
udev            4096034     452  4095582    1% /dev
tmpfs           4108973    2010  4106963    1% /run
/dev/sda2      16384000 5290517 11093483   33% /

Verify and clean build cache:

docker system df

docker system df
TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          86        86        55.15GB   955MB (1%)
Containers      149       47        9.23GB    7.688GB (83%)
Local Volumes   160       77        44.36GB   1.898GB (4%)
Build Cache     483       0         13.72GB   13.72GB

See docker prune document

Standard Cache Removal

To remove only “dangling” (unreferenced) build cache, use:

docker builder prune

Complete Cache Removal

To remove all build cache (including cache from existing images), use:

docker builder prune -a

.........
nq6vgyudufhxexl23m0cebydh
yii6c3yqn30on69y0o9u32mms
ntc2abbi11akf25e72d78tunn

Total reclaimed space: 13.72GB

docker system df
TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          86        86        55.15GB   955MB (1%)
Containers      149       47        9.23GB    7.688GB (83%)
Local Volumes   160       77        44.37GB   1.898GB (4%)
Build Cache     55        0         0B        0B

Remove Cache Older Than X Days 

To remove build cache entries created before a certain time (e.g., 7 days ago), run:

docker builder prune --filter "until=168h" -f

Set a Permanent Disk Limit

Instead of manual pruning, you can configure the Docker daemon to automatically keep your build cache under a specific size.

1. Edit your config: Open /etc/docker/daemon.json (Linux) or use Settings > Docker Engine in Docker Desktop.
2. Add the limit: Set defaultKeepStorage to your preferred maximum (e.g., 10GB):

{
  "builder": {
    "gc": {
      "defaultKeepStorage": "10GB",
      "enabled": true
    }
  }
}

Summary: