Docker Overlay2 Cleanup: 5 Ways to Reclaim Disk Space

If you are running Docker containers on a Docker container host, you may have seen issues with disk space. Docker Overlay2 can become a disk space hog if not managed efficiently. This post examines six effective methods for Docker Overlay2 cleanup to reclaim space on your Docker host.

Disk space issues on a Docker host

If you have run a Docker host in production or in your home lab, you may have seen something like the following. As you can see on this host, the root partition has all the space used. Interestingly, you an see as well the overlay filesystem listed.

What is the Overlay file system?

The Overlay File System, often shortened to OverlayFS, is a type of union file system. It helps the Docker engine manage containers and images by optimizing disk space usage.

In simple terms, it allows Docker to store and organize files in a way that keeps disk space usage minimal. It also helps ensure quick access to files when needed.

Here’s how it works:

Filesystem layers structure

OverlayFS operates on the principle of layers. Each action in Docker creates a new layer.

Each layer is stacked on top of each other. This forms a layered structure. When you modify a file, only the layer containing the file is updated, and the other layers are not modified. This layered file system is one of the ways Docker containers are easily moved between hosts and environments.

Disk space usage

With this, Docker makes sure that files shared across many different containers or images are stored just once. It saves a lot of disk space. When a container needs to read a file, it reads between the file layers from top to bottom until the file needed is found.

Similarly, when a file is modified, a new layer is created, ensuring that the original file remains unaltered and can be used by other containers.

Overlay and Overlay2

Docker has two versions of the Overlay file system: Overlay and Overlay2. Overlay2 is an improved version, providing better performance and reliability compared to its predecessor. The /var/lib/docker/overlay2 directory is where Docker stores all these layers on your host system.

Over time, as more containers and images are created and deleted, this directory can grow in size and become huge.

1. Utilizing Docker’s Built-In Commands for Docker Overlay2 Cleanup

Docker arms you with a suite of built-in commands designed to provide a quick overview and cleanup of disk space:

Docker System Overview

Initiate your cleanup crusade by understanding the space occupancy with the docker system df command. This command is your window into the space consumed by images, containers, and local volumes on your Docker host.

docker system df

# for a detailed view 
docker system df -v 

However, one thing you may find, if your host is 100% full, the docker system df command may not be able to run.

If you have the headroom for the command to run, you will see an output that looks like this:

Prune Unwanted Docker Objects

Docker prune is a built-in mechanism to reclaim space. Depending on your Docker version, The docker system prune command filters through your Docker system, removing stopped containers, networks unassociated with any container, and dangling images.

You can include the -a flag to remove all unused images. The -f flag is a force operation.

It is worth restating this WARNING:

WARNING! This will remove:
        - all stopped containers
        - all networks not used by at least one container
        - all dangling images
        - all build cache

docker system prune -a -f

Out of all the built-in tools and scripts I have tried, this yields the most total reclaimed space.

Addressing Dangling and Unused Images

Dangling and unused images are like the old books on your shelf collecting dust. The docker image prune command is your dusting tool, cleaning up these outdated images and reclaiming the lost space.

docker image prune -a

This may or may not yield much in terms of disk space.

Container and Volume Prune

Similarly, docker container prune and docker volume prune are your allies in removing stopped containers and unused volumes, respectively.

#Prune docker containers
docker container prune

#Prune docker volumes
docker volume prune

Results below of docker volume prune.

2. Manual Deletion

When the built-in commands fall short for Docker overlay2 cleanup, or you desire more control, manual deletion is the way to go.

Deleting Unused Images and Unused Containers

Manually remove dangling or unused images and containers with precision using the following commands:

# Remove dangling images 
docker rmi $(docker images -f "dangling=true" -q) 

# Remove exited containers 
docker rm -v $(docker ps -aq -f status=exited)

Checking Disk Usage

A wise Docker user is one who keeps tabs on disk usage. The df -h and du commands are your eyes into the disk space usage, helping identify the large files or directories.

df -h du -sch /var/lib/docker/overlay2

3. Log Management

The container’s logs are vital but can be space-consuming, even if not directly related to Docker overlay2 cleanup. Restrict log files size by modifying the log parameters in /etc/docker/daemon.json and restarting the Docker daemon.

# Edit the daemon.json file 
vi /etc/docker/daemon.json 

# Add the following entries 
{ "log-driver":"json-file", "log-opts": {"max-size":"3m", "max-file":"1"} } 

# Restart the Docker daemon 
systemctl daemon-reload systemctl restart docker

4. Specialized Cleanup in a Kubernetes Context

In a Kubernetes setup, a more nuanced approach is needed. Draining nodes, stopping the kubelet and Docker services, followed by a comprehensive Docker cleanup can reclaim significant disk space.

# Drain the node 
kubectl drain this_node --ignore-daemonsets --delete-local-data 

# Stop kubelet service 
kubelet stop 

# Restart Docker 
service docker restart
 
# Clean up Docker 
docker system prune --all --volumes --force

These methods, when employed judiciously, will help you keep a healthy disk space, ensuring Docker Overlay2 remains an asset, not a liability. Happy Dockering!

5. Completely refresh docker

You can completely refresh docker using the following steps:

To refresh Docker to a clean state and ensure Docker overlay2 cleanup, you can delete the entire directory, including but not limited to the Overlay2 directory.

# You are entering the danger zone - this will delete data
sudo -s
systemctl stop docker
rm -rf /var/lib/docker
systemctl start docker

Wrapping up

Running a Docker host is a great way to easily deploy applications. However, if you have run into the issue where your disk space is getting filled up on your Docker host, generally, the Overlay2 directory is the culprit. You can use the methods described to free up disk space on your Docker host. The docker system prune -a -f command is especially helpful.