A further discovery is that I have found over 4Gb of old docker-builder* files and directories under C:\Windows\Temp.

These will be related to building the images, but I would hope these were cleaned up as well.

I am seeing similar issues too. It seems some exited containers are skipped by docker system prune or docker ps -a.

Here is how I verified the issue.

1. stop all running containers
2. make sure docker ps -a returns none
3. run docker system prune and “y”
4. go to C:\ProgramData\docker\windowsfilter

On my test image (which I’ve been using only for several weeks) the directory has 11 left over folders. One of them contains ...\Files\inetpub, so this seems to be left over from an exited IIS container.

Let me know if you need any other information.

Docker version 17.03.1-ee-3, build 3fcee33

ping @johnstep @jstarks PTAL

Hello there,

We also see similar behavior.

I found that "windowsfilter" directory contains a lot of garbage on our production server.

I made a copy of this server, stopped and removed all containers, performed "docker system prune -a". And after that, this directory still contains 4204 sub-directories with size more then 800Gb. See following output

PS C:\Users\Administrator> docker images -a 
REPOSITORY TAG IMAGE ID CREATED SIZE 
PS C:\Users\Administrator> docker ps -a 
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 
PS C:\Users\Administrator> docker images 
REPOSITORY TAG IMAGE ID CREATED SIZE 
PS C:\Users\Administrator> docker info 
Containers: 0 
Running: 0 
Paused: 0 
Stopped: 0 
Images: 0 
Server Version: 17.03.1-ee-3 
Storage Driver: windowsfilter 
Windows: 
Logging Driver: json-file 
Plugins: 
Volume: local 
Network: l2bridge l2tunnel nat null overlay transparent 
Swarm: inactive 
Default Isolation: process 
Kernel Version: 10.0 14393 (14393.693.amd64fre.rs1_release.161220-1747) 
Operating System: Windows Server 2016 Datacenter 
OSType: windows 
Architecture: x86_64 
CPUs: 64 
Total Memory: 1.906 TiB 
Name: WIN-B7...skiped 
ID: 62EZ:...skipped
Docker Root Dir: E:\docker_storage_1_13 
Debug Mode (client): false 
Debug Mode (server): true 
File Descriptors: -1 
Goroutines: 23 
System Time: 2017-06-04T17:51:45.8250234Z 
EventsListeners: 0 
Username: aureadockerservice 
Registry: https://index.docker.io/v1/ 
Experimental: false 
Insecure Registries: 
127.0.0.0/8 
Live Restore Enabled: false 
PS C:\Users\Administrator> (dir E:\docker_storage_1_13\windowsfilter\).count 
4204 
PS C:\Users\Administrator> 

I assumed that it can be result of performing docker rm -f ... command as sometimes just docker rm fails with Access denied error and our developers have to use force flag to delete container.

But looks like it is common issue, as Eiichi and Alex were able to reproduce it.

ping @PatrickLang

@darrenstahlmsft, is this a known issue?

I've taken a look at this previously, but have not found the root cause. The container and image configs (c:\programdata\docker\containers and c:\programdata\docker\image) are still on disk even though they are not visible from the API, which is why prune thinks there is nothing available to clean up.

I'm not able to find a reliable way to cause the leaks, but it seems that some failure path doesn't remove the configs from disk correctly (and they are never reloaded into the daemon, even after a restart).

docker rm -f would cause this (and would be expected I think) as the -f flag removes tracking from Docker regardless of if it can clean up the backing image.

This is on my backlog of things to look at when I have time.

@darrenstahlmsft would this PR (part of 17.06) help with this #31012 ? (basically, if deleting fails, do not remove the container, so that it is still visible in docker ps -a)

While prevention is important, some guidance on manual clean up would be helpful too. We have customers running Windows containers in production already.

For example, is there any way to take a look at the folder and determine if it's safe to delete? I could write a KB article as a temp solution.

@ekitagawa this tool is good -> https://github.com/jhowardmsft/docker-ci-zap
But use with caution. Its completely destructive but does clean up all the space as far as I found. Not an ideal fit for production environments though (unless you have some good orchestration going on that will allow you to migrate all containers to a new server whilst you clean up). Perfectly fine for CI servers etc when used correctly.

@darrenstahlmsft have you found out more about this? Also, do you think #31012 might help address this?

@jhowardmsft or @darrenstahlmsft Any idea when you can come back to this?

Once LCOW is done.....

ok. i will ping you again then.

#31012 certainly should help in a lot of cases and make tracking this down easier, but as John said, both of our available cycles are going to be totally consumed by containerd and LCOW work. I'll continue to watch out for this as I work on other things, but don't expect to be able to dedicate the necessary time in the near future. I suspect some ref counting issues in the graphdriver are at fault for some parts of this, which others could feel free to look for in the mean time.

@pennywisdom if it's an option for you, can you try 17.06.1-ee release clients to see if #31012 helps?

They're available here:

• https://download.docker.com/components/engine/windows-server/17.06/docker-17.06.1-ee-1-rc1.zip
• https://download.docker.com/win/static/test/x86_64/docker-17.06.1-ce-rc2.zip

Here's approximately how to update:

$ProgressPreference = "SilentlyContinue"
Invoke-WebRequest -Uri "<url>" -OutFile "docker.zip"

Stop-Service docker
Remove-Item -Force -Recurse $env:ProgramFiles\docker
Expand-Archive -Path "docker.zip" -DestinationPath $env:ProgramFiles -Force
Remove-Item docker.zip

Start-Service docker

Hi @friism just got around to testing docker-17.06.1-ce-rc4.zip and the results are a lot better. There are a couple of bits of feedback::

1. the stats seem a little off, but this may be to do with orphaned artifacts not being applied. e.g. when i ran docker system df before pruning the stats showed that i had 17.09GB reclaimable. After I ran docker system prune -fa then get-volume closer to 35Gb of space had been cleaned up. Not a huge problem by slightly confusing.
2. After running docker system prune -fa i ran docker-zap docker-ci-zap.exe -folder C:\ProgramData\docker which found and recovered ~2Gb more space. However this is deleting more that just the image data so could be something in that. The reason I ran this after is that this was the only reliable way to previously reclaim free space and I wanted to see if it was still recovering significantly more space than the system prune.

Thanks for the update @pennywisdom!

@friism @darrenstahlmsft Hello Michael, Darren,
After installing 17.10 and performing docker system prune -af --volumes inside a clone of production server I still have 3015 sub-directories in windowsfilter directory with about 925 Gb of data.

Can we think about some separate Powershell/Go automation of cleaning this directories? I can test it inside clone test servers before applying it on production.

I just submitted a PR that I think solves this issue. If anyone who sees this regularly could verify that would be great. I've not leaked a single layer locally since applying my fix.

Link to PR; #36728

The issue still exists on 18.03.0-ce on Windows 10.

@vasicvuk could you open a new issue with details, and steps to reproduce/verify the bug?

@vasicvuk actually; make sure you're on 18.03.1-ce, because the fix was included in the 18.03.1 patch release; see docker-archive/docker-ce#508

This issue should be reopened as it was never fixed @thaJeztah and it is a very serious issue

Continues to occur in 19.03.1 and is very easy to reproduce

Yes I've got the issue too. I had 100Gig tied up in build cache and did docker system prune and it now shows 0Gig in build cache, but no hard drive space is freed up. My drive is 'full'. Agree with Cloudmersive, this is v serious issue

I've resolved this issue by manually recompacting the WSL2 ext4.vhdx size:

• Windows Pro
• Windows Home