# Docker host & no space left on device

**URL:** <https://forums.suse.com/t/docker-host-no-space-left-on-device/5006>\
**Category:** Rancher 1.x\
**Created:** [December 23, 2016, 2:25pm UTC](https://forums.suse.com/t/docker-host-no-space-left-on-device/5006 "2016-12-23T14:25:07Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![cabrinoob](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/cabrinoob/32/1343_2.png) [@cabrinoob](https://forums.suse.com/u/cabrinoob)\
**Post date:** [December 23, 2016, 2:25pm UTC](https://forums.suse.com/t/docker-host-no-space-left-on-device/5006/1 "2016-12-23T14:25:07Z")

</div>

Hi,  
I’am using Rancher to manage some EC2 hosts (4 nodes in an auto-scaling group) & to orchestrate containers. Everything works fine.

But, at some point, I have a recurrent problem of disk space, even if I remove unused and untagged images with this command

```
docker images --quiet --filter=dangling=true | xargs --no-run-if-empty docker rmi

```

Like I said, even if I run this command above, my hosts are continuoulsy running out of space :

```
Filesystem Size Used Avail Use% Mounted on
udev 7.9G 12K 7.9G 1% /dev
tmpfs 1.6G 1.4M 1.6G 1% /run
/dev/xvda1 79G 77G 0 100% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
none 5.0M 0 5.0M 0% /run/lock
none 7.9G 7.5M 7.9G 1% /run/shm
none 100M 0 100M 0% /run/user

```

I’am using rancher 1.1.4 and my hosts are running Docker 1.12.5 under Ubuntu 14.04.4. LTS.

Is there something I miss? What are the best practices to configure docker for production hosts in order to avoid this problem?

Thank you for your help.

---

<div class="post-metadata">

**Author:** ![bacheson](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/bacheson/32/2881_2.png) [@bacheson](https://forums.suse.com/u/bacheson)\
**Post date:** [December 23, 2016, 9:39pm UTC](https://forums.suse.com/t/docker-host-no-space-left-on-device/5006/2 "2016-12-23T21:39:45Z")

</div>

Install the janitor from the catalogue…it’s handles all this for you

---

<div class="post-metadata">

**Author:** ![sjiveson](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/sjiveson/32/966_2.png) [@sjiveson](https://forums.suse.com/u/sjiveson)\
**Post date:** [December 30, 2016, 9:16am UTC](https://forums.suse.com/t/docker-host-no-space-left-on-device/5006/3 "2016-12-30T09:16:15Z")

</div>

How much space should be used? What I mean by this is what is the total sum of the size of the images that you need available on a host? What requirements do the running containers have. Are there any stopped containers taking up space etc. etc.

---

<div class="post-metadata">

**Author:** ![ecliptik](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/ecliptik/32/1068_2.png) [@ecliptik](https://forums.suse.com/u/ecliptik)\
**Post date:** [December 30, 2016, 6:33pm UTC](https://forums.suse.com/t/docker-host-no-space-left-on-device/5006/4 "2016-12-30T18:33:54Z")

</div>

@sjiveson not sure if it will entirely fix the issue, but Ubuntu 14.04 uses the `devicemapper` Docker storage driver. We’ve had issues with this on some of our Ubuntu 14.04 Jenkins hosts, not specifically running out of space, but other storage related problems.

The solution it seems for this is to use the `overlay` storage driver instead, which unfortunately won’t work on the default Ubuntu 14.04 kernel. You can install the LTS kernel from Xenial and it will work, but that requires a kernel package upgrade and reboot.

See [https://docs.docker.com/engine/userguide/storagedriver/selectadriver/](https://docs.docker.com/engine/userguide/storagedriver/selectadriver/) for more information.

---

<div class="post-metadata">

**Author:** ![sjiveson](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/sjiveson/32/966_2.png) [@sjiveson](https://forums.suse.com/u/sjiveson)\
**Post date:** [January 1, 2017, 1:07pm UTC](https://forums.suse.com/t/docker-host-no-space-left-on-device/5006/5 "2017-01-01T13:07:45Z")

</div>

Interesting. Funnily enough I had a similar problem on a CentOS host a while back, which I was specifically using to avoid a hard links limit issue with overlay. I completely blew it up. Was also for Jenkins. I took a punt and switched back to RancherOS and overlay and we didn’t hit the hard link issue again, which was a pleasant surprise.

---

<div class="post-metadata">

**Author:** ![cabrinoob](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/cabrinoob/32/1343_2.png) [@cabrinoob](https://forums.suse.com/u/cabrinoob)\
**Post date:** [January 3, 2017, 10:32pm UTC](https://forums.suse.com/t/docker-host-no-space-left-on-device/5006/6 "2017-01-03T22:32:26Z")

</div>

I figured it out : My problem came from an “stdout” missconfigured in a logstash container and the generated log file (docker side) was always growing because there was no limit size on it. I did’nt understand why there’s no default config about that in the docker engine …

---

<div class="post-metadata">

**Author:** ![bizmate](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/bizmate/32/3073_2.png) [@bizmate](https://forums.suse.com/u/bizmate)\
**Post date:** [September 13, 2020, 9:05am UTC](https://forums.suse.com/t/docker-host-no-space-left-on-device/5006/7 "2020-09-13T09:05:27Z")

</div>

I have the same situation to yours. I destroyed the stack with the logstash container but the space has not been reclaimed. Did you do anything else to fix this?

I have also tried to use the janitor stack, it says it removed exited containers but the space has not been reclaimed yet.
