Results 1 to 4 of 4

Thread: "no space left on device" with btrfs setup - hog-hunting

  1. #1

    "no space left on device" with btrfs setup - hog-hunting

    On a test VM with SLES 12 SP1 with default btrfs setup I ran into a
    situation with "no space left on device" on /var/...

    As such a default setup will put together the whole installation on
    one btrfs filesystem ...

    /
    /.snapshots
    /var/tmp
    /var/spool
    /var/opt
    /var/lib/named
    /var/lib/mailman
    /var/crash
    /var/lib/pgsql
    /usr/local
    /tmp
    /srv
    /opt
    /boot/grub2/x86_64-efi
    /boot/grub2/i386-pc
    /var/log

    ... neither df nor "btrfs filesystem df" will help you hunting the
    space hogs, as all share one common pool.

    I was able to ease the pressure by removing snapshots that snapper
    created automatically:
    https://www.suse.com/documentation/s...s_volfull.html

    I would like to ask here if there is a good means to get an overview
    where the btrfs file system usage actually comes.

    In our case, /var/lib/docker with various docker images appears to be
    the culprit, but even there, it seems to be hard to get a realistic
    usage information: "du -ksh /var/lib/docker" is reporting far too much
    (I think because of the btrfs layering employed by docker), and
    "docker images" not enough to justify the usage.

    A think a good idea might be relocating /var/lib/docker to a separate
    partition out of scope of the btrfs default filesystem?

    Thanks for sharing your opinions.

    Regards, Tom

  2. #2

    Re: "no space left on device" with btrfs setup - hog-hunting

    On 07/25/2016 01:44 AM, swadm wrote:
    >
    > On a test VM with SLES 12 SP1 with default btrfs setup I ran into a
    > situation with "no space left on device" on /var/...


    Does this imply you have a separate filesystem setup for /var/... or
    /var/lib/docker at least? Doing so is recommended, whether or not you use
    BtrFS, per the documentation:
    https://www.suse.com/documentation/s...ckerquick.html

    > As such a default setup will put together the whole installation on
    > one btrfs filesystem ...
    >
    > /
    > /.snapshots
    > /var/tmp
    > /var/spool
    > /var/opt
    > /var/lib/named
    > /var/lib/mailman
    > /var/crash
    > /var/lib/pgsql
    > /usr/local
    > /tmp
    > /srv
    > /opt
    > /boot/grub2/x86_64-efi
    > /boot/grub2/i386-pc
    > /var/log
    >
    > ... neither df nor "btrfs filesystem df" will help you hunting the
    > space hogs, as all share one common pool.


    Have you tried the following?

    Code:
    btrfs fi usage /  #note the trailing slash to specify filesystem

    > I was able to ease the pressure by removing snapshots that snapper
    > created automatically:
    > https://www.suse.com/documentation/s...s_volfull.html


    If you have a lot of snapshots for whatever reason, then cleaning those up
    can help, though if you did not partition /var/lib/docker separately then
    those snapshots could be cleaning up data from all over the filesystem,
    which would especially apply as patches or other Yast/zypper-based changes
    are made to the system as snapshots are automatically generated then for
    rollback purposes. Also, the "timeline" snapshots, configurable via Yast
    or /etc/snapper/configs/* files, will generate a few more timeline-like
    snapshots than I would prefer. I typically turn down the month/year
    settings, and sometime even the day-based settings, depending on the
    system's purpose. Initial setup of a system often has a lot of patching,
    and tweaking of system settings with Yast, and all of those snapshots will
    be kept around longer than you expect, so my general practice is to get a
    system working, clear out extra snapshots, then move forward.

    > I would like to ask here if there is a good means to get an overview
    > where the btrfs file system usage actually comes.


    Let me know if the 'usage' (vs. 'fs') portion of 'btrfs filesystem' works,
    as shown above.

    > In our case, /var/lib/docker with various docker images appears to be
    > the culprit, but even there, it seems to be hard to get a realistic
    > usage information: "du -ksh /var/lib/docker" is reporting far too much
    > (I think because of the btrfs layering employed by docker), and
    > "docker images" not enough to justify the usage.


    If 'du' shows a lot of space used, I would not expect that to be just
    BtrFS-related. The last time I checked, 'du' (the command, just to be
    unambiguous) only looked at the current version of the filesystem, meaning
    the current snapshot, but perhaps that has changed.

    > A think a good idea might be relocating /var/lib/docker to a separate
    > partition out of scope of the btrfs default filesystem?


    Yes, and that's the official recommendation as well per the link above.

    Just so we can understand better, how often are you making changes to your
    images? Is this a box on which you are actively developing images to be
    used as containers going forward? If so, the BtrFS filesystem's CoW
    feature can be really nice for tracking changes, but it can also be a bit
    expensive for the same reason.

    Something else I recall from SUSECon last year, which I had not fully
    grasped yet, was that every command executed in a Dockerfile contributes
    to the final size of the image because, like git, it treats every command
    in a single Dockerfile as something to be remembered going forward, so
    chaining commands (with '&&') so make one big command, including commands
    to clean up the image filesystem (cached package files, etc.) can help
    reduce the final image size by having everything treated as a single
    commit instead of multiple commits. I doubt this is very related what you
    were seeing, but figured I would mention it in case it helps generally.

    --
    Good luck.

    If you find this post helpful and are logged into the web interface,
    show your appreciation and click on the star below...

  3. #3

    Re: "no space left on device" with btrfs setup - hog-hunting

    ab,

    thanks a lot for your numerous helpful hints!

    As I wrote, the setup is currently still default, so /var/lib/docker
    is on the same filesystem as / or /usr. You are referring to the
    follogin recommendation I guess:

    It is recommended to have /var/lib/docker mounted on a different file system to not affect the Docker host operating system in case of a file system corruption.
    I will do this when I can take down docker with all its containers for maintenance.

    "btrfs fi usage /" has some more info, but it will output the same result, regardless of where I point to in the FS, e.g.

    Code:
    host:~ # btrfs fi usage /usr
    Overall:
        Device size:		  40.00GiB
        Device allocated:		  40.00GiB
        Device unallocated:		   1.00MiB
        Device missing:		     0.00B
        Used:			  29.49GiB
        Free (estimated):		   8.23GiB	(min: 8.23GiB)
        Data ratio:			      1.00
        Metadata ratio:		      2.00
        Global reserve:		 512.00MiB	(used: 0.00B)
    
    Data,single: Size:29.10GiB, Used:20.87GiB
       /dev/sda2	  29.10GiB
    
    Metadata,DUP: Size:5.42GiB, Used:4.31GiB
       /dev/sda2	  10.84GiB
    
    System,DUP: Size:32.00MiB, Used:16.00KiB
       /dev/sda2	  64.00MiB
    
    Unallocated:
       /dev/sda2	   1.00MiB
    host:~ # btrfs fi usage /var/lib/docker
    Overall:
        Device size:		  40.00GiB
        Device allocated:		  40.00GiB
        Device unallocated:		   1.00MiB
        Device missing:		     0.00B
        Used:			  29.49GiB
        Free (estimated):		   8.23GiB	(min: 8.23GiB)
        Data ratio:			      1.00
        Metadata ratio:		      2.00
        Global reserve:		 512.00MiB	(used: 0.00B)
    
    Data,single: Size:29.10GiB, Used:20.87GiB
       /dev/sda2	  29.10GiB
    
    Metadata,DUP: Size:5.42GiB, Used:4.31GiB
       /dev/sda2	  10.84GiB
    
    System,DUP: Size:32.00MiB, Used:16.00KiB
       /dev/sda2	  64.00MiB
    
    Unallocated:
       /dev/sda2	   1.00MiB
    host:~ #
    I will adjust the retention of snapper snapshots, thanks for pointing
    to this, too.

    Our box is not heavily doing changes to images, so I daresay that the
    overhead of committing those changes is not really relevant to our
    situation.

    Thanks again,
    Tom

  4. #4

    Re: "no space left on device" with btrfs setup - hog-hunting

    hm, I did some looking around, but there seems to be no easy way to relocate btrfs subvolumes?
    Should I do a chain of btrfs send / receive commands?
    A simple mv would probably blow up the file system usage, as each pooled file is stored separately.

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •