Full LVM - hope you can help

[stefan_1304]

Hello,

I have big problem with lvm.
When I do "df -ah"


df: `/space': No such file or directory
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/system-root_lv 22G 22G 0 100% /
proc 0 0 0 - /proc
sysfs 0 0 0 - /sys
debugfs 0 0 0 - /sys/kernel/debug
udev 64G 164K 64G 1% /dev
tmpfs 190G 980K 190G 1% /dev/shm
devpts 0 0 0 - /dev/pts
/dev/sda1 152M 48M 96M 34% /boot
/dev/sda5 148G 119G 22G 85% /hana/shared
/dev/sda7 378G 195M 359G 1% /hana/data/HNA
/dev/sda8 130G 188M 124G 1% /hana/log/HNA
/dev/sda6 50G 15G 33G 31% /usr/sap
fusectl 0 0 0 - /sys/fs/fuse/connections
securityfs 0 0 0 - /sys/kernel/security
none 0 0 0 - /proc/sys/fs/binfmt_misc
none 0 0 0 - /var/lib/ntp/proc
gvfs-fuse-daemon 0 0 0 - /root/.gvfs


The /space partition I deleted. I do not wanted, but there where no important data. Thias partition had about 700 GB,. Where can I see the free space. I cannot open partitionen GUI on desktop because there is no space left.
What can I do? Several things I tried went wrong. Can I only change LVM size with live cd?

Hope this can help you?



hna:~ # lvextend -L 25G /dev/mapper/system-root_lv
Extending logical volume root_lv to 25.00 GiB
Insufficient free space: 769 extents needed, but only 2 available
hna:~ # umount /space
error writing /etc/mtab.tmp: No space left on device
hna:~ # lvextend -L 25G /dev/mapper/system-root_lv
/etc/lvm/archive/.lvm_hna_9989_559749969: write error failed: No space left on device
Volume group "system" metadata archive failed.





hna:~ # lvscan
ACTIVE '/dev/system/root_lv' [22.00 GiB] inherit
ACTIVE '/dev/system/swap_lv' [128.00 GiB] inherit
hna:~ # pvscan
PV /dev/sda2 VG system lvm2 [150.00 GiB / 8.00 MiB free]
Total: 1 [150.00 GiB] / in use: 1 [150.00 GiB] / in no VG: 0 [0 ]
hna:~ # vgscan
Reading all physical volumes. This may take a while...
Found volume group "system" using metadata type lvm2
hna:~ #
hna:~ #
hna:~ # lvreduce -L -2G /dev/system/swap_lv
WARNING: Reducing active and open logical volume to 126.00 GiB
THIS MAY DESTROY YOUR DATA (filesystem etc.)
Do you really want to reduce swap_lv? [y/n]: y
/etc/lvm/archive/.lvm_hna_10977_1150586696: write error failed: No space left on device
Volume group "system" metadata archive failed.
hna:~ # lvextend -L +2G /dev/system/root_lv
/etc/lvm/archive/.lvm_hna_10979_146382498: write error failed: No space left on device
Volume group "system" metadata archive failed.



I treid this, but it did not help
http://linuxblackbook.blogspot.de/2013/09/volume-group-vgname-metadata-archive.html

hope youc an help. thank you

[ab]

It looks like something about being out of space in / is causing your
problems with everything else. Find data that can be deleted from the
root filesystem (/) and delete it, for example under /tmp or /home in your
case. Once done, see if things behave better.
--
Good luck.

If you find this post helpful and are logged into the web interface,
show your appreciation and click on the star below...

[jmozdzen]

Hi stefan_1304,

best bet would be to handle this offline - and I mean your server system, not the discussion :D

So if you can, shut down the server and use some rescue boot of Linux (openSUSE via USB stick will work fine). Then all LVM operations will work, despite the full root FS (which then is no root fs, but just "some" FS :) ).

As long as you have free extends in your VG, you then can simply "lvextend -L +xG /dev/system/root_lv" and "resize2fs /dev/system/root_lv" (assuming this is an Ext3 root fs).

The blackbook link you reference just handles the case that the lvm command fails because of a read-only target area... in your case, there simply isn't enough free disk space. I've never tried "lvextend --force" under that condition... and you shouldn't do so either, unsless you have a good backup.

But you might want to give the *FS* resize a try anyhow, in case the *LV* resize worked (despite the error message) and just the follow-up action (like writing a new backup of the current LVM metadata) failed.

Another word of advice: split off at least /var into a separate LV/FS, instead of simply increasing "/". As you now have noticed, it's crucial to have some "spare room" in rootfs. Don't let increasing log files in /var/log or similar issues (like some application writing big or even endless amounts of data in /var/run) wreck your system integrity.

Regards,
Jens