PDA

View Full Version : Unable to Create VM



carnold6
25-Feb-2015, 23:44
SLES12 and latest updates. Installed Xen and Xen tools. Reboot and select the SLES12 with Xen option. Click on create VM in yast and select CDROM media install. Made 2048GB RAM with 108GB drive. Used br0 for the bridge. Click finish and got this error:

Unable to complete install: 'internal error: libxenlight failed to create new domain '2012r2lab''

Traceback (most recent call last):
File "/usr/share/virt-manager/virtManager/asyncjob.py", line 91, in cb_wrapper
callback(asyncjob, *args, **kwargs)
File "/usr/share/virt-manager/virtManager/create.py", line 1908, in do_install
guest.start_install(meter=meter)
File "/usr/share/virt-manager/virtinst/guest.py", line 426, in start_install
noboot)
File "/usr/share/virt-manager/virtinst/guest.py", line 490, in _create_guest
dom = self.conn.createLinux(start_xml or final_xml, 0)
File "/usr/lib64/python2.7/site-packages/libvirt.py", line 3297, in createLinux
if ret is None:raise libvirtError('virDomainCreateLinux() failed', conn=self)
libvirtError: internal error: libxenlight failed to create new domain '2012r2lab'

Oh, the guest is 2012R2 fullyvirtualized and the 2012R2 disc is in the drive. Any ideas?

jmozdzen
26-Feb-2015, 11:51
Hi carnold6,

> Any ideas?

what's in the related Xen logs? I'd assume you'll find there the actual error that occurred during VM creation.

Regards,
Jens

mhuels
26-Feb-2015, 13:15
same problem on my SLES 12 System. The VMs worked fine until the last update.

server:/var/log/libvirt/libxl # tail vm.domain.de.log
libxl: debug: libxl_create.c:1376:do_domain_create: ao 0x7f10580037c0: inprogress: poller=0x7f10780ec270, flags=ic
libxl: debug: libxl_event.c:1563:libxl__ao__destroy: ao 0x7f10580037c0: destroy
libxl: debug: libxl_create.c:1362:do_domain_create: ao 0x7f106c004c20: create: how=(nil) callback=(nil) poller=0x7f10780ec270
xc: error: xc_alloc_hypercall_buffer: mmap failed (11 = Resource temporarily unavailabl): Internal error

i cant see any ressource problem on my server.

mfg
Manfred

carnold6
26-Feb-2015, 13:18
Hi carnold6,

> Any ideas?

what's in the related Xen logs? I'd assume you'll find there the actual error that occurred during VM creation.

Not even sure where the corresponding log file is since this is the first time i have every used Xen. I did get a VM created and 2012r2 installed by rerunning the yast applet, install hypervisor and tools. The first time i installed Xen and xen tools only. The 2nd time i installed kvm and tools and then i was able to create the VM. What do you need to install in order to get a VM working? Xen and KVM or just one of those?

jmozdzen
26-Feb-2015, 13:27
Hi carnold6,

> Not even sure where the corresponding log file

my best guess would be the /var/log/xen directory, with individual log files per VM (only created when the ComU start was successful), xend.log for the management daemon and probably a separate log file for DomU configuration.

> What do you need to install in order to get a VM working? Xen and KVM or just one of those?

On SLES11 I know you'll only need Xen installed, not KVM. For SLES12, I have no practical experience, so I cannot tell if that has changed... but I'd call any requirement to manually add "kvm" to the list (in order to be able to create Xen DomUs) a dependency problem worth reporting ;)

Regards,
Jens

PS: Judging from Manfreds reply, /var/log/libvirt seems to be an additional / alternate location for related log files.

jmozdzen
26-Feb-2015, 13:47
Hi Manfred,

same problem on my SLES 12 System. The VMs worked fine until the last update.

server:/var/log/libvirt/libxl # tail vm.domain.de.log
libxl: debug: libxl_create.c:1376:do_domain_create: ao 0x7f10580037c0: inprogress: poller=0x7f10780ec270, flags=ic
libxl: debug: libxl_event.c:1563:libxl__ao__destroy: ao 0x7f10580037c0: destroy
libxl: debug: libxl_create.c:1362:do_domain_create: ao 0x7f106c004c20: create: how=(nil) callback=(nil) poller=0x7f10780ec270
xc: error: xc_alloc_hypercall_buffer: mmap failed (11 = Resource temporarily unavailabl): Internal error

i cant see any ressource problem on my server.

mfg
Manfred

"11" is EAGAIN, which by looking at mmap's man page ought to say "EAGAIN The file has been locked, or too much memory has been locked (see setrlimit(2))."

Does your invoking shell have too restricted ulimits ("ulimit -l")?

Regards,
Jens

mhuels
26-Feb-2015, 14:48
server:/var/log/libvirt/libxl # ulimit -l
64

i guess, it could be a bug like
http://lists.opensuse.org/opensuse-bugs/2014-04/msg00259.html
because the SLES 12 default installation takes btrfs on all partitions

/dev/sda5 on /var/tmp type btrfs (rw,relatime,space_cache)
/dev/sda5 on /var/spool type btrfs (rw,relatime,space_cache)
/dev/sda5 on /var/log type btrfs (rw,relatime,space_cache)
/dev/sda5 on /tmp type btrfs (rw,relatime,space_cache)
/dev/sda5 on /boot/grub2/x86_64-efi type btrfs (rw,relatime,space_cache)
/dev/sda5 on /boot/grub2/i386-pc type btrfs (rw,relatime,space_cache)


mfg
Manfred

jmozdzen
26-Feb-2015, 15:14
Hi Manfred,

server:/var/log/libvirt/libxl # ulimit -l
64

i guess, it could be a bug like
http://lists.opensuse.org/opensuse-bugs/2014-04/msg00259.html
because the SLES 12 default installation takes btrfs on all partitions

to me it doesn't look like the same thing.

In your case, the lib is complaining about not having access to sufficient shared memory space. Have you tried setting the limit to "unlimited" or to a much higher value than it currently is? What memory size have you specified for the DomU to be started?

Regards,
Jens

mhuels
26-Feb-2015, 17:07
Hi Jens
the problem dont care about Memory of the DomU. My VM, which works until last SLES update, has 4 GByte, but if i try to make a new VM with 1 GByte, the error remains the same.
I took a look to the configs of libvirt and found no entry to change ulimit for this process.
My server seems to have a lot of shared memory space

server:~ # ipcs -lm
------ Shared Memory Limits --------
max number of segments = 4096
max seg size (kbytes) = 18014398509481983
max total shared memory (kbytes) = 18014398509480960
min seg size (bytes) = 1

The server has no other function than to load the VMs.
I found no entry to increase the limit shown by ulimit -l

i tried changing
LIBVIRTD_NOFILES_LIMIT="8192"
in
server:/# cat /etc/sysconfig/libvirtd
but that had no effect.
mfg
Manfred

jmozdzen
26-Feb-2015, 17:55
Hi Manfred,

let me start by telling that I have not yet had to do this via SLES12.

> the problem dont care about Memory of the DomU.

While I can perfectly follow you, especially if one takes into consideration that the 64k ulimit for locked memory seems to be fairly default and the Internet isn't flooded with such messages, I had a look at the source code patch that introduced this message (it's from April 2014) and it suggests that you indeed stumbled over such a situation.

Have you opened a service request with SUSE, so that a product engineer can voice a qualified opinion on this?

> I found no entry to increase the limit shown by ulimit -l

As most likely the limit set for "libvirtd" applies, look up the pid of that process and dump the content of /proc/<pid>/limits... that way you can see what limits actually apply to that process. "ulimit -l" is just for the current session, not system-wide (as are the other limits adjustable via "ulimit").

Looking at /etc/init.d/libvirtd on an opensuse machine (I don't have a SLES12 at hand), I can see that during "start" handling the ulimit is already adjusted for the NOFILES limit... you could adopt that to set the locked memory limit, too.

> LIBVIRTD_NOFILES_LIMIT="8192"

Judging by the name of the variable, it will only influence the "number of open files" limit. How it's handled can be seen in /etc/init.d/libvirtd, at least on on an opensuse machine... my guess is that it's the same for SLES12 ATM.

Regards,
Jens

mhuels
27-Feb-2015, 11:15
Hi Manfred,

As most likely the limit set for "libvirtd" applies, look up the pid of that process and dump the content of /proc/<pid>/limits... that way you can see what limits actually apply to that process. "ulimit -l" is just for the current session, not system-wide (as are the other limits adjustable via "ulimit").



Ok, there is a hard limit of 64k for "max locked memory"

server:/var/tmp # cat /proc/2118/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 187199 187199 processes
Max open files 1024 4096 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 187199 187199 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us

i dont actually know the method to increase this value, but i hope dr. google will help me.

The hard limit is differing to the values on my old SLES 11 Servers. Perhaps i was a bit too fast to adopt the brandnew SLES 12 system. ;-)

mfg
Manfred

jmozdzen
27-Feb-2015, 11:42
Hi Manfred,


i dont actually know the method to increase this value, but i hope dr. google will help me.

Does changing it in the libvirtd start script help? And for good debugging measure, you might want to add a successive call to "(date;ulimit -a; echo "") >> /tmp/libvirtd.startup.ulimits.log" to have a log of the actual limits during startup...

Best regards,
Jens

mhuels
27-Feb-2015, 12:43
ok. SLES 12 is using systemd ;-(

server:/etc/init.d # systemctl show libvirtd.service | grep MEM
LimitMEMLOCK=65536

server:/etc/init.d # systemctl set-property libvirtd.service LimitMEMLOCK=262144
Unknown assignment MEMLOCK=262144.

mhuels
27-Feb-2015, 12:54
it needs to edit the file himself with
server:/etc/init.d # cat /usr/lib/systemd/system/libvirtd.service | grep MEM
LimitMEMLOCK=262144

and then
server:/etc/init.d # systemctl kill libvirtd.service
server:/etc/init.d # systemctl daemon-reload
server:/etc/init.d # systemctl start libvirtd.service

server:/etc/init.d # systemctl show libvirtd.service | grep Limit
StartLimitInterval=10000000
StartLimitBurst=5
StartLimitAction=none
MemoryLimit=18446744073709551615
LimitCPU=18446744073709551615
LimitFSIZE=18446744073709551615
LimitDATA=18446744073709551615
LimitSTACK=18446744073709551615
LimitCORE=18446744073709551615
LimitRSS=18446744073709551615
LimitNOFILE=4096
LimitAS=18446744073709551615
LimitNPROC=187199
LimitMEMLOCK=262144
LimitLOCKS=18446744073709551615
LimitSIGPENDING=187199
LimitMSGQUEUE=819200
LimitNICE=0
LimitRTPRIO=0
LimitRTTIME=18446744073709551615


but sadly its no use. The error remains.

jmozdzen
27-Feb-2015, 12:59
Hi Manfred,

> LimitMEMLOCK=262144
> but sadly its no use. The error remains.

have you tried setting it to "unlimited" or to at least the size of your DomU? From what I can see, the value above is in "bytes" and not megabytes ;)

Regards

mhuels
27-Feb-2015, 13:15
server:/etc/init.d # cat /usr/lib/systemd/system/libvirtd.service
# NB we don't use socket activation. When libvirtd starts it will
# spawn any virtual machines registered for autostart. We want this
# to occur on every boot, regardless of whether any client connects
# to a socket. Thus socket activation doesn't have any benefit

[Unit]
Description=Virtualization daemon
Before=libvirt-guests.service
After=network.target
After=dbus.service
After=iscsid.service
After=apparmor.service
Wants=xencommons.service
After=xencommons.service
Documentation=man:libvirtd(8)
Documentation=http://libvirt.org

[Service]
Type=notify
EnvironmentFile=-/etc/sysconfig/libvirtd
ExecStart=/usr/sbin/libvirtd $LIBVIRTD_ARGS
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
Restart=on-failure
# Override the maximum number of opened files
#LimitNOFILE=2048
# manni
LimitMEMLOCK=infinity


[Install]
WantedBy=multi-user.target

server:/etc/init.d # systemctl daemon-reload
server:/etc/init.d # systemctl kill libvirtd.service
server:/etc/init.d # systemctl start libvirtd.service
server:/etc/init.d # systemctl show libvirtd.service | grep Limit
StartLimitInterval=10000000
StartLimitBurst=5
StartLimitAction=none
MemoryLimit=18446744073709551615
LimitCPU=18446744073709551615
LimitFSIZE=18446744073709551615
LimitDATA=18446744073709551615
LimitSTACK=18446744073709551615
LimitCORE=18446744073709551615
LimitRSS=18446744073709551615
LimitNOFILE=4096
LimitAS=18446744073709551615
LimitNPROC=187199
LimitMEMLOCK=18446744073709551615
LimitLOCKS=18446744073709551615
LimitSIGPENDING=187199
LimitMSGQUEUE=819200
LimitNICE=0
LimitRTPRIO=0
LimitRTTIME=18446744073709551615
server:/etc/init.d #

sadly no effect

mhuels
27-Feb-2015, 13:20
same with libvirt-guests.service

server:/etc/init.d # cat /usr/lib/systemd/system/libvirt-guests.service
[Unit]
Description=Suspend Active Libvirt Guests
After=network.target libvirtd.service time-sync.target
Documentation=man:libvirtd(8)
Documentation=http://libvirt.org

[Service]
EnvironmentFile=-/etc/sysconfig/libvirt-guests
# Hack just call traditional service until we factor
# out the code
ExecStart=/usr/lib64/libvirt/libvirt-guests.sh start
ExecStop=/usr/lib64/libvirt/libvirt-guests.sh stop
Type=oneshot
RemainAfterExit=yes
StandardOutput=journal+console
# manni
LimitMEMLOCK=infinity


[Install]
WantedBy=multi-user.target
server:/etc/init.d #

jmozdzen
27-Feb-2015, 13:53
Hi Manfred,

> sadly no effect

my recommendation is to open a service request - this is something that SUSE engineers ought to help you with, and will do much better than I can.

In case this turns out to be something simple (although I truly doubt it), please let us know...

Regards,
Jens

mhuels
27-Feb-2015, 17:07
unfortunately we only have a basic support on Suse/Novell. So there is no support. I will try some ideas next week and will report success (if there is any).
mfg
Manfred

carnold6
01-Mar-2015, 03:35
Hi carnold6,

> Not even sure where the corresponding log file

my best guess would be the /var/log/xen directory, with individual log files per VM (only created when the ComU start was successful), xend.log for the management daemon and probably a separate log file for DomU configuration.

> What do you need to install in order to get a VM working? Xen and KVM or just one of those?

On SLES11 I know you'll only need Xen installed, not KVM. For SLES12, I have no practical experience, so I cannot tell if that has changed... but I'd call any requirement to manually add "kvm" to the list (in order to be able to create Xen DomUs) a dependency problem worth reporting ;)

PS: Judging from Manfreds reply, /var/log/libvirt seems to be an additional / alternate location for related log files.

So, on a completely different SLES 12 install, the exact same thing happens when creating a VM. Here is the log:

libxl: debug: libxl_create.c:1362:do_domain_create: ao 0x7f91b001a0f0: create: how=(nil) callback=(nil) poller=0x7f91b0012b50
xc: error: xc_alloc_hypercall_buffer: mmap failed (11 = Resource temporarily unavailabl): Internal error
libxl: error: libxl.c:570:libxl_list_domain: geting domain info list: Resource temporarily unavailable
libxl: error: libxl.c:314:libxl__domain_rename: unexpected errorchecking for existing domain
libxl: error: libxl_create.c:762:initiate_domain_create: cannot make domain: -5
libxl: error: libxl_dm.c:1494:kill_device_model: unable to find device model pid in /local/domain/1/image/device-model-pid
libxl: error: libxl.c:1507:libxl__destroy_domid: libxl__destroy_device_model failed for 1
libxl: debug: libxl_event.c:1591:libxl__ao_complete: ao 0x7f91b001a0f0: complete, rc=-3
libxl: debug: libxl_create.c:1376:do_domain_create: ao 0x7f91b001a0f0: inprogress: poller=0x7f91b0012b50, flags=ic
libxl: debug: libxl_event.c:1563:libxl__ao__destroy: ao 0x7f91b001a0f0: destroy
xc: debug: hypercall buffer: total allocations:34 total releases:33
xc: debug: hypercall buffer: current allocations:1 maximum allocations:3
xc: debug: hypercall buffer: cache current size:2
xc: debug: hypercall buffer: cache hits:31 misses:2 toobig:1

This is only with Xen and Xen tools installed (no KVM)

carnold6
01-Mar-2015, 16:45
So, on a completely different SLES 12 install, the exact same thing happens when creating a VM. Here is the log:

libxl: debug: libxl_create.c:1362:do_domain_create: ao 0x7f91b001a0f0: create: how=(nil) callback=(nil) poller=0x7f91b0012b50
xc: error: xc_alloc_hypercall_buffer: mmap failed (11 = Resource temporarily unavailabl): Internal error
libxl: error: libxl.c:570:libxl_list_domain: geting domain info list: Resource temporarily unavailable
libxl: error: libxl.c:314:libxl__domain_rename: unexpected errorchecking for existing domain
libxl: error: libxl_create.c:762:initiate_domain_create: cannot make domain: -5
libxl: error: libxl_dm.c:1494:kill_device_model: unable to find device model pid in /local/domain/1/image/device-model-pid
libxl: error: libxl.c:1507:libxl__destroy_domid: libxl__destroy_device_model failed for 1
libxl: debug: libxl_event.c:1591:libxl__ao_complete: ao 0x7f91b001a0f0: complete, rc=-3
libxl: debug: libxl_create.c:1376:do_domain_create: ao 0x7f91b001a0f0: inprogress: poller=0x7f91b0012b50, flags=ic
libxl: debug: libxl_event.c:1563:libxl__ao__destroy: ao 0x7f91b001a0f0: destroy
xc: debug: hypercall buffer: total allocations:34 total releases:33
xc: debug: hypercall buffer: current allocations:1 maximum allocations:3
xc: debug: hypercall buffer: cache current size:2
xc: debug: hypercall buffer: cache hits:31 misses:2 toobig:1

This is only with Xen and Xen tools installed (no KVM)

OK, not sure if this fixed it or not but i turned off apparmor and rebooted the server. Then i was able to create a VM

mhuels
02-Mar-2015, 12:03
the solution was to disable apparmor like carnold6 above ( unswitch yast -> apparmor config -> settings -> enable). Apparmor is started by default in SLES 12. It is possible to disable apparmor only for libvirtd.

mfg
Manfred

carnold6
02-Mar-2015, 15:03
the solution was to disable apparmor like carnold6 above ( unswitch yast -> apparmor config -> settings -> enable). Apparmor is started by default in SLES 12. It is possible to disable apparmor only for libvirtd.

mfg
Manfred

Hi manfred,
I have read on the internet where you can whitelist the libvirtd service in the apparmor profile but i just stopped it by disabling and then using:

sudo systemctl stop app.armor.service

on SLES12

jmozdzen
04-Mar-2015, 17:59
Hi carnold6,

> OK, not sure if this fixed it or not but i turned off apparmor and rebooted the server. Then i was able to create a VM

with Manfred reporting this as a work-around for him, too - this makes it look like the "wall" that's hit by mmap() is not with ulimtis (as suspected by me), but rather AppArmor.

Good find, and thank you for sharing the result!

Regards,
Jens