PDA

View Full Version : Changing Block device to Iscsi ?



stelgenkamp
07-Dec-2013, 20:56
So this server has been running for more then a year now, and I'm finding the throughput still slow.
We have SLES Dom0, with OES11 running as domu
Dom0 has a separate RAID, which is presented to the OES11 as Xen Block device.
On this block device the OES11 domu has created NSS volumes
When doing a server reboot the OES11, is taking forever to load the block device ( 5 minutes )

So I am thinking : what about Iscsi ?
So my question to you all is :
Can I turn the block device into a Isci share ?
Will the OES11 keep it's NSS volumes ?
How is this best done ?

Thanks, Stephan

KBOYLE
07-Dec-2013, 21:40
stelgenkamp wrote:

>
> So this server has been running for more then a year now, and I'm
> finding the throughput still slow.
> We have SLES Dom0, with OES11 running as domu
> Dom0 has a separate RAID, which is presented to the OES11 as Xen Block
> device.
> On this block device the OES11 domu has created NSS volumes

All of this seems quite normal

> When doing a server reboot the OES11, is taking forever to load the
> block device ( 5 minutes )

This is the issue that needs to be investigated.
>
> So I am thinking : what about Iscsi ?
> So my question to you all is :
> Can I turn the block device into a Isci share ?
> Will the OES11 keep it's NSS volumes ?
> How is this best done ?

You should not make random changes to the system with the hope the
problem will be corrected.


The first step is to diagnose the cause:

1. We need to know what software versions you are running. On both Dom0
and DomU please run "cat /etc/*release". Copy the output and paste it
into your reply using <code tags>.

<code tags>: When replying, click the "Go Advanced" button then click
on the "#" at the top of the reply window.

2. Look for any clues in /var/log/messages on both Dom0 and DomU.
Report any interesting messages the same way.



--
Kevin Boyle - Knowledge Partner
If you find this post helpful and are logged into the web interface,
show your appreciation and click on the star below...

stelgenkamp
12-Dec-2013, 10:50
Dear Kevin,

In /var/log/message, I could not find anything on this.

In startup log :
<6>[ 1.644885] xvdg: detected capacity change from 0 to 476999319552
<4>[ 6.565997] XENBUS: Waiting for devices to initialise: 295s...290s...285s...280s...275s...270s...265s...2 60s...255s...250s...245s...240s...235s...230s...22 5s...220s...215s...210s...205s...200s...195s...190 s...185s...180s...175s...170s...165s...160s...155s ...150s...145s...140s...135s...130s...125s...120s. ..115s...110s...105s...100s...95s...90s...85s...80 s...75s...70s...65s...60s...55s...50s...45s...40s. ..35s...30s...25s...20s...15s...10s...5s...0s...
<4>[ 301.567897] XENBUS: Timeout connecting to device: device/vbd/51840 (local state 3, remote state 2)
<4>[ 301.570968] XENBUS: Device not ready: device/vbd/51840

Version virtual OES :

OES server
LSB_VERSION="core-2.0-noarch:core-3.2-noarch:core-4.0-noarch:core-2.0-x86_64:core-3.2-x86_64:core-4.0-x86_64"
Novell Open Enterprise Server 11 (x86_64)
VERSION = 11
PATCHLEVEL = 0
SUSE Linux Enterprise Server 11 (x86_64)
VERSION = 11
PATCHLEVEL = 1

Version Xen Host :

LSB_VERSION="core-2.0-noarch:core-3.2-noarch:core-4.0-noarch:core-2.0-x86_64:core-3.2-x86_64:core-4.0-x86_64"
SUSE Linux Enterprise Server 11 (x86_64)
VERSION = 11
PATCHLEVEL = 2

KBOYLE
12-Dec-2013, 20:07
Hi Stephan,

Have you tried a Google search on "XENBUS: Waiting for devices to
initialise"? There are quite a few hits but I have no way to know if
one of them is your issue.

Current software releases are:
SLES11-SP3
OES11-SP1 on SLES11-SP2

They contain many bug fixes which may resolve your issue. Have you
considered upgrading to the current releases?

You said:
> So this server has been running for more then a year now, and I'm
> finding the throughput still slow.

Are you saying that this issue existed from the time your DomU was
initially configured?

If you running other DomUs, do any of them have similar issues?

Please run these commands on Dom0 and paste the output into your reply
using "code" tags.



fdisk -l
xm list
xm list -l <name of your OES11 DomU>




--
Kevin Boyle - Knowledge Partner
If you find this post helpful and are logged into the web interface,
show your appreciation and click on the star below...

stelgenkamp
15-Dec-2013, 20:03
Right, have updated the Xen Server to SP3

Only updating the OES server did give a bad side effect. :-(
So now the OES11 is booting without any issues,
but now the Gnome desktop looks screwed.
I cannot login. the login screen is shown, I login, it starts loading the desktop.
Somehow crashes and returns to the login screen.

jmozdzen
16-Dec-2013, 14:45
Hi Stephan,

> I cannot login. the login screen is shown, I login, it starts loading the desktop.
> Somehow crashes and returns to the login screen.

Have you had a look at the ~/.xsession-errors-* file(s), there might be some information on what program failed during desktop initialization.

Regards,
Jens

stelgenkamp
25-Dec-2013, 12:55
Christmas time ; so finally some off time to check out what happened.
./xsession-errors shows :


/etc/X11/xim: Checking whether an input method should be started.
/etc/X11/xim: use GDM_LANG=nl_NL.UTF-8
sourcing /etc/sysconfig/language to get the value of INPUT_METHOD
INPUT_METHOD is not set or empty (no user selected input method).
Trying to start a default input method for the locale nl_NL.UTF-8 ...
There is no default input method for the current locale.
Dummy input method "none" (do not use any fancy input method by default)
/usr/bin/gpg-agent: symbol lookup error: /lib64/libgcrypt.so.11: undefined symbol: gpg_err_set_errno

Have no idea how to fix an input method ...
Any help would be appreciated.
Stephan

stelgenkamp
25-Dec-2013, 14:15
Have found a TID on the subject
http://www.novell.com/support/kb/doc.php?id=7013143
Will try that tomorrow.

jmozdzen
25-Dec-2013, 16:06
Hi Stephan,

Have found a TID on the subject
http://www.novell.com/support/kb/doc.php?id=7013143
Will try that tomorrow.
not that I could have pointed you at that TID, but that error message caught my eye and made me more suspicious than the input device stuff. Looks to me like an upgrade problem, the TID is likely right on the spot.

Happy holidays,
Jens

stelgenkamp
27-Dec-2013, 19:20
OK, we're back.
Have solved the libcrypt 'challenge' and now are back in business.

However the : "XENBUS: Waiting for devices to initialise" 'challenge' has returned.


(domain
(domid 28)
(cpu_weight 256)
(cpu_cap 0)
(pool_name Pool-0)
(bootloader /usr/lib/xen/boot/domUloader.py)
(vcpus 4)
(cpus (() () () ()))
(on_poweroff destroy)
(description None)
(on_crash destroy)
(uuid ecf1c574-5b7b-d898-8415-9122fc4f513e)
(bootloader_args '--entry=xvda1:/boot/vmlinuz-xen,/boot/initrd-xen')
(name oes2l)
(on_reboot restart)
(maxmem 3072)
(memory 3072)
(shadow_memory 0)
(features '')
(on_xend_start ignore)
(on_xend_stop ignore)
(start_time 1388090521.29)
(cpu_time 24261.1702008)
(online_vcpus 4)
(image
(linux
(kernel '')
(args ' ')
(superpages 0)
(pci ())
(localtime 0)
(nomigrate 0)
(tsc_mode 0)
(device_model /usr/lib/xen/bin/qemu-dm)
(notes
(FEATURES
'writable_page_tables|writable_descriptor_tables|a uto_translated_physmap|supervisor_mode_kernel'
)
(VIRT_BASE 18446744071562067968)
(GUEST_VERSION 2.6)
(PADDR_OFFSET 0)
(GUEST_OS linux)
(HYPERCALL_PAGE 18446744071562080256)
(LOADER generic)
(INIT_P2M 18446719884453740544)
(SUSPEND_CANCEL 1)
(ENTRY 18446744071562076160)
(XEN_VERSION xen-3.0)
(MOD_START_PFN 1)
(SUPPORTED_FEATURES 2063)
)
)
)
(status 2)
(state -b----)
(store_mfn 8388923)
(console_mfn 8388922)
(device
(vif
(mac 00:16:3e:10:09:17)
(script /etc/xen/scripts/vif-bridge)
(uuid 340c40d7-ca68-9ce8-89be-34b6695c1dab)
(backend 0)
)
)
(device (vkbd (backend 0)))
(device
(tap
(protocol x86_64-abi)
(uuid 4f394d94-23af-09ff-5ca7-7437cb09b193)
(bootable 0)
(dev xvdh:cdrom)
(uname
tap:aio:/root/Installatie/OES11-SP1-addon_with_SLES11-SP2-x86_64-DVD.iso
)
(mode r)
(backend 0)
(VDI '')
)
)
(device
(console
(protocol vt100)
(location 2)
(uuid d3773870-144f-a7ff-c443-b77012a5b695)
)
)
(device
(vbd
(protocol x86_64-abi)
(uuid 934b82e4-848b-067b-05f6-1997c848c332)
(bootable 1)
(dev xvda:disk)
(uname phy:/dev/Linux/boot)
(mode w)
(backend 0)
(VDI '')
)
)
(device
(vbd
(protocol x86_64-abi)
(uuid aa14a6f0-9a1f-f172-fead-976ad410742f)
(bootable 0)
(dev xvdb:disk)
(uname phy:/dev/Linux/root)
(mode w)
(backend 0)
(VDI '')
)
)
(device
(vbd
(protocol x86_64-abi)
(uuid 001faa38-93c5-c952-ce20-f821c190823f)
(bootable 0)
(dev xvdc:disk)
(uname phy:/dev/Linux/swap)
(mode w)
(backend 0)
(VDI '')
)
)
(device
(vbd
(protocol x86_64-abi)
(uuid 9d5c94c5-8538-f6a3-f3b9-1db33c79a9e8)
(bootable 0)
(dev xvdd:disk)
(uname phy:/dev/Linux/opt)
(mode w)
(backend 0)
(VDI '')
)
)
(device
(vbd
(protocol x86_64-abi)
(uuid af5a5d8c-a878-49f8-ee36-63a74c781f07)
(bootable 0)
(dev xvde:disk)
(uname phy:/dev/Linux/home)
(mode w)
(backend 0)
(VDI '')
)
)
(device
(vbd
(protocol x86_64-abi)
(uuid 37784803-f92b-4880-3669-7e42ca00dfe8)
(bootable 0)
(dev xvdf:disk)
(uname phy:/dev/Linux/var)
(mode w)
(backend 0)
(VDI '')
)
)
(device
(vbd
(protocol x86_64-abi)
(uuid ff66c73f-8c4f-1c37-8b66-cddce5f5cb51)
(bootable 0)
(dev xvdg:disk)
(uname phy:/dev/sdb)
(mode w)
(backend 0)
(VDI '')
)
)
(device
(vbd
(protocol x86_64-abi)
(uuid eacb8f4b-ad49-b844-1d38-6a03efb98443)
(bootable 0)
(dev xvdi:cdrom)
(uname phy:/dev/sr0)
(mode r)
(backend 0)
(VDI '')
)
)
(device
(vfb
(vncunused 1)
(keymap en-us)
(vnc 1)
(uuid 73f26216-6efa-2bad-741c-ce83b198fb43)
(location 127.0.0.1:5902)
)
)
)


The device which is showing the XENBUS wait is : phy:/dev/sdb
This is a separate raid-5 on my xen host.
Any ideas ?

jmozdzen
28-Dec-2013, 00:39
Hi Stephan,

> However the : "XENBUS: Waiting for devices to initialise" 'challenge' has returned.

I assume this message appears inside DomU during boot and you have no access to /dev/xvdg from the DomU afterwards.

> Any ideas ?

Is there anything helpful in the DomU log (on Dom0: /var/log/xen/qemu-dm-oes2l.log) or xend's log (on Dom0: /var/log/xen/xend.log - search for the debug statements for your DomU's creation: "XendDomainInfo.create(['vm', ['name', 'oes2l'],...")?

Regards,
Jens

stelgenkamp
28-Dec-2013, 13:01
Hello Jens,

Yes the XENBUS: Waiting for devices to initialize, message appears on the Domu.
I do have access to the device afterwards, it just takes the server 5 extra minutes to boot
On the device are my nss volumes, and this is my 'main' server.
So all other servers, and virtual WS are logging into this server. However when this server takes 5 minutes to boot, the automatic logons I have created fail.

I have sifted through the logs, nothing in /var/log/xen/qemu-dm-oes2l.log
In /var/log/xen/xend.log :

[2013-12-26 21:40:30 11914] DEBUG (DevController:95) DevController: writing {'virtual-device': '51792', 'device-type': 'disk', 'protocol': 'x86_64-abi', 'backend-id': '0', 'state': '1', 'backend': '/local/domain/0/backend/vbd/27/51792'} to /local/domain/27/device/vbd/51792.
[2013-12-26 21:40:30 11914] DEBUG (DevController:97) DevController: writing {'domain': 'oes2l', 'frontend': '/local/domain/27/device/vbd/51792', 'uuid': '37784803-f92b-4880-3669-7e42ca00dfe8', 'bootable': '0', 'dev': 'xvdf', 'state': '1', 'params': '/dev/Linux/var', 'mode': 'w', 'online': '1', 'frontend-id': '27', 'type': 'phy'} to /local/domain/0/backend/vbd/27/51792.
[2013-12-26 21:40:30 11914] INFO (XendDomainInfo:2469) createDevice: vbd : {'uuid': 'ff66c73f-8c4f-1c37-8b66-cddce5f5cb51', 'bootable': 0, 'devid': 51808, 'driver': 'paravirtualised', 'dev': 'xvdg', 'uname': 'phy:/dev/sdb', 'mode': 'w'}
So no mention whatsoever concerning the XENBUS wait.
Anywhere else I can look ?
Stephan

jmozdzen
28-Dec-2013, 14:40
Hi Stephan,

while going through this thread I noticed that you initially quoted devid 51840 to fail; somewhere along you said it's /dev/sdb - but your latest quote says that /dev/sdb (Dom0) is 51808.

Did the device instance, named in the DomU error message (" XENBUS: Device not ready: device/vbd/51840") change after upgrade?

Would you mind quoting the complete message sequence from xend.log when creating the DomU, plus the current DomU messages concerning the delay (and anything that appears in DomU's "dmesg" around that delay)?

Regards,
Jens

Rickb
28-Dec-2013, 14:45
stelgenkamp wrote:

>
> OK, we're back.
> Have solved the libcrypt 'challenge' and now are back in business.
>
> However the : "XENBUS: Waiting for devices to initialise" 'challenge'
> has returned.
>
>
> Code:
> --------------------
> (domain
> (domid 28)
> (cpu_weight 256)
> (cpu_cap 0)
> (pool_name Pool-0)
> (bootloader /usr/lib/xen/boot/domUloader.py)
> (vcpus 4)
> (cpus (() () () ()))
> (on_poweroff destroy)
> (description None)
> (on_crash destroy)
> (uuid ecf1c574-5b7b-d898-8415-9122fc4f513e)
> (bootloader_args '--entry=xvda1:/boot/vmlinuz-xen,/boot/initrd-xen')
> (name oes2l)
> (on_reboot restart)
> (maxmem 3072)
> (memory 3072)
> (shadow_memory 0)
> (features '')
> (on_xend_start ignore)
> (on_xend_stop ignore)
> (start_time 1388090521.29)
> (cpu_time 24261.1702008)
> (online_vcpus 4)
> (image
> (linux
> (kernel '')
> (args ' ')
> (superpages 0)
> (pci ())
> (localtime 0)
> (nomigrate 0)
> (tsc_mode 0)
> (device_model /usr/lib/xen/bin/qemu-dm)
> (notes
> (FEATURES
> 'writable_page_tables|writable_descriptor_tables|
auto_translated_physmap|supervisor_mode_kernel'
> )
> (VIRT_BASE 18446744071562067968)
> (GUEST_VERSION 2.6)
> (PADDR_OFFSET 0)
> (GUEST_OS linux)
> (HYPERCALL_PAGE 18446744071562080256)
> (LOADER generic)
> (INIT_P2M 18446719884453740544)
> (SUSPEND_CANCEL 1)
> (ENTRY 18446744071562076160)
> (XEN_VERSION xen-3.0)
> (MOD_START_PFN 1)
> (SUPPORTED_FEATURES 2063)
> )
> )
> )
> (status 2)
> (state -b----)
> (store_mfn 8388923)
> (console_mfn 8388922)
> (device
> (vif
> (mac 00:16:3e:10:09:17)
> (script /etc/xen/scripts/vif-bridge)
> (uuid 340c40d7-ca68-9ce8-89be-34b6695c1dab)
> (backend 0)
> )
> )
> (device (vkbd (backend 0)))
> (device
> (tap
> (protocol x86_64-abi)
> (uuid 4f394d94-23af-09ff-5ca7-7437cb09b193)
> (bootable 0)
> (dev xvdh:cdrom)
> (uname
> tap:aio:/root/Installatie/OES11-SP1-addon_with_SLES11-SP2-x86_64-DVD.iso
> )
> (mode r)
> (backend 0)
> (VDI '')
> )
> )
> (device
> (console
> (protocol vt100)
> (location 2)
> (uuid d3773870-144f-a7ff-c443-b77012a5b695)
> )
> )
> (device
> (vbd
> (protocol x86_64-abi)
> (uuid 934b82e4-848b-067b-05f6-1997c848c332)
> (bootable 1)
> (dev xvda:disk)
> (uname phy:/dev/Linux/boot)
> (mode w)
> (backend 0)
> (VDI '')
> )
> )
> (device
> (vbd
> (protocol x86_64-abi)
> (uuid aa14a6f0-9a1f-f172-fead-976ad410742f)
> (bootable 0)
> (dev xvdb:disk)
> (uname phy:/dev/Linux/root)
> (mode w)
> (backend 0)
> (VDI '')
> )
> )
> (device
> (vbd
> (protocol x86_64-abi)
> (uuid 001faa38-93c5-c952-ce20-f821c190823f)
> (bootable 0)
> (dev xvdc:disk)
> (uname phy:/dev/Linux/swap)
> (mode w)
> (backend 0)
> (VDI '')
> )
> )
> (device
> (vbd
> (protocol x86_64-abi)
> (uuid 9d5c94c5-8538-f6a3-f3b9-1db33c79a9e8)
> (bootable 0)
> (dev xvdd:disk)
> (uname phy:/dev/Linux/opt)
> (mode w)
> (backend 0)
> (VDI '')
> )
> )
> (device
> (vbd
> (protocol x86_64-abi)
> (uuid af5a5d8c-a878-49f8-ee36-63a74c781f07)
> (bootable 0)
> (dev xvde:disk)
> (uname phy:/dev/Linux/home)
> (mode w)
> (backend 0)
> (VDI '')
> )
> )
> (device
> (vbd
> (protocol x86_64-abi)
> (uuid 37784803-f92b-4880-3669-7e42ca00dfe8)
> (bootable 0)
> (dev xvdf:disk)
> (uname phy:/dev/Linux/var)
> (mode w)
> (backend 0)
> (VDI '')
> )
> )
> (device
> (vbd
> (protocol x86_64-abi)
> (uuid ff66c73f-8c4f-1c37-8b66-cddce5f5cb51)
> (bootable 0)
> (dev xvdg:disk)
> (uname phy:/dev/sdb)
> (mode w)
> (backend 0)
> (VDI '')
> )
> )
> (device
> (vbd
> (protocol x86_64-abi)
> (uuid eacb8f4b-ad49-b844-1d38-6a03efb98443)
> (bootable 0)
> (dev xvdi:cdrom)
> (uname phy:/dev/sr0)
> (mode r)
> (backend 0)
> (VDI '')
> )
> )
> (device
> (vfb
> (vncunused 1)
> (keymap en-us)
> (vnc 1)
> (uuid 73f26216-6efa-2bad-741c-ce83b198fb43)
> (location 127.0.0.1:5902)
> )
> )
> )
>
> --------------------
>
>
> The device which is showing the XENBUS wait is : phy:/dev/sdb
> This is a separate raid-5 on my xen host.
> Any ideas ?
>
>

I know I am popping in her late? May I ask is if phy:/dev/sdb is in fact an
iscsi device? Is this an ncs setup?

stelgenkamp
29-Dec-2013, 18:26
Hello Jens,
To check :

<6>[ 1.854068] xvdg: detected capacity change from 0 to 476999319552
<4>[ 6.728256] XENBUS: Waiting for devices to initialise: 295s...290s...285s.
..280s...275s...270s...265s...260s...255s...250s.. .245s...240s...235s...230s...2
25s...220s...215s...210s...205s...200s...195s...19 0s...185s...180s...175s...170s
...165s...160s...155s...150s...145s...140s...135s. ..130s...125s...120s...115s...
110s...105s...100s...95s...90s...85s...80s...75s.. .70s...65s...60s...55s...50s..
.45s...40s...35s...30s...25s...20s...15s...10s...5 s...0s...
<4>[ 301.735243] XENBUS: Timeout connecting to device: device/vbd/51824 (local
state 3, remote state 2)
<4>[ 301.745724] XENBUS: Device not ready: device/vbd/51824

So it looks like 51824, i will check the xend.log for more info.

To Rick, Thanks for joining in :
No this is a raid disc, just presented to domu, but has difficulty loading.
Therefore the titel of the thread, if it would be possible to change this block device into an ISCSI.

Stephan

stelgenkamp
29-Dec-2013, 21:26
Xend log shows :

[2013-12-28 16:48:40 6126] INFO (XendDomainInfo:2479) createDevice: vbd : {'uuid': '59077e20-41e7-8526-4859-25f56c6899d7', 'bootable': 0, 'driver': 'paravirtualised', 'dev': 'xvdg', 'uname': 'phy:/dev/sdb', 'mode': 'w'}
[2013-12-28 16:48:40 6126] DEBUG (DevController:95) DevController: writing {'virtual-device': '51808', 'device-type': 'disk', 'protocol': 'x86_64-abi', 'backend-id': '0', 'state': '1', 'backend': '/local/domain/0/backend/vbd/4/51808'} to /local/domain/4/device/vbd/51808.
[2013-12-28 16:48:40 6126] DEBUG (DevController:97) DevController: writing {'domain': 'oes2l', 'frontend': '/local/domain/4/device/vbd/51808', 'uuid': '59077e20-41e7-8526-4859-25f56c6899d7', 'bootable': '0', 'dev': 'xvdg', 'state': '1', 'params': '/dev/sdb', 'mode': 'w', 'online': '1', 'frontend-id': '4', 'type': 'phy'} to /local/domain/0/backend/vbd/4/51808.

So could that be the problem ? That it links to a different backend ?
Ok, so know how do I fix it ?

jmozdzen
29-Dec-2013, 22:12
Hi Stephan,

it does look strange - is that ID available somewhere in Dom0's config? Probably configured for this DomU? Or does that ID exist nowhere in Dom0?

It might be useful to post a collection of all log entries (xend, domu syslog/dmesg, actual DomU config) rather than excerpts.

Regards,
Jens

KBOYLE
29-Dec-2013, 22:34
stelgenkamp wrote:

> However the : "XENBUS: Waiting for devices to initialise" 'challenge'
> has returned.

I see you have a CDROM and the SLES11-SP2/OES11-SP1 DVD attached to
your DomU. Have you seen this?

TID 7014030 - XENBUS: Waiting for device to initialize: 295s...
http://www.novell.com/support/kb/doc.php?id=7014030


> Situation
> When a VM is started it hits a 5 minute delay at:
>
> XENBUS: Waiting for device to initialize: 295s...
>
> If the VM has Open Enterprise Server (OES) installed and running NSS,
> a corresponding entry in the message file may be:
>
> jstcpd: Jet Stream starting with Connections NoActivity Timeout to be
> 300 seconds Resolution
> Check the VM configuration to see if there is a virtual CDROM
> attached with no actual media available at the location specified.
> (path to iso image is wrong/unavailable or there is no DVD in the
> device on the HOST).
>
> Remove the virtual CDROM device from the VM.


--
Kevin Boyle - Knowledge Partner
If you find this post helpful and are logged into the web interface,
show your appreciation and click on the star below...

stelgenkamp
26-Jul-2014, 09:32
Old post, but indeed Kevin, removing tge cd rom fixes the problem